Group Actions
Engagement
  • Author Author: javagoza
  • Date Created: Date Created
  • Views 2733 views
  • Likes 12 likes
  • Comments 12 comments
Related
Recommended

Blog 4 - UrbanRest Guardian - Classifying Urban Sounds

javagoza
Blog 4 - UrbanRest Guardian - Classifying Urban Sounds

Introduction

This is the fourth blog of my participation in the Experimenting with Extreme Environments challenge sponsored by Hammond Manufacturing and Element14 community. It was going to be my fifth blog but an injury to my right hand prevents me from making efforts with that hand and I have changed the planned activities.

In this blog, I describe the techniques I've learned for sound classification, specifically methods for classifying urban noises into specific categories using deep learning. This includes noises like car horns, sirens, and other typical sounds heard on the streets of any city. Deep learning is a type of artificial intelligence that mimics the way the human brain works, using artificial neural networks to learn from data. 

During this experimentation period, my main goals include testing a pre-trained model on the UrbanRest Guardian device, which is built inside a Hammond Manufacturing Enclosure using the provided kit for the Experimenting with Extreme Environments design challenge. Additionally, I aim to learn how to generate and train a model using raw audio data.

Table of Contents

Enhancements to the Graphical Interface

With the idea of making the experiments more visual I've implemented several enhancements to the graphical interface of the UrbanRest Guardian device, expanding the options available to the user. 

The enhancements to the user interface include:

  • Displaying the classification score and relative decibel value on the first line.
  • Presenting a graphical representation of sound pressure level on the second line.
  • Showing the identified class by the sound classifier on the third line.
  • Providing real-time updates of NTP and CPU temperature on the fourth line.
  • Adding a 12 key capacitive keyboard behind the transparent polycarbonate lid of the Hammond enclosure.

image

The screen is now divided into four lines of information:

  1. The first line displays the score, representing the probability score of the class identified in the sound classification process of the last 0.96 seconds processed. Additionally, it shows a relative value of decibels, indicating the maximum level of sound pressure referenced to a value obtained in silence. It's worth noting that this calibration is heuristic rather than scientific, as I lack the tools for precise calibration.
  2. The second line features a horizontal bar graph illustrating the sound pressure level in decibels, calculated using the root mean square (rms) of the last 1024 measurements. Sampling is conducted at 16 kHz, sampled values are 16 bits length integers.
  3. The third line displays the description of the class identified by the sound classifier, determined by the convolutional neural network. It indicates the class with the highest probability score within the last 0.96 seconds.
  4. The fourth line displays real-time updates obtained via NTP over the Wi-Fi connection, along with the CPU temperature of the Raspberry Pi4 compute module.

Me and Artificial Intelligence

In my academic years during the 80s and 90s, I delved into various facets of artificial intelligence (AI). One intriguing venture involved exploring fuzzy logic, a type of reasoning that handles approximate decisions rather than strict yes or no choices. Specifically, I applied fuzzy logic to enhance control systems for guiding vehicles in an automobile factory, where precise decisions weren't always feasible.

Another captivating project, sponsored by IBM, centered on crafting an expert system. These systems mimic human experts' decision-making skills within a specific field. My focus was on aiding lawyers with civil works claims by employing natural language processing to sift through legal documents and highlight potential breaches of contract. I utilized languages like Lisp, Prolog, and Crystal to build this rule-based system.

My journey into neural networks began with a research project involving predictions in a coal-fired power plant. Neural networks, adept at modeling complex behaviors, helped forecast power output based on factors like coal grinding levels and historical data.

Transitioning to my professional career, I ventured into developing an automated inventory replenishment system. Here, Bayesian inference played a pivotal role. This statistical method updates probabilities as new evidence emerges, aiding in predicting future sales based on past data and adjusting forecasts as sales evolve.

Throughout my career, statistical learning has remained a passion. I've recently revisited one of my favorite books on the subject: The Elements of Statistical Learning: Data Mining, Inference, and Prediction

imageimage

Sound Feature Extraction and Classification

Working with TensorFlow and Keras

For building the classification model, I've chosen to use TensorFlow with Keras.  TensorFlow is an open-source machine learning framework developed by Google. TensorFlow can be used for various tasks such as classification, regression, clustering, and more. It allow us to define and train deep neural networks. Keras is the high-level API of the TensorFlow platform.

image  image

The TensorFlow development cycle begins with data preparation, followed by model design, where the neural network architecture is defined along with activation functions and the loss function chosen. Next, the model is trained using training data and evaluated for performance using test data. Hyperparameters, like learning rate, batch size, number of layers, activation functions.. are adjusted and the model is optimized as needed. Then we can deploy the model in real-world scenarios. 

I've been working on Windows using Anaconda and JupyterLab notebooks. 

YAMnet, short for Yet Another Multilabel Network, is a deep learning model developed by Google for sound event recognition. YAMNet is an audio event classifier that takes audio waveform as input and makes independent predictions for each of 521 audio events from the AudioSet ontology. 

Started with this Yamnet Tutorial. https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/hub/tutorials/yamnet.ipynb

Let see some examples of the prediction outputs using this model.

First audio captured with the Raspberry Pi4 Compute Module and the Trust Microphone + USB Audio card

image

An example given by Google

image

Another sound captured with the Raspberry Pi4 Compute Module and the Trust Microphone + USB Audio card

image

Python code

import tensorflow as tf
import tensorflow_hub as hub
import numpy as np
import csv


from scipy.io import wavfile

# Load the model.
# model = hub.load('https://tfhub.dev/google/yamnet/1')
model = hub.load('./yamnet_model')


# Find the name of the class with the top score when mean-aggregated across frames.
def class_names_from_csv(class_map_csv_text):
  """Returns list of class names corresponding to score vector."""
  class_names = []
  with tf.io.gfile.GFile(class_map_csv_text) as csvfile:
    reader = csv.DictReader(csvfile)
    for row in reader:
      class_names.append(row['display_name'])

  return class_names

class_map_path = model.class_map_path().numpy()
class_names = class_names_from_csv(class_map_path)

def ensure_sample_rate(original_sample_rate, waveform,
                       desired_sample_rate=16000):
  """Resample waveform if required."""
  if original_sample_rate != desired_sample_rate:
    desired_length = int(round(float(len(waveform)) /
                               original_sample_rate * desired_sample_rate))
    waveform = scipy.signal.resample(waveform, desired_length)
  return desired_sample_rate, waveform


wav_file_name = 'recycling_truck.wav'

sample_rate, wav_data = wavfile.read(wav_file_name, 'rb')
sample_rate, wav_data = ensure_sample_rate(sample_rate, wav_data)

# Show some basic information about the audio.
duration = len(wav_data)/sample_rate
print(f'Sample rate: {sample_rate} Hz')
print(f'Total duration: {duration:.2f}s')
print(f'Size of the input: {len(wav_data)}')

# Listening to the wav file.
Audio(wav_data, rate=sample_rate)

# The wav_data needs to be normalized to values in [-1.0, 1.0]
waveform = wav_data / tf.int16.max

# Run the model, check the output.
scores, embeddings, spectrogram = model(waveform)

scores_np = scores.numpy()
spectrogram_np = spectrogram.numpy()
infered_class = class_names[scores_np.mean(axis=0).argmax()]
print(f'The main sound is: {infered_class}')

Installing TensorFlow on the Raspberry Pi4 Compute Module

Preparing the Python environment to run the sound classifier on the Raspberry Pi is very simple.

https://www.tensorflow.org/install/pip?lang=python3

python -m venv --system-site-packages pyenv

sudo apt-get install libhdf5-serial-dev

python3 -m pip install tensorflow

python3 -m pip install tensorflow_hub

It is a large installation

Installing PyAudio

PyAudio provides Python bindings for PortAudio v19, the cross-platform audio I/O library. With PyAudio, you can easily use Python to play and record audio on a variety of platforms, such as GNU/Linux, Microsoft Windows, and Apple macOS.

First install PortAudio v19

 sudo apt-get install portaudio19-dev

Then in your python environment pip install pyAudio

pip install PyAudio

Insall scipy

pip install scipy

Using Alsamixer to set microphone gain

alsamixer is a command-line audio mixer utility for the Advanced Linux Sound Architecture (ALSA) system, which is commonly used in Linux-based operating systems.

image

It provides a graphical interface within the terminal for adjusting various audio settings, such as volume levels, input/output selection, and channel balance.

We utilize it to adjust the microphone capture gain level, a critical aspect for our project. This adjustment determines what qualifies as silence and helps us prevent microphone saturation.

image

Field tests with the YAMnet model

Many of the tests conducted revolve around determining the feasibility of housing the microphone inside the box. Understandably, the acoustics inside the box differ, with more reverberation present. This often results in periods of silence, indicating that the microphone is situated within a confined space, essentially a box.

Moreover, it appears that the foam used to bed the battery improves the sound conditions. While increasing the microphone gain can partially compensate for the acoustic isolation provided by the box, it also amplifies signal noise.

Siren rising from https://freesound.org/

Street parade with music

Urban ambience. People talking

Pocket radio tuning

Night street sound

Summary and conclusions

After two weeks of experimentation, I'm thrilled with the excellent results. I had prior experience with sound recognition using TensorFlow Lite, but I couldn't achieve such impressive outcomes. It seems my hand injury is getting worse, soI still need to wait before finishing machining the box to attach the microphone externally.

Thanks for reading!

Parents Comment Children
  • in reply to javagoza

    Back in 2003, while attempting to finish a remodeling project, I had a scary accident with my table saw. The piece of wood that I was working on was kicked out by the blade (actually more of a planner/shaper), and my left hand (which was applying pressure to keep the wood aligned with the fence), not having anything to resist it, moved into the path of the blade.  Checking the wound, I was able to identify all of the fingers.  At the ER, the x-rays showed significant bone loss in distal (outermost) phalanges of my middle and ring fingers.  The doctor told me was going to remove both of them.  I went into shock and then into bargaining mode and begged him to try and save at least one of them.  He re-examined the x-rays and agreed to trying to save the middle finger. In surgery they removed the distal phalange of the ring finger and patched every else up.  The nail on the middle finger was very damaged and to this day misshaped, but I am happy to still have all of it.

    I loss about 25% of the ring finger, hence the counting to 9.75.  I am extremely happy with the outcome, as 9.75 is so much better that 9.5!

    To this day, I feel the best way of dealing with any injury or set back to resort to humor. So, thank you for noticing my humor.