*updated 24-May-2020 for Angry dog sound classification #11
Introduction
In this challenge, the target is by build a prototype of intelligent mailbox, to study how Cypress PSOC6 and AWS cloud can be simplify of iOT both hardware, administration and support costs.
In traditional IoT solution, normally with a MQTT broker served for a server role to obtain, admin and process of client data, if number of "Things" increase, the system become hard to manage. An IOT project is easy to engage such many things, for example, detect soil moisture of agriculture is easy require few hundred sensors for a farmer, make it scale like a mid-size network.
The AWS Cloud solution, not only become the MQTT broker, it also fully integrated with AWS cloud resources, the pay-as-you-go also reduce the product initial investment without require excess hardware facility to provide reliability IOT solution.
In our project, by use of AWS IoT shadow, Cognito, S3 storage and a lambda function to serve multi user, multi intelligent mailboxs and the mobile APP clients without investment of server, eliminate of hardware and support as well as enjoy high reliabilty AWS framework.
Block Diagrams
Above show the intelligent mailbox software components involved.
In additional of Cypress Pioneer Kit and the TFT shield, only a green led for mail detection is used.
For develop platform we run:
1. Mbed CLI
2. ModusToolbox 2.1
3. Xcode 11.4
Brief of the system working flow
Our System with three main parts, the cypress kit, AWS IoT Cloud and IOS APP:
Cypress
- Initiation of Mic Sound Card, WIFI and NTP, Motion sensors, Analog Input of Light Sensors
- Initiation of FRAM and retrieve last status
- Initialtion emwin LCD display
- Setup WIFI connection
- Start up AWS IoT MQTT, subscribe to shadow get and productmetadata
- Retrieve AWS shadow status
- A separated thread for AWSIoT network transfer, included read FRAM sound data for stream and publishing shadow, handling IOS APP pairing, when idle wait of subscribe incoming.
- main thread enter to 20ms forever loop for :
- Capsense read
- Sensor read
- Sound recorder control
- RTC update
9. Seperated thread for LCD update, which serve the main / awsiot thread for display task by event query
10. PDM MIC and I2S SOUND Interrupt for save data to RAM buffer, forwarding sound clip up to 12s under FRAM, only very small RAM buffer is required thanks of fast FRAM.
AWS IoT Cloud
- Security access managed MQTT broker
- Working for AWS shadow, provide two way synchronisation between IOS APP and Cypress Device
- Congnito user login for APP
- Kinesis stream for sound
- S3 Storage for storage sound clip recorded by cypress kit
- Lambda python function reconstruct sound steam, add wav header, build Transcribe job and publish result back to IOS APP and Cypress Kit
- Run under SageMaker for recognition of sound to classfield is or is not angry dog, if angry dog sound recognised, push alarm wav to IoT device and play it.
IOS APP
- Login Control by Congnito
- Pairing between the Kits (individual mailbox) and APP
- Control Cypress Kit and read sensor data (detect the mail) though AWS shadow
- Show transcript / AI result
The brief introduced of Previous Blog Working
Begin the journey : Hello the World by PSoC6 Kits
The first view of the kits
Mbed connect to AWS iOT, OS, driver, security .. #2
Work for Mbed, LCD driver and first view of AWS IoT
Mbed RTOS thread: Send CapSense Data to AWS #3
CapSense and send button/slider data to AWS IoT
Study of AWS shadow sync between iOS and the Cypress IoT device #4
Build IOS APP and sync/control by AWS shadow
Sense the mail, Send to cloud #5
Simply use the kit's Ambient Light Sensor and a LED for mail detection, and read the BMI160 Acc and Gyro motion data
Echo the sound by the pioneer kit #6
Work with PDM MIC and Sound Card, use of DMA for CPU-free transfer
Exploring of Cypress long life FRAM sound Recorder control from AWS IOT #7
Study use of FRAM for storage the sound data, use Interrupt read of PDM MIC and transfer data to I2S
Adventure of stream the sound clip to AWS #8
Stream the sound and storage to AWS Cloud
Fun with AWS Transcribe and IOS APP pairing #9
Sound data feed of AWS Transcribe for speech to text process and feedback to IOT device, also implement the IOS APP pairing for owner verifying.
Deep Learning for angry dog sound classification #11
Sound Recognition of Angry Dog by Amazon SageMaker, Tensorflow, AudioSet, Yamnet and custom training.
Source Code
Our git provide all the Mbed, IOS and Lambda code with explan for reference, detail can also read from above previous individual blog.
Final versions. Demo of run sound recognised under Amazon SageMaker
https://github.com/sicreative/ConnectedCloudChallenge
Demo of use AWS speech Translate
https://github.com/sicreative/ConnectedCloudChallengeTranslate
Conclusion
For the limited of time we are hard to build the whole new AI model for dog bites detect in this challenge, instead we fine-turning of Yamnet to get the binary result - AngryDog or Other with achieve around 90% accuracy. Our system run under of Amazon SageMaker for custom inference, as well as use AWS service for text transcription. We also with good performance of use AWS IOT shadow for remote control/monitoring IOT device by Mobile APP, as update time normally under two seconds.
By use of AWS IoT, Lambda functions, IAM, S3 our framework support unlimited user by serverless architecture, the system administration cost can be minimum also reduce complex of whole system. With proper design, opportunity of run a large number of IoT devices without high skill of computer technical knowledge such as design a mobile APP for the farmer work for organising hundred of sensors, support customer control and monitor under APP.
By using of AWS Kinesis and Lambda function, we are success use of AWS IoT MQTT to translate large sound data by segment with very low memory usage.
As the Cypress Pioneer Kit CPU only provide 288kb SRAM, sometime hard to do more as WHD,TLS and MQTT Socket require many heap size, the 1MB versions become best alternative for task require more ram such as Alexa.
Thanks for elements14, Cypress and AWS run of this Community Challenge.