Introduction
Last Blog, we discuss how to reroute the PDM mic signal to Sound Card which is require for our intelligence mailbox. To detect dog bites and play alarm scare away the poor dog or make notification to the Dog owner.
To meeting the target, as we don't want triggering the alarm just cat meow or Peking dog wow, so a good AI or sound recognition to detect vicious dog need. To fulfil the requirement, first require is collect the sound samples, which our IoT hardware job done, we also hope working for pre-process of sound signal such as trigger the sound recognition when the sound level high to certain level, so we need a buffer enough to storage about 5-8 second sound samples, which here try to build it.
What the Challenge
To reduce the power consumption and small footprint and keep the cost lowest, we only have very limit resource compare as PC, mobile phone or even Raspberry is too heavy for our application. The Raspberry 4 draw 2.85W under idle, compare as PSO6 CPU run under 60mw at full speed mode, even PSO6 power draw is so low, we also need to carefully adjust the power consumption to make the platform run power lower, for example we optimum our process a batch and make the system go to deep sleep ASAP.
As PSO6 such low power, don't thing that only allow to make simple task or re-route the signal to cloud just like a 8-bit MPU, as our explosion , the PSO6 good enough for working more, such as become a control centre, logging, analysis, filtering, troubleshooting, even work small AI task, we can decentralised working tasks down to near field and exchange only necessary data between centre (cloud) and the device, to make whole system more economic, simple and security.
In our application, which require monitoring the background sound to recognise the dog sound, require a forever buffer to storage sound sample,
Ref: https://raspi.tv/2019/how-much-power-does-the-pi4b-use-power-measurements
Our main program area, only have 288K SRAM for program usage included the global variance, heap and stack, in our current software, our program data + bss require 136kb, so we only have 140kb for heap+stack usage, the LCD UI use about 32kb, the WHD (WIFI driver) + AWS use almost 50kb with additional 10kb for other, we almost use near all of our SRAM, with no room for storage sound buffer, we need immediate transfer the PDM Mic data to external memory.
Under our kits, we have four devices of memory, which are SRAM and Internal Flash, also with 512KB FRAM and 64MB Flash connected by QSPI.
In our application, we use 16khz 16bit mono data, which 32KB/S, if we use all 64MB flash, we can store 2048S in a cycle, as 10K write cycle, we got 237 days out of life, also it write speed is relative slow, we require big SRAM buffer, which we can not afford.
FRAM
The FRAM is our solution, the Ferroelectric Random Access Memory is new type of non-volatile with very long endurance, very high speed and don't need erase before write, although it very expensive but cheaper that SRAM, make it good for using critical environment which require instant record, the FRAM give us at least ten thousand years record time, although current FRAM size very small, the 512KB FRAM can only record 16s, this enough our usage.
As the FRAM don't require slow block erase before write and don't require additional wear levelling process, as this latency fix and low, the power consumption of every write can be very low and don't require additional power loss protection circuit and backup battery, make this very suitable for IoT use, as many IoT frequency write small logging such as sensor data. As it very fast writing time, it can be faster return to sleep idle reduce the overall power consumption.
From the view of programming, it also reduce complex of software, as the status can save it instantly, quick and high endurance, no additional power loss recovery logic and verify is required.
The reason is amazing, our system only require 256 Bytes small buffer for record and playback under SRAM, as both of DMA is using by WIFI, as our experiment, the record/playback will affect the network, which make AWS jobs unstable, it is because the DMA is share for both Sound and WIFI SDIO. In here, we shift using traditional interrupt method for record and playback, because FRAM speed very high, as such small buffer size, we able working for AWS task, update LCD, logging sensor without destroy the playback or record.
How Sound Buffer Work
Because we don't need echo the sound, in this case we sample use a size 128 buffer for record/playback.
int16_t buffer[128];
We apply two interrupt PDM and I2S, for PDM we trigger interrupt when PDM's FIFO with 128 sample:
for(int i=0;i<128;i++){
buffer[i] = Cy_PDM_PCM_ReadFifo(CYBSP_PDM_HW);
}
fram_write_sound(sound_record_end_pos++,(void*)buffer);
We just fill our buffer 128 sample once a time and write it to FRAM.
Under fram.h, we define the data address, here we use end 3/4 address which enough to save 12s audio chip, as we prevent the front for storage our program status such as the record positions.
Every block size is 256 bytes and we total have 1536 block, because FRAM can overwrite the bit directly, so don't require erase the block before write like flash, our section can be very small with very low latency.
/*MEMORY MAPPING*/ #define RECORD_SOUND_ADDRESS_START 0x1FFFF #define RECORD_SOUND_DATA_PRE_BLOCK 256 #define RECORD_SOUND_TOTAL_BLOCK 1536 #define STATUS_ADDRESS 0xFFFF
Our recorder will erase early data when the record time more that 12s, the sound_record_start_pos and sound_record_end_pos use for tracking the start and end position, those also save to FRAM when record finish. When the record pos run to the end, the record will come back to 0 to form a forever the loop, it forever record but only last 12 seconds will be remained.
The start and end position have been saved in FRAM, so after power loss can retrieve the audio clip.
The playback like revising process of the record, we read the FRAM data and then duplicate the data feed to I2S Left/Right channel, because every data need send twice, we require two cycle for the I2S FIFO to consume one FRAM block, as we set the trigger level 128 and empty, the I2S interrupt will require us feed the I2S FIFO when empty or FIFO buffer only with 128 samples.
if (!half)
fram_read_sound(sound_playingblock,(void*)buffer);
for(int i=0+half*64;i<(64+half*64);i++){
Cy_I2S_WriteTxData(CYBSP_I2S_HW, buffer[i]);
Cy_I2S_WriteTxData(CYBSP_I2S_HW, buffer[i]);
}
When the playback arrival at the end, the interrupt change the global var call_stop_play.
Every 20ms, our main thread will call sound_update for update the sound recorder status, the sound_update will track the call_stop_play,
when the call_stop_play is true, the sound_update will call sound_stop_playback to close sound card and recovery UART.
Control of Sound Recorder
hw.cpp received AWS/Capsense user inputs, after that the hw.cpp will call sound.cpp change of relative status var.
The main thread will call sound_update every 20ms, the sound_update track any change, check and working relative process.
The User can control the recorder either from IOS APP though AWS IoT or Capsense, use the indirect control method mainly for prevent deal loops issue,
also ISR is limit on Mbed, for example ISR is not allow mutex.
The slider is using for volume control can be adjusted though AWS or by Capsense.
The Code
The source code available under our git, the source code also included the example of drive motion sensor / use the light sensor under mbed.
The PDM/I2S and relative GPIO are initiating by code to avoid dead because of UART, so don't select PDM/I2S under Cypress Device Configuration.
The codes also integrated with the WIFI Low power, the RTC with NTP server time sync by reference from Cypress examples.
https://github.com/sicreative/mbed-os-recorder
Next
The FRAM, a new type of long life non volatile memory with incredible write cycle, high speed, low latency good for instantly save application, as here we try to use this as sound buffer.
Although currently the FRAM only have small size (<8M Bit) and high cost, make it unattractive to high volumes low prices market,
as opposite the FRAM can be use in crucial product such as medical and industrial.
Such as currently new hearing aids product use of FRAM for audio noise cancel and adjustment, the application is actually some as this example.
Next step, try to send sound data though AWS IoT.....
Relative Cypress examples reference in here:
https://github.com/cypresssemiconductorco/mtb-example-psoc6-pdm-to-i2s
https://github.com/cypresssemiconductorco/mtb-example-psoc6-qspi-fram-access
https://github.com/cypresssemiconductorco/mbed-os-example-wlan-lowpower
https://github.com/cypresssemiconductorco/mbed-os-example-wifi-provisioning-via-ble