2Bee Or Not 2Bee
That is the question, lets see if machine vision can answer that question.
Background
Bees are incredible creatures that are fundamentally important to our food supply as well as the entire world's eco system, and they face many significant threats. I have a buddy who keeps bees and the only thing he worries about is mites getting into his hives. He indicates the best way to monitor this is to visually inspect the bees in the hive. This project challenge is about raising awareness within the technical community of the issues surrounding bees. However this project won't dwell on the plight of bees but rather it highlights some technical ideas that might be applied to mitigate some of the issues.
Intro - Setting the Stage
But how can we make the topic of bees entertaining for a technical crowd? Well strap in for the ride because on the way we are going to have conversations with a couple of bees named OhBeeOne and TwoBee. If their voices sound a little electronic, it is just an artifact of their universal translator. In fact I won't be talking to them beyond initial introductions, it will be our AI field team of Lora and Husky, who of course are going to sound like you would expect an AI to sound. We can follow all the action out in the field because Lora will be sending a play-by-play back to Lora's sister Crystal. Our AI field team will use machine vision and machine learning to recognize all the main characters in the field of view, including guest appearances by Spiderman and Cindy Cicada. Hopefully Grasshopper can hop on a plane and make an appearance. We might even persuade Lady Bug to drop in for a cameo if her flight arrives in time. Husky does the first contact heavy lifting while Lora does all the talking and of course communicating with us.
Lora has a brother named Ardu, who will be reporting live from the hive. He will be sending long range images of hive status back to his sister Crystal.
System Block Diagram
This is a block diagram showing the main components of the system.
There are three main blocks:
A base station with LoRa radio and LCD (left block)
A farmer system to monitor crop pollination and report back via LoRa (upper block)
A bee keeper system to monitor hive health and report back via LoRa (lower block)
Cast of Characters
This first video will introduce some of the main characters using speech synthesis, although there may be more characters before the project is complete.
Technology Objectives
Hopefully the interaction between all the characters will be entertaining, but just as hopefully the underlying technologies will showcase a couple of potentially useful applications for key stakeholders, which are:
- Farmer information regarding crop pollination which will use machine learning and machine vision to recognize bees and bee activity in crop fields and report back wirelessly
- Bee Keeper information regarding bee hive health which will use a low cost remote imaging system to provide visual evidence of bee health in the hive
Ultimately the project will use a lot of technology like machine vision, machine learning, speech synthesis, and long range image broadcasting, etc. but a key objective is to keep implementation simple because farmers and bee keepers have enough tasks on their plate without needing to obtain PhDs in computer science.
System Description
There are two distinct systems in this project, a farmer system and a bee keeper system. Each system will consist of a base station node and a satellite node communicating via LoRa. The base station node for both systems will be the same Arduino MKR WAN 1300 which will have an LCD to show what is happening out in the field at the satellite node. The base station will be called Crystal.
The Farmer System
The farmer system satellite node will include an Arduino MKR WAN 1310 and a machine vision module. Two different machine vision systems will be tried: One will be Huskylens and the other will be Arduino Nicla Vision. This node will be called either Lora and Husky or Lora Nicla depending on which configuration is in use. Both systems will employ machine learning to perform vision recognition of various insects and scenes. The MKR MCU will also control a speech synthesis module that will allow the various insects and instruments to hold audio conversations. The MKR will also have an LCD to provide local information.
The Bee Keeper System
The bee keeper system satellite node will include a MKR WAN 1300 connected to an Arducam serial camera to take pictures of bees and bee activity in a hive. These images will be transmitted back to Crystal over LoRa. This process will take time since LoRa is a low bandwidth communications system, but there is no need for real time video in this application. Transmitting images over LoRa is not normal, but this is an experiment to see if it is viable.
Speech Synthesis Breadboard
Although the plan is to integrate the various modules with a PCB, initially I needed to breadboard the speech synthesis system because the module had no manual and there is none available on the internet (that I could find). There are similar modules that have slightly better documentation, but it turns out they are not the same and it took a lot of fussing around to determine what this module can do and write software to control it properly. The result is demonstrated in the "cast of characters' video above. Dissappointingly, this module only has one voice, so I have ordered another one with more flexibiity. Hopefully it will arrive in time to be useful.
This module is a SYN6988 and here is the state of the code at this moment:
/*
* Speech Syth Controller - MKR 1300
* Uses a SYN6988 text-to-speech module to talk
* by Doug Wong 2023
*/
#include <SPI.h>
#include <LoRa.h>
#define PIN_SCE 6 //chip select pin for LCD
#define PIN_RESET 10 //reset pin for LCD
#define PIN_DC 7 //data / command select pin for LCD
#define PIN_SDIN 8 //serial data input pin for LCD (MOSI)
#define PIN_SCLK 9 //SPI clock pin for LCD
#define LCD_C LOW //this value select command mode on LCD DC
#define LCD_D HIGH //this value select data mode on LCD DC
#define LCD_X 84 //number of pixels on an LCD line
#define LCD_Y 48 //number of vertical pixels on LCD
char rsi[6]; // received signal strength at home base
char LoRaIn; // character received via LoRa
char LoRaInStr[2];
char loopc[6];
char LoRaChar;
char LoRaString[1];
double xpos;
double ypos;
double rad = 6371000;
int rssi; // recieved signal strength
int packetSize;
byte synback;
int loops;
void speak(char* msg) {
Serial1.write(0xFD);
Serial1.write((byte)0x0);
Serial1.write(2 + strlen(msg));
Serial1.write(0x01);
Serial1.write((byte)0x0);
Serial1.write(msg);
}
void waitForSpeech(unsigned long timeout = 60000) {
unsigned long start = millis();
bool done = false;
while ( ! done && (millis() - start) < timeout ) {
while ( Serial1.available() ) {
if ( Serial1.read() == 0x4F ) {
done = true;
break;
}
}
}
}
// pixel map of character set for LCD
static const byte ASCII[][5] =
{
{0x00, 0x00, 0x00, 0x00, 0x00} // 20
,{0x00, 0x00, 0x5f, 0x00, 0x00} // 21 !
,{0x00, 0x07, 0x00, 0x07, 0x00} // 22 "
,{0x14, 0x7f, 0x14, 0x7f, 0x14} // 23 #
,{0x24, 0x2a, 0x7f, 0x2a, 0x12} // 24 $
,{0x23, 0x13, 0x08, 0x64, 0x62} // 25 %
,{0x36, 0x49, 0x55, 0x22, 0x50} // 26 &
,{0x00, 0x05, 0x03, 0x00, 0x00} // 27 '
,{0x00, 0x1c, 0x22, 0x41, 0x00} // 28 (
,{0x00, 0x41, 0x22, 0x1c, 0x00} // 29 )
,{0x14, 0x08, 0x3e, 0x08, 0x14} // 2a *
,{0x08, 0x08, 0x3e, 0x08, 0x08} // 2b +
,{0x00, 0x50, 0x30, 0x00, 0x00} // 2c ,
,{0x08, 0x08, 0x08, 0x08, 0x08} // 2d -
,{0x00, 0x60, 0x60, 0x00, 0x00} // 2e .
,{0x20, 0x10, 0x08, 0x04, 0x02} // 2f /
,{0x3e, 0x51, 0x49, 0x45, 0x3e} // 30 0
,{0x00, 0x42, 0x7f, 0x40, 0x00} // 31 1
,{0x42, 0x61, 0x51, 0x49, 0x46} // 32 2
,{0x21, 0x41, 0x45, 0x4b, 0x31} // 33 3
,{0x18, 0x14, 0x12, 0x7f, 0x10} // 34 4
,{0x27, 0x45, 0x45, 0x45, 0x39} // 35 5
,{0x3c, 0x4a, 0x49, 0x49, 0x30} // 36 6
,{0x01, 0x71, 0x09, 0x05, 0x03} // 37 7
,{0x36, 0x49, 0x49, 0x49, 0x36} // 38 8
,{0x06, 0x49, 0x49, 0x29, 0x1e} // 39 9
,{0x00, 0x36, 0x36, 0x00, 0x00} // 3a :
,{0x00, 0x56, 0x36, 0x00, 0x00} // 3b ;
,{0x08, 0x14, 0x22, 0x41, 0x00} // 3c <
,{0x14, 0x14, 0x14, 0x14, 0x14} // 3d =
,{0x00, 0x41, 0x22, 0x14, 0x08} // 3e >
,{0x02, 0x01, 0x51, 0x09, 0x06} // 3f ?
,{0x32, 0x49, 0x79, 0x41, 0x3e} // 40 @
,{0x7e, 0x11, 0x11, 0x11, 0x7e} // 41 A
,{0x7f, 0x49, 0x49, 0x49, 0x36} // 42 B
,{0x3e, 0x41, 0x41, 0x41, 0x22} // 43 C
,{0x7f, 0x41, 0x41, 0x22, 0x1c} // 44 D
,{0x7f, 0x49, 0x49, 0x49, 0x41} // 45 E
,{0x7f, 0x09, 0x09, 0x09, 0x01} // 46 F
,{0x3e, 0x41, 0x49, 0x49, 0x7a} // 47 G
,{0x7f, 0x08, 0x08, 0x08, 0x7f} // 48 H
,{0x00, 0x41, 0x7f, 0x41, 0x00} // 49 I
,{0x20, 0x40, 0x41, 0x3f, 0x01} // 4a J
,{0x7f, 0x08, 0x14, 0x22, 0x41} // 4b K
,{0x7f, 0x40, 0x40, 0x40, 0x40} // 4c L
,{0x7f, 0x02, 0x0c, 0x02, 0x7f} // 4d M
,{0x7f, 0x04, 0x08, 0x10, 0x7f} // 4e N
,{0x3e, 0x41, 0x41, 0x41, 0x3e} // 4f O
,{0x7f, 0x09, 0x09, 0x09, 0x06} // 50 P
,{0x3e, 0x41, 0x51, 0x21, 0x5e} // 51 Q
,{0x7f, 0x09, 0x19, 0x29, 0x46} // 52 R
,{0x46, 0x49, 0x49, 0x49, 0x31} // 53 S
,{0x01, 0x01, 0x7f, 0x01, 0x01} // 54 T
,{0x3f, 0x40, 0x40, 0x40, 0x3f} // 55 U
,{0x1f, 0x20, 0x40, 0x20, 0x1f} // 56 V
,{0x3f, 0x40, 0x38, 0x40, 0x3f} // 57 W
,{0x63, 0x14, 0x08, 0x14, 0x63} // 58 X
,{0x07, 0x08, 0x70, 0x08, 0x07} // 59 Y
,{0x61, 0x51, 0x49, 0x45, 0x43} // 5a Z
,{0x00, 0x7f, 0x41, 0x41, 0x00} // 5b [
,{0x02, 0x04, 0x08, 0x10, 0x20} // 5c ¥
,{0x00, 0x41, 0x41, 0x7f, 0x00} // 5d ]
,{0x04, 0x02, 0x01, 0x02, 0x04} // 5e ^
,{0x40, 0x40, 0x40, 0x40, 0x40} // 5f _
,{0x00, 0x01, 0x02, 0x04, 0x00} // 60 `
,{0x20, 0x54, 0x54, 0x54, 0x78} // 61 a
,{0x7f, 0x48, 0x44, 0x44, 0x38} // 62 b
,{0x38, 0x44, 0x44, 0x44, 0x20} // 63 c
,{0x38, 0x44, 0x44, 0x48, 0x7f} // 64 d
,{0x38, 0x54, 0x54, 0x54, 0x18} // 65 e
,{0x08, 0x7e, 0x09, 0x01, 0x02} // 66 f
,{0x0c, 0x52, 0x52, 0x52, 0x3e} // 67 g
,{0x7f, 0x08, 0x04, 0x04, 0x78} // 68 h
,{0x00, 0x44, 0x7d, 0x40, 0x00} // 69 i
,{0x20, 0x40, 0x44, 0x3d, 0x00} // 6a j
,{0x7f, 0x10, 0x28, 0x44, 0x00} // 6b k
,{0x00, 0x41, 0x7f, 0x40, 0x00} // 6c l
,{0x7c, 0x04, 0x18, 0x04, 0x78} // 6d m
,{0x7c, 0x08, 0x04, 0x04, 0x78} // 6e n
,{0x38, 0x44, 0x44, 0x44, 0x38} // 6f o
,{0x7c, 0x14, 0x14, 0x14, 0x08} // 70 p
,{0x08, 0x14, 0x14, 0x18, 0x7c} // 71 q
,{0x7c, 0x08, 0x04, 0x04, 0x08} // 72 r
,{0x48, 0x54, 0x54, 0x54, 0x20} // 73 s
,{0x04, 0x3f, 0x44, 0x40, 0x20} // 74 t
,{0x3c, 0x40, 0x40, 0x20, 0x7c} // 75 u
,{0x1c, 0x20, 0x40, 0x20, 0x1c} // 76 v
,{0x3c, 0x40, 0x30, 0x40, 0x3c} // 77 w
,{0x44, 0x28, 0x10, 0x28, 0x44} // 78 x
,{0x0c, 0x50, 0x50, 0x50, 0x3c} // 79 y
,{0x44, 0x64, 0x54, 0x4c, 0x44} // 7a z
,{0x00, 0x08, 0x36, 0x41, 0x00} // 7b {
,{0x00, 0x00, 0x7f, 0x00, 0x00} // 7c |
,{0x00, 0x41, 0x36, 0x08, 0x00} // 7d }
,{0x10, 0x08, 0x08, 0x10, 0x08} // 7e ←
,{0x78, 0x46, 0x41, 0x46, 0x78} // 7f →
};
void LcdCharacter(char character) // display a character on the LCD
{
LcdWrite(LCD_D, 0x00); // display a blank space before character
for (int index = 0; index < 5; index++) // font uses 5 x 7 pixels
{
LcdWrite(LCD_D, ASCII[character - 0x20][index]); // display next column of pixels for this character
}
LcdWrite(LCD_D, 0x00); // display a blank space after character
}
void LcdClear(void) // display a blank screen
{
LcdWrite(LCD_C, 0x40 ); // command to set Y cursor position to 0
LcdWrite(LCD_C, 0x80 ); // command to set X cursor position to 0
for (int index = 0; index < LCD_X * LCD_Y / 8; index++)
{
LcdWrite(LCD_D, 0x00); // display a blank column of 8 pixels
} // the LCD automatically steps to the next position
}
void LcdInitialise(void)
{
pinMode(PIN_SCE, OUTPUT); // set the chip select pin to be an output
// pinMode(PIN_RESET, OUTPUT);
pinMode(PIN_DC, OUTPUT); // set the DC pin to be an output
pinMode(PIN_SDIN, OUTPUT); // set the MOSI pin to be an output
pinMode(PIN_SCLK, OUTPUT); // set the SPI clock pin to be an output
// digitalWrite(PIN_RESET, LOW);
// digitalWrite(PIN_RESET, HIGH);
LcdWrite(LCD_C, 0x21 ); // put LCD in Extended Commands mode.
LcdWrite(LCD_C, 0xCC ); // Set LCD Vop (Contrast). 0x80 - 0xFF
LcdWrite(LCD_C, 0x05 ); // Set Temp coefficent. 0x04 - 0x07
LcdWrite(LCD_C, 0x16 ); // LCD bias mode 1:48. 0x10 - 0x17
LcdWrite(LCD_C, 0x20 ); // LCD Basic Commands
LcdWrite(LCD_C, 0x0C ); // put LCD back in normal mode.
}
void LcdString(char *characters) // display a sting of characters on LCD
{
while (*characters)
{
LcdCharacter(*characters++);
}
}
void LcdWrite(byte dc, byte data) // display a byte as 8 vertical pixels
{
digitalWrite(PIN_DC, dc); // control the DC pin - should be data
digitalWrite(PIN_SCE, LOW); // apply chip select to LCD
shiftOut(PIN_SDIN, PIN_SCLK, MSBFIRST, data); // send byte as serial bit stream
digitalWrite(PIN_SCE, HIGH); // deselect LCD
}
void setup() {
Serial1.begin(9600); // this serial port is used for controlling speech
LcdInitialise(); // display a splash screen
LcdClear();
LcdString(" ARDUNIO ");
LcdString(" MKR WAN ");
LcdString(" 1300 ");
LcdString(" SPEECH ");
LcdString(" by ");
LcdString(" DOUG WONG ");
delay(4000); // show the splash screen for 4 seconds
LcdClear(); // clear the LCD
}
void loop() // main loop has a conversation
{
char buf[128]; // for text-to-speech text
LcdClear();
LcdString(" Doug "); // indicate who is talking - actually allowing time to dub in audio
delay(4000);
// This blog is a quick introduction to the cast of characters who will be featured in this project.
// First up we have one of the stars of the show, OhBeeOne:
LcdClear();
LcdString(" OhBeeOne "); // indicate who is talking
sprintf( buf, "[m3]Hello, I am ohbeewun. I am interested to see how you can help bees.");
speak(buf);
LcdString(" Speaking ");
waitForSpeech();
LcdClear();
LcdString(" Idle ");
delay(500);
LcdClear();
LcdString(" Doug "); // indicate who is talking
delay(4000);
// Lora and Husky will be collecting activity data so it can be correlated with environmental data, and Ardu will be looking out for mites.
LcdClear();
LcdString(" OhBeeOne "); // indicate who is talking
sprintf( buf, "[m3][s7][v7]That sounds good. May the forrce be with you.");
speak(buf);
LcdString(" Speaking ");
waitForSpeech();
LcdClear();
LcdString(" Idle ");
delay(500);
LcdClear();
LcdString(" Doug "); // indicate who is talking
delay(4000);
// Who do we have next? Is that Two Bee or not Two Bee?
LcdClear();
LcdString(" TwoBee "); // indicate who is talking
sprintf( buf, "[m0][s7][v7]Hi, of corse I am TooBee. As you can plainly see. When will we get a new universal translator? We all sound the same.");
speak(buf);
LcdString(" Speaking ");
waitForSpeech();
LcdClear();
LcdString(" Idle ");
delay(500);
LcdClear();
LcdString(" Doug "); // indicate who is talking
delay(4000);
// The new universal translator is definitely in the mail. I hope it gets here soon as well.
// Up next we have our field reporter Lora.
LcdClear();
LcdString(" Lora "); // indicate who is talking
LcdString(" Speaking ");
sprintf( buf, "[g2][m10][t8]Hi, I am Lora. Half of the Lora and Husky [f0]team");
speak(buf);
waitForSpeech();
LcdClear();
LcdString(" Idle ");
delay(500);
LcdClear();
LcdString(" Doug "); // indicate who is talking
delay(4000);
//
LcdClear();
LcdString(" Husky "); // indicate who is talking
LcdString(" Speaking ");
sprintf( buf, "[g2][m51][t8]Yo, I am Husky. I am trying to learn wut [f0]bees [f0]look like.");
speak(buf);
waitForSpeech();
LcdClear();
LcdString(" Idle ");
delay(500);
LcdClear();
LcdString(" Doug "); // indicate who is talking
delay(4000);
// Wow, look, it is Spiderman.
LcdClear();
LcdString(" Spiderman "); // indicate who is talking
LcdString(" Speaking ");
sprintf( buf, "[g2][m3][t8][s2]Hey there, I am Spiderman. Lets see if I get confused with a bee.");
speak(buf);
waitForSpeech();
LcdClear();
LcdString(" Idle ");
delay(500);
LcdClear();
LcdString(" Doug "); // indicate who is talking
delay(4000);
// I hope not.
// I think I see a cicada.
LcdClear();
LcdString(" Cindy "); // indicate who is talking
LcdString(" Speaking ");
sprintf( buf, "[g2][m3][t8]Hi I am Sindy Cicada. I hope I don't look like a bee.");
speak(buf);
waitForSpeech();
LcdClear();
LcdString(" Idle ");
delay(500);
LcdClear();
LcdString(" Doug "); // indicate who is talking
delay(4000);
// We will have to ask Husky.
// Not everyone has arrived to the party yet:
LcdClear();
LcdString(" Grasshopper"); // indicate who is talking
LcdString(" Speaking ");
sprintf( buf, "[g2][m20][t8]Hey, Grasshopper here, coming at you remotely. I will show up, as soon as I can hop on a plane.");
speak(buf);
waitForSpeech();
LcdClear();
LcdString(" Idle ");
delay(500);
LcdClear();
LcdString(" Doug "); // indicate who is talking
delay(4000);
//I hope you get here soon, we need you.
LcdClear();
LcdString(" Grasshopper"); // indicate who is talking
LcdString(" Speaking ");
sprintf( buf, "[g2][m36][t8]As my Master would say, patience grasshopper, patience.");
speak(buf);
waitForSpeech();
LcdClear();
LcdString(" Idle ");
delay(500);
LcdClear();
LcdString(" Doug "); // indicate who is talking
delay(4000);
// Okay, who is calling in now?
LcdClear();
LcdString(" LadyBug "); // indicate who is talking
LcdString(" Speaking ");
sprintf( buf, "[g2][m44][t8]Hello I am Ladybug. I hope to fly in for a cameo at some point. I am not to be confused with a bee. ");
speak(buf);
waitForSpeech();
LcdClear();
LcdString(" Idle ");
delay(500);
LcdClear();
LcdString(" Doug "); // indicate who is talking
delay(4000);
// It will be good to see you, everyone loves ladybugs.
}
Technology Summary
This project will use at least 5 MCUs, at least 3 LCDs, at least 3 cameras, 2 machine learning systems, probably 2 different speech synthesis modules and at least 3 variants of a custom PCB. 3D printed housings will be designed for each node.
Stretch Goals
I was intending to explore the Arduino ESLOV Visual Code Editor since both the MKR and Nicla include an ESLOV connector, but it appears this initiative, which started in 2016, has yet to be launched. Instead I will try to use the Arduino IoT Cloud so that bee info from the satellite nodes can be viewed by any internet connected device.
I had considered making waterproof enclosures and solar power supplies for outdoor deployment, but the timing of this project really won't allow enough time to perfect such systems during bee season. There would also be a lot of logistics for me to to test on site at my buddy's bee operation. But we will see how the schedule plays out.
Next Steps- The Plan
The project needs at least 5 blogs. The following blogs will be produced, although the various topics may be partitioned differently or split into separate blogs:
- This blog introducing the project and the main characters. Speech synthesis code.
- Kit unboxing, Machine learning/training, Husky demo of insect recognition
- Schematics and PCB design. Conversations with insects.
- Electronic systems build of 3 nodes, node software, 3D printed case design
- Farmer system demonstration, Bee Keeper system demonstration, project summary and discussion
Links:
Save the Bees - Machine Learning
Image Conversion to Integer Array for LCD Display
Top Comments