Honey Bee Safe: Wrapping up: Summary and Source Code

28 Apr 2023

Honey Bee Safe: Wrapping up: Summary and Source Code

This has been an interesting challenge, and it was fun learning about machine language models, using open MV and Edge Impulse, and playing with an artistically created stunt bee and hornet.

We originally set out to use computer vision to detect bees on fruit tree blossoms, as well as detecting invasive murder hornets, using artistic stand-ins for all the things because it was winter when we started.

These were our project goals:

Detect murder hornets.
Count the number of bees
Make the results available via a web browser

And I think we achieved what we set out to do! Yay us!

The results show what is possible. As it is right now, we're getting fairly good recognition, but with more images I'm sure we could get much better results. To do this with real bee and hornet (and other insect) pictures, it might be a good plan to enlist a few more people to help with the bounding boxes!

Unfortunately I was unable to find a local LoraWan receiver gateway for The Things Network, so I wasn't able to use Lora in this project, which would have been interesting too. But luckily using WiFi directly from the Nicla Vision works very nicely.

Overall though, I'm very happy with the results.

And I really like that I was able to save images of hornets when they are detected. This will help stop false positives becoming a worry.

I feel that this proof of concept could be updated to work in a real setting, by just changing all of the training images to use real bees and murder hornets. That would be quite a task, of course, given the number of images that will likely be required to get reliable real-life results with such little insects.

This could actually do as our project name implies - keeping our honey bees safe (from murder hornets).

Here is the Python script I used with openMV, that runs on the Nicla Vision board.

#
# Honey Bee Safe - keeping the honey bees safe.
#
# Uses bits of code from openMV examples, and Edge Impulse for the machine language model
#
# Extra details added by Nico teWinkel to detect bees and murder hornets and report to the server.
#

import sensor, image, time, os, tf, math, uos, gc, pyb

# for MQTT to PiServer
import network
from mqtt import MQTTClient

SSID='MyWiFiNetwork' # Network SSID ---> change this to your own WiFi network name
KEY='My WiFiPassword'  # Network key ---> change this to your own WiFi network password

# connect to the WiFi network
print("Trying to connect. Note this may take a while...")

wlan = network.WLAN(network.STA_IF)
wlan.deinit()
wlan.active(True)
wlan.connect(SSID, KEY, timeout=30000)

# We should have a valid IP now via DHCP
print("WiFi Connected ", wlan.ifconfig())

client = MQTTClient("mulberry", "piserver.local", port=1883)  # if .local doesn't work, use IP address
client.connect()

# prepare the camera sensor
sensor.reset()                         # Reset and initialize the sensor.
sensor.set_pixformat(sensor.RGB565)    # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA)      # Set frame size to QVGA (320x240)
sensor.set_windowing((240, 240))       # Set 240x240 window.
sensor.set_vflip(True) # Flips the image vertically
sensor.set_hmirror(True) # Mirrors the image horizontally
sensor.skip_frames(time=2000)          # Let the camera adjust.

net = None
labels = None
min_confidence = 0.5

try:
    # Load our model, that Edge Impulse built into the firmware
    labels, net = tf.load_builtin_model('trained')
except Exception as e:
    raise Exception(e)

colors = [ # Add more colors if you are detecting more than 7 types of classes at once.
    (255,   0,   0),
    (  0, 255,   0),
    (255, 255,   0),
    (  0,   0, 255),
    (255,   0, 255),
    (  0, 255, 255),
    (255, 255, 255),
]

clock = time.clock()
timecounter = 0
server_update_counter = 0
image_counter = 0

while(True):
    clock.tick()

    img = sensor.snapshot()

    # Since we don't want to double-count the bees, we will only take a picture every 10 seconds
    # We are assuming a bee will spend an average of 10 seconds on a flower - this is a wild guess

    if (timecounter > 33):
        # detect() returns all objects found in the image (splitted out per class already)
        # we skip class index 0, as that is the background, and then draw circles at the center
        # of our objects
        timecounter = 0
        server_update_counter = server_update_counter + 1

        for i, detection_list in enumerate(net.detect(img, thresholds=[(math.ceil(min_confidence * 255), 255)])):
            if (i == 0): continue # background class

            thing_count = len(detection_list)

            if (thing_count == 0): continue # no detections for this class?

            thing_found = labels[i]

            print("********** " + thing_found + " **********" )
            print("Number of %ss: " %thing_found, thing_count)

            for d in detection_list:
                [x, y, w, h] = d.rect()
                center_x = math.floor(x + (w / 2))
                center_y = math.floor(y + (h / 2))
                print('x %d\ty %d' % (center_x, center_y))
                img.draw_circle((center_x, center_y, 12), color=colors[i], thickness=2)

                # rectangles usually don't properly surround the thing that was detected
                # so may as well stick with the circles.
                #img.draw_rectangle(d.rect(), color=colors[i], thickness=2)

            # Check if this is something we need to send to the server
            if (thing_found == 'bee'):
                print("BEE! Send this to the server")
                json_string = "{\"sensor_id\":\"bee_sensor\",\"type\":\"bee\",\"count\":" + str(thing_count) + "}"
                client.publish("mulberry/bees", json_string)
                server_update_counter = 0

            if (thing_found == 'murderhornet'):
                print("MURDER HORNET! Send this to the server")
                json_string = "{\"sensor_id\":\"bee_sensor\",\"type\":\"hornet\",\"count\":" + str(thing_count) + "}"
                client.publish("mulberry/bees", json_string)
                server_update_counter = 0

                # Because this is supposed to be a rare and bad situation, we should try to send a picture to the server
                # Unfortunately, trying to b64 encode the image seems to hang the board. It might require too much memory.
                # so we will just save the pictures
                filename = "hornet_" + str(image_counter) + ".jpg"
                img.save(filename)
                image_counter = image_counter+1


    print(clock.fps(), "fps", end="\n\n")

    pyb.delay(300) # in milliseconds. This is just a small delay between sensor snapshots
    timecounter = timecounter + 1

    # even if nothing detected for 6 rounds, check in every minute
    if (server_update_counter > 5):
        print("Nothing! Send a checkin to the server")
        client.publish("mulberry/bees", "{\"sensor_id\":\"bee_sensor\",\"type\":\"checkin\"}")
        server_update_counter = 0

In case you want to start from the beginning, here is the list of blog posts I created for the challenge: