Introduction

In this episode, we will see the activation of another interesting feature of Seven of Nine; based on my experience, I can classify it as one of the most complex (and intriguing) integrations between the user, the machine and the environment. I should admit that this has been also the most challenging and complex part of the entire project and I had to make several tries approaching different paths before finding a way relatively simple and robust to provide the expected results for a long period of continuous usage (seven hours a day, five days a week during the two months of exhibition).

Edit media

Dimensions x Small Medium Large Custom

Subject (required) Brief Description Tags (separated by comma) Video visibility in search results Visible Hidden

Parent content

Art-a-Tronic Episode 12

Poster

Upload Preview

Ok

Cancel

A Window to the Outer World

SevenOfNine should be able to interact with the visitors and the surrounding environment in several ways. One of these is starting a contextual speaking to the visitor through a third subject creating an unexpected and surprising situation; in all the interaction events the human-machine communication is started involuntarily by the visitor that originate a sequence of connected events:

• The visitor is "forced" to note a magic mirror hang at about five meters from Seven Of Nine. it is impossible not to see it, especially watching toward the Borg-like.
• The visitor is attracted by the hanging mirror showing a text (refreshed every 30 seconds) and the current time and moved toward it.
• The mirror detects the subjects (via an ultrasonic sensor) and recognizes its moving direction
• As the visitor nearby to the magic mirror, a live image is overlayed on the mirror screen.
• While the visitor is focusing his attention to the magic mirror SevenOf Nine is notified of the event
• SevenOfNine takes a shoot, send a tweet and says some sentence, contextualized to the situation.

Below an example of the mirror-related sentenced (from the phrases list document)

42 And keep checking the mirror

43 What does this mirror do

44 It’s a mirror image

45 One time, I broke this mirror. Next week, my parents were separated

46 Hoy! Look in the mirror

47 Observe

48 We observe

As a matter of fact, the mirror is part of the SevenOfNine installation constantly interacting with the mannequin.

Making the Magic Mirror

It is not the first time I build a magic mirror (the last was the Scary Mirror for the past Element14 presents Halloween) but in this case, some complications arose and I had to face to unexpected issues.

The Base Design

The principle to create a "magic" mirror is relatively easy; if it is well designed the impact on the visitors, especially those who have no idea of what will happen, is really nice. I started revisiting my first Magic Mirror project – a Magic Mirror development platform to experiment with the Raspberry PI – to which I have applied some structural changes. In principle, a Magic Mirror is a screen behind a semi-transparent mirror; this makes possible to generate incredibly and surprisingly effects.

The semi-transparent mirror front side is firmly kept in place by a big MDF frame; the remaining visible rectangle hosts laser-cut plywood with a square hole for the Pi Camera lens and the 7" Raspberry PI LCD touch screen (in this case the touch features of the screen are not used). For more details on the structure, look at the Magic Mirror project mentioned above.

Sensing the Visitors

This Art-a-Tronic version uses the following hardware components:

• Raspberry PI 3B+Raspberry PI 3B+
• Raspberry PI 7"Raspberry PI 7"
• Pi CameraPi Camera
• An Ultrasonic distance sensor, part of my Elegoo box of sensors.

Dressing the Sensor in Style

Also for the sensor, I have designed the sensor support with Fusion360 following a similar style of the eye implant and the laser of SevenOfNine.

Above: the quoted sketch of the ultrasonic sensor support for the Magic Mirror.

Below: the 3D rendering of the finished support.

The sequence of images below shows the phases building the support and installing the sensor to its back.

Finishing the Assembly

To close the back of the mirror frame I have 3D printed a series of spacers to fix the back leaving internal air circulation.

Then, after gluing ten spacers on the border of the frame I have closed the mirror with a sheet of MDF 3 mm thick

In the below images: the magic mirror installed and "visible" by SevenOfNine

The Ultrasonic Sensor: Circuit and Software

The ultrasonic sensor HC-SR04HC-SR04 has four pins; the two power pins has been connected to the Raspberry PI 5Vcc and GND. The Raspberry PI pin 7 instead has been connected to the Trigger, to initialize the sensor emitter. The 3.3V provided by the Raspberry PI GPIO outputs are sufficient to activate the circuit. The Echo input instead, has been applied to a voltage divider as the sensor works at 5V while the Raspberry input pins accept 3.3V max. This small circuit has been assembled on a PCB breadboard attached to the connector and connected to the Raspberry GPIO as shown in the image below.

Driving the Ultrasonic Sensor

The principle on how the sensor works is easy: we start the ultrasonic emission changing the state of the trigger signal then see how many microsecond are needed before the echo signal goes high. Knowing the speed of the ultrasonic frequency we can calculate the distance of the subject intercepted by the sensor with the precision of a centimeter over a distance range of about 4 meters. This and other functions are managed by the distance_sensor.py Python script. Below the checkDistance() method that executes the distance calculation and optionally adds a short delay at the end of the reading.

# Executes a sensor reading cycle, inclusing the pause
# for the sensor settle.
# If the distance is between 2 and 400 cm (4 m) the reading
# is valid, else zero is returned.
#
# if isPause is true, after a reading pauses for a fixes period
# else return immediately to the callse
def checkDistance(isPause):
    # Set trigger to high then after 0.1 ms
    # set it to low as the sensor needs a
    # pulse of this length to start
    GPIO.output(PIN_TRIGGER, GPIO.HIGH)
    time.sleep(0.00001)
    GPIO.output(PIN_TRIGGER, GPIO.LOW)

    # define start/end time variables before excturing
    # a distance reading. This is unnecessary, but if
    # the variables are not initialized randomly an error
    # of variable use before its definition may occur.
    # This is probably due in cases when the start time
    # reading is not yet been finished processing and the
    # echo (end time) reading while is already started.
    pulse_start_time = 0
    pulse_end_time = 0

    # Save start time until a transitio does not occur
    # then save the time after the transition from the
    # echo input
    while GPIO.input(PIN_ECHO) == 0:
        pulse_start_time = time.time()
    while GPIO.input(PIN_ECHO) == 1 :
        pulse_end_time = time.time()

    # Calculate the pulse duration then calculate the distance
    # in cm accordingly to the ultrasound speed
    pulse_duration = pulse_end_time - pulse_start_time
    distance = round(pulse_duration * 17150, 2)


    # Ignore out o range calculations, distance should be
    # detected between 2 and 400 cm
    if(distance <= 2 or distance > 400):
        distance = 0

    # Wait the sensor to settle before starting another
    # reading cycle
    GPIO.output(PIN_TRIGGER, GPIO.LOW)
    if(isPause):
        time.sleep(0.5)

    # Return The reading
    return distance

Magic Mirror vs. SevenOfNine

The test of the sensor distance calculation got very good results but this is only a small part of the Magic Mirror software architecture. The entire setup includes other components that were not so easy to make all run concurrently together.

MagicMirror 2

MagicMirror2 by MichMich is the popular project showing well-organized stuff on the screen with a periodical refresh. After installation, it will run as a service starting at boot. After startup, the Raspberry PI display shows the black screen with the configured MagicMirror2 plugins and extensions. There are plenty of possibilities with this framework, including a template and examples of how to create custom modules. In this case, I have only placed the time and date and a set of sentences – contextualized for this specific project – with 30 seconds of refresh time.

All the details and instructions, as well as good examples and how-to, are accessible from the MaigcMirror2 GitHub repository.

Edit media

Dimensions x Small Medium Large Custom

Subject (required) Brief Description Tags (separated by comma) Video visibility in search results Visible Hidden

Parent content

Art-a-Tronic Episode 12

Poster

Upload Preview

Ok

Cancel

For this building, the Magic Mirror architecture has been used as the default circular show aiming to attract the attention of the visitors. But other background activities should run on the Raspberry PI...

The distance_sensor.py Python Script

As a matter of fact, this Python script includes all the methods needed to manage the sensor and the interaction with the other device, SevenOfNine; we will see in detail how the interaction works wirelessly in the next chapter.

When the distance of the visitor is nearby to the mirror the Python script uses the Pi Camera to show an overlayed image preview on the screen

# Shows the camera preview for a number of seconds
def camPreview(sec):
    camera.rotation = 180 # Image is bottom-top

    # Show the camera preview    
    camera.start_preview(alpha=192)
    # And brightness 0
    camera.brightness = 0

    # Progressively increase the brightness
    # to the maximum
    for i in range(0, 50):
        camera.brightness = i
        time.sleep(0.1)
    # Leave the preview visibile for the desired
    # number of seconds
    time.sleep(sec)
    camera-stop_preview()

The image persists for a few seconds then disappear; during this time, the mirror will send a notification to SevenOfNine that – seeing the visitor from his shoulders – says a phrase and, eventually (randomly decided) take a photo and send a tweet of the scene. I should say that the sequence, especially if unexpected, is really surprising and impacting.

Creating a Linux Service

This program should run in the background and the better solution is to leave the Linux system to decide the processes priorities so after some tests instead of setting the script launched on startup (via the Linux cron scheduler) I have created a service that manages the Python script, restart it if for some reason it stops and has a more stable behavior. I have based the service set up on the helpful article How to Run a Python Script as a Service in Raspbian-Jessie; indeed the procedure worked fine also with the last Raspbian-Stretch distribution for the Raspberry PI 3B+

The first step to run the script as a service is creating the corresponding service file: distance_sensor.service.

# Definition of the Python3 script
# distance_sensor as a service to run
# automatically in background
# This service definition file should be copied
# in /lib/systemd/system
# If for any reason the service is aborted, it is
# restarted automatically

[Unit]
Description=Ultrasonic distance sensor
After=multi-user.target

[Service]
Type=simple
ExecStart=/home/pi/distance_sensor.py
Restart=on-abort

[Install]
WantedBy=multi-user.target

After the service file has been copied to /lib/systemd/system we should assign the right attributes to the file:

sudo chmod 644 /lib/systemd/system/distance_sensor.service

Note that if the first line of a Python script is

#!/usr/bin/python3

Assigning to the script executable rights with chmod +x /home/pi/distance_sensor.py the script becomes a command itself and can be run without calling the Python command before. Now we should inform the Linux system to create a new service, accordingly with the details specified in the service file.

sudo systemctl daemon-reload
sudo systemctl enable distance_sensor.service
sudo systemctl start distance_sensor.service

At this point, the last thing we should set up is the communication between the two Raspberry PI, MagicMirror and SevenOfNine.

Connecting more PI on the Same Network

There are many methods we can approach to communicate wirelessly between two Raspberry PI connected on the same network: NodeJS, IP-Socket protocol and many others. I am always searching for the best and more efficient solution and in this case, the two key factors are the kind of communication (speed, frequency, timing) and the amount of data, impacting on the network traffic. In these particular circumstances, there are conditions to search for a lightweight solution:

• The communication is mono-directional, from the Magic Mirror to Seven Of Nine
• Communication is not time-critical
• The amount of data is very reduced: an integer value expressing the subject distance from the mirror (a numeric integer is sufficient)
• The communication occurs randomly but with a low frequency

Taking into account the above scenario I opted for a client-server connection based on the MQTT protocol where SevenOfNine is the broker and subscriber while the Magic Mirror is the publisher.

Installing Mosquitto

First of all, on both the machines I had to install mosquitto, the Python component to manage the MQTT protocol communication. Note that it is also possible to install the version of Mosquitto for the command line but in this case, I preferred to integrate the mechanism inside the already existing Python scripts. As usual, I installed mosquitto via pip, the Python package manager.

sudo pip install paho-mqtt

Setting Up SevenOfNine as the Receiver

The receiver program consists of a Python script that will be launched on startup (for now it is scheduled by cron at boot but also in this case creating a service is a better choice).

#!/usr/bin/python3
import paho.mqtt.client as mqtt
import subprocess

MQTT_SERVER = "localhost"
MQTT_PATH = "magic_mirror"
MIRROR_SERVER = "192.168.0.141"

# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc))

    # Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed.
    client.subscribe(MQTT_PATH)

# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
    print(msg.topic+" "+str(msg.payload))
    # says a phrase
    subprocess.call(["/home/pi/mirror.sh"])

client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message

client.connect(MQTT_SERVER, 1883, 60)

# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
client.loop_forever()

The script is relatively simple; there are two callback: one to manage the connection request by the Magic Mirror and another activated by the on_message event (when the Magic Mirror sends a signal). The main program instantiates the callback functions then start an infinite loop. When a publisher message is received the mirror-specific sentences are managed by the bash script mirror.sh called as a subprocess. Working in the Linux multi-tasking environment this is the best approach to control the logic flow in a smooth way.

Setting Up the Magic Mirror as Publisher

The MQTT protocol is managed by the Magic Mirror as part of the distance_sensor.py Python script running as a service. By the client side, the actions are very simple to do and also very low time-consuming.

GPIO.setmode(GPIO.BOARD)
PIN_TRIGGER = 7 # Distance sensor trigger pin
PIN_ECHO = 11 # Distance sensor echo pin
# Mosquitto server IP Address - Set this value accordingly
mqttServer = "192.168.0.241"
# Mosquitto server channel name
mqttChannel = "magic_mirror"
# Distance detection pause duration after a valid movement
# has been detected and a message has been sent to the mqtt
# server (7of9) in minutes
mqttPause = 2.5
camera = PiCamera()

After defining the MQTT parameters and connection path, as well as the SevenOfNine remote IP address in the initial block as shown above (together with the other initialization constants), the call for sending the message is incredibly simple and easy to integrate into any script.

# Send the distance value to the mqtt server
#
# Param: distance The value to send to the mqtt server
def mqttDistance(distance):
    publish.single(mqttChannel, distance, hostname=mqttServer)

Now it is time to give a personalized character to SevenOfNine but it is another story, See you in the next episode.

Previous Episodes

• Art-a-Tronic Episode 1
• Art-a-Tronic Episode 2
• Art-a-Tronic Episode 3

• Art-a-Tronic Episode 4
• Art-a-Tronic Episode 5
• Art-a-Tronic Episode 6

• Art-a-Tronic Episode 7
• Art-a-Tronic Episode 8
• Art-a-Tronic Episode 9
• Art-a-Tronic Episode 10

• Art-a-Tronic Episode 11

Next Episodes

• Art-a-Tronic Episode 13
• Art-a-Tronic Episode 14
• Art-a-Tronic Episode 15
• Art-a-Tronic Episode 16