Here is a list of the posts in this challenge
Gr0G - 03 - High-pressure system design
Gr0G - 07 - Playing with the Gertbot
Gr0G - 11 - Building the box (2)
Gr0G - 12 - Building the high-pressure system
Gr0G - 13 - Building the high-pressure system (2)
Source code available at https://github.com/ambrogio-galbusera/gr0g, https://github.com/ambrogio-galbusera/gr0g-ble-android and https://github.com/ambrogio-galbusera/gr0g-ble
In this challenge I spent a lot of time cutting, drilling and bolting and, for this reason, the software side of myself has been complaining a lot. So, to calm it down a little bit, and considering the relaxed rhythm the nature takes for seeds germination, I decided to implement a remote user interface to the Gr0G.
The only requirement I have is to be loyal to my first principle: build a device that is completely self-contained. The obvious choice was to configure the Raspberry Pi's Wifi to act as an access point, install a web server and create some web pages with one of the many tools available out there, but this was not going to be very satisfactory. Why not explore something new (at least for me), like BLE?
BLE
Raspberry Pi supports Bluetooth Low Energy because they integrate a combo Wi-Fi + Bluetooth chipset. The exact chipset supported varies depending on the board being used. For example, Raspberry Pi 4 has a CYW43455 chipset with support for 802.11ac and dual band (2.4GHz and 5GHz), whereas older Raspberry Pi boards used BCM43438 chipset from Broadcom. But... here comes the first problem: Raspberry Pi 4 uses this hardware configuration
The Bluetooth controller is an external chip and communicates with the main CPU on a UART which is, unfortunately, the same UART used by the Gertbot board. For this reason, I had to disable the Bluetooth interface (see this post for further details). Luckily, I have a USB Bluetooth dongle that I can use as a replacement.
BLE Network architecture
In a typical BLE network, there is a central device (e.g. a smartphone) and one or more peripheral device (e.g. the Gr0G box, or your smartwatch). The way the central device interacts with the peripheral device is defined by GATT (Generic Attribute Profile). GATT is based on the concept of ATT (Attribute Protocol)
ATT
ATT defines how a server exposes its data to a client and how this data is structured. There are two roles within the ATT:
- Server: this is the device that exposes the data it controls or contains (in our case, the Gr0G box). It is the device that accepts incoming commands from a peer device and sends responses, notifications, and indications
- Client: this is the device that interfaces with the server with the purpose of reading the server’s exposed data and/or controlling the server’s behavior. It is the device that sends commands and requests and accepts incoming notifications and indications. A mobile phone that connects to the Gr0G box and reads its temperature value is acting in the Client role.
The data that the server exposes is structured as attributes. An attribute is the generic term for any type of data exposed by the server and defines the structure of this data. For example, services and characteristics (both described later) are types of attributes. Attributes are made up of the following properties:
- Attribute type (Universally Unique Identifier or UUID): this is a 16-bit number (in the case of Bluetooth SIG-Adopted Attributes), or 128-bit number (in the case of custom attribute types defined by the developer, also sometimes referred to as vendor-specific UUIDs). For example, the UUID for a SIG-adopted temperature measurement value is 0x2A1C SIG-adopted attribute types (UUIDs) share all but 16 bits of a special 128-bit base UUID:00000000-0000-1000-8000-00805F9B34FBThe published 16-bit UUID value replaces the 2 bytes in bold in the base UUID. A custom UUID, on the other hand, can be any 128-bit number that does not use the SIG-adopted base UUID. For example, a developer can define their own attribute type (UUID) for a temperature reading as: F5A1287E-227D-4C9E-AD2C-11D0FD6ED640. One benefit of using a SIG-adopted UUID is the reduced packet size since it can be transmitted as the 16-bit representation instead of the full 128-bit value.
- Attribute Handle: this is a 16-bit value that the server assigns to each of its attributes — think of it as an address. This value is used by the client to reference a specific attribute and is guaranteed by the server to uniquely identify the attribute during the life of the connection between two devices. The range of handles is 0x0001-0xFFFF, where the value of 0x0000 is reserved.
- Attribute Permissions: permissions determine whether an attribute can be read or written to, whether it can be notified or indicated, and what security levels are required for each of these operations. These permissions are not defined or discovered via the Attribute Protocol (ATT) but rather defined at a higher layer (GATT layer or Application layer).
GATT
GATT is defined in terms of
- Services: a grouping of one or more attributes, some of which are characteristics. For example, the service BatteryService has a characteristic name BatteryLevel. A service also contains other attributes (non-characteristics) that help structure the data within a service (such as service declarations, characteristic declarations, and others). There are two types of services
- Primary: represents the primary functionality of a device
- Secondary: provides the auxiliary functionality of a device and is referenced (included) by at least one other primary service on the device
- Characteristics: a characteristic is always part of a service and it represents a piece of information/data that a server wants to expose to a client (e.g. the battery level characteristic represents the remaining power level of a battery). The characteristic contains other attributes that help define the value it holds:
- Properties: represented by a number of bits and which defines how a characteristic value can be used. Some examples include: read, write, write without response, notify, indicate.
- Descriptors: used to contain related information about the characteristic Value. Some examples include: extended properties, user description, fields used for subscribing to notifications and indications, and a field that defines the presentation of the value such as the format and the unit of the value.
- Profiles: is a grouping of services that are related to a specific behavior
These concepts are used specifically to allow hierarchy in the structuring of the data exposed by the server.
The GATT defines the format of services and their characteristics, and the procedures that are used to interface with these attributes such as service discovery, characteristic reads, characteristic writes, notifications, and indications.
GATT takes on the same roles as the Attribute Protocol (ATT). The roles are not set per device — rather they are determined per transaction (such as request ⟷ response, indication ⟷ confirmation, notification). So, in this sense, a device can act as a server serving up data for clients, and at the same time act as a client reading data served up by other servers (all during the same connection).
There are a number of adopted services and their characteristics, but for the Gr0G BLE services we will implement a simple service with the following characteristics
- Temperature [R]
- Temperature Setpoint [R / W]
- Humidity [R]
- Humidity Setpoint [R / W]
- Light [R]
- Light control [R / W]
BlueZ
BlueZ is the Open Source Bluetooth stack.
Raspbian already comes with Bluetooth packages that can be downloaded, but I will install BlueZ form source code, just as an exercise and to explore a little bit the implementation details.
I download BlueZ source code (BlueZ 5.50 which is the latest version at time of writing)
wget http://www.kernel.org/pub/linux/bluetooth/bluez-5.50.tar.xz
tar xvf bluez-5.50.tar.xz
Let's now install the libraries required to build the source code
sudo apt-get update
sudo apt-get install -y libusb-dev libreadline-dev libglib2.0-dev libudev-dev libdbus-1-dev libical-dev
Now we can build and install BlueZ
cd bluez-5.50
./configure --enable-library
make -j4
sudo make install
To check that eveything is working as expected, we can run btmon
NOTE: here I am working with the wireless chip mounted on the Raspberry Pi 4 board, namely the CYW43455
D-Bus
An application can query BlueZ or request commands by means of D-Bus, so it worth the while to spend some words about this technology.
D-Bus is a system for interprocess communication (IPC). Architecturally, it has several layers:
- A library, libdbus, that allows two applications to connect to each other and exchange messages.
- A message bus daemon executable, built on libdbus, that multiple applications can connect to. The daemon can route messages from one application to zero or more other applications.
- Wrapper libraries or bindings based on particular application frameworks. For example, libdbus-glib and libdbus-qt. There are also bindings to languages such as Python. These wrapper libraries are the API most people should use, as they simplify the details of D-Bus programming. libdbus is intended to be a low-level backend for the higher level bindings. Much of the libdbus API is only useful for binding implementation.
D-Bus messages are all sent to a broker: the dbus-daemon. An application sends a message to the dbus-daemon, and the daemon forwards the message to other connected applications as appropriate. The daemon is, a matter of fact, a sort of router.
The bus daemon has multiple instances on a typical computer. The first instance is a machine-global singleton (called "system"). This instance has heavy security restrictions on what messages it will accept, and is used for systemwide communication. The other instances are created one per user login session (called "session"). These instances allow applications in the user's session to communicate with one another
The systemwide and per-user daemons are separate. Normal within-session IPC does not involve the systemwide message bus process and vice versa.
Each server application can be uniquely identified by an object path. The idea of an object path is that higher-level bindings can name native object instances, and allow remote applications to refer to them. The object path looks like a filesystem path, for example an object could be named /org/kde/kspread/sheets/3/cells/4/5
Each object has members; the two kinds of member are methods and signals.
- Methods are operations that can be invoked on an object, with optional input (aka arguments or "in parameters") and output (aka return values or "out parameters").
- Signals are broadcasts from the object to any interested observers of the object; signals may contain a data payload.
Each object supports one or more interfaces. Interfaces define the type of an object instance.
Raspbian includes an tool (bluetoothctl) that can be used to get familiar with BlueZ interface (if missing, you can install with sudo apt-get install bluetooth). If you are interested in BlueZ D-Bus interface, you could run the command
sudo dbus-monitor --system "destination='org.bluez'" "sender='org.bluez'"
in a separate window and check the messages exchanged between bluetoothctl and BlueZ
BLE service
I developed the BLE service using Python. The gr0g-ble repository includes just two files:
- ble.py: a file created from BlueZ examples and this example.This file uses the BlueZ D-Bus interface whose specification is available here
- app.py: the implementation of the BLE service
Thanks to the classes in ble.py, the code in app.py is quite easy to read
First, let's set the mainloop object for the D-Bus communication. This loop will take care of processing D-Bus events
dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
Now let's get the bus we will use to communicate with BlueZ
bus = dbus.SystemBus()
Then, we need to find an object that implements the org.bluez.GattManager1 interface
adapter = find_adapter(bus)
Now we can get a D-Bus proxy object, that is to say a local reference to the actual D-Bus object we will use to invoke methods
adapter_obj = bus.get_object(BLUEZ_SERVICE_NAME, adapter)
We can now power on the interface
adapter_props = dbus.Interface(adapter_obj, "org.freedesktop.DBus.Properties")
adapter_props.Set("org.bluez.Adapter1", "Powered", dbus.Boolean(1))
With the proxy, we can also get the object that handles the implements the org.bluez.LEAdvertisingManager1 and handles service advertisements...
ad_manager = dbus.Interface(adapter_obj, LE_ADVERTISING_MANAGER_IFACE)
...and register our service. This code tells BlueZ where the advertisement object is (advertisement.get_path()) and sets the handlers for replies and errors
advertisement = Gr0GAdvertisement(bus, 0)
ad_manager.RegisterAdvertisement(
advertisement.get_path(),
{},
reply_handler=register_ad_cb,
error_handler=register_ad_error_cb,
)
To implement the service, let's first start to implement characteristic using the class Characteristic in ble.py. This class takes care of creating the objects that represent the characteristics on the system D-Bus that implement the org.bluez.GattCharacteristic1 interface.
Each characteristic has an UUID and a description, which is added as a property of the characteristic in the class constructor
class TemperatureSetpointCharacteristic(Characteristic):
uuid = "00002a06-0000-1000-8000-00805f9b36fc"
description = b"Get/set temperature setpoint"
def __init__(self, bus, index, service):
Characteristic.__init__(
self, bus, index, self.uuid, ["read", "write"], service,
)
self.value = []
self.add_descriptor(CharacteristicUserDescriptionDescriptor(bus, 1, self))
The ReadValue method is invoked when the client requests a read on a characteristic. The method reads data from Gr0G main application through D-Bus and returns to the BLE client. the gr0g_bus is a global variable (see below) and you may recognized in the code the object path (/gr0g) and the interface (com.ag.gr0g)
def ReadValue(self, options):
logger.info("Temperature setpoint read: " + repr(self.value))
try:
remote_object = gr0g_bus.get_object("com.ag.gr0g", "/gr0g")
status = remote_object.status()
self.value = bytearray(struct.pack("d", float(status["temperature_setpoint"])))
except Exception as e:
logger.error(f"Error getting status {e}")
return self.value
In the same way, the WriteValue method is invoked when the BLE client requests a write operation. Again, the new data is passed to the Gr0G main application using D-BUS invokation
def WriteValue(self, value, options):
logger.info("Temperature write: " + repr(value))
cmd = bytes(value)
# write it to machine
logger.info("writing {cmd} to temperature_setpoint")
data = {"cmd": "temperature_setpoint", "value": struct.unpack("d", cmd)[0]}
try:
remote_object = gr0g_bus.get_object("com.ag.gr0g", "/gr0g")
status = remote_object.cmd(data)
logger.info(status)
except Exceptions as e:
logger.error(f"Error updating machine state: {e}")
raise
Finally, we can initialize the service by extending the Service class
class Gr0GS1Service(Service):
GR0G_SVC_STATUS_UUID = "00001802-0000-1000-8000-00805f9b38fb"
def __init__(self, bus, index):
Service.__init__(self, bus, index, self.GR0G_SVC_STATUS_UUID, True)
self.add_characteristic(LightCharacteristic(bus, 1, self))
self.add_characteristic(LightControlCharacteristic(bus, 2, self))
self.add_characteristic(TemperatureCharacteristic(bus, 3, self))
self.add_characteristic(TemperatureSetpointCharacteristic(bus, 4, self))
self.add_characteristic(HumidityCharacteristic(bus, 5, self))
self.add_characteristic(HumiditySetpointCharacteristic(bus, 6, self))
and expose the service
app = Application(bus)
app.add_service(Gr0GS1Service(bus, 2))
D-Bus service
A D-Bus service is not a strict requirement. I could have integrated the code that implements the BLE service in the Gr0G's main application. However, the use of a separate process for this very specific feature brings all the advantages that are intrinsic in a modular design: reliability, clean interfaces, maintainability
The Gr0G main application will expose a D-Bus interface named com.ag.gr0g with just two methods
- status: returns a json object with the following fields
- temperature
- temperature_setpoint
- humidity
- humidity_setpoint
- light
- cmd: allows a client to execute a command. Supported command are
- setlight: switch on or off light, thus overridng automatic control
- temperature_setpoint
- humidity_setpoint
The class that implements the D-Bus interfase is TheGr0G and can be found in dbusservice.py
Some special decorations are needed to indicate that a method is exposed on the D-Bus
@dbus.service.method("com.ag.gr0g",
in_signature='', out_signature='a{sd}')
def status(self):
With this decoration, we are saying that
- the method is part of the com.ag.gr0g interface
- there are no input parameters (in_signature='')
- the method returns a set of tuples of (string, float) (out_signature='a{sd}')
For details about the supported types, you can refer to the d-bus tutorial
Because D-Bus service requires a special access to the mainloop, I had to move the Gr0G application logic to a separate thread
app = App()
def mainThread(runEvent):
while runEvent.is_set():
app.process()
app.handleScreens()
time.sleep(0.1)
return
try:
dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
session_bus = dbus.SessionBus()
name = dbus.service.BusName("com.ag.gr0g", session_bus)
object = TheGr0g(app.ds, app.sett, app.lightc, session_bus, '/gr0g')
runEvent = threading.Event()
runEvent.set()
mainT = threading.Thread(target=mainThread, args=(runEvent,))
mainT.start()
mainloop = gobject.MainLoop()
mainloop.run()
I can now test the D-Bus service using the dbus-send command line utility
The BLE service can be scanned and browsed by means of any standard BLE scanner. But I am already working on an Android app with a custom UI. More details in the next post!
Top Comments