Connecting a Raspberry Pi to AWS IoT Core

2 Oct 2020

I work with companies all over the world who need help building everything from interactive screens to industrial equipment. In recent years, I’ve seen that powering the best customer experiences and most reliable applications means leveraging cloud solutions like AWS IoT Core. Getting started working with cloud services from a hardware background can be an intimidating journey that requires learning a lot of new concepts and best practices. For that reason, I wanted to share my experience and help bridge the gap between those skillsets.

In this tutorial I’ll show you how to connect your Raspberry Pi Device with AWS IoT Core in order to communicate with it securely from the cloud. I’ll cover everything from the hardware and peripherals you need to how to use the AWS IoT Core console UI to control your Raspberry Pi, and everything in between!

Before You Start

Let’s make sure you have all the materials we’re going to be using in this tutorial. For the first section of this guide, you’ll need:

Hardware:

A Raspberry Pi 4 – I’m using the one found in this kit here which comes with a bunch of useful additions like a fan, enclosure, heat sinks, power cables, and micro SD card and it’s very easy to get started with. I have the 4GB starter kit but any of the three options listed should work.
A microSD card with 8GB of space or more (included with the kit above)
A power cable for the Pi (included with the kit above)
An HDMI to micro HDMI cable (included with the kit above)
A microSD card reader for your computer (included with the kit above). You can get a USB adapter like the one in the kit or an SD card adapter like this. If you have a new Mac computer you may want to get one like this.
A USB keyboard
A USB mouse
A monitor with an HDMI port to display the Raspberry Pi’s Screen
Something to integrate with the GPIO pins in your Raspberry Pi later when we configure the Raspberry Pi to interact with the cloud. In my case, I’ll be using a kit like this. That contains everything I need including a breadboard, LEDs, resistors, jumper wires a Raspberry Pi extension board and forty pin cable extension.

Software and Accounts:

An Amazon Web Services (AWS) Account. It’s free and you can create one here. After you create the account, make sure to save your login details somewhere as we’ll need them later.

Step 1: Setup Your Hardware

Before we get our Raspberry Pi connected to AWS we need to finish the process of booting up the machine and installing some dependencies.

If you’re working with the Raspberry Pi kit I linked above, here is an excellent video on how to get your Raspberry Pi setup inside the enclosure, how to add the heat sinks, install the fan, and get the operating system setup for the first time. One thing to note here is that we will eventually be taking the Raspberry Pi back out of this enclosure later to integrate it with other things such as the bread board and LEDs in the kit I link to above.

You should be able to boot up our Raspberry Pi and finish setting it up as shown in the video. After the operating system installs on the device, we connect to a Wi-Fi network, set the other basic settings, and then we can apply the updates it asks us if we’d like to install. When you’re done, reboot the device.

Step 2: Create an IoT Thing in Your AWS IoT Thing Registry

The next step is to register this IoT Device inside of AWS and get the credentials that will allow you to connect back and forth with the cloud.

Start by signing into your AWS Account while you’re on the Raspberry Pi. You can open up the Raspberry Pi OS web browser and go to aws.amazon.com before clicking “create account” or “sign in”.

On the create an account page there should be an option to sign into an AWS Account where you can enter in your credentials.

Once you’re signed in, then navigate to the “Services” dropdown and search for “IoT Core”:

In the IoT Core console, go to Secure > Policies and click “Create a policy”:

This will allow you to create a policy that will give your IoT device permissions to act on things inside the AWS cloud (such as IoT Core MQTT Topics) that allow you to send and receive data between the device and the cloud.

Let’s call our policy raspi-policy.

You’re going to give it some broad permissions that in a production environment we would scope down more substantially to only what the device needs. In this case though, you’re still playing around with everything and we want to be the flexibility to support any actions we might like to try. Because of this, you’ll set the Action as “iot:*” and the resource as “*” along with selecting the “Allow” checkbox.

Your screen should look like the image below:

I’ll break this down a little:

The Allow checkbox means that this policy will allow specific actions on AWS cloud resources. You could also set a policy that explicitly denies permissions to AWS but by default AWS denies all actions unless granted explicitly.
The “iot:*” action means that you are talking about ALL actions under the AWS IoT service umbrella. The asterisk indicates the “all” part here.
The Resource ARN indicates what specific AWS resources you can perform these IoT actions on. In this case, you’re referring to any AWS resource.

For the purposes of a device that is going to be living on your desk this might be acceptable. But you’re probably already starting to see how this policy would be too broad to be applied in a production setting. If you kept it without modification it could allow any of our devices make requests to other devices or interfere with things they shouldn’t – this is far from ideal.

For this demo though you can leave these values and the allow checkbox as shown above and press “Create”.

Next, go to Manage > Things and press “Register a Thing”:

AWS IoT Core provides lots of great tooling for managing IoT “Things” that can help you sort things in to “Types” or “Groups”. You can then use these settings to determine permissions these things might require and how we might want to interact with them. This can also be combined with some more appropriate permissions (as mentioned above) to help secure your IoT environments.

For right now though, you can use the easiest method of getting started and press “Create a Single Thing” and name your thing “raspi”:

Step 3: Secure Your Connection to AWS IoT

On the next page you have the option to add a certificate for your thing. This is why I suggested you log into AWS while still on the IoT device: because for the device to speak to the cloud, you’ll need the certificate on it locally.

At this point it might make sense to briefly summarize how public key cryptography works, how it applies to IoT devices and how that differs from typical web-based public key scenarios.

Securing Websites with Public Key Cryptography

If you have a website and you want to allow users to connect securely, you can request a certificate for your website from a Issuing Certificate Authority or CA. This CA goes through a very stringent process to securely create a public and private key and store the private key securely. There is then another layer called a Root CA which signs the issuing certificate authority’s certificate.

These Root CAs then work with hardware and software vendors like web browsers and consumer electronics like laptop manufacturers in order to bundle public keys in with the devices they sell. This allows a person to visit your website, review the website certificate you provide, see who signed it, review that CA certificate and check it against the Root CA certificate that was already bundled on their machine. After confirming the Issuing Certificate Authority’s certificate is good it will then use that certificate to verify yours and then establish a connection with your site using that information.

While this brief explanation is ignoring a few other layers of public key infrastructure (PKI) around how revoking certificates works and newer changes to PKI related to public listings of issued certificates, this is essentially how securing website works on the web.

But what happens when you are making the devices yourself?

Public Key Cryptography and Securing IoT Devices

Let me start by saying that there are other mechanisms for trying to implement security on IoT devices. Access keys or tokens can be used on embedded on devices to perform some of the same operations.

However, these methods rely on symmetric encryption and are considered less secure because the key must be protected on both the cloud and the device.

Additionally, when using access keys or tokens like this they are more likely to be abused and stored in plaintext or hardcoded in software which makes them more accessible to attackers. For these and other reasons, the public key cryptography approach is preferred

When connecting our device to AWS IoT we can use public key cryptography to do so. Within that, you have a few options for generating certificates:

Let’s see how these options work:

One-click certificate creation (recommended) – AWS generates a public/private key that we’ll be using for your device. It then takes the public key and makes a Certificate Signing Request or CSR that it signs with its own CA Private Key to generate a certificate we can use on our device. This all happens in the background and then AWS allows us to download the public/private key and the certificate it created for us so we can authenticate our device later on. We also download a Root CA certificate from AWS that we use to authenticate our connection with AWS.
Create with Certificate Signing Request – With this approach, we have our own public and private key already, either on our laptop or maybe provisioned already on our IoT device. We then use our keys to create a certificate signing request on our end before sending that CSR to AWS. AWS will then send us back a certificate which we can use in combination with our private key on the device. But like the first option, this requires us to have permissions to call AWS APIs for every new certificate we want to generate.
Use my Certificate – Alternatively, we can register our own certificate authority certificate so that we can sign one or many kinds of devices. Then, we can use that certificate to sign new certificate requests without having to make a request to AWS. This is the most common approach when working on production services where we are creating a large number of devices where we want to provision a unique certificate for each.

For right now, given that we’re only provisioning one device and that we’re not deploying this device into the wild beyond our home networks.

After you click this, you should download all the created certificates to your Raspberry Pi device:

Download all three of these files and then make sure to press “Activate”. This is key because AWS has already used the public key to help generate a certificate for you, and you will mainly be using the certificate and the private key to authenticate and sign your requests to AWS.

Also, on this page you need to open the link for the root CA download which is right above the “Activate/Deactivate” button:

From this page, you can download the Amazon Root CA 1. Right click the link and click “Save link as” and save it to the Raspberry Pi. This certificate will allow you to trust AWS IoT itself.

Once you’ve downloaded the Root CA certificate along with the three other certificates, you should have four total files on your Raspberry Pi. Go back to the “Certificate created!” page and make sure to press “Activate” if you haven’t already done so.

Then you can press the “Attach a policy” button. On this page, you can select the checkbox next to the “raspi-policy” we created earlier and then press “Register thing”:

Now you should have the certificates you need on your device to have it start talking to AWS and it should show up in the Things portion of the IoT Core:

Next, you need to download some dependencies and, yes, write some code!

Step 4: Installing Dependencies on the Raspberry Pi

Raspberry Pi OS should come with Python 3 installed on it already. If you’re working with another operating system on the Raspberry Pi, you will want to install Python 3 with at least version 3.5 or later.

Raspberry Pi OS also comes with git installed so if you’re using a different operating system, you’ll want to install that too.

The next step will be to install the AWS IoT SDK for Python. Before you do this, you’ll need to gather some dependencies that are not included on our Raspberry Pi OS. You’ll do this by running the following commands and press Y when prompted to add the dependencies to your system:

sudo apt-get update

sudo apt-get install cmake

sudo apt-get install libssl-dev

After you run these commands, you should be able to use pip3 to install the AWS IoT SDK:

pip3 install awsiotsdk

This installation process might take a few minutes but when it’s finished, you can get the code you need for this tutorial. To do this, run:

git clone https://github.com/Witekio/aws-iot-core-raspberry-pi-demo.git

Step 5: Syncing Your Raspberry Pi with an AWS IoT Shadow

Now that you have the code, you’ll need both code files in the same directory as your certificates.

cd raspberry-pi-aws-iot

mv ~/Downloads/<your-certificate-name>-certificate.pem.crt .

mv ~/Downloads/<your-certificate-name>-private.pem.key .

mv ~/Downloads/AmazonRootCA1.pem .

Replace “<your-certificate-name>” with the actual first part of your certificate names, of course!

At this point, you might also want to change the permissions on the downloaded files to make them only readable to the current user. You can do this by running the following commands in the same directory as the certificates:

chmod 400 <your-certificate-name>-certificate.pem.crt

chmod 400 <your-certificate-name>-private.pem.key

chmod 400 AmazonRootCA1.pem

Again, remember to replace the part of the commands with <these> in them.

In a production scenario, you probably wouldn’t even provision these files in this way. Instead, you would have a process for storing sensitive information like this inside something like a Trusted Platform Module or TPM. This allows you to be more confident that the certificates can’t be stolen off the device and used in some other system. While devices might still be compromised, with a TMP it becomes much harder to steal the keys and use them elsewhere without you noticing.

The last thing we’ll need to get before syncing our device to the cloud is the endpoint URL for our Thing. In order to find the endpoint URL you can click into the details of your new AWS IoT Thing and look under the “Interact” section. You’ll notice in this section there is an HTTPS URL and an MQTT section:

This is because AWS IoT supports connections using multiple protocols including HTTPS, MQTT and MQTT over WebSocket, for a full list you can refer to the documentation here.

On this page, copy the endpoint value under HTTPS down and use it, along with the names of the certificates you have in the present directory, to replace the “<your-certificate-name>” and “<endpoint>” sections of this command:

python3 shadow.py --endpoint <endpoint>-ats.iot.us-east-1.amazonaws.com \
--cert <your-certificate-name>-certificate.pem.crt \
--key <your-certificate-name>-private.pem.key \
--root-ca AmazonRootCA1.pem \
--thing-name raspi \
--shadow-property light

After you update it, it would look something like this:

python3 shadow.py --endpoint a1btsyszhken4p-ats.iot.us-east-1.amazonaws.com \
--cert fdd885eb3a-certificate.pem.crt \
--key fdd885eb3a -private.pem.key \
--root-ca AmazonRootCA1.pem \
--thing-name raspi \
--shadow-property light

Before you run this command, go over to the “Shadows” section of your “raspi” thing and click “Classic Shadow”. You should see something like this:

Now, run the command above and see what happens on your device.

You should see an output that describes the process of syncing with AWS IoT and then a prompt that allows you to update the state of the device from the device itself.

Type in “OFF” and press enter, then see what happens to the IoT Code Shadow state:

Now, click the “Edit” option in AWS IoT Core and update your shadow by editing the “Desired” property of “light” to “ON”. You should see the device report the new updated state in the terminal!

All of this means you’ve successfully connected our device to AWS IoT Core. In the background what’s happening here is that our “shadow” here has a desired and reported value. When we update the desired value the script running on the device polls an MQTT topic provided by AWS IoT Core to see if anything has changed.

If it sees a new “desired” value then it knows it needs to update the state on the device. This triggers the script on the device to update the “reported” value in the MQTT shadow for the device. In the long run, this will allow you to act on changes to the desired or reported state of a device and manage connections between the cloud and devices more effectively, especially in situations when network access cannot be maintained – something common for many IoT devices.

Now you need to set up our hardware and have the cloud control an LED on your device!

Step 6: Hardware Configuration and GPIO Testing

For this demo, I’ll be using some of the equipment that came in my Freenove kit linked in the first section of this guide. Specifically, I’ll be using:

A breadboard
A Raspberry Pi GPIO Extension Board
A 40-pin extension cable
One red LED
One 10K Ohm resistor
Two jumper wires

For the hardware portion of this tutorial, I’ll be remotely controlling one of the Raspberry PI’s GPIO pins in order to provide power for an LED when the “light” property of the device is set to “ON”. For right now, shutdown and unplug your Raspberry Pi so you can setup the breadboard and the LED.

I’ll be using the same pin configuration as the first project shown in the Freenove Ultimate Starter Kit for Raspberry Pi here.

You can download the project details and tutorial in PDF format from here.

Inside that ZIP file is a Tutorial.pdf file that includes the hardware configuration for the project we’re doing from page 52 to page 55 in “Chapter 1 LED”. If you haven’t done much work with hardware before, make sure that you read the instructions carefully and be sure to pick an appropriate resistor for your LED. In my case, the 10K Ohm resistor was appropriate for the red LED.

You can turn the Raspberry Pi back on after you have the breadboard configured as shown in the tutorial photo on page 55 here:

At this point, you can test the hardware configuration using some of the code we got from GitHub earlier. Change directories into the GitHub repo you cloned and make sure you see the “raspi.py” file.

With the terminal in the same directory as “raspi.py”, type “python3” and press enter in order to get into the python3 interpreter. Then, run the following commands (without the >>>) one at a time and wait for a second or two between each to look at your LED:

>>> from raspi import *
>>> setup()
>>> turn_on()
>>> turn_off()
>>> destroy()

How did it go? Did you see the LED turn on and off when you ran those corresponding commands?

If not, you might have a different hardware configuration than I do. This code assumes that you’ve followed the guide above and used GPIO 17 in order to power your LED. If you’re using some other GPIO then you will need to either change the script or update your hardware configuration to make it work. Make sure to leave the device off if you need to change some of the wiring!

If this works for you, we can proceed to the next step: doing this all from the cloud!

Step 8: Controlling the Raspberry Pi’s GPIO from the Cloud

Now that we have your hardware setup, our Raspberry Pi dependencies installed, and your device can connect to AWS IoT Core, you should be able to control the LED from the cloud!

The first step is to slightly modify the shadow.py file. Start by opening it up in the text editor on the Raspberry Pi. The easiest way to do this from the terminal is by running xdg-open shadow.py

Then, scroll down a little in the file and change the line that says “HARDWARD_CONFIGURED = False” to this:

“HARDWARD_CONFIGURED = True”

Make sure to use a capital “T” for “True”! Then save the file and go find that command you ran earlier that looks something like this except with your certificate names:

Run this command again.

Now when it prompts you for the “light” property enter “ON” or “OFF” from the Raspberry Pi terminal and you should see the LED turn on and off!

But for the best part, try updating the Shadow inside of the AWS IoT Console and change the desired state for the light property to “ON” or “OFF” and watch it be reflected in your LED.

Nice work! By now you’ve managed to sync the state of your Raspberry Pi between the device itself and AWS IoT Core. You’ve also used Python to control the GPIO pins on your Raspberry Pi and used it to enable the controlling of those pins from AWS. From here, you should feel comfortable doing more projects that combine AWS IoT with hardware connected to your device!