IoT live data upload to Plotly and Xively

Previous post: Internet of Things for the tomato greenhouse

Next post: CC3200-LAUNCHXL program uploading data to IoT and hibernating

Plotly is an online service allowing to draw beautiful graphs. It is free of charge. Just register at Plotly. Limitation I read somewhere is that it would allow to store 50 plots.

CC3200-LAUNCHXL roadtesters were adviced to use Plotly to graph Internet of Things data. In this post I show how to upload data to Plotly from a Linux PC bash script and from a CC3200-LAUNCHXL board. 

First experiences with Plotly

Am am trying Plotly for the first time and was impressed by the multitude of different plots it can generate! Ingenious!

I could not find a tutorial for CC3200 and followed a Python tutorial for Raspberry Pi on a Debian Linux PC. It did not work, I got stuck with en error message:

p = plotly.plotly(username, api_key)

TypeError: 'module' object is not callable.

I have previously used cURL bash scripts from Linux to upload json data, but for Plotly could not find documentation how to do it. Probably I could reverse ingeneer HTTP POST code from Arduino example.

Succeeded to automatically send data to Plotly from a Linux PC

Today succeeded to upload sample data automatically to Plotly. It is a first step and later I will try to learn how to use different options.

There are no Plotly examples for CC3200 yet, but there is an example for CC3000 and Arduino. I wanted to learn to upload sample data from a Linux (Debian, Ubuntu) PC using curl POST method in a bash script. First I tried to reverse ingeneer the Arduino code. There were many places causing confusion, for example, should one use /" /r/n in bash script or not.

Chris at Plotly helped by e-mail that should use https://plot.ly/rest/  POST /clientresp. Here is what could type on a Linux PC:

curl -X POST -d @/www/plotly.txt https://plot.ly/clientresp -H "HTTP/1.1" -k -v

curl -X POST -d @/www/plotly.txt http://plot.ly/clientresp -H "HTTP/1.1" -k -v

curl -X POST -d @/www/plotly.txt http://107.21.214.199/clientresp -H "HTTP/1.1" -k -v

Upladed data are saved in a text file  plotly.txt:

----------------------------------------------------------

un=janisalnis&key=p3r69ls929&origin=plot&platform=lisp&

args=[[0, 1, 2], [3, 4, 5], [1, 2, 3], [6, 6, 5]]&

kwargs={"filename": "plot from api", "fileopt": "overwrite", "style": {"type": "bar"},"traces": [1], "layout": { "title": "experimental data"}, "world_readable": true}

----------------------------------------------

HTTP response from connection to Plotly:

==============================

* About to connect() to plot.ly port 443 (#0)

*  Trying 107.21.214.199... connected

* Connected to plot.ly (107.21.214.199) port 443 (#0)

> POST /clientresp HTTP/1.1

> User-Agent: curl/7.21.0 (i486-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6

> Host: plot.ly

> Accept: */*

> Content-Length: 327

> Content-Type: application/x-www-form-urlencoded

< HTTP/1.1 200 OK

< Server: nginx/1.4.6 (Ubuntu)

< Date: Sun, 24 Aug 2014 13:28:47 GMT

< Content-Type: text/html; charset=utf-8

< Transfer-Encoding: chunked

< Set-Cookie: csrftoken=Np6qSSDYEFnfhghYdh9MJjTBiWVYXv1w; expires=Sun, 23-Aug-2015 13:28:47 GMT; Max-Age=31449600; Path=/

* Connection #0 to host plot.ly left intact

* Closing connection #0

* SSLv3, TLS alert, Client hello (1):

{"url": "https://plot.ly/~janisalnis/5", "message": "", "warning": "", "filename": "plot from api", "error": ""}

================================================

When syntaxis was wrong, and it was not working, the answer was

< HTTP/1.1 302 Found

< Content-length: 0

< Location: https://107.21.214.199/

* Closing connection 09

=================================================

Real-time temperature upload to Plotly with date as x-axis

I was uploading from a Linux PC every 5 min. Future improvement would be to count data points: 288 for 24 hours or 2016 for 1 week and to start a new plot afterwards.

https://plot.ly/~janisalnis/20

Many Plotly examples are here:

https://plot.ly/javascript-graphing-library/

The value of `data` variable in the javascript examples should be identical to the value of `args` in the url query string. The value of `layout` in the

examples will be equal to a the value of the `layout` key in `kwargs` in the url query string.

If you would like to use time as x-axis in Plotly then instead of x value supply time in this format  "2013-10-04 22:23:00"

Check out our Time Series example:

https://plot.ly/javascript-graphing-library/time-series/

file plotly.sh:

-------------------------------------------

#Upload to Plotly value with date as x axis

echo 'un=janisalnis&key=p3r69ls929&origin=plot&platform=lisp&args=[["' >/tmp/plotly.txt

t=$(date +"%Y-%m-%d %H:%M:%S")

echo $t  >> /tmp/plotly.txt

echo '"], [' >> /tmp/plotly.txt

t=20.3

echo $t >>/tmp/plotly.txt

echo ']]&kwargs={"filename": "temperature", "fileopt": "extend", "style": {"type": "scatter"},

"layout": {"title": "experimental data"}, "world_readable": "true"}'>>/tmp/plotly.txt

curl -X POST -d @/tmp/plotly.txt http://107.21.214.199/clientresp -H "HTTP/1.1" -k -v

--------------------------------------------------------------

Plotly upload from a html page

Another possibility is to compose such a html page and open it in a webbrowser.

It will not work on CC3200 because no webpage will be open. But it is a good try for syntaxis.

------------------------------------

<html> <body>Test upload to Plotly

<FORM action=" http://107.21.214.199/clientresp" method="post">

version <input TYPE="text" name="version" value="2.3"/><br>

origin  <input TYPE="text" name="origin" value="plot"/><br>

platform <input TYPE="text" name="platform" value="arduino"/><br>

un <input TYPE="text" name="un" value="janisalnis"/><br>

key <input TYPE="text" name="key" value="p3r69ls929"/><br>

args <input TYPE="text" name="args" size="150" value='[{"x": ["2014-08-26 15:18:01"], "y": [25], "type": "scatter", "stream": {"token": ["h2f92op17i", "vwya1hp6ma"],"maxpoints": 20}}]'/><br>

kwargs <input TYPE="text" name="kwargs" size="100" value='{"filename": "streaming API", "fileopt": "extend", "world_readable": true}'/><br>

<INPUT type="submit" value="Send">

</FORM></body></html>

------------------------------------

Server response printed on a web browser screen was:

{"stream-host": "http://stream.plot.ly", "stream-status": "All Streams Go!", "url": "http://107.21.214.199/~janisalnis/19", "filename": "streaming API", "warning": "token not in streamObj or len(token) != 10", "error": "", "message": ""}

Postman - REST Client app for Google Chrome

A very useful application. It allows to see what data are sent in GET and POST requests.

It was very important to verify what POST message actually needs to be sent (Length of the message is not shown, that needs to be calculated manually when programming in C code).

POST /clientresp HTTP/1.1

Host: plot.ly

Cache-Control: no-cache

Maxpoints in Plotly

After uploading 200 data points the plot stopped updating. Had to overwrite all data and start again.

Maxpoints did not work as I expected that older data will be discarded as the new data arrive.

Chris from Plotly repplied that one needs to use two URLs: one for setting up the plot another for streaming.

(Isn't it too complicated?}

Hey Janis --

maxpoints only applies to graphs made through the streaming endpoint at stream.plot.ly

It's a bit confusing, here are the steps to make a streaming graph

1 - Post to /clientresp with the a stream token embedded in every trace of the `data` object - there is a unique token for every trace.

2 - Once the graph has been created, it is ready to stream to. Now, you open a *new* connection to a *different* url: https://stream.plot.ly. This

connection will have a chunked encoding header. You will write to persistent connection. It is in this connection that the `maxpoints`

parameter will be obeyed.

Here is a more in depth example in Python:

http://nbviewer.ipython.org/github/plotly/Streaming-Demos/blob/master/IPython%20examples/http%20details/httplib.ipynb

Or, check out the Arduino API:

1 - make the post to /clientresp:

https://github.com/plotly/arduino-api/blob/master/plotly_streaming_wifi/plotly_streaming_wifi.cpp#L26-L205

2 - if that succeeds, then open a connection to stream.plot.ly or arduino.plot.ly:

https://github.com/plotly/arduino-api/blob/master/plotly_streaming_wifi/plotly_streaming_wifi.cpp#L206-L234

3 - write points to the open connection:

https://github.com/plotly/arduino-api/blob/master/plotly_streaming_wifi/plotly_streaming_wifi.cpp#L294-L301

Let me know if that helps!

Chris

On Tue, Aug 26, 2014 at 5:37 PM, <Janis.Alnis> wrote:

Hello Chris!

I have not managed to use maxpoints. I wanted to show only 10 last points on a plot but all points show up.

Using streaming API I get a strange warning": "token not in streamObj or len(token) != 10"

The plot is updated. I dont understand what is wrong with tokens. Why there are usually 2 tokens used and not just one.

Greetings, Janis.

Code Composer Studio project for CC3200-LAUNCHXL: success

The code was based on a tweaked GetWeather example. Because example was nicely written and commented.

I am attaching a project that is reading onboard temperature sensor, getting time from internet  and uploading a datapoint to Plotly.

Sleep mode is not implemented. Code crashes after some time due to unknown reasons.

My CCSudio code is attached to this blog post. Warning: code is quite messy and as a beginner I have been so stupid to change some include files.

Range of x-axis

To Chris@Plotly:

"Is it possible to scale x axis range to be:

"today" that would be 0-24h,

start "this monday", stop ="this_sunday"

"this_week", meaning Mo-Su

"this_month"  meaning whole month

I have previously been using "rrdtool" database where such options were implemented.

If it is not possible, may be could range time "last 24 hours", "1 week" as used in Xively."

From Chris@Plotly:

"It's not currently possible to update the range like this, although that is

a great suggestion! I just added it to our roadmap."

Summarising Plotly vs Xively

Plotly

+    It is very good for plotting scientific data.

+    Nice visualization possibilities.

+   Can plot multiple curves in a single plot.(does not help if you want to plot temperature and humidity, because units are different).

+   It is possible to generate nice plots by uploading, for example, 200 points at on1ce for the last 6 hours. Often noone cares what has happened longer time ago.

-    Long delay to re-load API every time one wishes to see the plot.

-    Plot not embeddable in a website as a simple png picture.

-    One needs to send date as x in form 2014-08-29 09:11:10. This means CC3200 has to connect to a time server to get date.

-    Older data are not discarded as new data arrive and plots stops updating when maxpoints are reached.

Xively

+   Formerly Pachube/Cosm was the first IoT platform that has become extremely popular (39M feeds!).

+   Automatically assigns date tag to a new datapoint.

+    Automatically manages time axis by showing new points as they come in.

+   One can get a link to a .png file and easily include a graph in any website.

+   One can download data as .csv or .json file with a simple http get command specifying start date and intervals.

+    Good documentation about formatting API for upload to Xively https://xively.com/dev/docs/api/

-    Can not show multiple curves, for example, temperatures from several sensors on a single plot.

-    Free version keeps data only for 3 months.

Conclusion

Plotly is great for visualizing data, but less convenient for continuous upload of streaming IoT data. After spending a week on trying to tweak Plotly, I have not reached a point of satisfaction and switched over to Xively that is much easier to use for IoT data.

How to get back data from Xively

https://xively.com/dev/docs/api/quick_reference/historical_data/

It is very simple using http get. Last current point from Xively:

https://api.xively.com/v2/feeds/39457629.csv?key=qpLG77lHBQVhhJlJ5yAhOAin_CggAaW5tnTVnyGj09k

1,2014-11-08T20:18:44.678712Z, 20.81

2,2014-11-08T20:18:44.678712Z,  1.05

3,2014-11-08T20:18:44.678712Z,  0.60

4,2014-11-08T20:18:44.678712Z,  0.02

5,2014-11-08T20:18:44.678712Z,  1.00

can select just one datastream:

https://api.xively.com/v2/feeds/39457629.json?datastreams=1 

6 hours interval:

https://api.xively.com/v2/feeds/39457629.csv?datastreams=1&duration=6hours&interval=0&limit=1000

1,2014-11-08T14:34:45.977566Z, 20.79

1,2014-11-08T14:35:14.240811Z, 21.03

1,2014-11-08T14:35:43.838346Z, 20.88

1,2014-11-08T14:36:12.814001Z, 20.79

1,2014-11-08T14:38:06.473953Z, 20.86

1,2014-11-08T14:38:37.194363Z, 20.88

.............

1,2014-11-08T20:32:59.927062Z, 20.72

24 hours duration, every 5 minutes:

https://api.xively.com/v2/feeds/39457629.csv?datastreams=1&start=2014-10-01&duration=24hours&interval=100&limit=1000

1week duration, every half an hour

https://api.xively.com/v2/feeds/39457629.csv?datastreams=1&start=2014-10-01&end=2014-10-07&interval=1800&limit=1000&interval_type=discrete

================================================

Uploading data to Xively from processing 2.2.1

There is library to upload to Xively. 

Search and download it.

Put it into lib folder in Processing.

Got Error: can not find library eeml

Solution:

import eeml library file into your project by opening it.

It will be saved in sketchbook under directory code/eeml.jar

================================================