Microchip AVR-IoT question: How to correctly format a POST request

Question

I've been working with the Microchip mini dev board in the Arduino environment [Side note, very nice board so far]. I am converting a program that currently runs using Wifi to this board. It does a POST to a service called Scriptr. The original stripped down code that does the POST is: 
 
 // Working code for Wifi (stripped of error checking and other extraneous stuff)
 const char* ssid = "myssid";
 const char* password = "mypw";
 String mymsg = "TestMSG";
 const char* bearer = "xxxxx...xxxxx"; // API Key

 WiFiSSLClient client;

 WiFi.begin(ssid, password);
 client.connect("api.scriptrapps.io", 443)) 

 client.print(String("POST ") + "/SNDSMS/?sms=" + mymsg + " HTTP/1.1
" + 
 "Host: api.scriptrapps.io
" + 
 "Authorization:bearer " + bearer + "
" + // 23 + 68
 "Content-type: application/x-www-form-urlencoded
" +
 "Connection: close

");
 
 The corresponding code for the Microchip mini dev board that I've tried is: 
 Lte.begin();
 HttpClient.configure(host, 443, true); // Have tried true and false for TLS

 HttpResponse response;

 // Following doesn't work
 response = HttpClient.post("/SNDSMS/","?sms=TestMSG HTTP/1.1
Host: api.scriptrapps.io
Authorization:bearer xxxxx...xxxxx
Content-type: application/x-www-form-urlencoded
Connection: close

" ); 

 // Following doesn't work either
 response = HttpClient.post("/SNDSMS","/?sms=TestMSG HTTP/1.1
Host: api.scriptrapps.io
Authorization:bearer xxxxx...xxxxx
Content-type: application/x-www-form-urlencoded
Connection: close

" ); 

 // Following doesn't work either
 response = HttpClient.post("api.scriptrapps.io","/SNDSMS/?sms=TestMSG HTTP/1.1
Host: api.scriptrapps.io
Authorization:bearer xxxxx...xxxxx
Content-type: application/x-www-form-urlencoded
Connection: close

" ); 

 
 I ran https_configure_ca so HTTPS should be enabled. 
 I"ve tried numerous guesses at how to format the Post request for the board, all either result in a response of 400 or 0. 
 The board is connecting just fine to AT&T and obviously sending the request to Scriptr or I wouldn't be getting 400 errors (at least some of the time). 
 Can anyone point me to how I should be formatting the POST request so that it does what the Wifi version does? 
 Thank you, Ira.

Gough Lui · Answer

I have modified the version of the library to allow for the extra parameters and it seems to work. First of all, the sample test code: #include #include #include #include #include #define DOMAIN "xxxxxxxx" # THIS IS SET TO YOUR SEVER void setup() { LedCtrl.begin(); LedCtrl.startupCycle(); Log.begin(115200); Log.info("Starting HTTP example"); // Start LTE modem and connect to the operator if (!Lte.begin()) { Log.error("Failed to connect to the operator"); return; } Log.infof("Connected to operator: %s ", Lte.getOperator().c_str()); // --- HTTP --- Log.info("---- Testing HTTP ----"); if (!HttpClient.configure(DOMAIN, 80, false)) { # I only tested with HTTP Log.info("Failed to configure http client "); } Log.info("Configured to HTTP"); uint8_t bodymsg[] = ""; // This is the payload uint32_t bodylen = strlen((const char*)bodymsg); // This is the payload length HttpResponse response = HttpClient.post2("/SENDSMS?sms=TESTMSG",bodymsg,bodylen,"application/x-www-form-urlencoded","Authorization: bearer 1234567890",1,30); Log.infof("POST - status code: %u, data size: %u ", response.status_code, response.data_size); // Add some extra bytes for termination String body = HttpClient.readBody(response.data_size + 16); if (body != "") { Log.infof("Body: %s ", body.c_str()); } } void loop() {} The result is the following from my test server: POST /SENDSMS?sms=TESTMSG HTTP/1.1 Host: xxxxxxxxx Authorization: bearer 1234567890 Content-Type: application/x-www-form-urlencoded Connection: close Content-Length: 0 This is pretty much exactly what you would want since your message is URL encoded, the body of the request has zero length. The function prototype for the updated function (I've called post2) is as follows: HttpResponse post2(const char* endpoint, const uint8_t* buffer, const uint32_t buffer_size, const uint8_t* post_param, const uint8_t* extra_hdr, const uint32_t disconnect, const uint32_t timeout); In order to add this to your library, you will need to locate the AVR-IoT-Cellular library folder, navigate to the src folder and edit http_client.h and http_client.cpp . I have bolded my additions. My http_client.h now looks like this: /** * @brief HTTP client for REST calls. */ #ifndef HTTP_CLIENT_H #define HTTP_CLIENT_H #include #include typedef struct { uint16_t status_code; uint32_t data_size; } HttpResponse; class HttpClientClass { private: HttpClientClass(){}; public: static HttpClientClass &instance(void) { static HttpClientClass instance; return instance; } enum StatusCodes { STATUS_OK = 200, STATUS_NOT_FOUND = 404, STATUS_INTERNAL_SERVER_ERROR = 500, }; /** * @brief Sets up the HTTP client with a host and port. * * @param host Can either be a host name resolved with DNS or an actual * server address in the form of "xxx.xxx.xxx.xxx". * @param port Port of the host, e.g. 80 or 443 (for HTTPS). * @param enable_tls Use tls, required for HTTPS. * * @return True if operation was successful. */ bool configure(const char *host, const uint16_t port, const bool enable_tls); /** * @brief Issues a post to the host configured. Will block until operation * is done. */ HttpResponse post(const char *endpoint, const uint8_t *buffer, const uint32_t buffer_size); // MODIFIED FUNCTION BY GOUGH LUI TO ENABLE ADDITIONAL POST HEADERS HttpResponse post2(const char* endpoint, const uint8_t* buffer, const uint32_t buffer_size, const char* post_param, const char* extra_hdr, const uint32_t disconnect, const uint32_t timeout); /** * @brief Issues a post to the host configured. Will block until operation * is done. * * @param messsage Needs to be a regular c string which is null terminated. * */ HttpResponse post(const char *endpoint, const char *messsage); /** * @brief Issues a put to the host configured. Will block until operation is * done. */ HttpResponse put(const char *endpoint, const uint8_t *buffer, const uint32_t buffer_size); /** * @brief Issues a put to the host configured. Will block until operation is * done. * * @param messsage Needs to be a regular c string which is null terminated. */ HttpResponse put(const char *endpoint, const char *message); /** * @brief Issues a get from the host configured. Will block until operation * is done. The contents of the body after the get can be read using the * readBody() function after. */ HttpResponse get(const char *endpoint); /** * @brief Issues a head from the host configured. Will block until operation * is done. */ HttpResponse head(const char *endpoint); /** * @brief Issues a delete from the host configured. Will block until * operation is done. */ HttpResponse del(const char *endpoint); /** * @brief Reads the body of a response after a HTTP call. Note that the * range for the buffer_size has to be between 64-1500. This is a limitation * from the Sequans LTE module. So if the data is larger than that, * multiple calls to this function has to be made. * * @param buffer Destination of the body. * @param buffer_size Has to be between 64-1500. * * @return bytes read from receive buffer. -1 indicates the buffer_size was * outside the range allowed. */ int16_t readBody(char *buffer, const uint32_t buffer_size); /** * @brief Reads the body of the response after a HTTP call. Will read @param * size amount of bytes at a time, so several calls to this method has to be * made in order to read responses greater in size than that, or the size * has to be increased. * * @param size How many bytes to read at a time. */ String readBody(const uint32_t size = 256); }; extern HttpClientClass HttpClient; #endif My http_client.cpp now looks like this: #include "http_client.h" #include "log.h" #include "sequans_controller.h" #include #include #include #include #include // We only use profile 0 to keep things simple we also stick with spId 3 // which we dedicate to HTTPS #define HTTP_CONFIGURE "AT+SQNHTTPCFG=0,\"%s\",%u,0,\"\",\"\",%u,120,1,3" // Command without any data in it (with quotation marks): 36 bytes // Max length of doman name: 127 bytes // Max length of port number: 5 bytes (0-65535) // TLS enabled: 1 byte // Termination: 1 byte // This results in 36 + 127 + 5 + 1 + 1 = 170 #define HTTP_CONFIGURE_SIZE 170 #define QUERY_SECURITY_PROFILE "AT+SQNSPCFG" #define SECURITY_PROFILE_PREFIX_LENGTH 11 #define HTTPS_SECURITY_PROFILE_NUMBER '3' #define HTTP_SEND "AT+SQNHTTPSND=0,%u,\"%s\",%lu" // MODIFIED FUNCTION BY GOUGH LUI TO ENABLE ADDITIONAL POST HEADERS #define HTTP_SEND2 "AT+SQNHTTPSND=0,%u,\"%s\",%lu,\"%s\",\"%s\",%lu,%lu" #define HTTP_RECEIVE "AT+SQNHTTPRCV=0,%lu" #define HTTP_QUERY "AT+SQNHTTPQRY=0,%u,\"%s\"" #define HTTP_RING_URC "SQNHTTPRING" #define HTTP_POST_METHOD 0 #define HTTP_PUT_METHOD 1 #define HTTP_GET_METHOD 0 #define HTTP_HEAD_METHOD 1 #define HTTP_DELETE_METHOD 2 #define HTTP_RECEIVE_LENGTH 32 #define HTTP_RECEIVE_START_CHARACTER '<' #define HTTP_SEND_START_CHARACTER '>' #define HTTP_RESPONSE_MAX_LENGTH 84 #define HTTP_RESPONSE_STATUS_CODE_INDEX 1 #define HTTP_RESPONSE_STATUS_CODE_LENGTH 3 #define HTTP_RESPONSE_DATA_SIZE_INDEX 3 #define HTTP_RESPONSE_DATA_SIZE_LENGTH 16 // These are limitations from the Sequans module, so the range of bytes we can // receive with one call to the read body AT command has to be between these // values. One thus has to call the function multiple times if the data size is // greater than the max size #define HTTP_BODY_BUFFER_MIN_SIZE 64 #define HTTP_BODY_BUFFER_MAX_SIZE 1500 #define HTTP_TIMEOUT 20000 HttpClientClass HttpClient = HttpClientClass::instance(); /** * @brief Generic method for sending data via HTTP, either with POST or PUT. * Issues an AT command to the LTE modem. * * @param endpoint Destination of payload, part after host name in URL. * @param buffer Payload to send. * @param buffer_size Size of payload. * @param method POST(0) or PUT(1). */ static HttpResponse sendData(const char* endpoint, const uint8_t* buffer, const uint32_t buffer_size, const uint8_t method) { HttpResponse http_response = {0, 0}; // Setup and transmit SEND command before sending the data const uint32_t digits_in_data_length = trunc(log10(buffer_size)) + 1; char command[strlen(HTTP_SEND) + strlen(endpoint) + digits_in_data_length]; sprintf(command, HTTP_SEND, method, endpoint, (unsigned long)buffer_size); SequansController.writeBytes((uint8_t*)command, strlen(command), true); if (!SequansController.waitForByte(HTTP_SEND_START_CHARACTER, HTTP_TIMEOUT)) { Log.error( "Timed out whilst waiting on delivering the HTTP payload. Is the " "server online?"); return http_response; } // Wait some before delivering the payload. The modem might hang if we // deliver it too quickly delay(100); // Now we deliver the payload SequansController.writeBytes(buffer, buffer_size); char http_response_buffer[HTTP_RESPONSE_MAX_LENGTH] = ""; char http_status_code_buffer[HTTP_RESPONSE_STATUS_CODE_LENGTH + 1] = ""; char data_size_buffer[HTTP_RESPONSE_DATA_SIZE_LENGTH] = ""; // Now we wait for the URC if (!SequansController.waitForURC(HTTP_RING_URC, http_response_buffer, sizeof(http_response_buffer))) { Log.warn("Did not get HTTP response before timeout "); return http_response; } // We pass in NULL as the start character here as the URC data will only // contain the payload, not the URC identifier bool got_response_code = SequansController.extractValueFromCommandResponse( http_response_buffer, HTTP_RESPONSE_STATUS_CODE_INDEX, http_status_code_buffer, HTTP_RESPONSE_STATUS_CODE_LENGTH + 1, (char)NULL); bool got_data_size = SequansController.extractValueFromCommandResponse( http_response_buffer, HTTP_RESPONSE_DATA_SIZE_INDEX, data_size_buffer, HTTP_RESPONSE_DATA_SIZE_LENGTH, (char)NULL); if (got_response_code) { http_response.status_code = atoi(http_status_code_buffer); } if (got_data_size) { http_response.data_size = atoi(data_size_buffer); } return http_response; } // MODIFIED FUNCTION BY GOUGH LUI TO ENABLE ADDITIONAL POST HEADERS //sendData2(endpoint, buffer, buffer_size, HTTP_POST_METHOD, post_param, extra_hdr, disconnect, timeout); static HttpResponse sendData2(const char* endpoint, const uint8_t* buffer, const uint32_t buffer_size, const uint8_t method, const char* post_param, const char* extra_hdr, const uint32_t disconnect, const uint32_t timeout) { HttpResponse http_response = {0, 0}; // Setup and transmit SEND command before sending the data const uint32_t digits_in_data_length = trunc(log10(buffer_size)) + 1; const uint32_t digits_in_disconnect = trunc(log10(disconnect)) + 1; const uint32_t digits_in_timeout = trunc(log10(timeout)) + 1; char command[strlen(HTTP_SEND2) + strlen(endpoint) + digits_in_data_length + strlen(post_param) + strlen(extra_hdr) + digits_in_disconnect + digits_in_timeout]; sprintf(command, HTTP_SEND2, method, endpoint, (unsigned long)buffer_size, post_param, extra_hdr, (unsigned long)disconnect, (unsigned long)timeout); SequansController.writeBytes((uint8_t*)command, strlen(command), true); if (!SequansController.waitForByte(HTTP_SEND_START_CHARACTER, HTTP_TIMEOUT)) { Log.error( "Timed out whilst waiting on delivering the HTTP payload. Is the " "server online?"); return http_response; } // Wait some before delivering the payload. The modem might hang if we // deliver it too quickly delay(100); // Now we deliver the payload SequansController.writeBytes(buffer, buffer_size); char http_response_buffer[HTTP_RESPONSE_MAX_LENGTH] = ""; char http_status_code_buffer[HTTP_RESPONSE_STATUS_CODE_LENGTH + 1] = ""; char data_size_buffer[HTTP_RESPONSE_DATA_SIZE_LENGTH] = ""; // Now we wait for the URC if (!SequansController.waitForURC(HTTP_RING_URC, http_response_buffer, sizeof(http_response_buffer))) { Log.warn("Did not get HTTP response before timeout "); return http_response; } // We pass in NULL as the start character here as the URC data will only // contain the payload, not the URC identifier bool got_response_code = SequansController.extractValueFromCommandResponse( http_response_buffer, HTTP_RESPONSE_STATUS_CODE_INDEX, http_status_code_buffer, HTTP_RESPONSE_STATUS_CODE_LENGTH + 1, (char)NULL); bool got_data_size = SequansController.extractValueFromCommandResponse( http_response_buffer, HTTP_RESPONSE_DATA_SIZE_INDEX, data_size_buffer, HTTP_RESPONSE_DATA_SIZE_LENGTH, (char)NULL); if (got_response_code) { http_response.status_code = atoi(http_status_code_buffer); } if (got_data_size) { http_response.data_size = atoi(data_size_buffer); } return http_response; } /** * @brief Generic method for retrieving data via HTTP, either with HEAD, GET or * DELETE. * * @param endpoint Destination of retrieve, part after host name in URL. * @param method GET(0), HEAD(1) or DELETE(2). */ static HttpResponse queryData(const char* endpoint, const uint8_t method) { HttpResponse http_response = {0, 0}; // Fix for bringing the modem out of idling and prevent timeout whilst // waiting for modem response during the next AT command SequansController.writeCommand("AT"); // Set up and send the query char command[strlen(HTTP_QUERY) + strlen(endpoint)]; sprintf(command, HTTP_QUERY, method, endpoint); SequansController.writeCommand(command); char http_response_buffer[HTTP_RESPONSE_MAX_LENGTH] = ""; char http_status_code_buffer[HTTP_RESPONSE_STATUS_CODE_LENGTH + 1] = ""; char data_size_buffer[HTTP_RESPONSE_DATA_SIZE_LENGTH] = ""; if (!SequansController.waitForURC(HTTP_RING_URC, http_response_buffer, sizeof(http_response_buffer))) { Log.warn("Did not get HTTP response before timeout "); return http_response; } bool got_response_code = SequansController.extractValueFromCommandResponse( http_response_buffer, HTTP_RESPONSE_STATUS_CODE_INDEX, http_status_code_buffer, HTTP_RESPONSE_STATUS_CODE_LENGTH + 1, (char)NULL); bool got_data_size = SequansController.extractValueFromCommandResponse( http_response_buffer, HTTP_RESPONSE_DATA_SIZE_INDEX, data_size_buffer, HTTP_RESPONSE_DATA_SIZE_LENGTH, (char)NULL); if (got_response_code) { http_response.status_code = atoi(http_status_code_buffer); } if (got_data_size) { http_response.data_size = atoi(data_size_buffer); } return http_response; } bool HttpClientClass::configure(const char* host, const uint16_t port, const bool enable_tls) { if (enable_tls) { char response[128] = ""; ResponseResult result = SequansController.writeCommand( QUERY_SECURITY_PROFILE, response, sizeof(response)); if (result != ResponseResult::OK) { Log.error("Failed to query HTTPS security profile"); return false; } // Split by line feed and carriage return to retrieve each entry char* ptr = strtok(response, " "); bool security_profile_found = false; while (ptr != NULL) { // Skip the prefix of '+SQNSPCFG: ' ptr += SECURITY_PROFILE_PREFIX_LENGTH; // Now we check if the entry has the third security profile if (*ptr == HTTPS_SECURITY_PROFILE_NUMBER) { security_profile_found = true; break; } ptr = strtok(NULL, " "); } if (!security_profile_found) { Log.error( "Security profile not set up for HTTPS. Run the " "'https_configure_ca' Arduino sketch example to set this up." "More information here: " "">iot.microchip.com/.../http"); return false; } } char command[HTTP_CONFIGURE_SIZE] = ""; sprintf(command, HTTP_CONFIGURE, host, port, enable_tls ? 1 : 0); return SequansController.writeCommand(command) == ResponseResult::OK; } HttpResponse HttpClientClass::post(const char* endpoint, const uint8_t* buffer, const uint32_t buffer_size) { return sendData(endpoint, buffer, buffer_size, HTTP_POST_METHOD); } HttpResponse HttpClientClass::post(const char* endpoint, const char* message) { return post(endpoint, (uint8_t*)message, strlen(message)); } // MODIFIED FUNCTION BY GOUGH LUI TO ENABLE ADDITIONAL POST HEADERS HttpResponse HttpClientClass::post2(const char* endpoint, const uint8_t* buffer, const uint32_t buffer_size, const char* post_param, const char* extra_hdr, const uint32_t disconnect, const uint32_t timeout) { return sendData2(endpoint, buffer, buffer_size, HTTP_POST_METHOD, post_param, extra_hdr, disconnect, timeout); } HttpResponse HttpClientClass::put(const char* endpoint, const uint8_t* buffer, const uint32_t buffer_size) { return sendData(endpoint, buffer, buffer_size, HTTP_PUT_METHOD); } HttpResponse HttpClientClass::put(const char* endpoint, const char* message) { return put(endpoint, (uint8_t*)message, strlen(message)); } HttpResponse HttpClientClass::get(const char* endpoint) { return queryData(endpoint, HTTP_GET_METHOD); } HttpResponse HttpClientClass::head(const char* endpoint) { return queryData(endpoint, HTTP_HEAD_METHOD); } HttpResponse HttpClientClass::del(const char* endpoint) { return queryData(endpoint, HTTP_DELETE_METHOD); } int16_t HttpClientClass::readBody(char* buffer, const uint32_t buffer_size) { // Safeguard against the limitation in the Sequans AT command parameter // for the response receive command. if (buffer_size < HTTP_BODY_BUFFER_MIN_SIZE || buffer_size > HTTP_BODY_BUFFER_MAX_SIZE) { return -1; } // Fix for bringing the modem out of idling and prevent timeout whilst // waiting for modem response during the next AT command SequansController.writeCommand("AT"); // We send the buffer size with the receive command so that we only // receive that. The rest will be flushed from the modem. char command[HTTP_RECEIVE_LENGTH] = ""; sprintf(command, HTTP_RECEIVE, buffer_size); SequansController.writeBytes((uint8_t*)command, strlen(command), true); // We receive three start bytes of the character '<', so we wait for // them uint8_t start_bytes = 3; while (start_bytes > 0) { if (SequansController.readByte() == HTTP_RECEIVE_START_CHARACTER) { start_bytes--; } } // Now we are ready to receive the payload. We only check for error and // not overflow in the receive buffer in comparison to our buffer as we // know the size of what we want to receive if (SequansController.readResponse(buffer, buffer_size) != ResponseResult::OK) { return 0; } return strlen(buffer); } String HttpClientClass::readBody(const uint32_t size) { char buffer[size]; int16_t bytes_read = readBody(buffer, sizeof(buffer)); if (bytes_read == -1) { return ""; } return String(buffer); } I did not change the existing functions so that none of the existing examples should break. Good luck in sending your SMSes. You'll need to change the example to match your hosts, authorisations and SSL/TLS. Hopefully that will be all, but perhaps you could run into other limitations such as length of the URI which is capped at 1000 bytes total. - Gough

Microchip AVR-IoT question: How to correctly format a POST request

Top Replies