Microchip AVR-IoT question: How to correctly format a POST request

I have modified the version of the library to allow for the extra parameters and it seems to work.

First of all, the sample test code:

#include <Arduino.h>
#include <http_client.h>
#include <led_ctrl.h>
#include <log.h>
#include <lte.h>

#define DOMAIN "xxxxxxxx" # THIS IS SET TO YOUR SEVER

void setup() {
    LedCtrl.begin();
    LedCtrl.startupCycle();

    Log.begin(115200);
    Log.info("Starting HTTP example");

    // Start LTE modem and connect to the operator
    if (!Lte.begin()) {
        Log.error("Failed to connect to the operator");
        return;
    }

    Log.infof("Connected to operator: %s\r\n", Lte.getOperator().c_str());

    // --- HTTP ---

    Log.info("---- Testing HTTP ----");

    if (!HttpClient.configure(DOMAIN, 80, false)) { # I only tested with HTTP
        Log.info("Failed to configure http client\r\n");
    }

    Log.info("Configured to HTTP");

    uint8_t bodymsg[] = ""; // This is the payload
    uint32_t bodylen = strlen((const char*)bodymsg); // This is the payload length

    HttpResponse response = HttpClient.post2("/SENDSMS?sms=TESTMSG",bodymsg,bodylen,"application/x-www-form-urlencoded","Authorization: bearer 1234567890",1,30);
    Log.infof("POST - status code: %u, data size: %u\r\n",
              response.status_code,
              response.data_size);

    // Add some extra bytes for termination
    String body = HttpClient.readBody(response.data_size + 16);

    if (body != "") {
        Log.infof("Body: %s\r\n", body.c_str());
    }
}

void loop() {}
The result is the following from my test server:

POST /SENDSMS?sms=TESTMSG HTTP/1.1
Host: xxxxxxxxx
Authorization: bearer 1234567890
Content-Type: application/x-www-form-urlencoded
Connection: close
Content-Length: 0

This is pretty much exactly what you would want since your message is URL encoded, the body of the request has zero length. The function prototype for the updated function (I've called post2) is as follows:

    HttpResponse post2(const char* endpoint,
                       const uint8_t* buffer,
                       const uint32_t buffer_size,
                       const uint8_t* post_param,
                       const uint8_t* extra_hdr,
                       const uint32_t disconnect,
                       const uint32_t timeout);

In order to add this to your library, you will need to locate the AVR-IoT-Cellular library folder, navigate to the src folder and edit http_client.h and http_client.cpp. I have bolded my additions.

My http_client.h now looks like this:

/**
 * @brief HTTP client for REST calls.
 */

#ifndef HTTP_CLIENT_H
#define HTTP_CLIENT_H

#include <Arduino.h>
#include <stdint.h>

typedef struct {
    uint16_t status_code;
    uint32_t data_size;
} HttpResponse;

class HttpClientClass {

  private:
    HttpClientClass(){};

  public:
    static HttpClientClass &instance(void) {
        static HttpClientClass instance;
        return instance;
    }

    enum StatusCodes {
        STATUS_OK = 200,
        STATUS_NOT_FOUND = 404,
        STATUS_INTERNAL_SERVER_ERROR = 500,
    };

    /**
     * @brief Sets up the HTTP client with a host and port.
     *
     * @param host Can either be a host name resolved with DNS or an actual
     * server address in the form of "xxx.xxx.xxx.xxx".
     * @param port Port of the host, e.g. 80 or 443 (for HTTPS).
     * @param enable_tls Use tls, required for HTTPS.
     *
     * @return True if operation was successful.
     */
    bool
    configure(const char *host, const uint16_t port, const bool enable_tls);

    /**
     * @brief Issues a post to the host configured. Will block until operation
     * is done.
     */
    HttpResponse post(const char *endpoint,
                      const uint8_t *buffer,
                      const uint32_t buffer_size);

// MODIFIED FUNCTION BY GOUGH LUI TO ENABLE ADDITIONAL POST HEADERS
    HttpResponse post2(const char* endpoint,
                       const uint8_t* buffer,
                       const uint32_t buffer_size,
                       const char* post_param,
                       const char* extra_hdr,
                       const uint32_t disconnect,
                       const uint32_t timeout);
    /**
     * @brief Issues a post to the host configured. Will block until operation
     * is done.
     *
     * @param messsage Needs to be a regular c string which is null terminated.
     *
     */
    HttpResponse post(const char *endpoint, const char *messsage);

    /**
     * @brief Issues a put to the host configured. Will block until operation is
     * done.
     */
    HttpResponse put(const char *endpoint,
                     const uint8_t *buffer,
                     const uint32_t buffer_size);

    /**
     * @brief Issues a put to the host configured. Will block until operation is
     * done.
     *
     * @param messsage Needs to be a regular c string which is null terminated.
     */
    HttpResponse put(const char *endpoint, const char *message);

    /**
     * @brief Issues a get from the host configured. Will block until operation
     * is done. The contents of the body after the get can be read using the
     * readBody() function after.
     */
    HttpResponse get(const char *endpoint);

    /**
     * @brief Issues a head from the host configured. Will block until operation
     * is done.
     */
    HttpResponse head(const char *endpoint);

    /**
     * @brief Issues a delete from the host configured. Will block until
     * operation is done.
     */
    HttpResponse del(const char *endpoint);

    /**
     * @brief Reads the body of a response after a HTTP call. Note that the
     * range for the buffer_size has to be between 64-1500. This is a limitation
     * from the Sequans LTE module. So if the data is larger than that,
     * multiple calls to this function has to be made.
     *
     * @param buffer Destination of the body.
     * @param buffer_size Has to be between 64-1500.
     *
     * @return bytes read from receive buffer. -1 indicates the buffer_size was
     * outside the range allowed.
     */
    int16_t readBody(char *buffer, const uint32_t buffer_size);

    /**
     * @brief Reads the body of the response after a HTTP call. Will read @param
     * size amount of bytes at a time, so several calls to this method has to be
     * made in order to read responses greater in size than that, or the size
     * has to be increased.
     *
     * @param size How many bytes to read at a time.
     */
    String readBody(const uint32_t size = 256);
};

extern HttpClientClass HttpClient;

#endif

My http_client.cpp now looks like this:

#include "http_client.h"
#include "log.h"
#include "sequans_controller.h"

#include <Arduino.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// We only use profile 0 to keep things simple we also stick with spId 3
// which we dedicate to HTTPS
#define HTTP_CONFIGURE "AT+SQNHTTPCFG=0,\"%s\",%u,0,\"\",\"\",%u,120,1,3"

// Command without any data in it (with quotation marks): 36 bytes
// Max length of doman name: 127 bytes
// Max length of port number: 5 bytes (0-65535)
// TLS enabled: 1 byte
// Termination: 1 byte
// This results in 36 + 127 + 5 + 1 + 1 = 170
#define HTTP_CONFIGURE_SIZE 170

#define QUERY_SECURITY_PROFILE "AT+SQNSPCFG"

#define SECURITY_PROFILE_PREFIX_LENGTH 11
#define HTTPS_SECURITY_PROFILE_NUMBER  '3'

#define HTTP_SEND    "AT+SQNHTTPSND=0,%u,\"%s\",%lu"
// MODIFIED FUNCTION BY GOUGH LUI TO ENABLE ADDITIONAL POST HEADERS
#define HTTP_SEND2   "AT+SQNHTTPSND=0,%u,\"%s\",%lu,\"%s\",\"%s\",%lu,%lu"
#define HTTP_RECEIVE "AT+SQNHTTPRCV=0,%lu"
#define HTTP_QUERY   "AT+SQNHTTPQRY=0,%u,\"%s\""

#define HTTP_RING_URC "SQNHTTPRING"

#define HTTP_POST_METHOD   0
#define HTTP_PUT_METHOD    1
#define HTTP_GET_METHOD    0
#define HTTP_HEAD_METHOD   1
#define HTTP_DELETE_METHOD 2

#define HTTP_RECEIVE_LENGTH          32
#define HTTP_RECEIVE_START_CHARACTER '<'
#define HTTP_SEND_START_CHARACTER    '>'

#define HTTP_RESPONSE_MAX_LENGTH         84
#define HTTP_RESPONSE_STATUS_CODE_INDEX  1
#define HTTP_RESPONSE_STATUS_CODE_LENGTH 3
#define HTTP_RESPONSE_DATA_SIZE_INDEX    3
#define HTTP_RESPONSE_DATA_SIZE_LENGTH   16

// These are limitations from the Sequans module, so the range of bytes we can
// receive with one call to the read body AT command has to be between these
// values. One thus has to call the function multiple times if the data size is
// greater than the max size
#define HTTP_BODY_BUFFER_MIN_SIZE 64
#define HTTP_BODY_BUFFER_MAX_SIZE 1500

#define HTTP_TIMEOUT 20000

HttpClientClass HttpClient = HttpClientClass::instance();

/**
 * @brief Generic method for sending data via HTTP, either with POST or PUT.
 * Issues an AT command to the LTE modem.
 *
 * @param endpoint Destination of payload, part after host name in URL.
 * @param buffer Payload to send.
 * @param buffer_size Size of payload.
 * @param method POST(0) or PUT(1).
 */
static HttpResponse sendData(const char* endpoint,
                             const uint8_t* buffer,
                             const uint32_t buffer_size,
                             const uint8_t method) {

    HttpResponse http_response = {0, 0};

    // Setup and transmit SEND command before sending the data
    const uint32_t digits_in_data_length = trunc(log10(buffer_size)) + 1;

    char command[strlen(HTTP_SEND) + strlen(endpoint) + digits_in_data_length];
    sprintf(command, HTTP_SEND, method, endpoint, (unsigned long)buffer_size);
    SequansController.writeBytes((uint8_t*)command, strlen(command), true);

    if (!SequansController.waitForByte(HTTP_SEND_START_CHARACTER,
                                       HTTP_TIMEOUT)) {
        Log.error(
            "Timed out whilst waiting on delivering the HTTP payload. Is the "
            "server online?");
        return http_response;
    }

    // Wait some before delivering the payload. The modem might hang if we
    // deliver it too quickly
    delay(100);

    // Now we deliver the payload
    SequansController.writeBytes(buffer, buffer_size);

    char http_response_buffer[HTTP_RESPONSE_MAX_LENGTH]                = "";
    char http_status_code_buffer[HTTP_RESPONSE_STATUS_CODE_LENGTH + 1] = "";
    char data_size_buffer[HTTP_RESPONSE_DATA_SIZE_LENGTH]              = "";

    // Now we wait for the URC
    if (!SequansController.waitForURC(HTTP_RING_URC,
                                      http_response_buffer,
                                      sizeof(http_response_buffer))) {
        Log.warn("Did not get HTTP response before timeout\r\n");
        return http_response;
    }

    // We pass in NULL as the start character here as the URC data will only
    // contain the payload, not the URC identifier
    bool got_response_code = SequansController.extractValueFromCommandResponse(
        http_response_buffer,
        HTTP_RESPONSE_STATUS_CODE_INDEX,
        http_status_code_buffer,
        HTTP_RESPONSE_STATUS_CODE_LENGTH + 1,
        (char)NULL);

    bool got_data_size = SequansController.extractValueFromCommandResponse(
        http_response_buffer,
        HTTP_RESPONSE_DATA_SIZE_INDEX,
        data_size_buffer,
        HTTP_RESPONSE_DATA_SIZE_LENGTH,
        (char)NULL);

    if (got_response_code) {
        http_response.status_code = atoi(http_status_code_buffer);
    }

    if (got_data_size) {
        http_response.data_size = atoi(data_size_buffer);
    }

    return http_response;
}

// MODIFIED FUNCTION BY GOUGH LUI TO ENABLE ADDITIONAL POST HEADERS
//sendData2(endpoint, buffer, buffer_size, HTTP_POST_METHOD, post_param, extra_hdr, disconnect, timeout);
static HttpResponse sendData2(const char* endpoint,
                             const uint8_t* buffer,
                             const uint32_t buffer_size,
                             const uint8_t method,
                             const char* post_param,
                             const char* extra_hdr,
                             const uint32_t disconnect,
                             const uint32_t timeout) {

    HttpResponse http_response = {0, 0};

    // Setup and transmit SEND command before sending the data
    const uint32_t digits_in_data_length = trunc(log10(buffer_size)) + 1;
    const uint32_t digits_in_disconnect = trunc(log10(disconnect)) + 1;
    const uint32_t digits_in_timeout = trunc(log10(timeout)) + 1;

    char command[strlen(HTTP_SEND2) + strlen(endpoint) + digits_in_data_length + strlen(post_param) + strlen(extra_hdr) + digits_in_disconnect + digits_in_timeout];
    sprintf(command, HTTP_SEND2, method, endpoint, (unsigned long)buffer_size, post_param, extra_hdr, (unsigned long)disconnect, (unsigned long)timeout);
    SequansController.writeBytes((uint8_t*)command, strlen(command), true);

    if (!SequansController.waitForByte(HTTP_SEND_START_CHARACTER,
                                       HTTP_TIMEOUT)) {
        Log.error(
            "Timed out whilst waiting on delivering the HTTP payload. Is the "
            "server online?");
        return http_response;
    }

    // Wait some before delivering the payload. The modem might hang if we
    // deliver it too quickly
    delay(100);

    // Now we deliver the payload
    SequansController.writeBytes(buffer, buffer_size);

    char http_response_buffer[HTTP_RESPONSE_MAX_LENGTH]                = "";
    char http_status_code_buffer[HTTP_RESPONSE_STATUS_CODE_LENGTH + 1] = "";
    char data_size_buffer[HTTP_RESPONSE_DATA_SIZE_LENGTH]              = "";

    // Now we wait for the URC
    if (!SequansController.waitForURC(HTTP_RING_URC,
                                      http_response_buffer,
                                      sizeof(http_response_buffer))) {
        Log.warn("Did not get HTTP response before timeout\r\n");
        return http_response;
    }

    // We pass in NULL as the start character here as the URC data will only
    // contain the payload, not the URC identifier
    bool got_response_code = SequansController.extractValueFromCommandResponse(
        http_response_buffer,
        HTTP_RESPONSE_STATUS_CODE_INDEX,
        http_status_code_buffer,
        HTTP_RESPONSE_STATUS_CODE_LENGTH + 1,
        (char)NULL);

    bool got_data_size = SequansController.extractValueFromCommandResponse(
        http_response_buffer,
        HTTP_RESPONSE_DATA_SIZE_INDEX,
        data_size_buffer,
        HTTP_RESPONSE_DATA_SIZE_LENGTH,
        (char)NULL);

    if (got_response_code) {
        http_response.status_code = atoi(http_status_code_buffer);
    }

    if (got_data_size) {
        http_response.data_size = atoi(data_size_buffer);
    }

    return http_response;
}


/**
 * @brief Generic method for retrieving data via HTTP, either with HEAD, GET or
 * DELETE.
 *
 * @param endpoint Destination of retrieve, part after host name in URL.
 * @param method GET(0), HEAD(1) or DELETE(2).
 */
static HttpResponse queryData(const char* endpoint, const uint8_t method) {

    HttpResponse http_response = {0, 0};

    // Fix for bringing the modem out of idling and prevent timeout whilst
    // waiting for modem response during the next AT command
    SequansController.writeCommand("AT");

    // Set up and send the query
    char command[strlen(HTTP_QUERY) + strlen(endpoint)];
    sprintf(command, HTTP_QUERY, method, endpoint);
    SequansController.writeCommand(command);

    char http_response_buffer[HTTP_RESPONSE_MAX_LENGTH]                = "";
    char http_status_code_buffer[HTTP_RESPONSE_STATUS_CODE_LENGTH + 1] = "";
    char data_size_buffer[HTTP_RESPONSE_DATA_SIZE_LENGTH]              = "";

    if (!SequansController.waitForURC(HTTP_RING_URC,
                                      http_response_buffer,
                                      sizeof(http_response_buffer))) {
        Log.warn("Did not get HTTP response before timeout\r\n");
        return http_response;
    }

    bool got_response_code = SequansController.extractValueFromCommandResponse(
        http_response_buffer,
        HTTP_RESPONSE_STATUS_CODE_INDEX,
        http_status_code_buffer,
        HTTP_RESPONSE_STATUS_CODE_LENGTH + 1,
        (char)NULL);

    bool got_data_size = SequansController.extractValueFromCommandResponse(
        http_response_buffer,
        HTTP_RESPONSE_DATA_SIZE_INDEX,
        data_size_buffer,
        HTTP_RESPONSE_DATA_SIZE_LENGTH,
        (char)NULL);

    if (got_response_code) {
        http_response.status_code = atoi(http_status_code_buffer);
    }

    if (got_data_size) {
        http_response.data_size = atoi(data_size_buffer);
    }

    return http_response;
}

bool HttpClientClass::configure(const char* host,
                                const uint16_t port,
                                const bool enable_tls) {

    if (enable_tls) {

        char response[128]    = "";
        ResponseResult result = SequansController.writeCommand(
            QUERY_SECURITY_PROFILE,
            response,
            sizeof(response));

        if (result != ResponseResult::OK) {
            Log.error("Failed to query HTTPS security profile");
            return false;
        }

        // Split by line feed and carriage return to retrieve each entry
        char* ptr                   = strtok(response, "\r\n");
        bool security_profile_found = false;

        while (ptr != NULL) {

            // Skip the prefix of '+SQNSPCFG: '
            ptr += SECURITY_PROFILE_PREFIX_LENGTH;

            // Now we check if the entry has the third security profile
            if (*ptr == HTTPS_SECURITY_PROFILE_NUMBER) {
                security_profile_found = true;
                break;
            }

            ptr = strtok(NULL, "\r\n");
        }

        if (!security_profile_found) {
            Log.error(
                "Security profile not set up for HTTPS. Run the "
                "'https_configure_ca' Arduino sketch example to set this up."
                "More information here: "
                "">iot.microchip.com/.../http");

            return false;
        }
    }

    char command[HTTP_CONFIGURE_SIZE] = "";
    sprintf(command, HTTP_CONFIGURE, host, port, enable_tls ? 1 : 0);
    return SequansController.writeCommand(command) == ResponseResult::OK;
}

HttpResponse HttpClientClass::post(const char* endpoint,
                                   const uint8_t* buffer,
                                   const uint32_t buffer_size) {
    return sendData(endpoint, buffer, buffer_size, HTTP_POST_METHOD);
}

HttpResponse HttpClientClass::post(const char* endpoint, const char* message) {
    return post(endpoint, (uint8_t*)message, strlen(message));
}

// MODIFIED FUNCTION BY GOUGH LUI TO ENABLE ADDITIONAL POST HEADERS
HttpResponse HttpClientClass::post2(const char* endpoint,
                                   const uint8_t* buffer,
                                   const uint32_t buffer_size,
                                   const char* post_param,
                                   const char* extra_hdr,
                                   const uint32_t disconnect,
                                   const uint32_t timeout) {
    return sendData2(endpoint, buffer, buffer_size, HTTP_POST_METHOD, post_param, extra_hdr, disconnect, timeout);
}

HttpResponse HttpClientClass::put(const char* endpoint,
                                  const uint8_t* buffer,
                                  const uint32_t buffer_size) {
    return sendData(endpoint, buffer, buffer_size, HTTP_PUT_METHOD);
}

HttpResponse HttpClientClass::put(const char* endpoint, const char* message) {
    return put(endpoint, (uint8_t*)message, strlen(message));
}

HttpResponse HttpClientClass::get(const char* endpoint) {
    return queryData(endpoint, HTTP_GET_METHOD);
}

HttpResponse HttpClientClass::head(const char* endpoint) {
    return queryData(endpoint, HTTP_HEAD_METHOD);
}

HttpResponse HttpClientClass::del(const char* endpoint) {
    return queryData(endpoint, HTTP_DELETE_METHOD);
}

int16_t HttpClientClass::readBody(char* buffer, const uint32_t buffer_size) {

    // Safeguard against the limitation in the Sequans AT command parameter
    // for the response receive command.
    if (buffer_size < HTTP_BODY_BUFFER_MIN_SIZE ||
        buffer_size > HTTP_BODY_BUFFER_MAX_SIZE) {
        return -1;
    }

    // Fix for bringing the modem out of idling and prevent timeout whilst
    // waiting for modem response during the next AT command
    SequansController.writeCommand("AT");

    // We send the buffer size with the receive command so that we only
    // receive that. The rest will be flushed from the modem.
    char command[HTTP_RECEIVE_LENGTH] = "";
    sprintf(command, HTTP_RECEIVE, buffer_size);
    SequansController.writeBytes((uint8_t*)command, strlen(command), true);

    // We receive three start bytes of the character '<', so we wait for
    // them
    uint8_t start_bytes = 3;

    while (start_bytes > 0) {
        if (SequansController.readByte() == HTTP_RECEIVE_START_CHARACTER) {
            start_bytes--;
        }
    }

    // Now we are ready to receive the payload. We only check for error and
    // not overflow in the receive buffer in comparison to our buffer as we
    // know the size of what we want to receive
    if (SequansController.readResponse(buffer, buffer_size) !=
        ResponseResult::OK) {
        return 0;
    }

    return strlen(buffer);
}

String HttpClientClass::readBody(const uint32_t size) {
    char buffer[size];
    int16_t bytes_read = readBody(buffer, sizeof(buffer));

    if (bytes_read == -1) {
        return "";
    }

    return String(buffer);
}

I did not change the existing functions so that none of the existing examples should break.

Good luck in sending your SMSes. You'll need to change the example to match your hosts, authorisations and SSL/TLS. Hopefully that will be all, but perhaps you could run into other limitations such as length of the URI which is capped at 1000 bytes total.

- Gough

I’m amazed (and grateful) that you did this. It worked perfectly, first time. The only thing I changed was the configure function call to use port 443 & TLS (Scriptr requires it).

I also added some bogus text for the body (all of the necessary information is actually passed in the header). Without it, the Post function call thought the call timed out, although it really worked anyway and still gave a 200 status code.  

Once again, a big THANK YOU! for taking the time to do this.

I’m amazed (and grateful) that you did this. It worked perfectly, first time. The only thing I changed was the configure function call to use port 443 & TLS (Scriptr requires it).

I also added some bogus text for the body (all of the necessary information is actually passed in the header). Without it, the Post function call thought the call timed out, although it really worked anyway and still gave a 200 status code.  

Once again, a big THANK YOU! for taking the time to do this.