Adding The Golioth Firmware SDK to Your ESP-IDF Project

One of the most common questions we hear from our community is how to add Golioth to existing ESP-IDF projects. Until recently, our go-to resource was a blog post that walked users through the process. While the fundamentals of adding Golioth to CMake-based projects remain the same, that guide has become outdated due to significant updates to the Golioth API. Naturally, many users have noticed this and have pointed out areas where the older guide no longer applies.

The most notable change is the evolution of the Golioth Firmware SDK. We’ve made significant improvements, including combining with our previously standalone Zephyr SDK, making it a unified SDK for multiple platforms. These changes make it easier for developers to switch between frameworks, leverage common tools, and maintain consistency across IoT projects. However, they also mean some of the specifics in the original ESP-IDF blog post no longer align with the latest practices.

For example, in step 4 of the original post, users were instructed to include the golioth.h header file to access the API functions. That file no longer exists, so if you’re using the latest version of the Golioth Firmware SDK, following the original guide will lead to confusion and build errors.

Another limitation of the original post is that it doesn’t address the needs of users who are new to CMake. If you’re just starting with ESP-IDF and want to leverage the common example code we provide in our SDK, the older guide won’t be of much help.

We’re here to remedy that. Whether you’re updating an existing project or starting fresh, this updated post will guide you step-by-step through the process of adding Golioth to your ESP-IDF project.

Getting Started with Golioth Integration

This guide focuses on adding Golioth to an existing ESP-IDF application, so we’re assuming you already have the ESP-IDF development environment set up on your system. If not, we recommend following the ESP-IDF Quickstart Guide to get up and running before proceeding.

For this walkthrough, we’ll use the /esp-idf/examples/wifi/getting_started/station example code as our starting point. This example provides a solid foundation for adding Golioth functionality. To begin, copy the example project to a new folder, which we’ll call esp_hello_golioth:

cp -r ~/esp-idf/examples/wifi/getting_started/station esp_hello_golioth

Step 1: Adding Golioth as a Component

To integrate Golioth into your project, we first need to include the Golioth Firmware SDK as a submodule. This allows you to pull in the latest SDK features while keeping your project organized. However, for this step to work, your project must be under version control. If it isn’t already, initialize a Git repository:

cd ~/esp_hello_golioth
git init

Next, navigate to your project directory and add the Golioth Firmware SDK as a submodule. Run the following commands to include the SDK and ensure all its dependencies are updated:

git submodule add https://github.com/golioth/golioth-firmware-sdk.git submodules/golioth-firmware-sdk
git submodule update --init --recursive

After running these commands, you’ll find the Golioth Firmware SDK located in the submodules folder, specifically pulled from the SDK’s main branch.

Working with Specific SDK Releases

While using the main branch ensures you’re working with the latest code, it’s often better to lock your project to a specific SDK release for stability and consistency. To do this, you can checkout a specific tag or release version after adding the submodule:

cd submodules/golioth-firmware-sdk
git checkout <release-tag>
git submodule update --init --recursive

Replace <release-tag> with the version of the SDK you wish to use, such as v0.15.0. This ensures your project always uses a stable and tested version of the Golioth SDK, avoiding potential issues with breaking changes or experimental updates.

With the Golioth SDK installed as a component, we are ready to move on to the project configuration.

Step 2: Configuring CMake for the Golioth SDK

To integrate the Golioth SDK into your ESP-IDF project, we need to modify the CMake configuration files to ensure the SDK is included during the build process. This involves updating the top-level CMakeLists.txt file and the CMakeLists.txt file in the main directory.

Add the Golioth SDK to the Component Directories

First, we need to tell CMake where to find the Golioth SDK components. Open the project’s top-level CMakeLists.txt file and add the following line before the project() directive:

list(APPEND EXTRA_COMPONENT_DIRS submodules/golioth-firmware-sdk/port/esp_idf/components)

This informs CMake to include the Golioth SDK’s ESP-IDF components, which are located in the submodules/golioth-firmware-sdk/port/esp_idf/components folder.

Specify Golioth as a Dependency

Next, we need to declare the Golioth SDK as a dependency in your project’s main/CMakeLists.txt file. For the ESP-IDF example code we’re using, there are no existing dependencies defined. To add Golioth, insert the following at the top of the main/CMakeLists.txt file:

set(deps
"golioth_sdk"
)

This step ensures that CMake properly resolves the Golioth SDK as a required component during the build process.

Why These Changes Matter

By making these adjustments, we’re effectively linking the Golioth SDK into your project’s build system. This allows you to call Golioth APIs and ensures that the SDK is correctly compiled along with the rest of your application.

Now, let’s move on to configuring the application!

Step 3: Configuring Kconfig Settings

To ensure the Golioth SDK functions correctly, we need to enable specific configuration options using ESP-IDF’s Kconfig system. These settings enable essential features like Mbed-TLS and Golioth-specific options. Instead of modifying configurations manually each time, we’ll replace the contents of the originalsdkconfig.defaults file in the top-level project folder with the following:

CONFIG_ESP_WIFI_SOFTAP_SUPPORT=n
CONFIG_MBEDTLS_SSL_PROTO_DTLS=y
CONFIG_MBEDTLS_PSK_MODES=y
CONFIG_MBEDTLS_KEY_EXCHANGE_PSK=y
CONFIG_MBEDTLS_TLS_CLIENT_ONLY=y
CONFIG_LWIP_NETBUF_RECVINFO=y
CONFIG_FREERTOS_USE_TRACE_FACILITY=y
CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS=y
CONFIG_COMPILER_OPTIMIZATION_SIZE=y
CONFIG_GOLIOTH_AUTO_LOG_TO_CLOUD=y
CONFIG_GOLIOTH_COAP_REQUEST_QUEUE_MAX_ITEMS=20

Explanation of Key Settings

• Mbed-TLS Configuration
  • Golioth relies on Mbed-TLS for secure communication. The first four CONFIG_MBEDTLS_ options enable support for DTLS, PSK modes, and client-only TLS.
• Networking Enhancements
  • CONFIG_LWIP_NETBUF_RECVINFO enables additional networking features needed by Golioth, while the FreeRTOS settings CONFIG_FREERTOS_USE_TRACE_FACILITY and CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS help monitor and debug system performance.
• Wi-Fi Adjustment
  • Since this example focuses on station mode, we disable unnecessary SoftAP support with CONFIG_ESP_WIFI_SOFTAP_SUPPORT=n.
• Code Optimization
  • The setting CONFIG_COMPILER_OPTIMIZATION_SIZE=y ensures the code is optimized for size, which is often critical for IoT devices.
• Golioth-Specific Settings
  • CONFIG_GOLIOTH_AUTO_LOG_TO_CLOUD enables automatic uploading of GLTH_LOGX log statements to the Golioth Logs service.
    CONFIG_GOLIOTH_COAP_REQUEST_QUEUE_MAX_ITEMS sets the maximum number of queued CoAP requests for smooth operation.

Why These Settings Matter

The configurations in sdkconfig.defaults ensure that all necessary features are enabled and optimized for using Golioth with ESP-IDF. The final two symbols are particularly useful for automatically uploading log messages to the Golioth Logs service, making debugging and monitoring easier.

With this step complete, your project is now configured to build with Golioth’s essential components.

Step 4: Using Golioth API Calls in Your Code

Now that the Golioth Firmware SDK is configured in your project, you can start incorporating its functionality into your application. The first step is to include the appropriate header file in your code to provide access to Golioth’s API, enabling communication with Golioth services:

#include <golioth/client.h>

For demonstration purposes, we’ll create a simple counter that runs in an infinite loop. Every five seconds, the counter value will be sent to Golioth as a Log message. Below is the modified app_main function, with the new code starting at line 14:

void app_main(void)
{
    //Initialize NVS
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
        ESP_ERROR_CHECK(nvs_flash_erase());
        ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);
    ESP_LOGI(TAG, "ESP_WIFI_MODE_STA");
    wifi_init_sta();

    //Everything above this line is unchanged from the ESP-IDF example code
    const char* psk_id = "your-golioth@psk-id";
    const char* psk = "your-golioth-psk";

    const struct golioth_client_config config = {
        .credentials = {
        .auth_type = GOLIOTH_TLS_AUTH_TYPE_PSK,
        .psk = {
            .psk_id = psk_id,
            .psk_id_len = strlen(psk_id),
            .psk = psk,
            .psk_len = strlen(psk),
    }}};

    struct golioth_client *client = golioth_client_create(&config);
    assert(client);

    uint16_t counter = 0;
    while(1) {
        GLTH_LOGI(TAG, "Hello, Golioth! #%d", counter);
        ++counter;
        vTaskDelay(5000 / portTICK_PERIOD_MS);
    }
}

For simplicity, in this example, we’ll hardcode the credentials (PSK-ID and PSK) directly in the application code. While this approach is sufficient for demonstration purposes, it’s never recommended for production use. In the next blog post, we’ll explore how to securely store these credentials using the ESP-IDF common code from the Firmware SDK to set them in the NVS (Non-Volatile Storage) subsystem.

If you’re following along, the next step is to set up your WiFi credentials. You can do this in one of two ways:

1. Use menuconfig to configure the WiFi settings interactively
2. Modify the default values directly in the main/Kconfig.projbuild file

Viewing the Results on the Golioth Console

Once your application is running, you can observe its behavior through the serial output. This output confirms that the Golioth client has successfully connected to the platform and is sending log messages as expected. Here’s a sample snippet of what the serial output might look like:

I (896) wifi:AP's beacon interval = 102400 us, DTIM period = 1
I (3316) esp_netif_handlers: sta ip: 192.168.8.113, mask: 255.255.255.0, gw: 192.168.8.1
I (3316) wifi station: got ip:192.168.8.113
I (3316) wifi station: connected to ap SSID:mpuric password:abcdeedcba
I (3326) golioth_mbox: Mbox created, bufsize: 2520, num_items: 20, item_size: 120
I (3336) wifi station: Hello, Golioth! #0
I (3396) golioth_coap_client_libcoap: Start CoAP session with host: coaps://coap.golioth.io
I (3396) golioth_coap_client_libcoap: Session PSK-ID: 20240403080418-esp32@dev
I (3406) libcoap: Setting PSK key

I (3416) golioth_coap_client_libcoap: Entering CoAP I/O loop
I (4376) golioth_coap_client_libcoap: Golioth CoAP client connected
I (8336) wifi station: Hello, Golioth! #1
I (13336) wifi station: Hello, Golioth! #2
I (18336) wifi station: Hello, Golioth! #3
I (23336) wifi station: Hello, Golioth! #4