Golioth continuously checks real hardware targets to ensure our new features and code fixes work across a wide swath of devices. The last time I checked, we’re running 541 hardware-in-the-loop tests on each pull-request of our SDK. This includes sample and integration tests for the Golioth Firmware SDK on 8 hardware devices (3 ESP-IDF and 5 Zephyr), two Zephyr native simulator devices, and on Linux. Understanding what tests failed–and finding out why–became almost impossible. So we added an open source tool to visualize each test and deliver historic insight as well. Today I’ll walk through how we use Allure Report to understand continuous integration tests.
Allure Report is a test reporting tool
Allure Report gathers files generated during each CI test, using them to generate an HTML site that visualizes the outcomes of every test. Better yet, error output and logs are available at the click of a button and the history of a test can be viewed to see if something is frequently failing.
Since the Golioth Firmware SDK is open source, you can use our approach to automatic Allure Report generation to your own CI.
Why are our tests failing frequently enough to need a reporting tool?
Great question. Hardware in the Loop (HIL) testing for Internet of Things (IoT) devices is a gauntlet of technologies. We’re testing Cellular, Ethernet, and WiFi, any of which can have connectivity issues. The range of hardware we’re testing relies on different programmers and flashing tools, and network latency is an ever-changing variable.
But the real reason to have a tool is to help clarify if a test failed due to one of the reasons above, or because of a legitimate bug that only rarely occurs. This could be an issue with the Firmware SDK, or something that changed on Golioth’s cloud platform. We want to know right away, and we want to be able to track how often we’re seeing the same types of failures.
Add Allure Report file generation to a test
All Golioth CI testing uses pytest which has support for Allure Report generation built in. However, there’s a ton of frameworks that are supported so if you’re running a different test automation, chances are you can use Allure Report.
pip install allure-pytest
With the pytest plugin installed, add a flag to your pytest command that specifies an output directory for the generated report files.
pytest path/to/your/test --alluredir=your-report-output-directory
That’s it, when you run this test, the Allure Report files will be generated.
Manually generate the Allure Report Website
Viewing test output locally is quite simple. Follow the Allure Report install instructions for your OS and then just run the serve command to show the test output:
allure serve /path/to/your-report-output-directory
A live server will open up in your web browser to visualize the report files you provided. This is a great preview to start poking around, but it’s missing one of the best features which is the ability to view report history. For that, you should add report history to CI so everything is handled automatically.
Add Allure Report to CI
Just as before, there are two parts to using Allure Report: generating the files and collecting those files into a website.
Generating report files in CI
As mentioned above, generating the report files is as simple as adding a flag to your pytest (or other test framework) command in your GitHub workflow. However, we run our tests in parallel, using at least 5 self-hosted runners and innumerable GitHub hosted runners. All of these generate report files and we want all of them in one place. For that we take the following approach:
- Individual tests all upload report files as artifacts with a common prefix in the name
- A separate job collects reports once all tests are complete
steps:
- name:
run: |
pytest path/to/your/test --alluredir=allure-reports
- name: Safe upload Allure reports
if: success() || failure()
uses: actions/upload-artifact@v4
with:
name: allure-reports-samples-zephyr-${{ matrix.platform }}
path: allure-reports
To collect files after each test, use the upload artifact action. Here the files are stored in a directory named allure-reports.
merge_reports:
runs-on: ubuntu-latest
needs:
- hil_sample_esp-idf
- hil_sample_zephyr
- hil_sample_zephyr_nsim
- hil_test_esp-idf
- hil_test_linux
- hil_test_zephyr
if: always()
steps:
- name: Gather reports
uses: actions/download-artifact@v4
with:
path: reports/allure-reports
pattern: allure-reports-*
merge-multiple: true
- name: Upload reports
uses: actions/upload-artifact@v4
with:
name: allure-reports-alltest
path: reports/allure-reports
The collection job should add each job that generates reports to the needs list. This causes the job to wait for the others to complete. It uses the download artifact action to download all reports and merge them into a single reports/allure-reports directory.
That collection is then uploaded using allure-reports-alltest as the artifact name. The upload step stores these artifacts so you can optionally download them from the GitHub action summary page and view them locally. However, we also want to automatically generate and host an Allure Report site.
Publishing the Allure Report site using CI
There are two GitHub actions useful for publishing your Allure Report site:
- simple-elf/allure-report-action: Generate the Allure Report Site
- peaceiris/actions-gh-pages: Publish Allure Report to GitHub pages
Publishing to GitHub pages is easy, but remember that anything shown in your test logs will be public on that page. We scrub our logs of all secrets before uploading the artifacts (that’s a story for a different day).
steps:
- name: Get Allure history
uses: actions/checkout@v4
if: always()
continue-on-error: true
with:
ref: gh-pages
path: gh-pages
- name: Generate new Allure Report site
uses: simple-elf/allure-report-action@v1
if: always()
id: allure-report
with:
allure_results: reports/allure-reports
gh_pages: gh-pages
allure_report: allure-report
allure_history: allure-history
- name: Deploy report to Github Pages
if: always()
uses: peaceiris/actions-gh-pages@v4
env:
PERSONAL_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PUBLISH_BRANCH: gh-pages
PUBLISH_DIR: allure-history
The steps above have the following effect:
- Check out the last time you published the Allure Report site (from the
gh-pagesbranch) so that the history is available for the newly generated page. - Generate the report using the downloaded/merged artifacts in the
reports/allure-reportsdirectory from the previous section (combine all of these steps together). The history that we checked out in this branch will be used in this build step. - Publish the changes to GitHub pages.
You can now go to your-org-name.github.io/your-repo-name to see the report. For instance, the last twenty Golioth merges and scheduled tests on the main branch are found at https://golioth.github.io/allure-reports.
Customizations: Suite names, tags, parameters
We heavily depend on Allure Reports so we’ve made a number of customizations to better organize the generated reports. The most important is to assign a dynamic parameter for the board name and platform so that we can differentiate the tests:
@pytest.hookimpl(wrapper=True)
def pytest_runtest_setup(item):
board_name = item.config.getoption("--allure-board") or item.config.getoption("--board")
platform_name = item.config.getoption("--platform")
suitename = item.config.getoption("--custom-suitename") or "hil"
allure.dynamic.tag(board_name)
allure.dynamic.tag(platform_name)
allure.dynamic.parameter("board_name", board_name)
allure.dynamic.parameter("platform_name", platform_name)
allure.dynamic.parent_suite(f"{suitename}.{platform_name}.{board_name}")
if runner_name is not None:
allure.dynamic.tag(item.config.getoption("--runner-name"))
yield
We are also publishing our GitHub pages to a different repo. The report that we generate separates the main branch from all other branches so that we don’t confuse PR test failures from merge and scheduled tests on main.
Here’s a list of files you might want to browse to see the current state of our Allure Report generation:
- test.yml: run the tests and execute report generation at the end
- reports-allure-publish.yml: collect report files, generate site, and publish
- pytest-hil/plugin.py: add dynamic tags/parameters/suites to integration tests
We care a lot about software and firmware quality here at Golioth and we want our users to feel confident that they are using an SDK that is reliable and tightly coupled to the cloud services that enable the underlying IoT Devices. If you decide to follow in our footsteps, we hope this will deliver a better understanding of your CI runs as it has for Golioth!
