Find Breaking Changes in Zephyr Using Git Bisect

One of Zephyr’s greatest strengths is the large number of maintainers and contributors and the frequency with which they are merging changes. In the past four years that I’ve been working with Zephyr, it’s become remarkably stable and feature-filled. However, the increased stability you get with each new release sometimes comes with breaking changes. Tracking those down quickly is a skill that will lower your stress level. Let’s discuss one approach to finding and resolving breaking changes in Zephyr: using git bisect.

What is Git Bisect?

Git bisect is a command built into git that pinpoints the commit at which a breaking change was introduced. Basically, you tell Git where the last time the code was working is located, where the non-working code is located. Git will then checkout new commits for you to label good or bad. Just run your test, label the commit, and git does the rest.

The astute reader will recognize this as a binary search. You take all of the commits and test one in the middle. Depending on whether that test succeeds or fails you eliminate half of the commits before or after and then start the process again on the smaller set.

Difference When Using Git Bisect with Zephyr

Git bisect is nearly a perfect troubleshooting tool for Zephyr. The maintainers of the project are strict about requiring clean and granular commits, in order to improve consistency during the review process. As a result, every commit should build and run. Ideally you’ll have very few cases where you get a false positive while bisecting.

However, the manifest system that Zephyr is based upon means you need to take a few extra steps while bisecting. That’s because git will change the Zephyr hash, but it will not run a west update to ensure all the modules match the current hash of Zephyr. Let’s jump in and look at a few approaches that you can use.

Your Project Setup Matters

This article came to mind when I worked on updating which version of Zephyr the Golioth Firmware SDK uses. Our SDK includes Zephyr as a module, which means we have our own west.yml (actually west-zephyr.yml) that includes a hash (revision) to lock Zephyr to a specific version.

manifest:
  projects:
    - name: zephyr
      revision: v4.3.0
      url: https://github.com/zephyrproject-rtos/zephyr
      west-commands: scripts/west-commands.yml
      import: true

  self:
    path: modules/lib/golioth-firmware-sdk

This is a great approach to use with your own projects, as it ensures you always know which version of Zephyr (and its inherited modules) was used for any given commit in your project.

But, when using git bisect on the Zephyr tree of this project, you can’t run west update because it will just check out the revision stored in this file. You can take two approaches to overcome this situation:

1. Manually change the number in your manifest file to match each hash selected by git bisect
   • This is prone to user error if you forget to change the hash or make a mistake when updating.
   • It doesn’t require any other changes to your project.
   • This approach is used in the walkthrough below.
2. Change your project to use the west manifest in the Zephyr tree
   • Add any additional configuration for your project into a manifest that you place in Zephyr’s submanifests/ folder. This can be complicated if your project manifest is rather involved.
   • The manual work of changing your project manifest file for each git bisect step is no longer needed.

Today we’re going to stick to the manual version. Let’s dive in!

Walkthrough: Git Bisect with Zephyr

For today’s walkthrough, let’s assume you have the v0.21.1 of the Golioth Firmware SDK set up as a manifest repository by following the directions in the README. The west manifest for this project is located in your_workspace_folder/modules/lib/golioth-firmware-sdk/west-zephyr.yml and the Zephyr tree is found in your_workspace_folder/zephyr.

Start the Bisect

We’re going to find where the Twister tests for the SDK broke between Zephyr v4.2.1 and Zephyr v4.3.0 so we open a terminal in the Zephyr tree to conduct the bisect:

$ git bisect start
status: waiting for both good and bad commits

$ git bisect good
status: waiting for bad commit, 1 good commit known

$ git bisect bad v4.3.0
Bisecting: a merge base must be tested
[413b789deb391d3a37d06b463288a5fe765ee57e] release: Zephyr v4.2.0

The process starts by running git bisect start, and you may abort this process at any time by running git bisect reset. We then constrain the bisect process by labeling a good commit with git bisect good (using the hash of currently checked out commit) and a bad commit with git bisect bad v4.3.0 (here we use a tag but you can also use a hash).

Now that we’ve labelled both good and bad commits, git picks out the next commit to test. Our case is a special one: git moved to an earlier commit which was just some internal git housekeeping.

Test the Commit, but Update Modules First!

Here’s where the special step for Zephyr comes in. We need to update the Zephyr modules. If you’re bisecting a project that uses the west manifest inside the Zephyr tree (zephyr/west.yml), you can update modules directly. But I need to manually update the Zephyr hash in my project manifest file.

# Find the hash of this commit
$ git rev-parse HEAD
413b789deb391d3a37d06b463288a5fe765ee57e

# Update ../modules/lib/golioth-firmware-sdk/west-zephyr.yml with the above commit for Zephyr

# Update Zephyr modules
west update

# Test this commit (change this for your use case!!)
$ scripts/twister --platform frdm_rw612 -T ../modules/lib/golioth-firmware-sdk/examples/zephyr/logging --prep-artifacts-for-testing
$ if test -e twister-out/frdm_rw612_rw612/zephyr/sample.golioth.logging/zephyr/zephyr.hex; echo "PASS"; else echo "FAIL"; end
PASS

Here you can see that I ask git for the hash of this commit. This is used to update the Zephyr revision in the west manifest file for the project. With that updated, we run west update, which will not change the Zephyr tree, but it will/may change what version of modules are checkout out in modules/lib, etc.

The test step is going to be different every time you bisect. In my case, I know that Twister tests place the compiled files in a different directory than our CI is expecting. So I’m just using a fish shell one-liner to test if the hex file is where I expect.

In this case the first test passed!

Mark the Commit Good or Bad to Continue the Bisect

Based on the results of the test from the previous section, mark the commit as good or bad and Git will continue the bisect process.

$ git bisect good
Bisecting: 3871 revisions left to test after this (roughly 12 steps)
[3a6ae2b01c9babd8be320a6912123de53ebba748] driver: sdhc: add sdmmc for sama7g5

Since my test from the previous section passed, I mark the commit as good and Git checks out a new commit. It also helpfully informs me that I will need to perform about 12 more tests to pinpoint the issue. Just run the same steps from the previous section again.

In my case, the twister build failed on the second step of the bisect so that is definitely a bad commit!

$ git bisect bad
Bisecting: 1933 revisions left to test after this (roughly 11 steps)
[56a527321ed977338d6ca4f0892114f988bf845a] include: drivers: input: enhance Doxygen documentation for CY8CMBR3xxx

Git has checked out another commit, so keep doing the test-and-mark loop until you get to the end of the bisect process.

Finishing the Bisect

When you get through all the bisect steps, git should tell you which commit is the first one with a breaking change.

$ git bisect good
9219c81b66f48523f0cd3b7b7cac8cf3dadb84b2 is the first bad commit
commit 9219c81b66f48523f0cd3b7b7cac8cf3dadb84b2
Author: Anas Nashif <[email protected]>
Date:   Fri Jul 11 11:36:03 2025 -0400

    twister: make no-detailed-test-id default

    Make this option default and reduce amount of details displayed by
    twister.
    This options has been enabled in CI for a while now with the intention
    of making it default.

    Signed-off-by: Anas Nashif <[email protected]>

 .github/workflows/clang.yaml                    | 2 +-
 .github/workflows/twister.yaml                  | 6 +++---
 .github/workflows/twister_tests_blackbox.yml    | 8 ++++----
 scripts/ci/test_plan.py                         | 4 ++--
 scripts/pylib/twister/twisterlib/environment.py | 4 ++--
 scripts/tests/twister/conftest.py               | 1 +
 6 files changed, 13 insertions(+), 12 deletions(-)

In my case, this is a perfect outcome, so perfect that I immediately knew I should blog about this. I was trying to find out what in Zephyr v4.3.0 broke all of the Twister tests that we were running in v4.2.1 and here is the smoking gun!

As you can tell from the excellent commit message, one of the Twister default values was changed. By looking at the code changes in this commit, I was able to discover that all of our tests now need to specify the --detailed-test-id flag.

All that’s left now is to return our Zephyr repository to normal functionality by resetting the bisect.

$ git bisect reset
Previous HEAD position was b68058787e9 counter: cmsdk_apb_dualtimer: Use clock freq from DT clocks
HEAD is now at 64e98eb5278 release: Zephyr v4.2.1

Don’t forget that if you have been manually updating your project manifest for each bisect hash, that change will need to be reverted as well.

A Strong Incentive for Disciplined Commits

This is a great argument for using strict discipline in your git repositories. Ensure that no commit breaks the build, that commits are granular, and commit messages are verbose. If you’re successful at this, then git bisect will someday help you pinpoint problems and deliver useful information in the process.

Bonus: Automation Using Git Bisect Run

I asked for feedback from the Golioth engineering team before I started writing this, to see if there were any tips on using bisect. Marcin Niestroj is the one who suggested using the Zephyr submanifests mentioned earlier in this post. Sam Friedman suggested using git bisect run which was also new to me. Certainly if you take the time to write a short script the run command is pretty powerful.

I was able to pinpoint the breaking change by running just four commands:

$ git bisect start
$ git bisect good
$ git bisect bad v4.3.0
$ git bisect run bash bisect_test.sh

Literally, the rest happens automatically and the offending commit just pops up on the screen! Of course you need to write the test script which I did using bash shell this time.

#!/bin/bash

# Get the hash
this_hash=$(git rev-parse HEAD)

# Write hash to project manifest for Zephyr revision number
sed -i "4s/.*/      revision: ${this_hash}/" ../modules/lib/golioth-firmware-sdk/west-zephyr.yml

# Update the modules
west update

# Build the twister test
scripts/twister --platform frdm_rw612 -T ../modules/lib/golioth-firmware-sdk/examples/zephyr/logging --prep-artifacts-for-testing

# Test to see if build worked
if [[ -f "twister-out/frdm_rw612_rw612/zephyr/sample.golioth.logging/zephyr/zephyr.hex" ]]; then
    exit 0
else
    exit 1
fi

Really the only special sauce here is the sed command which replaces line 4 of the manifest file. The rest is the same commands we ran manually, with the exception that we exit with an error code instead of echoing out a PASS/FAIL. This is slick if you take the time to set up your test!