Via Insita

Zephyr template project for event-driven application development

Sat, 03 Jan 2026 10:45:57 +0100

It’s been quite some time since I got a taste of working with Zephyr so I decided to revisit some old code I have written to see if I can turn it into something useful that other developers could use.

After a few interations, I’ve decided to reshape this old code into an event-driven template project for developing C++ applications. You can take a look at the template on my Github repository where you will see further instructions on building and flashing your project with the template.

It is basically a blinky with sane hardware support for user interaction (i.e. button and LED) so you can easily start from that point and then upscale the complexity of your project as needed.

If you would like to support the work I do, consider donating here.

Setting up Yocto for efficient embedded Linux systems development

Sun, 27 Jul 2025 06:24:13 +0200

In this blog post, I will explain how I’ve set up my workstation for building custom Linux image for STM32MP157D-DK1 and demonstrate a basic build in this setup.

Introduction

Similarly to my previous posts, this writeup is based on Bootlin’s Yocto labs. And as always, thanks to Bootlin for making their materials free and open source!

Before starting with Yocto, make sure you check all the required boxes for working with Yocto as described in Yocto’s system requirements reference manual.

This especially goes out to all Arch-based distro users as it doesn’t support natively!

Even though it is possible to work with Yocto on an Arch-based distribution, the build environment must be at least containerized which many users prefer and I would argue makes the builds more easily reproducible and transferable. If one opts no to use podman or docker to do do and still work with Yocto on Arch or similar non-supported distros, get ready to jump through some hoops because there will be quite few of them.

Also, make sure your home directory is not encrypted with eCryptFS since OpenEmbedded cannot be used on top of it due to limitation in file name lengths.

If you are using Ubuntu: Ubuntu 24.04 added an apparmor policy preventing usage of unprivileged user namespace restrictions to improve security. This prevents bitbake from working, because it uses namespaces to forbid untracked downloads outside of the do_fetch task (we will get to terminology and technicalities later, don’t worry if you don’t understand these terms). Ironically, bitbake does this to improve security. This results in operation not permitted error which can be disabled by running:

echo 0 | sudo tee /proc/sys/kernel/apparmor_restrict_unprivileged_userns

You will need to run this command time you reboot your machine or look at the Ubuntu 24.04 Release Notes to see how to disable this restriction permanently.

First Yocto build

SIDE NOTE: For a quick build, you can check out Yocto Project Quick Build.

The first prerequisite to building with a Yocto image for SMT32MP157D-DK1 is to install required packages and to download data which we will use for our board.

First, install the required packages:

$ sudo apt install gawk wget git diffstat unzip texinfo gcc build-essential \
chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils \
iputils-ping python3-git python3-jinja2 python3-subunit zstd liblz4-tool \
file locales libacl1

Then, obtain data we will use for working with target board in this setup:

cd
$ wget https://bootlin.com/doc/training/yocto-stm32/yocto-stm32-labs.tar.xz
$ tar xvf yocto-stm32-labs.tar.xz

After that, update your distribution:

$ sudo apt update
$ sudo apt dist-upgrade

Next, download Yocto (I used the scarthgap version of Poky reference distribution):

$ git clone https://git.yoctoproject.org/git/poky
$ cd $HOME/yocto-stm32-labs/poky
$ git checkout -b scarthgap-5.0.1 scarthgap-5.0.1

Then, return to project root directory and download the OpenEmbedded and STM32MP layers:

$ cd $HOME/yocto-stm32-labs/
$ git clone -b scarthgap https://git.openembedded.org/meta-openembedded
$ git clone https://github.com/STMicroelectronics/meta-st-stm32mp
$ cd meta-st-stm32mp
$ git checkout b820cf3a1a855d2bd95969251e6465e281502759

You now just need to set up the build environment:

$ cd $HOME/yocto-stm32-labs
$ source poky/oe-init-build-env

and you are now ready to use the bitbake which is Yocto’s main build engine.

To build specifically for SMT32MP157D-DK1, you must specify the target machine in the conf/local.conf configuration file.

To save disk space on your computer, you can add INHERIT += "rm_work" at the end of the same file which will remove package work directory once a package is built.

Also, the build configuration needs to be aware of the OpenEmbedded and STM32MP layers. This is done by editing the $BUILDDIR/conf/bblayers.conf configuration file by adding full paths to layers to the BBLAYERS variable.

If not done already, make sure to configure your git username and email as some recipe builds can fail without it:

$ git config --global user.name "Your Name"
$ git config --global user.email "your@email.com"

To test image build with set configuration, run the following from the sourced environment:

$ bitbake core-image-minimal

This build will initially last for quite some time. Mine was building for an hour if I recall correctly. Anyway, the build might fail for various unexpected reasons. One might be due to RAM limitations where you can build the failing recipe individually will all the RAM available for it. Simply rerunning bitbake core-image-minimal sometimes did the trick for me because bitake caches various stages of the build so it doesn’t have to rebuild the same recipes over again when rebuilding the full image.

In that case, make sure to search though the internet for your specific build issue if one occurs.

Once the build is finished, you can observe the output image under $BUILDDIR/tmp/deploy/images/stm32mp1.

Now, the easiest way to transfer the newly built image onto the board is by using an SD card. Insert your SD you plan to use to store the bootloader, kernel and root filesystem files into the SD card reader of your computer.

To generate the final image for the board run the following script located in $BUILDDIR/tmp/deploy/images/stm32mp1/scripts:

./create_sdcard_from_flashlayout.sh \
../flashlayout_core-image-minimal/extensible/FlashLayout_sdcard_\
stm32mp157d-dk1-extensible.tsv

Now, flash the SD card with image:

WARNING: make sure you are 100% sure you know which partition are you writing to because overwriting other paritions might cause irreversible changes!

$ umount /dev/mmcblk0p*
$ sudo dd if=../FlashLayout_sdcard_stm32mp157d-dk1-extensible.raw of=/dev/mmcblk0 bs=8M \
conv=fdatasync status=progress

To observe the booting process of our custom Linux distribution we have places onto the SD card, we first need to set up serial communication with SMT32MP157D-DK1. Plug the USB-A to micro USB-B cable on the Discovery board to micro USB port (CN11) which is actually ST-LINK. This is a debug interface and exposes multiple debugging interfaces including a serial interface. When you plug it in your computer, a serial device should appear. The exact name of it can be observed by looking ath the output of dmesg utility. Once we have identified the device, we can pass it to picocom in order to start the serial communication with the board:

picocom -b 115200 /dev/ttyACM0

You can now insert the SD card into the dedicated slot on the STM32MP157D-DK1 and then power up the baord by connecting the USB-C cable to the board (CN6). You should now see boot messages in picocom. Wait until the login prompt and then enter root as username and root as password.

Great! The board has booted and now we have access to the shell!

Build process and basic Yocto lexicon

in principle, you can think of Yocto as a build system that takes custom built and open-source components as an input and builds binary packages (i.e. distributions) as its output.

Yocto lexicon

bitbake: Yocto’s main build engine that functions like a task scheduler parsing text files (i.e. recipes) to what to build and how to build it
recipe: a text file that describes how to fetch and build a software component - program, library or an image
task: a specific step of the build process / a smaller execution unit of a recipe being built
- for example: fetch,configure, compile, package
metadata: configuration files, recipes, classes and include files organized in layers which act as an input to bitbake
layer: a set of configuration files, recipes, classes and include files with a common purpose
- for example, meta-stm32mp15x for STM32MP157x board support package
- openembedded-core is the core layer that all other layers a built on top of
poky: Yocto’s reference distribution (tools and metadata) which serves as a starting point for building custom embedded Linux system

Advanced Yocto configuration

Setting up Ethernet communication and NFS on the board

It’s very impractical to to reflash the root filesystem onto the SD card every time Yocto rebuilds an image. This is why it is very useful to set up networking between the development workstation (host) and the target machine because the target machine can then access workstation files using NFS:

We first need to set the kernel boot arguments U-Boot will pass to the Linux kenrel on the target at boot time. To do that, modify the mmc0_extlinux/extlinux.conf configuration file and cahnge the APPEND file:

APPEND root=/dev/nfs rw console=ttySTM0,115200 nfsroot=192.168.0.1:/nfs,vers=3,tcp ip=192.168.0.100

Setting up Ethernet communication on the workstation

Now with a network cable, connect the Ethernet port of the board to the computer over the USB Ethernet adapter if your computer already has a wired connection to the network. A new network interface should appear on your computer that you can check by running ip a.

The network interface name is enxxx.

You now need to make sure you set the static IP of that port to be in a different subnet of where your computer is connected. You can do that over the command line:

nmcli con add type ethernet ifname en... ip4 192.168.0.1/24

Change en... and 192.160.1/24 to fit your network configuaration.

Setting up the NFS server on the workstation

First install the NFS server on the workstation by running the following:

$ sudo apt install nfs-kernel-server
$ sudo mkdir -m 777 /nfs

Then make sure this directory is used and exported by the NFS server by adding the following line to the /etc/exports file:

/nfs *(rw,sync,no_root_squash,subtree_check) # NOTE: modify the last parameter to no_subtree_check if needed!

Finally, make the NFS server use the new configuration:

$ sudo exportfs -r

Adding SSH support

We will now add a package called dropbear to support connecting to the board over SSH by appending the IMAGE_INSTALL variable $BUILDDIR/conf/local.conf.

The Dropbear SSH server is now enabled and runs as a service on the STM32MP157D-DK1 board.

We first need to put the rootfs under the NFS root directory so that it is accessible by NFS clients. To do taht, uncompress the archived output image in the previously created /nfs directory:

sudo tar xpf $BUILDDIR/tmp/deploy/images/stm32mp1/\
core-image-minimal-stm32mp1.rootfs.tar.xz -C /nfs

Now boot the board and open a separate terminal to access it using ssh:

ssh root@192.168.2.100

Setting up TFTP

Setting up TFTP is extremely useful since it lets us bypass having to reflash the SD card for every test.

We first need to install a TFTP server on our host machine:

$ sudo apt install tftp-hpa

We then need tp copy the Linux kernel image and the device tree to the TFTP server home directory (which is /srv/tftp in my case as defined in /etc/default/tftpd-hpa) so that they are reachable by the TFTP server.

We then need to boot the board and interrupt the booting process to enter in the U-Boot shell and change the bootcmd variable to load the kernel image and the device tree over TFTP:

setenv ipaddr <your-client-ip>
setenv serverip <your-server-ip>
setenv bootcmd 'tftp 0xc2000000 zImage; tftp 0xc4000000 dtb; bootz 0xc2000000 - 0xc4000000'

And finally, set the bootargs variable as follows:

setenv bootargs root=/dev/nfs rw console=ttySTM0,115200 nfsroot=<your-server-ip>:/nfs,vers=3,tcp ip=<your-client-ip>
saveenv

For more information on how to set up TFTP, make sure you check out section on this link.

And that’s it. You now have a well configured and hassle-free setup for developing with Yocto!

Using Yocto to create a Linux-based gaming console

Mon, 21 Jul 2025 16:01:39 +0200

In this blog post, I will explain how I built the world’s worst gaming console (at least by today’s standards but it makes it one of the best anti-gaming consoles today in my opinion):

As you can see from the picture above, the user experience is terrible and this console is extremely poorly portable because of cabling and jumper wires used to connect the Wii Nunchuck and the STM32MP157D-DK1 board. Other than that there is only one game you can play and it’s nInvaders - a ncurses Space Invaders clone.

This is exactly why it’s one of the best anti-gaming consoles.

Introduction

in this blog post, I will be using the setup I describred in my Setting up workspace for developing and building custom Linux-based distributions with Yocto blog post.

Similary to my previous posts, this writeup is based on Bootlin’s Yocto labs. And as always, thanks to Bootlin for making their materials free and open source!

Before we continue, I would like to give a quick refresher of some basic terms used when working with Yocto as described in my previous post with regards the Setting up Yocto for efficient embedded Linux system development:

bitbake: Yocto’s main build engine that functions like a task scheduler parsing text files (i.e. recipes) to what to build and how to build it
recipe: a text file that describes how to fetch and build a software component - program, library or an image
task: a specific step of the build process / a smaller execution unit of a recipe being built
- for example: fetch,configure, compile, package
metadata: configuration files, recipes, classes and include files organized in layers which act as an input to bitbake
layer: a set of configuration files, recipes, classes and include files with a common purpose
- for example, meta-stm32mp15x for STM32MP157x board support package
- openembedded-core is the core layer that all other layers a built on top of
poky: Yocto’s reference distribution (tools and metadata) which serves as a starting point for building custom embedded Linux system

If you want more details, either check out the previous blog post or better yet, check out the official Yocto Project terms.

Writing a recipe to support nInvaders and integrating it to an existing layer

Before downloading nInvaders, make sure you run the follwing command in poky’s root directory:

source oe-init-build-env

This now lets you use bitbake for your builds.

Next, locate the recipes-extended directory and create a directory for nInvaders where you will download the source code of the game. In my case, this was:

/yocto-stm32-labs/meta-openembedded/meta-oe/recipes-extended/ninvaders

Next, inside the same directory, create a recipe which will tell bitbake how to handle building nInvders. The recipe name should follow the Yocto nomenclature - {recipe_name}_{recipe_version}.bb. More info on Yocto naming conventions can be found here.

The link to my recipe can be found on my corresponding GitHub page. Here is a preview of how I’ve written the recipe:

You can now build the recipe by running:

bitbake ninvaders

Output of the build process should look something like:

Build files of the recipe we have now built is in tmp/work/... folder:

A good practice before generating a new image which will end up on the target board is to verify in which layer is the recipe which contains our application present as well as is the application built for proper architecture which is arm in our case.

Running bibake-layers show-layer ninvaders from the build folder will show where is our recipe present:

Great, our recipe is included in the meta-oe layer. Even though this is not the best practice (and we will fix that in quite soon), we can be sure that it is included in the build since meta-oe is definately included in the build:

Finally, let’s verify that the ninvaders recipe has been built for proper architecture by running bitbake -e ninvaders | ^TARGET_ARCH=:

Once nInvaders recipe has been successfully built, generate a new rootfs image which will contain nInvaders by running:

bitbake core-image-minimal

The image with our ninvaders application is now in /tmp/deploy/images... and we can now extract the image to our NFS root directory on the host machine (/nfs):

where we see our ninvaders app in the third column in the lower mid section. Great!

We can now just connect over ssh to board, and run nInvaders from /usr/bin by running /usr/bin/ninvaders:

We can now use the keyboard to play nInvaders on STM32MP157D-DK1!

Creating and integrating a custom Yocto layer into the build

It is a good practice to leave your custom software fully decoupled from basic Yocto project files. This goes for both adding new recipes or customizing the existing ones. This is why I will go through explaining how to create a new layer with our application which we will integrate into the build.

Let’s add a new layer into the build. This can be done by running:

bitbake-layers reate-layer meta-bootlinlabs --priority 7

You can now verify that the layer has been added either by running bitbake-layers show-layers:

To integrate this newly created layer into the build, add it to the conf/bblayers.conf file:

We can now move the ninvaders recipe into our custom meta-bootlinlabs layer by simply copying all of the contents from /yocto-stm32-labs/meta-openembedded/meta-oe/recipes-extended/ninvaders to yocto-stm32-labs/meta-bootlinlabs. To can now either delete old files from the recipes-extended directory or leave them as they are. bitbake will build the ninvaders recipe with the higher priority first anyway. You can now check that ninvaders recipe is part of our new meta-bootlinlabs layer by running:

bitbake-layers show-recipes ninvaders

Applying patches to an existing recipe by adding joystick support

bitbake allows users to extend recipes by the means of .bbappend files. In this chapter we will go thorugh extending the linux-stm32mp recipe by applying a patch and including it into the build. This patch which will enable us to play nInvaders with Wii’s Nunchuck joystick.

First, we will create a linux-stm32mp_6.1.bbappend file in the /yocto-stm32-labs/meta-bootlinlabs/recipes-kernel/linux directory.

You can now run bitbake-layers show-appends to see all the available bbappend files and the recipe they apply to. There you can see the .bbappend file we have created for our kernel recipe:

The content of the .bbappend file can be seen here, and here is the preview for illustrative purposes:

Notice how the .bbappend file matches the exact recipe name of a file we want to extend. The needed patch for using the Nunchuck joystick as input to our console was provided by Bootlin and is present on this link.

As you can see, defconfig and patches are present in the source file list. Last two lines are used as an inidication to the linux-stm32mp recipe to use kernel configuration as defined by defconfig.

Treeview of the directory responsible for handling the nunchuck at the kernel level is now:

You can now rebuild the kernel to verify that the patches have been included into the build by running cat ... after rebuilding the kernel with bitbake virtual/kernel:

.bbappend file responsible for handling the nunchuck at the application level is here and looks like:

Treeview of the directory responsible for using the nunchuck in the game is now:

Transferring kernel image and device tree blob to TFTP

We can now rebuild the whole core-image-minimal and copy the newly generated kernel and device tree images to the TFTP server home directory (/srv/tftp in my case). Files you are looking for and that you want to transfer to the directory are:

zImage
stm32mp157d-dk1.dtb

Whith these files now present, the board’s bootloade can now load the proper image with proper hardware support.

We can now reset the board and observe the boot logs until we have access to the command line. Near the end of the boot, we should now observe the following.

Testing the Nunchuck

Connection diagram for the Nunchuck can be seen on following images provided by Bootlin:

You can now check that the Nunchuck is present and working by checking the presence of the js0 device file by running:

cat /dev/input/js0

Notice the random characters that appear while both playing with the Nunchuck’s joystick or by even moving the controller since the driver we integrated also handles acclerometer events.

And now finnaly, simply run /usr/bin/ninvaders and use the C button on the joystik to confirm and fire and Z to pause the game:

Amd that’s it.

Using valgrind with vgdb and address sanitizer to analyze memory errors

Tue, 20 May 2025 16:57:17 +0200

In this blog post, I will explain how I used valgrind with vgdb to detect memory errors leaks and misbehaviors:

as well as address sanitizer to do the same, but quicker:

In this post, I will not be focusing on giving a detailed tutorial on how to solve Bootlin’s debugging lab but rather present how I approached solving memory leaks of a running application and how I detected and fixed the root cause of it.

As usual, I would like to give a shout out to Bootlin for making their materials free and open source, including materials I used to demonstrate detecting and fixing memory leaks described in this blog post.

Introduction

In this blog post, I will be using the same setup as described in the GDB setup for debugging embedded applications blog post. As always, special thanks to Bootlin for providing the source code.

On your development host, change directory to valgrind folder, cross-compile valgrind.c and then run it on the target:

Even though there is no segfault, an application might leak memory or perform out-of-bounds access or just simply contain uninitialized memory blocks. One useful tool to detect these “invisible” memory issues is valgrind.

Using valgrind

When we run the application again with valgrind we get the following result:

$ valgrind --leak-check=full ./faulty_mem_app

The output indicates following occurences in faulty_mem_app:

invalid memory write
uninitialized memory
memory leak (memory was allocated but never freed) !slika.png

The call stack indicates the following origin and error propagation: main() -> do_something() -> clear_array()

To fix this issue, we have to check if array access is out of bounds and that malloc() is allocating enough space. Also, this memory has to initialized before use and freed in the end.

Backtracing the call stack to pinpoint where the issue occurs will, in majority of cases, most likely be enough to solve the memory issue.

However, in order to pinpoint each error exactly dynamically and be able to debug with gdb, vgdb can be used. vgdb is a bridge that allows gdb to attach to and control a running valgrind process.

valgrind and vgdb

Using valgrind with vgdb gives us dynamic interactive control by letting us stop execution at memory errors, inspect variables, set breakpoints, and trace exact causes, whereas using plain valgrind only gives a post-mortem report. We will do that remotely on the host using gdb-multiarch. First, run the following on the target:

$ valgrind --vgdb=yes --vgdb-error=0 --leak-check=full ./faulty_mem_app

In order to do remote debugging using vgdb, it has to be in listen mode. Start another terminal in SSH on the target and run:

# vgdb --port=1234

On the host side, run:

$ gdb-multiarch ./faulty_mem_app

and connect to vgdb using the following:

(gdb) target remote 192.168.0.100:1234

We can now debug each error using gdb with valgrind’s supervision which will interrupt the program each time it detects an error. The setup should now look something like this (same picture as the first one):

One more thing to note is that backtrace for leaks is not shown on the target because all libraries are stripped and thus do not have any debugging symbols anymore. This leads to the impossibility to use the DWARF information for backtracing.

First thing I always like to do is to set a breakpoint to main() and start the program to reach the application’s entry point. Then, we can set breakpoints to both functions mentioned in the backtrace above: clear_array() and do_something(). If we were to execute continue command, eventually, gdb would stop execution when valgrind detects an error as shown in the upper-right corner:

until we reach the end of execution:

Solution to memory issues is quite straightforward and consists of changing<= to < in the clear_array() function. You can see the fixed source code here.

recompiling and running valgrind on the recompiled app will now give the following result:

Address Sanitizer

An even simpler way to detect memory errors is to use address sanitizer.

First, make sure you obtain address sanitizer. I downloaded it via apt on my Ubuntu machine:

$ sudo apt install libasan* # replace "*" with corresponding version of gcc you are using

Once this was done, I opened the Makefile and simply added the following compiler and linker flags as shown below and just ran the app as I would when testing it’s behavior on the target:

Conclusion

Address sanitizer gave us a clear backtrace to the clear_array() function as the source of heap buffer overflow and more or less same information that valgrind provided. This is something I would first recommend someone to run their code through since it doesn’t require much effort to set up and yet gives quite a readable insight into where memory leaks lurk. If you want to go a step further and have observe leaky code dynamically, use valgrind with vgdb.

If you would like to support the work I do, consider donating here.

Using strace to analyze application's system calls

Wed, 26 Mar 2025 06:49:05 +0100

In this blog post, I will explain how I used strace to analyze an application for which I didn’t have the access to the source code:

Introduction

In this blog post, I will be using the same setup as described in the GDB setup for debugging embedded applications blog post. As always, special thanks to Bootlin for providing the source code.

In order to see which system calls strace_me application invokes, run the following in the nfsroot/root/strace directory and observe the output as shown in the figure below:

$ strace ./strace_me

Before making sense of the output, let’s see what syscalls actually are.

What is a syscall?

A system call (or just simply, “syscall”) is a special type of a call that an application makes to the operating system’s kernel in order to communicate with it. This is depicted in the following figure I found on Red Hat’s blog:

Applications live in the so called “userspace” and the kernel lives in the “kernelspace” which means that they reside in different protection rings due to the nature and consequences of operations in the respective spaces as depicted in this figure I found on stack overflow:

For example, a program can do a wide variety of things in the userspace like calculating complex math or manipulating data structures, but if it wants do anything with the network or the filesystem for example, it has to talk to the kernel and in order for the application to communicate with the kernel, As noted earlier, syscalls can be traced with a tool called strace which was used to trace syscalls that strace_me application has made.

NOTE: There is also an “inverse” of this communication userspace->kernelspace communication so to speak called signals which the kernel uses to send information to userspace applications. Signals are asynchronous software interrupts that notify a process about some events. A process is a running instance of an application in this case. It is created when you run strace_me application which in turn enables the kernel to send signals to strace_me.

Making sense of the output

Generally, everything up until openat() command is generally just application initialization. The real work of the application begins with the openat() which passes the path to the file which we want to open along with some other options. Make sure to check the file descriptor after the equals sign. The following are taken by:

0 -> stdin
1 -> stdout
2 -> stderr
3 -> anything else

Before diving deeper, focus on two calls as they are usually most telling of what the application effectively does:

openat()
read()

To see this in action, use grep to see which files did the strace_me application open:

I also used grep to see what was being read() and how may times was it called:

NOTE: The 2>&1 part of the strace command is used to redirect standard error stderr(2) to standard output stdout(1) because strace outputs syscalls to the stderr, not to stdout and grep searches lines only in stdout.

Application’s `strace` output analysis

Process execution

The first line that can be seen from the output is the system call that starts the strace_me binary. The return value of 0 indicates successful execution:

execve("./strace_me", ["./strace_me"], 0xbfef60d8 /* 14 vars */) = 0

Memory management

Next three lines are used for memory allocation:

brk(NULL) = 0x491000
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6eed000
mprotect(0xb6ebf000, 8192, PROT_READ) = 0

brk(NULL) checks the program break (heap start), while mmap2(...), the application allocates 8KB of anonymous private memory, most likely for stack or heap. Finally, mprotect(...) sets memory permissions by making a region read-only.

Filesystem access

Next two lines:

access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)

indicate that there is no custom shared library preloading and that no pre-cached library paths were found, meaning ld.so has to search directories manually.

The following line:

openat(AT_FDCWD, "/etc/fstab", O_RDONLY|O_LARGEFILE) = 3

indicates that strace_me successfully opened the /etc/fstab file which contains filesystem mount information.

This line:

openat(AT_FDCWD, "/tmp/tmp.GchbQ27l", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)

indicates that the application tries to open a temporary file called tmp.GchbQ27l but fails.

The following lines:

rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0xb6de2510}, 8) = 0
kill(0, SIGALRM) = 0

are used by strace_me to set a signal handler for SIGALRM and keep it at default behavior with SIG_DFL flag. It then sends SIGALRM to itself (kill(0, SIGALRM)) which leads to process termination. SIGALRM is a signal that the kernel sends to the userspace program when working with timers which in turn means that strace_me is dependent upon some kind of a timer in some way.

In the end, strace_me terminates normally.

Summary

We now have enough information about strace_me to make following conlusions and assumptions:

manages memory via mmap2() and mprotect(), likely enforcing stack security
dynamically linked, looks for shared libraries but doesn’t preload anything
checks system files (/etc/fstab) - might be inspecting mounts
tries to open temp files in /tmp/ but fails - possible optional config check
sets SIGALRM to default behavior - program will exit when it receives an alarm signal indicating some kind of a timeout

What strace_me is most likely doing

Based on the strace output, the strace_me is a test application that interacts with system files and memory. It and exits cleanly upon receiving a SIGALRM signal without doing much.

Conclusion

strace lets you trace system calls that an application makes which provides more information about how does the application interact with the system it runs on. Useful calls that provide useful information that you should have an eye on are openat() and read(). Generally, everything up until openat() syscall is basically just application initialization. Every syscall afterwards gives us more meaningful information about the application.

If you would like to support the work I do, consider donating here.

Using ltrace and software shims to bypass a credential-checking algorithm

Sun, 23 Feb 2025 08:57:50 +0100

Overview

In this blog post, I will explain how I have used ltrace to analyze dynamic library calls from an application requesting user credentials and managed to hack a credential-checking algorithm:

Introduction

The setup for demonstrating ltrace and software shims is described in GDB setup for debugging embedded applications blog post. As always, special thanks to Bootlin for providing the source code.

ltrace is a tool to trace shared library calls used by a program and all the signals it receives. This is especially useful when we don’t have access to the source files of the binaries we are analyzing. A software shim is a piece of code (a library to be more precise) that lives in between a program and some other library that the program uses. A shim intercepts calls from the program to the library allowing us to monitor and change the behavior of those calls.

These tools are extremely useful to analyze and change the behavior of an existing program for which we don’t have source code and that we cannot recompile.

My mind is unironically blown.

Running an authentication-constrained application

Since our application uses a local dynamic shared library which is not in the default paths expected by the linker, we provided that path using LD_LIBRARY_PATH by running the following on the target:

# export LD_LIBRARY_PATH=$PWD

And when we now run the application:

# ./authent

we can see that our application is failing to correctly authenticate the user:

Bypassing credential-checking algorithm

By using ltrace on the target, we can trace the application on the target in order to understand what is going on and based on that trace which function fails:

# ltrace ./authent

As you can see, the function that fails is al_authent_user() and this is the function which we will override in a new file, overload.c which will print the user, password and simply return 0:

In this case, I looked at the authent_library.h inside the ltrace\ directory to conclude what the function prototype should be.

Now, to use the overload we just created, compile your source file on the development host with the following command:

$ ${CROSS_COMPILE}gcc -fPIC -shared overload.c -o overload.so

This command generates a position-independent shared library overload.so which can be loaded at any valid memory address without requiring modification.

Finally, run the application and preload the new library using the following command on the target:

# LD_PRELOAD=./overload.so ./authent

This command sets the LD_PRELOAD environment variable which tells the loader to first check our newly compiled overload.so library when running the authent binary. When al_authent_user function is called in the authent binary, our function with the same name will be called and credentials will be printed on the console:

If you would like to support the work I do, consider donating here.

Using a coredump with GDB for post mortem crash analysis

Sat, 01 Feb 2025 21:37:19 +0100

I configured the Linux kernel core to dump a core file that I analyzed in order to find the root cause of a segmentation fault:

In this blog post, I will set up the working environment exactly the same as I described in previous post. A coredump is a file that that holds the recorded state of the program’s memory at moment of crash. This kind of static analysis is useful when code crashes occur in end-of-line testing, or worse yet, out in the field when it has been shipped to the customer. Wherever they occur, coredump files are sent to developers for a debugging session.

Introduction

One thing that has to be taken into account in software development is the difference between debug build and release build binaries. Debug builds have a larger memory footprint due to additional debug symbols that compiler uses for debugging purposes, whereas build binaries are often stripped off of this additional information. As suggested by Jacob Sorber in his Can I Debug Release Code? video, the best of both worlds approach would be to compile your release binaries with debug symbols, but then strip them afterward before shipping them to the customer which will decrease binary size but still leave you with an option to debug a program using a coredump. I highly recommend you check out this video if you I piqued your interest on this topic. As always, special thanks to Bootlin for providing the source code.

Running coredump through GDB

In order to configure the Linux kernel to generate such a file, run the following on the target machine:

# ulimit -c unlimited

which removes the size limit of coredumps that are going to be saved. By default, they are set to zero on most machines (i.e. they don’t get generated).

Them, run ./linked_list again and observe that a core file has been generated:

On your host machine, change the permissions by running:

$ sudo chown $USER:$USER core

and run the failed program with the coredump in GDB which will point you to the exact moment your program crashed:

$gdb-multiarch ./linked_list ./core

Coredump analysis is less dynamic than the usual way of debugging binaries we execute live on GDB. It nevertheless still allows you to pinpoint the exact moment when the crash occured by manipulating the coredump file. Since it is only a memory snapshot and the program is not running anymore, we cannot step through the code by using GDB commands such as continue or step but we can print variables that are in scope.

I have already demonstrated one way of finding and resolving the root cause of the segmentation fault in this post in this particular case and we know what the causes it. That is why I will take a different approach this time for demonstration purposes.

I will define custom commands in order to ease printing elements of the singly linked list we are working here with. Let’s backtrace the program execution a bit and define the command that will print out elements of our list:

where slh_first and sle_next allude to the following macros as described in sys/queue.h:

SLIST_ENTRY(name) next - this macro creates a structure containing a pointer called sle_next (“singly linked entry next”) pointing to the next node
SLIST_HEAD(name_list, name) name_list - this macro creates the list head that contains a pointer called slh_first (“singly linked head first”) which points to the first element of the list

Even though this is quite telling and enough to make conclusions if you were to look under the hood of the word_list and linked_list.c, I will lay out additional considerations that might be useful in pinpointing the exact root cause of the segmentation fault.

Additional considerations

Even though running the coredump GDB points you to the exact function in the source code where the crash occured, the assembly view of the memory, as well as examining the state of the CPU registers can provide additional useful information that can help us in backtracing the root cause of the segmentation fault.

To print the state of the registers at the moment of crash, run info registers:

Based on the pc (program counter) register value, we can conclude where the execution has stopped - 36 bytes into the display_linked_list() function.

Based on the lr (link register) register value, which holds the reutrn address, we can conclude that display_linked_list() was still executing when the appliction crashed.

Based on the r2 register value, which in this case stores function arguments, we can see that it equals to ASCII ’m' indicating that overflowing occurs at the ’m' character of the word “fermentum” as concluded in the previous post.

To access the assembly view of the function where the crash occured (i.e. display_linked_list()), run dissasemble display_linked_list:

0x4c8a04 <display_linked_list+36>: ldr r1, [r3] indicates that the CPU is trying to dereference the memory address stored in ‘r3’ register.

When we examine the value stored in register r3, we get familiar feedback: Cannot access memory at address 0x6c which is a suspiciously low memory address where no user-defined variables should reside indicating that an invalid pointer dereferencing might have occurred. And it did.

If you would like to support the work I do, consider donating here.

Detecting and fixing the cause of a segmentation fault on an embedded Linux system

Mon, 30 Dec 2024 23:03:48 +0100

I used GDB on my host machine to detect root cause of a segmentation fault on the STM32MP157D-DK1 (I am using gdb-dashboard here btw):

In this post, I will not be focusing on giving a detailed tutorial on how to solve Bootlin’s debugging lab but rather present how I approached solving a crash caused by a segmentation fault and how I detected and fixed the source code snippet that caused the segmentation fault.

Introduction

To demonstrate how to solve an application crash caused by a segmentation fault, I will be using source code that Bootlin provided for their Linux debugging, profiling, tracing and performance analysis training which I highly recommend you check out. Before continuing, I would like to express my gratitude towards everyone responsible for making Bootlin happen. I really appreciate their desire to contribute to the world of embedded Linux and that they keep their materials free and open source. Kudos!

Also, here is a picture of the STM32MP157D-DK1 I was working with:

Reproducing the segmentation fault

If you haven’t read GDB setup for debugging embedded applications, I advise you to read it so you get a better understanding of what will be laid out in the rest of this blog post.

On your host machine, make sure you prepare the current shell for cross-compiling by running: $ export CROSS_COMPILE=/home/$USER/debugging-labs/buildroot/output/host/bin/arm-linux-

Then, run make in the nfsroot/root/gdb directory to build the linked_list binary. Now, if you run linked_list on your target, you will get a segmentation fault:

Generally, a segmentation fault indicates that a program attempted to access a memory location that it was not allowed to access. That means that one of the following scenarios is the root cause of the application crash we trying to resolve:

access violation
- reading or writing to an invalid memory address such as a null pointer, an uninitialized pointer, a pointer to a memory address that has been freed or is out of scope, an out-of-bounds array index
improper memory usage
- recursive function calls can exhaust the stack memory causing it to overflow
- improper use of dynamic memory allocation or deallocation like using free multiple times on the same memory block or writing beyond the allocated memory causes heap corruption
- attempting to access data at an address not aligned to the data type’s requirements like reading a 4-byte integer from a 3-byte aligned address
logic error occurence in the source code
- dereferencing a pointer before it has been properly initialized (i.e. dereferencing a null pointer)
- passing invalid pointers to functions

Setting up the debug environment

As hinted at in the beginning of this blog post, we will be using a remote GDB setup since our target does not embed a fully featured GDB, but only a gdbserver that allows connecting with a remote fully featured version of GDB that we have running on our host machine. Let’s start our program on the target by using gdbserver in multi mode:

# gdbserver --multi :2000 ./linked_list

On the host side use gdb-multiarch to attach to the process on the target:

$ gdb-multiarch ./linked_list

when in gdb, enter the following:

(gdb) target extended-remote 192.168.0.100:2000
(gdb) set sysroot /home/<user>/debugging-labs/buildroot/output/staging/

You can now enter the TUI either by ctrl + x + a or by typing lay next and hitting enter a couple of times until you see the source code, assembly view and the GDB console:

NOTE: Besides GDB TUI, there are several other user interfaces with debugging options such as gdbgui and gdb-dashboard which I will be using the most from the next blog post onwards.

Application debugging

The first thing I recommend doing is setting a breakpoint in the main function so you can see where exactly the segfault occurs when you backtrace it after the program has crashed:

(gdb) break main

Then, run the program to reach the breakpoint at the main function and continue execution until the crash occurs:

(gdb) run
(gdb) continue

Once the program segfaults, show the backtrace to pinpoint the function that causes the segfault:

(gdb) backtrace

As you can see, the line causing the crash in the main program is located in the linked_list.c at 81. Add the breakpoint to that line:

(gdb) break linked_list.c:81

Restart the program in GDB by running start or run. You will reach the main function again since this is the first breakpoint we set and then run continue until you reach the function causing the problem which is display_linked_list(). Enter step to enter this function.

A very useful GDB command in this case is print which is used to print variables inthe current scope:

You can see strings being printed line after line in the target machine’s console. Stepping and printing will give you a glimpse of how the function is being executed sequentially. For education and demonstration purposes, feel free to step through every entry of the word list until you reach the segfault. Generally, if typing the same command becomes way too repetitive way too quickly, consider using conditional breakpoints.

When you reach the last entry being printed, notice the difference between what is being shown in the GDB console and on the target machine’s console. One says “fermentu” and the other “fermentum”. This should raise an eyebrow…

If you now step and try printing the next entry in the linked list, you will see that you are pointing to an address you cannot access which is causing the segmentation fault:

Now, if you take a look at the name struct in linked_list.c you will see that it has a member name which is an array of 8 characters:

And “fermentum” has 9 characters…eureka! The program is trying to display a word of 9 characters when it only has 8 character slots for a given entry from the word list. That is the reason why the program segfaults.

There are more intelligent ways of solving this problem but I will propose the simplest and quickest one. I looked at the world list being printed and none of the words contain more than 20 characters. Let’s change char name[8] to char name [20] and recompile the program. Now, we run the program we can see it execute correctly. Awesome!

As stated earlier, conditional breakpoints may provide a more elegant way of solving segmentation faults, especially when some parts of code are iterative in nature. In order to set a breakpoint when the pointer to the next element of the linked list is nonexistent, do the following:

Restarting the application in GDB will now result in:

Summary

There are many ways to go about solving an application crash on an embedded device, and this is one of them. In general, you want to consider the following:

compile you application with debug symbols (g3, ggdb if necessary)
establish connection between the development host and the target machine (via SSH or serial communication)
create a breakpoint in the main function and let it run to provoke the segmentation fault and use backtracing to pinpoint the function causing where the fault occurs
add new breakpoint to that function
run the program again and continue execution until you reach the newly added breakpoint and step into the function where the fault occurs
modify the source code, recompile it and run your faultfree application!

If you would like to support the work I do, consider donating here.

Setting up GDB for debugging embedded devices

Mon, 30 Dec 2024 23:03:48 +0100

In this blog post, I will explain how I have set up GDB on my host machine for debugging an application on STM32MP157D-DK1:

Development workstation setup

I have set up an NFS server on my development machine as well as configured U-Boot on STM32MP157-DK1 so that any built binaries of interest on the host are easily accessible on the target machine. Also, I have set up SSH to establish a connection with the target board. For more details, check out the first two chapters of this document from Bootlin.

GDB setup for embedded systems

By its default configuration, GDB targets binaries that are being run on the same architecture as the host system GDB is running on:

Screenshot captured from Memfault’s Interrupt page describing tools they use

When targeting an embedded device, GDB has to be configured in a way that it supports the target architecture of the program being debugged:

Screenshot captured from Memfault’s Interrupt page describing tools they use

Binaries generated for embedded devices are often stripped of debug information to save storage space. This is also why native GDB should not be installed on an embedded target (around 2.5 MB on x86).

For that reason, remote debugging is preferred. ARCH-linux-gdb is used on the development host instead of the native GDB allowing the developer to use all features that native GDB provides. On the target side, gdbserver is used (around 400 KB on arm). It allows connecting to a fully featured remote GDB. This server handles requests from the GDB client, such as setting breakpoints, reading memory or stepping through code.

Before we start debugging, keep in mind that there are several lightweight graphical interfaces that make debugging easier. One is the default TUI that comes built-in with GDB that you enter by typing lay next when in GDB and the other is gdb-dashboard that I have just discovered recently and which I am quite fond of. It creates a .gdbinit file and populates it with sensible configuration for debugging. It enables you to use well-known shortcuts within GDB like ctrl + r for reverse searching of GDB commands you typed previously which the out-of-the-box installation of GDB does not provide. It also appends additional debug information and coloring to your console that I find quite useful.

Running ./linked_list on the target will cause a segfault. Now, let’s start the program on the target using gdbserver in multi-mode and break down what each of the commands mean:

# gdbserver --multi :2000 ./linked_list

gdbserver starts the GDB server which acts as an intermediary between ./linked_list and a GDB client on the host machine
--multi flag allows multi-process debugging and launching sessions dynamically
- instead of automatically attaching to ./linked_list at startup, this option sets up the GDB server to accept multiple debugging sessions and wait for a remote GDB client to specify which executable to debug
:2000 specifies the port that the GDB server will listen on for incoming GDB client connections - ./linked_list normally specifies the program that GDB server will debug, but this argument is ignored because of the --multi option and the target program is not launched until the remote GDB client specifies it

In a separate terminal on your host machine, run export CROSS_COMPILE=/home/$USER/debugging-labs/buildroot/output/host/arm-linux-

this export needs to be done in each shell in which CROSS_COMPILE is going to be used or added to the shell configuration file (I am using bash, so .bashrc)
export command sets the CROSS_COMPILE environment variable and makes it available to all child processes indicating to the build system that the binary being built is meant to be run on a different architecture than the one you are working on
/home/$USER/debugging-labs/buildroot/output/host/ is the directory where Buildroot has placed the cross-compilation tools.

Next, install gdb-multiarch and run gdb-multiarch ./linked_list. You have now entered GDB and inside it, run the following:

(gdb) target extended-remote 192.168.0.100:2000

this allows connecting to a remote target GDB server and also reconnecting after the server restarts
- target sets the target system GDB will control
- extended-remote allows multi-process debugging where GDB starts new programs on the target or attaches to running processes
this command is commonly used with gdbserver started in --multi mode

NOTE: this applies if the connection with the board was established via SSH. If it was established over a serial connection, run target remote /dev/ttyUSB0, or which device it was mapped on when the board was connected to the host machine.

Finally, run:

(gdb) set sysroot /home/<user>/debugging-labs/buildroot/output/staging/

this tells GDB where to find the root filesystem of the remote target on the local, host machine where files for debugging are located
when debugging remote programs, GDB might need access to shared libraries, binaries and other files from the target’s root file system which are not available on the local host machine’s filesystem
- by using this setting, GDB now points to a mirror of the target’s filesystem from where it can load the correct library versions, display complete stack traces and resolve symbols for debugging

Automating repetitive tasks

Instead of retyping exact and specific commands every time you establish a connection with a target you device, you can edit your .gdbinit file which does it for you. Also, GDB has powerful Python support that it can be used to automate application debugging more efficiently by utilizing GDB’s Python API

Useful resources

A couple of useful sites I have found regarding using GDB is Jacob Sorber’s Debugging C Programs playlist, Memfault’s Interrupt GDB blogs and of course Bootlin’s Linux debugging, profiling and tracing and performance analysis training. I highly recommend you check out other topics of theirs if you are interested in embedded software development. Special thanks to Bootlin for providing the source code.

If you would like to support the work I do, consider donating here.

About

Mon, 30 Dec 2024 22:57:35 +0100

My name is Martin Cvitić and I am an embedded software developer from Croatia.

Via Insita is my personal website where I mostly write about embedded systems focusing mainly on embedded systems running Linux.

General areas of my technical interest include open source software and hardware as well as cybersecurity.

Thanks for stopping by!

If you would like to support my work, consider:

buying me coffee
donating to my Monero address:

48wxtnuXnv18xeYMBjoLHqZ3ySt5fMSUvPLbSSbmWacL2GDwJaBv49PPzPurCsvdBMajpk93aQgHAW8DzS6wHBNuUQFmHYS

Via Insita

Zephyr template project for event-driven application development

Setting up Yocto for efficient embedded Linux systems development

Table of Contents

Introduction

First Yocto build

Build process and basic Yocto lexicon

Yocto lexicon

Advanced Yocto configuration

Setting up Ethernet communication and NFS on the board

Setting up Ethernet communication on the workstation

Setting up the NFS server on the workstation

Adding SSH support

Setting up TFTP

Using Yocto to create a Linux-based gaming console

Table of Contents

Introduction

Writing a recipe to support nInvaders and integrating it to an existing layer

Creating and integrating a custom Yocto layer into the build

Applying patches to an existing recipe by adding joystick support

Transferring kernel image and device tree blob to TFTP

Testing the Nunchuck

Using valgrind with vgdb and address sanitizer to analyze memory errors

Table of Contents

Introduction

Using valgrind

valgrind and vgdb

Address Sanitizer

Conclusion

Using strace to analyze application's system calls

Table of Contents

Introduction

What is a syscall?

Making sense of the output

Application’s strace output analysis

Process execution

Memory management

Filesystem access

Summary

What strace_me is most likely doing

Conclusion

Using ltrace and software shims to bypass a credential-checking algorithm

Overview

Table of Contents

Introduction

Running an authentication-constrained application

Bypassing credential-checking algorithm

Using a coredump with GDB for post mortem crash analysis

Table of Contents

Introduction

Running coredump through GDB

Additional considerations

Detecting and fixing the cause of a segmentation fault on an embedded Linux system

Table of Contents

Introduction

Reproducing the segmentation fault

Setting up the debug environment

Application debugging

Summary

Setting up GDB for debugging embedded devices

Table of Contents

Development workstation setup

GDB setup for embedded systems

Automating repetitive tasks

Useful resources

About

Application’s `strace` output analysis