A thousand crashes per night: anti-cascade as a tested requirement

On a command node, the failure of one module must not become the failure of the node. OC2-Edge states this property as a requirement: a crash stays confined to the perimeter of the module that suffers it, supervision detects it, cleans up its resources and restarts it according to its policy, with no effect on the other modules nor on the COP (Common Operational Picture, the shared tactical picture). An untested requirement is an opinion. This one is tested every night, by fault injection, in the public repository.

Deterministic supervision: pure state machine, injected time

OC2-Edge supervision is the component that decides the fate of each module: start, monitoring, confinement, restart, revocation. Its core is a pure state machine: a function that receives a state and an event, and returns a new state and a list of actions. It does not read the clock, does not touch the filesystem, does not open a socket. Time is injected into it as one event among others.

This architectural constraint has two direct consequences.

First, determinism: given an identical sequence of events, supervision produces an identical sequence of decisions. Restart strategies, escalation delays and revocation thresholds are tested without starting a single real process, by unrolling event sequences in memory. A complex failure scenario (crash bursts, crossed timeouts, revocation during a restart) fits in a unit test that runs in milliseconds.

Second, replayability: any anomaly observed in production or in a test campaign reduces to a sequence of events. That sequence replays as is against the state machine, in the repository, as a regression test. Yesterday’s bug becomes tomorrow’s test, identically, without depending on a platform or on a thread race. This approach belongs to fault injection testing: causing controlled failures to verify that the system responds to them as specified [7].

The pure state machine tests the decision logic. It does not test the real world: processes that actually die, file descriptors that leak, cgroups left orphaned. That is the role of the chaos campaigns.

Chaos v0: a thousand cycles, zero orphaned resources

The first campaign, chaos v0, targets the endurance of the elementary lifecycle. Every night, CI runs 1,000 crash and restart cycles of a real module on a real node (emulated x86-64 and ARM64). The principle follows chaos engineering: state a steady-state hypothesis, inject real failures, verify that the hypothesis holds [1].

Each cycle applies the same sequence: the module starts, the harness kills it (brutal termination, no notice, no cleanup), supervision detects the death, cleans up, restarts, and the harness verifies the return to steady state. The success criterion rests on two invariants, checked after each cycle and at the end of the campaign:

• availability: the node’s other services respond throughout the campaign; no crash propagates outside the module’s perimeter;
• zero orphaned resources: after the last cycle, the inventory of the node’s processes, mount points, virtual network interfaces and cgroups is identical to the initial inventory. A leak of one resource per cycle would be invisible in a unit test; over 1,000 cycles, it becomes a blatant inventory gap. That is the reason for the volume: the number does not aim to impress, it aims to accumulate.

Chaos v1: sixteen modules, crash storm, convergence

The chaos v1 campaign extends the model to concurrency. The node hosts 16 modules; the harness triggers a crash storm: simultaneous terminations, in bursts, on random subsets of modules, including during the restarts caused by the previous crashes. This is the regime where naive supervisors degrade: crossed locks, lagging event queues, restart loops that saturate the node.

The success criterion is convergence: after the injections end, the node returns within bounded time to a state where every module is either active or stopped in accordance with its policy, and where the zero orphaned resources invariant holds. The random seed of each campaign is logged: a storm that reveals a defect replays identically, which brings concurrency debugging back into the realm of the reproducible.

The negative control: proving that the test bites

A test that cannot fail proves nothing. A chaos harness can pass for the wrong reasons: an injection that does not actually kill the process, a check that queries the wrong node, an invariant that compares two empty inventories. This risk has a classic remedy in experimental method: the negative control.

Each chaos campaign therefore includes a run where isolation is deliberately broken. The harness starts a module in a configuration stripped of one isolation layer, such that a crash there has observable effects outside its perimeter. The expected result of this run is test failure. If the harness declares this run successful, the harness itself is defective, and the entire campaign is invalidated. The negative control verifies that the test bites: that it actually detects the class of defect it claims to detect, and that the green of the other runs has evidential value.

Onion-layered isolation

The confinement verified by these campaigns rests on successive layers, each with its own role. No layer suffices alone; their composition provides the defense in depth.

• Namespaces: the Linux kernel gives the module a private view of the system (processes, mount points, network, identifiers) [2]. The module does not see the other modules; what it cannot see, it can neither read nor kill.
• Cgroups: the kernel bounds the module’s resources (CPU, memory, number of processes, input-output) [3]. A module that leaks or loops exhausts its own budget, not the node’s. This is the layer that turns a resource failure into a local event.
• Seccomp: a filter restricts the system calls the module can invoke, to the reduced list it needs [4]. The kernel attack surface exposed to the module shrinks accordingly; a compromised module has fewer levers.
• MicroVM: for the highest-risk modules, execution moves into a lightweight virtual machine, on the model of Firecracker [5]. The boundary is no longer a structure of the shared kernel but a hardware virtualization interface: an escape requires crossing the hypervisor.
• WASI: modules compiled to WebAssembly run under WASI (WebAssembly System Interface), a model where every capability (file, network, clock) must be granted explicitly [6]. By default, the module can do nothing; denial is the initial state and authorization the exception.

The chaos harness exercises these layers with the node’s real modules. The negative control described above consists precisely of removing one of these layers and verifying that the harness notices.

What this commits to

The chaos v0 and v1 campaigns are part of the public repository’s CI chains, alongside the other suites (~930 automated tests passing, 9 CI chains). Their harnesses, invariants and execution logs are in the repository: a DTIB (defense technological and industrial base) instance can replay them on its own configuration, with its own modules, and publish its own results. Article no. 2 describes how to reproduce the single-fault scenario on an emulator [see /en/blog/rejouer-la-demo-emulateur/]; the full campaigns run with the same tools. The anti-cascade property is not a marketing argument: it is a requirement, a harness that verifies it, and a negative control that verifies the harness.

Sources

1. Principles of Chaos Engineering, “Principles of Chaos Engineering”, 2019, https://principlesofchaos.org/
2. man7.org, “namespaces(7), Linux manual page”, https://man7.org/linux/man-pages/man7/namespaces.7.html
3. man7.org, “cgroups(7), Linux manual page”, https://man7.org/linux/man-pages/man7/cgroups.7.html
4. The Linux Kernel documentation, “Seccomp BPF (SECure COMPuting with filters)”, https://www.kernel.org/doc/html/latest/userspace-api/seccomp_filter.html
5. Firecracker, “Firecracker: Secure and fast microVMs for serverless computing”, https://firecracker-microvm.github.io/
6. WASI, “WebAssembly System Interface”, https://wasi.dev/
7. A. Basiri et al., “Chaos Engineering”, IEEE Software, vol. 33, no. 3, 2016, https://arxiv.org/abs/1702.05843