Major U.S. Aerospace Prime

A major U.S. aerospace prime was in the final integration and debug push for a custom QEMU system emulator hosted by LynxOS-178 and running within a LynxSecure subject. The emulator is heavily multi-threaded, with separate threads handling emulated code execution, filesystem I/O, and multiple forms of inter-subject communication over virtual network links and LynxSecure FIFOs.

Correct operation depends on tight real-time behavior: the I/O threads must respond promptly while the “vcpu” thread responsible for the emulated code must receive as much execution time as it demands. Balancing these competing demands against LynxOS-178’s priority-preemptive scheduler surfaced several elusive timing and threading issues that were very difficult to observe by conventional means.

To tune and balance system behavior, the team used thread priorities to take advantage of the LynxOS-178 priority-preemptive scheduler, and relied on SpyKer-TZ — Lynx’s event-tracing and trace-viewing/analysis utility, built on Percepio Tracealyzer technology — to capture live event traces and visualize how threads were being scheduled. This gave the team rapid, direct insight into complex, simultaneous thread interactions that would otherwise be extremely hard to diagnose.

SpyKer-TZ quickly exposed a LynxOS-178 device driver that implemented a poor spinloop while blocked waiting for I/O. This caused the read thread to consume a large amount of CPU while blocked, starving lower-priority threads. The behavior was easily identified in the SpyKer-TZ trace viewer; after the customer switched to a non-blocking mode of the driver, follow-up trace viewing confirmed the errant behavior was eliminated and lower-priority threads were no longer starved.

A tougher, late-stage hurdle was a timeout on a customer-designed inter-subject communication path, caused by a software heartbeat not arriving when expected — seemingly indicating that the vcpu thread was not getting enough execution time or was not being properly woken by QEMU. SpyKer-TZ analysis instead showed the vcpu thread receiving ample execution with no obvious gaps. By adding user-specified event generation from within the application, the team tightly correlated SpyKer-TZ profile data with specific instrumented waypoints in the code, revealing that the vcpu thread was spending an inordinate amount of time on a task unrelated to running the emulated application.

That insight pinpointed the true cause: QEMU’s code-generation buffer, which holds translated code, was filling and taking an unusually long time to flush before newly translated code could continue executing — and the flush delay was triggering the inter-subject communication timeout.