Not long ago I had the opportunity to stand in front of hundreds of senior executives — leaders drawn from across the high-integrity software industry, convened by one of the companies at the center of it. It was the kind of room where people certify software for a living: software for systems that fly, drive, and keep people alive. At Lynx. we build the foundational software those systems run on, so this was very much my world. What follows is the case I wanted to make to that room.
The Most Misleading True Thing
Nearly every organization can now recite the same number: time to first draft, down by some impressive percentage. Code, requirements, test cases — a working draft appears faster than any tool we have ever had. It is true. It is also the most misleading true thing you can say about AI in our world.
Because in our world, “it works” has never been the bar. The bar is that you can prove it works and prove how you built it. At the highest assurance levels, writing the code was never what consumed the schedule. What consumed it was the evidence: full traceability from requirement to design to code to test; structural coverage proving every path was exercised and justified; qualification of any tool that touches the certified artifacts. That is the expensive part. That is what gates a release. None of it was ever the typing.
So, when AI compresses the time to a first draft, it compresses the part that was already cheap. The assurance work is untouched. The bottleneck did not disappear — it moved. A great deal of our industry has quietly booked a productivity win on an activity that was never on the critical path.
AI collapses the cheap slice and leaves the expensive one intact — the win comes off the part that never gated a release.
This is not just our experience. A widely cited 2025 randomized controlled trial found that experienced developers were actually slower with AI assistance — even as they were convinced, they were faster. The gap between how AI feels and what it delivers is itself worth sitting with.
The Lifecycle has a Direction — and AI Reverses it
One layer deeper is the part that rarely gets said out loud. Our development lifecycle assumes a direction. Requirements come first; design follows; code exists because a requirement demanded it. Traceability is a by-product of the order you worked in — you can walk back up from any line of code to the intent that justified it. Code with no requirement above it is, by definition, extraneous: a finding, regardless of whether it functions correctly.
AI inverts that order. It hands you the implementation first and leaves you to reverse-engineer the requirement that fits it. It is worth being precise about where this shift takes place. High-level requirements — the true requirements, born of elicitation, architecture, and testability judgment — are still largely human work. The inversion lands at the boundary between low-level design and code. That is where AI produces the artifact first and the intent gets backfilled afterward.
AI delivers the bottom of the V first. The danger is that the inversion climbs — as AI moves up into design, and eventually requirements.
The Prototype Trap
Every engineer has felt the pull here. There is always a prototype that mostly works, and deriving the requirements from it is the path of least resistance. You write the requirement to describe what the code already does. The trace link that results is perfect — and hollow. It describes what the code does instead of governing what the code should do. The requirement was fitted to the implementation, so of course they agree. The authority that is supposed to flow from intent down to code is now running backwards.
A requirement fitted to working code produces a flawless-looking trace that certifies nothing.
A Hollow Trace is Worse Than a Missing One
A missing link is visibly incomplete. It gets caught in review, flagged, and fixed — that is what review is for. A hollow link is the dangerous one, because it looks finished. It passes review precisely because it is fluent. Researchers have shown that developers using AI assistants can produce less secure code while feeling more confident it is correct. Fluency reads as correctness. AI has raised the floor of “looks done” without touching the floor of “is correct” — and in an assurance regime, that is exactly the wrong thing to get better at.
Where This Caught Us
The honest version of this is the useful one, and it happened to us. In our own development, we set out to use AI to help write low-level requirements. We did the sensible thing and fine-tuned a model on our own past requirements, drawn from across projects — who better to learn from than our own design history? And we were not naive: we kept a human in the loop, with experienced engineers reviewing the output as the safety net.
The output was good. It reproduced the patterns faithfully, read like our own requirements, and matched the examples it had learned from. That is exactly why it slipped through. Because it looked familiar, it drew less scrutiny, not more — from our own experts. But it was a new project, and the project-specific checks and quality expectations had shifted. The model had learned to replicate the structure of past requirements beautifully. That is not the same as meeting the requirements of the system in front of us. We had a perfectly shaped artifact that did not actually govern the thing we were building — the hollow trace, from the inside.
We caught it. But the lesson was not “review harder,” because the whole problem was that the artifact was built to pass review. A human gate, on its own, is not enough when the output is fluent. So, we changed two things: we curate the training data for the intended use rather than feeding the model everything we had ever written, and we added an automated check into the pipeline that scores generated requirements against the quality criteria our experts actually care about, improving on them iteratively. We stopped trusting fluency — and the fix was not a better prompt.
The Fix is Structural: Anchor the Intent
If the problem is that AI reverse-fits intent to implementation, the answer is to give it an anchor it cannot reverse-fit. That means ordering, not exhortation. The design vocabulary comes first — a controlled schema of the allowed requirement types and their attributes, tied to the architecture patterns, so that generated design is built to transform and trace to code. The model then generates against that schema: an implementer satisfying a fixed structure, never the origin of it. When the design layer is governed before generation, traceability holds by construction. Nothing is orphaned, and nothing is reverse fitted.
Freeze the intent before the model runs, and the trace is genuine by construction rather than reconstructed after the fact.
One Principle — and The Questions it Leaves Open
If there is a single line to take from all of this, it is this: AI is an implementer of intent — never its origin. Wherever that line holds, AI is a genuine gift. Wherever it slips, you are manufacturing confidence you have not earned. And it generalizes well beyond aviation to any domain that must answer for its software — avionics, medical, rail, automotive, security.
The hard parts are not solved. How do you freeze intent fast enough to keep the speed AI just gave you? What counts as sufficient human authority over a frozen requirement? Where does tool qualification land when the generator is probabilistic? How do you detect a hollow trace at scale, in review rather than after? And the hardest in practice: in a brownfield system with no clean architecture or schema to anchor to, how do you stay disciplined enough to treat working code as inspiration only?
Keep AI Accountable to Engineering Intent
AI can help accelerate development, but high-assurance systems still depend on requirements that govern the design, traceability that holds through implementation, and evidence that stands up to review.
Talk with us about applying AI within a disciplined, high-assurance development process.