“AI hallucinates. I cannot use something like that in safety-relevant development.” We hear this sentence in almost every first conversation with heads of development — and it is not a prejudice, but sound engineering judgement. Anyone who builds products where mistakes are expensive or dangerous is right to distrust a system that can produce convincing-sounding errors.
Nevertheless, the reverse conclusion — keeping AI out of development altogether — would be wrong. The right question is not whether, but where: at which points of the V-model does AI carry reliably — and at which does it do more harm than good? This article is our honest map. It comes not from vendor slides, but from 15 years of test engineering in the embedded field and the daily use of AI in our own organisation.
The basic rule: Checking is easier than creating
Almost all sensible uses of AI in engineering follow one pattern: AI produces a proposal; a human with professional judgement checks and decides. This works well wherever checking a result is considerably faster than producing it — an engineer assesses twenty proposed test cases in a fraction of the time it would have taken to devise them independently.
Conversely: where checking is as laborious as doing it yourself — or where nobody checks any more because the result looks so convincing — the benefit tips over. That is precisely where the stories arise that feed engineers’ distrust.
Where AI carries reliably today
Requirements reviews: the strongest lever
Checking requirements for unambiguity, contradictions, completeness and testability is classic legwork: important, tiring, and in practice always under-resourced. Here AI is in its element — it reads a thousand requirements without fatigue, finds the wording that admits two interpretations, and the contradiction between requirement 47 and 812.
Crucially: the result is a list of findings, not a verdict. Each finding can be checked against the source in seconds. A false finding costs one click; a discovered contradiction saves weeks. The risk-benefit ratio is hard to beat — which is why we almost always recommend requirements reviews as a pilot project.
Test case derivation: proposal, not sign-off
Deriving test cases from a clean requirement — normal cases, boundary values, error cases — is structured work with clear rules. AI proposes more complete sets of test cases than emerge by hand under time pressure, and does not forget the inconvenient edge cases.
But: the AI proposes; the test engineer decides what is adopted into the test specification. Responsibility for test coverage remains with the human — the AI ensures they exercise it with better material.
Documentation and traceability upkeep
Keeping the links between requirement, implementation and test result up to date is the work that is needed in every audit and deferred in every project. AI can suggest links, report orphaned requirements, flag outdated documentation sections and pre-draft change descriptions. The appeal: all of this happens within the existing ALM environment — Polarion, Codebeamer, Jira — and strengthens auditability rather than endangering it.
Code review as an additional instance
An AI review does not replace a human reviewer — it is an additional instance before them. It finds the forgotten null check, the deviation from the coding guideline and the copy-paste error before a colleague spends their time on them. Human reviews do not become superfluous as a result, but better: they can concentrate on architecture and logic instead of formalities.
Finding triage in the pipeline
When the nightly build is red, detective work begins: which commit, which module, genuine defect or flaky test? AI can pre-analyse build logs, cluster findings, name suspects and write the summary needed in the morning stand-up. It is allowed to be wrong here — it is a prioritisation aid, not a decision.
Knowledge search across the project archive
“Didn’t we solve this problem once before, back in 2019?” AI-assisted search across Confluence, legacy projects and ticket history answers such questions in seconds, with source references. Especially in teams where experienced colleagues are the project memory, this is worth its weight in gold — more on this in our article on knowledge loss.
Where AI does not carry — and where it deceives
The other half of the map is just as important. We advise against these uses — in some cases as a matter of principle, in others given the current state of the technology:
Final sign-offs and safety decisions. No AI system signs anything off with us. Sign-offs require responsibility, and responsibility requires a person with a name. This is not a technical limit, but a fundamental organisational decision — and under the EU AI Act, for many applications, simply mandatory.
Inventing requirements. AI can check, structure and rephrase requirements. But it must not create domain requirements that no subject-matter expert has dictated — otherwise a plausible-sounding conjecture sits at the start of the traceability chain. That is the classic hallucination accident in the engineering context.
Calculation and safety evidence. Design calculations, worst-case analyses, safety cases: what counts here is not plausibility, but correctness. Language models do not calculate — they formulate. For evidence, verified tools and the four-eyes principle belong at the table, not AI prose.
“The AI says the test is green.” Test results must be reproducible and traceable. AI may summarise results and report conspicuous patterns — but the statement “passed” must come from the test system, not from an interpretation.
Areas without a data foundation. Where processes are undocumented and data unstructured, AI has nothing to work on. Those who introduce it anyway automate the chaos. First optimise the processes for AI, then introduce AI — in that order, never the reverse.
The four test questions for every use case
Whether a specific use belongs in the left or the right column can be decided with four questions:
- Checkability: Can a domain expert check the AI result quickly and reliably against the source? If not — hands off.
- Cost of errors: What does it cost if the result is wrong and slips through? The higher, the tighter the human control.
- Reversibility: Can the effect be undone? A discarded proposal is harmless; shipped firmware is not.
- Data situation: Is there structured, well-maintained data for the AI to work on? Without a foundation, nothing can be built.
Anyone who asks these four questions consistently needs no faith in AI — and no fear of it. They need a method. That is exactly what our staged model is for: governance and data maturity first, then the process, then — for those who want it — the product.