22Governance Across the AI System Life Cycle

“A model that worked on launch day may be wrong six months later without a single line changing.”

An AI system is not a product delivered and forgotten. Data changes, users discover new uses, and an external model is updated by its provider. That is why the sixth domain of Annex A requires governance to be embedded in every stage of the life cycle, just as we embedded security in the software development life cycle.

Text in this figure

Planning · Purpose and use · Data · Source, quality, rights · Design and build · Fairness and security · Verify and evaluate · Pre-launch criteria · Deployment · Disclosure, human oversight · Operate and monitor · Drift and incidents · Across the cycle: impact assessment · documentation and versions · event logs · Figure 28

From Idea to Retirement

• Concept: What is the problem? Is AI the right solution at all? What level of impact is expected?
• Data: Collection and preparation with clear rights, checks for representativeness and bias, and documented provenance.
• Build or select: Developing the model or choosing a ready-made one and its supplier, with performance, security and fairness requirements.
• Verification: Performance and robustness testing, red teaming, and review of the impact assessment before approval.
• Deployment: Disclosure to users, usage limits, and explicit approval from the system owner.
• Operation and monitoring: Event logs, monitoring for drift and complaints, and periodic review of the impact assessment.
• Retirement: A plan for withdrawing or replacing the system, and what happens to the data and model afterwards.

Integration with the Software Life Cycle

Teams that have applied security across the software life cycle are halfway there. They only need to add an AI question at each gate: an impact question in requirements, a human oversight question in design, fairness and robustness questions in testing, and a drift question in operation.

Managing AI Incidents

An AI incident may not be a breach: an offensive output published in a client’s name, an agent that sent an email nobody asked for, or a model that started rejecting a group of users. It is handled with the same approach we learned in Chapter 13: contain, assess, escalate, communicate and review. With special evidence: the prompts, the outputs and the model version at the time of the incident.

Lessons Learned

1. 1The AI life cycle has seven stages from concept to retirement.
2. 2Every gate in the software cycle needs an AI question.
3. 3AI incidents include harmful outputs, not just breaches.
4. 4Agents are service accounts with least privilege and a full log.