How AI Agents Erode Software Governance and Audit Trails
JFrog research warns that autonomous AI agents in the SDLC vanish after sessions, leaving organizations without the accountability needed for compliance.
Key points
- AI agents perform SDLC tasks but their reasoning and tool calls disappear when sessions end.
- Traditional git logs capture code changes but not the agent’s decision-making process or model used.
- New supply chain risks include prompt injections and malicious Model Context Protocol tools.
JFrog research indicates that the speed of software development has outpaced the standards designed to govern it. Autonomous AI agents now handle pull requests, dependency selection, and infrastructure changes across the entire Software Development Life Cycle. These agents act as team members, yet they are ephemeral, meaning their reasoning and decisions vanish immediately after a session ends.
Why do AI agents break traditional governance models?
Traditional SDLC trust relies on human accountability, such as peer reviews and traceable intent linked to human identities. Compliance frameworks like SOC 2 and FedRAMP were built on the assumption that a human made every decision. JFrog notes this assumption no longer holds because agents are autonomous. When humans review thousands of lines of agentic code, fatigue can lead to rubber-stamping approvals, bypassing the safety checks originally intended for human-led processes.
What specific risks do ephemeral agents introduce?
The primary risk is the loss of provenance. Git logs show final code states but do not record which AI model was used, which tools were called, or the prompt iterations that produced the result. This lack of visibility makes it impossible to hold ephemeral processes accountable. Additionally, agents may make non-deterministic decisions or introduce latent vulnerabilities that pass functional tests. These issues create new attack surfaces, including prompt injections and malicious Model Context Protocol servers.
How are regulations addressing these gaps?
Regulatory bodies are beginning to require stricter logging for AI systems. The EU AI Act mandates that high-risk AI systems allow for automatic event recording and effective human oversight during use. Similarly, the OWASP AI Security Verification Standard requires verification that AI interactions are logged with session context and that audit logs capture security-critical actions, including approver identity and decision outcomes. These rules align with the security-by-design expectations of the EU Cyber Resilience Act.
What changes are needed for supply chain security?
Organizations must attach provenance data directly to SDLC entities like commits, artifacts, or released versions. This attachment enables automatic release gating through policies and supports post-release troubleshooting. Without this linkage, the new actors in the supply chain—agents, harnesses, and large language models—remain unverified. JFrog suggests that current practices must evolve to meet the governance approach outlined in ISO/IEC 42001 and other emerging standards.
Background: Large language models
Large language models are statistical engines that predict the next token based on patterns in their training data. They do not "know" facts; they generate plausible continuations. You must verify their outputs, secure their inputs against injection, and manage their context windows carefully to prevent data leakage.
Read the full guide: Large Language Models: 8 Technical FAQs for Secure Development
What to do and how to stay safe: JFrog
- Audit current logging mechanisms to ensure they capture AI agent session context and tool usage, not just final code commits.
- Implement policy-based release gates that require verified provenance data before deploying artifacts produced by autonomous agents.
- Review compliance frameworks against EU AI Act and OWASP AISVS requirements to identify gaps in AI oversight and record-keeping.
- Monitor for new supply chain attack vectors specific to AI, such as malicious Model Context Protocol tools or prompt injection attempts.
Step-by-step guide: How AI Risk Management Works: The Hidden Failure Points
General security guidance from the Sandbox Dispatch newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
Do git logs capture AI agent decision-making processes?
No, git logs only capture the final code state and do not show the agent’s reasoning, tools invoked, or model used.
What regulatory requirements apply to high-risk AI systems?
The EU AI Act requires automatic event recording and human oversight, while OWASP AISVS mandates logging of AI interactions and security-critical actions.
How can organizations maintain accountability for AI-generated code?
Teams should attach provenance data directly to commits and artifacts to enable automatic release gating and post-release troubleshooting.
