Article Updated 01.04.2026

"AI Audit Kit: a reproducible standard for AI auditing of large codebases"

Abstract

AI Audit Kit can be described as a portable standard operating framework for AI-assisted security and reliability auditing. Unlike the typical scenario, where an agent receives one long prompt, finds some of the problems, and loses coherence by the next step, this approach formalizes the full lifecycle of working with defects: from the initial audit to triage, orchestrated fixing, and independent re-verification.

The main merit of the project lies not in the volume of source code, but in the fact that it defines a process contract. Findings turn into artifacts, fixes go in batches, progress is recorded in a tracking document, and every important change goes through a reviewer loop. As a result, AI stops being merely an idea generator and becomes part of a reproducible engineering loop.

1. Problem statement

Practical AI auditing has a recurring problem: the quality of the result depends heavily on the specific session, the context window, random prompt engineering, and the operator's patience. Even when the agent has found real bugs, several systemic weaknesses arise next:

  • findings are not normalized and stay inside the chat;
  • severity and priorities are not collected into a single backlog;
  • fixes scale poorly when defects number in the dozens or hundreds;
  • there is no independent mechanism that checks the completeness of each fix;
  • the process is hard to port to another repository without rewriting the workflow almost from scratch.

AI Audit Kit addresses exactly this: not "how to ask the model to look for bugs," but "how to make an AI audit repeatable and operationally viable."

2. Method

The project defines a four-phase cycle:

Phase 1: Audit
Phase 2: Triage
Phase 3: Fix Pipeline
Phase 4: Review and Hardening

In the Audit phase, the agent goes through the codebase and saves each problem as a separate markdown report with fields for severity, category, location, root cause, impact, reproduction, and suggested fix. This is a simple decision, but a critical one: a finding becomes not an ephemeral fragment of the model's answer, but a full unit of work.

In the Triage phase, the reports are aggregated into TRACKING.md, where the total number of findings, the distribution by severity, and the current processing status are visible. This layer is needed for planning and observability: without it, a large-scale AI audit falls apart into disconnected episodes.

In the Fix Pipeline phase, bugs are grouped into batches by files, domains, and repositories. Then each step is executed sequentially: the implementer makes the fix, writes tests, runs the quality gates, and creates a commit, while the reviewer analyzes exactly that commit and returns a formal verdict.

The Review and Hardening phase closes what should not depend on the local success of an individual step: running the full set of checks, spot-checking, and final validation of the result.

3. Architectural advantage

The key engineering idea of AI Audit Kit is the separation of roles and artifacts.

First, the bug report is separated from the fix. This means that audit and remediation are no longer mixed in a single model response.

Second, the implementer and the reviewer are separated from each other. One agent makes the fix, another assesses the completeness of the fix and the quality of test coverage. This reduces the risk of the false sense of completion that is characteristic of single-agent scenarios.

Third, the whole process leaves material artifacts behind:

  • bug reports;
  • tracking dashboard;
  • step logs;
  • review verdicts;
  • fix iteration logs.

This is exactly what makes the project closer to an operational standard than to a set of prompt templates.

4. Why this is better than ad hoc prompt engineering

A one-off prompt can be useful on a small repository, but it ports poorly to a large system. The bigger the codebase, the more noticeable three fundamental problems become.

The first problem is loss of state. The model may find useful defects, but by the next step the operator has to manually restore context, re-select priorities, and decide which findings are compatible in one fix batch.

The second problem is the absence of proper quality control. If the same agent that wrote the fix also asserts its correctness, the probability of missing an incomplete fix or a regression stays high.

The third problem is reproducibility. Most "successful" AI audits are not portable: another developer, another repository, or even another week of work effectively requires reassembling the method from scratch.

AI Audit Kit removes these three classes of problems by formalizing the structure of the process.

5. Real-world use on a large closed project

The value of AI Audit Kit is clearest not on a toy repository, but in real work on a large closed multi-repo project. As of April 1, 2026, this standard is used there as an active working loop, not as a demonstration template.

The practical process is arranged as follows.

First, the large codebase was cut into dozens of functional analysis zones. Then the audit ran over several iterations, in each of which separate agents were launched in parallel on different module groups. After each pass, findings were collected into standardized markdown reports and consolidated into a shared tracking document.

In the current live run, this loop has already covered 33 module groups and cross-cutting concerns, gone through 4 audit iterations, and used 24 agent invocations for a systematic review of the codebase. The result was hundreds of findings across security, reliability, correctness, and observability. This matters not as a marketing figure, but as confirmation that the approach holds up not for a single local case, but for sustained work on a large system.

Next the fix pipeline comes into play. For critical and high-priority defects, separate orchestrator scripts are launched. They work not in "let the agent try to fix something" mode, but in a mode of sequential steps:

  1. a batch of related bugs is taken;
  2. the implementer reads the bug reports and the relevant code;
  3. the implementer makes the fix, writes tests, and passes the quality gates;
  4. a commit is created;
  5. the reviewer analyzes exactly that commit and returns a structured verdict;
  6. if the fix is incomplete, the step goes to the next fix iteration.

What is especially important, the reviewer loop in real use really does catch defects that the implementer missed on the first pass. The logs of the current run show that some high-severity steps went through several review iterations before the verdict became PASS. In other words, the independent check here is not decorative but functional.

6. Empirical observations

Real-world use supports several practical conclusions.

First observation: on large codebases, the main value of AI arises not at the stage of initially finding a single bug, but at the stage of normalizing a large volume of findings into a single pipeline.

Second observation: the reviewer loop pays off. Even a strong implementer regularly leaves incomplete fixes, insufficiently strict tests, or locally correct but conceptually flawed solutions.

Third observation: tracking and step-by-step logs radically improve the manageability of the process. When a step hangs on a timeout, gives a partial result, or fails review, this becomes visible immediately rather than being discovered by accident several commits later.

Fourth observation: portability is achieved not through a universal "magic" prompt, but through a minimal set of stable artifacts and orchestration rules.

7. Limitations of the approach

AI Audit Kit does not remove the need for final engineering validation. It does not guarantee that every discovered bug will be fixed automatically and perfectly, and it does not free the team from making architectural decisions.

The approach also has operational limitations:

  • quality depends on the model's ability to read and hold real code context;
  • large steps can run into execution time and budget limits;
  • quality gates must be understood and reproducible in advance;
  • security- and auth-related fixes still require careful human review.

In other words, the project does not replace a senior engineer, but it significantly increases their throughput.

8. Conclusion

The main advantage of AI Audit Kit is that it moves AI auditing out of improvisation mode and into a standardized engineering process.

This matters especially where the codebase is large, defects are many, and fixes must be not just fast but also observable, verifiable, and portable across projects. In that sense, AI Audit Kit is useful not as "a set of files for working with an agent," but as an operational framework for systematic bug discovery and orchestrated remediation.

That is exactly why the project is valuable: it captures not only what the agent should do, but also how this process should be arranged so that the result can be repeated, verified, and scaled.

Project repository: ascorblack/ai-audit-kit

← Back to list