Better CI Triage Requires Real Evidence Before AI Classifiers

Better CI Triage Requires Real Evidence Before AI Classifiers

Clearing a wall of red builds before a morning standup often turns into a repetitive cycle of rerunning tests until they pass without ever identifying the root cause of the initial failure. This standard operating procedure relies on the assumption that most failures are transient, often caused by minor infrastructure hiccups or “flaky” tests that eventually resolve themselves upon repetition. When a nightly build produces dozens of errors, the immediate pressure to unblock the development team leads to a culture of rerunning rather than investigating. However, this approach masks the underlying instability of the testing environment and the application itself. If thirty tests fail but twenty-five pass on the second attempt, those twenty-five are typically ignored, while only the remaining five receive any engineering attention. This creates a significant gap in diagnostic intelligence where recurring issues are never properly addressed because they are not persistent enough to warrant a ticket. The emergence of decision-based AI models like Jev offers a potential escape from this manual triage fatigue, but these classifiers are only effective when provided with the raw evidence required to make a logical choice. Without structured data, automated triage becomes a faster method for mislabeling critical product defects as harmless noise.

1. Categorizing Failures for Specific Owners

The modern continuous integration pipeline frequently groups all failures under a single “failed” status, which serves as a significant hurdle for efficient resolution. To move beyond this bottleneck, organizations must differentiate failures into specific categories that point toward a clear path for remediation. Sai Krishna, Director of Engineering at TestMu AI, emphasizes the necessity of breaking these down into three primary buckets: product defects, environmental noise, and agent errors. Product defects represent genuine regressions in the software that can be consistently reproduced on the same code commit. Environmental noise, by contrast, involves external factors such as network latency, backend API outages, or intermittent infrastructure failures that do not reflect the health of the application code itself. The third category, agent errors, has become increasingly prominent in 2026 as more pipelines integrate agentic workflows. These occur when the reasoning or actions of an AI agent become inconsistent despite a healthy environment and stable code. Distinguishing between these three categories is the first step toward building a triage system that does more than just report failure; it identifies responsibility.

Assigning an owner to each of these buckets ensures that failures are routed to the teams best equipped to fix them. When a failure is labeled a product defect, it is immediately directed to the feature team responsible for the relevant code module. If the cause is environmental noise, the platform or infrastructure team takes charge of stabilizing the network or the specific service that went down. Agent errors require a different kind of expertise, often involving the engineers who manage the prompts, the model versions, or the underlying logic of the autonomous agents. By defining these owners early, teams can avoid the common trap of passing failures between departments like a hot potato. This structured approach requires the pipeline to capture specific signals—such as trace logs, dependency health checks, and retry results—to support each label. Without these signals, a classifier cannot accurately decide which team should receive the notification. Therefore, the implementation of a triage model must be preceded by a clear strategy for data collection and ownership that reflects the unique architecture of the organization’s delivery pipeline.

2. Identifying Hidden Test Defects in Parallel Systems

Scaling a test suite from local development machines to large parallel grids often introduces a fourth category of failure that is frequently misidentified as environmental noise. Srinivasan Sekar, Director of Engineering at TestMu AI, points out that shared state across parallel workers is a primary source of unanticipated test instability. In a local environment where tests run sequentially or with minimal concurrency, global fixtures and singletons rarely cause issues. However, when 200 workers operate simultaneously in a cloud environment, tests may compete for the same database records or global state, leading to race conditions and order-dependent failures. These are not product defects or infrastructure outages; they are test defects. Because they appear intermittently and often resolve themselves during a single-test rerun, many teams mistakenly bucket them with environmental noise. This prevents the team from addressing the actual problem, which usually involves isolated test data setup or idempotent teardown procedures. Identifying these defects requires a specific diagnostic step that separates the impact of the grid from the health of the application.

A practical method for isolating test defects involves rerunning any failed test in a single-worker mode as part of the triage process. If a test fails in the parallel grid but passes consistently when running in isolation, it provides a strong signal that the failure is related to shared state or resource contention. This simple procedural change turns an ambiguous “flaky” result into a clear directive for the test authors to refactor their data management logic. Addressing test defects at the source—rather than treating them as transient noise—improves the overall reliability of the CI pipeline and reduces the total cost of execution by preventing unnecessary retries. It also ensures that when an AI classifier is eventually introduced, it has a distinct label for issues that reside within the test code itself. By providing the model with the outcome of both parallel and isolated runs, engineers can train the system to recognize the specific patterns associated with concurrency issues, further streamlining the path from failure detection to resolution.

3. Capturing Decision Traces for Agentic Pipelines

The rise of agent-based automation has made traditional logging insufficient for diagnosing why a pipeline failed. When an autonomous agent makes a decision that leads to an error, the evidence is often hidden within the agent’s internal reasoning process rather than standard system logs. Tom Howe, Director of Insight Engineering, suggests that capturing a cryptographically secure trace of every action an agent takes is essential for modern triage. These traces provide a step-by-step account of the agent’s logic, the tools it utilized, and the context it considered before arriving at a conclusion. By comparing these traces against previously successful runs, engineers can identify exactly where the agent’s logic diverged or where its decision-making became brittle. This level of insight is critical because an agent error might not look like a traditional crash; it could manifest as the agent simply choosing the wrong sequence of steps to achieve a goal. Without a detailed trace, determining whether the agent failed due to a poor prompt, an updated model version, or a subtle change in the UI becomes guesswork.

To facilitate automated triage, this diagnostic data should be packaged into a structured evidence bundle for every failure. This bundle acts as a comprehensive snapshot of the test’s execution state, including the specific code commit, the versions of the models and prompts being used, the health of external dependencies at the time of execution, and the retry outcomes. In 2026, many teams are including a diff of the agent’s steps against the last known “passing” trace directly in this bundle. For example, if an agent skipped a critical login step that it had previously performed correctly, the bundle highlights this discrepancy. This structured data allows both human engineers and AI classifiers to make informed judgments about the nature of the failure. Instead of asking a model to guess why a test failed based on a generic error message, the evidence bundle provides the specific facts needed to verify the failure mode. This shift toward high-fidelity data collection is what enables advanced triage agents to perform at or above human accuracy levels in complex CI environments.

4. Implementing Automated Triage Through Calibration

Once the foundation of structured evidence is in place, organizations can begin the process of automating the routing of failures to their respective owners. This transition must be managed carefully to ensure that the classifier’s decisions are trustworthy and grounded in reality. The implementation begins with a manual labeling phase, where engineers tag failures from the previous month using the established categories: product defect, environmental noise, agent error, and test defect. During this period, the triage job is configured to automatically attach the relevant evidence bundles to each record. This creates a high-quality dataset that reflects the unique failure modes of the team’s specific stack. Following this, a decision model should be deployed in “shadow mode” for several weeks. In this state, the model analyzes every failure and generates a prediction and a confidence score, but it does not actually route any tickets or notify any teams. This allows engineering leadership to compare the model’s performance against the manual labels provided by the staff, identifying areas where the classifier might be struggling.

The final requirement for enabling automatic routing is a rigorous calibration check to ensure the model’s confidence scores are accurate. A well-calibrated model that reports 90 percent confidence should be correct approximately 90 percent of the time. If a calibration report reveals that the model is overconfident—for instance, achieving only 70 percent accuracy when it claims 90 percent confidence—the team must either refine the evidence bundle or adjust the automated routing threshold. Shahram Anver, Co-Founder and CEO of Cleric, points out that as evidence grows richer, the number of failure labels can expand to provide even more granular routing. For example, agent errors could be split further into tool access problems and reasoning errors. By only allowing the classifier to route failures when its confidence exceeds a high, verified threshold, the team can automate the bulk of their triage work without the risk of flooding departments with incorrect notifications. Lower-confidence results are sent to a human triage queue, ensuring that the system remains both efficient and reliable while it continues to learn from new data.

5. Advancing Pipeline Resilience with Evidence-Based Logic

Determining if a failure originates from an agent error requires a specific understanding of how non-deterministic systems behave in a controlled environment. An agent error is typically confirmed when the application code and environment remain static, yet the agent’s trajectory or final output changes between runs. This inconsistency often points to limitations in the prompt engineering or the underlying model’s ability to handle specific edge cases within the UI. Parallel runs frequently complicate this diagnosis by exposing race conditions that are not present in sequential execution. When multiple tests attempt to modify the same global fixtures or shared data records, they create a state of instability that mirrors environmental noise but is actually a flaw in the test’s design. By rerunning these failures in isolation, teams can distinguish between a flaky environment and a lack of isolated test data. This level of granularity in the evidence bundle—including commit IDs, prompt versions, and API health—allows for a robust classification process that reduces the “flaky” tag to a relic of the past.

The transition to an evidence-first triage system transformed the way engineering departments handled their delivery cycles. By prioritizing the collection of detailed traces and environmental snapshots, organizations successfully reduced the time spent on manual investigation by more than sixty percent. The implementation of calibrated decision models allowed for the automatic routing of high-confidence failures, which ensured that feature teams were only alerted when genuine product regressions occurred. Meanwhile, the platform teams utilized the environmental noise data to identify and stabilize brittle infrastructure components that had previously gone unnoticed. This structured approach did not just solve the problem of red builds; it created a transparent feedback loop that improved the quality of both the software and the automated tests themselves. Ultimately, the teams discovered that the key to effective AI-driven triage was not the complexity of the model, but the quality and accessibility of the data it was asked to analyze. By building this foundation, they turned their CI pipelines into a source of actionable intelligence rather than a wall of frustrating red signals.

Subscribe to our weekly news digest.

Join now and become a part of our fast-growing community.

Invalid Email Address
Thanks for Subscribing!
We'll be sending you our best soon!
Something went wrong, please try again later