The traditional pull request (PR) review process, long considered the gold standard for quality assurance, has reached a breaking point as agentic workflows become the standard for software development. Engineering managers who once prided themselves on reading every line of code now find that the sheer volume of output from autonomous fleets exceeds the cognitive limits of even the most dedicated human teams. This shift represents a transition from a model where humans are the primary actors to one where they act as high-level governors of a highly automated “Software Factory.”
In this new reality, the role of the technical leader is no longer defined by manual inspection but by the ability to design and maintain the automated gates that govern the delivery pipeline. As the queue of changes lengthens, the manager often becomes the only station on the line that cannot scale, creating a persistent bottleneck that threatens the agility of the entire organization. Adopting a strategy of supervising by exception is not just a tactical adjustment; it is a survival mechanism for the modern engineering manager who must balance rapid delivery with the rigorous requirements of safety and security.
The Transition from Manual Review to Gate-Based Supervision
Supervising by exception requires a fundamental pivot in the mental model of engineering management, moving away from the exhaustive review of every pull request toward the strategic management of automated gates and risky outputs. Instead of acting as a manual checkpoint for every individual change, the manager now focuses on the configurations that route high-risk agent output to human judgment. This approach acknowledges that while agents can implement code at an unprecedented scale, the final authority for high-impact decisions must remain with the human supervisor who understands the broader business context.
The central challenge of this transition is the “human bottleneck” that emerges when the speed of agentic output far outpaces the capacity for human processing. When an engineering manager attempts to read every diff generated by a fleet of agents, the approval chain inevitably stalls, and the quality of the reviews begins to suffer as fatigue sets in. By Monday morning, a queue that was cleared on Friday often returns twice as long, leading to a cycle of burnout and decreased vigilance. To break this cycle, the manager must own the gates themselves, ensuring that only the most critical deviations reach their desk.
Moreover, this shift forces a redefinition of what it means to be a “gate owner” in an agentic SDLC. The engineering manager must write down, in advance, which signals will trigger a human intervention and what thresholds will define a “routine” change versus an “exceptional” one. This allows the vast majority of agent-authored code to flow through the system without manual reading, provided it clears the automated safety and style checks. The manager’s labor is then concentrated on sampling the routine output and performing deep-dive investigations into the specific records of agent behavior.
Context and Importance of Agentic Governance
The background of the “Software Factory” model highlights a critical evolution from “Human in the Loop” configurations to “Human on the Loop” oversight. In the previous era, human developers were the primary authors of every line of code, with AI tools serving only as assistants. Today, the configuration has flipped: agents handle the bulk of the implementation, while humans sit at the control console, monitoring the flow and intervening only when necessary. This structural change is vital for preventing reviewer fatigue and maintaining long-term code quality as the per-capita throughput of pull requests continues to rise.
This research is particularly critical for Engineering Managers and CTOs who are tasked with scaling their departments without sacrificing the integrity of their codebases. As per-capita PR throughput has doubled in many organizations, the pressure on human reviewers has reached a point where the traditional “read-everything” approach is no longer viable. Without a structured governance framework, organizations risk a silent erosion of quality as reviewers begin to skim or “rubber stamp” changes just to keep the pipeline moving. Establishing clear agentic governance ensures that accountability remains a priority even as the speed of production increases.
Furthermore, the implementation of these governance structures involves a move toward authorship delegation, where specific humans own the specification and verification of a task while delegating the implementation to an agent. This requires a clear division of roles, such as specification owners, verification owners, and service owners, each with the authority to halt the workflow if the agents deviate from the intended path. By formalizing these roles, the engineering manager can budget supervision bandwidth as a finite resource, ensuring that the organization’s human capital is directed toward the areas of greatest risk and complexity.
Research Methodology, Findings, and Implications
Methodology
The qualitative and quantitative approaches used in this research were anchored by a 2026 practitioner interview study conducted across 12 diverse organizations. This study involved 20 in-depth interviews with engineering leaders who are actively managing agentic fleets, providing a detailed look at how daily workflows have changed since the mass adoption of autonomous coding tools. To supplement these qualitative insights, the research included a broad survey of the current state of the industry, capturing the sentiment of hundreds of practitioners.
In addition to the interview data, the study analyzed a massive enterprise PR mandate dataset involving 802 developers and nearly 200,000 pull requests. This dataset provided the statistical weight needed to identify trends in human review coverage and the impact of increased AI output on traditional oversight mechanisms. The analysis focused on how the ratio of human reviews to total changes shifted as agent involvement increased, revealing a significant drop in manual oversight when no new supervisory strategy was implemented.
To establish performance thresholds and evaluate the effectiveness of human oversight, the research utilized the Cosmos platform’s session records and followed the NIST AI 600-1 guidelines. These guidelines provide a framework for setting minimum performance thresholds as part of a go/no-go deployment approval process. By reviewing the retained and reopenable records of agent actions—known as Cosmos Sessions—the researchers were able to compare the agent’s internal steps against the final diff, offering a more comprehensive view of “risk” than code inspection alone could provide.
Findings
The study discovered that human review coverage drops significantly, falling from 89% to 68%, when AI output increases without a corresponding change in supervisory strategy. This indicates that human review naturally becomes a sampling process by default rather than by design when volume exceeds capacity. This “unintentional sampling” is dangerous because it lacks the systematic rigor needed to catch high-risk errors. When reviewers are overwhelmed, they are less likely to spot subtle logic flaws or security vulnerabilities, leading to a false sense of security.
Another key finding was the “inverted U” relationship between the escalation load and overall system safety. The data showed that over-escalation—sending too many minor issues to human reviewers—actually leads to a decrease in system security. This occurs because the human supervisors become desensitized to alerts, treating them as noise rather than signals. Conversely, a system that escalates too little fails to catch genuine threats. The research suggests that the highest level of safety is achieved when the volume of escalations is carefully tuned to match the cognitive bandwidth of the human on the loop.
Finally, the research identified the emergence of “supervisory engineering work” as a distinct professional category. This new role focuses on the direction and evaluation of AI output rather than manual code implementation. This work involves refining specifications, setting task boundaries, and evaluating the agent’s reasoning process. The findings suggest that as implementation becomes cheaper and more automated, the value of engineering talent will increasingly shift toward these supervisory tasks, requiring a new set of skills centered on oversight and verification.
Implications
The move toward authorship delegation represents a major shift in how engineering teams are structured and how work is assigned. In this model, the human engineer owns the specification—the “what” and the “why”—while the agent handles the “how.” This requires the engineering manager to manage the relationship between the spec and the output, ensuring that the agent remains within the boundaries defined by the human. This delegation allows teams to scale their output without needing a linear increase in human headcount, provided the verification mechanisms are robust.
Practical application of these findings involves the use of Risk Score thresholds and signal-class tuning to manage the flow of changes. By differentiating between human-authored diffs and agent-authored code, managers can apply stricter thresholds to the latter, ensuring that AI-generated logic is subjected to more rigorous automated scrutiny. For a flagged change, the manager can reopen the specific session record to see the agent’s reasoning, rather than just the final diff. This provides a deeper level of insight into whether the agent followed the intended path or arrived at the solution through a “hallucinated” logic.
Organizational design must also evolve, as EMs must now budget supervision bandwidth alongside traditional compute capacity in their planning cycles. If the volume of agent output exceeds the team’s ability to verify it, the team begins to accrue “verification debt,” which is as dangerous as technical debt. Managers must be empowered to tune their sampling rates and adjust their spans of control based on the quality of the signals they receive. If a sample reveals too many defects, the manager must have the authority to slow down the agents until the supervisory system is recalibrated.
Reflection and Future Directions
Reflection
The study highlighted the pervasive challenge of AI complacency, often referred to as the “rubber stamp” phenomenon. When agents consistently produce high-quality code, human reviewers can fall into a state of habituation, assuming that the next change will be correct without performing a thorough check. This complacency was driven primarily by a lack of personal accountability for the AI’s output. To combat this, the research suggests that managers must explicitly sign off on their personal samples of agent records, creating a paper trail that reinforces the responsibility of the human on the loop.
Another point of reflection was the subjective nature of “risk” in agentic output, as evidenced by a relatively low Fleiss’ kappa agreement of 0.52 among reviewers. This indicates that different engineers often disagree on what constitutes a “safe” or “risky” agent action. This subjectivity makes it difficult to set universal thresholds and underscores the need for localized tuning within specific teams and codebases. The “Halt Drill” emerged as a necessary tool to ensure that teams are prepared to intervene effectively. A service owner who waits for permission before pulling the cord has not yet mastered the autonomy required for a safe agentic SDLC.
The necessity of the Halt Drill ensures that the human-led intervention remains a practiced muscle rather than a theoretical capability. In an era of high automation, the ability to stop the line is as important as the ability to keep it running. The research found that teams that regularly practiced their halt and restart procedures were more confident in their ability to manage agents and more likely to catch subtle sabotages or errors that bypassed automated checks. This proactive approach to failure management is essential for maintaining trust in a system that produces code faster than it can be fully understood.
Future Directions
Research into long-term “verification debt” will be essential as we move from 2026 to 2028 and beyond. We need to understand how agent-generated codebases evolve over time and whether they become more difficult for humans to maintain as the original human authorship of the logic recedes. If agents are writing the code and humans are only verifying the outputs, the long-term impacts on architectural integrity and “tribal knowledge” remain unknown. Future studies should focus on the maintainability of code that has been primarily authored and reviewed by non-human entities.
The evolution of career ladders and promotion packets also requires immediate investigation. Traditional coding skills, such as syntax proficiency and manual debugging, are becoming less relevant than “prompt quality” and the ability to define precise task boundaries. Organizations will need to develop new ways to evaluate junior and senior engineers based on their ability to direct and supervise agents rather than their ability to write code from scratch. This shift will likely change the way we interview, hire, and train the next generation of software professionals.
Finally, the scaling limits of manager spans must be investigated as the agent-to-human ratio continues to rise. There is a ceiling on how many agents and human supervisors one manager can effectively oversee before the quality of the “supervision by exception” begins to degrade. Identifying these limits will be critical for organizational planning, as it will determine the ideal size of a “software factory” squad. As the technology matures, the definition of a “manager” may shift entirely toward that of a systems auditor, focused on the health of the automation rather than the output of individuals.
Summary and Final Perspective on Supervisory Engineering
The shift toward owning the gates and the exception list represented a fundamental change in the identity of the modern engineering manager. This research demonstrated that supervising by exception was not merely a productivity hack but a requirement for maintaining safety and accountability in an automated SDLC. By focusing on the signals that indicated a deviation from the norm, managers successfully navigated a landscape where the volume of code was no longer a metric of success but a logistical challenge to be solved.
The successful technical leader managed the system that produced the code, rather than attempting to manage the code itself. This transition was supported by the implementation of rigorous sampling techniques and the formalization of roles such as specification and verification owners. The focus moved away from the mundane task of manual PR reviews toward the more complex work of threshold calibration and risk analysis. This approach ensured that human judgment was preserved for the cases where it mattered most, preventing the “rubber stamp” culture that often accompanies unmanaged automation.
Ultimately, the goal of supervisory engineering was to create a resilient and scalable software delivery process that could handle the velocity of AI without sacrificing the principles of sound engineering. Organizations that embraced these gate-based strategies found themselves better equipped to handle the complexities of agentic workflows. By treating the agent fleet as a force multiplier that required careful governance, engineering leaders were able to maintain high standards of quality and security, even as their output reached unprecedented levels. The focus remained on owning the process, the exceptions, and the ultimate accountability for the system’s performance.
