Engineers at inDrive found that a pull request involving 206 modules could be completed in 44 minutes when executed in a single parallel dependency layer. This milestone highlights the sheer scale of the infrastructure managed by the global mobility platform, which encompasses a total of 14,551 distinct Terragrunt modules. For a significant period, the organization managed its extensive Infrastructure as Code repository using a manual serial process that relied on a static configuration file. This system required every update to follow a rigid, pre-defined path, creating substantial delays as the number of modules grew. The manual orchestration was governed by a complex set of patterns that determined the order of execution for every network setup, database configuration, and cluster deployment. As the cloud footprint expanded across multiple regions, this linear approach became an unsustainable barrier to rapid innovation, leading the engineering team to seek a more automated alternative for handling deployments.
The Evolution: From Static Orders to Dynamic Flow
The legacy infrastructure management system at inDrive centered around a 113-line text file known as order.txt, which utilized glob patterns to establish a hierarchy for resource deployment. This file dictated that core modules and virtual private clouds were processed first, followed by transit gateways and service-specific resources like identity management roles or messaging topics. While this manual orchestration provided a semblance of control during the earlier phases of growth, it ultimately proved to be both brittle and high-maintenance. Every time a new category of module was introduced, an engineer had to manually update the sorting file to ensure the new resource would be deployed in the correct sequence. Furthermore, inconsistent naming conventions often led to pattern-matching errors, where some modules would inadvertently bypass the intended order or fail to be included altogether. This manual intervention introduced a significant risk of human error in production.
Beyond the maintenance burden, the serial execution model imposed by the static ordering file created artificial bottlenecks that slowed down the entire development lifecycle. In this linear world, every module in a rollout had to wait for the preceding one to complete its execution, regardless of whether a functional relationship existed between them. For instance, a minor update to a monitoring configuration in a secondary region would be forced to wait for a complex database migration in the primary region to finish. This lack of concurrency meant that large pull requests involving dozens of unrelated resources would take hours to process, leaving engineers in a state of constant waiting. The team identified that the primary inefficiency was the failure to recognize which components could safely run in parallel. Moving away from this rigid structure required a complete paradigm shift toward a system that could intelligently interpret the underlying code to determine dependencies.
Graphing Connectivity: Automating Dependency Awareness
To transition toward a more efficient deployment model, the engineering team turned to the inherent capabilities of Terragrunt’s native dependency blocks. These blocks provide an explicit declaration of relationships between different infrastructure modules, allowing the system to understand which resources must exist before others can be initialized. By extracting these declarations from the codebase, the team was able to construct a comprehensive directed acyclic graph that represented the entire infrastructure hierarchy. They applied Kahn’s topological sorting algorithm to this graph, which allowed for the programmatic determination of the optimal execution order for any given pull request. This automated approach replaced the fallible manual lists with a rigorous, code-driven logic that could adapt to changes in real-time. Instead of guessing the correct sequence, the pipeline could now mathematically prove the safest and fastest path for resource deployment across the cloud environment.
Implementing a reliable dependency graph required more than just running an algorithm; it necessitated a massive cleanup of the existing codebase to expose hidden relationships. During the initial audit, the team discovered numerous instances where modules were reading outputs from other resources without an explicit dependency declaration, a practice that would cause parallel runs to fail. To address this, a widespread effort was undertaken to formalize all inter-module connections and ensure that every reference was correctly mapped within the configuration files. To protect the integrity of the new system, the team integrated Open Policy Agent checks directly into the continuous integration pipeline. These automated safeguards are designed to scan every pull request for circular dependencies, broken paths, or missing declarations. If a violation is detected, the pipeline automatically rejects the change, ensuring that the dependency graph remains a trustworthy foundation for parallel execution.
Orchestrating Parallelism: Matrix Deployment Strategies
With a validated graph as the engine, the deployment workflow was restructured into distinct dependency layers to maximize concurrent execution within GitHub Actions. In this innovative model, modules are categorized based on their position in the hierarchy, with Layer 0 containing all resources that have no prerequisites within the current pull request. Subsequent layers are populated with modules that depend only on those found in the preceding tiers. The orchestration logic utilizes the platform’s matrix feature to dynamically generate multiple jobs that run simultaneously for every module within a single layer. This means that if a developer submits a change involving twenty independent resources, the pipeline can trigger twenty parallel runners to handle the work at once. The system is designed to wait for the entire layer to reach a successful completion state before the next set of dependent modules is allowed to begin, maintaining a strict but efficient order of operations.
While the goal was to maximize speed, the team had to account for the physical and logical constraints of the cloud environment and the CI platform. Launching hundreds of parallel Terraform processes simultaneously can lead to aggressive throttling from AWS API endpoints, potentially causing deployments to fail due to rate-limiting policies. To mitigate this risk, the team implemented a strategic concurrency cap using the strategy.max-parallel attribute in GitHub Actions. This allows the organization to control the number of active runners at any given moment, ensuring that the infrastructure provider is not overwhelmed by sudden bursts of activity. Additionally, this cap helps manage the pool of available GitHub runners, preventing a single massive infrastructure update from exhausting the organization’s entire compute quota. This balanced approach ensures that the pipeline remains fast and responsive while adhering to the operational limits of the underlying services and management tools.
Analyzing Results: Performance Metrics and Tradeoffs
The transition to parallel execution yielded remarkable results, which the team quantified over a period of several months involving hundreds of unique rollouts. By comparing the actual duration of parallel applies against a baseline of estimated serial execution times, the organization recorded an aggregate 2.7× increase in infrastructure delivery speed. The data showed that for large-scale operations, the time savings were transformative for engineering productivity. In one instance, a rollout involving 64 modules was completed in just 31 minutes, whereas the traditional serial method would have required nearly five hours of processing time. This drastic reduction in the feedback loop allows developers to iterate on infrastructure changes with the same agility typically reserved for software application code. The cumulative effect of these improvements saved over a hundred hours of total elapsed time, demonstrating the high return on investment for the parallelization effort.
Despite the clear advantages for massive changes, the performance data also revealed that parallelism is not a universal solution for every pull request size. For smaller updates involving fewer than four modules, the overhead associated with dependency discovery, policy validation, and the startup time for multiple GitHub Action runners often exceeded the duration of the apply steps themselves. In these cases, the serial execution path remained more efficient because it avoided the complexity of orchestrating multiple jobs. Recognizing this reality, the engineering team decided to maintain both execution paths, making the parallel apply an opt-in feature rather than a mandatory default. This flexible strategy allows engineers to choose the most efficient method based on the scope of their work. Guidance was issued to the team suggesting that parallelization should be reserved for larger architectural changes where the concurrency benefits far outweigh the initial orchestration overhead.
Future Directions: Actionable Insights for Scale
The journey toward optimizing Terragrunt execution provided critical insights into the necessity of maintaining a clean and well-audited codebase. The organization discovered that the success of any parallelization effort was directly proportional to the accuracy of the underlying dependency graph. Without the rigorous audit of hidden references and the implementation of automated validation through policy-as-code, the move to parallel execution would have introduced unacceptable risks to production stability. Furthermore, the decision to keep the orchestration logic within existing CI tools proved beneficial, as it allowed the team to leverage established approval workflows and reporting mechanisms. Organizations looking to replicate these results were encouraged to prioritize the formalization of module relationships before attempting to scale execution. The experience demonstrated that automation must be built on a foundation of architectural clarity to ensure that speed does not come at the cost of reliability.
Looking forward, the focus shifted toward further minimizing the barrier effect inherent in layered execution models. While the current system significantly improved delivery times, the team identified that a more fluid, non-layered approach could potentially unlock even greater efficiencies. This would involve moving toward a fully asynchronous execution model where each module starts as soon as its specific prerequisites are met, rather than waiting for an entire layer to finish. Additionally, the implementation of more granular API rate-limiting awareness was suggested to allow for even higher levels of concurrency during off-peak hours. The organization ultimately proved that the bottleneck in large-scale infrastructure management was not the complexity of the resources themselves, but the rigidity of the tools used to deploy them. By embracing code-driven orchestration and data-backed decision-making, the team established a scalable framework that successfully supported the global demands of a modern mobility platform.
