You need to test your application against 4 different browsers, 3 different operating systems, and 2 different database versions. That’s 24 unique job configurations. You could manually write 24 jobs—but what happens when you add a 5th browser? You are back to editing YAML for hours. There is a better way: The Dynamic Matrix.
The standard Azure DevOps strategy: matrix is a powerful tool for runtime parallelism, but it is typically static. You define the combinations in the YAML, and the engine runs them. However, in an enterprise environment, your test requirements often come from an external data source or a complex parameter object. The “Dynamic Matrix” pattern combines the compile-time expansion of ${{ each }} with the runtime parallelism of matrix, allowing you to generate hundreds of test jobs from a single, simple parameter list. This guide teaches you how to build a self-scaling test engine that “pivots” your data into a parallel execution plan.
1. The Static Matrix vs. the Dynamic Matrix
To appreciate the dynamic approach, we must first look at the limitation of the standard model.
1.1: Why Static is Not Enough
A static matrix requires you to hardcode every configuration key and value. This leads to “YAML Bloat” and is incredibly difficult to maintain when combinations change frequently. Furthermore, there is no way to conditionally add a matrix dimension—like a “Legacy Browser” suite—based on a pull request’s metadata without duplicating the entire job definition.
1.2: The Core Concept: ${{ each }} + matrix
The “Dynamic Matrix” works because of the pipeline evaluation order. The ${{ each }} loop expands at compile-time, creating the matrix object that the runtime engine then uses to spawn parallel agents. By looping over a parameter to generate the matrix keys, you gain the ability to scale your test suite by simply updating a JSON or YAML list, rather than modifying the pipeline’s logic.
2. Implementing the “Matrix Pivot” Pattern
The “Matrix Pivot” is the architectural move of turning a list of items into the top-level keys of a strategy: matrix block.
The Matrix Pivot (Parameter -> Runtime Jobs)
2.1: Transforming a Parameter into a Matrix
Consider a parameter named browsers. You can transform it into a matrix like this:
strategy:
matrix:
${{ each browser in parameters.browsers }}:
${{ browser }}:
browserName: ${{ browser }}
During expansion, if browsers is ['Chrome', 'Edge'], the engine generates two entries: Chrome and Edge. Each instance of the job will then have a local variable $(browserName) available to its tasks.
2.2: Multi-Dimensional Dynamic Matrices
For complex scenarios like “Browser x OS,” you can nest loops within the matrix block.
Multi-Dimensional Matrix (Cartesian Product)
matrix:
${{ each os in parameters.osList }}:
${{ each browser in parameters.browsers }}:
${{ format('{0}_{1}', os, browser) }}: # Ensure unique, valid keys
osName: ${{ os }}
browserName: ${{ browser }}
Pro Tip: Use the format() function to ensure matrix keys are unique and contain only valid characters (A-Z, 0-9, and underscores). Azure DevOps strictly prohibits hyphens and spaces in matrix identifiers.
3. Managing Parallelism at Scale
Generating 100 jobs is easy; running them without crashing your infrastructure is the challenge.
3.1: The maxParallel Property
If your tests hit a shared resource, such as a staging database or a limited Selenium Grid, you must throttle execution. Use the maxParallel property to limit concurrency. Even if your matrix generates 50 jobs, setting maxParallel: 5 ensures that only 5 run at any given time, preventing resource exhaustion.
3.2: Sizing Your Agent Pools
Remember the formula: Actual Concurrency = min(maxParallel, Available Parallel Jobs). If your organization only has 10 parallel job licenses, a matrix of 100 legs will queue 90 of them. Monitor your “Job acquisition” time to detect if your dynamic matrices are being “starved” of agents.
4. Result Aggregation and Dependencies
Matrix jobs follow a special naming convention for their outputs: JobName_MatrixKey.
4.1: Collecting Matrix Outputs
To wait for an entire matrix to finish, use dependsOn: JobName (without the matrix key) in your subsequent job. This creates a “Fan-In” pattern. However, if you need to read a specific variable from every leg of the matrix, you must use a “Collector Job” that iterates through the dependencies JSON object.
Matrix Dependencies (Fan-Out / Fan-In)
4.2: Conditional Logic Based on Matrix Success
Use succeeded() and failed() at the matrix level. A common pattern is running a “Cleanup” job with condition: failed() to tear down test environments only if at least one leg of the matrix encountered an error.
5. Advanced Optimization: The “Slice” Strategy
For hyper-scale testing (200+ legs), the “Initialization Lag” becomes significant. To reduce the parsing overhead, consider the “Slice” strategy: use ${{ each }} to generate three separate jobs, each containing a smaller, 50-leg matrix. This parallelizes the orchestrator’s work as well as the agent’s work, resulting in faster start times.
Hands-On Example: The “Universal” Test Suite
To implement a self-scaling test engine, create a template that accepts an object representing your test targets. By combining dynamic matrix generation with the convertToJson() function, you can transform a simple list of microservices into a global, parallel validation suite that automatically adjusts its scope based on which services were modified in the current PR.
Key Takeaways
- Pivot Your Data: Use
${{ each }}to turn parameters into matrix keys for automated scaling. - Naming Rules: Matrix keys must start with a letter and contain only underscores/digits. No hyphens or spaces.
- Throttle Wisely: Use
maxParallelto protect shared resources and manage agent pool consumption. - Depend on the Whole: Use the base
JobNameindependsOnto wait for the entire parallel suite.