Dynamic Matrix Strategies: Generating Runtime Tests

Your test suite covers 6 browsers. You hard-coded them in a static matrix. Today the spec added a seventh. You edited the matrix, committed, and waited for CI. Tomorrow there will be an eighth.

A static strategy.matrix works until the test dimensions change — then it becomes a maintenance liability. Every new browser version, OS target, or SDK release requires a YAML edit. For SRE and QA automation teams managing suites that run against a changing configuration surface, editing YAML to add a test target is a feedback loop that slows releases.

The solution is a matrix that generates itself from data: a parameter list that callers control, a template that consumes it, and a parallelism cap that keeps costs from scaling unchecked.

This article covers:

• The difference between compile-time ${{ each }} job generation and runtime strategy.matrix.
• Patterns for generating a strategy.matrix object dynamically from a YAML parameter.
• Parallelism control using maxParallel on dynamic matrices.
• Result aggregation from matrix legs in downstream jobs.
• A complete self-scaling browser compatibility test pattern.

Two Mechanisms, Two Phases

strategy.matrix — A Runtime Mechanism

strategy.matrix is evaluated during the Initialize phase, after the agent is provisioned but before any steps run. The matrix entries spawn parallel job copies — called legs — each receiving a different set of variables. From Azure DevOps’s perspective, a job with a 3-entry matrix is one job definition that produces three runtime copies.

Each matrix leg is a named key under strategy.matrix: with a nested mapping of variables. All matrix variable names become pipeline variables accessible via $(varName) macro expansion in any step. The leg name appears as the job display name suffix in the pipeline run graph.

# Static 3-entry matrix — three identical job structures, different variable values
jobs:
- job: RunTests
  strategy:
    matrix:
      chrome_latest:          # Leg name — shown in run UI, required in output variable refs
        browserName: chrome
        driverVersion: '114'
      firefox_stable:
        browserName: firefox
        driverVersion: '115'
      safari_current:
        browserName: safari
        driverVersion: '16'
    maxParallel: 3            # All three run concurrently when agents are available
  pool:
    vmImage: ubuntu-latest
  steps:
  - script: |
      echo "Testing on $(browserName) v$(driverVersion)"
      npx playwright test --browser $(browserName)
    displayName: "Playwright tests: $(browserName) v$(driverVersion)"

${{ each }} — A Compile-Time Mechanism

${{ each }} generates discrete, separate job definitions at parse time. Each iteration produces a fully independent job with its own name:, pool:, and steps:. The generated jobs are visible in the Expanded YAML as separate entries.

# ${{ each }} generates 3 SEPARATE job definitions — not matrix runtime copies
parameters:
- name: testTargets
  type: object
  default:
  - browser: chrome
    version: '114'
  - browser: firefox
    version: '115'
  - browser: safari
    version: '16'

jobs:
- ${{ each target in parameters.testTargets }}:
  - job: RunTests_${{ target.browser }}   # Three distinct job names in Expanded YAML
    pool:
      vmImage: ubuntu-latest
    steps:
    - script: |
        echo "Testing ${{ target.browser }} v${{ target.version }}"
        npx playwright test --browser ${{ target.browser }}
      displayName: "Test ${{ target.browser }}"

The key distinction: strategy.matrix produces one YAML definition that Azure DevOps clones at runtime. ${{ each }} produces multiple YAML definitions at compile time.

Choosing the Right Mechanism

The most powerful pattern for data-driven test suites is the combination: use ${{ each }} to build the matrix object at compile time from a parameter array, and let strategy.matrix handle the runtime parallelism.

Static Matrix — The Baseline

Structure and Variable Access

Each matrix leg is a named key with a nested variable mapping. Variable names become runtime variables accessible via $(varName).

jobs:
- job: RunTests
  strategy:
    matrix:
      ubuntu_node18:
        os: ubuntu-latest
        nodeVersion: '18.x'
        testSuite: unit
      windows_node18:
        os: windows-latest
        nodeVersion: '18.x'
        testSuite: integration
    maxParallel: 2
  pool:
    vmImage: $(os)               # Matrix variable used in pool configuration
  steps:
  - task: NodeTool@0
    inputs:
      versionSpec: $(nodeVersion)
  - script: |
      echo "OS: $(os) | Node: $(nodeVersion) | Suite: $(testSuite)"
      npm run test:$(testSuite)
    displayName: "Run $(testSuite) tests"

maxParallel: 2 instructs Azure DevOps to run at most 2 legs concurrently. Without it, all legs start simultaneously, potentially exhausting the agent pool.

Matrix include and Limitations

matrix.include adds extra legs to a matrix without rewriting the full definition. However, include and exclude cannot be combined with dynamically generated matrices. If you inject the matrix dynamically, include and exclude are ignored. Build every leg directly into the dynamic object instead.

Generating a Matrix Dynamically

Building the Matrix Object at Compile Time

Placing a ${{ each }} expression directly inside strategy: matrix: generates the matrix content from a parameter array at compile time.

parameters:
- name: testTargets
  type: object
  default:
  - browser: chrome
    version: latest
  - browser: firefox
    version: stable

jobs:
- job: RunTests
  strategy:
    matrix:
      # Each iteration contributes one leg to the matrix object
      ${{ each target in parameters.testTargets }}:
        ${{ target.browser }}_${{ target.version }}:
          browser: ${{ target.browser }}
          version: ${{ target.version }}
    maxParallel: 4
  pool:
    vmImage: ubuntu-latest
  steps:
  - script: npx playwright test --browser $(browser)
    displayName: "Playwright: $(browser) $(version)"

Adding a third entry to parameters.testTargets generates a third leg. No template changes are needed.

Leg Name Generation

Matrix leg names must be unique. Azure DevOps leg names allow only alphanumeric characters and underscores ([A-Za-z0-9_]). Dots and hyphens break output variable reference paths because the accessor uses dots as separators.

Sanitize names using a separate id property on each parameter entry that the caller ensures is underscore-safe:

parameters:
- name: testTargets
  type: object
  default:
  - id: chrome_114
    label: "Chrome 114"
    browser: chrome
    version: '114'

jobs:
- job: RunTests
  strategy:
    matrix:
      ${{ each target in parameters.testTargets }}:
        ${{ target.id }}:                  # Caller-controlled, guaranteed safe
          browser: ${{ target.browser }}
          version: ${{ target.version }}

Filtering the Matrix

Add an enabled boolean property to each entry. The loop filters disabled entries at compile time with an explicit equality check eq(..., true) to prevent string truthiness issues.

strategy:
  matrix:
    ${{ each target in parameters.testTargets }}:
      ${{ if eq(target.enabled, true) }}:
        ${{ target.id }}:
          browser: ${{ target.browser }}
          version: ${{ target.version }}

Controlling Parallelism

maxParallel on a Dynamic Matrix

maxParallel limits concurrent agents. On a dynamically generated matrix, declare it statically alongside the matrix: key. Note that maxParallel is a best-effort limit — actual concurrency depends on available agent slots in the pool.

parameters:
- name: parallelism
  type: number
  default: 4

jobs:
- job: RunTests
  strategy:
    matrix:
      ${{ each target in parameters.testTargets }}:
        ${{ if eq(target.enabled, true) }}:
          ${{ target.id }}:
            browser: ${{ target.browser }}
    maxParallel: ${{ parameters.parallelism }}

Total agent-minutes are constant — parallelism trades infrastructure spend rate against wall-clock time. For Microsoft-hosted agents, each leg carries ~30-60 seconds of provisioning overhead. Batch small tests into fewer legs to amortize this cost.

Consuming Matrix Results

Aggregating Status

A downstream job declaring dependsOn: RunTests waits for all legs. Use condition: always() for aggregation jobs to ensure they run even if some legs fail.

- job: TestAggregation
  dependsOn: RunTests
  condition: always()
  steps:
  - download: current
    patterns: 'test-results-*/**'
    displayName: "Download all leg artifacts"

Unique Artifact Naming

Artifact names must be unique per leg to avoid overwriting. Use $(System.JobName), which resolves to the matrix leg key (e.g., chrome_latest) and is safe for file paths. Do not use $(Agent.JobName), as it includes spaces and parentheses (e.g., RunTests (chrome_latest)).

  - publish: $(Build.SourcesDirectory)/test-results
    artifact: test-results-$(System.JobName)
    condition: always()

The Two-Phase Matrix Pattern

Use this advanced pattern only when test targets are discovered at runtime (e.g., “all microservices currently in staging”).

1. Discovery Job: Queries an API and outputs a JSON object (keys = leg names, values = variable objects).
2. Matrix Job: Injects the JSON via $[ dependencies.Discovery.outputs['findTargets.matrixJson'] ].

Diagram: The Two-Phase Dynamic Matrix Pattern

This diagram visualizes how runtime discovery enables a pipeline to scale its parallelism based on external data (e.g., current staging microservices or available browser versions) without any YAML changes.

Visual Notes:

• Job 1: The script must produce a valid JSON object (not an array).
• Job 2: The strategy.matrix is evaluated during the Initialize phase of the job. It spawns parallel copies (“legs”) based on the JSON keys.
• Job 3: Aggregation uses System.JobName to ensure artifacts from parallel legs don’t overwrite each other.

# Phase 1: Discovery (PowerShell)
$matrix = @{}
foreach ($pkg in $packages) {
  $legName = "v$($pkg.version -replace '[.\-]', '_')"
  $matrix[$legName] = @{ serviceVersion = $pkg.version }
}

if ($matrix.Count -eq 0) {
  Write-Host "##vso[task.logissue type=error]Zero targets found"
  exit 1
}

$matrixJson = $matrix | ConvertTo-Json -Compress
Write-Host "##vso[task.setvariable variable=matrixJson;isOutput=true]$matrixJson"

Critical Constraint: The output must be a JSON object {...}, not an array [...]. Ensure the Project Collection Build Service account has “Read” access to the artifact feed if querying Azure Artifacts.

Hands-On: Self-Scaling Browser Suite

parameters:
- name: testTargets
  type: object
  default:
  - { id: chromium_desktop, browser: chromium, viewport: desktop, enabled: true }
  - { id: firefox_desktop, browser: firefox, viewport: desktop, enabled: true }
- name: parallelism
  type: number
  default: 6

jobs:
- job: RunTests
  strategy:
    matrix:
      ${{ each target in parameters.testTargets }}:
        ${{ if eq(target.enabled, true) }}:
          ${{ target.id }}:
            browser: ${{ target.browser }}
            viewport: ${{ target.viewport }}
    maxParallel: ${{ parameters.parallelism }}
  steps:
  - script: npx playwright test --browser $(browser) --project $(viewport)
    continueOnError: true     # Publish results even if tests fail
    env:
      BROWSER: $(browser)
      VIEWPORT: $(viewport)
  - publish: $(Build.SourcesDirectory)/playwright-report
    artifact: results-$(System.JobName)
    condition: always()

- job: TestSummary
  dependsOn: RunTests
  condition: always()
  steps:
  - download: current
    patterns: 'results-*/**'

Best Practices

• Separation: Define the matrix in a parameter, not hardcoded in the template. Callers add targets without template PRs.
• Sanitization: Enforce machine-safe id properties for leg names.
• Unique Artifacts: Always use $(System.JobName).
• Fail Loudly: In the Discovery Job, explicitly fail if zero targets are found to avoid “empty” success.

Sources

• Microsoft Learn: strategy.matrix schema reference
• Microsoft Learn: Pipeline limits and constraints
• GitHub: Azure Pipelines YAML — matrix strategy design
• Microsoft Learn: PublishTestResults task reference