Test Reporting in CI: Formats, Tools, and Integration

An upload step without if: always() skips when tests fail, leaving you with no HTML report and no screenshots at the exact moment you need them. CI test reporting has three independent layers: artifact storage for post-run inspection, JUnit XML for dashboard parsing, and PR comments that surface failures where developers look. This article covers all three with configuration examples for GitHub Actions and GitLab, plus Slack notifications and programmatic result parsing.

The Problem with Raw Test Output

Default CI output for a failed Playwright run looks like this:

  ✘ tests/login.spec.ts > shows error for wrong password (15234ms)
    Error: Timed out 5000ms waiting for expect(locator).toBeVisible()

    Call log:
      - expect.toBeVisible with timeout 5000ms
      - waiting for locator('[data-testid="error-message"]')

You have to read through hundreds of lines of output to find the failures. In a suite of 200 tests, that's painful.

Report Formats

JUnit XML

The universal format — every CI system understands it.

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="Login Tests" tests="5" failures="1" time="12.5">
    <testcase name="shows error for wrong password" time="15.2">
      <failure message="Timed out waiting for error-message">
        Error: Timed out 5000ms waiting for expect(locator).toBeVisible()
        ...
      </failure>
    </testcase>
    <testcase name="successful login redirects to dashboard" time="3.2" />
  </testsuite>
</testsuites>

Generate from Playwright:

// playwright.config.ts
reporter: [
  ['junit', { outputFile: 'test-results/results.xml' }],
  ['list'],  // Also show output in console
],

HTML Report

Playwright's built-in HTML report — visual, with screenshots and traces.

reporter: [
  ['html', { outputFolder: 'playwright-report', open: 'never' }],
],

Generate and open locally:

npx playwright show-report

JSON

Machine-readable format for custom processing:

reporter: [
  ['json', { outputFile: 'test-results/results.json' }],
],

Useful for building custom dashboards or parsing results programmatically.

Multiple Reporters

Usually you want several formats:

reporter: process.env.CI
  ? [
      ['junit', { outputFile: 'test-results/results.xml' }],
      ['html', { outputFolder: 'playwright-report', open: 'never' }],
      ['list'],  // Live output in CI logs
    ]
  : [
      ['html'],  // Local: just the visual report
      ['list'],
    ],

GitHub Actions Integration

Basic setup with artifact upload

name: Playwright Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      
      - name: Run tests
        run: npx playwright test
        env:
          BASE_URL: ${{ secrets.STAGING_URL }}
      
      # Always upload report, even when tests fail
      - name: Upload HTML report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30
      
      - name: Upload JUnit results
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: test-results/results.xml

The if: always() is critical — without it, the upload step skips when tests fail.

JUnit test annotations in GitHub

GitHub Actions can parse JUnit XML and annotate your PR with failures directly in the diff:

- name: Report test results
  uses: dorny/test-reporter@v1
  if: always()
  with:
    name: Playwright Tests
    path: test-results/results.xml
    reporter: java-junit
    fail-on-error: true

This shows a summary in the GitHub PR interface and annotates failed tests in the code.

GitLab CI Integration

GitLab has native JUnit support:

test:
  image: mcr.microsoft.com/playwright:v1.44.0-jammy
  script:
    - npm ci
    - npx playwright test
  artifacts:
    when: always
    reports:
      junit: test-results/results.xml  # Native JUnit support
    paths:
      - playwright-report/            # Upload HTML report too
    expire_in: 1 week

GitLab reads the JUnit XML and shows a test summary in the pipeline UI — no extra tools needed.

Slack Notifications

Send failure summaries to Slack when CI tests fail:

- name: Notify Slack on failure
  if: failure()
  uses: slackapi/slack-github-action@v1
  with:
    payload: |
      {
        "text": "❌ Playwright tests failed on ${{ github.ref_name }}",
        "attachments": [{
          "color": "danger",
          "fields": [
            {
              "title": "Branch",
              "value": "${{ github.ref_name }}",
              "short": true
            },
            {
              "title": "Run",
              "value": "${{ github.run_id }}",
              "short": true
            },
            {
              "title": "Report",
              "value": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
            }
          ]
        }]
      }
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

Parsing Results Programmatically

Sometimes you need custom logic — only notify on new failures, not re-failures:

// scripts/parse-results.js
const fs = require('fs');
const xml2js = require('xml2js');

async function parseResults() {
  const xml = fs.readFileSync('test-results/results.xml', 'utf8');
  const result = await xml2js.parseStringPromise(xml);
  
  const testsuites = result.testsuites.testsuite;
  const failures = [];
  
  testsuites.forEach(suite => {
    (suite.testcase || []).forEach(testcase => {
      if (testcase.failure) {
        failures.push({
          name: testcase.$.name,
          suite: suite.$.name,
          message: testcase.failure[0].$.message,
          time: testcase.$.time,
        });
      }
    });
  });
  
  console.log(`Total failures: ${failures.length}`);
  failures.forEach(f => {
    console.log(`- [${f.suite}] ${f.name}`);
    console.log(`  ${f.message}`);
  });
  
  process.exit(failures.length > 0 ? 1 : 0);
}

parseResults();

Test Summary Comment on PRs

Automatically comment a test summary on every PR:

- name: Test summary comment
  uses: marocchino/sticky-pull-request-comment@v2
  if: always()
  with:
    header: test-results
    message: |
      ## Test Results
      
      ${{ steps.test-results.outputs.summary }}
      
      [View full report](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})

Retention and History

Raw results are only useful if you keep them long enough.

GitHub Actions:

- uses: actions/upload-artifact@v4
  with:
    retention-days: 30  # Keep for 30 days

For longer history:

Allure TestOps — paid, keeps unlimited history
Self-hosted S3/GCS bucket — store results.json and generate trend reports
Datadog/Grafana — send metrics from test runs for dashboards

What to Put in Every CI Report

A useful test report answers:

1. How many tests passed/failed/skipped? — Summary counts

2. Which tests failed? — Full list of failures with test names

3. Why did they fail? — Error messages and stack traces

4. What does it look like? — Screenshots for UI tests

5. How long did it take? — Execution time by test

6. Is it a new failure? — Comparison with previous run

Playwright's HTML report covers 1–5. Trend tools (Allure, TestOps) add 6.

Summary

| Format | Best for |

|--------|---------|

| JUnit XML | CI system integration, PR annotations |

| HTML | Human reading, screenshots, traces |

| JSON | Custom processing, dashboards |

| List | Real-time console output |

Minimum setup for CI:

1. Generate JUnit XML (CI can parse it)

2. Upload HTML report as artifact

3. Use if: always() on upload steps

Enhanced setup:

Allure report with trends
Slack notification on failure
PR comment with summary
Test annotation in diff view