Skip to main content

RUNNING TESTS

Test Reports

Generate detailed test reports in multiple formats for debugging, CI/CD integration, and performance analysis. Karate provides comprehensive reporting out of the box.

Report Types​

HTML Reports (Default)​

Rich, interactive reports for development and debugging:

  • Request/response details — complete HTTP interaction data
  • Step-by-step execution — visual test flow with timing
  • Inline debugging — print statements and variable values
  • Screenshots — automatic UI test captures
  • Timeline visualization — Gantt-style thread utilization analysis
  • Dark mode (v2) — Bootstrap 5.3 with light/dark theme toggle
  • Tag filtering (v2) — interactive tag-based filtering in the dashboard

JUnit XML Reports​

Standard format for CI/CD tool integration:

Java
SuiteResult results = Runner.path("classpath:features")
        .outputJunitXml(true)
        .parallel(5);

Cucumber JSON Reports​

Structured data for custom dashboards and third-party tools:

Java
SuiteResult results = Runner.path("classpath:features")
        .outputCucumberJson(true)
        .parallel(5);

JSONL Event Stream (v2)​

Memory-efficient event stream for real-time progress monitoring and integration with external reporting tools:

Java
SuiteResult results = Runner.path("classpath:features")
        .outputJsonLines(true)
        .parallel(5);

Outputs karate-events.jsonl — one JSON object per line, each representing a test lifecycle event (suite start, scenario enter/exit, step results, etc.). This format is replayable and can be used for:

  • Real-time progress dashboards
  • Aggregating results from distributed test runs
  • Integration with platforms like Allure, ReportPortal, or JIRA/X-Ray

Report Location​

Default Directory Structure​

target/karate-reports/
├── karate-summary.html       # Main overview
├── karate-timeline.html      # Performance analysis
├── com.example.UserTest.html # Individual feature reports
└── *.json, *.xml            # JUnit/Cucumber outputs

Custom Report Directory​

Java
SuiteResult results = Runner.path("classpath:features")
        .outputDir("build/custom-reports")
        .outputJunitXml(true)
        .outputCucumberJson(true)
        .parallel(5);

HTML Report Features​

Console Output Access​

After test execution, the console shows direct report links:

html report: (paste into browser to view)
-----------------------------------------
file:///project/target/karate-reports/karate-summary.html

Report Contents​

HTML reports include:

  • Execution summary - Pass/fail counts and timing
  • Request details * headers, method, URL, body
  • Response data - Status, headers, complete response
  • Debug information * print statements and variables
  • Performance metrics - Response times and efficiency

Timeline Analysis​

Understanding Timeline Reports​

The karate-timeline.html provides visual thread utilization:

Thread 1: |████████████████████| Feature A (2.1s)
Thread 2: |██████████████████  | Feature B (1.8s)
Thread 3: |███████████████████ | Feature C (1.9s)

Efficiency: 85% (good utilization)

Key metrics:

  • Thread utilization - How well threads are used
  • Feature distribution - Load balancing across threads
  • Performance bottlenecks - Slow tests that block others

CI/CD Integration​

Jenkins Configuration​

Configure Jenkins to publish Karate reports:

Jenkinsfile
pipeline {
    stages {
        stage('Test') {
            steps {
                sh 'mvn test'
            }
            post {
                always {
                    publishHTML([
                        reportDir: 'target/karate-reports',
                        reportFiles: 'karate-summary.html',
                        reportName: 'Karate Reports'
                    ])
                    publishTestResults(
                        testResultsPattern: 'target/surefire-reports/*.xml'
                    )
                }
            }
        }
    }
}

GitHub Actions Configuration​

.github/workflows/test.yml
- name: Run tests
  run: mvn test

- name: Publish test results
  uses: dorny/test-reporter@v1
  if: always()
  with:
    name: Karate Test Results
    path: target/surefire-reports/*.xml
    reporter: java-junit

- name: Upload HTML reports
  uses: actions/upload-artifact@v3
  if: always()
  with:
    name: karate-reports
    path: target/karate-reports/

Report Configuration​

Enabling Multiple Formats​

Java
@Test
void comprehensiveReporting() {
    SuiteResult results = Runner.path("classpath:features")
            .outputJunitXml(true)          // For CI/CD
            .outputCucumberJson(true)      // For dashboards
            .outputHtmlReport(true)        // For debugging (default)
            .outputDir("target/ci-reports")
            .parallel(8);

    assertFalse(results.isFailed());
}

Report Verbosity Control​

Control what appears in reports:

Gherkin
Feature: Report configuration

Scenario: Verbose reporting
  * configure report = { showLog: true, showAllSteps: true }
  * print 'This will appear in reports'

Scenario: Minimal reporting
  * configure report = { showLog: false }
  * print 'This will not appear in reports'

Third-Party Integration​

Cucumber Reporting Plugin​

Generate enhanced reports using Maven plugin:

pom.xml
<plugin>
    <groupId>net.masterthought</groupId>
    <artifactId>maven-cucumber-reporting</artifactId>
    <version>5.7.2</version>
    <executions>
        <execution>
            <phase>verify</phase>
            <goals>
                <goal>generate</goal>
            </goals>
            <configuration>
                <projectName>API Test Suite</projectName>
                <outputDirectory>target/cucumber-reports</outputDirectory>
                <inputDirectory>target/karate-reports</inputDirectory>
                <jsonFiles>
                    <param>**/*.json</param>
                </jsonFiles>
            </configuration>
        </execution>
    </executions>
</plugin>

Common Gotchas​

  • Missing reports: Check if tests actually executed
  • Large report files: Reduce logging verbosity with configure report
  • Broken report links: Use relative paths and proper directory structure
  • CI report access: Ensure proper artifact archiving in CI/CD pipelines

Next Steps​

Master test reporting for effective debugging and monitoring:

On This Page