Confusing test summary when using subtests

[tsvikas]

Problem

The current summary format double-counts failures by adding test-level failures to subtest-level failures.

Scenario: Running 3 tests:

• Test A: no subtests, fails
• Test B: 3 subtests, all pass
• Test C: 3 subtests, 1 fails and 2 pass

Current output:

With pytest -q:

3 failed, 1 passed, 5 subtests passed in 0.03s

With pytest:

3 failed, 1 passed in 0.03s

Why this is misleading:

The "3 failed" adds:

• 2 regular test failure (Test A and C)
• 1 failed subtests (in Test C)

This double-counts Test C's failure at both the test level and subtest level, making it unclear how much actually failed.

Suggested improvements

Several options to make the summary clearer:

1. Omit subtests from summary:
   2 failed, 1 passed
   this assumes the division is by tests. a failed test with 1000 failed subtests and 1000 passing subtests will still show as 1 fail.

2. Like 1, but add the subtests counts (maybe in parentheses):
   2 failed, 1 passed (1 subtest failed)
   or
   2 failed, 1 passed (1 subtest failed, 5 subtests passed)

3. Treat each subtest as an individual test:
   2 failed, 5 passed
   This count each subtest execution as a separate test result. In this case: A (failed) + C.1 (failed) vs B.1, B.2, B.3, C.2, C.3 (all passed). This approach treats subtests equivalently to regular tests.

Environment

pytest 9.0.0

Reproduction

def test_fail():
    assert False

def test_pass(subtests):
    for i in range(3):
        with subtests.test(msg="all pass", i=i):
            assert True

def test_mixed(subtests):
    for i in range(3):
        with subtests.test(msg="1 fail", i=i):
            assert i != 0

Run with pytest -q to see the summary line.