Pillar — Testing

Prefix: TST- Catalog: v1.1 deltas (additive to v1.0 Testing). What it measures: whether the test suite still gives an agent a reliable signal when the subject is a product portfolio — coverage that cannot be gamed, contracts between packages, and a named split between local fakes and real staging.

v1.0 already covers a test framework, coverage measurement, an 80% threshold, end-to-end journeys, resilient selectors, isolation, flake detection, and property-based tests. These v1.1 criteria close the gaps that appear once an agent is changing several applications that share packages.

Criteria in this pillar

TST-100 — Coverage include matches the source tree

Level: 3 · Scope: application · Check: deterministic
Coverage include lists every production module the application owns. Thresholds cannot be satisfied by covering a stub, a barrel file, or a hand-picked subset while the rest of the tree sits outside the gate.
Rationale: an 80% threshold (TST-030) is vacuous if the include set is three files. Agents use coverage to decide where to add tests; a gamed include sends them to the wrong place and hides untested paths.
Evidence expected: coverage config include (or equivalent) matches the application's source glob; CI fails when a new production file is added outside the include without an explicit, reviewed exclusion.

TST-110 — Contract tests for published packages

Level: 4 · Scope: portfolio · Check: deterministic
Every published workspace package (or equivalent shared library) has a contract test that consumers of that package can run. A breaking change to an exported signature or documented behaviour fails that test before the change lands.
Rationale: portfolio subjects share code. An agent that "fixes" a package without a consumer-side contract will green the package suite and break every application that imported the old shape.
Evidence expected: a contract-test job per published package; the job is required on changes to that package; at least one assertion against a documented export.

TST-120 — System under test is not mocked

Level: 3 · Scope: application · Check: deterministic
Tests do not mock the module they claim to test. Collaborators are replaced at documented seams (constructor injection, test fakes, in-memory bindings). A lint or review gate fails a test file that replaces the system under test with a mock.
Rationale: a test that mocks the unit under test is a test of the mock. Agents copy that pattern and ship changes that never ran against real code. Local fakes at seams keep the suite fast without lying about what passed.
Evidence expected: a documented seam policy; lint or grep that fails mock-of-SUT patterns; tests for non-trivial modules exercise the real module with injected fakes.

TST-130 — Hybrid fake and staging strategy documented

Level: 4 · Scope: portfolio · Check: manual
Unit and integration tests use local fakes. A documented, scheduled path (nightly or equivalent) exercises real staging — or an equivalent isolated environment — for the portfolio's critical journeys. The two layers are named, owned, and have a stated cadence.
Rationale: local fakes make agents fast; only a real environment proves the wiring. A portfolio that has one but not the other either blocks on slow suites or ships integration bugs no unit test could see.
Evidence expected: a committed test-strategy document naming both layers; schedule or workflow for the staging path; critical journeys listed and owned.

TST-140 — Portfolio coverage floor

Level: 4 · Scope: portfolio · Check: deterministic
Every application in the portfolio meets the coverage floor, or is marked not_applicable with a one-line justification. A portfolio report shows k/n applications at or above the floor. A newly added application cannot ship below the floor.
Rationale: a single green application does not make a product agent-ready. Portfolio scope (CSPC-31) is the scoring unit; the coverage floor has to be true of the whole subject, not of the one package an agent happened to touch.
Evidence expected: a portfolio coverage report or CI matrix; k/n with per-application detail; not_applicable entries justified; the floor matches TST-030 unless a documented exception exists.

v1.0 Testing — the published TST-010 through TST-080 criteria.
v1.1 pillars — draft catalog, including these deltas and the six new pillar stubs.
v1.0 specification — the published standard.

Criteria in this pillar

Level: 3 · Scope: application · Check: deterministic

Coverage include lists every production module the application owns. Thresholds cannot be satisfied by covering a stub, a barrel file, or a hand-picked subset while the rest of the tree sits outside the gate.

Rationale: an 80% threshold (TST-030) is vacuous if the include set is three files. Agents use coverage to decide where to add tests; a gamed include sends them to the wrong place and hides untested paths.

Evidence expected: coverage config include (or equivalent) matches the application's source glob; CI fails when a new production file is added outside the include without an explicit, reviewed exclusion.

Level: 4 · Scope: portfolio · Check: deterministic

Every published workspace package (or equivalent shared library) has a contract test that consumers of that package can run. A breaking change to an exported signature or documented behaviour fails that test before the change lands.

Rationale: portfolio subjects share code. An agent that "fixes" a package without a consumer-side contract will green the package suite and break every application that imported the old shape.

Evidence expected: a contract-test job per published package; the job is required on changes to that package; at least one assertion against a documented export.

Level: 3 · Scope: application · Check: deterministic

Tests do not mock the module they claim to test. Collaborators are replaced at documented seams (constructor injection, test fakes, in-memory bindings). A lint or review gate fails a test file that replaces the system under test with a mock.

Rationale: a test that mocks the unit under test is a test of the mock. Agents copy that pattern and ship changes that never ran against real code. Local fakes at seams keep the suite fast without lying about what passed.

Evidence expected: a documented seam policy; lint or grep that fails mock-of-SUT patterns; tests for non-trivial modules exercise the real module with injected fakes.

Level: 4 · Scope: portfolio · Check: manual

Unit and integration tests use local fakes. A documented, scheduled path (nightly or equivalent) exercises real staging — or an equivalent isolated environment — for the portfolio's critical journeys. The two layers are named, owned, and have a stated cadence.

Rationale: local fakes make agents fast; only a real environment proves the wiring. A portfolio that has one but not the other either blocks on slow suites or ships integration bugs no unit test could see.

Evidence expected: a committed test-strategy document naming both layers; schedule or workflow for the staging path; critical journeys listed and owned.

Level: 4 · Scope: portfolio · Check: deterministic

Every application in the portfolio meets the coverage floor, or is marked not_applicable with a one-line justification. A portfolio report shows k/n applications at or above the floor. A newly added application cannot ship below the floor.

Rationale: a single green application does not make a product agent-ready. Portfolio scope (CSPC-31) is the scoring unit; the coverage floor has to be true of the whole subject, not of the one package an agent happened to touch.

Evidence expected: a portfolio coverage report or CI matrix; k/n with per-application detail; not_applicable entries justified; the floor matches TST-030 unless a documented exception exists.