Skip to main content
Documentation menu

Prefix: TST- Catalog: v1.1 deltas (additive to v1.0 Testing). What it measures: whether the test suite still gives an agent a reliable signal when the subject is a product portfolio — coverage that cannot be gamed, contracts between packages, and a named split between local fakes and real staging.

v1.0 already covers a test framework, coverage measurement, an 80% threshold, end-to-end journeys, resilient selectors, isolation, flake detection, and property-based tests. These v1.1 criteria close the gaps that appear once an agent is changing several applications that share packages.

Criteria in this pillar

TST-100 — Coverage include matches the source tree

  • Level: 3 · Scope: application · Check: deterministic
  • Coverage include lists every production module the application owns. Thresholds cannot be satisfied by covering a stub, a barrel file, or a hand-picked subset while the rest of the tree sits outside the gate.
  • Rationale: an 80% threshold (TST-030) is vacuous if the include set is three files. Agents use coverage to decide where to add tests; a gamed include sends them to the wrong place and hides untested paths.
  • Evidence expected: coverage config include (or equivalent) matches the application's source glob; CI fails when a new production file is added outside the include without an explicit, reviewed exclusion.

TST-110 — Contract tests for published packages

  • Level: 4 · Scope: portfolio · Check: deterministic
  • Every published workspace package (or equivalent shared library) has a contract test that consumers of that package can run. A breaking change to an exported signature or documented behaviour fails that test before the change lands.
  • Rationale: portfolio subjects share code. An agent that "fixes" a package without a consumer-side contract will green the package suite and break every application that imported the old shape.
  • Evidence expected: a contract-test job per published package; the job is required on changes to that package; at least one assertion against a documented export.

TST-120 — System under test is not mocked

  • Level: 3 · Scope: application · Check: deterministic
  • Tests do not mock the module they claim to test. Collaborators are replaced at documented seams (constructor injection, test fakes, in-memory bindings). A lint or review gate fails a test file that replaces the system under test with a mock.
  • Rationale: a test that mocks the unit under test is a test of the mock. Agents copy that pattern and ship changes that never ran against real code. Local fakes at seams keep the suite fast without lying about what passed.
  • Evidence expected: a documented seam policy; lint or grep that fails mock-of-SUT patterns; tests for non-trivial modules exercise the real module with injected fakes.

TST-130 — Hybrid fake and staging strategy documented

  • Level: 4 · Scope: portfolio · Check: manual
  • Unit and integration tests use local fakes. A documented, scheduled path (nightly or equivalent) exercises real staging — or an equivalent isolated environment — for the portfolio's critical journeys. The two layers are named, owned, and have a stated cadence.
  • Rationale: local fakes make agents fast; only a real environment proves the wiring. A portfolio that has one but not the other either blocks on slow suites or ships integration bugs no unit test could see.
  • Evidence expected: a committed test-strategy document naming both layers; schedule or workflow for the staging path; critical journeys listed and owned.

TST-140 — Portfolio coverage floor

  • Level: 4 · Scope: portfolio · Check: deterministic
  • Every application in the portfolio meets the coverage floor, or is marked not_applicable with a one-line justification. A portfolio report shows k/n applications at or above the floor. A newly added application cannot ship below the floor.
  • Rationale: a single green application does not make a product agent-ready. Portfolio scope (CSPC-31) is the scoring unit; the coverage floor has to be true of the whole subject, not of the one package an agent happened to touch.
  • Evidence expected: a portfolio coverage report or CI matrix; k/n with per-application detail; not_applicable entries justified; the floor matches TST-030 unless a documented exception exists.