This section describes how a CARL assessment is performed. Assessor implementations — deterministic engines, AI-assisted evaluators, human auditors, or any combination — MUST conform to this methodology to produce a result that can be compared across subjects.

Assessor types

A conformant assessment MAY be performed by:

• A deterministic assessor — a tool that evaluates every criterion by reading files, running commands, and parsing outputs, without involving an LLM. The reference implementation, @wentzel/carl-engine, is deterministic.
• An AI-assisted assessor — a tool that uses a large language model to evaluate criteria whose judgment cannot be fully encoded as a rule. The three reference prompts at /prompts/ are the blessed AI-assisted evaluators for v1.0.
• A human assessor — an auditor who reads the catalog, evaluates the subject by hand, and produces the conformant report format.
• A mixed assessor — any combination of the above, typically a deterministic engine augmented with one or more AI-assisted passes for the remaining criteria.

All four produce results in the same report format. The assessor's type field in the report MUST be set to the combination actually used (deterministic, ai-assisted, human, or mixed).

Determinism requirement

At least 80% of criteria in the standard catalog MUST be deterministic. This is a requirement on the catalog, not on any particular assessor — an assessor is free to evaluate every criterion deterministically, or to use AI for many of them, as long as the catalog's deterministic floor is met.

The reason: reproducibility. Two assessors that evaluate the same subject against the same catalog version should produce the same level within a narrow variance. Pure LLM-based evaluation is not reproducible enough at scale to anchor a comparable standard.

AI-assisted variance

Assessors that use AI-assisted evaluation MUST identify which criteria were AI-assisted in the report and SHOULD state measured variance. The standard recommends variance below 2% (0.02). Variance is measured by running the AI-assisted portion of the assessment n times on a fixed fixture repository, with temperature pinned and random seeds controlled where possible, and reporting the fraction of criteria whose result differs between runs.

Applicability marking

Criteria MAY be marked not applicable (not_applicable) when they genuinely do not pertain to a subject. Examples:

• A pure-Python repository marks TypeScript-specific criteria not_applicable.
• A documentation-only repository marks runtime observability criteria not_applicable.
• A monolithic repository marks monorepo-specific criteria not_applicable.

Applicability markings MUST be:

• Stable. The same criterion on the same subject should produce the same applicability marking across assessments, absent structural change to the subject.
• Justified. The assessor's report MUST state why each not_applicable marking was chosen. "Not applicable" without reason is not a valid marking.
• Reviewable. The applicability marking is a judgment call; third parties re-assessing a subject MAY disagree and re-mark it. The report format captures the justification so disagreements are visible.

Criteria marked not_applicable are excluded from both the numerator and denominator of the 80% gating rule — they neither help nor hurt the level calculation.

Evaluation frequency

Subjects claiming conformance MUST be re-assessed at least every 90 days or after any substantial infrastructure change, whichever comes first.

"Substantial infrastructure change" is deliberately broad. Examples that trigger re-assessment:

• A new linter, test runner, or CI system is adopted.
• A monorepo is split into multiple repositories, or several repositories are consolidated.
• The default branch is renamed, or branch-protection rules are materially changed.
• A package manager or runtime is swapped (e.g., npm → pnpm, Node 18 → Node 22).

Minor changes — a dependency bump, a bug fix, a new feature — do not require re-assessment. The spirit of the rule is that a published report should still describe reality at the time someone reads it, within a reasonable envelope.

Produced artifact

Every conformant assessment produces a report in the format described at /spec/v1.0/report-format/. The report's fields are minimum; assessors MAY produce richer structured output (per-file breakdowns, remediation priorities, trend data) as long as the minimum fields are present and well-formed.

Related

• Report format — exact output contract.
• Conformance claims — what a published conformance statement must and must not say.
• Reference Prompts — the three AI-assisted evaluators blessed for v1.0.