Evaluation artifacts¶

English | 日本語

evaluations/ holds reviewed inputs and selected evidence needed by current consumers. It is not a per-model or per-run archive. The repository's source maintainers own this policy; Issue #1011 records its alternatives and initial exact-file disposition.

Ownership and lifecycle¶

evaluations/lifecycle.json is the single family-scoped ownership catalog. Every retained artifact belongs to exactly one family and lifecycle. Each family names its purpose, owner, update trigger and retirement condition; each artifact names its actual consumers. A consumer is an existing test, script, fixture input/tool, or an explicit manual evaluation or historical review owner. Merely listing an artifact in this catalog or the source inventory is not a consumer.

Lifecycle	Purpose and source of truth	Update and retirement
forward-fixture	Authored scenarios for the family's current evaluation contract; evaluator expectations remain separate from agent input. A manual evaluator is explicit when there is no automated test.	Extend the family when its contract or failure scenarios change. Retire only after consumer and unique-scenario review.
regression-fixture	Representative inputs or pinned compatibility tooling used by current tests. A lockfile is generated from its package declaration; it is not an independently authored result.	Update with the consuming behavior or explicitly selected compatibility version. Preserve the input/tool/lock relationship in any migration.
current-baseline	The one baseline per family currently selected by a report or comparison consumer. Its capture date and unknown values remain historical facts, not fresh measurements.	Replace through reviewed comparable evidence and update all consumers together; preserve any uniquely required prior evidence separately.
immutable-history	Selected original observations or retired scenarios whose exact provenance or historical meaning still has an explicit consumer. Every retained file has a specific fidelity reason.	Do not rewrite observations to match current behavior. Retire only after all consumers and unique evidence have a proven disposition. Never use it as current benchmark input or model routing inventory.

The catalog classifies evidence; the original artifact owns its observations and schema. Existing embedded historical lifecycle declarations must agree. The hash-based source inventory remains the owner of source completeness, hashes and packaging identity; the catalog does not duplicate those hashes or grant distribution authority. Evaluation tooling and its dependencies stay outside the distributed Plugin.

Generated comparisons are disposable outputs until deliberately selected as a current baseline or immutable historical evidence. Keep transient reports in the established temporary/ignored output location or the owning CI artifact store. Do not retain raw transcripts, provider payloads, every trial, or another model inventory in this directory.

Extend a family before adding a file¶

Add scenarios to the existing family input. Compare repeated models or versions in one family comparison, preserving each selected observation's original identity and limitations. A new standalone historical file is justified only by a distinct consumer and concrete fidelity requirement, not a model name, date or another run. Record that reason in historical_fidelity.

The two existing execution-economics model observations share one family but retain their original filenames and bytes: one proves unavailable-model non-substitution; the other preserves two bounded-inference observations with unknown effective identity and cost. Their schemas, provenance and existing regression consumers differ. Combining their payloads would rewrite evidence without removing a runtime dependency. The same rule protects the dated calibrated-delegation comparison and DESIGN.md research snapshot.

measurement-cases.json is retained only for maintainers reviewing the retired #79 mechanism and its removal in PR #649. It has no current automated or benchmark consumer. On the next change to that family, review the five original scenarios against the retained Git/Issue history before proposing removal. This is an event-triggered review, not a scheduled task or permission to restore measurement collection.

Growth review¶

The catalog owns the numeric review limits; the validator reports current artifact count, bytes, family count and lifecycle counts. Crossing either limit requires a family/consumer review before accepting the candidate. The initial limits provide four additional files and roughly one largest existing fixture of byte growth over the inspected baseline. They are local review triggers, not universal size rules, a per-run retention quota or permission to delete.

Review only affected families. Consolidate redundant inputs where fidelity permits, or explain why the added evidence is needed. A reviewed change may revise the limits with its measured inventory and rationale in the Issue/PR; the validator never raises a limit automatically. This also catches expansion inside an existing file. A new historical split requires its fidelity reason even below the directory limits.

Lifecycle validation¶

Purpose: reject incomplete ownership and accidental promotion of history while preserving the existing evidence. The enforcing boundary is scripts/evaluation_lifecycle.py. It reuses the source inventory's Git path projection and safe-file resolution; it does not create a second hash index or execute the declared consumers.

Invariants: all tracked evaluation files except the catalog itself have exactly one family/lifecycle, a resolvable tracked consumer with a compatible role, and complete ownership fields. Untracked evaluation source, competing current baselines, historical/current declaration conflicts and unsafe paths fail. History admits only regression or historical-review consumers. Runtime routing is not an admitted consumer role. Semantic review must still verify that a declared consumer actually uses the evidence and that no undeclared dynamic loader promotes it; existence and role validation alone cannot prove that.

Quantitative bounds: one catalog is at most 128 KiB, with at most 64 families, 512 artifacts and 32 consumers per artifact. Each descriptive value is at most 2,000 characters. One invocation uses the existing bounded Git inventory reads and one catalog traversal; it makes zero model/provider API calls, installs, deletions, artifact rewrites, automatic retries or background operations. These input bounds are distinct from the catalog's growth-review limits.

Failure containment and observable evidence: malformed or unknown input, duplicate identity, missing consumer/file, ambiguous lifecycle or symlinked path returns a nonzero result and a bounded diagnostic. Growth returns review_required with the measured counts and trigger names. Success returns valid with those counts. Neither result claims a benchmark passed, a model was selected, an artifact may be deleted, or long-term growth was prevented.

Authority and recovery: this check is read-only. Correct the exact declaration, restore missing evidence from Git, or obtain the ordinary reviewed migration; then rerun it. Missing unique-evidence or consumer disposition blocks retirement. No deletion command or automatic age-based expiry exists. Reverting this additive catalog/check/policy restores the previous source layout without changing historical payloads or provider state.

Compatibility and migration: the initial migration retains all pre-existing artifacts byte-for-byte; exact paths are the catalog entries, consolidated metadata is family-scoped, and moved/retired/deleted paths are empty. Later migrations enumerate exact retained, consolidated and retired files in their Issue/PR, account for every consumer and unique observation, and update the source inventory and tests together. Unknown provenance blocks removal. Never choose deletion targets by glob or regard a missing catalog entry as evidence that a file is disposable.

Acceptance tests use real temporary files and Git inventories to cover valid classification, unclassified or missing evidence, missing/unsafe consumers, duplicate ownership, historical-as-active use and growth triggers. They also validate this repository's complete catalog through ordinary test discovery, so required and CI profiles enforce the check offline without invoking a model or external API. Existing consumer tests retain ownership of their behavioral assertions; the catalog check does not replace them.

python3 scripts/evaluation_lifecycle.py

Stage exact new source paths before source-inventory generation and final catalog validation. Both checks intentionally reject untracked source omissions. A review does not publish documentation, install the Plugin or change release authority.

Historical candidate classification¶

Purpose and authority: source maintainers admit a narrow evidence-only design classification through scripts/quality_pr_preflight.py, shared by local pre-push composition and the hosted design step. The portable Plugin validator continues to reject arbitrary material negative declarations. The evaluation lifecycle owner proves this repository's catalog transformation; this is not a manual-review override or a new Plugin capability.

Invariants: one exact base/head diff may add Markdown immutable-history artifacts and their catalog registration, accompanied only by documentation and the exact generated source inventory. Existing families' fields and artifact entries, schema, growth limits, owners and consumer authority remain unchanged. New families reuse an existing owner; additions use an already registered historical-review consumer. Existing evaluation payloads cannot change. The current catalog must pass its normal completeness and growth checks, and the source inventory must pass its existing path/hash checks. A change solely to that exact, valid generated projection is also eligible.

The files inspected by these owners must match the requested head in the tracked working files. A divergent merge checkout is not evidence for this exception: update the candidate or provide a matching checkout. Unknown, malformed, untracked, renamed/deleted, non-history, wrong-path, or mixed executable/configuration evidence retains review_required; positive design classification still follows the original source/reference/failure-test chain.

Bounds and observable evidence: one request examines one base/head pair and at most 512 changed paths, retaining the catalog's existing byte/family/artifact limits. It performs zero provider effects, retries, source writes or automatic approvals. Accepted negatives retain revision/diff identity and report stable reason historical_evaluation_addition or generated_source_projection. Historical evidence is not execution authority, product success or release acceptance; ordinary semantic review and exact-head hosted gates remain required.

Recovery and compatibility: correct the exact evidence and rerun the same source-owned gate. No input schema, persisted state or Plugin runtime migrates. Rollback removes this source composition and returns these candidates to review; it never rewrites evidence or bypasses the portable gate. Real Git candidate tests cover historical addition, projection-only evidence, changed ownership/retention/entries, stale and malformed evidence, mixed configuration and unchanged positive classification. Local and hosted entry points use the same classifier.

Thin reliability comparison¶

Maintainers reviewing #1122 use the selected historical comparison to distinguish the seven treatments, their source/scenario decision results, and the decision not to promote a new Plugin layer. Its eight cases per arm are not production outcomes; command counts are self-reported, and latency and full context size are unavailable. Preserve those limitations when reconsidering structural providers, principles, hooks or scenario generation. The review retains no experimental runtime or new normative Skill requirement.