Evaluation Plan

Frozen Scope

• Dataset: KP_FROM_EQUILIBRIUM_MOLES_V1@1.1.0, with explicit ordered StudentStep values, alignments, validation results, marks, ECF applications, and tool-call traces.
• Problem: CAIE9701-KP-EQM-001@1.0.0.
• Policy: CAIE9701-KP-EQM-001-MP@1.0.0.
• Source evidence: internal authored draft aligned to the named CAIE 9701 syllabus family; no past-paper mark-scheme equivalence claimed.
• Review status: AI_DRAFT pending chemistry and curriculum review.

No marking or ECF result from this dataset may be described as CAIE agreement until qualified review evidence is attached.

Metrics

Metric	Unit	Initial gate
Numeric result accuracy	exact/tolerance case accuracy	100%
Expression equivalence	accepted/rejected pair accuracy	100%
Unit validation	case accuracy	100%
Significant-figure validation	case accuracy	100%
Student-step alignment	node mapping accuracy	>= 95% after reviewed labels
First-invalid-step localisation	exact node accuracy	>= 95% after reviewed labels
Marking agreement	mark-node exact match	Pending expert baseline
Bounded ECF agreement	rule/application exact match	Pending expert baseline
Misconception classification	macro F1	Observe, then set gate
Hint appropriateness	reviewed ordinal agreement	Observe, then set gate
Next-problem relevance	reviewed top-1 relevance	Unavailable until reviewed bank exists
Tool-call correctness	required/forbidden call accuracy	100%
Trace completeness	complete authoritative traces	100%
Unsupported-marking-claim rate	unsupported authoritative claims	0%

Evaluation Layers

1. Unit tests for every deterministic tool and reason code.
2. Graph tests for node/edge validity, acyclicity, and lineage.
3. Golden attempt tests covering reordered, missing, propagated, and independent errors.
4. Policy tests for mark-node and ECF boundaries.
5. Orchestrator tests for routing, calls, retries, and trace structure.
6. Human review for alignment ambiguity, misconception labels, hints, and follow-up relevance.

scripts/run-evals.py replays every frozen case through scripts/reference_harness.py. The reference harness is evaluation infrastructure in this repository, not the product implementation; Demo PR 2 must later run the same dataset through the product harness in standard-trainer-demo.

Change Control

Any problem, graph, tool, policy, prompt, or model change increments its version where semantics change, reruns the frozen set, reports regressions, and links reviewer status. Confirmed failures become cases before closure.