Featured
Scorecery
A fine-tuned open model generates structured SAT items; a 5-layer automated eval gate (parse, schema, self-consistency, numeric-consistency, LLM-as-judge) scores and gates every item before a student ever sees it. Feeds a full digital-SAT-faithful adaptive exam app with IRT-based scaled scoring. Fixing a training/inference format mismatch was the unlock: it lifted valid structured output from ~0% to ~96% in a single alignment fix, the change that made every downstream stage possible.
# Eval gate: every generated item passes all five stages before shipping
gate = EvalGate([
ParseCheck(),
SchemaCheck(),
AnswerSelfConsistency(),
NumericRationaleConsistency(),
LLMAsJudge(model="judge-model"),
])
result = gate.run(item)
if not result.passed:
reject(item, reason=result.first_failure)