The score is the easy part
We first built a benchmark engine for structured questions. There, the mark is simple. The score matches the teacher's score, or it does not.
Essay feedback is harder to measure. Feedback can be accurate and still useless, such as "the argument is weak". Feedback can also be genuinely helpful although it uses none of the teacher's words. You must answer three questions. Did the AI catch the same issues the teacher caught? Did it explain them clearly? Did it reach the right score?
These three questions do not always move together. An AI that scores perfectly can still write vague comments. An AI that writes detailed feedback can under-mark every time.
For that reason we built a separate benchmark for essay feedback. It measures all three independently. Its design matters as much as the marking engine itself.
Why this matters: more than a number
Most claims about AI quality are hard to verify. "Our model is accurate" is easy to say and hard to challenge. A benchmark makes the claim falsifiable.
It tracks genuine improvement. We change one thing in the marking engine: a prompt, a model version, or a feedback structure. We then run the same essays through the same benchmark. If the score rises, the change helped. If the score falls, we revert it. Without the benchmark, "our AI got better" is an assertion. With it, the statement is a measurement.
It detects invented feedback. This is one of the most serious failure modes. The model writes fluent, confident feedback that has no relation to the student's essay. The AI judge catches this. Its score collapses to near zero, even when other components show accidental partial matches.
It aligns the AI with human quality. The benchmark measures agreement with real teacher judgement. A high overall score means the feedback is close to what an experienced teacher writes. That is a measurement, not a claim.
Three questions, one score
Essay feedback quality reduces to three questions a teacher asks:
- Did it catch the right things? → Issue Recall
- Was the feedback useful? → AI Judge Score
- Did it reach the right grade? → Score Accuracy
We weight them in that order. Recall counts most and score accuracy counts least.
The order is deliberate. A student reads AI feedback before a teacher reviews it. A missed structural problem does lasting damage, and no score correction repairs it. A thesis that contradicts itself is one example. An unsupported central claim is another. Generic feedback does the same damage. A wrong mark, by contrast, takes seconds to correct.
Issue Recall: catch what the teacher caught
Issue recall measures whether the AI identified the same problems the teacher marked. We penalise a miss, never an extra. The ideal AI catches everything the teacher caught, and possibly more. An extra AI observation must not reduce the score. Only a missed teacher comment reduces it.
Comments also carry unequal weight. A missed point about the student's thesis costs more than a missed comma. We weight comments about structure, argument, evidence, and conclusion more heavily. We weight surface corrections such as typos and punctuation less. This encodes a belief. If the AI misses what matters most, strong recall on minor issues must not conceal that.
AI Judge: what a number cannot capture
The most important question is qualitative. Would this feedback help the student?
We answer it with an AI judge. A separate model reads the teacher's marking and the AI's feedback. It evaluates issue identification, feedback quality, and missed issues independently. As with recall, we instruct the judge to penalise only a miss, never an extra. The bar is one-sided: do at least as much as the teacher, and more if you can.
Score Accuracy: why a grade-band error costs more
A 2-mark error on a 25-mark essay is a rounding issue. A 6-mark error can move a student into a different grade band. These two problems are not proportionally different. The second is categorically worse.
We apply exponential decay once an error exceeds a small tolerance. This rewards consistent proximity and punishes a large divergence sharply.
| AI score (teacher = 18/25) | Score accuracy |
|---------------------------|---------------|
| 17.75 (1% off) | 99/100 |
| 18.5 (2% off) | 98/100 |
| 20.5 (10% off) | 67/100 |
| 22.5 (18% off) | 28/100 |
Why recall is harder than it sounds
Our first benchmark measured similarity one comment at a time. It matched each AI comment to the nearest teacher comment with semantic embeddings. An embedding is a numerical representation of text that captures meaning instead of exact wording. It scores "unclear thesis" and "the argument lacks a central claim" as similar, although they share no words.
The method worked in simple cases. It then under-counted recall in a common case. The AI often writes one detailed paragraph that covers what the teacher marked in three separate annotations.
The arithmetic is punishing. Match that paragraph to its best-fitting teacher comment, and it cannot match the other two. Two of three issues then count as missed, although the AI addressed all three. That gives 33% recall on a response that was substantively correct.
Length normalisation does not repair this. Once a comment matches, it is consumed. You cannot divide one AI comment across several teacher points by a word-count adjustment. The problem is structural, not a matter of metric choice.
Our fix decomposes every comment into atomic issues before the match. Take one teacher annotation: "the thesis is too hedged — commit to a position, and explain why the constructivist framing is necessary". That annotation contains two distinct points. An AI comment that addresses both inside a longer passage now matches both. The first point no longer consumes it.
We also kept one design decision from the original benchmark. Where embedding similarity falls in a grey zone, a lightweight model call verifies the match. This is better than a hard threshold, which drops legitimate matches. On the essay below, that step resolved three further matches that embeddings alone missed.
A last adjustment adds a lookahead window across adjacent paragraphs. A teacher sometimes annotates the start of a paragraph for an issue that continues into the next one. The AI can address that issue a paragraph later. Without lookahead, this counts as a miss. With lookahead, it counts as a match.
In practice: a real essay
The essay is a JC Knowledge and Inquiry paper on moral epistemology. The prompt reads: "We can know what is right for us individually, but we can never know what is right for the whole of humanity." The student takes a constructivist position. They reject moral realism, but argue that collective, pragmatic knowledge of right and wrong remains possible.
The teacher left 25 comments across 12 paragraphs. These cover extreme framing in the introduction, requests for reasoning on specific claims, word choice corrections, and one structural concern. That concern is that the constructivist argument, the essay's central contribution, is too brief for the material before it.
The AI matched 19 of those 25 comments. It added 55 observations the teacher did not make, including:
- Philosophical enrichment. The AI noted that the divine command section must engage with the Euthyphro dilemma, a canonical counter-argument the teacher did not annotate.
- Argumentation critique. The AI gave a fuller treatment of Moore's open question argument, and showed where the student's use of it falls short.
- Internal contradiction. The AI found a tension in the constructivist framing. A claim to collective pragmatic knowledge risks a collapse into majoritarianism, and the essay must address that.
- Structural signposting. The AI advised the student to separate feasibility critiques from adequacy critiques more clearly in the middle sections.
The six missed comments were mostly specific to the teacher's reading. They cover the "yardsticks for progress" framing and over-extreme claims in the introduction. They also cover the teacher's note that the central constructivist argument is underdeveloped, plus several word choices the teacher circled.
The AI judge wrote this assessment:
The AI provides detailed, actionable philosophical and language feedback and often aligns with the teacher's concerns — over-extremity, need to explain reasoning, clarify terms. However, it misses several teacher-specific points: the teacher repeatedly flags claims as too extreme, asks for specific clarifications and attributions, and emphasises that the constructivist solution is the main focus and is underdeveloped. Overall, AI feedback is higher-quality and broader than the teacher's, but does not capture all of the teacher's idiosyncratic corrections and emphasis on proportion and focus.
The overall score is 74/100. Issue recall is 76%. The AI judge gave 71/100. The AI under-marked by 3 points, 22 against 25 out of 30, which is a 10% gap on score accuracy. Three of the 19 matched pairs were borderline embedding cases that the verification step resolved.
How we keep the metric itself honest
A benchmark is only useful if it separates good marking from bad marking reliably. We verify this with sanity checks that run on every release. The test is deliberately extreme. The worst case feeds in AI comments from a completely different essay. The best case feeds in the teacher's own marking as the AI output.
| | Worst case | Best case |
|---|-----------|-----------|
| Comment recall | 16.7% | 94.4% |
| Score accuracy | 33.4/100 | 98.3/100 |
| AI judge score | 2.0/100 | 97.0/100 |
| Overall score | 15.6/100 | 96.2/100 |
Three of these numbers need an explanation.
Worst-case recall is 16.7%, not zero. Comments from a different essay still contain generic educational phrases, such as "the argument is unclear" or "more evidence needed". A phrase like that can clear an embedding threshold by accident. Every embedding-based match system has this limit. It is one reason the benchmark uses three components instead of one.
Worst-case score accuracy is 33.4/100, not zero. The AI marked the wrong essay and still produced a number. That number landed near the teacher's mark by coincidence, not by comprehension. Exponential decay cuts it to 33.4, but it does not collapse to zero.
The AI judge catches both cases. It scores 2.0/100 on completely irrelevant feedback. It is the one component that surface coincidence cannot fool. Chance can move recall and score accuracy, but the judge reads the content and gives a verdict. The 80-point gap between worst and best, 15.6 against 96.2, is the number that matters.
Still improving
The benchmark described here is not the final version. Publication is part of the work, because a transparent measurement is one that others can challenge and refine.
Multi-teacher calibration. Two teachers who mark one essay rarely write identical feedback. The current benchmark measures against a single teacher's standard, so it merges genuine AI error with normal teacher-to-teacher variation. A richer version compares against several teachers and separates the two.
Cross-subject expansion. The current benchmark covers argumentative essays. Narrative writing, scientific reports, and structured exam responses each carry different quality dimensions. A wider test set is the next step.
Longitudinal quality. Does the AI hold the same quality at its 500th essay as at its first? Consistency at scale is a different question from accuracy on a sample, and schools that deploy at volume need the answer.
The deeper goal is never an AI that replaces teacher judgement. The goal is an AI that extends it. It must catch what a teacher catches, and sometimes catch what the teacher had no time to catch. The benchmark is how we stay honest about our progress. We will continue to publish these results as the work matures.
Do you want to know how Ren marks essays at your school? Get in touch to learn more.