Sample report
Jonas Erikson
- Task
- Payment events
- Opening
- Backend engineer, payments
- Harness HarnessThe app the candidate used to work with an AI agent, such as Claude Code, Codex or OpenCode. The choice of harness does not affect the score.
- Claude Code
- Started
- 2026-11-13 10:02
- Submitted
- 2026-11-14 15:48
- Working time
- 6 h 40 min
- Model cost Model costThe cost of the AI model used during the trial.
- $11.80
Overall score
Overall score, 52 out of 10052
What the overall score covers
The overall score reflects the six dimensions with enough evidence. Decision making is excluded.
Evaluator's summary
Jonas asked the agent how the payment system retries events before any code was written. He had a repeated event checked and, when the last check showed two failures, had the refund status fixed and checked again. The daily totals for finance were still failing at the submission, and he accepted the agent's assumption that a refund always follows its payment.
A person makes the final decision. A gap of a few points between candidates is too small to interpret.
Dimensions
Each dimension has four levels. The evaluator, an AI agent, uses the evidence to choose a level for each dimension.
| Dimension | Level LevelOne of four descriptions of what the evidence shows for a dimension. Each level carries points. | Weight WeightHow much a dimension counts towards the overall score. | Evidence |
|---|---|---|---|
| How the candidate worked | |||
| Understanding of the problem | Working understanding Level 3 of 4 | Weight 15% | Evidence unavailable in this sample |
| Analytical thinking | Cases worked through Level 3 of 4 | Weight 20% | Evidence unavailable in this sample |
| Decision making | Not enough evidence Not enough evidenceThe session record and the submission provide too little evidence to choose a level. This dimension is excluded from the overall score. | Weight 10% | Excluded from the overall score |
| Own ideas | Adjustments only Level 2 of 4 | Weight 10% | Evidence unavailable in this sample |
| Verification | Checked and followed through Level 3 of 4Above level 2: Jonas read what the checks showed and had a failing one fixed and checked again. Below level 4: one failing check was left as it was, and the agent's assumption about refunds went unquestioned. | Weight 15% | See the evidence for Verification |
| What the candidate submitted | |||
| Meets the requirements | Partly met Level 2 of 4 | Weight 15% | Evidence unavailable in this sample |
| Quality of execution | Usable after rework Level 2 of 4 | Weight 15% | Evidence unavailable in this sample |
Verification
Level 3 of 4 · Checked and followed through
Rationale
Level 3 and not 2: Jonas asked for checks, read what they showed and had a failing one fixed and checked again. Level 3 and not 4: the checking began once the service existed, one failing check was left as it was, and the agent's assumption about refunds went unquestioned.
Observations ObservationA specific question about the trial, answered using a quote from the session record or the submission.
The evaluator uses all the available evidence to choose a level. Open a quote to see its context.
- V5
Was a case other than the basic one checked at the candidate's request?
YesThe candidate wrote send the same payment.captured twice, ten ms apart, and show me the rows
Show surrounding messagesHide surrounding messages
- AI agent
The handler stores each event and updates the order. Shall I move on to refunds?
- Candidate quoted
send the same payment.captured twice, ten ms apart, and show me the rows
- AI agent
One row for the payment; the second event was recognised as a repeat and ignored.
- AI agent
- V4
Did the candidate catch a defect or a false statement of the agent?
NoThe AI agent wrote Events can arrive out of order; I assume a refund always follows its payment.
The AI agent wrote this. The candidate replied "ok, continue".
Show surrounding messagesHide surrounding messages
- Candidate
now handle events that arrive in the wrong order
- AI agent quoted
Events can arrive out of order; I assume a refund always follows its payment.
- Candidate
ok, continue
- Candidate
- V6
When a check failed or showed a defect, did the candidate have it fixed and checked again?
PartlyThe candidate wrote fix the refund status, then run the whole list from the brief again
The refund status was fixed and checked again. The daily totals were left failing.
Show surrounding messagesHide surrounding messages
- Candidate
run the whole list from the brief against the service and show me which ones fail
- AI agent
Two of seven fail: the daily totals and the refund status.
- Candidate quoted
fix the refund status, then run the whole list from the brief again
- AI agent
Six of seven pass. The daily totals still fail.
- Candidate
Integrity signals Integrity signalAn observation with its evidence for you to consider. Integrity signals have no effect on the overall score.
Observations with their evidence for you to consider. They have no effect on the overall score.
- Content from outside the session
Evidence The file docs/runbook.md in the submission was never written or read in the session record.
This can also be honest manual work. It has no effect on the overall score.
How the candidate spent their time
The stages of the trial in order. Working time: 6 h 40 min.
- Understanding58 min 14%
- Planning34 min 9%
- Building3 h 8 min 47%
- Verifying1 h 36 min 24%
- Handing in24 min 6%
Limits of this evaluation
This evaluation is based on the submission and the recorded exchange with the AI agent. The evaluator read the submission without running it. The report shows excerpts from the session record with their surrounding context.
This sample shows detailed evidence for one dimension, Verification.