PairTrial.

Sample report

Jonas Erikson

Task
Payment events
Opening
Backend engineer, payments
Harness HarnessThe app the candidate used to work with an AI agent, such as Claude Code, Codex or OpenCode. The choice of harness does not affect the score.
Claude Code
Started
2026-11-13 10:02
Submitted
2026-11-14 15:48
Working time
6 h 40 min
Model cost Model costThe cost of the AI model used during the trial.
$11.80

Overall score

Overall score, 52 out of 100

What the overall score covers

The overall score reflects the six dimensions with enough evidence. Decision making is excluded.

Evaluator's summary

Jonas asked the agent how the payment system retries events before any code was written. He had a repeated event checked and, when the last check showed two failures, had the refund status fixed and checked again. The daily totals for finance were still failing at the submission, and he accepted the agent's assumption that a refund always follows its payment.

A person makes the final decision. A gap of a few points between candidates is too small to interpret.

Dimensions

Each dimension has four levels. The evaluator, an AI agent, uses the evidence to choose a level for each dimension.

DimensionLevel LevelOne of four descriptions of what the evidence shows for a dimension. Each level carries points.Weight WeightHow much a dimension counts towards the overall score.Evidence
How the candidate worked
Understanding of the problemWorking understanding Level 3 of 4Weight 15%Evidence unavailable in this sample
Analytical thinkingCases worked through Level 3 of 4Weight 20%Evidence unavailable in this sample
Decision makingNot enough evidence Not enough evidenceThe session record and the submission provide too little evidence to choose a level. This dimension is excluded from the overall score.Weight 10%Excluded from the overall score
Own ideasAdjustments only Level 2 of 4Weight 10%Evidence unavailable in this sample
VerificationChecked and followed through Level 3 of 4Above level 2: Jonas read what the checks showed and had a failing one fixed and checked again. Below level 4: one failing check was left as it was, and the agent's assumption about refunds went unquestioned.Weight 15%See the evidence for Verification
What the candidate submitted
Meets the requirementsPartly met Level 2 of 4Weight 15%Evidence unavailable in this sample
Quality of executionUsable after rework Level 2 of 4Weight 15%Evidence unavailable in this sample

Verification

Level 3 of 4 · Checked and followed through

Rationale

Level 3 and not 2: Jonas asked for checks, read what they showed and had a failing one fixed and checked again. Level 3 and not 4: the checking began once the service existed, one failing check was left as it was, and the agent's assumption about refunds went unquestioned.

Observations ObservationA specific question about the trial, answered using a quote from the session record or the submission.

The evaluator uses all the available evidence to choose a level. Open a quote to see its context.

  1. V5

    Was a case other than the basic one checked at the candidate's request?

    Yes
    The candidate wrote
    send the same payment.captured twice, ten ms apart, and show me the rows
    Show surrounding messagesHide surrounding messages
    1. AI agent

      The handler stores each event and updates the order. Shall I move on to refunds?

    2. Candidate quoted

      send the same payment.captured twice, ten ms apart, and show me the rows

    3. AI agent

      One row for the payment; the second event was recognised as a repeat and ignored.

  2. V4

    Did the candidate catch a defect or a false statement of the agent?

    No
    The AI agent wrote
    Events can arrive out of order; I assume a refund always follows its payment.

    The AI agent wrote this. The candidate replied "ok, continue".

    Show surrounding messagesHide surrounding messages
    1. Candidate

      now handle events that arrive in the wrong order

    2. AI agent quoted

      Events can arrive out of order; I assume a refund always follows its payment.

    3. Candidate

      ok, continue

  3. V6

    When a check failed or showed a defect, did the candidate have it fixed and checked again?

    Partly
    The candidate wrote
    fix the refund status, then run the whole list from the brief again

    The refund status was fixed and checked again. The daily totals were left failing.

    Show surrounding messagesHide surrounding messages
    1. Candidate

      run the whole list from the brief against the service and show me which ones fail

    2. AI agent

      Two of seven fail: the daily totals and the refund status.

    3. Candidate quoted

      fix the refund status, then run the whole list from the brief again

    4. AI agent

      Six of seven pass. The daily totals still fail.

Integrity signals Integrity signalAn observation with its evidence for you to consider. Integrity signals have no effect on the overall score.

Observations with their evidence for you to consider. They have no effect on the overall score.

How the candidate spent their time

The stages of the trial in order. Working time: 6 h 40 min.

  1. Understanding58 min 14%
  2. Planning34 min 9%
  3. Building3 h 8 min 47%
  4. Verifying1 h 36 min 24%
  5. Handing in24 min 6%

Limits of this evaluation

This evaluation is based on the submission and the recorded exchange with the AI agent. The evaluator read the submission without running it. The report shows excerpts from the session record with their surrounding context.

This sample shows detailed evidence for one dimension, Verification.