AI vs Human-Graded Language Tests

When you test a candidate's English, something has to turn their answers into a score: a trained human rater, an automated system, or a mix of both. The skill being measured is the same. The evidence you get, the cost, and the time it takes are not. This article compares the two approaches for hiring, sets out what the testing standards ask of automated scoring, and suggests when to use each.

Two ways to score open-ended answers

Multiple-choice questions can be marked against a key, so the method makes little difference. The choice matters for speaking and writing, where there is no single correct answer.

• Human grading: trained raters apply published criteria to each response. IELTS, for example, has examiners score Writing and Speaking on four equally weighted criteria each (IELTS scoring in detail).
• Automated scoring: a model scores each response against defined criteria. The same rules are applied to every response.

What human grading does well, and where it strains

An experienced rater can judge things that are hard to write down as rules: a response that is loose in grammar but very effective, a strong accent that never gets in the way, or a candidate who makes up for limited vocabulary with good strategy. For a few high-stakes decisions, that judgement is worth paying for.

The costs show up with volume:

• Raters do not always agree. Two qualified raters can score the same borderline response differently. Providers control this with training, calibration, and double-marking, but it does not disappear.
• Cost grows with volume. Each response takes rater time, so testing more candidates needs more raters.
• Results wait for a rater. The more responses there are, the longer the queue.

What automated scoring changes

• Consistency. The same model applies the same rules to the first candidate and the five-hundredth. There is no difference between an easy and a hard marker, and no end-of-day drift.
• Scale. Testing ten candidates or a thousand needs the same setup.
• Cost per test. Software does the scoring, so testing every applicant becomes realistic.
• No wait for a rater. Results do not depend on someone being available.

The open question is whether the machine scores what a human rater would. That is an empirical question, and there is published evidence. In an ETS research report on automated scoring for the TOEFL Junior Comprehensive Writing and Speaking Test, machine scores correlated with human scores at r = .81 for speaking and r = .83 for writing, compared with r = .89 and r = .90 between two human raters. In that study, the machine came close to human agreement without fully reaching it.

What the testing standards ask of automated scoring

The Standards for Educational and Psychological Testing, published by AERA, APA and NCME and available open access, have a standard specifically for this. Standard 4.19 says that when automated algorithms score complex responses, the characteristics of responses at each score level should be documented, along with the theoretical and empirical basis for using the algorithm. The comment adds that automated scoring should have research support, such as agreement rates with human scorers, and evidence that it does not introduce systematic bias against some subgroups.

That gives a hiring team three concrete questions to ask of any AI-scored test:

1. What criteria does the system score against?
2. What evidence is there of agreement with human scorers?
3. What evidence is there that it treats different groups of candidates fairly?

Screening versus final decisions

A language test can do two different jobs.

Screening filters many candidates to a shortlist against one threshold, such as "B2 or above". Consistency, cost, and speed matter most, and automated scoring fits well.

High-stakes final decisions concern a few candidates, often borderline ones, where a person reviewing the evidence adds confidence. Here the extra time and cost of human judgement can be worth it.

A common pattern is to screen every applicant with an automated test, then use an interview or a human-reviewed task for the small number of borderline or final candidates.

Where AI is used in the International English Test

The method is set out on our "How we score" page:

• Grammar, vocabulary, reading and listening are multiple-choice and marked against a verified key, with no AI.
• Speaking and writing are scored by AI language models from Google (Gemini) and Anthropic (Claude). Speaking is scored on fluency, coherence, grammar, vocabulary and pronunciation; writing on grammar, vocabulary, content and organisation.
• Each criterion is scored from 0 to 100, the scores are averaged with equal weight, and the CEFR level is calculated in code from a fixed threshold table. The model does not choose the level.

You can see the AI-scored parts from the candidate's side on the speaking test and the writing test. The multiple-choice English level test does not use AI scoring. For using the tests across a team, see English assessment tests for companies.

Key points

• Human grading brings judgement on unusual answers; automated scoring brings consistency and scale.
• Neither is reliable by default: ask for evidence of agreement with human scorers.
• Standard 4.19 of the testing standards asks for agreement evidence and checks for bias against groups of test takers.
• A practical pattern is automated screening for everyone, with human review for the few borderline or final candidates.