Key takeaways
- These are the limits of a properly built instrument used correctly, which is a different complaint from the failures of an unsupervised web quiz.
- Coverage is the first limit: a battery reports only on what it sampled, and abilities it never tested cannot appear in the score.
- Bias is checked item by item using differential item functioning, which compares groups at every level of ability rather than comparing averages. In one study of 2,773 questions and 13,694 candidates, only eight items — 0.29 per cent — showed notable effects.
- That result cuts both ways: it shows item-level unfairness can be small while group differences remain, which means a score difference is not by itself evidence about a test's fairness.
- Norms expire, and the correction convention is contested. A 2024 analysis of the WAIS-5 validity studies found about 1.2 IQ points per decade, well below the three-points-per-decade figure used in some intellectual disability and capital cases.
- Predictive validity is not decision utility. A score can relate to an outcome and still fail to improve the decision it is being used for.
Every limitation below applies to a well-constructed instrument, administered correctly by a qualified person, with current norms. This is not the case against bad web quizzes — that argument is made in are online IQ tests accurate. It is the harder and more useful question: what remains true when everything is done properly.
Five limits survive good practice, and each of them changes how a score should be read.
The instrument reports only on what it sampled
A battery is a fixed set of short tasks. Whatever it did not sample cannot appear in the result, no matter how important that ability is to the person's life.
This sounds obvious and is routinely ignored, because the reported number has no visible boundary on it. A composite score looks like a statement about a person. It is a summary of performance on seven or so subtests, chosen years earlier by a publisher for reasons of practicality as much as theory.
The same compression problem applies inside the test. A composite averages away a profile. Two people with identical composites can have opposite strengths across the domains that built them, and the profile is frequently the more informative object. Anyone reading a single number is reading the least detailed thing the assessment produced.
The number is an estimate, not a reading
Measurement error is not an occasional failure. It is a permanent property of testing, quantified rather than eliminated.
A score reported as a single integer implies precision the instrument does not claim, which is why a competent report gives a confidence interval. The consequence is sharpest at boundaries: a result near a cut point can sit on either side of it on a second administration without anything about the person changing, as discussed in percentile, standard score and raw score.
Bias is tested, and the testing is more interesting than the argument
"Are IQ tests biased?" is usually debated as though it were a matter of opinion. In psychometrics it is a technical property with an established method.
The method is differential item functioning. It asks whether test takers from different groups who are at the same measured ability level respond differently to a particular item. That qualifier is what makes it useful: comparing group averages tells you nothing about the item, because the groups may genuinely differ in the ability the item measures. Hope and colleagues make the methodological point directly in their 2018 study in BMC Medical Education — DIF compares performance at every level of the ability curve, where a simple average can conceal differences at the lower end, where candidates are competing to pass, and at the upper end.
Their result is worth knowing. Across 2,773 questions answered by 13,694 candidates, only eight items — 0.29 per cent — showed notable differential item functioning, and a blinded panel of clinician reviewers could identify no plausible explanation for the differences that were found. The authors concluded that group-level differences in performance could not be explained by unfairness at the question level.
Read that carefully, because it is easy to over-claim in both directions. It does not show that the assessment was fair in every sense; it shows that item-level unfairness was not the explanation for the group differences observed. Fairness at the item level and fairness of the whole enterprise are different questions, and a finding about one does not settle the other.
Test publishers treat this as process rather than argument. ETS states that it is committed to ensuring its tests and products are as free of bias as possible, that every product is formally evaluated against criteria covering offensive content, stereotypical representation, and racial, ethnic, gender and socioeconomic bias, and that staff involved in creating and evaluating questions receive extensive training in its fairness review process.
Norms age, and the correction for it is contested
A score is a rank against a particular sample of people at a particular time. Raw performance drifts across generations, so instruments are renormed — and until they are, the score a person earns depends partly on how old the norms are.
The standard correction assumed roughly three points per decade. Winter, Trudel and Kaufman's 2024 analysis in the Journal of Intelligence unsettles it. Examining two counterbalanced validity studies from the WAIS-5 Technical Manual — 186 adolescents and adults aged 16 to 90, and 98 sixteen-year-olds — they found the same unexpected result in both: approximately 1.2 IQ points per decade.
The stakes are not academic. The authors note that Flynn-effect adjustments feature in intellectual disability determinations under contemporary diagnostic guidance, and in capital cases where those determinations govern eligibility. A correction of three points where the true drift is 1.2 is an over-correction with consequences. They raise the question rather than resolving it, and the unresolved state is itself a limitation worth naming: the standard fix for an ageing norm is currently uncertain. The underlying mechanism is covered in why the average IQ score is always 100.
Predicting an outcome is not the same as improving a decision
The most under-appreciated limit is about use rather than measurement.
A 2026 framework paper in the Journal of Intelligence separates incremental validity — the predictive gain from adding a cognitive measure to an existing evidence base — from decision utility, whether using the score actually improves the decision being made. Cognitive measures relate moderately to strongly with academic outcomes, and often add predictive value at transition points. But that gain varies with range restriction, grading differences and unequal opportunity; where a target outcome is rare, false positives multiply; where places are capped, the operational meaning of a prediction changes; and a measure can improve average prediction while worsening outcomes for a subgroup.
The authors' conclusion is a discipline worth borrowing: testing should be conditional and sequential, used only where it can plausibly change what happens next. A score that cannot change a decision is not neutral — it consumes time, money and attention, and it confers unearned authority on whatever is done anyway.
What the limits mean for our own app
IQ Test & Brain Training Games is our own app, built by Reign Creative, and it includes an IQ-style test, so the limits above apply to it directly.
Its IQ-style test covers numerical reasoning, logic, patterns, vocabulary, working memory, spatial reasoning and processing speed, and the app also has a memory test, a reaction time test and a personality questionnaire. Its Google Play listing states that these results are estimates based on app activities, not medical, psychological, clinical, educational-placement, employment or professional diagnostic assessments. The listing cites no norm sample, technical manual, reliability evidence or validity studies, and a test taken on a phone has no standardised administration, so a result from it carries every limit described above and should be read as an estimate from the app's own questions, never as a score to base a decision on.
Its games report accuracy, streaks, response speed and personal bests against your own history. Those are practice statistics. Keeping them, and the test estimates, clearly apart from standardised psychological assessment is not a disclaimer bolted on afterwards; it is the only honest way to read them. Our broader position on evidence in this category is set out in do brain training apps work.
What a careful reader takes away
A good test, used well, produces a bounded estimate of performance on a defined sample of tasks, with a known margin of error, against a comparison group of a known vintage. That is genuinely valuable and it is much less than the word "IQ" implies in ordinary conversation.
For the construct behind the instrument, see what IQ tests actually measure. The rest of the series sits in the IQ tests and reasoning hub, with practice-focused material across the education and brain training section.
Good to know
Frequently asked questions
What are the main limitations of IQ testing?
Are IQ tests biased?
Does a group score difference prove a test is unfair?
How should you read the results in IQ Test & Brain Training Games?
Sources
- Using differential item functioning to evaluate potential bias in a high stakes postgraduate knowledge based assessment (opens in a new tab)BMC Medical Education (Hope, Adamson, McManus, Chis & Elder, 2018), via PubMed Central — accessed
- Wait, Where's the Flynn Effect on the WAIS-5? (opens in a new tab)Journal of Intelligence (Winter, Trudel & Kaufman, 2024), via PubMed Central — accessed
- From Incremental Validity to Decision Utility: A Framework for Intelligence Testing in Education (opens in a new tab)Journal of Intelligence, via PubMed Central — accessed
- Fairness | About ETS (opens in a new tab)Educational Testing Service — accessed
About the author
Reign Creative LLC — The independent app studio that builds the apps covered on this site and publishes every article here. On Google Play it publishes as Reign Collective Apps.
We use AI tools to help research, draft and edit our articles. Each article lists the sources its factual claims rely on, and details about our apps come from their live Google Play listings. No article on this site has been reviewed by an independent expert unless the reviewer is named on it.
Spotted an error? Tell us — how corrections are handled is set out in our editorial policy.