Skip to content

Reliability and Validity: The Two Questions to Ask Any Test

Reliability asks whether a test repeats itself. Validity asks whether the conclusion drawn from the score is justified. Two questions to ask of any test.

By Published Updated 5 min read

Reign Creative LLC makes IQ Test & Brain Training Games, which this article links to, so we have a commercial interest in it. How we write and check articles.

Key takeaways

  • Reliability asks whether a test gives a consistent answer across occasions, items and scorers. Validity asks whether the interpretation someone wants to draw from the score is actually supported.
  • A test can be highly reliable and still invalid for a given use. Consistency is necessary and not sufficient, and this is the confusion that lets a stable-looking number carry an unsupported conclusion.
  • The Standards for Educational and Psychological Testing, published jointly by AERA, APA and NCME in its 2014 edition, is the reference professionals use for how test development and use should be documented.
  • Documentation is the practical test. Published instruments carry technical manuals, norm samples and qualification requirements — Pearson restricts its 2024 Wechsler adult battery to qualification level C purchasers.
  • Our own app, IQ Test & Brain Training Games, includes an IQ-style test, a memory test and a reaction time test, and its listing cites no reliability or validity evidence for them; it calls the results estimates based on app activities, not professional assessments, so ask this article's two questions of them before reading anything into a result.

Two questions separate a measuring instrument from a quiz that produces a number. Reliability: would this test give a similar answer if it were repeated? Validity: is the conclusion someone wants to draw from the score actually supported by evidence? Ask both of anything that offers to tell you something about yourself, and most of what circulates online answers neither.

The order matters. Reliability is a precondition — a result that swings wildly cannot support any interpretation — but it is not sufficient. A test can be beautifully consistent and still be measuring the wrong thing, or be measuring the right thing and still not support the decision being made with it.

Reliability: does the test repeat itself?

Reliability is estimated in several ways, and each answers a slightly different version of "would we get this again".

Test-retest reliability compares scores from the same people on two occasions. The interval is a design decision with consequences: short intervals inflate agreement because people remember items, long ones risk the underlying attribute genuinely changing. Practice effects sit right in the middle of this, which is why a higher score on a second sitting is ambiguous evidence.

Internal consistency asks whether the items within a test agree with one another — whether people who get one item right tend to get the others right. It is cheap to compute from a single administration, which is why it is the figure most often quoted, and it says nothing about stability over time.

Alternate-form reliability compares two supposedly equivalent versions. This is why item banks matter: a test used repeatedly needs many interchangeable items, a problem now often solved by generating them systematically, as abstract reasoning item construction describes.

Inter-rater reliability applies wherever a human judges the response. It is irrelevant to a multiple-choice quiz and central to anything scored by an examiner.

Every reliability estimate implies a margin of error around an individual score. A score of 104 from an instrument with a meaningful error band is not distinguishable from 99 in any way that should change a decision, and reporting it as a bare number hides that.

Validity: is the interpretation justified?

The modern framing is the one that trips people up. Validity is not a property a test owns permanently; it attaches to a specific interpretation for a specific use. The same instrument can be well supported for one decision and unsupported for another.

Several kinds of evidence contribute. Content evidence asks whether the items represent the domain claimed — a numeracy test made entirely of percentage questions covers a narrow slice. Criterion evidence asks whether scores relate to an outcome they are claimed to predict. Construct evidence asks whether the pattern of relationships behaves as theory says it should: scores should correlate with measures of related things and not with measures of unrelated things.

The last of those is where popular instruments fail most often. A published critical review of emotional-intelligence measurement makes the problem concrete for one field, noting that "one clear challenge faced by early EI test developers was constructing emotion-focused questions that could be scored with objective criteria", and that "One disadvantage of self-report measures is that people are not always good judges of their emotion-related abilities". A questionnaire can be highly internally consistent while measuring self-perception rather than the ability it names. That case is developed further in IQ and emotional intelligence.

Fairness is the third question

Professional practice treats fairness as a technical requirement rather than a courtesy. The Standards for Educational and Psychological Testing, published in its 2014 edition by the American Educational Research Association with the American Psychological Association and the National Council on Measurement in Education, addresses professional and technical issues of test development and use across education, psychology and employment, and a companion volume in the same series is devoted specifically to fairness.

Test publishers document their own compliance machinery for the same reason. ETS describes its Standards for Quality and Fairness as providing "benchmarks of excellence" used throughout design, development and delivery to produce "technically fair, valid, and reliable tests", and states that its Office of Professional Standards Compliance "performs audits of all ETS tests". Whether or not you find a given publisher persuasive, the existence of published standards and an audit function is itself the signal — it is what a claim of validity looks like when someone intends it to be checkable.

Applying the two questions to anything

A short checklist does most of the work.

  1. Is there a technical manual? Not a marketing page — a document describing item development, scoring and evidence.
  2. Is there a norm sample, and who was in it? A score is a comparison, so the comparison group is part of the result.
  3. Are reliability coefficients published, with the method named?
  4. Is validity evidence tied to a stated use? "Validated" with no use specified is a slogan.
  5. Is administration restricted? Published cognitive batteries usually are. Pearson lists its 2024 Wechsler adult battery, which reports a Full Scale IQ derived from seven subtests, at qualification level C.

Applied to a free web quiz, this checklist usually terminates at step one. That is not a scandal; it is a category difference, and it is the same distinction that runs through who invented the IQ test and through what IQ tests actually measure.

What our own numbers are

IQ Test & Brain Training Games is our own app, built by Reign Creative, and neither its Google Play listing nor its app page cites any reliability or validity evidence. That matters, because the app is an educational game that also includes an IQ-style test, a memory test and a reaction time test, and its listing says the reaction time test measures speed and consistency. The listing also states that IQ test, memory test and other results are estimates based on app activities, not medical, psychological, clinical or professional diagnostic assessments, and the app keeps personal bests, accuracy, streaks, response speed and an assessment history.

Those figures are useful for exactly one purpose, which is comparing your own runs on the same mode at the same difficulty. Read across modes and they mislead — a 60-second sprint and a 75-second trainer are different tasks. Read them as a statement about your abilities and they mislead more. The scepticism that the evidence review in do brain training apps work applies to the category applies to us as well, and the specific case of a performance number dressed as an age estimate is dismantled in what a brain age score means.

The rest of this material sits in the IQ tests and reasoning hub, with the wider practice and memory articles in the education and brain category, and the modes described on the IQ Test & Brain Training Games app page.

Good to know

Frequently asked questions

What is the difference between reliability and validity?
Reliability is consistency: would the test produce a similar result if you sat it again next month, if a different set of equivalent items had been used, or if a different person had scored it? Validity is justification: does the evidence support the specific conclusion someone wants to draw from the score? A bathroom scale that reads three kilograms heavy every time is perfectly reliable and invalid as a measure of weight.
What is test-retest reliability?
It is the agreement between scores from the same people tested on two occasions. It is reported as a correlation, and the interval matters: too short and people remember the items, too long and the thing being measured may genuinely have changed. Practice effects are the main complication, because a second sitting is never the first sitting, and an apparent gain can be familiarity rather than change.
How can I tell whether a test is validated?
Look for documentation rather than a claim. A genuinely validated instrument has a technical manual, a described norm sample, published reliability coefficients, validity evidence for stated uses, and usually a restriction on who may administer it. Pearson, for example, sells its 2024 Wechsler adult battery at qualification level C. A product that asserts it is validated without publishing any of that has told you nothing.
Does IQ Test & Brain Training Games publish reliability or validity evidence?
No. Neither its Google Play listing nor its app page cites any reliability or validity evidence, even though the app now includes an IQ-style test, a memory test and a reaction time test. The listing describes those results as estimates based on app activities, not medical, psychological, clinical or professional diagnostic assessments. Without published evidence, treating a result as a measurement of a trait would be the error this article is written to prevent; use it to compare your own runs.

Sources

  1. Standards for Educational and Psychological Testing (2014 Edition) (opens in a new tab)American Educational Research Association, with APA and NCME — accessed
  2. Fairness Review Publications (opens in a new tab)ETS — accessed
  3. Wechsler Adult Intelligence Scale, Fifth Edition (WAIS-5) (opens in a new tab)Pearson Assessments — accessed
  4. The Measurement of Emotional Intelligence: A Critical Review of the Literature and Recommendations for Researchers and Practitioners (opens in a new tab)Frontiers in Psychology — accessed

About the author

— The independent app studio that builds the apps covered on this site and publishes every article here. On Google Play it publishes as Reign Collective Apps.

We use AI tools to help research, draft and edit our articles. Each article lists the sources its factual claims rely on, and details about our apps come from their live Google Play listings. No article on this site has been reviewed by an independent expert unless the reviewer is named on it.

Spotted an error? Tell us — how corrections are handled is set out in our editorial policy.

Keep reading