In the book Doing research in the real world by David E. Gray, there is a section on experiment design. When discussing validity and reliability, the author defines “criterion validity” as

This is where we compare how people have answered a new measure of a concept, with existing, widely accepted measures of a concept.

and a little later, in the “Reliability” part, there is a subheading “Equivalence”, which says

Another way of testing the reliability of an instrument is by comparing the responses of a set of subjects with responses made by the same set of subjects on another instrument (preferably on the same day).

So if I got this right, we are both times measuring if there is a difference between the answer on our new instrument and another, existing instrument.

Is there a practical difference between the two concepts, or only a philosophical one? And whether practical or philosophical, what is the actual difference?

Update The author discusses validity and reliability in general, then lists 7 different aspects of validity and 5 different aspects of reliability. "Criterion validity" and "reliability equivalence" are only one type of each, respectively. Please consider in your answer that this question is not about validity vs. reliability in general, but only about these two specific aspects.

  • 4
    This question appears to be off-topic because it is about experiment design and not academia.
    – StrongBad
    Feb 7, 2014 at 12:39
  • 3
    The list of topics doesn't say that research methodology is off-topic, and doing proper research is part of being in academia. Is there an existing policy against "how is research done" questions, and where can I read about it? I don't personally see a good argument why being a good university teacher should be on topic but being a good university researcher should be off topic.
    – rumtscho
    Feb 7, 2014 at 12:53
  • 2
    @StrongBad is there any SE website where you would see this question fit better than AC.SE? Just curious. Cross Validated does not seem to be the place (in case).
    – user7112
    Feb 7, 2014 at 12:54
  • 3
    Anyhow, I often feel we spend too much time putting questions into silo. Answer submitted. Feb 7, 2014 at 13:12
  • 3
    To anybody interested, I posted a question on meta, meta.academia.stackexchange.com/questions/779/…
    – rumtscho
    Feb 7, 2014 at 13:31

3 Answers 3


Criterion validity concerns with measuring the right thing. For instance, GPA is likely to have criterion validity to measure a student's academic understanding. While the change in weight in the last semester has much less criterion validity to measure the same trait. Basically, if the measurement you use and the trait you want to measure has a high correlation, then there is likely criterion validity.

Reliability concerns mostly with measuring the thing right. For instance, if GPA can measure a student's academic understanding, and percent attendance can also measure a student's academic understanding, then GPA and percent attendance should correlate, aka, they are reliable. Before subjected to reliability assessment, the tests are usually checked if they are criterion-valid. However, it's possible to have two tests that are highly correlated (reliable) but invalid. Such as using dietary fat intake and serum lipid to predict a college graduate's earning potential.

Notice that there a few different types of reliabilities, the one you cited is more about alternate forms reliability, there are also test-retest reliability and inter-rater reliability, etc.

Practically, they are not interchangeable. Validity happens between the true trait (or behavior) and the measurements. Reliability happens between two measurements (or modes/instances of measurement.)


Validity is comparing the new results with the existing literature, without doing extra experiments.

Reliability is comparing the new results with some extra experiments that you carry on with some other settings/devices.

I agree with @StrongBad that this question is off-topic, but there is no SE site on research in general and I think this question is quite interesting.

  • I don't know where this view comes from, but it is certainly not the one the book author had in mind. In another subchapter, he insists that external validity has to be shown through comparison with a series of new studies. As for the off-topic suggestion, please share your view on Meta, meta.academia.stackexchange.com/questions/779/….
    – rumtscho
    Feb 9, 2014 at 22:58

Slightly off-topic as the terms you ask for are more specific, but:

Unfortunately, there is some ambiguity as to what exactly is meant by different terms in this quality control/validation/method context.


  • in machine learning the "validation set" is often used to optimize parameters - as opposed to proving whether or not the model "does its job" (a shortened version of one definition of validity). The latter is measured with the "test set" (again, in my opinion, a rather ambiguous name).

  • The Handbook of validation in analytical chemistry spends several pages to compare and discuss differences between several definitions given in literature and norms specific to the field of analytical chemistry. The bottom line of these definitions is that in analytical chemistry, validity is not only about measuring the right thing (as @Penguin_Knight nicely explained), but also about measuring the right thing right.

I'd therefore recommend that you state what you are speaking about rather than relying on these terms.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .