8

In the book Doing research in the real world by David E. Gray, there is a section on experiment design. When discussing validity and reliability, the author defines “criterion validity” as

This is where we compare how people have answered a new measure of a concept, with existing, widely accepted measures of a concept.

and a little later, in the “Reliability” part, there is a subheading “Equivalence”, which says

Another way of testing the reliability of an instrument is by comparing the responses of a set of subjects with responses made by the same set of subjects on another instrument (preferably on the same day).

So if I got this right, we are both times measuring if there is a difference between the answer on our new instrument and another, existing instrument.

Is there a practical difference between the two concepts, or only a philosophical one? And whether practical or philosophical, what is the actual difference?

Update The author discusses validity and reliability in general, then lists 7 different aspects of validity and 5 different aspects of reliability. "Criterion validity" and "reliability equivalence" are only one type of each, respectively. Please consider in your answer that this question is not about validity vs. reliability in general, but only about these two specific aspects.

  • 4
    This question appears to be off-topic because it is about experiment design and not academia. – StrongBad Feb 7 '14 at 12:39
  • 3
    The list of topics doesn't say that research methodology is off-topic, and doing proper research is part of being in academia. Is there an existing policy against "how is research done" questions, and where can I read about it? I don't personally see a good argument why being a good university teacher should be on topic but being a good university researcher should be off topic. – rumtscho Feb 7 '14 at 12:53
  • 2
    @StrongBad is there any SE website where you would see this question fit better than AC.SE? Just curious. Cross Validated does not seem to be the place (in case). – dgraziotin Feb 7 '14 at 12:54
  • 3
    Anyhow, I often feel we spend too much time putting questions into silo. Answer submitted. – Penguin_Knight Feb 7 '14 at 13:12
  • 3
    To anybody interested, I posted a question on meta, meta.academia.stackexchange.com/questions/779/… – rumtscho Feb 7 '14 at 13:31
7

Criterion validity concerns with measuring the right thing. For instance, GPA is likely to have criterion validity to measure a student's academic understanding. While the change in weight in the last semester has much less criterion validity to measure the same trait. Basically, if the measurement you use and the trait you want to measure has a high correlation, then there is likely criterion validity.

Reliability concerns mostly with measuring the thing right. For instance, if GPA can measure a student's academic understanding, and percent attendance can also measure a student's academic understanding, then GPA and percent attendance should correlate, aka, they are reliable. Before subjected to reliability assessment, the tests are usually checked if they are criterion-valid. However, it's possible to have two tests that are highly correlated (reliable) but invalid. Such as using dietary fat intake and serum lipid to predict a college graduate's earning potential.

Notice that there a few different types of reliabilities, the one you cited is more about alternate forms reliability, there are also test-retest reliability and inter-rater reliability, etc.

Practically, they are not interchangeable. Validity happens between the true trait (or behavior) and the measurements. Reliability happens between two measurements (or modes/instances of measurement.)

0

Validity is comparing the new results with the existing literature, without doing extra experiments.

Reliability is comparing the new results with some extra experiments that you carry on with some other settings/devices.

I agree with @StrongBad that this question is off-topic, but there is no SE site on research in general and I think this question is quite interesting.

  • I don't know where this view comes from, but it is certainly not the one the book author had in mind. In another subchapter, he insists that external validity has to be shown through comparison with a series of new studies. As for the off-topic suggestion, please share your view on Meta, meta.academia.stackexchange.com/questions/779/…. – rumtscho Feb 9 '14 at 22:58
0

Slightly off-topic as the terms you ask for are more specific, but:

Unfortunately, there is some ambiguity as to what exactly is meant by different terms in this quality control/validation/method context.

E.g.

  • in machine learning the "validation set" is often used to optimize parameters - as opposed to proving whether or not the model "does its job" (a shortened version of one definition of validity). The latter is measured with the "test set" (again, in my opinion, a rather ambiguous name).

  • The Handbook of validation in analytical chemistry spends several pages to compare and discuss differences between several definitions given in literature and norms specific to the field of analytical chemistry. The bottom line of these definitions is that in analytical chemistry, validity is not only about measuring the right thing (as @Penguin_Knight nicely explained), but also about measuring the right thing right.

I'd therefore recommend that you state what you are speaking about rather than relying on these terms.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.