Chapter 6: Assessing the Validity and Reliability of Measures
(F) Day of the week: Tuesday Class: IS303 Created Time: December 8, 2020 12:47 PM Database: Class Notes Database Date: December 8, 2020 12:47 PM Days Till Date: Passed Last Edited Time: June 9, 2021 10:42 AM Type: Presentation Notes, Reading Notes
Introduction
In translating mental abstractions into empirical ideas we need to look at problems of accuracy during translation.
- Validity: the extent to which it is measuring what is claimed to measure
- Reliability: ability to deliver consistent or stable results
1. Measurement Validity
To test whether a variable measure what it claims to measure:
Ordering from least to more rigorous assessment
1.1. Face Validity
The assessment of validity by seeing if it looks right (logical) on the surface
- Lack Validity
It’s the minimum a research should do. It is subjective, different people have different assessment
Ex: You cannot measure people’s happiness by whether people stomp their feet.
1.2. Content Validity
Assesses if conceptual and nominal definition over lap each other:
- Conceptual Definition: theoretical clarification of concepts
- Nominal Definition: steps involved in documenting the concept
If the purpose of the question isn’t the same as the theory in question, its not content valid
How would you measure if it related or overlap, is it subjective?
1.3. Criterion Validity
Assessment using empirical evidence
-
Predictive Validity: using empirical data/evidence to create predictions (picking the right variables)
If the outcome is the same as the prediction it’s valid
-
Concurrent Validity: using existing valid measure to demonstrate accuracy of another measure of the same variable
How would you do that? Ex?
1.4. Construct Validity
Demonstrating accuracy by producing the same results as theoretically based hypotheses or predictions
Gathering empirical data that comes to the same result to the theory
Ex:
2. Reliability Check
The ability to produce or yield consistent or stable results
2.1. Test-Retest
Simplest technique
Assessing consistency by measuring twice and looking for similarities and difference on the same subject
Same results = reliable
The importance of time delay:
- Long Time Delay: subjects could change due to time since last test
- Short Time Delay: subjects may be able to remember the answers they picked
2.2. Multiple-Forms Test
Assessing consistency by using two forms for the same subject asking for the same information
Changing the way they ask questions
Asking for age:
- What is your age?
- In what your were you born?
Problem: changing the wording of questions might change the meaning or measurement of the question itself
2.3. Split-Half Test
Assessing consistency splitting measure into two halves of a test to see if both groups of answers correlate to the same result
Cronbach’s Alpha: the coefficient that is frequently used to report reliability
- From 0 - 1
-
0.7 is considered reliable
3. Measurement Errors
3.1. Noise Bias
Measurement is the estimate of the true value
There are many measurement errors
3.2. Noise
The error that has no direction or reason
Unintended error due to random chance
- With many measures noise will cancel each other out and will not be significant
3.2.1. Research Subject
Noise from the subjects’ incompetency in answering:
- Subject is too tired to provide normal competent answers
- Subject may be too young to provide accurate answer
- Subject is inattentive, or not concentrating
3.2.2. Human Diversity
different people will answer differently, if the sample group is too diverse it’s very noisy
3.2.3. Poorly Designed Questions
Random Questions that leads to random answers, not relating to the point of the research
3.3. Bias
Error that has a pattern or consistent direction (overly or under estimate)
Sources of Bias
- Expectation: Researchers have their theory in mind and might use vague/ambiguous data to confirm their biased theory
- Biased Tool: can have bias towards one answer or another
- Threatening Condition: Subjects might answer questions that researchers might want or expect
Reduce bias: by introducing noise as it can cancel each other out
Noise is better than Bias
- Bias might be malicious in nature and intentional, hard to correct
- Noise isn’t intentional and will cancel itself out in time and scale