Reliability and Validity

Reliability and validity are both concepts that are not new to me.  However, after reading this week’s chapters, I realized how much I really did not know about them!  I also realized how relevant they are to my job at this point.

When reading about true ability vs. observed score in the Reliability chapter, I immediately thought about a 7^th grade student that I have. He is very emotionally immature and has low confidence in his abilities. He does not like school at all and states this often.  From working with him, one would think that he has low ability based on the way he performs in class both in motivation/task completion and his grades. However, he was evaluated at the beginning of this school year, and his full-scale IQ was a 95.  His scores in all areas of reading, written expression, and math were average on the KTEA-II.  This seems like a great example of true ability vs. an observed score, with the observed score being his performance in class.  The two are like polar opposites for this student.

When reading about test-retest as a method of estimating reliability, I wondered if the test that is given is the exact same test, or if the test is just a similar one.  When reading about simultaneous administration on page 89 (Drummond & Jones, 2010), I gather that the test-retest measure uses the same assessment.  I wonder, then, how long they have testers wait before re-taking the test?  Some people have extraordinary memories, and a curiosity that might warrant them to seek answers to particular questions they were asked on the test. If they sought out information or practice on certain questions or skills themselves, can the test-retest measure really show reliability for some groups of people?  Related to re-test measures, the book stated that a limitation is that many tests do not have alternate forms.  When reading about this, I thought about my school’s re-take policy, which was implemented last year.  We are required to allow students to re-take tests when they go through a particular process.  We were told as teachers to have an alternate form of each test so that students who re-take will take the alternate.  I agree with the book: this is extremely time-consuming!  So time-consuming that I wonder just how reliable some of the teacher-made alternate tests are? 

When reading about predictive validity and the example of SAT scores as a predictor with college success, I was thinking, “Yeah, but SAT scores has nothing to do with personal choices about whether to focus more on classes or partying.”  I was not surprise to then read that while it can predict academic success, it’s a poor predictor of morality (Drummond & Jones, 2010).  With anything there could be confounding variables that could skew the correlation between two items.  Also related to confounding variables was the thought that, in group differentiation studies, there is an expectation that children with ADHD would score lower than children in the standardization sample. The special education teacher in me wonders if a child’s environment is modified to limit distractions when taking this test.  Are they given extended time to complete the test?  Do they have ADHD paired with a learning disability?  These are all variables that I think would need to be considered.  Lastly, relating to confounding variables were my thoughts on criterion measures as uncontaminated.  It makes sense that a measure should not be influenced by any external factors.  I am curious about how achievement or IQ testing could be affected by the fact that a student might be an English Language Learner.  All students that I have had whom have undergone evaluations have been fluent in English.  I guess I have never asked a school psychologist how they would evaluate a student who speaks minimal English, especially if it is a more obscure language that they speak.  Is the test interpreted?  Are there forms of tests in students’ native languages and psychologists who specialize in administering such assessments?  This could certainly affect not only the level of “contamination,” but also the reliability of the scores.

The more I learn in this class, the more questions I have, and the less I realize I know.  I am left wondering: How many of the concepts in this textbook will be concepts that we will need to understand and use on a regular basis as school counselors?

Drummond, R. J. and Jones, K. (2010). Assessment procedures for counselors and

            helping professionals. (7th ed.). Upper Saddle River, NJ: Pearson.