No Qualified Employee Left Behind --- How Scientific Testing Works

Reliability and validity are two fundamental concepts associated with the use of tests or assessments for pre-employment screening. Unfortunately, these concepts can be quite difficult to understand and are not widely addressed by many tests and assessment vendors.

Measuring Reliability

In general, there are three ways to measure or understand the reliability of a test or assessment.

1. The first, internal consistency, refers to whether the questions in a test or assessment measure the same attribute in the same way. For example, a math test would not reliably measure math ability if it also includes geography questions. 
2. A second way to understand reliability is the degree to which a test or assessment is consistent across time, or test-retest reliability. The most common way to estimate a test’s reliability over time is to use the test-retest method, where an assessment is administered twice to the same group of people. 
3. The third way to understand reliability is referred to as inter-rater reliability. In this sense, reliability is about the relative level of agreement between raters or with one rater over time.

Understanding Validity

In addition to reliability, it is important to understand validity in your test and assessment. Validity is generally defined as the extent to which a test measures what it claims to measure. Validity can also refer to a process of finding evidence that allows us to interpret the results of a test over time. Estimating the validity of a test helps us to understand what the results mean.

Much like reliability, there are three different strategies for collecting evidence of validity on a test or assessment.

1. The first, content-related validity, refers to evidence that questions on the test or questionnaire actually reflect aspects of the attribute being measured. Evidence of content-related validity can be built into the test by creating a process for test construction, whereby one can document that an effort was made to find out about the attribute and to include only those questions that measured the attribute. 
2.  The second method, criterion-related validity is assessed when the test or questionnaire is used to predict something else, such as job performance. Evidence of criterion-related validity is gathered in two ways: predictive validation, or that the test in question is administered and the criterion is assessed at a later point in time, and concurrent validation, or that the test and criterion are assessed at the same time. 
3.  And finally, construct-related validity can be thought of as the “big picture” validity. Evidence of construct-related validity is accumulated over time, and is demonstrated through convergent validity, or the degree to which the test in question is correlated with other things, and by divergent validity, or the degree to which the test results are not correlated with events, behaviors or other tests that should not be related.

Understanding how reliability and validity relate to your assessment or test can greatly improve the effectiveness of their results.

Chris Wright is the founder, President and CEO of Reliant (www.reliantlive.com). Dr. Wright leads the development of Reliant's Strategic Talent Management products and content.