Last time in this space, I offered a brief field guide to testing: what varieties of test are appropriate for hiring, what exactly they measure, what they can be expected to do, and what happens when they go astray. This month I’d like continue the discussion by offering a sort of “buyer’s guide.”

Unfortunately, buying a hiring test is a bit like shopping for a used car. There are few businesses where incomplete information, misleading claims, and plain old bad advice are quite so prevalent. Almost everyone has a different recommendation that they claim is the very best thing for the job. And since many tests currently used to make hiring decisions were developed in the infancy of selection science, for a variety of totally unrelated purposes, that cream puff has often literally been sitting on the lot for a few decades. Plus, there’s never any warranty.

So instead of an attempt to catalogue the wild world of test vendors (that’s an entire business in itself), what follows is a guide to decoding the sales pitches you’re likely to hear in the testing marketplace. To be fair, people who sell tests are seldom test experts themselves, so they are plagued with a mental condition in which they don't know what they don't know. This means that their claims can be very sincere, sound highly enthusiastic and plausible, but still be completely wrongheaded.

Test Vendor Claims vs. Reality
Claim #1: "Test users don't really need to worry about lawsuits."
Reality: Sure, the numbers of cases that get settled by court order are relatively small and most have to do with wrongful termination. But losses in court are just a fraction of the real cost. Win, lose, or draw, each of 8,000 or so cases filed annually take time, legal expense, investigation, depositions, reporting – not to mention ill will and bad publicity.

Claim #2: "Pencil and paper tests are accurate predictors of job performance." Reality: A summary* of the validity of common selection procedures shows the following levels of predictability:

• Traditional interviews: 4%
• Personality tests: 4%
• Motivation tests: 4%
• Mental ability tests: 25%
• Content-valid simulations: 64%

Pencil and paper tests can be useful tools, but hundreds of controlled studies have shown they are among the least accurate hiring tools. (*Adapted from a meta-analysis conducted by Hunter and Hunter, Psychological Bulletin, Vol. 96, 1984. Percentages have been rounded. "Predictability %" refers to the explained variance. This data was updated in 1998, but does not break out data for personality and motivation tests separately.)

Claim #3: "Test users can use the same test for everyone regardless of the job."
Reality: There are three kinds of validity (i.e., ways to correlate test scores with job performance). Here is some advice from the always-relevant Uniform Guidelines on Employee Selection Procedures: "There should be a review of job information to determine measures of work behavior(s) or performance that are relevant to the job or group of jobs in question...to the extent that they represent critical or important job duties, work behaviors or work outcomes as developed from the review of job information."

"A criterion-related validity study should consist of empirical data demonstrating that the selection procedure is predictive of or significantly correlated with important elements of job performance... A content validity study should consist of data showing that the content of the selection procedure is representative of important aspects of performance on the job for which the candidates are to be evaluated... A construct validity study should consist of data showing that the procedure measures the degree to which candidates have identifiable characteristics which have been determined to be important in successful performance in the job for which the candidates are to be evaluated."

Sorry about all the "governmentese" language, but the Guidelines are important – even if they were written by bureaucrats. Now, you decide: should you use the same test(s) for every position just because someone tells you it's okay? Have you conducted your own criterion, content, or construct validity study? If not, what level of hiring risk are you willing to accept?

Claim #4: "The candidate cannot cheat on this test." Reality: This is just too silly to believe. You can’t fake a typing test, sure, but do you really think applicants are totally honest on a test that cannot be verified? Any self-respecting test developer designing a hiring test will include a truthfulness factor to minimize (not eliminate) faking. But even that is not perfect – so look for “a truthfulness scale” to help account for lying.

Claim #5: "The EEOC has approved such-and-such a test."
Reality: The EEOC is not a certifying body. It investigates specific discrimination claims to see if they have merit. The only “approval” even implied by the EEOC would be of a test that was challenged and found non-discriminatory in a specific application for a specific job in a specific company.

Claim #6: The EEOC "likes" instruments that measure mental constructs.
Reality: Constructs are deep-seated, unobservable, underlying mental traits that a test developer will use to describe job behavior. Mental constructs include things like personality, values, attitudes, satisfaction, etc. Here is what the Guidelines say about using construct-oriented tests, "There is at present a lack of substantial literature extending the [construct validity] concept to employment practices...users should be aware that the effort to obtain sufficient empirical support for construct validity is both an extensive and arduous effort involving a series of research studies, which include criterion related validity studies and which may include content validity studies."

Claim #7: "This test reduces turnover by XX%, increases productivity by YY%” (It also gets 65 mpg and never needs an oil change).
Reality: The Guidelines state, "Under no circumstances will the general reputation of a test or other selection procedures, its author, its publisher, or casual reports of its validity be accepted in lieu of evidence of validity...this includes validity based on a procedure's name or descriptive labels; all forms of promotional literature; data bearing on the frequency of a procedure's usage; testimonial statements and credentials of sellers, users, or consultants; and other non-empirical or anecdotal accounts of selection practices or selection outcomes." In other words, “your mileage may vary.” Ask for a thorough test drive.

Claim #8: "You don't need professional help to build a valid selection system."
Reality: The Guidelines state, "Professional supervision of selection activities is encouraged but is not a substitute for documented evidence of validity. The enforcement agencies will take into account the fact that a thorough job analysis was conducted and that careful development and use of a selection procedure in accordance with professional standards enhance the probability that the selection procedure is valid for the job."

Claim #9: "Here is an attorney's site recommending my test."
Reality: See Claim #7 above.

Claim #10: "The test vendor is responsible for the validity of the test."
Reality: See Claim #7 above.

Claim #11: "Employment agencies are exempt."
Reality: The Guidelines state, "The use of an employment agency does not relieve an employer or labor organization or other user of its responsibilities under Federal law to provide equal employment opportunity or its obligations as a user under these Guidelines."

Conclusion
Congratulations! You survived the onslaught of dull text and technical standards (you might just qualify for law school). You have the data, now you decide. Do you believe the Guidelines define how to effectively use decent, job-related, validated tests? Or do you prefer to believe the enthusiastic advice of people who don't know what they don't know? If the latter, please step this way – I have a real beauty to show you, only driven to church on Sundays…

Recommended Reading
The 1999 Standards for Educational and Psychological Testing define best practices for test developers and are recommended reading for any test user. Here are a few excerpts from the section on using employment tests:

"Validity is the most fundamental consideration of developing and evaluating a test...it consists of accumulating evidence to provide a sound scientific basis for score interpretations" (i.e., no home-based evidence = no test credibility).

"A clear statement of the test objective should always be made prior to development of a test" (i.e., a hiring test should always be based on its ability to predict job performance)

"Selecting the right test should always be based on a job analysis" (i.e., you should never test for something that is not firmly grounded on business need and job requirements).

If these sound like good ideas to you, then you’ll find the detailed information in the Standards well worth the price of admission.