What's the "AHA!" REPORT all about?

This series of newsletters contains AHA! information to help people and organizations hire the best employees, make the best promotion decisions, retain the most qualified people, maintain the widest applicant pool, follow best practices, and (if you are subject to US law) remain aware of EEOC hot-spots.

In the News
Library
ERE articles
Media Kit

Spring is in the air, and despite the economy, everyone in town seems to be out on the roads. This year several young people of my acquaintance happen to have crisp, new driver’s licenses, so they’re out there too. They’re all fine boys and girls. Their reflexes are quick, their eyes are clear, and their judgment is the best possible in a 16-year-old. So naturally, I’m terrified at the thought of them behind the wheel.

If you really stop and think about it, piloting a two-ton automobile at near-relativistic speeds is a risky business indeed. If cars were invented today, they would probably be illegal (along with caffeine and many other comforts of modern life.) But because we’ve built our society around them, practically any teenager with a pulse, an ID, and some pocket change can obtain a license. The driver’s test is not much more than a quaint rite of passage.

Numerous studies have shown that seriously beefing up the licensing requirements would save thousands of lives per year. Given our love of cars and our tradition of civil liberty, it will probably never happen. But what if it did?

If I-O Psychologists Ran the DMV…
Suppose some very important, highly enlightened government officials decided to put you in charge of reducing traffic accidents. They gave you the power to renew current licensed drivers nationwide, as well as issue new permits. They said you could use any criteria you like. Where do you start?

First, you might want to investigate the job. Naturally, you would talk to drivers with good safety records and experienced driving instructors. After many interviews, you would learn that the following competencies were critical:

• Eyesight
• Physical coordination and reaction time
• Basic operator knowledge
• Knowledge of driving laws
• Analysis and judgment

Knowing vs. Showing
Knowing that impairment of any single factor could lead to driving accidents, you decide to administer a broad range of tests:

• Personal opinion. Each candidate would be recommended by one or more licensed drivers. This would help screen out the blatantly unqualified (assuming you can trust the references).
• The personal interview. Each candidate would be interviewed about driving skills, basic operator knowledge, driving laws, and past and/or future driving decisions. This would screen out people who did not know the right answers, but you would have no proof other than their word.
• Basic skill tests. Each candidate would be tested for eyesight, coordination, physical reaction time, operator knowledge, and driving laws. This would be considerably more valid and reliable. Candidates who, under controlled conditions, demonstrated physical skills and knew driving laws would be much safer than those who didn’t.
• Simulations. Each candidate would “drive” a realistic simulator, not just around the block, but under a wide variety of road, weather, and traffic conditions. You could be reasonably sure candidates who successfully completed the simulation would be safer drivers.
• Psychological evaluations. Each candidate would be evaluated for his or her potential for road rage, use of drugs that would impair judgment, physical health, emotional stability, and so forth. Again, you could be reasonably sure candidates who successfully completed the evaluations would be safer than those who did not.

Now, what would happen to traffic accidents and deaths if only the people who passed this battery were permitted to drive? Right! Millions of dollars and tens of thousands of lives would be saved annually.

But what might critics claim? It’s way too expensive, and too much work to implement. Too many candidates fail. People need to drive. They’ll get better once they get some more experience, or coaching, or training.

But we can’t just license everybody. How about a compromise: drop everything except the interview. Yeah. That should be OK.

Not Safe for Work
Now let’s apply this argument to organizations, and more specifically, employment and promotion decisions. Just as driver competency correlates with traffic accidents, employee competency correlates with turnover and productivity. From the shop floor to upper management, getting the right person into the right job requires multi-trait, multi-method testing.

This strategy depends on a few simple concepts: carefully determine job requirements; use several testing methods; and measure critical competencies more than once. You don’t need to be a rocket scientist to see that when something is left out of that equation, employee quality (e.g., productivity and turnover) will suffer.

What Kind of Multi-Trait, Multi-Method Test?
Everything that separates one candidate from another is a test. This includes references, interviews, photographs, application forms, resumes, sources, written questions and answers, and so forth. Picking the first person who responds is a test, of speed anyway. Put another way, unless you hire everyone who applies, you are using some kind of test.

Once we agree that every recruiter and hiring manager is in the testing business, the decision becomes much simpler: how predictive is the test he or she is using? Predictive accuracy is not complicated. All we want to know is whether the test measures something critically important to the job, whether scores are accurate, and whether the test is stable over time.

On the Road Again
Let’s return to the example of the new-and-improved driving test. Does the “personal opinion” component measure something critically important to the job? Is it accurate and consistent? Probably not – unless the candidate behaves like the lead character in a Steven King novel.

Personal opinions are filled with subjectivity. Some subjectivity is good; but unless the job is very simple, personal opinions are weak predictors. Why? It’s a people flaw. Humans tend to think their personal opinions are good indicators of job competencies. Although we all like to think we are excellent judges of character, research shows otherwise.

How about the interview portion? What about the people who are brimming with confidence about their interview skills? Can they be wrong? As amazing as it seems, sometimes better-crafted interviews lead to better predictions, but interviews are still based on “knowing” not “showing.”

Knowing and Showing
Which brings us to the skill tests and simulations. (Remember those – the ones we eliminated because they were too hard?) The more a test measures showing instead of knowing, the more decision quality increases. That’s because test error decreases.

Test error is an unfortunate fact of life. Every test, of whatever kind, contains some amount of error. Error can never be completely eliminated, but it can be reduced by following a few key rules:

• Interview job-content experts such as trainers and job holders to clearly understand the key competencies required for the job (i.e., skip the managers).
• Measure every key competency using tests that are job-related and validated (i.e., make sure scores predict job performance).
• Use more than one test type (i.e., cross-check).

Clearly understand that “showing” (what you see) is better than “knowing” (what you hear)