What's Accurate and What's Not...

We all tend to think we are a pretty good judge of skills…. if we measured results, though, most of us would have a bleak track record. For example, how many of us are in a second marriage, have mistakenly promoted a technician to manager or over estimated a friend's job skills?

It's normal to make mistakes. It's foolish not to learn from them. And, it is irresponsible for hiring professionals not to strive to get better.

Ever heard it said that impressions are made within the first 30 seconds of meeting? Our brains come pre-programmed to evaluate others based on a quick "snapshot." This tendency to form quick opinions might be time-efficient, but it is often highly inaccurate.

The following chart illustrates some of the most common selection methods and their average "predictability". The numbers were gathered from hundreds of controlled studies.

Selection Method	"Predictability"
Handwriting Analysis	0%
Age	0%
Amount of Education	0%
Self Assessment	3%
Projective Tests	3%
Traditional Interviews	4%
Grade Point Averages	4%
Expert Recommendations	4%
Personality Tests	4%
Motivation	4%
Reference Check	6%
Biographical Data	9%
Situational Interviews	9%
Behavioral Event Interviews	10%
Mental Ability Tests	25%
Content Valid Simulations	64%
Adapted from a meta-analysis conducted by Hunter and Hunter, Psychological Bulletin, Vol. 96, 1984. Percentages have been rounded. Predictability refers to the explained variance.

Behind the Numbers

You don't have to have a Ph.D. to guess why most of these figures are so low. For example, among the first three methods, you would have to make a giant leap of faith to believe that handwriting, age, or amount of education could be directly related to job performance. In fact, two of the most incompetent managers I've ever known were among the most educated - one was an attorney and the other had a Ph.D. in business. This is not to say that good managers are not well educated, but it does say that degrees don't guarantee good management skills.

The next seven selection methods are slightly more predictive. As you can see self-assessment, projective tests, and traditional interviews are all self-presentations of how a person would like to be seen. These methods are easy to fake and provide little job-skill data. Grade point averages tend to predict grade point averages - useful for admission to grad school, but not for job performance. Expert recommendations, personality tests, motivation tests provide about the same percentage of usable data. They still aren't great. Their low predictability either comes from "halo" or not understanding how personality affects performance.

The next four methods (reference checks, bio-data, situational interviews, and behavioral event interviews) are more accurate - not great, but several times better than the first 10 methods. If you look closely, these methods require more rigor, some training, are more focused on the elements of the job. This makes them more accurate and harder to fake. Still, as good as they seem, they leave about 90% to chance.

The last two methods are among the most powerful performance predictors. Unfortunately, if you look at mental alertness tests through a demographic filter, you will see a long history of adverse impact. Is this a problem? Not if you have done your homework. The government does not require you to hire unqualified people, but it does require you to show that intelligence test scores are correlated with acceptable job performance. That is good for both applicants and the organization.

Simulations are among the most accurate predictors. Simulations are widely used to qualify and train aircraft pilots, tank crews, ship crews, transport navigators, military specialists, sales people, managers, customer service reps, and a host of other occupations. Simulations are particularly good when specialized skills are needed for high job performance.