What's the "AHA!" REPORT all about?

This series of newsletters contains AHA! information to help people and organizations hire the best employees, make the best promotion decisions, retain the most qualified people, maintain the widest applicant pool, follow best practices, and (if you are subject to US law) remain aware of EEOC hot-spots.

In the News
Library
ERE articles
Media Kit

The social networking revolution has spawned a number of curious cultural phenomena. Cyber-stalking, celebrity tweets, and photo postings of what your friends had for breakfast all come to mind. On a more positive note, the mountains of detailed biographical data amassed by social networks have given programmers the opportunity to develop very clever techniques for matching up people and things.

Just a few years ago, automated affinity matching was confined to things like Amazon’s recommendations, and was often laughably inaccurate. Google made some headlines by using algorithm-based calculations in its hiring process, hoping to identify candidates who were not just smart and accomplished, but also likely to be a good fit in the company’s culture. At the time, this seemed pretty close to science fiction. Now it’s the kind of thing teenagers take for granted on Facebook. So perhaps it would be useful to revisit the subject of bio-data in the hiring process, and consider how it’s working out.

Bio-data in the Real World
The basic approach to using biographical data or “bio-data” is simple:

• Survey current employees on a variety of characteristics and traits, including teamwork, biographical information, past experiences, and accomplishments.
• Statistically determine which of these many traits differentiate employee performance.
• Develop an online survey to gather bio-data.
• Score applicant responses based on the number of performance indicators each candidate possesses.

All in all, this is supposed to be a more “scientific” approach to hiring. Bio-data surveys are well-researched in the literature. They work on the same principle as behavioral interviewing: what was done in the past predicts what will be done in the future. The big differences between bio-data questionnaires and behavioral interviews is that a good behavioral interview is backed by a thorough job analysis, interviewers can ask follow-up and clarification questions, and multiple interviewers coordinate the information.

A bio-data form depends entirely on the people who created the items, the scoring algorithm, trained analysts who look for trends, and the specific position. More about these later. Both bio-data questionnaires and behavioral interviews are self-reported information, subject to applicant creativity and being in touch with reality. In general, they both have about the same degree of predictive accuracy.

Bark if this hurts
One problem arises almost immediately: understanding the reasons for employee performance. In a way, identifying low performance in an organization is much like diagnosing a sick pet. Fluffy whimpers. Dr. Dolittle is on vacation, so we don’t know when it started, where it hurts, or whether there are other symptoms. We just know Fluffy is in pain.

In the workplace we can evaluate employee satisfaction, voluntary turnover, training success, or terminations. But these are all end results. They don’t actually tell us the root cause of the performance problem.

Ultimately, performance problems can usually be traced back to bad management (incompetent managers, conflicting goals), unpleasant working conditions (wages, benefits, environment, and insufficient resources), lack of training, and/or poor job skills. There are many reasons why people under-perform.

Finding the root of the problem is the most important part of developing a pre-hire test. If we don’t know the root cause of low performance (or the root cause of high performance, for that matter) any hiring solution will be half-baked because it won’t address the issue.

What’s Performance?
Moving back a step, how do we even know a company has low performance? Is performance measured by growth? By profitability? Perhaps today’s buzzword, “sustainability?” There are too many environmental factors for us to assume that employees are the only growth factor.

At an individual level, how about performance appraisals? We all know performance appraisals suffer from the “no one here is perfect,” “everyone here is perfect,” or “forced-rank” syndrome. In addition, performance appraisals tend to be part fact and part management opinion. Basically, we can never really know what performance appraisals measure. Two managers may rate the same employee differently.

Let’s say we have objective productivity data available such as units per hour, mistakes per 1,000 cold calls per month, cross-selling revenue, or customer service surveys. These are better indicators of performance because they are less likely to be affected by other factors, but we still have to account for things that might influence them.

Did the machinery malfunction or was it newly renovated? Were mistakes suddenly calculated on a different basis? Was the territory newly acquired or was there a company promotional campaign? You get the picture. Accurately defining performance and controlling for outside factors is absolutely critical. Otherwise, you run the risk of measuring garbage.

What Can Current Employees Tell Us?
Assuming that root-cause data and performance data are under control, let’s look at current employees.

Current employees are a great deal alike. That is, they are all “good enough” to stay hired. The differences between high- and low-performing current employees (assuming we are exceptionally clear on the definition of performance) are generally very small. So small, in fact, that performance differences might be due to pure chance (now, wouldn’t that mess up the recipe for success?). Applicants, on the other hand, are very different.

In addition to the applicant-employee difference, not all jobs have the same skill requirements. Does it come as any surprise that singing in the glee club may have nothing to do with administrative skills? I know sales managers who only hire salespeople if they played athletics in high school (the poor man’s bio-data test). Fifty percent consistently fail within the first year. That’s no better than chance. Google might ask about the chess club instead, but they’d probably get the same result.

The Job Matters. So Does the Context.
Pick up any good book on bio-data and you’ll see that trustworthy bio-data scores are exquisitely sensitive to positions. In other words, salespeople, first-line managers, and administrative support all might have completely different bio-data profiles associated with job performance (there’s that p-word, again).

High performers are usually specialized beasts who do not conform to any norm. They are often so good that they operate on automatic pilot; or they cut corners to achieve their goal. I recall a marketing manager who stole his prior employer’s product secrets and used them to reduce development time. There’s a good high-performance role model? Right?

You may think that you should figure out what your corporate culture is, and then examine whether applicants fit that. But companies are not static. They start as small enterprises founded by highly motivated entrepreneurial folks who dine on the vending machine goodies, shave in the bathrooms, and sleep on cots.

After a while, the free-wheeling entrepreneurial environment changes into a bureaucracy, then it changes again, and so forth. Anyone who recalls the rise in dot-com businesses, or considers how big business fares when leadership changes, knows that today’s cultural fit may not last.
                                  
I once worked for a company that hired smart, highly motivated people for plant start-ups. Two years later, the plant management complained they had “all leaders and no followers!” Be careful what you measure: you just might get it.

Statistical Sense and Nonsense
Statistics are dumb -- but useful. They can tell if two numbers are correlated; but they cannot tell if one number causes the other. This is really important if you want to develop a test that predicts job performance.

I can statistically show that blue eyes and blond hair are correlated, but we all know that blue eyes do not cause blond hair. Jan Lethen, a statistics professor at Texas A&M, cites more correlations as an example of statistical nonsense: shark attacks are correlated with ice cream sales; skirt lengths with stock prices; and cavities with vocabulary size.

When a broad sample of items are given to a broad sample of people and statistically analyzed, some correlations are inevitable. But if the items do not cause the behavior, they are bogus. They end up screening out qualified people and screening in unqualified ones.

Other problems include sample sizes. Statistics represent general trends between two groups, each of which must have the same bell-curve. Bell-curves need about 25 people at the minimum. They really work when the numbers get closer to 250. And, yes, that sample should be job-specific, not just the whole company.

When Does Bio-Data Work?
Bio-data questionnaires provide the best results when the following criteria are met: The process covers a tight-knit group of similar jobs and uses a tight-knit definition of job performance; a skilled analyst interviews multiple people looking for causal bio-data items; bio-data items are administered to a large number of current employees and analyzed for performance differentiation; the test is given to a large number of applicants who are hired regardless of their scores; and after a period of adjustment, bio-data scores and job performance are statistically compared.

So, why hasn’t bio-data revolutionized hiring yet? For one thing, you can’t just download the software. Like most things of value, it takes a pretty significant investment to implement for specific positions in a specific company. In the future, it may become ubiquitous and highly effective. But at this point, bio-data seems to work much better for selling online ads than for providing meaningful insights into people’s behavior at work. Which is not surprising, given that it started with Google.