Does Psychological Assessment Work?

The Research

Summary
Research has shown that staff vary widely in their productivity, and that psychometric tests can effectively predict this variation. Research has also calculated the financial benefits of psychometric selection, and has confirmed what many have known all along: good staff are critical to organisational success. Two of the most cost-effective and widely-used methods of selecting staff are ability testing and personality testing. Recommendations for the effective use of these tools are made.

Introduction

Why psychometric selection?
Most business people agree that recruiting the right employees is critical to business success. Recently, researchers have tackled the complex business of putting a dollar value on good employees. Certain estimates emerge from this research with remarkable consistency(5).

• The difference in production between a good employee and an average employee is worth half of that persons salary*. The same difference exists between an average employee and a poor employee.
• In most industries, the average employee does work worth twice their salary each year (because of costs, the business wouldn’t be able to survive otherwise).

So for this position, a good employee produces $20,000/year more than an average employee, and $40,000/year more than a poor employee.

These estimates are conservative and hold for skilled blue collar (eg, a junior builder) or administrative white collar jobs (eg, an administrative assistant). The differences in productivity are larger for professional jobs. They are smaller for jobs where employees follow very strict routines (eg, working on an assembly line).

These differences in productivity can spell the difference between a business thriving, and a business failing. Do the maths for the number of employees in your business – can you afford not to select the best?

Psychometric selection is the most reliable method of choosing employees
This graph shows how well different methods of selecting employees work. The combination of ability and personality testing appears to be the most effective method. These tools are far more cost-effective than other methods with similar validity – structured interviews and job knowledge and work sample tests need to be professionally produced for each particular job type.

Researchers have devised empirically based methods of calculating the economic benefits of different selection methods(5,14). These calculations can become somewhat complex. However, take an example of recruiting a secretary on a salary package worth $40,000, where there are 10 candidates for one position. On average, choosing the successful candidate using ability and personality testing combined, compared to using interview alone, would result in over $15,000 worth of increased output per year the employee stays with the company. Additional benefits with direct financial consequences include decreased staff turnover, increased trustworthiness of staff, and better team cohesion and morale. Typically, psychometric methods of selection result in large financial benefits across a broad range of jobs(13). It is not just academic psychologists who have observed this fact. The financial benefits of better selection are very real and noticeable, and as a consequence psychometric tests have been almost universally adopted by large corporations and military forces around the world. The impact on overall organisational performance is often dramatic(13). Smaller organisations are less likely to have first-hand experience of the benefits of psychometric selection, but more are discovering them each year.

APP psychologists can discuss the applicability of psychometric methods of selection to your specific organisation.

Why don't interviews work better?
Nearly all employers use employment interviews as part of the staff selection process(17). Most interviews are carried out in a fairly unstructured, subjective fashion. Traditionally, psychological researchers have been very sceptical about the predictive value of unstructured job interviews. More recent evidence has suggested that interviews do have some value, but that they are not nearly as predictive as ability tests, or as ability tests combined with personality tests(20). On the potentially useful side, the impressions of interviewers tend to be influenced by factors like the social adeptness of the employee, extraversion, and to a small extent by cognitive ability. On the negative side, interviewers are also influenced by impression management and outright deception on the part of candidates, by similarity between the interviewer and interviewee, and by superficial characteristics like the physical attractiveness of the candidate. Unstructured interviews are notoriously subjective – different interviewers tend to come to different conclusions about applicants(17). Interviewers, including trained psychologists, also find it very difficult to determine the conscientiousness, trustworthiness, and emotional stability of applicants. For these reasons, it is better to use an interview as part of an assessment process, rather than by itself.

Some things to know when thinking about selection
Traditionally, research into which selection methods are the most useful has concentrated on predicting overall job performance. More recent research(3) has made a distinction between whether employees

• can do the job (‘core task proficiency’, eg a plumber can install a sink)
• will do the job (eg employee follows rules, gets along with other team members)

These performance criteria are both important, but they are sometimes predicted by different things.

It is also important to know that over the last two decades substantial progress has been made in personnel selection. In particular, it was once widely believed that personality tests were not useful for personnel selection; it is now known that they can be very useful(19).

Testing Ability

Evidence for the usefulness of Ability Tests
The validity of tests of general mental ability in predicting job performance is one of the best established findings of psychology(21). The US Army has conducted a 20 year research project, involving thousands of participants, into the prediction of job performance across a wide variety of military jobs(3). The conclusion of this project was that measurement of general mental ability is the core, most critical tool for staff selection. Research by the US Employment Service, involving over 32,000 applicants to nearly every kind of job that exists, has also concluded that general mental ability is a near-universal predictor of job performance.

General mental ability versus specific abilities
Mental ability is not unitary – someone can be good with words, bad with numbers, but good at rotating maps in their head. However, all known tests of mental ability are positively correlated(16) – that is, people who are good at one kind of task involving mental ability also tend, on average, to be good at other tasks which appear to be completely different. General mental ability is essentially the tendency to be good at a wide range of cognitively demanding tasks, and it can be effectively estimated with certain tests.

Historically, there has been some debate about whether specific abilities or general ability are more important. Some important conclusions can now be drawn.

• General mental ability is important for any but the most simple of jobs(20).
• Verbal and numerical ability are important in administrative and management roles(3).
• Psychomotor ability is important in hands on jobs, eg driving a truck(5).
• Spatial ability is also important in some specific jobs, eg a tank gunner(3).

Verbal and numerical ability combined provide a good indicator of general mental ability(16), and provide sufficient information about ability for most selection decisions. APP psychologists can provide further advice on the relevancy and availability of tests of various specific abilities.

Ability level and job type
As previously indicated, general ability is more important for some jobs than for others. As a general rule(7), a higher level of general ability is required for tasks that are

• More complex
• Require continuous learning and adaptability

Studies have identified the average IQ of applicants for various jobs. A rule-of-thumb that will give your organisation a competitive advantage is to select people from the upper half of the distribution of ability amongst applicants for that position. In general, candidates for a position will possess a wide range of ability test scores. Candidates with an IQ that is unusually high for a position (15 or more IQ points above the median) may become bored with the role if they are not given sufficient opportunity for extra responsibility and advancement.

(adapted from information in Gottfredson (1997)).

Why not just use educational background?
Many organisations use educational background to select staff. This is appropriate and necessary for many situations: employees of course need to have learnt the specific technical knowledge required by a job. Employers sometimes also use educational level to try to measure the kinds of skills that are in fact a function of general mental ability: the ability to reason, solve novel problems, and learn new material. Psychometric testing has several advantages over this procedure:

• It is far more predictive of job performance (see graph above)
• It compares participants fairly and objectively, bypassing the complexity of comparing academic records from institutions which may have very different standards
• Candidates who have invested more time in tertiary education quite reasonably expect to be compensated with higher salaries

There is evidence that there is a wide range of IQs among people of all levels of education(16). It does not make economic sense to exclude someone who clearly has the ability to do a job, simply because they have not completed an arbitrary number of years at university.

Evidence for the validity of the APP online ability tests
The APP ability tests assess two areas of mental ability

• Verbal Reasoning measures how well someone can see relationships and solve problems using words and language.
• Numerical Reasoning measures how well someone can see relationships and solve problems using numbers.

Verbal and numerical reasoning are perhaps the most widely applicable aspects of mental ability in today’s workforce, and are particularly predictive of performance in administrative and managerial tasks(3). There is also considerable evidence that overall performance on these tests is strongly related to general mental ability(16), and thus is predictive of performance on a wide variety of job tasks.

The APP online ability tests were written and tested by APP’s psychologists. The items they contain have been trialled and normed using a representative sample of the Australian population. The tests are scored using Item Response Theory (IRT), which allows them to accurately discriminate between bright individuals.

The reliability of the APP ability tests compares favourably to other ability tests(15). APP have also conducted a study which found that their ability tests predict scholastic achievement (as measured by VCE exam performance) with an accuracy that is similar to that of other well-regarded ability tests(15).

Testing Personality

Personality and Job Performance
Research has shown that certain personality measures can predict job performance, and that combining these tests with ability tests can result in better prediction than the use of either measure alone(3). In most cases, core task proficiency is best predicted using ability tests, but personality measures come into their own when predicting other aspects of job performance which are equally important. Personality measures are particularly useful for predicting rule breaking and disciplinary problems(3), which are poorly predicted by measures of IQ. They are also predictive of staff turnover(3), absenteeism(3), and of ability to work productively in a team(2). In general, it is particularly important to use personality testing when selecting people for jobs requiring teamwork or a lot of social interaction. The inclusion of a single disruptive team member can have a dramatic negative impact on team effectiveness(2).

Beware! Some personality tests are predictive of job performance, and some are not(5). It is important to look at what research (if any) supports the use of a particular personality test for personnel selection. There are several personality tests marketed in Australia that are not supported by adequate research. In particular, personality tests that do not contain lie detection and validity scales are unlikely to be very useful(3). Also, it is important that any test used contains a valid measure of dependability and/or conscientiousness, as research has shown these constructs to be the most consistently useful in staff selection(3). Finally, it is important that the results of personality tests are interpreted by a psychologist or another suitably qualified individual.

Personality testing versus EQ testing
The concept of emotional intelligence has become very popular in recent years. At this stage, however, there has been comparatively little research conducted into emotional intelligence, compared to research into personality and job performance(20). In particular, there is disagreement about what emotional intelligence really is, whether it is distinct from personality, and whether, in applied settings, it can add to the prediction of job performance(20). At this stage, if EQ tests are used, it would be prudent to use them in conjunction with conventional ability and personality tests. It should be noted that the CPI (a personality test) contains measures of constructs which appear identical to some conceptions of emotional intelligence.

Evidence for the validity of the CPI and of Spectrum 260

• Background to the CPI

The CPI is probably the most thoroughly researched personality measure in the world(10). It has fared well under independent critical review, both for use in predicting behaviour in general1 and as a tool for personnel selection(5). It is used by the California Highway Patrol and by many other law enforcement agencies around the world, and by at least one Victorian fire authority. The US Army chose parts of it to adapt in the development of their own personality measure for recruitment(3). APP have used this test successfully for about 25 years. Spectrum 260 is a recently developed short version of the CPI, and there is evidence that it produces largely equivalent descriptions of individuals. The average correlation between corresponding scales on Spectrum 260 and on the CPI is 0.946.

• Research studies on the predictive validity of the CPI

Unlike many psychological tests, the CPI was specifically designed to predict real world behaviour, and was developed accordingly(9). The test has a high degree of predictive validity in predicting everyday behaviour; for example, the Dominance scale predicts whether someone will be described as ‘Behaves in an assertive fashion’ by their peers with a predictive validity of 0.42; conversely it predicts ‘Is vulnerable to real or fancied threat, generally fearful’ with r = -0.32. These predictive validities have not been corrected for unreliability in the criterion variable. People are often struck by the detail, descriptiveness and accuracy of the CPI.

The validity of the CPI for predicting job performance has been established in studies looking at a wide variety of occupational groups. Examples include police officers(11), armoured car company employees, telephone service representatives, and managers(10). Best estimates of the CPI’s corrected predictive validity can be derived from meta-analytic reviews: about 0.40 for overall job performance(19,22), and about 0.56 for problem behaviours(12). Given the evidence that Spectrum 260 produces very similar descriptions to the CPI(6), it can be assumed that this instrument is also a good predictor of work performance. Although Spectrum 260 is a short version of the CPI, it is still similar in length to other personality inventories which are widely used for personnel selection.

Finally, there is evidence that the CPI is a useful predictor of serious criminal behaviour. The instrument discriminates between criminals and noncriminals in general(8), and between white collar criminals and noncriminal executives(4). It also predicts self-reported sexual aggression amongst men who have not been convicted of any offence(18). This research suggests that the CPI is a useful screening tool to use when selecting individuals for positions of trust.

Further Reading
A good overview of personnel selection is provided by

Cook, M. (1998). Personnel selection: Adding value through people (3rd ed.). NY: John Wiley & Sons.

References

click here to close window

click here for a printable version