|
Methods for selecting personnel: what works and what doesn't.
By Professor Chris Dewberry
_What do we know about the extent to which different selection methods predict job performance? Over the last few decades there has been a great deal of research on this, examining many thousands of job applicants applying for many different types of jobs. For each selection method, the statistical technique of meta-analysis has been used to examine the relationships between a particular selection process (e.g. unstructured interviews) and job performance. With this technique, the results are not derived from a single research study, but rather from the results of many such studies.
According to these extensive meta-analyses, the rank order of selection methods (best to worst) in terms of the extent to which they are able to predict the high performing job candidates are as follows:
1. Cognitive ability tests (.53),
2. Assessment centres dimension scores (.45)
3. Structured interviews (.44)
4. Assessment centres post wash-up overall assessment scores (.37)
5. Biodata (.35)
6. Situational judgment tests (.34)
7. Work samples (.33)
8. Unstructured interviews (.33)
9. Emotional intelligence (.23)
10. Personality based on the best single personality predictor, conscientiousness (.22)
The figures in brackets are the correlation coefficients between each selection method and job performance. They are all taken from Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262-274 with the exception of the correlation for assessment centre dimension scores which are taken from Arthur, W., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56(1), 125-154; and the correlation for work samples which are taken from Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity: Updating and integrating some classic literature. Personnel Psychology, 58(4), 1009-1037.
What should we make of these findings?
Well one immediate conclusion might be that organizations should switch from whatever assessment method they are currently using to cognitive ability tests. After all, such tests are relatively cheap, quick to administer, can be used on several candidates simultaneously, and by some margin represent the best single predictor of job performance. Unfortunately however, there is a catch, or rather two catches. The first is that as well as being the best single predictor of job performance, cognitive ability tests also have the highest level of adverse impact of all selection methods.
Many studies, most of them carried out in North America, show that the performance of white candidates is significantly better than that of people describing themselves as black. Furthermore, this difference is not marginal - it is strikingly large. It should quickly be added that this is not research with racist motives. The people examining these issues were examining whether various types of selection method discriminate against social groups, including ethnic minorities.
So why should this difference in the ability of people from white and ethnic minority backgrounds exist? An important clue is offered in the work of James Flynn. This psychologist discovered that over the last couple of generations, average scores on cognitive ability tests have increased massively.
For example, compared to undergraduates in the 1950’s, undergraduate students in the 1980’s obtained much better scores on the same cognitive ability tests. What’s more, this increase is not due to current students having greater knowledge: the tests used in Flynn’s research are culture free, with the people being tested analysing relationships between visual patterns. The ability to analyse these patterns successfully does not depend on having a good vocabulary, or a good knowledge of mathematics. This finding, now known as the Flynn Effect, strongly suggests that cognitive ability is strongly related to environmental influences during our development. And if people from white and black ethnic backgrounds in North America tend to have systematically different educational and social backgrounds it is not surprising that they differ in their ability to perform well at cognitive ability tests. The upshot of this is that cognitive ability tests, whilst certainly having a number of useful features, should be used with considerable care.
The other problem with a sole reliance on cognitive ability tests is that for better or worse organizations are not happy to simply employ people who obtain very high scores on tests of intelligence – they want to see them and talk to them first. Two selection methods which allow this, and which are also relatively good at identifying the best job candidates, are structured interviews and assessment centres.
Research suggests that structured interviews work best when candidates are asked how they would perform in specific situations. These situations are designed to examine their ability to meet the key demands of the job. Answers can be matched to known response types by trained interviewers, and scores given for relatively good and relatively poor answers. In addition any steps taken to increase the reliability of interviews, and the extent to which they focus tightly on the competencies required in the job, are a good thing. Such steps include not only obvious steps such as interviewer training, but also the use of standardized scoring systems, and multiple interviews.
Another relatively effective selection method is the assessment centre. Assessment centres became popular in the 1980's and they are now very thick on the ground. A brief consideration of the features of assessment centres helps to explain their rapid growth. By using a team of trained assessors to evaluate each candidate against a set of job-relevant competencies, across a range of specially developed exercises, assessment centres appear to provide a comprehensive and objective way to assess job applicants. The applicant is satisfied because he or she is given a good chance to perform well in a variety of assessments, and the organization is happy because the selection system is thorough and scientific. Indeed many view assessment centres as the flagship of selection systems, arguing that whilst they may be time consuming and expensive to develop, and resource intensive and expensive to run, it is worth all this money and effort in order to ensure that the best possible selection system is put in place.
An interesting feature of assessment centres is that the extent to which they are predictive of job performance appears to depend partly on how they are designed. When the scores that people obtain from assessors across exercise are correlated with later job performance scores, research suggests that assessment centres are among the best of the selection methods currently available, but when overall assessment centre scores are used (i.e. the scores obtained by candidates after wash-up consensus discussions), research shows that the performance of assessment centres is no better, and may be worse. The implication of this is that wash-up sessions, which are used in the great majority of assessment centres, are not adding value to the assessment centre process, and may actually be undermining the effectiveness of the centres.
Why might this be?
One possible explanation on which I have recently been carrying out research myself is that as well as the formal processes in which assessors have been trained taking place at wash-ups, informal processes are operating also. To investigate what these processes might be I interviewed several experienced assessors. They indicated that in their experience several informal processes take place in wash-ups, including:
• Candidates are discussed in wash-ups when one or more of the assessors who observed them is not present.
• Assessors discuss candidates in ways that are not directly and explicitly related to the competencies under consideration.
• Assessors openly decide whether a particular candidate should be appointed or not, and then adjust that candidate's overall exercise grades or ratings accordingly.
• Assessors talk about candidates before the wash-up session begins.
• Assessors refer to things they believe they have learnt about candidates outside of the formal assessment process (e.g. by observing candidates at a pre-assessment centre meeting, or by overhearing discussions between candidates during the assessment centre).
• Candidates are evaluated when all assessors are clearly very tired and want to leave
• Assessors spend more time discussing the first one or two candidates to be considered than the last one or two
I then asked a further 84 experienced assessors, each working in a different organization, to indicate how often these informal processes take place. The results indicate that they are surprisingly common. For example, over 60% of the assessors indicated that assessors at least occasionally discuss candidates in ways that are not directly and explicitly related to the competencies under consideration, and that over 50% said that assessors at least occasionally refer to things they believe they have learnt about candidates outside of the formal assessment process.
These findings suggest that wash-ups are not always meetings in which information gathered about candidates is processed in a highly rational and objective manner. Instead a variety of processes and events which are not strictly relevant to the assessment process can intervene, processes and events which may undermine the effectiveness of the assessment process. The extent to which this is a problem is still unclear at present, but is certainly worthy of further research.
The selection methods which don’t make the top four in terms of their ability to predict the best job candidates can of course still make a useful contribution. One reason for this is that ability of these methods to predict the best candidates isn’t the only consideration. Selection systems should also be cheap, fair, reliable, applicable to just about any job in the organization using it, and acceptable to both the organization and the job candidate. Another is that it is of course possible to combine information from several selection methods rather than relying on just one.
However, the debate about the value of wash-ups in assessment centres draws our attention to a critical issue here. Good personnel selection is not just about using good selection methods – it is also about the steps taken to combine information obtained from different sources about each candidate – the CV, the application form, the interview, the personality test and so on. It is easy to overlook the nature of this critical information-combining process, and it warrants just as much attention as the identification, development, and maintenance of each of the selection methods utilized by an organization.
______________________________________
June 2007 |
