The Next Evolution of Computer Adaptive Testing
May 1, 2009 – As published in Talent Management
By Dr. Michael Fetzer, Ph.D. - Vice President of Content Development, PreVisor
Today’s talent management leaders are under increasing pressure to hire top candidates for critical positions — the first time. There is little margin for error in the current economy; it’s too expensive. This requires more cost- and time-efficient processes to separate the best from the rest of the candidate pool.
Many organizations have embraced pre-employment testing as a talent measurement tool in part due to decreased time-to-hire but, more importantly, to increase the quality of hires. For example, the use of pre-employment testing can impact on-the-job performance, such as higher sales and productivity, lower turnover and better customer service.
Lately, more organizations are using the Internet to deliver pre-employment testing to screen out the unqualified and identify candidates worthy of progression in the hiring process before they are brought on-site. This has created tremendous efficiency gains in time-to-fill and recruitment costs, but it is not without potential trade-offs such as exposure of test items and susceptibility to cheating.
Fortunately, advances in pre-employment testing science and technology, such as computer adaptive testing (CAT), where each test question is dynamically selected based on the applicant’s responses to the previous question, are mitigating these trade-offs and providing additional benefits.
For example, consider an organization that needs to test thousands of applicants globally to fill approximately 100 positions annually. These are competitive positions, so test security and process efficiency are paramount. Online computer adaptive tests provide the ideal solution, and are credited with saving the organization hundreds of thousands of dollars in wasted applicant travel costs by screening out those who ultimately would not be hired.
With the dramatic rise of the Internet as a recruiting tool, many talent management functions traditionally conducted in person or over the phone are now handled online. Pre-employment testing is one example, as now candidates can complete tests from home when it’s convenient for them. This process is known as unproctored, or unsupervised, Internet testing (UIT).
UIT is mutually beneficial to candidates and the hiring organization. It allows candidates greater schedule flexibility and reduces time spent traveling from one potential employer to another. Further, UIT provides more opportunities for employment outside of a candidate’s immediate area.
For the hiring organization, UIT greatly increases the candidate pool, resulting in a better chance of recruiting the best candidates. UIT also reduces costs associated with on-site testing and candidate travel expenses. Finally, organizations benefit from a more complete picture of candidate qualifications as more types of tests can be administered in an unproctored environment.
As with any type of program in which tests are used to determine who is screened out and who moves on, there is a potential for cheating and test-taker collusion with UIT. Candidates with less-than-honorable intentions may attempt to obtain test items in advance to increase their chances of success. There is no shortage of examples of cheating attempts in education, certification and licensure testing, despite highly secured proctored or supervised testing programs.
A recent Wall Street Journal article indicated that pre-employment testing is not immune to cheating attempts. In the article, accounts from cheaters included their efforts to obtain test items and share them with friends or, worse, post them on social networking sites. The answers to the tests targeted were not readily apparent because they were personality tests and thus didn’t have one correct answer, but having these test items publicly available was undesirable for the hiring organizations and test publishers.
Most of the targeted tests were static tests, meaning every candidate gets the same test items, most often in the same order. Static tests are prone to cheating since less-than-honorable candidates can obtain all the items after completing the test once. Once the security of a static test has been compromised, often the entire test becomes invalid and has to be removed from the process.
Computer adaptive testing mitigates the potential for cheating and is rapidly becoming the standard for UIT. Essentially, CAT uses sophisticated algorithms backed by large amounts of data and extensive item pools to deliver a test that adapts to the candidate. For example, as a candidate completes a computer adaptive test for cognitive ability, the items presented will be determined by the candidate’s responses. If the candidate answers an item correctly, he or she will be given a more difficult item from the item pool. Conversely, if the candidate selects the wrong answer, he or she will be given an easier item. This process continues until the candidate produces a set of item responses that is reliable enough to calculate a score.
Since CAT draws items from vast pools of hundreds or even thousands of items, the probability of two candidates seeing the same set of items can exceed one in a million. Thus, any attempt to copy items from one CAT administration would be almost worthless. In this respect, CAT is much more secure than static tests. Further, if any items are compromised, they can be removed from the item pool and replaced with new items without having to abandon the entire test.
CAT’s ability to reduce opportunities to cheat is only one of several benefits. Computer adaptive tests also are shorter, and the scores produced are more reliable than static tests. Since the test adapts to the candidate, items that are too easy or too difficult are not part of that individual’s test. A typical static test usually contains items that are easy, of average difficulty and very difficult. For candidates with high ability, the easy and average difficulty items don’t provide useful information since high-ability candidates tend to get all those items correct. Thus, the time they spend answering those questions is essentially wasted.
With CAT, only items that provide information to differentiate among high-ability candidates are administered. The same is true for low-ability candidates. Compared to static tests, fewer items are needed to determine a reliable score for candidates at any ability level.
This is especially relevant in a hiring situation. The search for candidates with the highest qualifications requires that talent managers be able to differentiate between prospects. Static tests often fall short in this requirement since the best candidates tend to achieve the highest possible scores. Thus, static test scores become essentially useless if all of the top candidates have the same score. With CAT, the scores more closely reflect the candidate’s ability level, which helps talent managers differentiate those at the higher levels more effectively.
The use of CAT may be new to the talent management arena, but it has been used for years in military, educational, certification and licensure settings with great success. What is new on the CAT front is the technology to administer adaptive tests that go beyond measuring hard skills and abilities. Specifically, computer adaptive tests of personality traits can be added to adaptive skills/abilities tests to provide a more holistic picture of candidate qualifications. Evaluating candidates on these characteristics enables talent managers to gain insight into candidates’ propensity to succeed on the job, above and beyond the hard skills and abilities required.
CAT personality assessments utilize similar algorithms, item pools and supporting data to more quickly zero in on a candidate’s personality traits. Many static personality measures can contain upward of 300 items, but CAT personality measures can evaluate the same personality aspects with as little as 50 performance items. This greatly reduces test administration time, enhances the candidate experience and provides hiring managers with more reliable measures of a candidate’s soft skills.
For example, one organization did not want its managerial applicants to spend more than an hour during the testing phase of its hiring process. Rather than have applicants spend that time filling out lengthy personality measures, the organization was able to combine a shorter computer adaptive personality measure with realistic video-based coaching simulations, an online inbox exercise and a computer adaptive test of cognitive ability without going over the 60 minute time limit.
Preliminary research indicates CAT personality assessments are strongly correlated with job performance. In other words, individuals who score well on these assessments tend to be rated higher by their managers, have more productive interactions with co-workers and be viewed as better leaders. These results not only impact bottom-line performance, they offer evidence to validate this method for evaluating candidate personality characteristics.
This recent breakthrough in CAT technology takes the whole-person approach to testing to a new level. The whole-person approach to testing uses a battery of skills, cognitive ability and personality assessments combined into a single test to evaluate various candidate qualifications, rather than a single measurement such as those typically provided by stand-alone tests. The combined measures in a single computer adaptive test battery take less time and are more predictive than static tests of the same applicant qualifications. Getting more information in less time not only benefits the hiring organization, it provides a much better testing experience for applicants.
CAT is quickly becoming the standard for the whole-person approach to pre-employment testing. Increased levels of test security, score accuracy and reliability and the ability to predict on-the-job performance — coupled with decreased administration time and costs — provide tremendous benefits for organizations, both short and long term.
It is likely the trend toward CAT technology will continue, expanding into other types of pre-employment assessments such as simulations and situational judgment tests.
CAT Testing Helps Ensure Consistent Quality Hires
By Dr. Michael Fetzer
IT staffing firm TEKsystems believes in providing quality talent to its base of more than 5,000 clients. The organization has found that test results are crucial if it is to recruit and place the best.
Quality is the core tenet of the company’s business and brand. Headquartered near Baltimore, with offices in more than 90 locations in North America, TEKsystems places more than 50,000 technical professionals annually. Further, the organization administers more than 40,000 online tests a year as part of its process to ensure it provides top talent.
Michelle Webb, director of technical professional programs, said TEKsystems’ field offices have been using online assessments since 2000. The company measured skills prior to this, but it was an onerous paper-based task that made it difficult to aggregate and use specific data.
“Contingent staffing represents the majority of our client contracts with a strong presence in services,” Webb explained. “Our quality assurance process in the recruiting phase is the most important part of the equation. While we monitor our placed professionals, we are not managing them on a day-to-day basis per project — the client is. Understanding our customers’ specific needs up-front and matching the right person to the job is critical to our success. We can uncover individual strengths much more readily using online computer adaptive testing (CAT).
The organization hires talent for all types of IT positions, and 100 percent of the skills tests administered online are done in an unproctored setting. Webb said the adaptive testing method provides the highest-quality assessment results to better qualify candidates because it adapts to users’ responses. It is also difficult to cheat since tests are always different.
“Another benefit, especially relevant for us in IT, is that higher-level candidates — those with the most experience — are often frustrated when given an online test where they have to answer many basic questions before they get to the end,” Webb said.
“Adaptive testing can measure skills more quickly, only presenting them with relevant questions. We feel very strongly about striving to be the highest-quality IT staffing source and maintaining that brand image. Professionals in a competitive IT world need to know we only place the best.”
TEKsystems tests skills for all IT functions, as well as clerical, call center and some general business applications. They do not test for any personality or behavioral items. Other parts of the hiring quality assurance process include a live interview with every candidate placed and two past work references.
“In the world of IT staffing, when our customers see the candidates have gone through a specific test and/or achieved certification in that skill area, it tells them the candidate is highly qualified,” said Webb. “CAT has been particularly useful when testing developers, analysts, etc., who are more difficult to assess. Very specific strengths can be found and matched more appropriately to a TEKsystems client need.”
Webb said she is convinced testing has increased the value of TEKsystems’ staffing services. “Our acceptance rate began to increase after we implemented our current quality recruitment process. We pride ourselves on our quality; utilizing the CAT tool is a major component of ensuring that. We want our customers to know we are doing everything we can to give them the best person for the job.”