Knowledge Factor Takes the Guesswork Out of Testing

In college, there is a type of exam students jokingly refer to as the "multiple guess test." Such multiple-choice assessments present the test taker with at least four potential answers to choose from. If a student doesn't know the exact answer, he can use the process of elimination to select the correct one or simply guess, and there's a good chance he'll guess correctly.

In the public safety world, if new recruits pass such tests to receive certification, those relying on these new employees believe these individuals come with a basic set of skills and knowledge. After all, they did pass an exam certifying they knew the material. But this may not be true, warns Brian Webster, vice president of marketing and business development at Knowledge Factor Inc., a Denver, Colorado, company that has developed new testing protocols to ensure assessments accurately reflect what test takers actually know.

"There are some people who are very lucky test takers, who can actually guess the correct answers on certifications without really knowing the information," Webster explains. "They may receive credit for information they don't truly hold. Now, that's not a serious deal in college, but it becomes a major risk with first responders, where quick decisions may mean the difference between life and death."

Three years ago, Knowledge Factor uncovered a learning methodology, developed by a UCLA professor over 20 years, which eliminates guesswork from multiple choice testing and helps ensure students have mastered the subject material.

The company calls this training method Confidence-Based Learning (CBL), which it acquired and patented three years ago. This technique ensures learning by measuring precisely what people know and don't know — without guesswork and doubt contaminating the results. Once this is complete, CBL rapidly remediates learners' gaps in knowledge and confidence.

"We individualize training and learning by first understanding what someone knows or doesn't know," explains Webster. "We then use these results to tailor a curriculum to the individual's specific needs."

Knowledge quality = confidence

According to Webster, CBL accomplishes learning by measuring knowledge quality in four different domains:

1. Mastery. This is where individuals are both correct and confident in their knowledge. "Individuals with confidently held, accurate information will perform effectively and predictably based on that knowledge in the field," Webster says.
2. Doubt. The person retains the correct knowledge but does not believe with full confidence the information is accurate and thus is not confident enough to act on it.

3. Misinformed. This is when people confidently believe the wrong information is correct. Confidently held misinformation is dangerous because individuals will act on that knowledge but won't have the correct outcomes because their facts are incorrect, says Kevin Warr, CEO and technology officer at Knowledge Factor.

   Misinformation is absolutely unacceptable in law enforcement, Webster concurs. "We could probably call it the headlines quadrant," he says. "Chiefs of police would be reading headlines about it in the paper if their officers acted that way."

4. Uninformed. This is where someone lacks any knowledge of the subject.

Under the traditional means of assessment, in order to determine whether two separate individuals, Person A and Person B, are competent in a given law enforcement discipline, an instructor would administer an exam.

Now let's say both individuals scored a 90 percent on this assessment. It would then be assumed they were equal in their abilities. But what if you learned that 96 percent of the time when Person A answered correctly, he felt confident about his answers, compared to Person B, who only felt certain of his answers 90 percent of the time and the other 10 percent of the time had guessed correctly?

Now what if you found out that of the 10 percent that Person A answered incorrectly, he simply didn't know the information, but of the 10 percent Person B answered incorrectly, he was confident in his answer but the information he held was incorrect? "In this case, who is a greater danger to the public?" asks Warr.

Knowledge Factor's CBL system creates awareness of what liabilities and strengths are for a given individual and their information quality. "It is our position that mastery is the only acceptable outcome for training, before we put law enforcement professionals on the street," Warr says. "In the last example, Person B scored 90 percent, which is enough to certify for law enforcement, but if that individual scored in doubt, his ability to perform is substantially lower than what we thought it would be."

To truly learn something, an individual must leave the classroom with a level of confidence in the subject matter. To illustrate this point, Webster describes a college student discussing the Theory of Relativity in a learning environment. The individual leaves the classroom with some knowledge of this theory but lacks enough training to teach a class on it. "That's not because he doesn't know it at all," he explains. "It's because he lacks confidence in that knowledge."

If education occurs by simply having an instructor lecture from the front of the room, the teacher tells students what they need to know before they enter the field to practice it. But if nothing is done to nurture or truly measure student performance, when learners enter the workforce, they may not perform as expected. "Confidence-Based Learning increases knowledge but also nurtures confidence," says Webster.

The CBL difference

The entire Knowledge Factor learning program is conducted online, eliminating the need for customers to load software. Knowledge Factor works with its clients to create content for specific programs and then develops a portal for students to retrieve their customized learning program. "People have access to their content any time they want," Webster says.

At the beginning of every Knowledge Factor course, an assessment is performed to ascertain what learners know about a given topic, and the quality at which they hold the information. Students receive the assessment results to find out what learning opportunities remain. The program then uses these findings to tailor curriculum to each student's individual needs. Whatever questions learners miss the first time are covered in the new material, and when they complete this coursework, they are tested again. "This becomes an interactive process that guarantees every single person has the opportunity to reach the level of mastery," says Webster.

"What we've done is eliminate the bell curve," explains Warr. "Nobody is left behind with Knowledge Factor. It's simply a matter of going through the program the number of times necessary to close out all the gaps."

Exam questions, as shown in the Sample Question at the tope of Page 158, appear slightly different than traditional "multiple choice" questions. As illustrated in this example, students can answer each question in seven different ways. They can answer with certainty that A, B or C is correct, select a combination of answers — A or B, B or C, or A or C — with partial confidence, or simply answer "I don't know."

"This type of evaluation program promotes truth telling," says Warr. "It encourages people to answer honestly."

Once all questions are answered, a weighting system goes on behind the scenes. Students receive maximum credit if they answer correctly with confidence. If they select an answer with confidence and answer incorrectly, they receive a maximum penalty. If they note they are partially sure, it means they have some knowledge of the subject but don't hold it to the degree of commitment they need in order to act upon it. In this case, students receive a partial reward for an accurate answer; because though they are correct, their confidence is low. But if their choices are completely wrong, the system assumes they are guessing and assesses a maximum penalty. If students select "I don't know," however, the program assesses no reward and no penalty.

"Just having the option of answering 'I don't know' is fairly unique to most certifications," Webster points out. "In most tests, if you don't know the answer, you are forced to guess, but that testing format promotes contamination of its own data. It doesn't honestly reflect what the test taker actually knows." The benefit of promoting such honesty, he notes, is that students begin to view the testing process as their "friend" in helping them move through the material and master the information.

"Ultimately, it's not the score we are after; it's the learning," he adds. "The CBL method gives you accurate data on what people actually know."

In the program's second step, as shown in Student Feedback at the top of Page 160, Knowledge Factor provides rich feedback to students, who quickly see where they are misinformed, uniformed, or had doubts or mastery. The "Question, Answer, Explanation and Link" displays, as shown on Page 160, for each test question show the chosen answer and correct choice, coupled with an explanation as to why it's correct. Online links to additional materials relating to the subject are also provided. This knowledge enables users to focus on learning what they most need to know — not information they have already mastered.

In the third step, Knowledge Factor delivers the entire course back to the learner. Its primary focus is to compel students to hone in on areas of misinformation and lack of knowledge, not subject matter where they've already achieved mastery.

"What is the return on investment to teach people stuff they already know?" asks Warr. "Think about a course you've attended where you were frustrated because the instructor was teaching material you'd already learned. What happened? You started to check out."

George Epp, executive director of the County Sheriffs of Colorado Inc., an association charged with developing training initiatives for Colorado sheriff's offices, knows full well the frustration of sitting through training he didn't need. "As an officer, I've sat through 8-hour training classes where the first 7 hours covered stuff I already knew," he says. "Finally, in the last hour, they got to the stuff I found valuable. Knowledge Factor's training is designed to reduce this time waste."

After completing the program's third step, students select "retake" to have the system administer a new test with a fresh set of questions. Knowledge Factor writes a large pool of questions for each module. Every time students go through the system, they receive a random set of questions.

This virtually eliminates the ability for people to beat the test, says Warr. "In order to cheat with Knowledge Factor you'd need to spend more time cheating than actually learning," he says.

Once students are retested, they revert back to step two, and hopefully their pie chart shows a greater percentage of green or mastery levels. Each time students go through the program, more green should appear in the chart until they achieve 90 percent mastery or better — whatever mastery score their department requires. "Once they reach that goal, students are done," Warr says. "At 90 percent, they have shown they not only know, but confidently know, the material."

Retention rates skyrocket when information is presented to students in this way, says Warr, who notes CBL increases retention of knowledge by seven to 10 times that of traditional training programs. "We are building confidence around the knowledge," he says. "It's the difference between telling someone something and allowing them to discover it on their own."