Exams are a widely used method for summative assessment in college education, especially in introductory courses. However, at many universities, introductory courses are large (e.g., 200+ students). Running traditional pencil-and-paper exams at this scale presents management challenges that include requesting space, printing exams, proctoring, timely grading, and handling conflict exams. These practical concerns often have more influence on how assessment is performed than pedagogical concerns.
In this talk, we'll discuss an effective, secure, and efficient alternative to traditional pencil-and-paper exams. The College of Engineering at the University of Illinois has been running a Computer-Based Testing Facility (CBTF) for more than four years. The CBTF has been running at scale (25+ courses, 50,000+ exams/semester) for the past couple years. The CBTF is a proctored, "locked-down" computer lab operated as a service to courses. The CBTF has changed how we teach, leading to improved student learning and enabling the introduction of more project and group work in large classes, because graduate TAs are freed from routine proctoring and grading.
The goal of the CBTF is to make assessment with exams better for everyone involved--students, faculty, and course staff. Four concepts are key to achieving this goal. First, by running the exams on computers, we can write complex, authentic (e.g., numeric, programming, graphical, design) questions that are auto-gradable, allowing us to test a broad set of learning objectives with minimal grading time and providing students with immediate feedback. Second, we write question generators that use randomness to produce a collection of problems, allowing us to give each student different questions and permitting the problem generators to be used semester after semester. Third, because each student has a unique exam, we allow students to schedule their exams at a time convenient to them within a specified day range, providing students flexibility and avoiding the need to manage conflict exams. Finally, because exam scheduling and proctoring is completely handled by the CBTF, once faculty have their exam content, it is no more effort to run more frequent, smaller exams, which reduces anxiety for some students, as well as offering second-chance exams to reduce failure rates by allowing struggling students an opportunity to review and demonstrate mastery of concepts that they missed on an exam.
In this talk, Craig Zilles will discuss the basic operation of the Illinois CBTF and the key components that make it work. He'll present findings on aggregate student behavior in the CBTF and data on increased learning gains and reduced failure rates in specific courses. He'll discuss mechanisms and policies for maintaining security, supporting testing accomodations, and minimizing faculty disruption. Finally, he'll present findings from surveys of faculty and students and discuss the cost of operating the CBTF and how it compares to traditional exams and online services.
Craig Zilles is an Associate Professor in the Computer Science department at the University of Illinois at Urbana-Champaign. His current research focuses on applying computing and data analytics to education, including the development of the Computer-based Testing Facility (CBTF). Previously, his research focused on the interaction between compilers and computer architecture, and he developed the first algorithm that allowed rendering arbitrary three-dimensional polygonal shapes for haptic interfaces (force-feedback human-computer interfaces). He received the IEEE Education Society's 2010 Mac Van Valkenburg Early Career Teaching Award and an NSF CAREER award. At Illinois, he has received a wide range of teaching awards, including a 2018 Campus Award for Excellence in Undergraduate Teaching, a 2013 Illinois Student Senate Teaching Excellence Award, and the College of Engineering's Rose Award (2007) and Everitt Award (2008) for Teaching Excellence. He holds 5 patents, and his research has been recognized by a best paper awards from ASPLOS in 2010 and 2013 and by selection for inclusion in the IEEE Micro Top Picks from the 2008 Computer Architecture Conferences.