Jan Costenbader

Adaptive Learning using the Knewton Engine in MyMathLab

We have been exploring the use of the Knewton Adaptive Learning engine built into Pearson’s MyMathLab. We began with a limited study during the summer of 2016 with a trial in 4 developmental math courses. The results from the trial courses using Knewton were compared to sections of the same courses in which the adaptive engine was not used. Before continuing, you may be wondering what is this adaptive learning?

According to “What Is Adaptive Learning,” a post from David Kunz in The Knewton Blog:

Adaptive learning makes content dynamic and interactive, placing the student at the center of his or her individual learning experience. The platform monitors how the student interacts with the system and learns, leveraging the enormous quantities of data generated by a student’s online interactions with ordinary (textbook-like) and extraordinary (game- and social-media-like) content, with teachers and peers, and with the system itself. It assesses not only what a student knows now, but also determines what activities and interactions, developed by which providers, delivered in what sequence and medium, most greatly increase the possibility of that student’s academic success.

Differentiated instruction lets students focus on the concepts that they are struggling with, rather than bogging them down with busywork on topics on which they have already demonstrated understanding. In addition to supporting students who are falling behind with targeted remediation, It also gives advanced students the opportunity to move forward at their own pace.

Knewton and MyMathLab

One of the most time-consuming aspects, aside from creating the engine itself, is mapping the pathway through the content, so that prerequisite knowledge is identified for each concept. Each objective in a MyMathLab section is mapped, through a rigorous process, to one or more prerequisite objectives. If a student masters a prerequisite objective but later forgets the concept when it is needed for mastery of a later objective, the Knewton engine, based on this mapping, will direct the student back for additional study on this previous objective. A simple mapping may look something like this:

Flow Chart: Knewton and MyMathLab mapping Summer 2016 Trial

Four courses in the Summer First Year Academic Success program were implemented using the Knewton engine. These were all developmental math courses and were offered in a compressed 4 or 5 week program. These were compared against regular sections of the same courses. The courses were structured as follows:

  • All use MyMathLab
  • Traditional Course
    • Lectures (either online or F2F)
    • MML Homework—60% score required to take corresponding quiz
    • MML Quizzes—three attempts possible—highest score
    • Midterm, Final
  • Knewton Adaptive Courses
    • Lectures (either online or F2F)
    • Adaptive Study plan—Mastery of x objectives required to take quiz
    • MML Quizzes—three attempts possible—highest score
    • Midterm, Final

Setting up the Study Plan

There was a fair amount of work on the part of the instructors to initially set up the study plan, and training was offered to each of them to assist in the development. Essentially, the process is as follows:

  • Set coverage to objective level
  • Enable Study Plan and set Mastery standards
  • Set question coverage and scoring
  • Set prerequisites
  • Define study plan
  • Review Settings

We had to first identify which sections and which objectives would be covered:

Sections and objectives identification

Next we defined the Mastery Standards. Note that the mastery standards default to 100%. This is not the recommended approach as students struggle to achieve 100% mastery. A lot of this is due to the unforgiving nature of the answer format in MyMathLab.

MyMathLab mastery level settings
Note that we also define the number of questions and level of difficulty for the Quiz Me, which is used to determine if the student masters that particular objective.

Next, we select the questions from each section that will determine the pool from which both the Quiz Me and practice problems will be drawn. Since the courses were taught using the traditional MyMathLab homework and quizzes, we selected the problems that were already used in the homework, section quizzes, and tests.

Each section has an associated quiz (separate from the Quiz Me). In the traditional sections, the student has to get a 60% on the section homework in order to be able to take the quiz. With the adaptive study plan, the homework was replaced by the study plan, and the student is then required to demonstrate mastery of the number of objectives specified for that section.

Edit quiz dialog window Step 1


Edit quiz dialog window Step 2
This is indicated to the student when they attempt to take a particular quiz:

Quiz alert message to student

The student view

When a student views the study plan for a particular section, they are presented for each objective with an opportunity to practice problems for that objective, and/or take the Quiz Me to demonstrate mastery of the objective. In some cases, as shown below, publisher video is available for each objective.

Student view of study plan

As the student passes each Quiz Me, the objective moves below the line to indicate objectives mastered. In this case, 3 of the 4 objectives have been mastered and the student is ready to take the quiz for the section.

Study plan student view showing objectives mastered

If a student fails to master the objective, the Quiz Me is grayed out and they must complete at least one practice problem to be able to take the Quiz Me again. Note that traditional MyMathLab help is available in the practice problems, but no learning aids are available in the Quiz Me.

Quiz Me grayed out to indicate objective not mastered

Preliminary Results

The summer trial was extremely limited, both in terms of the number of courses and the number of students. It was further confounded by differences in session length as well as the type of student. In the First Year program, we typically see entering freshmen, where this is the first university level course they have encountered.

  • Aligned to same instructor—regular vs adaptive
  • Very small sample—for summer courses 12-20 students
  • Difference in session length
    • 10 weeks regular sessions
    • 4 weeks Summer I
    • 5 weeks Summer II
  • Difference in types of students
    • FYAS
    • Regular session

Overall, our results tracked what we have seen in other studies of a similar nature, that is, the student scores were better and they spent less time on task overall. The time on task is reflected in the fact that if the students already understood the material, they could prove mastery by taking the Quiz Me—without having to do homework problems just to be able to take the quiz.

This was a fully online course, in both the adaptive and regular course.

FYSM 094 Adaptive Quiz Results

As noted, time on task was significantly less than the traditional course. In this particular course, one student finished the entire course in 8 days with a score of 98%!

Median Time Spent per Chapter MAT 094

Similar results were seen for the other courses:

MAT 100 Quiz Scores graph

 

Graphs of Median Time Spent MAT 100 and MAT 101 Quiz Scores

Median Time Comparison MAT 101 graph

One of the instructors commented on the student performance:

“Comparing results of the same class (MAT 100) from Autumn 2015 (with homework assignments) and Summer 2016 (with study plan) I came to conclusion that students perform better using the study plan, because they can work on their weakest part of the sections more in order to master them. Also at the same time they don’t need to spend a lot of time on parts of the sections where they are already familiar with the material.”

Student comments were likewise favorable:

  • Hi and thanks! I was nervous at first to take an online math class, especially since it’s my weakest subject but the study plan and quiz me has been helping me a lot! I find it really simple to use and it’s very helpful with the subject being taught.
  • I definitely do find them helpful, at first I tried to go right into taking some of the quizzes without practicing and I was unsuccessful, so yes I find them helpful to get the juices flowing.
  • I have been working hard on MyMathLab and taken lots of notes. I find that watching the video, taking the practice and quiz me, in that sequence, really helps me learn the material and prepare me for the real quiz.

Next steps

The initial trial was limited in scope. During the Winter of 2016–2017, we will be implementing the same trial in 4 courses with larger enrollments and traditional students. We are also including Math 130, the pre-calculus course. Other disciplines, such as Environmental Science, Chemistry, and Psychology are investigating the use of adaptive technologies built into their publisher materials.

Watch this space for the results coming this Spring.

Jan Costenbader

About Jan Costenbader

Jan came to DePaul from California State University, Chico in November of 2010. There, he taught Mathematics and developed an online hybrid Mathematics course for General Education Mathematics. He also assisted faculty in course design as an instructional designer. Currently, he provides instructional design consultation to the College of Science and Health, the Quantitative Reasoning program and several departments within the College of Liberal Arts and Social Sciences. In addition, he teaches fully online developmental Mathematics and blended Quantitive Reasoning courses.

Leave a Reply