The most recent issue of IRRODL included an article titled Effectiveness of OER Use in First-Year Higher Education Students’ Mathematical Course Performance: A Case Study, by Juan I. Venegas-Muggli and Werner Westermann. Quoting from the article:

The main aims of this research were to examine the effect of OER use among higher education students and to analyze teacher and student views on OER use in order to better understand how these resources are used and valued. This was justified by the fact that there is a lack of empirical evidence to support expanding the use of OER. Moreover, recent societal demands to improve education quality in Chile have made this a relevant case study environment in which to examine the potentials of OER.

In relation to the first aim, the most important result is that students in face-to-face arithmetic/statistics courses using Khan Academy resources achieved significantly better exam grades than students who did not use any extra resources (p < 0.05) or those who used open textbooks as an extra resource (p < 0.01). The fact that the final exam was the same for everyone makes this a valid comparative measure of students’ performance.

As I read the article, I couldn’t help but reflect on a recent article in PLOS One by Phil Grimaldi and his colleagues at OpenStax, which challenged us to be more specific in our theorizing about why OER adoption might improve student learning. Despite popular rhetoric along the lines of “you can’t learn from materials you can’t afford,” their rather ingenious experiment showed that increasing access to learning materials by adopting OER instead of traditionally copyrighted resources (TCM) will almost never measurably improve learning. Learning will only improve measurably if you ALSO do something else.

So why did the Chilean math students who used Khan Academy do better than those who used either an open textbook or a traditionally copyrighted textbook? The underlying explanatory mechanism is extraordinarily simple. Students who have more opportunities to practice, receive feedback, reflect on that feedback, and practice again learn more than students who don’t have as many of those opportunities.  Whether you’re talking about conducting a literature review, playing a musical instrument, solving math problems, or literally anything else, practice with feedback is critical to learning.

Why, then, did the Chilean math students who used Khan Academy do better than those who used an open textbook? The students who used Khan Academy – specifically, students who used Khan Academy’s online homework system – had significantly more opportunities to practice, receive feedback, reflect, and practice again than students who used an open textbook. Unsurprisingly, the students with more opportunities to engage in online interactive practice did significantly better on the final exam. You have to read a bit between the lines of the article to see that students using Khan Academy weren’t just watching OER videos, but were also using the online interactive practice. I reached out to Werner and specifically verified this. He said teachers were also tracking student progress using Khan Academy’s analytics dashboards.

When Grimaldi et al. call on us to make the explanatory mechanisms underlying our research clearer (rather than just saying “OER was better”), this is the kind of clarification they’re calling for. If you didn’t know how Khan Academy worked before reading this article, you could easily have missed the role the online interactive practice system played in improving student learning and just thought “OER was better!”

This is not the first example, nor will it be the last example, of peer-reviewed research demonstrating that OER plus a system that provides opportunities to engage in online interactive practice results in better student learning than OER alone. And it shouldn’t surprise anyone that it’s true.

Research at CMU

A series of research papers from Carnegie Mellon going back almost 15 years has established this over and over again. In a 2016 article, Koedinger and colleagues wrote:

The “doer effect” is an association between the number of online interactive practice activities students do and their learning outcomes, that is not only statistically reliable but has much higher positive effects than other learning resources, such as watching videos or reading text… [We] provide generalizable evidence across four different courses involving over 12,500 students that the learning effect of doing [online interactive practice activities] is about six times greater than that of reading.

If you’re not familiar with educational research, 6x is a massive effect.

Other research from CMU explicitly connects the dots between (1) the significant impact of doing online interactive practice activities and (2) the somewhat common student practice of printing out their online learning materials.  Unsurprisingly, students who print out their online learning materials don’t do as many of the online interactive practice activities (“Volqs” in the path diagram below). Consequently, students who print do worse on both quizzes and final exams than their peers who work online.

For more research along these lines from CMU, read just about anything by Ken Koedinger.

If you’ve heard that research says print is superior to online in terms of supporting student learning, take a closer look at that research. It generally compares reading print to reading online. But the CMU research, the Chilean research, and related research is comparing doing online interactive practice to reading online (in the CMU case) and reading offline (in the Chilean case). Regardless of the format of the materials students are reading, actively engaging in online interactive practice is more effective than simply reading. Again, this should not be a surprise to anyone.

Homework Systems

Unfortunately, there’s something akin to a conspiracy theory floating around the open education community about the systems that provide these opportunities to students. The statement of the conspiracy goes something like this. “Since publishers can’t really sell static content for exorbitant prices anymore, they’ve started selling homework systems. Far from adding value to student learning, these systems simply put a paywall in front of activities required to pass the course – like submitting homework and doing exams – making it impossible for students to look for cheaper alternatives. These systems are really just attempts by publishers to increase revenues by forcing students to buy access codes that can’t be resold.”

As with every good conspiracy theory, there’s some truth in that statement. But there are some huge falsehoods, as well.

First, homework systems aren’t a recent development. They’re not a response to the way OER caused the bottom to fall out of the textbook market. They’ve been around for a very long time, and they emerged in order to serve market demands that textbooks did not meet (and do not meet, and cannot meet). First, they provide students the opportunity to engage in online interactive practice and therefore have the potential to greatly increase student learning as described above. Second, they automate a large portion of the grading in a course, giving faculty back a large amount of time to do higher value activities, like meeting with students during office hours (or evaluating OER to adopt in place of their current traditionally copyrighted textbook).

Second, publishers can’t force students to do anything. Ever. Publishers don’t get to make adoption decisions, only faculty get to make adoption decisions. And for every homework system out there, there are dozens of static textbooks available that faculty are completely free to choose. However, faculty choose to require homework systems for their courses because they see benefits for both their students and themselves. So, if you’re looking to criticize someone for “making students pay to submit their homework,” 100% of your criticism should be directed at faculty – they’re the only ones who can require students to submit their homework in a particular way.

Now, do these systems need to be priced unconscionably high, as publishers have done historically? Absolutely not. While there absolutely are real, ongoing costs associated with hosting, maintaining, and securing online systems that hold student data, these costs are something like an order of magnitude lower than what publishers have traditionally charged. So if you want to criticize homework systems, but you don’t want to criticize faculty for choosing to adopt learning materials that actually improve student learning, criticize publishers for the way they price these systems. That’s the part that publishers control and, as we have seen with textbooks, it’s something publishers can and will change in response to market pressure.

And we have to talk a little more about costs.

From Open to Free

You may not have discerned it yet, but the open education movement is pulling apart. There’s a growing group of people and institutions whose primary commitment has shifted from open to free. You can see this primarily in the emergence of the “zero textbook costs / Z degree / Zed cred” movement, which generally has a different primary goal than the OER movement. Rather than focusing primarily on supporting the adoption of OER in order to both lower costs and facilitate OER-enabled pedagogy and other open educational practices, these programs focus on eliminating learning materials costs for students without regard for whether the materials used by faculty are open or merely free.

Sister causes, to be sure. Siblings, certainly. But not the same movements.

The interaction of this shift from open to free with what we know (and are continuing to confirm) about systems that enable students to participate in online interactive practice has a terrible unintended consequence. As an example, see this question and answer from a FAQ for California’s ZTC grant program:

12. Do fees for access to online materials disqualify a program? Do materials fees for lab classes and would it matter if they are actual materials fees or items purchased in a bookstore?

Fees for access to online instructional materials like MyMathLab are not allowed because components of these type programs [sic] include e-textbooks. Fees for materials in certain labs are allowed such as materials kits in cosmetology courses/labs.

Given what the research says about the significant learning impact online interactive practice systems can have, ZTC and similar policies can cross a line that OER advocates said we would never cross – sacrificing student learning for cost savings. Traditionally, one of the cornerstones of OER advocacy has been that faculty should never adopt OER in order to save students money if they thought that doing so would harm student learning. If they’re not careful, ZTC and related policies can abandon that foundational commitment, replacing it with something closer to “free at any cost.” And I don’t believe any of us want to be painted with this “free at any cost” (to student learning) brush.

The way ZTC / Z degree / Zed cred program policies can make it difficult for students and faculty to use and benefit from these systems seems to be a genuinely unintended consequence. Fortunately, like all policies, these policies aren’t laws of nature and can be changed after unintended negative consequences like this are identified.

A Concluding Thought

I know it makes people unhappy when I say it, but we’ve got to stop talking about “textbooks” if we want to improve student learning. Textbooks are universally understood by both faculty and students to be static compilations of words, tables, and images. Yes, the first conversation with a faculty member (or policymaker) can be easier when you say “textbook” rather than “learning materials” or “learning resources” or “courseware” or something else. But the price of the lubricant you use to grease the skids of that first conversation is a restriction of imagination that precludes the possibility of something interactive. Something that will improve student learning. The language of “textbooks” prevents faculty, administrators, policymakers, students, and others – including OER advocates – from seeing the adjacent possible. Maybe we needed to use the language of “textbooks” 20 years ago when we first started advocating for faculty to adopt open content. But, for the sake of student learning, it’s time to re-evaluate that decision.