I read an article back in June (reference below) that prompted some memories and catalyzed some additional thoughts. In the mid-late aughts and early teens, when I was still serving as chair for doctoral students, I often had conversations like this:

Student (bursting into my office): I have an exciting idea for my dissertation research!

Me: Let’s hear it!

Student: I’ll study whether students learn better with OER than with traditional course materials!

Me (sighs): Sit down here and let’s have a chat…

At this point I would pull a random book down from my shelf and open to the copyright statement near the front of the book.

Me: If I cross out these lines about the publisher’s copyright and write “Licensed CC BY” above it, have I made this book more effective at supporting student learning?

Student: (ponders briefly) I guess not?

Me: Why not?

Student: (questioningly) Because you didn’t change the instructional design of the book in any way?

This conversation was a wonderful jumping off point to discuss the characteristics of an educational resource that actually function to support student learning. It generally didn’t take long for these doctoral students who were knee-deep in the literature about effective instructional design to understand that copyright status is not one of the characteristics of a learning resource that directly affects student learning. We would also discuss people’s uncontrollable urge to conduct media comparison studies – studies comparing student learning  in the context of iPads versus printed books, or online classes versus face to face classes, or interactive video disc versus televised courses, or some other setup in which the medium du jour is compared with the medium of yesteryear.

We’d talk about why these studies almost always result in “no significant difference.” In brief, this is because it matters far less which materials students are using than what students are doing with them. We learn by doing, so if you’re doing the same things (e.g., reading chapters and doing end of chapter homework) with a new set of materials that you had been doing with the old ones, you shouldn’t expect to find a difference in learning. There’s actually an entire website dedicated to the hundreds of media comparison studies  that have found no significant difference in student learning.

(And if you’re thinking, “just increasing access to course materials will make a significant difference in student learning!”, see this article where researchers from OpenStax “demonstrate that even if there is a learning benefit of [increasing access to] OER, standard research methods are unlikely to detect it.” In other words, when you read OER research that shows a difference in student learning, look harder – there’s likely more going on than just a change in materials. You’ve likely crossed over into the realm of OER-enabled pedagogy.)

Research that compares student learning with OER to student learning with traditionally copyrighted materials (TCM) are generally a form of media comparison study because the pedagogy generally stays the same after the switch from a proprietary textbook to an open textbook (meaning you’re only comparing the effects of the media themselves). All other things being equal there is no reason to expect that OER will support significantly better student learning than TCM. I spent several years actively discouraging students from doing this kind of research – until I realized how incredibly important it was and started doing it myself. Then my colleagues and I started inviting everyone whose attention we could get to join in the work.

So why did I come to believe this research is so important to do? First, some historical context.

When they first hear about OER, many faculty have a hard time believing that anything you can get online for free could be as valuable and effective as the traditionally-produced, incredibly expensive, traditionally copyrighted learning materials sold by publishers. This instinct is understandable, because in many cases it’s true that you get what you pay for. As OER started to grow in popularity, publishers’ adopted the strategy of reinforcing this natural impulse in order to create fear, uncertainty, and doubt (FUD) in the minds of faculty with regard to OER.   

This “you get what you pay for” campaign leveraged faculty’s widespread lack of understanding of instructional design to redefine “quality.” When it comes to materials whose designed purpose is to support learning, there can be one and only one meaningful definition of their quality – whether students who use the materials actually learn. As part of their PR attacks on OER, publishers shifted the quality conversation away from student learning and toward the publication process. In this narrative, “quality” becomes a function of how many peer reviewers read a book, how many technical writers edit a book, how many design professionals work on a book’s visual layout, images, and cover design, etc. Perhaps most importantly, by defining quality as equivalent to a specific production process, and describing that process primarily in terms of the dozens of highly paid professionals who participate in it, publishers tried to make it impossible for most OER to be “quality” – since OER are generally produced by a single author or a very small group of people. As soon as you accept the argument that quality equals a process that involves dozens of highly paid professionals, you unwittingly admit that the overwhelming majority of OER in the world can never be “quality.”

The “open textbooks” sub-movement emerged about the same time publishers were hitting their stride with this messaging. Whether consciously or not, many organizations that worked in open textbooks responded to the publishers’ FUD campaign by being very clear with faculty that, while their textbooks were openly licensed, they had gone to great lengths to ensure that their open textbooks were exactly like traditional textbooks in every other regard. The scope of coverage was the same. The sequence in which topics were introduced were the same. The interior design features were the same. The composition of the team that produced the books was the same. The peer review process was the same. Literally everything about the books were exactly the same except for the open license. In making this argument, open textbook publishers may have actually been reinforcing publishers’ messages about what constitutes “quality” and unwittingly working against the success of smaller, faculty-led OER initiatives.

Which brings us back to the question – if we already know the likely outcomes of these studies before we do them, why are media comparison studies of OER vs TCM so important to do? There’s nothing new for those who have expertise in instructional design to learn by doing OER vs TCM research. It is, essentially, simply the most recent addition to the growing body of media comparison studies that are bound to find no significant difference. And the repeated messaging that “open textbooks are exactly the same as proprietary textbooks, just free” certainly encourages faculty to continue using with a pedagogical approach that is ‘exactly the same,’ further strengthening the chances of finding no significant difference.

The critical thing to realize is – the overwhelming majority of people who will encounter OER in their professional life have never studied instructional design and have never heard of media comparison studies. It’s an oft-repeated truism that the overwhelming majority of faculty have no formal training whatsoever in teaching and learning. And far fewer of them have any formal training in issues related to the design of effective learning materials. For the faculty member, or department chair, or dean, or provost who is worried about what might happen to student learning when OER is adopted, you can’t provide them with a collection of research on iPads vs desktop computers or online courses vs classroom courses and ask them to extrapolate. For the knowledge about the impacts of OER on student learning to rapidly spread across the disciplines and actually take hold in the disciplines, the research needs to be replicated within the disciplines. Then it will be written in the languages that faculty and department chairs and deans and provosts speak natively. 

So when I see articles in places like the Journal of Forestry titled Student and Instructor Generated Open Educational Resources Compare Favorably to a Traditional Textbook, I’m not exasperated that someone has done yet another media comparison study. On the contrary, I’m excited that someone is helping to translate these important findings in ways that faculty in each discipline can easily understand. Ensuring that all faculty understand that typical OER are likely to support just as much (or as little) student learning as typical TCM is a critical first step in facilitating the widespread adoption of OER across all courses in all departments.

Once this research helps faculty open their minds to the possibilities of OER, and people and organizations start developing OER that no longer look or work like printed textbooks, and students and faculty start doing different things with them, then we can start doing research into the impacts of OER on student learning where we will expect to find differences. “Does student engagement in this OER-enabled pedagogy activity, which is possible only because my students and I are using OER, increase student learning?”