Pittsburgh, PA—For a moment this week, the provost of Carnegie Mellon University looked a bit like a game show host as he grabbed the lever of an oversized switch and called on an audience to join him in a countdown—“5, 4, 3, 2, 1.” Then, he toggled the cardboard lever and declared open a new website, one that gave away software that took more than $100 million in grant funding to develop.

It was an unusually theatrical moment for a gathering to announce the release of software tools to help professors improve their teaching. But the organizers were playfully acknowledging the size of their project’s ambition—which they hope will spark a more data-driven and experimental approach to teaching at colleges around the country. And the flair was fitting, since success will end up being based not so much on how well the software works, but on how well its creators can attract momentum to their cause—and change the culture of the academic profession to make teaching an area professors are excited to make discoveries around.

The software tools released at the event are called the OpenSimon Toolkit, and they can help colleges create digital courseware and measure how well their course materials work. The various pieces can work together or stand alone, depending on what colleges need, and the source code will be made open so that colleges can customize the tools to fit their own needs.

“We can’t build something and throw it over the fence and expect it to work in your context,” Richard Scheines, dean of Carnegie Mellon’s College of Humanities and Social Sciences, told the audience of leaders from colleges around the U.S.

The university announced in March that it would release the software, but officials held this week’s gathering, along with a group called the Empirical Educator Project, to give an overview of the tools and how to use them.

The software list includes more than a dozen packages, including the university’s pioneering adaptive-learning project, the Open Learning Initiative, as well as a learning analytics platform LearnSphere.

Instead of spending time highlighting software features, though, many of the presenters outlined a way to bring the scientific method to the act of teaching. The idea is that each semester, professors will create hypotheses for what might lead to better learning in their courses, adapt their course materials to try to achieve that result, monitor student behavior to see if it worked, and then analyze the results and make adjustments based on what happened. The visual that was put on slide after slide was a circular flow chart with arrows moving between the words, Design, Develop, Deliver, Discover (and then back to Design).

Carnegie Mellon presented a process for how to bring the scientific method to teaching practices.

In one presentation, Ken Koedinger, director of CMU’s LearnLab, described a course in which the professor set out to answer the question, Which student choices of what to do within the course materials resulted in the most learning? The answer: students who spent the most time answering practice problems did far better than those who did things like watching lecture videos or on text explanations. “Learning by doing appears to have a 6x better [outcome] than learning by watching or reading,” Koedinger said. He and his colleagues published an academic paper with the finding called “Learning is Not a Spectator Sport.”

Koedinger said he now knows to encourage students to do problems—and he even presents his data to his classes to stress the point.

As much as CMU showed off its own successes at the event, it also repeatedly called for help. The word “community” came up again and again during the two-day event, as CMU organizers called on colleagues at other institutions to join them in adopting and improving not just the software, but also in developing playbooks for how best to use it and to encourage professors to take on the new task of “learning engineering.”

Reaction at the event was enthusiastic, but sober about the challenges of bringing the tools to their campuses.

“A lot of the science and the technology they use is spot on” for what colleges need, said Michael Bridges, executive director of online learning and strategy at Duquesne University. “They’ve developed the process and tools, and they’ve refined those and collected data. Now the question is, How do we get it out there?”

After the presentation he went over to his colleague, Brian Bolsinger, director of instructional technology at Duquesne, and asked for suggestions of which professor they could send to a summer workshop at CMU to learn to use the OLI software and other tools. The trick, the pair agreed, was finding just the right professor, who might go on to set an example that others will want to follow. But Bridges noted that dealing with analytics is not in every professor’s wheelhouse. “If it’s not accessible in a way that makes it a cost-benefit win for them, they’re not going to do it,” he said.

Bolsinger noted that Duquesne was already using the main tool CMU was talking about, the Online Learning Initiative platform, but only in a few courses. “The challenge is it’s hard to scale,” he confessed. “We found a couple of niche faculty that were really engaged and were interested in this and they ran with it, but it’s hard to get to a broader audience.” He said he was encouraged that the improvements CMU has made the software tools easier to use than in the past.

As Yakut Gazi, associate dean of learning systems at Georgia Tech, watched the presentations, she emailed the list of software tools to a colleague and asked which ones might make sense for the university to adopt.

“We’re really intrigued, and what I want to understand deeper is the mechanics of how this would work,” she said. The university gets an increasing amount of data in its online courses, especially from the large numbers of students taking its low-cost $7,000 Masters program in computer science. That data currently runs through a mix of systems, but she wanted to know if the CMU tools could help bring all the data together so that they could better learn from it. “So far the insights that we’ve generated are really superficial or, ‘yeah, duh.’ So I’m really intrigued. I think we need to figure out a way to put [the CMU tools] on top of what we have. This is taking all of us in the right direction.”

For her faculty, the challenge is providing the incentives to spend more time on teaching rather than research and consulting. “They can generate consulting money way faster than what they are doing for us, and there’s not enough money in our budgets to incentivize them in such a way that this is their priority.”

Erin DeSilva, assistant director of learning design and technology at Dartmouth College, said whether her college will adopt these tools is “TBD.”

“We’re still trying to find our place in this community. We’re small, and there isn’t a lot of pressure the way there can be in different places for change,” she added. “We might be one of those one or two or three research scientists or there.”

Other institutions at the even expressed a greater urgency in finding a way to use the tools. Coppin State University, for instance, a small historically black college in Baltimore, sent its president, Maria Thompson, who said she feels like her institution has to find a way to adopt these practices. “If you’re small and you have a limited budget, you have to find ways to improve what you’re doing, and we’re always on the lookout for technology that helps us multiply our effect,” she said.

And she said she was there to share some of the innovations at her campus. “I’ve democratized almost all of our data, and I’m probably the only campus here who has total transparency in data, so we can learn from each other.”

Even so, she acknowledged that making the shift to learning engineering will be tough. “There’s going to be a lot of cultural resistance,” Thompson said. “It’s just that simple.”

Why Now?

Plenty of others have tried in the past to bring the principles of engineering to college teaching, though with limited success. In fact, the effort at Carnegie Mellon is named for Herbert Simon, a longtime professor at Carnegie Mellon who won a Nobel Prize in economics and devoted his energy and academic capital to trying to spread his ideas about turning teaching from a solo sport to a team effort. But it didn’t catch on widely in his lifetime.

And many of the tools announced this week have been around for more than a decade, though they have not been open source and many have been recently improved to make them more ready for prime time.

“We tried OLI five years ago for intro biology,” said Marco Molinaro, assistant vice provost and director of the Center for Educational Effectiveness Center, at the University of California at Davis. While it worked well, he said, professors complained that they could not customize it to fit their courses. He was impressed that CMU has made it easy to author or modify, and he called the latest release “a really dramatic” improvement. “Now the question is: can I convince them to come back and try it again?” he said of the faculty he worked with before.

So why do Carnegie Mellon officials think they can make inroads now?

Michael Feldstein, the founder of the Empirical Educator Project who helped organize the meeting at CMU, argued that colleges are now under increased pressure to improve their teaching outcomes, and that is forcing institutions to take a closer look at these kinds of tools and approaches.

And he challenged participants as the event began to think of the gathering as more than just a time to sit and listen but not follow up once they get back to their campuses. “If we do not go home with something for our students, then everything we’ve done here has been a waste of time and money.”

For the last couple of hours of the meeting, participants broke into groups and brainstormed how they would take next steps.

One member of the audience asked whether the software being given away by CMU should be thought of as free like a free kitten, or free like free ice cream? In other words, is adopting the software going to require a long-term and expensive commitment, or is it something that can be easily consumed today.

Koedinger, director of CMU’s LearnLab, suggested thinking of the give-away more like a free plant, noting that implementing it will take care and attention, though not to the extent of bringing a baby animal home.

One official noted that the commercial alternative to what CMU has developed also costs colleges nothing, and requires less work on their part. That option is in fact the status quo: for professors to assign commercial textbooks and courseware add-ons that include online homework and quizzes.

In that scenario, professors simply assign the book and courseware and require students to buy them.

When colleges adopt commercial courseware, though, often it is a “major black box” as far as how the software is designed, and colleges often have little or no control of how the data is used, said Molinaro, of U.C. Davis.

And such courseware and homework systems have been in the news lately, with some professors and students complaining about being forced to use them. And the publishers that make them are merging and consolidating their offerings lately.

Sara Finney, associate director of the Center for Assessment and Research Studies at James Madison University, said substantial change will only come when colleges begin to signal that they value empirical teaching practices.

“It’s going to take a university saying, You need to be trained in curriculum and instruction,” she said. “But most institutions don’t do this.”