New Meta-Analysis on Debunking — Still an Unclear Way to Potency

A new meta-analysis on debunking was released last week, and I was hoping to get clear guidelines on how to debunk misinformation. Unfortunately, the science still seems somewhat equivocal about how to debunk. Either that, or there's just no magic bullet.

Bigstock-Clown-businessman-in-funny-con-67646752

Let's break this down. We all know misinformation exists. People lie, people get confused and share bad information, people don't vet their sources, incorrect information is easily spread, et cetera. Debunking is the act of providing information or inducing interactions intended to correct misinformation.

Misinformation is a huge problem in the world today, especially in our political systems. Democracy is difficult if political debate and citizen conversations are infused with bad information. Misinformation is also a huge problem for citizens themselves and for organizations. People who hear false health-related information can make themselves sick. Organizations who have employees who make decisions based on bad information, can hurt the bottom line.

In the workplace learning field, there's a ton of misinformation that has incredibly damaging effects. People believe in the witchcraft of learning styles, neuroscience snake oil, traditional smile sheets, and all kinds of bogus information.

It would be nice if misinformation could be easily thwarted, but too often it lingers. For example, the idea that people remember 10% of what they read, 20% of what they hear, 30% of what they see, etc., has been around since 1913 if not before, but it still gets passed around every year on bastardized versions of Dale's Cone.

A meta-analysis is a scientific study that compiles many other scientific studies using advanced statistical procedures to enable overall conclusions to be drawn. The study I reviewed (the one that was made available online last week) is:

Chan, M. S., Jones, C. R., Jamieson, K. H., & Albarracin, D. (2017). Debunking: A meta-analysis of the psychological efficacy of messages countering misinformation. Psychological Science, early online publication (print page numbers not yet determined). Available here (if you have journal access: http://journals.sagepub.com/doi/10.1177/0956797617714579).

This study compiled scientific studies that:

  1. First presented people with misinformation (except a control group that got no misinformation).
  2. Then presented them with a debunking procedure.
  3. Then looked at what effect the debunking procedure had on people's beliefs.

There are three types of effects examined in the study:

  1. Misinformation effect = Difference between the group that just got misinformation and a control group that didn't get misinformation. This determined how much the misinformation hurt.

  2. Debunking effect = Difference between the group that just got misinformation and a group that got misinformation and later debunking. This determined how much debunking could lesson the effects of the misinformation.

  3. Misinformation-Persistence effect = Difference between the group that got misinformation-and-debunking and the control group that didn't get misinformation. This determined how much debunking could fully reverse the effects of the misinformation.

They looked at three sets of factors.

First, the study examined what happens when people encounter misinformation. They found that the more people thought of explanations for the false information, the more they would believe this misinformation later, even in the face of debunking. From a practical standpoint then, if people are receiving misinformation, we should hope they don't think too deeply about it. Of course, this is largely out of our control as learning practitioners, because people come to us after they've gotten misinformation. On the other hand, it may provide hints for us as we use knowledge management or social media. The research findings suggest that we might need to intervene immediately when bad information is encountered to prevent people from elaborating on the misinformation.

Second, the meta-analysis examined whether debunking messages that included procedures to induce people to make counter-arguments to the misinformation would outperform debunking messages that did not include such procedures (or that included less potent counter-argument-inducing procedures). They found consistent benefits to these counter-argument inducing procedures. These procedures helped reduce misinformation. This suggests strongly that debunking should induce counter-arguments to the misinformation. And though specific mechanisms for doing this may be difficult to design, it is probably not enough to present the counter-arguments ourselves without getting our learners to fully process the counter-arguments themselves to some sufficient level of mathemagenic (learning-producing) processing.

Third, the meta-analysis looked at whether debunking messages that included explanatory information for why the misinformation was wrong would outperform debunking messages that included just contradictory claims (for example, statements to the effect that the misinformation was wrong). They found mixed results here. Providing debunking messages with explanatory information was more effective in debunking misinformation (getting people to move from being misinformed to being less misinformed), but these more explanatory messages were actually less effective in fully ridding people of the misinformation. This was a conflicting finding and so it's not clear whether greater explanations make a difference, or how they might be designed to make a difference. One wild conjecture. Perhaps where the explanations can induce relevant counter-arguments to the misinformation, they will be effective.

Overall, I came away disappointed that we haven't been able to learn more about how to debunk. This is NOT these researchers' fault. The data is the data. Rather, the research community as a whole has to double down on debunking and persuasion and figure out what works.

People certainly change their minds on heartfelt issues. Just think about the acceptance of gays and lesbians over the last twenty years. Dramatic changes! Many people are much more open and embracing. Well, how the hell did this happen? Some people died out, but many other people's minds were changed.

My point is that misinformation cannot possibly be a permanent condition and it behooves the world to focus resources on fixing this problem -- because it's freakin' huge!

------------

Note that a review of this research in the New York Times painted this in a more optimistic light.

------------

Some additional thoughts (added one day after original post).

To do a thorough job of analyzing any research paradigm, we should, of course, go beyond meta-analyses to the original studies being meta-analyzed. Most of us don't have time for that, so we often take the short-cut of just reading the meta-analysis or just reading research reviews, etc. This is generally okay, but there is a caveat that we might be missing something important.

One thing that struck me in reading the meta-analysis is that the authors commented on the typical experimental paradigm used in the research. It appeared that the actual experiment might have lasted 30 minutes or less, maybe 60 minutes at most. This includes reading (learning) the misinformation, getting a ten-minute distractor task, and answering a few questions (some treatment manipulations, that is, types of debunking methods; plus the assessment of their final state of belief through answers to questions). To ensure I wasn't misinterpreting the authors' message that the experiments were short, I looked at several of the studies compiled in the meta-analysis. The research I looked at used very short experimental sessions. Here is one of the treatments the experimental participants received (it includes both misinformation and a corrective, so it is one of the longer treatments):

Health Care Reform and Death Panels: Setting the Record Straight

By JONATHAN G. PRATT
Published: November 15, 2009

WASHINGTON, DC – With health care reform in full swing, politicians and citizen groups are taking a close look at the provisions in the Affordable Health Care for America Act (H.R. 3962) and the accompanying Medicare Physician Payment Reform Act (H.R. 3961).

Discussion has focused on whether Congress intends to establish “death panels” to determine whether or not seniors can get access to end-of-life medical care. Some have speculated that these panels will force the elderly and ailing into accepting minimal end-of-life care to reduce health care costs. Concerns have been raised that hospitals will be forced to withhold treatments simply because they are costly, even if they extend the life of the patient. Now talking heads and politicians are getting into the act.

Betsy McCaughey, the former Lieutenant Governor of New York State has warned that the bills contain provisions that would make it mandatory that “people in Medicare have a required counseling session that will tell them how to end their life sooner.”

Iowa Senator Chuck Grassley, the ranking Republican member of the Senate Finance Committee, chimed into the debate as well at a town-hall meeting, telling a questioner, “You have every right to fear…[You] should not have a government-run plan to decide when to pull the plug on Grandma.”

However, a close examination of the bill by non-partisan organizations reveals that the controversial proposals are not death panels at all. They are nothing more than a provision that allows Medicare to pay for voluntary counseling.

The American Medical Association and the National Hospice and Palliative Care Organization support the provision. For years, federal laws and policies have encouraged Americans to think ahead about end-of-life decisions.

The bills allow Medicare to pay doctors to provide information about living wills, pain medication, and hospice care. John Rother, executive vice president of AARP, the seniors’ lobby, repeatedly has declared the “death panel” rumors false.

The new provision is similar to a proposal in the last Congress to cover an end-of-life planning consultation. That bill was co-sponsored by three Republicans, including John Isakson, a Republican Senator from Georgia.

Speaking about the end of life provisions, Senator Isakson has said, “It's voluntary. Every state in America has an end of life directive or durable power of attorney provision… someone said Sarah Palin's web site had talked about the House bill having death panels on it where people would be euthanized. How someone could take an end of life directive or a living will as that is nuts.”

That's it. That's the experimental treatment.

Are we truly to believe that such short exposures are representative of real-world debunking? Surely not! In the real world, people who get misinformation often hold that misinformation over months or years while occasionally thinking about the misinformation again or encountering additional supportive misinformation or non-supportive information that may modify their initial beliefs in the misinformation. This all happens and then we try our debunking treatments.

Finally, it should be emphasized that the meta-analysis also only compiled eight research articles, many using the same (or similar) experimental paradigm. This is further inducement to skepticism. We should be very skeptical of these findings and my plea above for more study of debunking -- especially in more ecologically-valid situations -- is reinforced!

 

 

 

 

Major Research Review on eLearning Effectiveness

Is elearning effective? As effective as classroom instruction -- more or less effective? What about blended learning -- when elearning and classroom learning are combined?

 

ELearning Research Report Cover 2017.


These critical questions have now been answered and are available in the research report, Does eLearning Work? What the Scientific Research Says!

In this research review, I looked at meta-analyses and individual research studies, and was able to derive clear conclusions. The report is available for free, it includes an executive summary, and research jargon is kept to a minimum.

 

Click here to download the report...

 

 

 

Note that the August 10, 2017 version of this report incorrectly cited the Rowland (2014) study in a footnote and omitted it from the list of research citations. These issues were fixed on August 11, 2017. Special thanks to Elizabeth Dalton who notified me of the issues.

 

Prompting Learning When Our Learners Play Games

Another research brief. Answer the question and only then read what the research says:

 

In a recent study with teenagers playing a game to learn history, adding the learning instructions hurt learning outcomes for questions that assessed transfer, but NOT recall. The first choice hurt transfer but not recall. Give yourself some credit if you chose the second or third choices.

Caveats:

  • This is only one study.
  • It was done using only one type of learner.
  • It was done using only one type of learning method.
  • It was done with teenagers.

Important Point:

  • Don't assume that adding instructions to encourage learning will facilitate learning.

Research:

Hawlitschek, A., & Joeckel, S. (2017). Increasing the effectiveness of digital educational games: The effects of a learning instruction on students’ learning, motivation and cognitive load. Computers in Human Behavior, 72, 79-86.

Research Reviews of the Spacing Effect.

I've been following the spacing effect for over a decade, writing a research-to-practice report in 2006, and recommending the spacing effects to my clients and in the guise of subscription learning (threaded microlearning).

One of the fascinating things is that researchers continue to be fascinated with the spacing effect producing about 10 new studies every year and many research reviews.

Here are a list of the research reviews from most recent to earliest.

  • Maddox, G. B. (2016). Understanding the underlying mechanism of the spacing effect in verbal learning: A case for encoding variability and study-phase retrieval. Journal of Cognitive Psychology, 28(6), 684-706.
  • Vlach, H. A. (2014). The spacing effect in children's generalization of knowledge: Allowing children time to forget promotes their ability to learn. Child Development Perspectives, 8(3), 163-168.
  • Küpper-Tetzel, C. E. (2014). Understanding the distributed practice effect: Strong effects on weak theoretical grounds. Zeitschrift für Psychologie, 222(2), 71-81.
  • Carpenter, S. K. (2014). Spacing and interleaving of study and practice. In V. A. Benassi, C. E. Overson, & C. M. Hakala (Eds.), Applying science of learning in education: Infusing psychological science into the curriculum (pp. 131-141). Washington, DC: Society for the Teaching of Psychology.
  • Toppino, T. C., & Gerbier, E. (2014). About practice: Repetition, spacing, and abstraction. In B. H. Ross (Ed.), The psychology of learning and motivation: Vol. 60. The psychology of learning and motivation (pp. 113-189). San Diego, CA: Elsevier Academic Press.
  • Carpenter, S. K., Cepeda, N. J., Rohrer, D., Kang, S. H. K., & Pashler, H. (2012). Using spacing to enhance diverse forms of learning: Review of recent research and implications for instruction. Educational Psychology Review, 24(3), 369-378.
  • Kornmeier, J., & Sosic-Vasic, Z. (2012). Parallels between spacing effects during behavioral and cellular learning. Frontiers in Human Neuroscience, 6, Article ID 203.
  • Delaney, P. F., Verkoeijen, P. P. J. L., & Spirgel, A. (2010). Spacing and testing effects: A deeply critical, lengthy, and at times discursive review of the literature. In B. H. Ross (Ed.), The psychology of learning and motivation: Vol. 53. The psychology of learning and motivation: Advances in research and theory (pp. 63-147).
  • Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354-380.
  • Janiszewski, C., Noel, H., & Sawyer, A. G. (2003). A Meta-analysis of the Spacing Effect in Verbal Learning: Implications for Research on Advertising Repetition and Consumer Memory. Journal of Consumer Research, 30(1), 138-149.
  • Dempster, F. N., & Farris, R. (1990). The spacing effect: Research and practice. Journal of Research & Development in Education, 23(2), 97-101.
  • Underwood, B. J. (1961). Ten years of massed practice on distributed practice. Psychological Review, 68(4), 229-247.
  • Ruch, T. C. (1928). Factors influencing the relative economy of massed and distributed practice in learning. Psychological Review, 35(1), 19-45.

 

 

Giving Learners More Control in eLearning: What Does the Research Say?

Does giving learners more control of the way they navigate through an elearning program help them or hurt them?

Before I tell you what the research says, challenge yourself with this one-item quiz question:

 

 

How did you do? I ask because I want to give you maximum control of this learning experience.

Or maybe, I just like telling a joke. SMILE.

A recent meta-analysis (a scientific study that looks at many other scientific studies) found NO benefit for learner control. And contrary to what we were told by Malcolm Knowles and all those who tried to sell us on the adult-learner-knows-best baloney, the meta-analysis showed slightly more of a tendency for learner control to be beneficial for kids, NOT adults (but still a virtually non-existent benefit).

Our job then as elearning designers is NOT to give up control to our learners, but to design a learning experience that uses proven research-based techniques to guide learners through an effective repertoire of learning experiences. Certainly, the research finding shouldn't be construed to mean that we should never give learners control, but it does mean that as an over-riding design principle, it's a bad idea.

Interestingly, the meta-analysis broke down elearning into its methodological parts, and found no benefit to learner control in any of the components.

"The current study, in keeping with the previous meta-analysis (Niemiec et al., 1996), found near zero effects for all components of instruction (pacing, time, sequence, practice, review). Thus, there does not seem to be an advantage to giving the learner control over any particular instructional component." (p. 404)

Of course, with research as with most things in life, some circumspection should be in order. My first worry is that research on elearning tends to be done on very short learning programs where learner motivation doesn't really come into play. Who can't keep attentive for 15 minutes, maintaining their motivation? Some learner control might help in longer learning programs where it might support motivation to engage the learning. Also, we wouldn't want to throw out the idea of learner control completely. There may be some specific opportunities where it is worthwhile.

 

The Research Reviewed

Karich, A. C., Burns, M. K., & Maki, K. E. (2014). Updated meta-analysis of learner control within educational technology. Review of Educational Research, 84(3), 392-410.

 

Remember This!

If you're developing learning or developing your learning team, don't forget to seek out trusted research-to-practice experts to help you.

Five Reasons Learners Experience the Spacing Effect

The spacing effect, if not the most studied learning factor, is certainly in the top five. As Harry Bahrick and Lynda Hall said in 2005, “The spacing effect is one of the oldest and best documented phenomena in the history of learning and memory research.”

The spacing effect is the finding that repetitions that are spaced over time produce better long-term remembering than the exact same repetitions spaced over a shorter amount of time or massed all together.

About 10 new scientific studies are carried out each year on the spacing effect (I counted 31 in the last three years). Why such frenzied dedication to exploring the spacing effect? Scientists want to know what causes it! It's really rather fascinating!

To prepare for my upcoming conference presentation at the UK Learning Technologies conference, where they've asked me to speak on the spacing effect as it relates to mobile learning and microlearning, I'm doing another review of the scientific research. Here, on my blog, I'll share tidbits of my translational research effort, especially when I find articles that are particularly interesting or informative.

In this post, I'm looking at Geoffrey Maddox's review of the spacing research, which was published just last year in 2016 and is the most recent review available (although given the interest in spacing, there are many reviews in the scientific literature). He does a spectacular job making sense of the many strands of research.

In it, he finds that there are six main theories for why the spacing effect occurs. I've simplified his list into five theoretical explanations and I've ignored his somewhat jargony labels to help normal folks like me and you grok the meaning.

Five Theoretical Explanations for the Spacing Effect

1.Spacing Prompts More Attention
Learners may exert more attention to spaced items (compared with massed items).

2.Spacing Prompts Retrieval
Learners may be forced to retrieve spaced items (compared with massed items that need no retrieval—because they are still top of mind).

3.Spacing Prompts More-Difficult Retrieval
Learners may be prompted to engage in more difficult retrieval of spaced items (compared with massed items) and longer-spaced items (compared with shorter-spaced items).

4.Spacing Involves More Contextual Variability
Learners may create more retrieval routes (or more varied retrieval routes) when prompted with spaced items (compared with massed items).

5.Spacing Prompts Retrieval and Variability
Spacing benefits learners through both retrieval and variability, but variability, because it induces weaker traces may lead to more retrieval failure, thus lowering retrieval rates when spaced intervals are too long.

Maddox concludes that the only one that even comes close to explaining all the phenomenon in the scientific literature on spacing is the last one, which is really a combination of #2 and #4.

Count me as a skeptic. I just think we may be asking too much to push ourselves toward a unifying theory of spacing at this point. While a ton of research has been done, there are so many aspects to spacing and so much more authentically realistic research left to do that we ought to hold off on tying a pretty bow around one theory or another.

Evidence supporting my skepticism was found in the very next article that I read, where Metcalfe and Xu found more mind-wandering during massed practice than during spaced practice. This would fall into theory #1 above, not 5 (as Maddox recommended) or #2 or #4, which comprise 5.

Practical Implications

This article was not focused on providing practical implications, so it's probably too much to ask of it. Nevertheless, it does show the complexity of spacing at a cognitive level.

Also, Maddox was pretty clear in describing how robust the scientific research is in terms of the spacing effect. He wrote, Because of its robustness, the spacing effect has the potential to be applied across a variety of contexts as a way of improving learning and memory.”

He also detailed the ways that the science of spacing is so strong, including the following:

  • The spacing effect is: “observed in different animal species,”

  • “across the human lifespan”

  • “with numerous experimental manipulations”

  • “observed with educationally relevant verbal materials”

  • “observed in the classroom”

  • and observed with memory impaired populations”

We as learning professionals can conclude that the spacing effect (1) is real, (2) that it applies to all human beings, (3) that it is relevant to most situations, (4) that it is a powerful learning factor, and (5) that we ought to be utilizing it in our learning designs!

So folks, as I wrote in 2006, we ought to be figure out ways to Space Learning Over Time, using spaced repetitions, perhaps in a subscription-learning format.

 

Research Cited:

Bahrick, H. P., & Hall, L. K. (2005). The importance of retrieval failures to long-term retention: A metacognitive explanation of the spacing effect. Journal of Memory and Language, 52, 566-577.

Maddox, G. B. (2016). Understanding the underlying mechanism of the spacing effect in verbal learning: A case for encoding variability and study-phase retrieval. Journal of Cognitive Psychology, 28(6), 684-706.

Metcalfe, J., & Xu, J. (2016). People mind wander more during massed than spaced inductive learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42(6), 978-984.

Thalheimer, W. (2006, February). Spacing Learning Events Over Time: What the Research Says. Available at: http://work-learning.com/catalog.html.

Will’s History of Spacing Out on Learning

As you know, if you've dabbled into my work for a few years, I've closely followed the research finding, "The Spacing Effect," both from a research perspective and a practical perspective. Indeed, I was one of the first in the workplace learning field to recognize its practical significance, which I wrote about as early as 2002. In 2006 I published the research-to-practice report entitled Spacing Learning Over Time, which should -- if there was justice in the world (LOL) or viable trade organizations (OUCH) -- be enshrined in the Learning and Development Hall of Fame. Snickers are welcome. Taza Chocolate even better. A few years ago, still wanting to advocate for the practical use of the spacing effect, I began speaking about Subscription Learning at conferences and I developed a website (SubscriptionLearning.com) to encourage folks in the learning field to utilize the spacing effect in their learning designs. 

I am grateful to the enlightened organizations who have supported my work over the years and specifically to the individuals who continue to encourage the reading of the 2006 research report. Feel free to share yourself (http://willthalheimer.typepad.com/files/spacing_learning_over_time_2006.pdf)

Now in 2017, I am grateful to another organization, Learning Technologies (in the UK) who is sponsoring me to speak on the spacing effect at their conference starting in a few weeks. As part of my efforts, I am developing a new presentation and I am updating my research compilation on the spacing effect. Stay tuned to this blog as I'm likely to share a few of my findings as I dig into the research.

Indeed, the research on spacing is some of the most interesting I've studied over the years. The first thing that fascinates is that there is so much damn research on the spacing effect, also referred to as spaced learning, distributed practice, and interleaving. In 1992, Bruce and Bahrick counted up the number of scientific studies on spacing and found over 300 articles at that time. Every year, there are more and more scientific articles published on spacing. By my rough count of journal articles cited on PsycINFO (a primary social-science database), over the last three years there have been 31 new articles published on the spacing effect (7 in 2014, 14 in 2015, and 10 in 2016).

One of the main reasons that so many research articles are published on the spacing effect is that the phenomenon is so intriguing. Why would spacing repetitions over time produce so much more remembering than giving the learners the exact same repetitions but simply massing them all at once or spacing them with less time in between? Freakin' fascinating! So researchers keep digging into the complexities.

Harry Bahrick and Lynda Hall announced in 2005 that, “The spacing effect is one of the oldest and best documented phenomena in the history of learning and memory research.” And, just last year in a scientific review article, Geoffrey Maddox wrote, Because of its robustness, the spacing effect has the potential to be applied across a variety of contexts as a way of improving learning and memory.”

Stay tuned, I'll be spacing my research compilations over time...

 

Research

Bahrick, H. P., & Hall, L. K. (2005). The importance of retrieval failures to long-term retention: A metacognitive explanation of the spacing effect. Journal of Memory and Language, 52, 566-577.

Maddox, G. B. (2016). Understanding the underlying mechanism of the spacing effect in verbal learning: A case for encoding variability and study-phase retrieval. Journal of Cognitive Psychology, 28(6), 684-706.

Thalheimer, W. (2006, February). Spacing Learning Events Over Time: What the Research Says. Available at: http://work-learning.com/catalog.html.

 

Fitness Training Supports Improved Cognitive Functioning in Older Adults

A 2003 meta-analysis found that fitness training was likely to improve cognitive functioning in older adults.

I'm reprising this because it is one of Psychological Science's most cited articles as recently as September 1, 2016.

Fitness and Aging

The researchers examined 18 scientific studies and 197 separate effect sizes. They categorized measures of cognitive functioning into four categories as depicted above in the graph, including:

  • Executive functioning (the ability to plan, schedule, and generally engage in high-level decision-making).
  • Controlled processing (the ability to engage in simple decision-making).
  • Visuospatial processing (the ability to transform visual or spatial information).
  • Speed processing (the ability to make quick reactions).

As you can see in the graph above, overall the groups that exercised outperformed those who didn't.

 

Some Details:

  • Results were stronger for people 66-80 than for those 55-65 (judged by effect size), although all groups showed significant benefits from exercise.
  • Exercise for less than 30 minutes produced very little benefit compared to exercise for 30-60 minutes.
  • Females seemed to get more benefits from exercising, but the way comparisons were made makes this conclusion somewhat sketchy.
  • Those who engaged in both weight-training and cardio-training had slightly better results than those who did cardio alone.

 

Research Citation:

Colcombe, S., & Kramer, A. F. (2003). Fitness effects on the cognitive function of older adults: A meta-analytic study. Psychological Science, 14, 125-130.

 

More Information

Check out a 2009 blog post I wrote on aging and cognition, and test your knowledge with the quiz!!

It’s Behavior Change Stupid! Transfer that!

Those of us in the learning professions are naturally enamored with the power of learning. This is all fine and good--learning is necessary for human survival and for our most enlightened achievements--but too narrow a focus on learning misses a key responsibility. Indeed, learning without sustained behavior change is like feeding a man who's planning to jump off a bridge. Nice, but largely besides the point. 

The bottom line is that we learning professionals must not only look to the science of learning, but also the science of behavior change.

My friend and colleague Julie Dirksen has been thinking about behavior change for years. Here's a recent article she wrote:

Here is another recent resource on behavior change:

It's good to keep this all in perspective. Science often moves slowly and in fits and starts. There is great promise in the many and varied research areas under study. You can see this most fully in the health-behavior-change field. There's a ton of research being done. Here's a quick list of research being done on behavior change:

  • Cardiac health
  • Obesity
  • Clean cooking
  • Asthma
  • HIV and STD prevention
  • College-student drinking
  • Cancer prevention
  • Child survival
  • Recycling
  • Use of hotel towels
  • Young-driver distraction
  • Hand washing
  • Encourage walking and cycling
  • Smoking cessation
  • Healthy pregnancy behaviors
  • Promoting physical activity

Okay, the list is almost endless.

One of the findings is not surprising. Lasting behavior change is very difficult. Think how hard it is to lose weight and keep it off, or stop an internet addition. So it's great that researchers are looking into this.

In the learning field, we have our own version of behavior-change research. It's called transfer. We've already learned a lot about how to get people to transfer what they've learned back to their jobs or into their lives. We're not done learning, of course.

One thing we do know is that training by itself is rarely sufficient to produce lasting change. Sometimes our learners will take what they've learned, put it immediately into practice, deepen their own learning and continue to learn and engage and use what they've learned over time. Too often, they forget, they get distracted, they get no support.

 

Summary

My four messages to you are these:

  1. Keep your eyes open for Behavior Change Research.
  2. Keep your eyes open for Transfer-of-Learning Research.
  3. Don't be a fool in thinking that training/learning is enough.
  4. We learning professionals have a responsibility to enable usable behavior change.

 

Interview with Karl Kapp on Games, Gamification, and LEARNING!

Dr. Karl Kapp is one of the learning field's best research-to-practice gurus! Legendary for his generosity and indefatigable energy, it is my pleasure to interview him for his wisdom on games, gamification, and their intersection.

His books on games and gamification are wonderful. You can click on the images below to view them on Amazon.com.

 
 

The following is a master class on games and learning:

Will (Question 1):

Karl, you’ve written a definitive exploration of Gamification in your book, The Gamification of Learning and Instruction: Game-Based Methods and Strategies for Training and Education. As I read your book I was struck by your insistence that Gamification “is not the superficial addition of points, rewards, and badges to learning experiences.” What the heck are you talking about? Everybody knows that gamification is all about leaderboards, or so the marketplace would make us believe… [WINK, WINK] What are you getting at in your repeated warning that gamification is more complex than we might think?

Karl:

If you examine why people play games, the reasons are many, but often players talk about the sense of mastery, the enjoyment of overcoming a challenge, the thrill of winning and the joy of exploring the environment. They talk about how they moved from one level to another or how they encountered a “boss level” and defeated the boss after several attempts or how they strategized a certain way to accomplish the goal of winning the game. Or they describe how they allocated resources so they could defeat a difficult opponent. Rarely, if ever, do people who play games talk about the thrill of earning a point or the joy of being number seven on the leaderboard or the excitement of earning a badge just for showing up.

The elements of points, badges and leaderboards (PBLs) are the least exciting and enticing elements of playing games. So there is no way we should lead with those items when gamifying instruction. Sure PBLs play a role in making a game more understandable or in determining how far away a player is from the “best” but they do little to internally motivate players by themselves. Reliance solely on the PBL elements of games to drive learner engagement is not sustainable and not even what makes games motivational or engaging. It’s the wrong approach to learning and motivation. It’s superficial; it’s not deep enough to have lasting meaning.

Instead, we need to look at the more intrinsically motivating and deeper elements of games such as: challenge, mystery, story, constructive feedback (meaningful consequences) strategy, socialization and other elements that make games inherently engaging. We miss a large opportunity when we limit our “game thinking” to points, badges and leaderboards. We need to expand our thinking to include elements that truly engage a player and draw them into a game. These are the things that make games fun and frustrating and worth our investment in time.

 

Will (Question 2):

You wrote that “too many elements of reality and the game ceases to be engaging,”—and I’m probably taking this out of context—but I wonder if that is true in all cases? For example, I can imagine a realistic flight simulator for fighter pilots that creates an almost perfect replica of the cockpit, g-forces, and more, that would be highly engaging… On the other hand, my 13-year-daughter got me hooked on Tanki, an online tank shot-em-up game, and there are very few elements of reality in the game—and I, unfortunately, find it very engaging. Is it different for novices and experts? Are the recommendations for perceptual fidelity different for different topic areas, different learning goals, et cetera?

Karl:

A while ago, I read a fake advertisement for a military game. It was a parody. The fake game description described how the “ultra-realistic” military game would be hours of fun because it was just like actually being in the military. The description told the player that he or she would have hours of fun walking to the mess hall, maintaining equipment, getting gasoline for the jeep, washing boots, patrolling and zigging and cleaning latrines. Now none of these things are really fun, in fact, they are boring but they are part of the life of being in the military. Military games don’t include these mundane activities. Instead, you are always battling an enemy or strategizing what to do next. The actions that a military force performs 95% of the time are not included in the game because they are too boring.

 If games where 100% realistic, they would not be fun. So, instead, games are an abstraction of reality because they focus on things within reality that can be made engaging or interesting. If a game reflected reality 100%, there would be boring game play. Now certainly, games can be designed to “improve” reality and make it more fun. In the game, The Sims, you wake up, get dressed and go to work which all seems pretty mundane. However, these realistic activities in The Sims are an abstraction of the tasks you actually perform. The layer of abstraction makes the game more exciting, engaging and fun. But in either the military game case or The Sims, too much reality is not fun.

The flight simulator needs to be 100% realistic because it’s not really a game (although people do play it as a game) but the real purpose of a simulation is training and perfection of skills. A flight simulator can be fun for some people to “play” but in a 100% realistic simulator, if you don’t know what you are doing, it’s boring because you keep crashing. For someone who doesn’t know how to fly, like me. If you made a World War II air battle game which had 100% realistic controls for my airplane, it wouldn’t be fun. In game design, we need to balance elements of reality with the learning goal and the element of engagement.

For some people, a simulator can be highly engaging because the learner is performing the task she would do on the job. So there needs to be a balance in games and simulations to have the right amount of reality for the goals you are trying to achieve.

 

Will (Question 3):

In developing a learning game, what should come first, the game or the goals (of learning)?

Karl:

Learning goals must come first and must remain at the forefront of the game design process. Too often I see the mistake of a design team becoming too focused on games elements and losing site of the learning goals. In our field, we are paid to help people learn, not to entertain them. Learning first.

Having said that, you can’t ignore or treat the game elements as second class citizens, you can’t bolt-on a points system and think you have now developed a fun game—you haven’t. The best process involves simultaneously integrating game mechanics and learning elements. It’s tricky and not a lot of instructional designers have experience or training in this area but it’s critical to have integration of game and learning elements, the two need to be designed together. Neither can be an afterthought.

 

Will (Question 4):

Later we’ll talk about the research you’ve uncovered about the effectiveness of games. As I peruse the literature on games, the focus is mostly on the potential benefits of games. But what about drawbacks? I, for one, “waste” a ton of time playing games. Opportunity costs are certainly one issue, but maybe there are other drawbacks as well, including addiction to the endorphins and adrenaline; a heightened state of engagement during gaming that may make other aspects of living – or learning – seem less interesting, engaging. What about learning bad ideas, being desensitized to violence, sexual predation, or other anti-social behaviors? Are there downsides to games? And, in your opinion, has the research to date done enough to examine negative consequences of games?

Yes, games can have horrible, anti-social content. They can also have wonderful, pro-social content. In fact, a growing area of game research focuses on possible pro-social aspects of games. The answer really is the content. A “game” per-say is neither pro- or anti-social like any other instructional medium. Look at speeches. Stalin gave speeches filled with horrible content and Martin Luther King, Jr. gave speeches filled with inspiring content. Yet we never seem to ask the question “Are speeches inherently good or bad?”

Games, like other instructional media, have caveats that good instructional designers need to factor when deciding if a game is the right instructional intervention. Certainly time is a big factor. It takes time to both develop a game and to play a game. So this is a huge downside. You need to weigh the impact you think the game will have on learner retention or knowledge versus another instructional intervention. Although, I can tell you there are at least two meta-analysis studies that indicate that games are more effective for learning than traditional, lecture-based instruction. But the point is not to blindly choose game over lecture or discussion. The decision regarding the right instructional design needs to be thoughtful. Knowing the caveats should factor into the final design decision.

Another caveat is that games should not be “stand-alone.” It’s far better for a learning game to be included as part of a larger curriculum rather than developed without any sense of how it fits into the larger pictures. Designers need to make sure they don’t lose site of the learning objective. If you are considering deploying a game within your organization, you have to make sure it’s appropriate for your culture. Another big factor to consider is how the losers are handled in the game. If a person is not successful at a game, what are the repercussions? What if she gets mad and shuts down? What if he walks away halfway through the experience because he is so frustrated? These types of contingencies need to be considered when developing a game. So, yes, there are downsides to games as there are downsides to other types of instruction. Our job, as instructional designers, is to understand as many downsides and upsides as possible for many different design possibilities and make an informed, evidence-based decision.

 

Will (Question 5):

As you found in your research review, feedback is a critical element in gaming. I’ve anointed “feedback” as one of the most important learning factors in my Decisive Dozen – as feedback is critical in all learning. The feedback research doesn’t seem definitive in recommending immediate versus delayed feedback, but the wisdom I take from the research suggests that delayed feedback is beneficial in supporting long-term remembering, whereas immediate feedback is beneficial in helping people “get” or comprehend key learning points or contingencies. In some sense, learners have to build correct mental models before they can (or should) reinforce those understandings through repetitions, reinforcement, and retrieval practice.

Am I right that most games provide immediate feedback? If not, when is immediate feedback common in games, when is delayed feedback common? What missed opportunities are there in feedback design?

You are right; most games provide immediate, corrective feedback. You know right-away if you are performing the right action and, if not, the consequences of performing the wrong action. A number of games also provide delayed feedback in the form of after-action reviews. These are often seen in games using branching. At the end of the game, the player is given a description of choices she made versus the correct choices. So, delayed feedback is common in some types of games. In terms of what is missing in terms of feedback, I think that most learning games do a poor job of layering feedback. In well-designed video games, at the first level of help, a player can receive a vague clue. If this doesn’t work or too much time passes, the game provides a more explicit clue and finally, if that doesn’t work, the player receives step-by-step instructions. Most learning games are too blunt. They tend to give the player the answer right away rather than layers choices or escalating the help. I think that is a huge missed opportunity.

 

Will (Question 6):

By the way, your book does a really nice job in describing the complexity and subtlety of feedback, including Robin Hunicke’s formulation for what makes feedback “juicy.” What subtleties around feedback do most of us instructional designers or instructors tend to miss?

Our feedback in learning games and even elearning modules is just too blunt. We need more subtlety. Hunicke describes the need for feedback to have many different attributes including the need for the feedback to be tactile and coherent. She refers to tactile feedback as creating an experience where the player can feel the feedback as it is occurring on screen so that it’s not forced or unnatural within the game play. Instructional designers typically don’t create feedback the player or learner feels, typically, they create feedback that is “in your face” such as “Nice job!” or “Sorry, try again.” She describes coherent feedback as feedback that stays within the context of the game. It is congruent with on screen actions and activities as well as with the storyline unfolding as the interactions occur. Our learning games seem to fail at including both of these elements in our feedback. In general, our field needs to focus on feedback that is more naturally occurring and within the flow of the learning.

 

Will (Question 7):

Do learners have to enjoy the game to learn from it? What are the benefits of game pleasure? Are there drawbacks at all?

Actually, research by Tracy Sitzmann indicates (2011) that a learner doesn’t have to indicate that he or she was “entertained” to learn from a serious game. So fun should not be the standard by which we measure the success of game. Instead, she found that what makes a game effective for learning is the level of engagement. Engagement should be the goal when designing a learning game. However, there are a number of studies that indicate that games are motivational. Although, one meta-analysis on games indicated that motivation was not a factor. So, I am not sure if pleasure is a necessary factor for learning. Instead, I tend to focus more on building engagement and having learners make meaningful decisions and less on learner enjoyment and fun. This tends to run counter to why most people want a learning game but the reason we should want learning games is to encourage engagement and higher order thinking and not to simply make boring learning fun. Engagement, mastery and tough decision making might not always be fun but, as you indicated in your questions about simulations, it can be engaging and learning results from engagement and then understanding the consequences of actions taken during that engagement.

 

Will (Question 8):

As I was perusing research on games, one of my surprises was that games seemed to be used for health-behavior change at least as much as learning. What they heck’s going on?

Games are great tools for promoting health. We all know that we should focus on health and wellness but we often let other life elements get in the way. Making staying healthy a game provides, in many cases that little bit of extra motivation to make you stay on course. I think games for health work so well because they capitalize on our already existing knowledge that we need to stay healthy and then provide tracking of progress, earning of points and other incentives to help us give that extra boost that makes us take the extra 100 steps needed to get our 10,000 for the day. Ironically, I find games used in many life and death situations.

 

Will (Question 9):

In your book you have a whole chapter devoted to research on games. I really like your review. Of course, with all the recent research, maybe we've learned even more. Indeed, I just did a search of PsycINFO (a database of scientific research in the social sciences). When I searched for "games" in the title, I found 110 articles in peer-reviewed journals in this year (2016) alone. That's a ton of research on games!!

Let's start with the finding in your book that the research methodology of much of the research is not very rigorous. You found that concern from more than one reviewer. Is that still true today (in 2016)? If the research base is not yet solid, what does that mean for us as practitioners? Should we trust the research results or should we be highly skeptical -- OR, where in-between these extremes should we be?

The short answer, as with any body of research, is to be skeptical but not paralyzed. Waiting for the definitive decision on games is a continually evolving process. Research results are rarely a definitive answer; they only give us guidance. I am sure you remember when “research” indicated that eggs were horrible for you and then “research” revealed that eggs were the ultimate health food. We need to know that research evolves and is not static. And, we need to keep in mind that some research indicated that smoking had health benefits so I am always somewhat skeptical. Having said that, I don’t let skepticism stop me from doing something. If the research seems to be pointing in a direction but I don’t have all the answers, I’ll still “try it out” to see for myself.

That said the research on games, even research done today, could be much more rigorous. There are many flaws which include small sample sizes, no universal definition of games and too much focus on comparing the outcomes of games with the outcomes of traditional instruction. One would think that argument would be pretty much over but decade after decade we continue to compare “traditional instruction” with radio, television, games and now mobile devices. After decades of research the findings are almost always the same. Good design, regardless of the delivery medium, is the most crucial aspect for learning. Where the research really needs to go, and it’s starting to go in that direction, is toward comparing elements of games to see which elements lead to the most effective and deep learning outcomes. So, for example, is the use of a narrative more effective in a learning game than the use of a leaderboard or is the use of characters more critical for learning than the use of a strategy-based design? I think the blanket comparisons are bad and, in many cases, misleading. For example, Tic-Tac-Toe is a game but so is Assassin’s Creed IV. So to say that all games teach pattern recognition because Tic-Tac-Toe teaches pattern recognition is not good. As Clark Aldrich stated years ago, the research community needs some sort of taxonomy to help identify different genres of games and then research into the learning impact of those genres.

So, I am always skeptical of game research and try to carefully describe limitations of the research I conduct and to carefully review research that has been conducted by others. I tend to like meta-analysis studies which are one method of looking at the body of research in the field and then drawing conclusions but even those aren’t perfect as you have arguments about what studies were included and what studies were not included.

At this point I think we have some general guidelines about the use of games in learning. We know that games are most effective in a curriculum when they are introduced and described to the learners, then the learners play the game and then there is a debrief. I would like to focus more on what we know from the research on games and how to implement games effectively rather than the continuous, and in my opinion, pointless comparison of games to traditional instruction. Let’s just focus on what works when games do provide positive learning outcomes.

 

Will (Question 10):

A recent review of serious games (Tsekleves, Cosmas, & Aggoun, 2014, 2016) concluded that their benefits were still not fully supported. “Despite the increased use of computer games and serious games in education, there are still very few empirical studies with conclusive results on the effectiveness of serious games in education.” This seems a bit strong given other findings from recent meta-analyses, for example the moderate effect sizes found in a meta-analysis from Wouters, van Nimwegen, van Oostendorp, & van der Spek (2013).

Can you give us a sense of the research? Are serious games generally better, sometimes better, or rarely better than conventional instruction? Or, are they better in some circumstance, for some learners, for some topics – rather than others? How should us practitioners think about the research findings?

Wouters et al. (2013) found that games are more effective than traditional instruction as did Stizmann (2011). But, as you indicated, other meta-analysis studies have not come to that conclusion. So, again, I think the real issue is that the term “games” is way too broad for easy comparisons and we need to focus more on the elements of games and how the individual elements intermingle and combine to cause learning to occur. One major problem with research in the field of games is that to conduct effective and definitive research we often want to isolate one variable and then keep all other variables that same. That processes is extremely difficult to do with games. New research methods might need to be invented to effectively discover how game variables interact with one another. I even saw an article that declared that all games are situational learning and should be studied in that context rather than in an experimental context. I don’t know the answer but there are few simple solutions to game-based research and definitive declarations of the effectiveness of games.

However, having said all that, here are some things we do know from the research related to using games for learning:

  • Games should be embedded in instructional programs. The best learning outcomes from using a game in the classroom occur when a three-step embedding process is followed. The teacher should first introduce the game and explain its learning objectives to the students. Then the students play the game. Finally, after the game is played, the teacher and students should debrief one another on what was learned and how the events of the game support the instructional objectives. This process helps ensure that learning occurs from playing the game (Hays, 2005; Sitzmann, 2011).

  • Ensure game objectives align with curriculum objectives. Ke (2009) found that the learning outcomes achieved through computer games depend largely on how educators align learning (i.e., learning subject areas and learning purposes), learner characteristics, and game-based pedagogy with the design of an instructional game. In other words, if the game objectives match the curriculum objectives, disjunctions are avoided between the game design and curricular goals (Schifter, 2013). The more closely aligned curriculum goals and game goals, the more likely the learning outcomes of the game will match the desired learning outcomes of the student.

  • Games need to include instructional support. In games without instructional support such as elaborative feedback, pedagogical agents, and multimodal information presentations (Hays, 2005; Ke, 2009; Wouters et al., 2013)., students tend to learn how to play the game rather than learn domain-specific knowledge embedded in the game. Instructional support that helps learners understand how to use the game increases the effectiveness of the game by enabling learners to focus on its content rather than its operational rules.

  • Games do not need to be perceived as being “entertaining” to be educationally effective. Although we may hope that Maria finds the game entertaining, research indicates that a student does not need to perceive a game as entertaining to still receive learning benefits. In a meta-analysis of 65 game studies, Sitzmann (2011) found that although “most simulation game models and review articles propose that the entertainment value of the instruction is a key feature that influences instructional effectiveness, entertainment is not a prerequisite for learning,” that entertainment value did not impact learning (see also Garris et al., 2002; Tennyson & Jorczak, 2008; Wilson et al., 2009). Furthermore, what is entertaining to one student may not be entertaining to another. The fundamental criterion in selecting or creating a game should be the learner’s actively engagement with the content rather than simply being entertained (Dondling, 2007; Sitzmann, 2011).

 

Will (Question 11):

If the research results are still tentative, or are only strong in certain areas, how should we as learning designers think about serious games? Is there overall advice you would recommend?

First of all, I’d like to point to the research that exists indicating that lectures are not as effective for learning as some believe. So practitioners, faculty members and others have defaulted to lectures and held them up as the “holy grail” of learning experiences when the literature clearly doesn’t back up the use of lectures as the best method for teaching higher level thinking skills. If one wants to be skeptical of learning designs, start with the lecture.

Second, I think the guidelines outlined above are a good start. We are literally learning more all the time so keep checking to see the latest. I try to publish research on my blog (karlkapp.com) and at the ATD Science of Learning blog and, of course, the Will at Work blog for all things learning research are good places to look.

Third, we need to take more chances. Don’t be paralyzed waiting for research to tell you what to do. Try something, if you fail, try something else. Sure you can spend your career creating safe PowerPoint-based slide shows where you hit next to continue but that doesn’t really move your career or the field forward. Take what is known from reading books and from vetted and trusted internet sources and make professionally informed decisions.

 

Will (Question 12):

Finally, if we decide to go ahead and develop or purchase a serious game, what are the five most important things people should know?

  1. First clearly define your goals. Why are you designing or purchasing a serious game and what do you expect as the outcome? After the learners play the game what should they be able to do? How should they think? What result do you desire? Without a clearly defined outcome, you will run into problems.

  2. Determine how the game fits into your overall learning curriculum. Games should not be stand-alone; they really should be an integral part of a larger instructional plan. Determine where the serious game fits into the bigger picture.

  3. Consider your corporate culture. So cultures will allow a fanciful game with zombies or strange characters and some will not. Know what your culture will tolerate in terms of game look and feel and then work within those parameters.

  4. If the game is electronic, get your information technology (IT) folks involved early. You’ll need to look at things like download speed, access, browser compatibility and a host of other technical issues that you need to consider.

  5. Think carefully and deeply before you decide to develop a game internally. Developing good, effective serious games is tough. It’s not a two-week project. Partner with a vendor to obtain the desired result.

  6. (A bonus) Don’t neglect the power of card games or board games for teaching. If you have the opportunity to bring learners together, consider low-tech game solutions. Sometimes those are the most impactful.

 

Will (Question 13):

One of your key pieces of advice is for folks to play games to learn about their power and potential. What kind of games should we choose to play? How should we prioritize our game playing? What kind of games should we avoid because they’ll just be a waste of time or might give us bad ideas about games for learning?

I think you should play all types of games. First, pick different types of games from a delivery perspective so pick some card games, board games, causal games on your smartphone and video games on a game console. Mix it up. Then play different types of games such as role-play games, cooperative games, matching games, racing games, games where you collect items (like Pokémon Go). The trick is to not just play games that you like but to play a variety of games. You want to build a “vocabulary” of game knowledge. Once you’ve built a vocabulary, you will have a formidable knowledge base on which to draw when you want to create a new learning game.

Also, you can’t just play the games. You need to play and critically evaluate the games. Pay attention to what is engaging about the game, what is confusing, how the rules are crafted, what game mechanics are being employed, etc.? Play games with a critical eye. Of course, you will run the danger of ruining the fun of games because you will dissect any game you are playing to determine what about the game is good and what is bad but, that’s ok, you need that skill to help you design games. You want to think like a game designer because when you create a serious game, you are a game designer. Therefore, the greater the variety of game you the play and dissect, the better game designer you will become.

 

Will (Question 14):

If folks are interested, where can they get your book?

Amazon.com is a great place to purchase my book or at the ATD web site. Also, if people have access to Lynda.com, I have several courses on Lynda including “The Gamification of Learning”. And I have a new book coming out in January co-authored by my friend Sharon Boller called “Play to Learn” where we walk readers through the entire serious game design process from conceptualization to implementation. We are really excited about that book because we think it will be very helpful for people who want to create learning games.

 

You can click on the images below to view Karl's Gamification books on Amazon.com.

 
 

 

Research

Sitzmann, T. (2011). A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel Psychology, 64(2), 489–528.

Tsekleves, E., Cosmas, J., & Aggoun, A. (2016). Benefits, barriers and guideline recommendations for the implementation of serious games in education for stakeholders and policymakers. British Journal of Educational Technology, 47(1), 164-183. Available at: http://onlinelibrary.wiley.com/doi/10.1111/bjet.12223/pdf

Wouters, P., van Nimwegen, C., van Oostendorp, H., & van der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249-265. http://dx.doi.org/10.1037/a0031311