© Merlot Biology

Tamim, R. et al. (2011) What Forty Years of Research Says About the Impact of Technology on Learning: A Second-Order Meta-Analysis and Validation Study Review of Educational Research, Vol. 81, No. 1

This study found that looking at studies over 40 years, there is a slight tendency for students who study with technology to do better than students who study without technology.

Don’t get too excited about this. This is a ‘second-order meta-analysis.’ A meta-analysis collects a wide range of research publications on the topic, aggregates the data then runs statistical tests to see if across all the studies there are consistent statistically valid results, even if individual studies didn’t find clear differences or produced mixed or contradictory results. Meta-analyses sometimes find differences ‘missed’ in the original studies, because the larger the sample, the smaller the differences needed to be statistically significant. The results of the meta-analysis depend heavily on the criteria used to chose the original studies.

Note though that this paper  is a meta-analysis of previous meta-analyses, so that although the aggregated sample size of participants is very large, the study is now two levels of analysis away from the original research. You begin to wonder what this really means, especially since that even at the second-order level of analysis, the measured difference between ‘effect’ (studying with technology) and ‘control’ (studying without technology) is quite weak (0.35 on a range of .00 to 1.0). As the researchers themselves conclude:

‘It is important to note that these average effects must be interpreted cautiously because of the wide variability that surrounds them. We interpret this to mean that other factors, not identified in previous meta-analyses or in this summary, may account for this variability….Thus, it is arguable that it is aspects of the goals of instruction, pedagogy, teacher effectiveness, subject matter, age level, fidelity of technology implementation, and possibly other factors that may represent more powerful influences on effect sizes than the nature of the technology intervention.’

Right on, but, as always, I recommend you read the article in full if you believe the results could be important.

RELATED ARTICLESMORE FROM AUTHOR

11 COMMENTS

  2. I think you’d agree Tony, that even if there was no correlation between “studying with technology” (whatever that is) and “doing better” (whatever that is), the education system owes it to students to incorporate technology because technology is what’s out there.

    • Well, actually, Elizabeth, I wouldn’t. I’m definitely in favour of teachers using technology, but only if it is used in an appropriate manner. Too often, though, it is used because it’s there or seems cool.

      Comparative research of the kind I quoted is like asking whether it’s better to use a car or walk. It all depends on the circumstance. This is why the results of carefully controlled comparisons are either of no significant difference or weakly associated. As the authors of this study acknowledge, it’s more useful to identify the circumstances or conditions where the use of technology is most beneficial.

      My big worry is that people draw unjustified conclusions from such studies, such as online learning is as good as or better than face-to-face teaching’ (or vice versa). Bad teaching online is not better than good face-to-face teaching (or vice versa). Both need to be done well, and that means looking at other factors as well as the means of delivery.

      • Yikes Tony,

        I was definitely not clear 🙂 I hope my track record (e.g. http://elizabethtweets.wordpress.com/) shows that I would never imply that any means of delivery, on its own, would automatically lead to good teaching/learning.

        But I *do* think the system owes it to students to address technology (and to model technology use) in an effective way, when appropriate. Not just because it’s out there and is cool but because students are going to use it anyway and without guidance they may use it badly and acquire lifelong bad habits. The best example I can think of is Wikipedia. I get really annoyed at teachers who banish it, particularly when they don’t even understand what it is or how it works. Students would learn more from, and have more respect for, a teacher who showed expertise with Wikipedia by explaining the concept of user-generated content and describing situations when Wikipedia might be reliable enough as well as those where a different research approach is required.

        I have intimate personal contact with kids who see school as antiquated and irrelevant for exactly the reason I’ve described (the school librarian doesn’t allow Wikipedia. Ever.)

        And I completely agree with your worry about unjustified conclusions based on a study like the one described. I don’t even understand the concept of clustering educational technology studies over a 40-year window. It seems a bit like looking at the effectiveness of cancer drugs over a 100-year window. Things have changed so much recently that the early data doesn’t apply.

  3. Personally I’ve never read anything more astonishing in regards to education as the claim that 40 years of comparative technology for teaching – weak.
    This study obviously came out of a system that was born before the cusp of the new direction education is now taking. It’s not soon enough.

LEAVE A REPLY

Please enter your comment!
Please enter your name here