Learning analytics: where is the evidence?

IMG_3917.jpgOur main paper at the LAK conference looked at the state of evidence in the field. Drawing on the work collated in the LACE project Evidence Hub, it seems that there is, as yet, very little clear evidence that learning analytics improve learning or teaching. The paper concludes with a series of suggestions about how we can work as a community to improve the evidence base of the field.

The room was full to overflowing for our talk and for the other two talks in the session on the ethics of learning analytics. If you weren’t able to get in and you want to understand the links between jelly beans, a dead salmon, Bob Dylan, Buffy the Vampire Slayer and learning analytics, I shall share the link to the recorded session as soon as I have it.

Ferguson, Rebecca and Clow, Doug (2017). Where is the evidence? A call to action for learning analytics. In: LAK ’17 Proceedings of the Seventh International Learning Analytics & Knowledge Conference, ACM International Conference Proceeding Series, ACM, New York, USA, pp. 56–65.

Abstract

Where is the evidence for learning analytics? In particular, where is the evidence that it improves learning in practice? Can we rely on it? Currently, there are vigorous debates about the quality of research evidence in medicine and psychology, with particular issues around statistical good practice, the ‘file drawer effect’, and ways in which incentives for stakeholders in the research process reward the quantity of research produced rather than the quality. In this paper, we present the Learning Analytics Community Exchange (LACE) project’s Evidence Hub, an effort to relate research evidence in learning analytics to four propositions about learning analytics: whether they support learning, support teaching, are deployed widely, and are used ethically. Surprisingly little evidence in this strong, specific sense was found, and very little was negative (7%, N=123), suggesting that learning analytics is not immune from the pressures in other areas. We explore the evidence in one particular area in detail (whether learning analytics improve teaching and learners support in the university sector), and set out some of the weaknesses of the evidence available. We conclude that there is considerable scope for improving the evidence base for learning analytics, and set out some suggestions of ways for various stakeholders to achieve this.

Leave a comment

LAK17: Failathon

Screen Shot 2017-03-31 at 10.20.18Monday 13 March was the day of the second LAK Failathon, this time held at the LAK17 conference at Simon Fraser University in Vancouver. This year, we took the theme ‘Beyond Failure’ and the workshop led into a paper later in the conference and then to a crowd-sourced paper on how we can work to avoid failure both on individual projects and across the learning analytics community as a whole.

We also took a consciously international approach, and so workshop leaders included Doug Clow and I from Europe, Mike Sharkey from North America, Cecilia Aguerrebere from South AMerica, Kirsty Kitto from Australia and Yong-Sang Cho from Asia.

Clow, Doug; Ferguson, Rebecca; Kitto, Kirsty; Cho, Yong-Sang; Sharkey, Mike and Aguerrebere, Cecilia (2017). Beyond failure: the 2nd LAK Failathon. In: LAK ’17 Proceedings of the Seventh International Learning Analytics & Knowledge Conference, ACM International Conference Proceeding Series, ACM, New York, USA, pp. 504–505.

If you can’t access the workshop outline behind the paywall, contact me for a copy.

Abstract

The 2nd LAK Failathon will build on the successful event in 2016 and extend the workshop beyond discussing individual experiences of failure to exploring how the field can improve, particularly regarding the creation and use of evidence. Failure in research is an increasingly hot topic, with high-profile crises of confidence in the published research literature in medicine and psychology. Among the major factors in this research crisis are the many incentives to report and publish only positive findings. These incentives prevent the field in general from learning from negative findings, and almost entirely preclude the publication of mistakes and errors. Thus providing an alternative forum for practitioners and researchers to learn from each other’s failures can be very productive. The first LAK Failathon, held in 2016, provided just such an opportunity for researchers and practitioners to share their failures and negative findings in a lower-stakes environment, to help participants learn from each other’s mistakes. It was very successful, and there was strong support for running it as an annual event. This workshop will build on that success, with twin objectives to provide an environment for individuals to learn from each other’s failures, and also to co-develop plans for how we as a field can better build and deploy our evidence base.

,

Leave a comment

LAK17: doctoral consortium

Screen Shot 2017-03-31 at 09.19.17A very busy week in Vancouver at the LAK17 (learning analytics and knowledge) conference kicked off with the all-day doctoral consortium on 14 March (funded by SoLAR and the NSF). I joined Bodong Chen and Ani Aghababyan as an organiser this year and we enjoyed working with the ten talented doctoral students from across the world who gained a place in the consortium.

  1. Alexander Whitelock-Wainwright: Students’ intentions to use technology in their learning: The effects of internal and external conditions
  2. Alisa Acosta: The design of learning analytics to support a knowledge community and inquiry approach to secondary science
  3. Daniele Di Mitri: Digital learning shadow: digital projection, state estimation and cognitive inference for the learning self
  4. Danielle Hagood: Learning analytics in non-cognitive domains
  5. Justian Knobbout: Designing a learning analytics capabilities model
  6. Leif Nelson: The purpose of higher education in the discourse of learning analytics
  7. Quan Nguyen: Unravelling the dynamics of learning design within and between disciplines in higher education using learning analytics
  8. Stijn Van Laer: Design guidelines for blended learning environments to support self-regulation: event sequence analysis for investigating learners’ self-regulatory behavior
  9. Tracie Farrell Frey: Seeking relevance: affordances of learning analytics for self-regulated learning
  10. Ye Xiong: Write-and-learn: promoting meaningful learning through concept map-based formative feedback on writing assignments

The intention of the doctoral consortium was to support and inspire doctoral students in their ongoing research efforts. The objectives were to:

  • Provide a setting for mutual feedback on participants’ current research and guidance on future research directions from a mentor panel
  • Create a forum for engaging in dialogue aimed at building capacity in the field with respect to current issues in learning analytics ranging from methods of gathering analytics, interpreting analytics with respect to learning issues, considering ethical issues, relaying the meaning of analytics to impact teaching and learning, etc.
  • Develop a supportive, multidisciplinary community of learning analytics scholars
  • Foster a spirit of collaborative research across countries, institutions and disciplinary background
  • Enhance participating students’ conference experience by connecting participants to other LAK attendees

,

Leave a comment

CBA: impact on student engagement

Our new paper is now – lead author Quan Nguyen – is available online in Computers in Human Behavior. It examines the designs of computer-based assessment and its impact on student engagement, student satisfaction and pass rates.

Computers in Behavior is locked behind a paywall, so contact me for a copy if you can’t get access to the paper.

Abstract

Many researchers who study the impact of computer-based assessment (CBA) focus on the affordances or complexities of CBA approaches in comparison to traditional assessment methods. This study examines how CBA approaches were configured within and between modules, and the impact of assessment design on students’ engagement, satisfaction, and pass rates. The analysis was conducted using a combination of longitudinal visualisations, correlational analysis, and fixed-effect models on 74 undergraduate modules and their 72,377 students. Our findings indicate that educators designed very different assessment strategies, which significantly influenced student engagement as measured by time spent in the virtual learning environment (VLE). Weekly analyses indicated that assessment activities were balanced with other learning activities, which suggests that educators tended to aim for a consistent workload when designing assessment strategies. Since most of the assessments were computer-based, students spent more time on the VLE during assessment weeks. By controlling for heterogeneity within and between modules, learning design could explain up to 69% of the variability in students’ time spent on the VLE. Furthermore, assessment activities were significantly related to pass rates, but no clear relation with satisfaction was found. Our findings highlight the importance of CBA and learning design to how students learn online.

Nguyen, Quan; Rienties, Bart; Toetenel, Lisette; Ferguson, Rebecca and Whitelock, Denise (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior (Early access).

Leave a comment

Innovating pedagogy: China

The Innovating Pedagogy 2016 report. Now in Chinese.

First page in Chinese

Leave a comment

Lea’s Box and Eco

Images from the two projectsJust back from a couple of trips to Luxembourg, where I was one of the team carrying out final reviews for the Lea’s Box and Eco projects. This was my third year reviewing Lea’s Box, but I only joined the Eco team for their final review.

Lea’s Box was ‘a 3-year research and development project (running from March 2014 to [January] 2016) funded by the European Commission. The project focussed on (a) making educational assessment and appraisal more goal-oriented, proactive, and beneficial for students, and (b) on enabling formative support of teachers and other educational stakeholders on a solid basis of a wide range of information about learners.’

Eco was ‘a European project based on Open Educational Resources (OER) that gives free access to a list of MOOC (Massive Open Online Courses) in 6 languages […] The main goal of this project is to broaden access to education and to improve the quality and cost-effectiveness of teaching and learning in Europe.’

,

Leave a comment

Introducing learning analytics

image of the online articleAfter talking about learning analytics at the BETT show, I was invited to write about them for the Public Service Executive magazine. The hard copy of PSE goes out to 9,000 subscribers, while the online version goes out to a database of 50,000.

This article provides a short introduction to learning analytics for people considering introducing analytics at their institution. It introduces six areas for action, and briefly outlines what needs to be done in each of these:

Areas for action

  • Leadership and governance
  • Collaboration and networking
  • Teaching and learning
  • Quality assurance
  • Capacity building
  • Infrastructure

 

Leave a comment