December 2025 – Room TBC

Evaluating Interactive Digital Narratives: New Approaches to Address an Old Issue
Workshop

This is a focused 90 mins workshop with two aims:

  1. discuss current issues in IDN evaluation
  2. start a process to develop improved evaluation methods

A problematic aspect of IDN research has been the scarcity of specific evaluation methods. Applying methods designed for non-interactive narratives carries the danger of not capturing what is specific about IDN and thus misrepresenting the effects of a work. At the same time, several methods frequently used in the evaluation of games and interactive narratives – based on flow theory, affect theory and self-determination theory – have been criticized recently for lacking proper theoretical grounding and for being used in problematic ways (Deterding & Cutting 2023, Tyack & Mekler 202, 2024, ). In addition, sample sizes in the field of IDN research and related areas are often too small to fully support reported results (Koenitz, Roth & Mekler 2024), especially when compared to established fields such as Sociology. Moreover, replication studies – considered a necessity in other fields – are extremely rare. This workshop aims to address this “evaluation crisis” by starting a “big tent” approach where many researchers in the field collaborate to improve the situation.

We also aim to use the workshop itself as a focus group study of the current state of affairs in IDN evaluation.

References

Deterding, S., & Cutting, J. (2023). Objective Difficulty-Skill Balance Impacts Perceived Balance but Not Behaviour: A Test of Flow and Self-Determination Theory Predictions. Proceedings of the ACM on Human-Computer Interaction, 7(CHI PLAY), 419:1179-419:1205. https://doi.org/10.1145/3611065
 
Koenitz, H., Roth, C., & Mekler, E. (2024). Alternate realities in interactive digital narratives – understanding and improving design and prosocial effects through empirical methods. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-024-18884-8
 
Tyack, A., & Mekler, E. D. (2020). Self-Determination Theory in HCI Games Research: Current Uses and Open Questions. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–22. https://doi.org/10.1145/3313831.3376723
 
Tyack, A., & Mekler, E. D. (2024). Self-Determination Theory and HCI Games Research: Unfulfilled Promises and Unquestioned Paradigms. ACM Transactions on Computer-Human Interaction, 31(3), 1–74. https://doi.org/10.1145/3673230

OBJECTIVES

  • Create an understanding of current issues in evaluating IDNs
  • Form a task force developing improved evaluation methods as a follow-up to the workshop

SCHEDULE

  • Introductions (5 mins)
  • Short intro: How do we currently evaluate IDNs? What is missing? (10 mins) 
  • Focus group: IDN eval 2030:  (20 mins)
    • What do we want to understand and do not currently? 
    • What do we need and currently do not have?
    • What strategies for improvement can we think about?
  • Group formation (5 mins)
  • Small group exercises (using Miro + recording for analysis later)
    • What do we want to know? (10 mins)
    • What are the obstacles to knowing that? (10 mins)
    • How can we start implementing? (10 mins) 
  • Sharing outputs and creating shared action plan, including a shared publication (15 mins)

PARTICIPATION

Participation will be limited to approximately 20 people to ensure active engagement.

Please register your interest here: https://forms.gle/54Rk3qJXb7oqFhYh8

ORGANISERS

Sarah Brown University of Florida

Mia Consalvo Concordia University

Lindsay Grace University of Miami

Hartmut Koenitz Södertörn University

Mirjam P Eladhari Stockholm University

Lissa Holloway-Attaway University of Skövde

Ágnes Karolina Bakk MOME University

Joshua A. Fisher Ball State University

Andrew Phelps American University

Jonathan Barbara St Martin's Institute

Iva Georgieva Bulgarian Academy of the Sciences

Christian Roth University of the Arts Utrecht