Experiencing pop-up and movable books through mixed reality technology

The joy of interacting with pop-up and movable books

Interacting with a pop-up or movable book is not like reading a ‘regular’ book. Elements popping up, even beyond the borders of the page, surprise young and old, spark joy, and trigger anticipation for what might be on the next page. Interactive, movable elements in books also show great diversity, from simple flaps and pull tabs to ingenious hinged mechanisms, leaving people in awe on how they were designed and fabricated.

“I like the surprise turning the page and not knowing what will come out. Usually it's not very disappointing, something just explodes out of the page. So it makes you wonder, like alright, what's gonna be on the next [page].” – Participant (27 years old), after interacting with pop-up books

“I'm always very interested in what you can realize with paper, with cardboard, […] and making it three-dimensional is especially what's happening here.” – Participant (63 years old), after interacting with pop-up books

Safely stored, what about experiencing them?

The fact that these are paper artifacts and intended for interaction, make them exceptionally vulnerable. This is especially true for such books in library collections, like the KB, where direct, hands-on interaction is usually prohibited, because of preservation concerns. The KB holds a substantial collection of such books, with early children’s pop-up books dating back to the 1860s. However, conventional approaches to digitisation and access such as 2D/3D imaging/videos, and using web browsers or e-readers, fall short in providing realistic user experiences and interactions, which is also the case for this KB special collection. 

Research project with KB

During my research residency at the KB, I investigate how we might employ mixed reality (XR) technology to experience pop-up and movable books. My research focuses on the dynamic - or temporal - materiality of these artefacts, meaning how these books change under the influence of interaction, and how we might translate the various aspects of such an experience to a digital or mixed reality environment. My research has three parts:

1. A literature and desktop study into the use of digital technologies for experiencing tangible cultural heritage, and approaches to book augmentation
2. Observation and interview study into adults and children interacting with physical and virtual pop-up and movable books
3. Design, development, and evaluation of three mixed reality prototypes, showcasing a book’s dynamic material qualities, which can be triggered through user interaction

In this first blog post, I will tell you something about my finding from the literature and desktop study, and the observation and interview study we conducted last month, at TU Delft and KB.

Digital technologies for cultural heritage experiences

In the past months, I have researched how digital technologies have been employed (throughout the past decades), to enhance the experience of cultural heritage artifacts. This has led to an extensive overview of case studies, with various approaches to employing state-of-the art (of that time) digital technologies. There is a wide range of approaches: some focus on developing an all-purpose multi-sensory device to experience a complete collection (e.g. [1]), and others focus on adding a multi-sensory ‘layer’ (e.g. using touch or smell) to one specific collection item (e.g. [2-3]). Many researchers emphasize the challenge of combining digital and physical elements, exploring different options ranging from overlaying them (e.g. a physical construction, the shape of a showcase, overlayed with a virtual ‘skin’, e.g. [4]) or placing virtual content ‘next to’ physical elements in the space (e.g. text pop-ups next to an artifact, e.g. [5]). Also, the interplay between the ‘original’ and digital content is often debated, whether for instance the digital content might form too much of a distraction from ‘the original’, and how that might be mitigated (e.g. [5]). Moreover, researchers have proposed various frameworks, to support the development of interactive museum experience, for instance focusing on tangible interaction – using physical elements to control digital content – [6-7], embodied museology – how various forms of bodily movement can be employed in museum experiences – [8], or narrative design –using principles of storytelling to design interactive (multi-sensory) experiences [9].

Though many of these studies and cultural heritage professionals acknowledge the key importance of the materiality of cultural heritage artifacts – also one of the key reasons we archive them -, none of the found approaches put the materiality of the artifacts they intend to augment, at the center point of their framework or approach. Moreover, the dynamic - or temporal - material qualities of such experiences have not seen much research yet. Therefore, I will take two frameworks on material experience [10], and temporal form [11] as starting points to creating an experience via digital/hybrid tools.

Digital technologies for enhancing book experiences

Through desktop research I found a great variety in book augmentation, which often make use of the various nifty features of mobile devices. For example, they use the embedded gyroscope in the iPad for interaction (e.g. Alice for the iPad), or interlace various content types (e.g. Leonardo da Vinci: Anatomy app, Insta Novels). Also augmented reality technology is used, for instance, using image/marker-triggered animations (e.g. augmented reality coloring books [12]), or interactive digital-physical hybrids (e.g. mixed-reality book systems [13-15]). 

In contrast to many of these examples, which we might see as the inventions of new (hybrid) book forms or (reading) experiences, my research will focus on using XR technology to augment the temporal material experience of historical pop-up and movable books. Nonetheless we might take inspiration from the technological capabilities applied in these examples.

Observing people interacting with physical and virtual pop-up books

To understand what makes interacting with pop-up and movable books so special, we organized three days of observation and interview studies. One day with participants at TU Delft library, another with employees of KB, and a third day with children visiting the Children’s Book Museum (Dutch: Kinderboekenmuseum). We had 36 participants: 27 adults (26-63y. old) and 9 children (4-10y. old, with accompanying adults). All participants were observed during, and interviewed after interacting with physical pop-up and movable books, and the adults also experienced and evaluated an earlier developed VR prototype, based on four pop-up and movable books [16]. The results were analyzed on four levels of material experience: sensorial (i.e. how people sense materials, incl. the vision, touch, and auditory experience), interpretive (i.e. meanings evoked by materials), affective (i.e. emotions elicited by materials), and performative (i.e. actions elicited by materials) [10]. 

Observations and feedback provided use with rich insights into reading the books, navigating the VR experience, and points of comparison between them. Participants for instance commented on the (lack of) sound and unnatural rigidness with turning the pages (sensorial level), and the restricted ability to ‘pick up’ the books in VR (performative level). Adults interpreted the books as nostalgic, referring back to their childhood, and/or reading to (grand)children (interpretive level). Children referred to comparable books they read (interpretive level). Emotions of amusement, anticipation, and surprise were universal across participants (affective level). Other (micro)emotions were observed/mentioned such as fear (affective level) of damaging a book, which made them act with caution or in a hesitant manner (performative). They were also frustrated with the controls in VR. Performative patterns with the physical books differed between adults and children. The latter took more time per book, read them, and also engaged in question-answer conversations. Comparing the VR experience with books, we conclude the latter offers a more intuitive and much wider range of performative actions, including repetitive open-close actions, and interaction with diverse mechanisms.

“Pretty amazing for me because every page has different ways to read, and the origami things are also pretty.” – Participant (26 years old) talking about interacting with pop-up and movable books

“But [they] make me a little happier, and a little calmer, and then I also feel a little nicer” – Participant (10 years old) after interacting with pop-up and movable books

What’s next?

In the next phase we will distill the distinct temporal material qualities of pop-up and movable books, and use them to develop three variants of a book reading experience, with increasing virtuality levels: using 1) a (facsimile) book (physical, control condition), 2) augmented reality (digital-physical hybrid), and 3) virtual reality (digital), to be evaluated in a comparative study. We have used the same book for all three variants, having 1) images and text, 2) pop-up elements, and 3) various interaction elements (e.g. pull tabs, rotational wheels). 

Acknowledgements

A special thanks to everybody helping me out with the research, including three enthusiastic students from Industrial Design Engineering (TU Delft):Yulia Zelenina, Maarten van Geene, and Jiaqing (Alex) Zhao; the TU Delft VR zone; and employees from the KB, including Dorien Haagsma, Mirjam Cuper, and Karin Vingerhoets. Thanks to the KB atelier (Loes van Eijk), and independent developers (Justin Zijlstra and Tijmen Lohmeijer), for allowing me to use the VR prototype in my observation and interview study. And of course thanks to all the participants who took part in the study. Part of the above text, is based on a short abstract text, submitted to DH Benelux 2023 conference. 

References

1. Joao M.F. Rodrigues, Celia M.Q. Ramos, Joao A.R. Pereira, Joao D.P. Sardo, and Pedro J.S. Cardoso. 2019. Mobile Five Senses Augmented Reality System: Technology Acceptance Study. IEEE Access 7, (2019), 163022–163033. DOI:https://doi.org/10.1109/ACCESS.2019.2953003
2. Mariza Dima, Linda Hurcombe, and Mark Wright. 2014. Touching the past: Haptic augmented reality for museum artefacts. In VAMR 2014: Virtual, Augmented and Mixed Reality. Applications of Virtual and Augmented Reality. DOI:https://doi.org/10.1007/978-3-319-07464-1_1 
3. Chi Thanh Vi, Damien Ablart, Elia Gatti, Carlos Velasco, and Marianna Obrist. 2017. Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition. Int. J. Hum. Comput. Stud. 108, (2017), 1–14. DOI:https://doi.org/10.1016/j.ijhcs.2017.06.004
4. Paul Tennent, Sarah Martindale, Steve Benford, DImitrios Darzentas, Pat Brundell, and Mat Collishaw. 2020. Thresholds: Embedding Virtual Reality in the Museum. J. Comput. Cult. Herit. 13, 2 (2020). DOI:https://doi.org/10.1145/3369394
5. Areti Damala, Eva Hornecker, Merel van der Vaart, Dick van Dijk, and Ian Ruthven. 2016. The loupe: Tangible augmented reality for learning to look at ancient Greek art. Mediterr. Archaeol. Archaeom. 16, 5 (2016), 73–85. DOI:https://doi.org/10.5281/zenodo.204970
6. Eva Hornecker and Jacob Buur. 2006. Getting a grip on tangible interaction: A framework on physical space and social interaction. In Conference on Human Factors in Computing Systems - Proceedings, ACM, Montreal, Quebec, Canada, 437–446. DOI:https://doi.org/10.1145/1124772.1124838 
7. Eva Hornecker. 2010. Creative idea exploration within the structure of a guiding framework: The card brainstorming game. TEI’10 - Proc. 4th Int. Conf. Tangible, Embed. Embodied Interact. (2010), 101–108. DOI:https://doi.org/10.1145/1709886.1709905 
8. Sarah Kenderdine. 2015. Embodiment, Entanglement, and Immersion in Digital Cultural Heritage. In A New Companion to Digital Humanities, Susan Schreibman, Ray Siemens and John Unsworth (eds.). John Wiley & Sons, Ltd., 22–41. DOI:https://doi.org/10.1002/9781118680605.ch2 
9. Jean Ho Chu and Ali Mazalek. 2019. Embodied engagement with narrative: A design framework for presenting cultural heritage artifacts. Multimodal Technol. Interact. 3, 1 (2019). DOI:https://doi.org/10.3390/mti3010001
10. Elisa Giaccardi and Elvin Karana. 2015. Foundations of materials experience: An approach for HCI. Conf. Hum. Factors Comput. Syst. - Proc. 2015-April, (2015), 2447–2456. DOI:https://doi.org/10.1145/2702123.2702337 
11. Anna Vallgårda, Morten Winther, Nina Mørch, and Edit E. Vizer. 2015. Temporal form in interaction design. Int. J. Des. 9, 3 (2015), 1–15. Retrieved from http://www.ijdesign.org/index.php/IJDesign/article/view/2212 
12. Stéphane Magnenat, Dat Tien Ngo, Fabio Zünd, Mattia Ryffel, Gioacchino Noris, Gerhard Rothlin, Alessia Marra, Maurizio Nitti, Pascal Fua, Markus Gross, and Robert W. Sumner. 2015. Live Texturing of Augmented Reality Characters from Colored Drawings. IEEE Trans. Vis. Comput. Graph. 21, 11 (2015), 1201–1210. DOI:https://doi.org/10.1109/TVCG.2015.2459871 
13. Yedan Qian. 2017. Wonder-LAND: A Tangible Mixed-Reality Book System For Explorative Learning in Science. Umeå University, Sweden & MIT, Boston (MA), USA. Retrieved from http://urn.kb.se/resolve?urn=urn:nbn:se:umu:diva-137766 
14. Adrien Mondot and Claire Bardainne. 2020. Acqua Alta - Crossing the mirror - A pop-up and augmented reality book. Druka printing house, Klaipeda, Lithuania. Retrieved from https://www.am-cb.net/projets/acqua-alta-popup-book
15. WoongjinThinkbig. ARpedia. Paju, Gyonggi Province, China. Retrieved January 25, 2023 from https://www.arpediabook.com
16. Marzia Loddo, Foekje Boersma, Martijn Kleppe, and Karin Vingerhoets. 2022. Experimenting with 360° and virtual reality representations as new access strategies to vulnerable physical collections: Two case studies at the KB, National Library of the Netherlands. IFLA J. 48, 2 (2022), 332–338. DOI:https://doi.org/10.1177/03400352211023080