In my previous blog post, I introduced my research project, on characterising the materials experience of pop-up and movable books, and designing digitally-mediated interactions with their digital representations. In this post, I will give an update on all activities since the last post. Best news is that our paper describing this research, has since been accepted and was presented last week at world’s largest and peer-reviewed human-computer interaction conference, called CHI (pronounced ‘kai’). The full paper (describing the complete research project), can be found here, and be referred to as follows:
Willemijn S. Elkhuizen, Jeff Love, Stefano Parisi, and Elvin Karana. 2024. On the Role of Materials Experience for Novel Interactions with Digital Representations of Historical Pop-up and Movable Books. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 619, 1–18. https://doi.org/10.1145/3613904.3642142
The paper provided many details on the conducted case study, but here I will describe the process (hopefully) in a bit more accessible fashion here.
Making sense of the data
The last blog left off at the end of the data collection and preliminary data analysis phase. So let’s pick up from here. After the observation and interview data was collected, three great student research assistants (and some AI transcription tooling J) supported me with the transcription and coding of the data. For the data coding we made use of a software tool called Atlas.ti, which allows you combine and annotate all kinds of data types (incl. video, audio, and text files).
For the data coding we used a theoretical thematic analysis approach, meaning that we searched for themes in the data, guided by a pre-existing theoretical framework. In our case we made use of the materials experience framework (Giaccardi and Karana, 2015), which was initially designed to be used in the context of material-driven design (Karana et al, 2015), but was transferred to the cultural heritage context for the first time in this project. In particular, we made use of the frameworks’ materials experience levels, which recognizes that materials experience takes place on four different interrelated levels, namely the:
- Performative level, i.e. what actions the books evoke.
- Sensorial level, i.e. how the materials is sensed through for instance vision, sound, and touch).
- Affective level, i.e. the emotions elicited by the books, such as joy, boredom, excitement.
- Interpretive level, i.e. the meanings ascribed to the books, such as old, fragile, and expensive.
We annotated the observational videos and interview data on the different interactions, sensorial characteristics, emotions and associations that could be observed or that were mentioned by the participants.
Interactions with pop-up and movable elements
In the video below, you can see several examples of the diverse interactions participants had with the physical books.
Once we identified all the different interactions, they were grouped per type of interactive element, such as a pop-up structures, pull tabs, or rotations wheels. See figure 1 for some examples of the different interactions with pop-up elements. In this diagram we also added when specific sequences of actions could help observed (e.g. repetitive actions), and whenever actions have notable temporal characteristics (e.g. taking place fast or slow).
Figure 1: Interactions with pop-up elements in the books (complete image can be viewed in the paper).
Similarly, we identified and grouped the different interactions that could be observed with the pop-up and movable books in VR (see figure 2). These interactions together can be denoted as the performativity of the books (i.e. one of the four materials experience levels).
Figure 2: interactions with the virtual pop-up and movable books
Paper qualities exploited in pop-up and movable books
Next, we tried to understand how all these identified characteristics relate to the different material qualities of pop-up and movable books. Through an iterative process we arrived at five key qualities of paper exploited in pop-up and movable books, namely:
- Fold-ability, related to the ability of paper to fold from a flat page, into 3D structures, and which is associated with repetitive actions, associations such as it being ‘like an explosion’, and emotions such as amusement, admiration, but also boredom.
- Slide-ability, related to the ability of paper, to slide in front and behind each other, also eliciting repetitive actions, it being described as ‘toylike’, ‘playful’, and ‘nostalgic’, ‘intuitive’, and eliciting emotions like amusement.
- Tear-ability, related to the ability of paper to damage (easily), associated with the ripping & scratching sounds, slowly page turning, fear of damaging, and the association that the books are actually not (very) suitable for children.
- Age-ability, relates to paper’s ability to show signs of use and where, which participants associate with ‘fragileness’, ‘old’, visual imperfections, and an (absence of) old-book smell.
- Print-ability, relates to paper’s ability to be printed (with letters and images), and can thereby be associated with (not) reading, colorful and nostalgic associations, leading to sensory pleasure, but also sometimes experienced as overwhelming and confusing.
These characteristics were summarized in five diagrams, such as can be seen in figure 3, which summarises the materials experience characteristics and their interrelationships (if identified), for the fold-ability quality.
Figure 3: Materials experience on four levels, relating to the fold-ability of paper, as used in pop-up and movable books.
Designing a mixed reality and two VR experiences
Based on these characterisations, several ideas for demonstrators were conceptualised. I created one mixed reality experience, and two master graduation students both created their own VR experience as part of their master graduation project, one of which is also featured in the paper. Here, I must also acknowledge Jeroen Boots and Yosua Andoko from the TU Delft XR zone, who supported us all greatly, with the development of all the prototypes. I will explain the mixed reality demonstrator in a bit more detail here.
Firstly, we digitized the book ‘Tip+top boven de wolken’, from 1964, by Vojtěch Kubašta, and used this as a basis for all the prototypes. Initially, we took a whole series of photographs of the original book from the national library collection. We then tried to reconstruct the 2D planes (i.e. the non-deformed, flat paper cut-outs), that make up the 3D shapes in the book (we already reasoned that 3D scanning is not suitable in this case). Reconstructing the planes turned out to be rather challenging using this approach (they need to have the correct dimensions, otherwise the book cannot fold correctly). In second instance, we used an identical book, bought from an antiques bookshop, which we put under a printer flatbed scanner. As all the elements are cut from one page, this was achievable. From this we traced all the parts which made up the book (in Blender). These could then be animated in the Unreal Engine, to create the mixed reality and VR prototypes.
In the mixed reality prototype we aimed to create a memetic material experience, meaning the experience aims to mimic the interactions with the physical book. The XR prototype combines a dummy book, printed with fiducial markers, and a mixed reality headset (Image 4a). As the reader opens the book and turns the pages, the pop-ups are visualised as overlays onto the physical pages (Figure 4b). The movable parts can be interacted with via the eye-tracking functionality – though preferebly we would have liked to rely on more memetic hand tracking (this did not work reliably enough). On three pages, we also explored enhancing different material qualities. On this page we aimed to enhance the fold-ability. By having parts of the page change color, linked to the opening angle of the page, aiming to create surprise, and trigger the repeated actions of opening and closing pages (Figure 4c-d). On another page, we aimed to enhance the age-ability of the book, where the first user sees a page, restored to its original appearance, but with repeated interaction, the pages slowly age and degrade (Figure 4e). Finally we have a page which emphasises the tear-ability of paper, as the plane on the page flies off, you see and hear it ripping loose from the page (Figure 4f).
Figure 4: XRLibris,a mixed reality prototype of the book ‘Tip+top boven de wolken’, uit 1964, origineel geschreven door Vojtěch Kubašta.
An early version of this prototype was demonstrated at The DH Benelux conference, held last year June (2023), in Brussels.
Implications, limitations and future work
From the characterisation and the development of these prototypes we show that the Materials Experience Framework, can well transferred from the (material) design context, and also be used in the context of cultural heritage. We see this case study as a first step towards a possible material-driven design methodology for the augmentation of tangible cultural heritage.
Some of the challenges we encountered relate to disentangling the different material qualities, and also separating experiences triggered by the books’ materiality, and other sources of experience (such as the text or depictions eliciting associations and emotions).
Finally we must acknowledge some limitations of this case study. In the demonstrators we do not tackle the practical challenges, that need to be resolved for actual implementation of such an experience in a library or museum context. This includes the important aspect of enabling social interaction, which is currently not (well) addressed but deemed important in cultural heritage experiences (i.e. people like to share cultural heritage experiences, or experience things in larger groups).
Another (potential) limitation is that we used surrogate books to characterise the materials experience. This might not be possible or viable for many other heritage artifacts (i.e. we might not have surrogates available, or they might be to expensive to create).
We hope to extend on this work in the future, by for instance exploring other ways of materials experience characterisation, and expanding the characterisation beyond the ‘use’ of the CH artifacts (i.e. also look the role of materiality in making, or conservation of CH artifacts). We also aim expand on the design phase of the methodology, exploring how designers might be better supported in translating the materials experience insights into meaningful, digitally-mediated experiences. Finally, the evaluation of the prototypes on their materials experience, is currently ongoing, which might provide us with further insight, into how well we are able to capture and translate the materials experience in the end.
Thanks to…
I would really like to thank the KB for providing me this opportunity to work with their collections and people, which led to this fantastic result. I really enjoyed working at the KB and with people from the KB, also getting an insight into all the other research topics they work on. This work has really helped me to shape my further research directions.
I would also like to thank my co-authors, for their excellent contributions to the paper, making the research not only better content wise, but also greatly helping articulate this in writing. Finally, I would like to thank the heirs of Vojtěch Kubašta, who kindly let us use his work for the XR demonstrators, the KB imaging department, and the XR Zone for helping us realise the demonstrators.
