top of page
Tangible Future
Materiality in Embodied Experiences
Keywords: Materiality; Embodied Experience; Tangible Media; Haptics;
Unpredictability; Uniqueness; Material Intelligence; Human-Material Interaction
figure01-aura1.png

I.    INTRODUCTION

 

i.    Corporeal in Cyber Era and Body-Mind Dualism

​

We live in a cyber era. We used to live in a perceived physical world, with our body as the primary site of knowing and experiencing (Merleau-Ponty, 1962). Yet the rapidly expansive domain of cyberspace beckons us to spend more time immersing ourselves in virtual experiences, where the corporeal dimension of our body and senses are gradually marginalised.

With the rapid development of virtual and digital media, artists and scientists are working diligently to create highly immersive embodied experiences in virtual realms. Due to the varying levels of technological development and efficiency, the experiences are primarily mediated through the mesmerising glow of headsets and screens. In this way, users` exploration and interaction within the cyber world are reduced to simplified interface actions with limited feedback. These visually dominant landscapes tend to capture our focus and prioritise our perception of sight. The new ascendancy towards visual emphasis results in “a break in the ratio among all our senses, a kind of loss of identity” (McLuhan, 1962).

Moreover, some virtual experiences also involve individually crafted simulations of various sensory perceptions, whereas real-world encounters are “almost synchronously multimodal and holistic” (Nanay, 2018). This detachment from the physicality of our experiences undermines their authenticity and gradually diminishes our connection to the tangible world. Consequently, we find ourselves immersed in an emerging phenomenon that can be termed artificial body-mind dualism[1].

In this paradigm, the once-clear boundaries between our physical bodies and digital experiences blur, resulting in a division where the tangible and the virtual no longer harmoniously coexist. The intimate relationship between our bodies and the world, once the foundation of our understanding, is gradually eroding as we increasingly rely on mediated and simulated encounters. This artificial body-mind dualism limits our sensory engagement, and separates us from the richness and profundity of our lived bodily experiences.

 

ii.    Medium and the Global Village

​

Nowadays, a multitude of mediums has emerged with the aim of bridging the divide between the tangible and the virtual, between our bodies and the world. As Marshall McLuhan, a prominent writer on media theory, proposed, the transformative power lies in the medium itself rather than its content, stating, "The medium is the message" (McLuhan, 1964). These mediums, as extensions of ourselves, shape our perceptions and alter “the way we think and act - the way we perceive the world” (McLuhan & Fiore, 1967).

Before the invention of the printing press, the pace of this transformation was relatively slow, but with the advent of the electronic age, the rate of change has exponentially accelerated. This acceleration is placing a severe strain on the processing capacities of individuals, also reforming the modern world as a “global village” (McLuhan, 1962). This global village represents a state where access to information from anywhere on the planet is nearly instantaneous.

However, within this global village and the digital realm, our experiences are always unavoidably pre-programmed with limited possibilities: Television and radio programs are prerecorded and played repetitively, video games rely on fixed inputs and offer a finite range of story developments within predefined virtual worlds, and even the most immersive VR and AR experiences are limited in the ways they allow interaction and often involve repetitive programmed experiences.

From this perspective, all intangible media create homogeneous embodied experiences. The dominance of well-programmed mediums and interactions imposes a certain impersonal narrative on our encounters. This thoughtlessness marked by predictable and standardized interactions, which stands in stark contrast to the richness of genuine physical interactions, ultimately leads to a loss of authenticity and uniqueness in digital experiences.

​

To rekindle the appreciation of the authenticity and uniqueness of real-world experiences, I am deeply engaged in research centred on tangibility and materiality. I seek to explore tangible forms of objects and experiences that consider bodily perception to mitigate body-mind dualism. By shifting the paradigm from pre-programmed and filtered media to more responsive and adaptable physical forms of engagement, the richness and authenticity, which are often compromised in our increasingly digital and cyber-dominated era, are aiming to be restored.

 

​

​

II.    AURA OF OBJECTS AND EXPERIENCES

​

In this chapter, I delve into Walter Benjamin's concept of the "object aura" and conduct a comprehensive analysis of the inherent Aura in different (digital vs. physical) forms of tangible and intangible entities, respectively using contemporary art and performance as examples.
First, I compare various methods of reproducing objects and artworks, and demonstrate how mechanical reproduction brings in a new inherent aura compared to the dim aura of digital and virtual reproduction. Then, with research and personal experience in performance, I investigate the manifestations of aura in both theatre and film performances, from the perspectives of both the audience and the performers. 
Based on these comparative analyses, I argued that the messages conveyed by digital formats are an interpretation with great deduction of their innate uniqueness and richness. The prevalence of pre-programmed procedures results in a dissipation of Aura, potentially leading to a monotonous and low-entropy future. In contrast, tangible forms of the objects and experiences, with a deliberate emphasis on perceptual richness, exhibit a heightened capacity to infuse the experiential dimension with a profound sense of embodiment, thus mitigating the artificial body-mind dualism. 

​

i.    Object Aura and Mechanical Reproduction


Walter Benjamin proposed “Object Aura” referring to an object`s presence in time and space and its unique existence at the place where it happens to be (Benjamin, 1968). This Aura is “an individualised atmosphere that envelopes the authentic object, a subtle but distinct sensation received in the presence of the original” (Naegele, 1998) rather than the object itself. And in the age of mechanical reproduction, the aura of the work of art withers. A simple example, a hand-made sculpture has a certain unique aura, while its mechanical, visual or virtual reproductions reduce the aura and uniqueness. However, the reduction of each reproduction way is quite different.
The visualisation is usually using cameras to capture the appearance of the object. Yet, whether it be a still image or a video, these reproductions can only show one fraction of the 3-dimensional appearance of the objects in one frame. Besides, the visual information it transmits is always limited by the resolution of the camera, which is much smaller than the amount of information that a physical object takes on. 
Due to this limitation of visualisation, people have created more digital technologies for reproduction in recent years. Taking museum VR visits as an example, they usually scan the original object and synthesise its three-dimensional Point Cloud model in software, then put it into VR so that people can see it in all directions and even interact with it. This digitisation does show the full appearance of objects at a higher resolution, but still only conveys the visual fraction of the rich tapestry of sensory experiences that physical objects offer. It is still a dimensional reduction of the aura.

The mechanical reproduction, whether through processes like injection moulding or programmed machine automation, no matter how closely resembles the original item, the very nature of the production process leaves traces of the tools and machinery used, imparting a distinct imprint on the object. While every reproduction method inherently diminishes the original's aura, the unique marks left by physical manufacturing, as well as the spatial composition with other physical objects, give rise to, in a sense, a new aura. All reproduction is interpretation with unavoidable reduction, but tangible reproductions have the potential to create another sense of novelty and uniqueness.

 

Figure 1: Original Objects (left) vs. Mechanical Reproduction (centre) vs. Visual & Digital Reproduction (right)

ii.    Performance’s Aura in Theatre and Film


Performance lives in its present. While both theatre and film are recognised as venues for performance, they embody distinct characteristics.
I used to work in a theatre continuously doing photography for the same show for a month. I was always surprised by the profound differences in each live performance, even when the cast and scripts were the same. After I started doing theatre performances, I gradually realised how actors are deeply influenced and responsive to the subtlest of cues – the audience's reactions, emotional shifts, the climate of the room, and even the smallest details like eye contact with fellow actors and crew. 
Theatre unites performers and spectators in a shared temporal and spatial reality, fostering a sense of togetherness and immediacy. Even performances with identical scripts, casts, and crews can be distinctly different from one another, as each instance is shaped by the unique interactions and dynamics of the live experience. Any attempt to preserve or document a performance through film or other representation fundamentally alters its nature, transforming it into something other than a live performance (Phelan, 2021).
Film, as one of the most successful representation attempts of performance, does open new avenues for democratising art with more gorgeous audio-visual effects and reaching wider audiences. However, it tends to weaken the aura of original performance. Films maintain their inherent identity regardless of the viewing location, as the camera becomes the absolute dictator of attention (Kiley, 2019). This leads to the participatory audience becoming a sea of disconnected individuals, detached from the live presence and the embodiment of emotions and stories.
In recent years, performances in theatres have seen a renaissance. Other than the emerging immersive theatres and interactive theatres, more and more performative experiences are also added in various realms. People succumbed to the aura of live performance which seems to possess deeper dimensions. During a live performance, each movement and expression of actors, each object on the stage, from lighting to orchestral accompaniment, even the subtle sounds of costumes rubbing and the collective breaths of the audience, contribute to a highly distinctive experience that evokes individual emotional and physical participation. In this way, live performance is also a tangible embodied experience encompassing a collective aura, an aura of the event itself.  

 

Figure 2 (left): Actors interact with audience and modify their performance, from Elisabeth (musical) (2005). https://www.youtube.com/watch?v=DINlFcyJQpc&ab_channel=ChristophKreuter 
Figure 3 (right): A contemporary mode of Interactive theatre that audience(with masks) walk and explore freely in a promenade theatre while immersing and interacting within it, for instance, Punchdrunk’s Sleep No More(2012). Photo by Yaniv Schulman. https://www.punchdrunk.com/project/sleep-no-more/ 
 

微信截图_20240213121214.png

Those investigations into diverse forms of tangible and intangible entities reveal that, the essence of Aura is deeply engraved upon specific temporal and spatial contexts.  Yet, it also becomes apparent that their ontological significance extends beyond mere temporospatial considerations. These entities inherently possess a broad spectrum of values and narratives “elicited in the course of making” and materialising (Jung & Stolterman, 2012). Their ontological contexts continue to develop, encompassing their relationship among objects, individuals and interactions within this framework. Consequently, the broader analysis of ontology ultimately emerges a perspective hierarchically beyond the aura, where "materiality" constructs at the forefront. 

 

 

III.    MATERIALITY IN INTERACTION

 

Materiality is generally defined as “the material existence” (Oxford English Dictionary, 2023) and “the physical properties” of existence (Wikipedia, 2020). Within the domain of human-computer interaction (HCI), the research around materiality has gained prominence, especially with the trend of the growing emphasis on developing tangible modes of interaction and integrating computing into material forms (Wiberg, 2018). Jung and Stolterman suggested that materiality should be a foreground concern for interaction design, which is because beyond the inherent material qualities of humans or objects, materiality pertains to “the relationship in between” (2012).

Therefore, considering the various definition aspects and the focus on the “in-between”, the concept of materiality in the context of creating embodied experiences can be examined across three layers:

  1. the Materiality of Interactive Objects;

  2. the Materiality of the Human Body;

  3. the Materiality of the Entire Experience.

To enhance the materiality of the human body, full-sensory perception and embodiment need to be prioritised, which leads to a rediscovery of tangibility and haptics (Chapter III). The materiality of the overall experience process necessitates various technical methodologies for its advancement, which will be explored in more detail in Chapter IV.

​

The present Chapter primarily concentrates on the materiality of the interactive objects. Throughout this chapter, I explored the limitations and challenges posed by the reliance on remote communication media during the lockdown period. This research drew particular attention to full-sensory communication within the physical world. To achieve such tangible communication, endowing interactive objects with tangible forms seems to have more potential compared with virtual perceptual simulations. This exploration thus leads to a more comprehensive investigation into tangible media.

 

i.    Lockdown, Communication and Media

​

During the COVID-19 lockdown period, the significance of pervasive and mobile media in connecting people with each other and the world has been undeniably acknowledged. Screens have become a lifeline, facilitating real-time communication with distant family members, providing access to global news through social media, and offering virtual worlds as an escape from the confines of our homes. Yet, a recent study found that social media and video communication cannot compensate for normal real-life social, because the lockdown has exacerbated the co-presence of online and offline social spaces (Brandtzaeg & Lüders, 2021). Despite the high-resolution imagery and real-time interactions, many individuals still experience feelings of loneliness, isolation, and a sense of being adrift. This discrepancy is primarily attributed to the touch deprivation in digital interactions, which as a result, cannot fully replace the depth and richness of face-to-face encounters.

​

Therefore, researchers have explored the role of tactile sensation in this discrepancy, leading to the development of haptic simulation technology. Meta Reality Labs, for instance, has developed haptic gloves equipped with ridged and inflatable actuators

that create vibrations and pressure on the hand`s

skin by adjusting the level and duration of inflation.

When combined with visual and audio rendering,

these gloves aim to create the illusion of physical

touch (Meta, 2021). Numerous other haptic gloves

are also being researched and invented. However,

despite ongoing efforts to enhance different aspects

such as actuators, response time, render frequency

(M, Venusamy, S, S, & O, 2023), and simulation of

various tactile sensations, digital haptic reproduction

remains constrained and homogeneous in comparison

to the unique and intricate sensations experienced

through whole-skin haptic interactions in the real

world. In some cases, simple tangible objects or

representations have proven to be more efficacious

and verisimilar substitutes for these intricate tactile

encounters.

RL___Haptic_Glove_1scaled_edited.jpg

Figure 4: Meta’s haptic glove. Simulating haptic perception for immersion in VR and AR. Photo from Meta Reality Labs. (2021) https://www.theverge.com/2021/11/16/22782860/meta-facebook-reality-labs-soft-robotics-haptic-glove-prototype

ii.     Tangible Media

​

One remarkable innovation in this domain is the InFORM, a shape display capable of physically rendering 3D content (Leithinger, Follmer, Olwal, & Ishii, 2014).  It relies on 900 arrays of motorized pins linked with actuators through nylon push-pull rods, which enable the transmission of bi-directional force, to achieve shape rendering. The physical representations inFORM can come from digital 3D models, other user interfaces, or the shapes of remote objects or people captured by a depth camera in real-time. In this way, inFORM achieves real-time interaction with digital information and the surrounding physical world in a tangible manner (Follmer, Leithinger, Olwal, Hogge, & Ishii, 2013).

​

Figure 5 (left): the composition of inFORM. Photo from Tangible Media Group. (2015) https://tangible.media.mit.edu/project/inform/

Figure 6 (right): how inFORM achieves remote and real-time shape rendering, and the interaction with remote objects (top), and digital information (bottom). Photos from Tangible Media Group. (2015) https://tangible.media.mit.edu/project/inform/

微信截图_20240213183037.png

InFORM adopts the way of shape displays, which allows direct physical contact with the rendered simulacra. The tangible physical rendered interface gives users an intuitive and visually faithful reproduction through touchable physical shapes. Users can touch and interact with the physical shapes while observing their transformation in response to applied forces. The microcontrollers will process the force and modify the resistance of pins to re-render shapes. The touch and hand gestures are also captured by the depth camera and trigger some digital actions, such as moving an object or changing it. The digital actions will be projected to the interface via the overhead projector, allowing users to see and touch the feedback in real-time. Furthermore, this real-time and highly tangible interaction mode addresses the inherent challenge in the haptic interactive system of “keeping the physical and digital states of objects synchronised” (Leithinger, Follmer, Olwal, & Ishii, 2015). The synchronisation and directness achieved by the tangible interface have great potential to reduce the loss of Aura in the distant communication process.

The group inventing and developing inFORM is the Tangible Media Group at MIT. This group was among the first to conceive the notion of the tangible user interface(TUI) and employ it as a conduit to achieve a more embodied form of human-computer interaction. They recognised the inherent loss with the advent of personal computers in the 1990s. It is like us humans donning spacesuits, floating individually within a surreal digital cosmos where our senses remain unscathed, yet our physicality is stripped away. Our experiences, once expansive and free, are now ensnared within the confines of a limited interface. Researchers attribute this loss and deduction in the standardised digital interaction to the absence of materiality, and try to bridge the chasm separating the cyber world from the physical realm through the medium of tangible interfaces (Ishii & Ullmer, 1997).

 

Over the past two decades, a proliferation of tangible media has emerged, each offering diverse perspectives on endowing tangible forms to interactive objects. These innovations offer physical objects that people can easily and directly interact with and receive real-time, synchronized-perceptual feedback. Furthermore, the renaissance of tangible interaction mode ultimately leads to creating unique embodied experiences. In conclusion, with consideration of synchronised multi-sensory perception, tangible media offers a genuine and efficacious remedy for artificial mind-body dualism. It ushers in a revival of its materiality enriching the depth and authenticity within our perceptual experiences.

​

​

IV.    MATERIALITY AND TANGIBILITY

 

The field of interaction design has increasingly emphasized the materiality of human body over the years within theoretical and practical realms, such as activity theory and user­centred design (Wiberg, 2018). This chapter goes back to the philosophical analysis of the human body and its dialogue with the environment. Through the research on how the human body employs full-sensory perception to engage with the physical world, particular attention is drawn towards tangibility and haptic perception. Furthermore, the chapter employs my project, "Haptic Mirror", as a practical case to argue that: prioritizing perceptions in embodied experiences, with a particular consideration on haptics, facilitates a more profound connection with the human body. This enhanced connection, in turn, augments user engagement and embodiment, ultimately fortifying the materiality of the human body within the context of interactive experiences.

 

i.    Body, Physical World, and Haptics​

​

We inhabit our bodies, living with them and through all their perceptions to know and experience a complex physical world (Merleau-Ponty, 1962).  Through perception, the body always instinctively engages, focuses and reacts to the world before any verbal reflection or any thoughts and actions (Nixon, 2020).  Among all the perceptions, haptic perception often remains an overlooked conduit for this sensory exchange. When touching happens, our body (through the biggest sensory organ, skin) triggers a multitude of sensory experiences within the material world, encompassing sensations like temperature, vibrations, weight, texture, and more. Many of these sensations are intricately intertwined with other senses, such as the interplay between visual stimuli and perceived temperature, or auditory cues and vibrations in the air.

The term "tangibility" aptly characterises this dual nature of touch. On one hand, individuals initiate haptic interactions by reaching out to touch objects or surfaces. On the other hand, our skin, functioning as the sensory receptor, continuously receives tactile stimuli from the environment. This dynamic process embodies a bidirectional exchange, where individuals simultaneously engage in both active and passive roles, revealing the inseparable relationship between initiating touch and being touched. This interplay shows the reciprocal nature of haptic interaction, where individuals are both actors and recipients in a dynamic, sensory dialogue with the material world.

In the early Western philosophy and interactive design, there was a tendency to take vision (and audio) as our primary or higher-order sense, while often relegating touch to a subordinate position and attributing to its perceived limitations in abstraction and detachment. However, the recognition rooted in phenomenology and the fervour for creating embodied experiences has elevated haptic perception to an equally critical (if not higher) sensory position, claiming it as a reminder of the materiality and embodiment of our beings (Hornecker, 2011).

The growing interest in tangible media (see Chapter III) has propelled Tangible Embodied Interaction (TEI) into an establishing realm of research and design practice (Hornecker, 2011). TEI delves into the implications and emerging prospects of Human-Computer Interaction (HCI) within the physical world, prioritizing embodied experiences and embracing the multifaceted concept of materiality. It melds physical form and computational elements, culminating in interactive and responsive manifestations (Baskinger & Gross, 2010). Furthermore, with the emphasis on haptics, Tangible Interaction blurs the boundary between the external and internal worlds, the physical and digital realms, providing a foundational avenue for experiencing and reinforcing our embodiment and existence (Becker B. , 2003).

 

ii.    Tangible Embodiment

​

Within the framework of embodied interaction, the significance of full-sensory engagement is evident. To raise the materiality of the human body to a limit, I choose to prioritise a haptic-centred experience, and make the material, rather than the human, take the lead in initiating tactile exchanges.

In pursuit of this goal, I extensively explored various tangible mechanisms capable of actively engaging with the human body in physical interactions. These explorations encompassed the development of various tactile actions, including grabbing, tapping, and stroking etc. Throughout the investigative process, an observation emerged that tactile actions invariably commence with contact between two surfaces. The subsequent distinctions in these actions are determined by variations in their intensity and duration, each assigned different names and meanings.

My research has been particularly focused on grabbing action. This decision is due to not only its inherent advantage in building connections, but also its capacity to offer a wider spectrum of dynamic and flexible movement possibilities. For instance, a brief surface contact corresponds to tapping; a continuous and slow surface contact corresponds to caressing; sustained and forceful engagement corresponds to grabbing. These actions may assume distinct characteristics and connotations when applied at different scales; for instance, small-scale seizing on objects or hands is categorised as grabbing, while larger-scale application to human body is categorized as hugging.

To achieve controlled and adjustable grabbing movements, we experimented with various mechanical structures, including inflatable system, soft mechanism, and rigid mechanism. Ultimately, based on the criteria of controllability, stability, and flexibility, we opted for a mechanical structure composed of a combination of stiff units and soft joints, which is inspired by the Yale Open Hand project (The Yale GRAB Lab, 2019). We further fabricated structures of varying scales and forms to enable more extensive bodily interactions.

​

微信截图_20240213183922.png
微信截图_20240213190114.png

The Haptic Mirror project represents an embodiment of these extensive research efforts and seeks to establish connections between the human body and material through touching interactions. The central installation is comprosed of six sets of arms, a supportive metallic stand, and six stepper motors. Each arm consists of rigid dactylopodites connected by soft joints and actuated by tensioned cables. Importantly, each arm is independently activated by individual motors, enabling a wide range of versatile movements, thus facilitating diverse interaction modes:

As people enter the scene, the arms are set in motion. One curious arm may reach out first trying to emulate the iconic “Creazione di Adamo”. As people approach closer, certain arms may execute quick and delicate bending-releasing movements, just like some subtle tapping. When people fully immerse themselves in the installation domain, all arms may collectively tighten and enclose around people, resembling a trapping or a hug. The adaptability of the soft joints allows some arms to modify their positions and gently transition to another part of the body, which is like a caressing or petting motion. Since each arm is actuated independently, the combination of touching movements can be more diverse and captivating. Furthermore, beyond the consideration of haptics, the incorporation of inner and outer surfaces with varying textures and visual shapes is undertaken to create a multitude of sensory experiences.

moving mode2.jpg

Figure 12: Diverse Haptic Interaction Modes:

in a & b, one arm reaches out to touch people or being touched;

In c, d,e, Arms move in various frequency in different order to create unique interaction & experiences;        

In f & g, all arms simultaneously enclose around people forming a trap or a hug.

​

​

 

In summary, the installation is purposefully created to engage with humans through haptic interactions. Its repertoire of diverse touch modalities, adaptable combinations, and a wide array of material components significantly enrich its capacity to foster profound emotional connections within embodied experiences. The enhanced connection and high consideration of holistic bodily perception, especially haptics, greatly fortify and amplify the materiality of human body.

V.    MATERIALITY RESPONDS TO UNPREDICTABILITY

 

The materiality of real-world experience is rooted in its authenticity and richness, which characterized by the unpredictability and unlimited possibilities in every step of the process. Therefore, when artificially creating embodied experiences, the aim should be restoring this beautiful unpredictability and avoiding preprogrammed or default methods. This chapter discussed various methods throughout the entire process that can actively embrace unpredictability. By thoughtfully designing input, mechanisms, control systems and materials responding to unpredictability, I endeavour to ensure that each embodied experience is as unique and rich as those encountered in the real world.

 

i. Allowance of Unpredictable Input 

​

In many intangible or tangible interactions, user inputs often conform to a predefined set of limited options. For instance, video games typically rely on specific keyboard and mouse actions, VR systems necessitate specialized controllers or gesture recognition, and tangible interfaces often require button presses or specific predefined commands. This approach demands that users undergo prior learning and execute precise operations, deviating from the natural spontaneity of random movements and the unpredictability inherent in the real world.

​

Figure 13: samples of limited input

limited input.png

Consequently, such interactions are inherently constrained from the outset. Users are confined to exploring preconfigured and circumscribed narratives, resulting in restricted sensory experiences. Their bodies are reduced to executing limited predefined movements, detracting from the materiality of their embodied presence. The overall materiality of the experience, which should ideally facilitate comprehensive exploration and full sensory immersion, is consequently diminished.
Therefore, there arises a need for an interaction paradigm that accommodates random and analog bodily inputs while retaining the ability to respond to unpredictability, producing high-responsive and uniquely tailored experiences. 
In the Haptic Mirror project, another stationary installation is made to collect real-time unpredictable bodily input. It is a pole with 6 belt-form pressure sensors all under its surface. When people engage with this pole, whether through touching, hugging, punching or any other form of haptic interaction, the sensors capture data regarding the strength and duration of pressure applied, store the data and remap it within a specific domain to be the touch strength and duration of the robotic Arms. 

微信截图_20240213200613.png

This approach allows for the acceptance of spontaneous and random forms of bodily movement as input, and the resulting output, in terms of the movements and interaction modes of the robotic arms, becomes fully responsive to the bodily input. Importantly, each interaction yields a unique and irreproducible experience, thereby preserving and enhancing the materiality of the overall user experience.

微信截图_20240213200903.png

However, it is worth noting that in this setup, haptic contact remains a necessary trigger for the interaction, although it accommodates a wide range of random and unpredictable movements. It is still limited in this case. Therefore, for the next version of this project, I use the existence of the user`s body as input, with no need for any advanced operation to trigger. The interaction mode is the arms initiating hugs and other modes of tactile interaction, adapting to the user's random bodily data such as position, distance from the machine, gestures, or body part utilised. The transcendent over the limitations of predefined actions and embraces the spontaneity of bodily movements, help to build rich, responsive, and distinctive user experiences. To achieve this technically, an adaptable dynamic system needs to be developed.

​

​

ii.    Adaptable Mechanism

​

In pursuit of a versatile tactile interaction akin to human hugging, the underlying mechanism can be conceptualised as a simplified robotic gripper. Traditional approaches to building a robotic gripper are using rigid material equipped with an array of sensors, encompassing depth cameras, distance sensors, and force sensors, among others, for the purpose of capturing data pertaining to target objects (Tai, El-Sayed, Shahriari, Biglarbegian, & Mahmud, 2016). These systems also rely on numerous motors attached to each joint to control their degree of bending and positioning. Implementing such robotic systems necessitated intricate programming to process the abundant visual and tactile data, issue motor commands, and continuously adjust based on real-time feedback. Consequently, some of these developed robotic hands became exceedingly complex and costly, sometimes featuring over 120 sensors (OH, YI, YIP, & DAYEH , 2020) and 20 actuators (Shadow Robot, 2021). Moreover, when confronted with the task of grasping objects with diverse shapes, these systems often proved unwieldy and challenging to control due to the substantial software overhead.

In contrast, the human hand can easily grasp objects at a low level of cognition (Heyl, 2018) – Humans can effortlessly locate and grasp fragile items, such as delicate glass, without consciously contemplating the process or the precise joint angles and force required to avoid breakage or dropping. Recognizing the inherent simplicity and efficiency of this biological mechanism, soft robotic researchers have been striving to construct more proficient grippers through more simple and intelligent means.

​

One method is the inflatable mechanism. The basic version is this kind of inflatable grabbing hand usually made of silicone with one side of the gripper finger having thicker silicone than the other. The thinner side of the surface tends to perform bigger deformation, resulting in the bending motion towards the thicker side driven by the air pressure actuator. Multiple fingers can bend simultaneously to achieve grasping. Importantly, when the grasping force balances with the air pressure difference between the finger and the external environment, the inflation process naturally halts, eliminating the need for precise control via programming and feedback systems. Furthermore, owing to the flexibility and softness of silicone rubber, this mechanism can adapt to a wide variety of object shapes without causing damage. Based on this principle, more advanced and capable robotic systems have been developed, such as the tentacle robot from SEAS (Harvard John A. Paulson School of Engineering and Applied Sciences). This soft robot uses a collection of thin, tentacle-like inflatable filaments to entangle and grasp objects (Becker, et al., 2022). These filaments are hollow, rubber tubes also with one side. So once inflated, each tube curls and entwines with the others and the target object. This cooperative effort allows multiple weak individual contacts to combine into a robust and adaptable grip suitable for securing fragile and irregularly shaped objects.

微信截图_20240213201255.png

Inspired by these advancements, we started our research on adaptable soft mechanisms. However, a rapid decline in strength and flexibility always happens when the inflatable mechanisms are scaling up to a whole-body size. So we redirected our focus toward alternative solutions within the realm of the combination of rigid and soft robotics.

A significant source of inspiration emerged from the SDM (Shape Deposition Manufacturing) adaptive robotic hand. These hands incorporate fingers designed with passive compliance and are actuated by tendon cables (Dollar & Howe, 2009). The embedded viscoelastic flexure joints provide an easier way for each finger to bend, while the use of pulleys enables one single force to drive the movement of all four fingers. Building upon this mechanism, the final version of the Arms was developed.

微信截图_20240213202802.png

The developed prototype of “Haptic Mirror” comprises eight sets of arms, with each arm consisting of fourteen 3D-printed rigid links and embedded viscoelastic joints made from urethane rubber. Eight pulling cables, as the actuators of the arms, converge into a pulley system that enables a single force to actuate all arms and all arms to flexibly adapt to various target shapes. This condenses what was once a complicated process involving digital sensing, heavy computing and multiple actuating into a single unified mechanism. In this way, the arms can move in a way that is unpredictable for software, making them more responsive to random inputs, which, in this interaction, pertains to the variations in object shapes.

微信截图_20240213202955.png

Following successful mechanism testing with a desktop prototype, we scaled up the system to a whole-body size. Besides, the direction of the pulley system is also adjusted to facilitate horizontal force application rather than vertical. In this scaled-up mechanism, only one stepper motor is needed to drive all arms. Once the motor starts, All sets of arms are actuated and bent together, and finally enclosing user`s body. This mechanism significantly bolstered its adaptability, making it highly responsive to unpredictable bodily inputs, including user position, proximity to the machine, gestures, and various body parts etc.

微信截图_20240213203245.jpg

For further enhancements, soft inner surfaces attaching to the pole structure of the link unit are also added to enhance the adaptability of the whole system. This modification enables the inner surface to flex and pivot around the pole in response to touch.

 

​

iii.    Material Intelligence

​

The concept of "Intelligent Materials" has gained prominence in light of the rapid developments in microprocessors and artificial intelligence. Researchers defined it as materials that could optimally adapt to environmental changes and autonomously exhibit various functions, including sensing, processing, effecting, and providing feedback, through their embedded software systems (Takagi, 1990). However, in the field of interactive design, the inherent intelligence of material is always underestimated. It is a property unique to the materials (Takagi, 1990), the nature of tangible objects offering whole-sensory experience, engaging human senses, and evoking emotions and memories. This material intelligence, once considered and applied effectively, underscores the richness of multi-sensory engagement and can profoundly impact the creation of distinctive embodied experiences.

One particularly potent aspect of material intelligence lies in its perceptual dimension. When individuals interact with tangible objects, they encounter a rich array of sensations, including textures, temperature, visual and haptic feedback, all of which convey information about the object's intrinsic and extrinsic properties. For example, a piece of glass contains intricate information related to transparency, colour, shapes, density, stress resistance, and strain, which depend on the manufacturing process of the glass, the duration and temperature of firing, and the element atoms its batch contains. These rich sources of information contribute to building complex perceptual layers in the embodied experience.

Additionally, based on the huge amount of information and human`s inherent ability to associate with personal experiences, materials could evoke unique personal emotions and memories, thereby forming immediate emotional connections between the human body and the object (Crippa, Rognoli, & Levi, 2012). For instance, a wooden surface may evoke warmth and familiarity, while worn paper may elicit childhood nostalgia or melancholy. These emotional responses are not solely derived from an object's form or function but are deeply intertwined with the materiality of the object itself.

​

​

Material Perceptual Experiment

In light of these material capabilities, we conducted a material perceptual experiment, inspired by rubber-hand illusion (Ehrsson, Holmes, & Passingham, 2005), to investigate how the visual characteristics of materials affect human expectations regarding the tactile feel of those materials.

In the beginning, we used the same method as the rubber hand illusion to achieve the illusion of bodily ownership transfer – hiding participants' left hands out of sight while presenting a lifelike rubber left hand in front of them, and stroking them synchronously and the same direction with a paintbrush. After participants started to perceive the rubber hand as their own, we used four sets of contrasting materials, and applied them with the same pace and pressure simultaneously to the participants' real hands and the rubber fake hand.

Most participants expressed that the contrasting experience between visual and actual tactile sensations left a strong impression on them. When they saw the material outside, they had a prediction about how it should feel in their mind, while their hidden (real) hand was feeling some texture beyond their expectation. In this context, this contrast also happened between their expectation and real bodily experiences. As a result, 8 out of 12 participants agreed that this contrast made them pay more attention to their haptic experience, which are often overlooked in daily life. And as the experiment progressed, their anticipation and curiosity about the forthcoming hidden materials and tactile interactions continued to grow.

There`s also 2 participants claimed it was difficult for them to build the perceptual transfer to the fake hand. So the whole process are just some separately visual-haptic experiences for them.

Based on the results of the material perceptual experiment, we recognized that:

Contrasting visual appearances and real tactile textures could somehow enhance the tactile experience.

This enhancement with the material`s property of evoking multi-layered personal emotions and memories, could ultimately create a richer embodied experience. Therefore, we decided to make the soft and adaptable arms with a deliberately contrasting and uncomfortable appearance, to amplify participants' sensations of being touched by this contrast, and establish a deeper and richer connection between their bodies and the materials through this interaction.

 

Feedback of Haptic Mirror in Exhibition

The final installation took on a skeleton-like form, featuring different-textured materials, including 3D printed PLA as links, translucent urethane rubber (hardness: shore 60A) as joints, and translucent silicone rubber (hardness: shore 30A) as the inner surface, and heat-moulded 3D printed PLA as the outer surface. We showcased this installation during the London Architecture Festival. We`ve got approximately 100 people engaged with our project, and the feedback we received was quite interesting.

Most participants found the aesthetically spiky and skeleton-like form to be creepy and expected an uncomfortable tactile experience. The translucent materials visually conveyed fragility, leading participants to anticipate a subtle and delicate touch. However, everyone was pleasantly surprised by the installation's ability to provide strong and adaptable touches. While lying down and experiencing the touch, more than two-thirds of the participants perceived the experience as akin to a hug, and desired even tighter, longer, and more extensive interactions. However, not all participants enjoyed the soft and comfortable touch. Nearly half of the participants mentioned that the texture of the soft silicone (as the inner surfaces) reminded them of some unsettling soft-bodied organisms, such as octopuses. The high adaptability and strength of the touches caused them to feel pressed and trapped. A few participants also noted that the soft yet powerful characteristics of the touch reminded them of unpleasant past experiences of being touched by unexpected individuals.

Despite these varying responses, all participants agreed that the contrast between visual appearance and tactile expectations heightened their sensory experiences. Furthermore, they were particularly drawn to the tangible interactions encompassing haptic experiences, which could be attributed to this heightened material intelligence greatly evoking their personal emotions and providing a multisensory experience. The substantial enhancement of materiality during the interaction process facilitated highly embodied and unique experiences.

​

​

VI.    HUMAN-MATERIAL INTERACTION & HUMAN-HUMAN INTERACTION

 

The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.

—Mark Weiser (1991)

​

Mark Weiser introduced the concept of “Ubiquitous Computing” as a vision for the near future of human-computer interaction. In this framework, technology seamlessly vanishes, hides, and blends in everything and everywhere. In the context of this article, which explores the creation of embodied experiences, I am also trying to put technology in backstage, while highlighting the interactive body and objects, as well as the interaction between them. The “indistinguishable” purpose of computing weaving into the world serves as strong evidence for prioritizing tangible materials over digital ones, which practised in my research.

In the realm of interaction design theory, for instance, activity theory (Nardi, 1995), the analysis mostly focuses on entities (human & interactive objects), with interactive technologies merely as mediating tools to accomplish tasks or activities.

Both perspectives underscore that the core of the experience is the interactive human and interactive material. Technologies (in the form of computers) is just a mediator between the users and their goals. Regarding the notion of “material”, this paper primarily pertains to tangible forms and their inherent properties. Extensively, it can also involve the integration of physical and digital materials in the design of new interactive technologies and objects (Wiberg, 2018). Therefore, innovative technologies that blend computations with intelligent materials or artificial intelligence are highly welcomed for future consideration. The discussion of Materiality does not negate the importance of technology; rather, it is a reminder that technology, regardless of its form, should function as a medium or tool, with the user experience remaining paramount.

Within this context, there is a rapid evolution of Human-Material Interaction (HMI) which transcends beyond the conventional paradigm of Human-Computer Interaction (HCI). Hiroshi Ishii and his Tangible Media Lab were pioneers in this area. Ishii`s “radical atoms” envision a future where all digital interactions occur through material substrates, leading to a shift from HCI to HMI (Holmquist, et al., 2019). Within this materiality-centred interaction, the boundaries between the physical, digital, virtual, and intellectual realms continue to blur, with tangible entities in the spotlight.

Furthermore, the initial analysis in this paper categorised the materiality of experiences into three separate aspects: human, object, and interaction. However, given the blurred boundaries in HMI, I recognised that such divisions should be avoided. In further research, I aim to find a synchronised and direct method for enhancing all three layers of materiality concurrently and blurring them together.

One idea is making interactive objects as a reflection of the human body. In this interaction, the dynamic installation will be designed as a functional deconstruction of the flesh body,  doing human-like movements and interacting with users in a multi-sensory, tactile-driven, and performative manner. Input is derived directly from the user`s body itself, with various sensors recording bodily data and transmitting it to the installation. Each dynamic component of the installation moves based on its interpretation of the data, resulting in seemingly anthropopathic haptic interactions, visual effects, and choreographic movements. This interaction will be an endless embodied loop with the boundary between the human body and interactive objects blurring. Any enhancement methods on materiality will promote the holistic materiality of the experience.

 

Building on the experiences involving blurred boundaries, the focus on the user experience is a call to reconnect individuals with the tangible world of “atoms” under the increasingly expanded landscape of “bits”, and appreciate the richness and authenticity of the real world. Ultimately, Human-Material Interaction represents a manifesto on human-human interaction and human’s interaction with the physical world.

​

​

VII.    CONCLUSION

 

The future we are heading to appears to be a monotonous and low-entropy world with materiality gradually diminishing: The visually-dominant experiences result in artificial body-mind dualism; the pre-programmed interactions lead to the loss of uniqueness and authenticity within the physical world. This decline in materiality is happening across three domains: the human body, interactive objects, experiences.

My research seeks to reestablish the significance of corporeal reality and envisions a tangible future. To achieve this goal, I advocate for the utilization of tangible media to maintain the materiality of the interactive objects. Also, I concentrate on the fully sensory consideration on interaction mode to enhance the materiality of body. Besides, I propose the incorporation of a variety of technical design throughout the whole interaction process, including unpredictable input, adaptable mechanisms, analog control systems and research on material intelligence, to strengthen the materiality of the entire experience.

Within this framework, the concept of human-material interaction emerges as a central theme, performing as a promising pathway towards the tangible future. By prioritizing tangible interactions that are attuned to bodily perception and responsive to unpredictability, unique embodied experiences are created and beckoning people to immerse themselves within. In these experiences, the bits recede into the backstage; the richness and authenticity of the physical world back to the scene, asserting itself as a powerful enhancement to reality, rather than a replacement.

​

​

Bibliography

​​

  • Baskinger, M., & Gross, M. (2010). tangible interaction= form+ computing. Interactions, 6-11. doi:http://doi.acm.org/10.1145/1649475.1649477

  • Becker, B. (2003). Marking and crossing borders: Bodies, touch, and contact in cyberspace. Body, Space & Technology. doi:http://doi.org/10.16995/bst.218

  • Becker, K., Teeple, C., Charles, N., Jung, Y., Baum, D., Weaver, J. C., . . . Wood, R. (2022). Active entanglement enables stochastic, topological grasping. Proceedings of the National Academy of Sciences, 119(42). doi:https://doi.org/10.1073/pnas.2209819119

  • Benjamin, W. (1968). The Work of Art in the Age of Mechanical Reproduction. In H. Arendt (Ed.), Illuminations (pp. 214-240). London: Fontana.

  • Brandtzaeg, P. B., & Lüders, M. (2021). Young people's use and experience of the Internet during the COVID-19 lockdown: Well-being and social support. First Monday, 26. doi:https://dx.doi.org/10.5210/fm.v26i12.11755

  • Crippa, G., Rognoli, V., & Levi, M. (2012). Materials and Emotions: A Study on the Relations Between Materials and Emotions in Industrial Products. ut of Control : Proceedings of 8th International Design and Emotion Conference, Design and Emotion Society, 1-9. Retrieved from https://re.public.polimi.it/handle/11311/686090

  • Dollar, A. M., & Howe, R. D. (2009). The SDM Hand: A Highly Adaptive Compliant Grasper for Unstructured Environments. Experimental Robotics (pp. 3-11). Berlin, Heidelberg: Springer Berlin Heidelberg. doi:https://doi.org/10.1007/978-3-642-00196-3_2

  • Ehrsson, H. H., Holmes, N. P., & Passingham, R. E. (2005, November 9). Touching a Rubber Hand: Feeling of Body Ownership Is Associated with Activity in Multisensory Brain Areas. Journal of Neuroscience, 25(45), 10564-10573. doi: https://doi.org/10.1523/JNEUROSCI.0800-05.2005

  • Follmer, S., Leithinger, D., Olwal, A., Hogge, A., & Ishii, H. (2013, 10 8). inFORM: Dynamic Physical Affordances and Constraints through Shape and Object Actuation. Proceedings of the 26th annual ACM symposium on User interface software and technology, pp. 417-426.

  • Heyl, P. (2018, November 30). Analog Futures: Materiality in the Digital Arts. Retrieved from Interactive Architecture Lab: http://www.interactivearchitecture.org/analog-future-materiality-in-the-digital-arts.html

  • Holmquist, L. E., Zuckerman, O., Ballagas, R., Ishii, H., Ryokai, K., & Zhang, H. (2019). The Future of Tangible User Interfaces. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-6). Glasgow Scotland UK: ACM. doi:10.1145/3290607.3311741

  • Hornecker, E. (2011). The role of physicality in tangible and embodied interactions. Interactions, 19-23. doi:https://doi.org/10.1145/1925820.1925826

  • Ishii, H., & Ullmer, B. (1997). Tangible bits: towards seamless interfaces between people, bits and atoms. Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (pp. 234-241). Atlanta Georgia USA: ACM.

  • Jung, H., & Stolterman, E. (2012). Digital form and materiality: propositions for a new approach to interaction design research. Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design (pp. 645-654). Copenhagen Denmark: ACM. doi:10.1145/2399016.2399115

  • Kiley, B. (2019, 10 10). What’s so special about theater? Retrieved from Intimate Excellent, the fountain theatre blog: https://intimateexcellent.com/2019/10/10/whats-so-special-about-theater/

  • Leithinger, D., Follmer, S., Olwal, A., & Ishii, H. (2014, 10 5). Physical Telepresence: Shape Capture and Display for Embodied, Computer-mediated Remote Collaboration. Proceedings of the 27th annual ACM symposium on User interface software and technology, pp. 461-470.

  • Leithinger, D., Follmer, S., Olwal, A., & Ishii, H. (2015). Shape Displays: Spatial Interaction with Dynamic Physical Form. EEE Computer Graphics and Applications, 35, 5-11. doi:10.1109/MCG.2015.111

  • M, S., Venusamy, K., S, S., S, S., & O, N. K. (2023). A Comprehensive Review of Haptic Gloves: Advances, Challenges, and Future Directions. 2023 Second International Conference on Electronics and Renewable Systems (ICEARS), 227-233. doi:10.1109/ICEARS56392.2023.10085607

  • McLuhan, M. (1962). The Gutenberg galaxy: The making of typographic man. Toronto: University of Toronto Press.

  • McLuhan, M. (1964). Understanding Media. Boston: MIT press.

  • McLuhan, M., & Fiore, Q. (1967). The Medium is the Massage. New York: Random House.

  • Merleau-Ponty, M. (1962). Phenomenology of perception. London: Routledge.

  • Meta. (2021). Inside Reality Labs Research: Meet the team that’s working to bring touch to the digital world. Retrieved from https://tech.facebook.com/reality-labs/2021/11/inside-reality-labs-meet-the-team-thats-bringing-touch-to-the-digital-world/

  • Naegele, D. (1998). Object, Image, Aura. Retrieved from HARVARD DESIGN MAGAZINE: https://www.harvarddesignmagazine.org/issues/6/object-image-aura

  • Nanay, B. (2018, 8). Multimodal mental imagery. Cortex, 105, 125-134. doi:10.1016/j.cortex.2017.07.006

  • Nardi, B. A. (1995). Context and Consciousness: Activity Theory and Human-computer Interaction. MIT Press. doi:https://doi.org/10.7551/mitpress/2137.001.0001

  • Nixon, D. (2020, December 7). The body as mediator. Retrieved from aeon: https://aeon.co/essays/the-phenomenology-of-merleau-ponty-and-embodiment-in-the-world

  • OH, H., YI, G.-C., YIP, M., & DAYEH , S. A. (2020, November 13). Scalable tactile sensor arrays on flexible substrates with high spatiotemporal resolution enabling slip and grip for closed-loop robotics. SCIENCE ADVANCES, 6(46). doi: 10.1126/sciadv.abd7795

  • Oxford English Dictionary. (2023, July). Materiality, n. doi:https://doi.org/10.1093/OED/2064247878

  • Phelan, P. (2021). The ontology of performance: representation without reproduction. In P. Phelan, The ontology of performance (pp. 147-166).

  • Shadow Robot. (2021, February). Dexterous Hand Series. Retrieved from Shadow Robot: https://www.shadowrobot.com/dexterous-hand-series/

  • Tai, K., El-Sayed, A.-R., Shahriari, M., Biglarbegian, M., & Mahmud, S. (2016). State of the Art Robotic Grippers and Applications. Robotics, 5(2). doi:https://doi.org/10.3390/robotics5020011

  • Takagi, T. (1990). A Concept of Intelligent Materials. Journal of Intelligent Material Systems and Structures, 1(2), 149-156. doi:10.1177/1045389X9000100201

  • The Yale GRAB Lab. (2019, June 28). Model Q. Retrieved from YALE OPENHAND PROJECT: https://www.eng.yale.edu/grablab/openhand/model_q.html

  • Weiser, M. (1991, September). The Computer for the 21 st Century. Scientific American, 265(3), 99-105. Retrieved from http://www.jstor.org/stable/24938718

  • Wiberg, M. (2018). The Materiality of Interaction: Notes on the Materials of Interaction Design. The MIT Press. doi:10.7551/mitpress/9780262037518.001.0001

  • Wikipedia. (2020, February 13). Materiality. Retrieved from Wikipedia: https://en.wikipedia.org/w/index.php?title=Materiality&oldid=940523026

Abstract

​

This paper questions the homogeneity and perceptual constraints inherent in the embodied experiences created by the pre-programmed intangible media. By discussing the medium and the global village idea of Marshall McLuhan, I question the burgeoning digital technologies within the realms of performance, contemporary art, media, and interactive design. I argue that the message they carried and the experiences they created are an interpretation with great deduction of uniqueness. In the pre-programmed process, the uniqueness and real-time presence, or the materiality of the experience, are gradually dissipating, resulting in a monotonous and low-entropy future. The ignorance of perceptual richness and personal embodiment deepens the dualism of body and mind.

 

My research tries to draw a picture of a more tangible future in this increasingly digital world. I advocate for enhancing the materiality of all layers of experiences (which are the Materiality of the Human Body; the Materiality of Interactive Objects; the Materiality of the Entire Experience) as a main pathway to this vision. The inclusion of random and personal input re-emphasises the physical existence of the human body. Through the research on soft robotics and adaptable mechanism, I demonstrate how analog mechanisms swiftly respond to unpredictable input. Additionally, through the creation of multisensory encounters, with a particular emphasis on haptics and materiality, I test how the materials evoke personal emotions and memories, ultimately fostering a reconnection of body and mind. In conclusion, my research shows how tangible media with an emphasis on haptics has greater potential to restore the uniqueness and authenticity of embodied experiences, leading to the future of human-material interaction where reality is enhanced rather than replaced.

​

bottom of page