Xiao Xiao (Media Lab, Massachusetts Institute of Technology, Boston, USA). Reflecting Music Through Movement: A Body-Syntonic Approach to Playing [with] the Piano.

This talk presents methods for the capture and reproduction of music on the piano that maintains a tight coupling between the body and the physical instrument. Music playing originates in the body, which plays a central role in both performance and pedagogy. However, technologies for the capture and reproduction of music has largely ignored the intricate relationship between the sound, the body, and movement in the making and appreciation of music.

Drawing from research in telepresence, tangible interfaces, and augmented reality, I present two projects that bring the movement of the human body back into the picture by projection-mapping on a player piano. The first, MirrorFugue, is inspired by reflections on the surface of a lacquered grand piano and simulates the presence of a virtual pianist who plays the physically moving keys. The other, Andante, depicts musical phrases as silhouettes of miniature animated figures that walk and dance across the piano keyboard. Drawing on human empathy and the importance of integrating sensory modalities, both installations are designed as immersive “sandboxes” for the playful exploration of musical ideas, embodied cognition, and the role of motion and emotion human experience. I discuss experiments and experiences of both projects for music learning and enjoyment.

Beyond these projects, this talk explores music as a source of insights on strategies for learning across domains. Inspired by Marvin Minsky’s concept of the Society of Mind and Seymour Papert’s concept of body-syntonicity, I use the process learning music as a case study of building “infrastructure” within the mind-body. I discuss an application of these ideas in the domain of technology-assisted language learning, presenting preliminary results from my current research at LAM on learning Chinese tones with the help of Vokinesis, a vocal synthesizer controlled by hand-writing movements from a tablet and stylus.