Gesture-Driven Interaction Using the Leap Motion to Conduct a 3D Particle System: Evaluation and Analysis of an Orchestral Performance

Abstract

In this work, we present and evaluate an interactive simulation of 3D particles conducted by the Leap Motion, for an orchestral arrangement. A real-time visual feedback during gesture entry is generated for the conductor and the audience, through a set of particle emitters displayed on the screen and the path traced by the captured gesture. We use two types of data input: the captured left and right hand conducting gestures (some universal movements, such as the beat patterns for the most common time signatures, the indication of a specific section of the orchestra, and the cutoff gestures), which are responsible for setting the tempo and dynamics of the orchestra; and a MIDI file, which contains information about the score of an orchestral arrangement, defining which notes each musical instrument should play. As regards the gestural input of the conductor, we have considered two musical elements of expression: tempo and dynamics. Besides performing functional testing, we analyzed the simulation results focusing on the occurrence of false positives, false negatives, positional deviations and latency. Moreover, a professional conductor evaluated our system and provided qualitative feedback about it.