Read more...

Eyes-free interaction with gestural control and auditory feedback

 

Second Thumbnail label

Cras justo odio, dapibus ac facilisis in, egestas eget quam. Donec id elit non mi porta gravida at eget metus. Nullam id dolor id nibh ultricies vehicula ut id elit.

‹ ›

References

{bibtex}virtualacoustics/eyesfree_page.bib{/bibtex}

Using Conductor Follower can be briefly described as conducting a virtual orchestra. The system uses a Kinect (or any other OpenNI compatible device) to capture hand movement. This data is analyzed, and the infromation used to process score data from a MIDI file. A synthesizer is then controlled using the modified MIDI data.

The final outcome of the projet is a VST plugin, which produces a stream of MIDI events. The project was coded in C++, using OpenNI for hand tracking, and Juce for VST and MIDI related functionality. The Boost C++ libraries were also heavily utilized. The source code and documentation for the project are available online on GitHub.

Demo video

The demo video below is a screen capture, including the visualization part of the plugin. It shows the basic functionality as I conduct part of Mozart's Trio for piano, clarinet and viola in E flat major, KV 498.

http://www.youtube.com/watch?v=DoGmnLHjI6w

Acknowledgment

The project was implemented as part of my master's thesis, and has received funding from the European Research Council under the European Community's Seventh Framework Programme (FP7/2007-2013) / ERC grant agreement no. [203636].

References

Sakari Bergen, "Conductor Follower: Controlling sample-based synthesis with expressive gestural input ", Master's Thesis, Aalto University, 2012

Sakari Bergen, The thesis presentation, made with Prezi.