Gesture synth on iPhone: air instruments you play with your hands
Musaic turns the iPhone's front camera into seven instruments you play in the air, without touching the screen: Chords, Penta, Halo, Theremin, Bellows, Handpan and Rails. The camera tracks your hands, and in a video project it films you at the same time, so the take is a video of you playing and not only the sound. You can sing while you play, and your voice lands on its own layer called Voice. The Air instruments are free and marked Beta.
Musaic - Recording Studio is free for iPhone and iPad. Where a step needs Musaic Pro, the guide says so.
You need: The front camera, your hands in the picture and some light. Put the phone down so you have both hands free.
-
Open Air instruments
Tap Add layer and choose Air instruments ("Played with your hands in front of the camera").
-
Pick an instrument
Choose one of the seven. Each has a short explainer that shows the moves.
-
Show your hands
Until the camera sees a hand, the recorder says Hold a hand up to the camera.
-
Play and sing
Record while you play. The instrument and your voice can land as two layers: the instrument, and a layer called Voice with your singing.
-
Chords Beta
The left hand chooses the chord by the number of fingers you show, from I to VII; a fist is a rest. The right hand shapes it: one to four fingers for the voicing, the thumb for an octave lower, the height of the hand for the volume.
-
Penta Beta
Each finger is a note, from the thumb (low) to the little finger (high). Straighten a finger to play it, curl it to stop; several fingers make a chord. The left hand plays an octave lower.
-
Halo Beta
Twelve notes form a ring around your palm, the lowest at the top, going clockwise. Reach a finger out through the ring to play the note in that direction.
-
Theremin Beta
Slide one hand left and right for the pitch, without steps. Lift the other hand to play louder, drop it to fade.
-
Bellows Beta
Pull your hands apart: wider is higher. Press your palms together to stop. It snaps to the song's key unless you turn that off.
-
Handpan Beta
The notes lie in front of you, low on the left, high on the right. Rest a hand on one to hold it, or strike it from above for a harder hit.
-
Rails Beta
Two bars, one per hand, low on the left and high on the right; higher up means a higher note. Hold a hand in a field to play it; both hands make a chord.
-
Is this the Gesture Synth from Instagram?
No. Gesture Synth is a browser instrument by Eric Wei. Musaic's Chords instrument plays chords with the same kind of hand shapes, on the iPhone, and records the video and your voice with it. Musaic and Gesture Synth compared.
-
Which camera?
The front camera, so you see yourself while you play. No Face ID sensor or extra hardware is needed.
-
Is there a theremin app for iPhone that I play with my hands?
Yes. Theremin is one of the seven Air instruments: one hand slides left and right for the pitch, the other lifts for the volume, and the front camera reads both.
-
Can I play a handpan on my iPhone?
Yes, in the air. Handpan lays its notes out in front of you, low on the left and high on the right. Rest a hand on a note to hold it, or strike it from above for a harder hit.
-
Does it film me while I play?
In a video project, yes: the camera records the picture together with the instrument, and the take becomes a layer like any other. In an Audio project only the instrument is recorded.
-
Does the camera picture leave my iPhone?
No. The hand tracking runs on the iPhone itself, with Apple's Vision framework.
-
Do they cost extra?
No. The Air instruments are free, like the other instruments.
Musaic - Recording Studio is free on the App Store for iPhone and iPad, and so are the Air instruments. Musaic Pro removes the watermark from the exported video and adds lyrics, animations and chord detection.
- Play instruments on your iPhone, on screen or with a MIDI keyboardGuide
- Detect the chords of your song and show them in the videoGuide
Written for Musaic 1.4.0.