Pianoverse
How can we assist piano learning and make it more accessible using VR and haptic wearables?
Role
VR Prototyper
Overview
Pianoverse is a VR Piano Tutor program that allows users to play a virtual piano with an accompanying haptic glove. Pianoverse offers visual cues to aid with piano learning, and the haptic glove will dynamically guides finger movements in sync with note sequences to develop muscle memory for specific chords/songs.
Pianoverse is aimed for beginners who want to break into learning piano but have limited resources for a real instrument and piano lessons, or for those who want to try out before deciding to commit. The system offers a budget-friendly and compact alternative to conventional piano learning.
Team
Jamie Dinh
Mrunal Dhaygude
Arnav Mittal
Ritik Chopra
Rohan Mathur
Timeline
Aug – Dec 2023
Tools
Unity3D
Arduino IDE
Autodesk Fusion 360
"What should we do?"
At the conception of our project, we were actually unsure of where to take our ideas. As a group we were pretty set on using VR or AR technology in combination with haptics to create an immersive experience, but were unsure of how we wanted to apply those technologies in a meaningful way. We wanted to experiment with the applications of VR and haptics in regards to music and instruments, and came up with a few ideas.
How might we assist piano learning through Virtual Reality and Wearables?
After brainstorming, we settled on the idea of a VR piano tutor with a wearable EMS (Electric muscle simulation) glove. The project aims to provide beginners a budget-friendly and space-efficient alternative to conventional piano learning.
We present an interactive virtual piano with a complete range of 88 notes. The program facilitates piano learning by indicating which note to press based on a music sheet, while the wearable glove synchronizes the user’s finger movements in real time with the note sequences. Visual indicators will be provided to signify whether user has correctly or incorrectly pressed the notes.
Demo
Building the VR scene
The VR scene includes a pre-made piano prefab with 88 keys. Utilizing Meta Quest’s hand tracking and collider component, we enabled interaction between each piano key and Oculus’s hand prefab, ensuring that the keys respond whenever the hand makes collision. Additionally, we integrated audio corresponding to each note, ensuring that the respective sound plays each time a key is touched.
In practice mode, we created a note sequence corresponding to the music sheets, and assigned delay time based on the length of each note. Visual cues, such as falling key visualization, highlight which note should be pressed. Furthermore, notes change color to red or green to signify incorrect or correct pressing by the user.
The Wearable EMS glove
At first, we visualized the idea with a low-fi prototype using low-cost material. Later on, we examined Electric Muscle Simulation (EMS) pads to induce finger movements.
Milestone #1
Early work
The gloves are meant to hold the fingers back when the EMS is activated.
Steps to restrain fingers:
- A servo motor is used to rotate a piece of string tied around the servo horn
- The string is connected to a ring that can be fastened around the user’s finger
- This restricts the user’s range of motion, so that they can’t bring the targeted finger all the way down
Milestone #2
Testing the EMS
We conducted research and experimentation to determine the optimal placement of EMS pads on the hand and arm, targeting specific fingers and nerves. Additionally, we explored the possibility of using EMS to rotate or move the entire arm, providing additional sensory feedback to guide users on where their hands should be positioned.
Final prototype
We have wirelessly integrated VR technology with a wearable device that sync together in real-time. The wearable was achieved through a combination of servo motor control and EMS feedback, providing an immersive and interactive platform for teaching piano muscle memory.
Device control mechanism
- Device Control: Linked VR headset with the wearable device via a Peer-to-Peer network and ESP32, allowing real-time wireless communication for instruction transmission from the VR headset to the ESP32.
Software Integration
- Communication Setup: Established a Wi-Fi hotspot, enabling connection between the VR headset and the ESP32. Employed Arduino WebSockets library for server creation and Native WebSockets library for Unity integration.
- Instruction Transmission: Enabled the VR headset to send control commands to the ESP32 over the WebSocket connection, facilitating finger and hand manipulation based on the VR scene.
Arduino Code Execution
- Global String Handling: Stored the most recent sent value in a global string on the Arduino side.
- Note Activation: Activated finger movements corresponding to notes played in the practice mode using EMS and servo motors.
- Reset Functionality: Implemented code to reset all servos and EMS intensity back to default after the completion of the note sequence.
