Demo 🔊 Turn On Your Volume
FINAL DESIGN
Role
Concept Design
Interaction Design
Audio Design
Duration
2 weeks - Dec. 2022
Tool
Unity 3D, Figma, Blender, Adobe Premiere Pro, Ableton Live, C#
Team
Yolanda Yu - Designer
Clayton Izuka - Audio Engineer
Sin-Yu Deng - Video Editor
Thomas Giberson - Developer
XR Device
Oculus Quest
OVERVIEW
360° MR Concert
This project is a mixed reality concert production for the song, Lemon, by Kenshi Yonezu with Unity. It is covered by 4 Chinese traditional instruments musicians.
In this 360° cyberpunk style environment, users can control the real performance as a DJ/mixer to play solo, mute, reverb and adjust volume on each instrument by interacting with a hand menu. Also they can enjoy the aesthetic environment to know more about stories behind the song, like context and lyrics.
CHALLENGE
Create an interactive mixed reality concert experience with audio and video files of a music performance captured with a VR camera.
Visual:
-
Use the panoramic skybox shader to display video materials in VR.
-
Add an interaction layer where user can manipulate their concert experience.
-
Add other mixed reality assets in accordance with the music to further blend real and virtual elements in VR.
Audio:
Use Resonance Audio to position the individual musician sound source in a 3D spatial audio scene relative to the placement of the performers in the video.
REFLECTION
Technical Challenges
For development, we are excited about creating a mixer-like experience with features such as solo, mute, and volume sliders. We are constantly exploring ways to emulate the user experience of Mixer. While different DAWs have varying workflows, we found that aligning with Ableton Live's solo button functionality worked best for our project. This decision allows us to focus on delivering a high-quality experience without the complexity of accommodating multiple scenarios. We remain committed to enhancing our coding skills to further perfect our project.
CONTRIBUTION
What I am responsible for?
As a product designer of the team, I initiated and developed the concept idea with my members. Technically, I am primarily responsible for UI and interaction design in Unity 3D . Also I implemented resonance audio in the environment. During testing, I helped debug the scripting on UI controls in the whole process.
For work inquiries, or to grab a coffee chat to know more about me, feel free to or email me or connect on Linkedin ! 😊
✨Thank you for visiting.
UI DESIGN
Global UI Panel Design
Local Panel Design
How did I design this interactive control panel?
We aimed for a minimalistic and modern UI design that would be easily visible in the dark environment. Inspired by digital music players, I created a local control panel for each musician to merge with the environment. Additionally, I designed a global panel resembling a mixer to control all instruments. Both designs worked well, but due to technical implementation limitations, the team decided to use the global panel solely as a hand menu.The hand menu is accessible by pressing the "X button" on the left hand controller and can be closed with one more press. Users can use laser to interact with the hand menu by holding the right hand controller's index trigger.
Other Works
🏆 Best Interaction Design (Oculus)
🚗 Find your car smartly w/o internet
🌱Plant to compose your music
ENVIRONMENT DESIGN
BEFORE
AFTER
How did we generate this concept?
We discovered that the original raw video implemented in the panorama skybox lacked user comfort due to its monochromatic color, light noise, and classroom settings. To improve the experience, we edited the 360 video using "blender" and created a cyberpunk-style environment inspired by New York Times Square. Our immersive world view featured Japanese-style billboards and posters related to the song "Lemon" by Yonezu Kenshi. Moreover we used LED-like rendering, added sci-fi elements, and included instrument names in the video for clear correspondences with the interactive interface.
OCULUS QUEST2 GAMEPLAY INTERACTION
