Visit here the installation I created with UXRzone in December 2020.

Animated 3D Models of the characters

After a discussion with UXRzone’s technician, Marco Modena, we decided that instead of using videos of the musicians, dancers, and actors, we will make 3D models of them, animated between the boundaries allowed in Mozilla Hubs. It will create a more unified ambiance, a whole and consistent reality in 3D. Keeping 2D videos of different qualities would break this unity.

The challenge was to take the videos sent by the artists and turn them into an animated 3D model. Marco discovered a very recent algorithm that allows us to capture the motion of a 2D video and translate it into a 3D model. And so we started with improvised movements sent by Berlin-based dancer Deva Schubert. 

Deva’s video was our first tryout. The results were not as expected, as the software has problems with occluded extremities. Deva’s video was full of turns; therefore, her extremities were interfering with each other. 

Marco made changes to the rigging, which is the internal connection system that acts as the model’s skeleton. This edition made the movements credible, although they differ a bit from the original dancer’s movements. 

Thanks to this experience, we know now that the videos must avoid occlusion as much as possible, and the joints (elbows and knees) must be evident and visible. 

Deepmotion Animate 3D

Since Mozilla doesn’t allow us to upload a long animation into the scene, we decided to choose small loops to animate the models. Fundamental movements that are expressive and capable of conveying important actions and emotions are selected for this. 

To create the characters, we decided to use the technology offered by Ready Player Me to create a 3D avatar from a 2D picture for video games. 

All the models for the opera, including all the team involved, including mine and that of the character of Snowden, are then subject to texture modifications by Marco, using the software Blender

The motion capture from the 2D videos is applied to the 3D models that still retain the facial characteristics of the artists and Snowden but have a completely different texture. We could see this as the costumes of the characters of the Opera.

3D Stage and The Dome Remixed

The Dome Remixed

The original Dome

We decided to use UXRzone’s venue for the opera. But this doesn’t mean that we have to use the venue as usual operas do. Instead of changing the stage with different props for each act, the dome itself becomes a different space. A sort of REMIX of the room itself, let’s say. 

For the “Office in Hawaii” act, when Snowden copies in a USB the data to be exposed, The Dome becomes a Point Cloud structure, as does almost everything in it, including Snowden’s office. 

A point cloud is a set of data points in a 3D coordinate system used to represent an object’s surface and doesn’t contain any internal features like color, materials, etc. A point cloud is a perfect allegory of the realization that humans have become simply data points for big machineries of control, losing their internal identity. The awful realization that humans can be deprived of their sentient being status, their privacy, and their free will, is what moves Snowden into taking the heroic risk of making what is right for the good of all. 

Avatars that the audience can select

Everyone can enter the opera with an avatar of their choosing. Still, we will provide three alternatives, in case they want to participate in the opera as a specific character:

  1. Snowden Avatar

The central teaching from Edward Snowden is that we all can be Snowden; we all can be heroes. A truly heroic act is not done to glorify the ego but for the good of all. We all can be heroes by choosing to do what is right for a dignified life on this planet, no matter how big these acts can be.

  1. Corporate Avatar

This avatar represents the greed and the sense of lack that instigate corporations and institutions to look for ways to control and reduce the freedoms of the individuals. It’s the corrupt military that kills thousands of people with just one order; it’s the demon behind each one’s ear that chooses corruption in exchange for security.

  1. Human individual Avatar

This avatar represents the individual escaping its state as a simple data point. The individual that awakens to the reality that the freedom we think we live in is not exactly as we imagine. It is the individual that takes the reigns of their life, reads, talks, researches, and decides to question their culture.

Alexandra Cárdenas

composer/programmer/improviser/live coder/algoraver