'Paradise' tech lead visualizes mood

Mood-generated, computer-rendered fireflies appear on stage in “Paradise Interrupted.”

No two performances of “Paradise Interrupted” are the same. The opera begins with soprano Qian Yi singing of her desire on a plain white stage. The atmospheric visuals that ripple behind her appear differently at every performance for one simple reason: they are interactive.

Directed and designed by Jennifer Wen Ma and composed by Huang Ruo, the installation-opera made its world debut at Spoleto Festival USA on May 22, and audiences have two more chances to see the fluid visuals for themselves. The last two performances are Friday and Sunday.

Creative Technology Director Guillermo Acevedo, 48, is in charge of the interactive components. Acevedo spent the last three years customizing sound recognition software and writing code that teaches a computer to recognize emotion. He then uses software to generate images that change in response to the moods conveyed in Qian Yi’s voice.

How do you teach a computer to recognize someone’s mood? It helps that Acevedo has seen computers as thinking, if not living, machines since he was a kid.

“I always wondered when a computer crashed if it was in pain, or getting a migraine,” he said. “Right now we think of computers as inanimate objects, but what if they are not?”

Acevedo worked with an open source visual programming language, Pure Data, to capture Yi’s volume, tone and pitch and analyze her tempo in order to access her mood as she sings. The software helps him create visuals that complement her performance, a capability he calls “emotional computing.”

He equates programming with problem solving: The job’s not done until you fix what doesn’t work. For Jennifer Wen Ma, who conceived of “Paradise Interrupted,” interactivity was the solution she was looking for.

“I really like to bring technology and contemporary ideas to ancient philosophies or cultural relics,” Ma said. “Early on, I knew I wanted to create this world and this garden where a woman was in search of an unattainable ideal, but I kept thinking, for a contemporary opera it needed that cutting-edge element. For ‘Paradise Interrupted,’ Guillermo was the missing link.”

Luckily for Acevedo, Qian Yi signed on early in the project, which allowed them to begin working on the emotionally responsive programming before “Paradise Interrupted” was even written. Yi sang the libretto from the classical Chinese opera “The Peony Pavilion” instead. After Acevedo developed the coding for the projections, he worked extensively with projection designer Austin Switser to bring the mix of interactive and non-responsive visual effects to life.

Ma and Acevedo first collaborated on “Nature and Man in Rhapsody of Light at the Water Cube,” a 2013 light and color installation that transformed Beijing’s National Aquatics Center into a larger-than-life cultural mood ring. Acevedo used a programming toolkit suited for large media environments and real-time motion graphics called vvvv to determine which color will pulse through the exterior of the water cube each day. The software combs through multiple Chinese social media platforms to gauge the users’ general mood.

Acevedo also uses vvvv to generate the “Paradise” visuals. Unlike those for the water cube, the visuals in the opera were crafted to have a natural, almost random feel, he said. For audiences, there are a few moments where the specialized projections are noticeable, but they are not meant to be pronounced.

“It’s about (the heroine’s) search and her voice, her breath and her energy,” Ma said. “I wanted the projections to manifest in a way that will really show what breath is like, which is all the time changing.”

According to Acevedo, who sang in a choir and tried his hand at acting in college, the projections should be subtle, enhancing the mood, not distracting the audience.

“I approached the visuals in ‘Paradise’ as if they were one more character,” he said. “When you’re onstage you react to the other actors. When one actor is performing with high energy you respond to that as an actor. I wanted that to be a part of ‘Paradise.’ The projections aren’t just there. They are reacting and responding to Qian Yi and her performance just as a fellow actor would.”

Lauren Cavalli is a Goldring Arts Journalist from Syracuse University.