“We believe that, by leveraging the collaborative nature of remix composition and musically-oriented computer programming, EarSketch may provide a successful alternative to the cultural issues that computer games have in the engagement of minorities,” said Brian Magerko, Assistant Professor of digital media in the IAC School of Literature, Communication, and Culture and Principle Investigator on the project.  Working with Magerko is co-investigator Jason Freeman, Assistant Professor of music in the College of Architecture School of Music.

The project involves development of a software package and curricula. Students will be taught how to use a digital audio workstation to control musical loops and beats by writing small bits of programming code.  

“We’ll also provide online resources through which students can share their works, encouraging them to learn to remix with other students’ codes and sounds,” said Magerko. “The end result will be music that they can be proud of and that helped students learn introductory concepts in computer science.” 

Funded by the National Science Foundation’s Computing Education for the 21st Century (CE21) Program, EarSketch will be piloted in 2014 in Atlanta's Lanier High School.  Industry support for the project came from Cockos, who has supplied discounted licenses and technical assistance for their Reaper digital audio workstation software which is the development environment for EarSketch.

Photo: Shannon Yao and Tom Jenkins, graduate students in the LCC Digital Media program, working on development of EarSketch software

The new software, called ZooZ Beat, allows the user to play and record a variety of instrumental sounds by shaking and tilting the phone. It also allows entering and transforming voice recordings and sharing the music in a group. The software interprets the movements and manipulates the music accordingly.

Weinberg says he created the software after realizing how his previous research on musical expression and creativity for novices could be applied to cell phones, which have become much more ubiquitous and powerful than in years past.

"My research focuses on developing algorhythms that would allow musical instruments to analyze and interpret a player's intention," said Weinberg. "With this technology, you don't have to have a lot of skill or know a lot of music theory to become musically creative. You can just use your own expressive, intuitive gestures to create music that you can relate to."

The new software, called ZooZ Beat, allows the user to play and record a variety of instrumental sounds by shaking and tilting the phone. It also allows entering and transforming voice recordings and sharing the music in a group. The software interprets the movements and manipulates the music accordingly.

Weinberg says he created the software after realizing how his previous research on musical expression and creativity for novices could be applied to cell phones, which have become much more ubiquitous and powerful than in years past.

"My research focuses on developing algorhythms that would allow musical instruments to analyze and interpret a player's intention," said Weinberg. "With this technology, you don't have to have a lot of skill or know a lot of music theory to become musically creative. You can just use your own expressive, intuitive gestures to create music that you can relate to."

Weinberg says that he always wanted to put this technology into the hands of everyone, but his previous instruments were too expensive and difficult to maintain.

"Cell phones have become so powerful as far as their capabilities, which led me to think that I could bring some of my research ideas into this realm," says Weinberg. "I don't have to develop the hardware, and everyone already has a cell phone. By making the software easily accessible, people will be able to create, manipulate and share music in a very intuitive and expressive manner."

Weinberg has also applied the software technology to gaming by using the cell phone device as a game controller for PC games.

Georgia Tech Venture Lab has supported the commercialization of Weinberg's technology, and he has a number of patents pending. The software will be available to the general public at http://www.zoozmobile.com.

The department's director, Frank Clark, is attracting faculty and students who have talents in both the music and technology sector.

"We're looking for people who are genuinely interdisciplinary, people who are interested and passionate about more than just one thing," said Clark. "That's really important because you can be a musician and a computer scientist, a musician and a mechanical engineer, a musician and a mathematician. Georgia Tech's Music Department and Center for Music Technology are creating ways for those skills to come together in real, meaningful ways."

Clark practices what he preaches. The director is an accomplished musician, but his research delves into ways to bring together various elements of multimedia simultaneously.

"What that means is, let's say I'm interested in rhythm and color," said Clark. "What then does rhythm, color and music look like? What is rhythm, color and graphics? What is rhythm, color and animation? My research is trying to find ways to bring those elements together in performances that actually have substance and meaning."

Clark says that technology in the music industry is changing daily and the new center will provide a place for researchers from across campus to see how their expertise can be applied to music.

"One of the things we firmly believe is that the vast majority of careers that combine
music and technology in the 21st century have yet to be invented," said Clark. "The center will provide a place for researching areas that haven't been explored yet. You'll have material scientists working side by side with musicians, computer scientists and mechanical engineers."

According to Clark, researchers will be studying everything from improving existing instruments to creating new instruments. There will be robotic projects, interactive performances and even applications for mobile devices that will all be associated with the new Center for Music Technology.

Shimon is able to interact with his human counterparts on a much more social level. The robot's head is made to be more interactive and give fellow musicians social cues that relate to the music it's playing.

"I really wanted to make this robot more socially dynamic to enrich the interaction experience for the human musicians," said Gil Weinberg, director of Music Technology. "We start with simple social cues such as recognizing a beat and moving the robot's head. Sort of getting itself into the groove."

Shimon is able to interact with his human counterparts on a much more social level. The robot's head is made to be more interactive and give fellow musicians social cues that relate to the music it's playing.

"I really wanted to make this robot more socially dynamic to enrich the interaction experience for the human musicians," said Gil Weinberg, director of Music Technology. "We start with simple social cues such as recognizing a beat and moving the robot's head. Sort of getting itself into the groove."

"There is really a back channel of social cues that go on between musicians," said Andrea Thomaz, assistant professor in the School of Interactive Computing, who is collaborating with Weinberg on building the robot's head. "Shimon's head is really meant to take Gil's robotic musicians into that realm of being a true social collaborative music partner."

"When a guitar player and a drummer want to finish a piece together, there are synchronization and anticipation social cues given," said Weinberg. "With Shimon, there are four arms that stretch over a large instrument that give other musicians anticipatory cues of what is going to happen next."

Shimon is able to interact with the environment around it, analyze rhythm, melodies and harmony and use his musical understanding to improvise with humans.

Weinberg says that it can help study the way we think and play music because it expands the knowledge we have about music making and the musical mind.

Haile, Weinberg's first robotic percussionist, played in venues all around the world and has led to additional research in human-robotic interaction. The Robotic Musicianship project, which led to the development of Shimon, was supported by NSF and by the GVU Research Innovation grant.

The gadget was successfully used by individuals with limited feeling or movement in their hands due to tetraplegia. These individuals had sustained their injury more than a year before the study, a time frame when most rehab patients see very little improvement for the remainder of their lives.  Remarkably, the device was primarily used while the participants were going about their daily routines.

The device is called Mobile Music Touch (MMT). The glove, which looks like a workout glove with a small box on the back, is used with a piano keyboard and vibrates a person’s fingers to indicate which keys to play. While learning to play the instrument, several people with SCI experienced improved sensation in their fingers.

Researchers at Georgia Tech and Atlanta’s Shepherd Center recently completed a study focusing on people with weakness and sensory loss due to SCI.

“After our preliminary work in 2011, we suspected that the glove would have positive results for people with SCI,” said Ph.D. graduate Tanya Markow, the project’s leader. “But we were surprised by how much improvement they made in our study. For example, after using the glove, some participants were able to feel the texture of their bed sheets and clothes for the first time since their injury.”

Markow worked with individuals with SCI who had limited feeling or movement in their hands. Each suffered a spinal injury more than a year prior to the study. The eight-week project required study participants to practice playing the piano for 30 minutes, three times a week.  Half used the MMT glove to practice; half did not.

The MMT system works with a computer, MP3 player or smart phone. A song, such as Ode to Joy, is programmed into a device, which is wirelessly linked to the glove. As the musical notes are illuminated on the correct keys on the piano keyboard, the gadget sends vibrations to “tap” the corresponding fingers. The participants play along, gradually memorizing the keys and learning additional songs. 

However, these active learning sessions with MMT were not the primary focus of the study.  The participants also wore the glove at home for two hours a day, five days a week, feeling only the vibration (and not playing the piano).  Previous studies showed that wearing the MMT system passively in this manner helped participants learn songs faster and retain them better.  The researchers hoped that the passive wearing of the device would also have rehabilitative effects. 

At the end of the study, participants performed a variety of common grasping and sensation tests to measure their improvement.  Those who used the MMT system performed significantly better than those who just learned the piano normally.

“Some people were able to pick up objects more easily,” said Markow. “Another said he could immediately feel the heat from a cup of coffee, rather than after a delay.”

Markow believes the increased motor abilities could be caused by renewed brain activity that sometimes becomes dormant in persons with SCI. The vibration might be triggering activity in the hand’s sensory cortex, which leads to firing in the brain’s motor cortex. Markow would like to expand the study to include functional MRI results. 

The glove has evolved in recent years under the leadership of Georgia Tech’s Thad Starner and Ellen Yi-Luen Do, as well as Deborah Backus, director of multiple sclerosis research at Shepherd Center. The initial concept, Piano Touch, developed with the team by then master’s student Kevin Huang, demonstrated that people could easily learn to play the piano by wearing the glove and feeling its vibrations. It didn’t take long for Starner to see the larger health benefits.

“Equipment used for hand rehabilitation may seem monotonous and boring to some, and doesn’t provide any feedback or incentive,” said Starner, who oversees the Contextual Computing Group. “Mobile Music Touch overcomes each of those challenges and provides surprising benefits for people with weakness and sensory loss due to SCI. It’s a great example of how wearable computing can change people’s lives.”

Starner is an associate professor in the School of Interactive Computing. Do is a professor in the Schools of Interactive Computing and Industrial Design.

Earsketch was designed by School of Music Associate Professor Jason Freeman and School of Literature, Communication and Culture Assistant Professor Brian Magerko. The software was created to address the nation’s shortage of high school students, especially females and minorities, who are interested in computer science careers.

“We believe that we can get people more motivated in this field by placing introductory computing education into a really interesting, fun context,” said Freeman. “Instead of writing programs that sort lists or crunch numbers, students learn all of these skills while making music.”

Earsketch utilizes the Python programming language and Reaper, a digital audio workstation program similar to those used in recording studios throughout the music industry. Students are remixing samples and loops to create their own tracks.

“We use hip-hop music as a core musical domain because of its history of remixing music and its strong roots in Atlanta,” said Magerko. “But students can work in any genre that speaks to them.”

Georgia Tech was awarded a three-year grant in 2011 by the National Science Foundation to test and implement Earsketch. The software and curriculum will be piloted this spring at Lanier High School in Gwinnet County as part of the school’s music technology program.

The Earsketch workshop is one of several summer camps organized on campus by the Institute for Computing Education. The program is overseen by Barbara Ericson, Director of Computing Outreach in the College of Computing.

Freeman is a faculty member in the College of Architecture. Magerko is a faculty member in the Ivan Allen College of Liberal Arts.

Over the years, researchers in Georgia Tech’s Sonification Lab (SonLab) have converted numerical data into sounds to analyze stock market prices, election results and weather data. When the reggae/rock band Echo Movement called wanting to turn the movements of celestial bodies into music, SonLab looked to the heavens.

“The Sonification Lab receives a lot of requests to convert scientific data into sound, but this one was truly unique,” said Bruce Walker, a professor in the Schools of Psychology and Interactive Computing. “It’s not often that we have a chance to help an actual star compose music.”

Although pitches, tempos and rhythms could be created and tweaked, the band insisted that the finished product remain true to all data and feature a musically appealing, “heavenly” sound. With those restrictions in place, the musicians and Walker’s team of students went to work with existing data gathered by NASA’s Kepler telescope. Focused on a binary star (Kepler 4665989), Kepler recorded its brightness levels for more than a year. The star dimmed and brightened each time its companion star crossed its path, providing varying brightness measurements.

“Those numerical values were loaded into our Sonification Sandbox software to create sequences of sonified musical pitches,” said Riley Winton, a psychology student and leader of the project. “The process put us on the right track. When the band reviewed it and requested timbres instead of pitches, we audified the data.

In other words, the team played the varying brightness levels as waveforms to create a different sound. The lab then cleaned the signal and removed some of the ambient sound before sending audio pitches to the band. Echo Movement looped the sounds and composed them into a four-part harmony.

For the final step, the students used a different binary star (Kepler 10291683) to adjust the timbre even further by adding a tremolo effect. This created a shuddered, natural sound rather than a flat, computerized noise.

The final result is a melody that will be used in the intro of Echo Movement’s song “Love and the Human Outreach,” which will be released in September.

“People have made music with space sounds before, but largely using pulsars and space events that can be recorded in the radio spectrum. We wanted something completely off the chart,” said band member David Fowler, who was encouraged by Edna DeVore at the SETI Institute to look at the Kepler Mission. “Discovering planets around other stars is a relatively new science worthy of everyone's attention and digs deep at the core of humanity's most basic quest to orient itself in reality,” he said.

The Georgia Tech team will present the sonification process at the International Conference on Auditory Display (ICAD) in Atlanta June 18 – 21, 2012.

The project’s goal, to create an authentic, aesthetic sound, was a success. The melody is further proof that sonification can be a valuable tool when working with large data sets.

“Sound is the best pattern recognition tool we have,” said Walker. “Instead of visually scanning through a long list of numbers, looking for patterns or random occurrences, sometimes it’s easier to create an audio file and listen for them. Very interesting patterns can often be discovered by using sound.”

The Festival will feature a variety of arts experiences at both indoor and outdoor venues. The work can be independently produced by students or faculty affiliated with Georgia Tech or produced as part of an academic or research project. Projects can showcase the convergence of art and science/technology, or be solely artistic in nature.

Exhibit proposals can be submitted at http://arts.gatech.edu/content/techarts-exhibition and performance proposals at http://arts.gatech.edu/content/techarts-performance. The deadline for full consideration of submissions is February 15.

 “This is a Georgia Tech arts festival, so while we expect some projects will feature the very talented musicians or visual artists who are part of the Tech community, other projects will showcase the coming together of arts and technology,” explains George Thompson, Director of the Office of the Arts and Festival producer. The Festival is sponsored by the Office of the Provost, the Council of the Arts and the Office of the Arts.  The annual event is open to the campus and the Atlanta community.

The 2013 Festival coincides with the annual Margaret Guthman Musical Instrument Competition. Events already slated for the festival include Student View, a visual art exhibition by freshmen in the Writing Communication Program, a concert by the Georgia Tech School of Music, the musical production of DramaTech’s How to Succeed in Business Without Really Trying, selected works from the Art Crawl sponsored by the Clough Undergraduate Learning Center, and Automaton, a graduate project featuring dance and robots.  Please call 404-894-2787 for more information.