In this experiment, Bonnie probes the intersection of music and language, exploring how percussion instruments can imitate and stand in for the human voice and function as an extension of the body. She utilizes and highlights the real-time misunderstandings generated by voice recognition AI technology. Here, she asks the smart speaker to cycle through randomized applications (news, weather, inspirational quotes, Jurassic Bark, reader, Shopping List, My Questions, Metronome Lite, etc.) while she cycles through randomized improvisational musical directives (imitate, accompany, cover, long-tones, various dynamic shadings, pointillistic activity, etc.)
In other experiments not documented above, she utilizes and highlights the real-time misunderstandings generated by voice recognition AI technology. Through both text to speech and speech to text applications, she reveals the smart speaker’s unnerving proclivity to advertise. When the Amazon Echo speaker was asked to record and read back excerpts from Kurt Schwitters’ mid-1920’s landmark sound poem Ursonate, for example, rather than the poet’s nonsense syllables we heard the AI’s closest approximation, yielding words like “Bezos” and “Whole Foods.” Speaking along with the smart speaker (and through her instruments), Bonnie emphasizes the uncanny valley that separates humans and machines, working to hack and manipulate the speaker as a generator of abstract sound poetry and music.
As previously reported, we brainstormed some great ideas and some possible ideas for what to do with the voice assistants and data archives.
My approach is to make a series of relatively short movements or “miniatures” using said data sets. The data sets were used as the generative material to create the fixed media track which you can listen to below. All sounds you hear in the track come directly from the voice clips and are then further processed for desired sounds and effects. There are no samples or sounds that came from other sources. The intention is to continue generating fixed media under that parameter.
To accompany the fixed media track will be a graphic notation score for performer(s). If you’ve never seen one, here are some examples: https://www.classicfm.com/discover-music/latest/graphic-scores-art-music-pictures/. While I am still in the process of sketching the score, it will be loosely based off a spectrum analysis of the frequencies found in the various voice clips you hear in the fixed media. The idea for instrumentation for this miniature is leaning towards resonant sounds of some type. I hope to post a page from the score soon so please check back.
The Turk Interpreter is a system that plays back voice commands issued by Mechanical Turk workers. This blog post will briefly overview its features and present a short demo.
Turk Worker GUI
Output -> Specifies the output of the turk interpreter. There are two options, human and Computer. Human will play back recordings of voice commands issued by turk workers, computer will read the commands with Mac’s built in text to speech software.
Person -> Specifies the specific turk worker for playback.
Search -> Allows user to search turk worker data for specific word/phrase. Playback is then refined to results of this search.
Result -> Results of search. Text can be edited to alter READ functionality.
Play -> Plays back random voice command based on search results with varied output depending on output settings.
Read -> Reads whatever is in results window with computer
Stop -> Kills all audio
Turk Worker Demo
The person’s name has been blacked out to preserve their autonomy
Working in the Developer Console for Alexa, Audrey worked on ‘hacking’ (or modifying) Alexa’s voice. Through simple commands, it is possible to make the voice whisper, slow down, speed up, or change pitch. It is also possible to change the voice itself, with the opportunity to hear from Ivy, Joanna, Joey, Justin, Kendra, Kimberly, Matthew, Salli. The commands are explained here.
Audrey used sketching as a way to visually map the various ideas we have had and to open new proposals for what a future performance could hold. She also started to collect ideas for points of departure for our design and artistic process. These points include:
A lot of AI responses are hard coded
People ask many things to voices assistants that are not planned for (not hard coded)
Voice assistants are still not very good at understanding human language, generating many bloopers
Voice assistants lack context and memory
Data is part of an ecosystem of algorithms, data collection, voice detection, voice to text technology, AWS….
Corporations benefit from clean and ‘true’ data
Voice assistants can’t pretend to be human — instead, how can they honor the reality that they are software.
It has been a little while since we’ve posted and we have a lot to share! Recently, a few students with interest in this project joined the research group on voice assistants and experimental performance. The following students have backgrounds in music composition, experimental percussion and digital arts and experimental media:
While Bonnie and Afroditi were attending and presenting at the Transmediale Festival (a festival and year-round project that draws out new connections between art, culture, and technology), Audrey and the team got together for a brainstorming session in an attempt to identify the trajectory and next steps for this project. We felt like we came up with some great initial ideas and concepts, here are a few: voice assistants and language manipulation, sound manipulation using digital audio workstations, voice assistants within the context of defined spaces.
The ideas are flowing and we’ve got momentum! Can’t wait to share more soon!
Experiments with VoicePad application, percussion, and voice
Part I: Bonnie + Whale Fall
The smart speaker reads original text, Whale Fall, by writer David Crouse. Bonnie experiments with speaking in unison, reading against the speaker, and creating a percussive soundscape based on long tones.
Part II: Bonnie + Kurt Schwitters’ UrSonate
The smart speaker reads a passage from Schwitters’ landmark sound poem, UrSonate (1922-1932) while Bonnie reads the poem.
We started this project in October 2019. We meet weekly and combine discussion with making and demos. We will share on this blog elements of our process and work-in-progress images, videos and thoughts.