Iain Emsley

Submitted Entry for 2015 Competition


Visualisation is an accepted way of presenting data and structures. Sonification is complementary way of presenting the data.

Structures created from queries performed on manuscripts and letter collections can be sonified to present the data and add context to it. The audio rendering of tones or notes depends on the query and the type of collection being queried. The resulting file would be transformed into different sources for the user to reuse or share. These would be stored with the data and links to the metadata. Cues in pitch and timing can help users understand changes between versions of texts and types of events within the metadata, as would choice of music.

This builds upon explorations made into sonifying structures in Bodleian’s First Folio Hamlet text. The initial application derived audio and video representations of the stage directions and speakers. Audio can be used as an analytical tool to help users with exploratory data searching or comparison of data sequences. The sequences would be linear with a pre-defined note set to each type of event that the system would expect to find.

Building on the existing structural work with the Hamlet text, the social context of the events can be sonified and their links made more obvious to the listener. This data and recording would be taken from a known provenance. Linking the data to the audio using the stream name and time transcludes the audio into the metadata.

Relationships between the metadata can be linked acoustically using timing and sounds to highlight the relationships. Mixing streams for different collections allows the listener to interact with the data psycho-acoustically.

Time can be represented by the metre and rhythm to guide the reader through the collections. Alternate sounds, samples and music can be used to denote the type of event, whether it is a document, letter or photograph, as well as editorial changes. Pitch can be used to denote whether the speaker or author is male or female. Collection metadata could use these approaches to provide understanding of the relationships, and to link the data to context such as the period or event type.

The BL labs residency will be used to build a prototype using the identified metadata and collections for an analytical tool to represent the metadata and network links as part of a hyperstructure.

Assessment Criteria

The research question / problem you are trying to answer

Please focus on the clarity and quality of the research question / problem posed:

I want to focus on the way that the sonification can be used to create novel interfaces to understand the underlying structures in variations or collections texts. Using psychoacoustics, the user can be alerted to subtle changes between versions of the in depth textual structures. Using a mix of streams, an individual can compare text variants.

We can help the user to understand the make up of a collection of texts using cues to understand the types of events, such as the gender or type of item, and to provide the context of the data, such as time - whether this is chronological time or cultural period. Sonification may provide context to an event using rhythm, pitch, metre or culturally relevant musical form (be this historical or use of different culture’s musical and sound experiences, rather than just using Western classical or DJ mixes, where relevant) to enrich the user’s experience of the data without them necessarily having to use a key or legend. Using multiple streams, we can help the user listen to the changes in two or more streams of text.

Using the novelty of sound rather than just visuals allows the collection to be viewed as part of a continuum and for hints to be given to the listener about how it might change or intersect with other collections.

This question intersects with concerns about historicism and accuracy when reconstructing data and events. Clearly this question cannot be resolved in this project but if there are available sound samples, we might put these together for an extract to demonstrate the effects.

Please explain the ways your idea will showcase British Library digital collections

Please ensure you include details of British Library digital collections you are showcasing (you may use several collections if you wish), a sample can be found at http://labs.bl.uk/Digital+Collections

This idea will take the metadata of the scientific letter collections from the nineteenth century, such as Charles Babbage and Michael Faraday, or the letters and manuscript metadata from the Gothic writers, such as Anne Radcliffe, M G Lewis and Horace Walpole.

The metadata will be sonified to show the intersections of the networks. Using a standard time scheme across the collections, each collection will have a track of its own. Using chords and tones per the writer and correspondent, these will play in unison. If there is an intersection, the notes will converge between the tracks.

Using different underlying sounds, context can be added for the user so that they might understand the underlying science or art. We can use sound to add subtleties of context for the listener, either through the gender or use of a sample or style to indicate a period or place.
Using psychoacoustics, the audio samples can help the user understand the make up a collection of texts over a period of time as well as using relevant records on the BL Sound Archive. Using pitch, we can help them understand the gender of the speaker or the type of document (such as letter or photograph or note). Changing the rhythm and metre allows them to experience the times of the texts differently. Should these collections not be available, it would be possible to use the Rene Navarre metadata to develop a similar sonification.

Please detail the approach(es) / method(s) you are going to use to implement your idea, detailing clearly the research methods / techniques / processes involved

Indicate and describe any research methods / processes / techniques and approaches you are going to use, e.g. text mining, visualisations, statistical analysis etc.

The data, either XML or CSV, would be parsed using a set of rules and the event data will be transformed into integers for conversion into sound and digital file. A set of rules will be used to link the event to a integer to be converted into a frequency. An initial approach may be to hardcode some of the transforms, such as the gender of a correspondent or speaker in a play, and to automate others, such as dates and links to periods, using text mining. Developing this protocol may need some revisions once the prototype is developed.

Using LAME and a domain specific language such as Chuck, the wave can be converted into a file and converted into other forms. The Haskell language might be used to convert the data into sound as a second line of enquiry. The transformation pipeline will either be driven from a static file of integers or a streaming interface.

It appears possible to link the data and audio together using TEI and SMIL so that the dat a can be re-used and possibly embedded.

Please provide evidence of how you / your team have the skills, knowledge and expertise to successfully carry out the project by working with the Labs team

E.g. work you may have done, publications, a list with dates and links (if you have them).

I am a Research Associate at the Oxford e-Research Centre, writing my own software or using Open Source software. I would also like to use some of this work towards a project for the Masters in Software Engineering. Apart from some technical advice on what is possible at the British Library and some of the linking of data together, I believe that the main help required would be curation.

An initial application has been developed using the First Folio XML text from the Bodleian Library with rudimentary audio and visual output (http://austgate.co.uk/development/hamletvis.mp4).

An audio only version of the same data was created as well using different instruments for events. (http://austgate.co.uk/development/hamlet.wav).

An early version of this work was presented in the Bodleian Ideas Hack (http://www.bodleian.ox.ac.uk/eebotcp/early-english-books-hackfest/), entitled “If Music be the food of Loue” (http://www.bodleian.ox.ac.uk/eebotcp/files/2015/02/If-Music-be-the-food-of-Loue-Sonifying-Drama.pdf).

Please provide evidence of how you think your idea is achievable on a technical, curatorial and legal basis

Indicate the technical, curatorial and legal aspects of the idea (you may want to check with Labs team before submitting your idea first).

Some assistance may be required in creating and optimising the queries to retrieve the data.

Using scripting languages such as Python or PHP, the data can be transformed into the relevant source for the audio services.

The audio services might be constructed using domain specific languages such as Chuck, or a more general language such as Haskell with its many music libraries. This is the most technically challenging part of the project and in part depends on the architecture.

The conversion of the audio files can be completed by the open source LAME or Vorbis programmes.

An issue to discuss is whether the audio file could be released using Torrents if the size is large.

Curatorial expertise may be required to help identify and collate the data sources and to identify the correct outputs and formats.

The letter collection metadata would need to be collated and any relevant objects identified. Using the Charles Babbage letters, the texts on Gutenberg could be linked to the letters by date.

Access to the sound archive would be useful as would advice on relevant tracks that might be sampled.

Using open data sets and tones / sounds would avoid the need to gain clearance to use samples. The resulting output would need to be correctly licensed so that it can be output and reused correctly. I may need help to license samples should this be thought a relevant in any early discussions.

Please provide a brief plan of how you will implement your project idea by working with the Labs team

You will be given the opportunity to work on your winning project idea between June 2015 - October 2015.

June 2015
BL: Help with the identification and curation of the data objects alongside Babbage letters.
BL: Provide interface details and arrange access with IE.
BL & IE: Discuss the project, focusing on the data and any restrictions, technical or curation.
BL & IE: Agree on the formats for representing the query results and audio files.
IE: Set up the query interface.

July 2015 - September 2015
IE: Work on the transformation pipelines from the raw data into the preparations.
IE: Work on the protocol and interfaces for the events.
IE: Work on the sonification using gender, document type, multiple streams and times.
IE: Linking XML to audio.
IE: Integration of the pipelines and transforming into HTML.

Ideally during this period, there will be the opportunity to demonstrate the current work and discuss any issues with curators and technical staff.

September 2015
IE: Be in a position to demonstrate the prototype.
IE and appropriate BL staff - write paper on the aims and methodology.

October 2015

Although October is for tidying up, I do anticipate using it for any tidying up or slippage should this happen.

IE: Packaging the software.
IE: Write up the documentation.
IE: Look at using the software for other collections and projects.