DML (Digital Music Laboratory) - Information management component, server and browser (Category: Research)

Name of Submitter(s): Samer Abdallah
Organisation: University College London

The DML is system for doing large scale computational musicology, bringing together large collections of music in recorded or symbolic form, and a collection of computational analysis tools, and allowing users to define sets of recordings or works of interest, decide what kind of analysis to do on them, and explore and build on the results. It is the result of joint project involving several partners including the British Library.
The information and computation management component of DML, which was my responsibility and is my entry to the Lab Awards, has the job or organising and keeping track of the recordings, their metadata, and the details of any computations done on them. In addition, it is responsible for triggering new computations when required. It is also a web server that allows the information to be explored in a web browser, using sound and graphics to make the content more accessible, for example, showing computer encoded scores in standard music notation and allowing this to be sonified, or sonifying the results of an automatic transcription. It also provides a browser-based laboratory environment which allows free experimentation with the collections and the computations available, to the extent of allowing users to think up their own new kinds of analysis, implement them in code, possibly making use of high-performance numerical computations in Matlab or R, and explore the results graphically. The results of such experiments can be recorded in sharable 'notebooks', supporting collaborative efforts.
The DML was built around several music collections that we had access to,
and since the British Library was one of the project partners, the BL digital
audio collection was included. The result is that the DML web interface
provides a distinctive way to explore the collection, and to move seamlessly from the recordings themselves to a wealth of related information obtained by computational analysis.
The DML information management component was built on a Cliopatria, a Semantic Web application framework which provides many of the basic services required and a system for extending this to support specific applications. Using this, many new capabilities were added to provide rich support for music-related data; these are described in more detail in the section on approaches and methods, which also includes URLs to pages on one of the current DML servers.
URL for Entry:



Job Title: Research Associate

Background of Submitter:

I am a postdoctorate researcher in music informatics and computational music analysis, and have been working in this field since starting my PhD on the use probabilistic models and information theoretic methods in computational modelling of the perception and cognition of music (completed in 2001). I have had an interest in music since taking up the saxophone at the age of 19, and later became interested in the questions of how we humans come to perceive, understand, make judgements about, and enjoy (or not, as the case may be) different kinds of music.
This led to an interest in brain-inspired computational models and hence a Masters course in neural networks and information processing, where I was introduced to probabilistic models, information theory, and machine learning.
During and following my PhD research, I studied the use of unsupervised
learning methods (those which learn about the structure of a problem domain without being given specific answers or background information about the application) for automatic transcription of polyphonic music
[1,2], onset detection [3,4], source separation [5,6], and segmentation [7].
It was the challenge of managing the data required to run machine learning experiments on collections of audio data that first motivated my interest in knowledge representation and laboratory workflow environments of the sort now embodied by DML. The knowledge representation aspects eventually led, in joint work with intern and then PhD student Yves Raimond, to the Music Ontology and the Event Ontology [8,9]. The workflow management aspects led to the prototype 'Knowledge Machine' [8,10], and eventually the more open and collaborative information management component of DML.
From about 2007 I began exploring the use of information theoretic methods for the characterisation of pattern and structure in music, and their
relationship with pyschological experiences like tension, complexity and interestingness [11-14]. More recently I have been investigating probabilistic models of symbolic music, using probabilistic programming languages to implement probabilistic grammars and their generalisations [15].
[1] Samer A. Abdallah and Mark D. Plumbley. Polyphonic transcription by
non-negative sparse coding of power spectra. In 5th Internation Symposium Music Information Retrieval (ISMIR 2004), pages 318–325, Barcelona, Spain, October 2004.
[2] Samer A. Abdallah and Mark D. Plumbley. Unsupervised analysis of
polyphonic music using sparse coding in a probabilistic framework. IEEE
Trans. on Neural Networks, 17(1):179–196, 2006.
[3] Samer A. Abdallah and Mark D. Plumbley. Probability as metadata: Event detection in music using ICA as a conditional density model. In 4th Intl. Symp. on Independent Component Analysis and Signal Separation (ICA2003), Nara, Japan, 2003.
[4] J. P. Bello, L. Daudet, S. Abdallah, C. Duxbury, M. Davies, and M.
Sandler. A tutorial on onset detection in music signals. IEEE Transactions in Speech and Audio Processing, 13(5):1035–1047, 2004.
[5] Samer A. Abdallah and Mark D. Plumbley. Application of geometric
dependency analysis to the separation of convolved mixtures. In 5th Intl.
Symp. on Independent Component Analysis and Signal Separation (ICA2004), Grenada, Spain, 2004.
[6] Maria. G. Jafari, Emmanuel Vincent, Samer A. Abdallah, Mark D. Plumbley, and Mike E. Davies. An adaptive stereo basis method for convolutive blind audio source separation. Neurocomputing, 71:2087–2097, 2008.
[7] Samer Abdallah, Christophe Rhodes, Mark Sandler, and Michael Casey. Using duration models to reduce fragmentation in audio segmentation. Machine Learning, 65:485–515, 2006.
[8] Samer Abdallah, Yves Raimond, and Mark Sandler. An ontology-based approach to information management for music analysis systems. In Proc. 120th AES Convention, Paris, France, 2006.
[9] Yves Raimond, Samer Abdallah, Mark Sandler, and Frederick Giasson. The music ontology. In International Conference on Music Information Retrieval, ISMIR 2007, 2007.
[10] M. Sandler, Y. Raimond, and S. Abdallah. A method of analyzing audio, music or video data, 2010. US Patent App. 11/917,601.
[11] Samer A. Abdallah and Mark D. Plumbley. Information dynamics: Patterns of expectation and surprise in the perception of music. Connection Science, 21(2):89–117, 2009.
[12] Samer A. Abdallah and Mark D. Plumbley. A measure of statistical
complexity based on predictive information with application to finite spin
systems. Physics Letters A, 376(4):275 – 281, 2012.
[13] Kat Agres, Samer Abdallah, and Marcus Pearce. An information-theoretic account of musical expectation and memory. In Proceedings of the Annual Meeting of the Cognitive Society, COGSCI 2013, Berlin, Germany, 2013.
[14] Henrik Ekeus, Samer A Abdallah, Mark D Plumbley, and Peter W McOwan. The melody triangle: Exploring pattern and predictability in music. In Proc. Musical Metacreation (MUME), 1st International Workshop on, Palo Alto, USA, pages 35–42, 2012.
[15] Samer A. Abdallah and Nicolas E. Gold. Comparing models of symbolic music using probabilistic grammars and probabilistic programming. In Proceedings of the 2014 In- ternational Computer Music Conference (ICMC/SMC 2014), pages 1524–1531, Athens, Greece, September 2014.

Problem / Challenge Space:

The broader question that the DML as a whole was designed to address is how to provide an effective environment for large-scale computational musicology. This entails bringing together a wide variety of computational music analysis tools, contributed by resarchers all over the world, with a large collection of recorded or symbolic music, and then providing means to explore the results of that analysis, using modern statistical methods and visualisation tools, in order to answer musicological questions on a scale much larger than that possible by manual analysis. An important aspect of this was a desire to accommodate collaborative working, and to be able to add collections and analysis methods easily when the need arises.
The information management component of the DML was designed to address the question of how to keep track and provide access to the many kinds of information involved in designing and running a musicological experiment, including:
- the selection of the recordings to be analysed (perhaps on the basis of composer, form, composition or recording dates, genre etc.)
- the selection of a set of analysis methods to be applied to each recording;
- the mechanics of running the different computations and providing a way to check on the progress of multiple long-running computations;
- the managing of results produced by analysis, including keeping track of which inputs and which parameter settings were used to produce which results;
- the design or selection of aggregate analysis methods to produce statistical summaries of a set of results, or apply other types of collection-level level analysis such as machine learning methods.
Given the many types of information being managed, one of the challenges was how to organise it in a way that makes it easy to explore and browse, but also flexible enough both to accommodate the complexities of music related information structures and to support arbitrarily complex queries that might be needed, for example, to define a certain set of recordings for analysis in an experiment, or to define a certain subset of computed results.
Finally, having conducted music analysis experiments for many years, I was very keen to provide an environment which allows for free experimentation, rather than just providing a fixed set of tools and options to choose from. In my opinion, this requires that users be able to express new ideas directly in code, in a high-level programming language, and see the results immediately visualised in a way of their choosing---this, to me, is the fullest expression of the idea of a 'digital laboratory'.

Approach / Methodology:

1. Overview
The approach to information management adopeted in the DML is to combine a relational data model with a system for managing computations based on memoisation. This approach was based on my previous work on the Music Ontology [1] and the 'Knowledge Machine' [2]. Unlike that previous work, however, the DML is built on a Semantic Web server application framework and supports web-based interaction, collaborative working, and the essential building blocks for distributed data and computation.
2. The Semantic Web
The Semantic Web [1] is a response to the observation that, while there is
a huge amount of information available on the world-wide web, most of it is in the form of text and images that are readily understood only by human readers, and generaly quite difficult for computers to process. The proposed solution is supplement this data with relational data in the form of 'triples', which are simple logical statements of the form
such as
(<J. S. Bach>,<composed>,<The Goldberg Variations>).
The entities, or 'resources', referenced in these particular triples are not literal text, but unique global identifiers, called 'uniform resource identifiers' or URIs, denoting the various entities such as people, events, places, musical works, etc. For example, the identifier <J. S. Bach> will itself be
described by other triples, for example
(<J. S. Bach>,<is a>,<Person>),
(<J. S. Bach>,<surname>,"Bach"),
(<J. S. Bach>,<first name>,"Johann"),
where "Bach" and "Johann" are literal character strings, and <Person> is a class of entity. Even the predicates themselves can be described by other triples, such as
(<composed>,<subproperty of>,<created>),
which indicate that composition is a form of creation and that the things
which have surnames are people. In this way, complex self-describing knowledge representations can be built up. The Music Ontology [2] is such a system that I was involved in creating. The general name for this triple-based format is RDF, and a collection of triples is called an RDF database or triple store.
3. Implementation framework
The programming language adopted for this project is Prolog, a language that originated in 1970s with the aim of allowing programmers to describe and solve problems at a high level of abstraction ('Prolog' is a contraction of 'PROgramming in LOGic'. Though an old language, still, it represents a way of thinking and working quite different from most commonly used procedural and functional languages, and has successfully taught as a first
programming language [3]. (This is relevant here because the laboratory environment described in section 7 is a Prolog programming environment.) Prolog's data representation and programming model makes it a very good match for managing relational data, avoiding the so-called 'object-relational impedance mismatch' which has been discussed in the literature [4,5]. Furthermore, the implementation we used, SWI Prolog [6], provides a substantial set of libraries for managing an RDF database [7], communicating with external databases, and building websites [8]. The
Cliopatria framework [5], written in Prolog by the main developer of SWI
Prolog, brings together many of the components required to build a semantic web application into a modular and extensible system. The DML information management component (DML-Cliopatria) is written as a Cliopatria plugin that provides many facilities for managing, exploring, and presenting music-related data. Some of these extensions are described in section 6 below.
4. Musicbrainz-Music ontology integration
Musicbrainz ( is a large online database of information about commercially available recordings. It also uses a relational data model, with unique identifiers for all the main entities, such as artists, recordings, album and works, but one based on SQL tables using their own schema, rather than the Music Ontology. It can be queried using the Musicbrainz XML service. Given a Musicbrainz identifier of a recording, the DML-Cliopatria can dynamically retrieve information surrounding that recording and add it to the RDF database. This includes detailed information about the factors which went into producing a recording, such as dates, the roles of the people involved, who played what instrument, which musical works were recorded, and other events that performers and composers may have been involved in. DML-Cliopatria provides a way to explore the information in Musicbrainz, and, if the audio itself is available locally, design musicological experiments on the basis of this information.
Cliopatria provides a mechanism where the page for a resource can include a graphical diagram of entities linked to the current resource via various properties, and in DML-Cliopatria, this is customised to provide a convenient rendering of the events and entities surrounding a given recording, person, or work.
Obtaining Musicbrainz identifiers for all recordings in the DML database
is currently not feasible, due to the unreliable nature of matching names and title to people and works on the basis of text matching alone, but DML-Cliopatria does support importing a database of recording from the open-source tool Beets ( Beets fulfils a role somewhat like iTunes in managing a user's music collection, but as part of the import process, it can interactively match up albums and recordings to their Musicbrainz identifiers. Once these identifiers are imported into DML-Cliopatria, they can be used to pull in more information from Musicbrainz.
The DML server currently includes the beets database for my own music collection (but without the audio). For example,
links to the page for a recording of Billy Eckstein's 'I want to talk about you' by John Coltrane. Scrolling down to the 'dml:file_recording' property we see a Musicbrainz URI for a recording. Clicking on this leads to a page with information from Musicbrainz and a context graph at the bottom. The nodes in the graph are clickable, and can yield more information about the entities involved. For example, the node on the far right with many sub-events represents the physical event of recording the track. Clicking on it yields more information and a new context graph. Clicking on the node linked by propory 'mo:performance_of' brings up the page for musical
work and a complex graph showing all known recordings of it.
[Unfortunately, dynamic updates are currently restricted to logged in users as the feature could be abused by a malicious user.]
5. Computation management
A major component of DML-Cliopatria is managing the computations that go into a musicological experiment. The idea is that the relational data model is ideal for recording all the information about a computation, including the input audio files and parameters, the output(s) produced, and 'metadata' about the computation such as when it was done and how long it took. This provides all the necessary information for further analysis of the results. It also supports the re-use of previously done computations to save time later, for example, a transcription analysis might be useful for doing further melodic or harmonic analysis, but need only be computed once. This process of remembering the result of a computation in order to avoid repeating it later is called 'memoisation'.
In DML-Cliopatria, computations done using the VAMP system of audio analysis plugins [10] is managed using the RDF database, as VAMP plugins are already described by RDF documents. For example, a page
describing the Silvet transcription plugin is here:
A VAMP 'transform' is a binding of a plugin to some specific parameter values When a new VAMP computation is requested using a given transform and recording, the system creates all the necessary files, calls Sonic Annotator, manages the resulting files, and adds an entry to the RDF database, all automtically. For example, this is a page for a particular transcription transform which includes at bottom a table of all the computations that were done using it:
The resources in this table can be followed to find out more about the
computation and its result.
Other computations that rely on the results of VAMP computations are memoised in the Prolog database, as this is a more efficient way of
memoising arbitrary computations (which may be written in Prolog, Matlab,
R or Python) with many parameters. A table of all such memoised functions is here:
The repository of results computed from VAMP plugin outputs is csv_op_memo/3. Clicking on this yields some information about it and clicking on the table icon shows its contents.
These memoised results can be used in the SWISH laboratory environment (see Sect 7) and are used to satisfy requests from the DML Visualisation web application.

6. Extensions to Cliopatria
Cliopatria usually displays resources using the resource's <label> property
if this is available, or the literal URI if not. However, it provides an extension mechanism to customise the web-page displayed form of the URI. DML uses this to contruct meaningful labels on the basis of resource
properties, e.g. for using title and composer for Humdrum symbolic scores
or constructing descriptions of Music Ontology events, and to decorate resources with e.g. an audio player, or links to relevant pages, such as analysis results in full, a graphical rendering of a musical score, or a link to a web page on the BL website.
Similarly, Cliopatria also provides an extension mechanism for adding content to the web page for each resource, which normally includes only a list of triple in which the resource appears. This is used to add various elements depending on the type of resource:
- Audio recordings with accessible audio get an audio player and an audio spectrogram image computed on demand using Matlab.
- Recordings with a Musicbrainz identifier but no local audio are looked up
on Spotify and, if found, a Spotify audio player is included, providing
access to the audio even if a local audio file is not available.
- Symbolic scores get a pitch class histogram plot generated using R and a
score sonification interface that allows the music to be synthesised with
control over tempo, transposition and instrument.
- Analysis functions get a link to a table of previous computations and documentation for the function.
- Computation results files get a link to page showing the results of that
computation, eg
- Automatic transcription results get a piano-roll rendering of the
transcription and an sonification interface.
A number of external, open source packages are used to provide various services:
- Audio processing programs Sox, oggenc, faac, and lame are used for audio
format conversions.
- Humdrum tools is used for converting Humdrum scores to other formats
such as MusicXML, Lilypoond and MIDI.
- The Lilypond music typesetting system is used to render scores in
standard music notation, eg
- Sonic Annotator is used for run VAMP plugins to do audio analysis.
Interfaces are provided to various languages for numerical computations and plotting:
- Matlab, via my Prolog-Matlab interface library plml.
- R, via Nicos Angelopoulos's Prolog-R interface package Real.
- Python, via standard shell commands with communication by JSON
objects on standard input and output streams.
7. DML-SWISH - a laboratory environment
SWISH ( ) is a web application that provides a
Prolog programming environment in a web browser, using modern Javascript application frameworks and a network protocol for making remote Prolog language calls to a 'Pengine' (Prolog engine) server [9]. It provides an environment for editing, runnning and debugging Prolog programs, with the option to do so collaboratively with people in other locations. SWISH runs Prolog code in a 'sandbox' environment: the code is dynamically analysed to make sure it is safe to run without compromising system security. SWISH is available as an add on to Cliopatria and immediately makes it possible to query the RDF database and do computations on it using the full power of Prolog.
DML-Cliopatria includes a SWISH server and exploits the extensible 'term
rendering' mechanism to help visualise music related computations. Firstly,
it inherits from Cliopatria the resource display customisation, so if, for example, a query produces the URI for an audio recording, the result includes an audio player. In addition, certain data structures are recognised as describing a graphical plot in Matlab or R, in which case, the graphic is created using Matlab or R and included on the web page. Alternatively, recent versions of SWISH support term renderings using C3, a Javascript library for high-quality interactive plots.
Using SWISH, a researcher can explore the DML database, design and code and debug a new idea for an experiment, possibly using results already in the database, obtain graphical results, and share the code with collaborators, all in a web application. This is a realisation of the idea of a 'digital laboratory' which aims to impose as few restrictions on the imagination of the researcher as possible.
8. Acknowledgements
I would like to acknowledge the contribution of the partners on the DML project, especially Daniel Wolff, Alex Kachkaev and Emmanouil Benetos for their hands-on work in building the DML system as a whole. I should emphasise that in this entry for the BL Lab Awards, I am submitting only the information and computation management server that was my contribution to the system.
8. References
[1] Tim Berners-Lee, James Hendler, Ora Lassila, et al. The semantic web.
Scientific American, 284(5):28–37, 2001.
[2] Yves Raimond, Samer Abdallah, Mark Sandler, and Frederick Giasson. The music ontology. In International Conference on Music Information Retrieval, ISMIR 2007, 2007.
[3] Arthur Fleck. Prolog as the first programming language. ACM SIGCSE
Bulletin, 39(4):61– 64, 2007.
[4] Christopher Ireland, David Bowers, Michael Newton, and Kevin Waugh. A classification of object-relational impedance mismatch. In Advances in
Databases, Knowledge, and Data Applications, 2009. DBKDA’09. First
International Conference on, pages 36–43. IEEE, 2009.
[5] Jan Wielemaker, Wouter Beek, Michiel Hildebrand, and Jacco van Ossenbruggen. Cliopatria: A SWI Prolog infrastructure for the semantic web. Semantic Web Journal.
[6] Jan Wielemaker, Tom Schrijvers, Markus Triska, and Torbjörn Lager.
SWI-Prolog. Theory and Practice of Logic Programming, 12(1-2):67–96, 2012.
[7] Jan Wielemaker, Guus Schreiber, and Bob Wielinga. Prolog-based
infrastructure for rdf: scalability and performance. In The Semantic Web-ISWC 2003, pages 644–658. Springer, 2003.
[8] Jan Wielemaker, Zhisheng Huang, and Lourens Van Der Meij. SWI-Prolog and the web. Theory and practice of logic programming, 8(03):363–392, 2008.
[9] Torbjörn Lager and Jan Wielemaker. Pengines: Web logic programming made easy. TPLP, 14(4-5):539–552, 2014.
[10] Chris Cannam, Christian Landone, Mark B Sandler, and Juan Pablo Bello. The sonic visualiser: A visualisation platform for semantic descriptors from musical signals. In ISMIR, pages 324–327, 2006.

Extent of showcasing BL Digital Content:

The DML currently includes several collections of recorded and symbolic music including:
- The British Library's collection of nearly 50,000 digitised audio recordings including classical music, British folk music, recordings made by ethnomusicologists of traditional musics from all over the world, oral history testimonies, and other genres.
- The collection created by the AHRC Research Centre for the History and Analysis of Recorded Music (CHARM - ), which contains about 4900 recordings of classical music digitised from 78rpm discs recorded between 1902 and 1962.
- The Mazurka dataset, curated by CHARM, containing 2,700 recordings of Chopin's mazurkas.
- The I Like Music (ILM) collection of 217,000 commercial records.
- A collection of 12,000 scores in Humdrum/Kern format taken from the KernScores website ( ), which holds 109,000 scores.
Sine the recordings in the ILM collection are not publicly available for
download, the BL collection (about 35,000 are publicly available for streaming from the British Library Sounds website ) is by far the largest and most varied of those available to fully explore on the DML website.
Using the browser interface, it is possible to explore the BL collection in ways that are not possible on the website: the recordings are cross linked in multiple ways on the basis of any of their metadata
properties, such as languages spoken or place of recording. Using these links, any property of interest of the current recording can be used to find other recordings with the same property value, or other possible values of that property. In addition, all of the analysis results associated with a recording can be explored, some of them with graphical or sonic renderings, for example the automatic transcriptions can be rendered as a piano roll image or via MIDI synthesis.
The browser-based lab environment provides another, user programmable way to explore the collections, allowing the user to define arbitrarily complex searches on the basis of any information in the database, and bring up individual audio players or a playlist player directly within the laboratory.
The variety of recordings in the BL collection, including as it does
recordings of traditional musics from all over the world and folk music
recorded in homes and pubs across the UK makes it a particularly absorbing collection to explore using the DML browser interface.

Impact of Project:

An early version of the DML informationation management component was demonstrated at UCL Digifest in November 2014 ( ).
The DML as a whole was presented at a workshop in the British Library in March 2015 ( ), where 50 attendees consisting of musicologists, music technologists and other researchers were given an opportunity to use the DML Visualisation web application, which relies on the information management component as a server.
It was also presented at a seminar in the series of seminars associated with the FAST_IMPACt (Fusing Audio and Semantic Technologies for Intelligent Music Production and Consumption) project ( ).
Work using the DML was presented [1] at the International Workshop on Folk Music Analysis in June 2015 ( ).
Also, after the end of the DML project, it was used to support the ASyMMuS project on music similarity, which was concluded with a workshop in July 2015. ( ).
Other events related to the DML project are described in the DML blog
( ).
[1] S. Abdallah, A. Alencar-Brayner, E. Benetos, S. Cottrell, J. Dykes, N.
Gold, A. Kachkaev, M. Mahey, D. Tidhar, T. Weyde A. Tovell, and D. Wolff.
Automatic transcription and pitch analysis of the British Library World &
Traditional Music Collection. In Proc. 5th International Workshop on Folk
Music Analysis, pages 10–12, June 2015.

Issues / Challenges faced during project(s):

The initial phase of adding the various music collections to the system
involved writing code to interpret each collection's metadata format.
Unavoidably in such large collections, there were inconsistencies and errors which only emerged during the import, and required a mixture of corrections to the metadata or special cases in the importing code to deal with them correctly. This served to highlight the desirability of a standardised vocabulary for music metadata, like the Music Ontology, to ensure that, for example, dates and date ranges are encoded consistently, and that entities such as composers, performers and works have unique unambiguous identifiers.
During the run up to the final DML workshop, we were plagued a by bug that was causing the information server to crash under load. SWI Prolog, the underlying implementation language, is very stable and almost impossible to crash in most circumstances, but the nature and pattern of
the crashes indicated a low-level problem. It only emerged several weeks later that the problem was caused by an unanticipated multi-threading interaction between the system garbage collector and my (C++ language) Prolog-Matlab communication library. As this crash had us frantically redirecting and restarting services a few times during the workshop, it was particularly satisfying to eventually pin down and correct the problem!