Telling Sounds

Telling Sounds - A digital research platform for the documentation and processing of audio-visual documents, including oral history materials, relevant for Austrian music history

Since the beginning of the 20th century, our (surrounding) world has been increasingly determined by media-mediated sounds and images. Their digital storage has considerably increased the diversity and scope of this repertoire. Music and sound shape such audiovisual media and thereby influence our social relationships: they have functions for the cultural memory of groups and individuals, they are associated with feelings and mark special rituals and events.

The embedding of music in radio programmes, reports, documentaries and films points to the historicity of musical phenomena that can be observed beyond the grand narrative of “sound heroes” operating in cultural centres. The documents (“clips”) we use are also related to music(s) in multiple and very different ways: In clips composed of different acoustic, visual or audiovisual elements, these may each refer to different times, institutions, events, places, people and repertoires, the coincidence of which in itself calls into question a linear understanding of history.

An important objective of the Telling Sounds project is the development of the so-called LAMA (“Linked Annotations for Media Analysis”), a digital research platform for collaborative work with audio(visual) clips. Digitally available clips from various archives and platforms (“Österreichische Mediathek”, “Phonogrammarchiv der Österreichischen Akademie der Wissenschaften” or “Archivio Luce”) are first recorded with their metadata; the media files themselves are not saved on the platform, but only referenced.

The clips are then enriched with additional, content-related metadata, which often only becomes accessible by watching or listening to the clip. This is, for example, which persons, places or sounds (various types of sounds, music or noises) occur in them and in what way (for example, whether someone is speaking or can be seen, or whether a sound is heard or mentioned). The entities (people/sounds/places etc.) are - in contrast to purely text-based metadata - linked to a corresponding data set in order to be able to uniquely identify them and in turn benefit from further semantic data linked to these entities (in the sense of Linked Open Data). In addition to “objective” findings (who or what occurs and how), derived meanings and connotations can also be recorded - in the form of so-called “topics”, which can be (historical) times, ideologies, topoi or places of memory, for example.

The content annotations result in a network of connections between clips and the entities and themes that appear in them. This network - and thus the interaction of sound and its contexts - is to be visualised. The visualisation is intended both to support the discovery of new connections that are not obvious at first glance and to make clear the variety of different elements - often from very different times - that make up a clip (which presents itself as an apparently organic whole).

In addition to visualisation, queries should be possible, especially which entities and meanings occur together (i.e. simultaneously in the same clip), such as: “Which clips contain images of Leonard Bernstein performing Beethoven’s music?”, “Which types of music are heard in the context of national socialism?”, “What image of Austrian identity is constructed in Jimmy Berg’s interviews through music?” or “In which clips does Austropop appear as a lieu de mémoire?“.