Project lead: Christoph Steindl
Institution: Austrian National Library (ÖNB)
Project duration: 01.06.2023 – 30.11.2023
GLAM institutions like the Austrian National Library (ÖNB) are responsible for archiving a vast amount of objects and data. Much information about these objects is available in digital form, and in many cases the object itself is already digitized. To manage, explore and analyze these collections of data, machine learning (ML) approaches have been developed to extract new information in oder to create new views on certain collections.
Many GLAM institutions use the International Image Interoperability Framework (IIIF) in order to give access to metadata and digital material. IIIF allows for an easy integration of digital resources in websites and sharing across different institutions in workflows. Thus IIIF-ready data is ideal for ML in order to retrieve and process it. The following graphic illustrates the stages of a conventional ML workflow (see the original machine learning workflow here) and the potential areas of application of the IIIF (ie. different IIIF APIs) within.
Many ML pipelines that are publicly available (e.g. on GitHub) use Jupyter notebooks tot rain, test and apply their ML models. Jupyter notebooks are also very common in DH, in particular to document the generation, manipulation or analysis of datasets. In addition, Jupyter notebooks are used to fulfill learning aspects. As an example, the NewsEye project with a focus on European newspapers published a notebook collection with an in-depth- analysis of their corpus (e.g. text classification or text similarity) dedicated for the use at university courses.
The main goal of this project is to combine the different technologies – IIIF, ML, Jupyter notebooks – in order to support researchers in generating new knowledge from the digitized cultural assets. Bringing together these technologies has many benefits: (1) it provides an easy, standardized and reusable way to integrate IIIF materials into ML applications in general and (2) it publishes these ML pipelines as Jupyter notebooks. They will be (3) well documented and can therefore (4) be used as a boilerplate for new projects and (5) can easily be applied by other institutions that support IIIF for their data. In addition to raw source code the project also aims to use interactive widget components in the notebooks in order to make the software suite easy to use for users with less previous knowledge in computer sciences.
Machine learning applications are and will be an essential part of data analysis in the scientific context. Regardless of whether small data collections or big data are processed, it is necessary to train users to use machine learning modules. This way, it is possible to understand complex and multidimensional problems and thus generate new perspectives on collections. The project ecourages applying these innovative methods to already digitized data.
(intermediary) Outcomes
The aim of the project is to create several small applications to illustrate machine learning applications in the GLAM area, which use IIIF resources as a data set and are easy to use for the potential user or can be customised to their use case, thus lowering the threshold for machine learning in the digital humanities area.
For access to the IIIF API, i.e. extraction, download and handling of resources, a small library in Python will be created to simplify the handling of manifests and collections. In addition, the example projects are made available as Jupyter notebooks with interactive components or as Gradio applications (Python library for interactive demonstration of machine learning applications). This guarantees easy reusability of the developed components and simple utilisation of the developed use cases, even for non-technical users.
As specific use cases, an existing code for image colouring by means of deep learning was adapted and made available in the form of a Jupyter notebook. The application uses black and white postcards from the ÖNB Akon Collection, which are available via IIIF, and colours them using an existing and freely available model and makes the result available to the user as a download. Another notebook gives the user the opportunity to train a colourisation model by using a IIIF collection of images and then applying it to an image.
Furthermore, an application will be developed that enables the user to use existing models for the segmentation and classification of images by specifying an IIIF resource and to export these results as annotations to the IIIF manifest. Possible resources are, on the one hand, the manifests and collections available at the ÖNB and, on the other hand, resources from other libraries.
