“Marking up” text, specifically the process of assigning metadata to text spans in order to encode meaning, has been central to digital humanities practice for decades. Thousands of person-hours have been devoted to this task using standards such as TEI, either performing the process manually or sometimes partially assisted using conventional NLP software.
Huge advances in neural networks and deep learning have profoundly impacted NLP since the early 2010s. This has greatly expanded the range of language tasks which can be performed automatically or semi-automatically. In particular, deep learning models known as “Transformers”, using a multi-head “attention” mechanism [1] have improved the performance of tasks such as Named Entity Recognition (NER) and Relation Extraction (RE) to the point where markup tasks previously performed by human researchers can potentially be automated or partially automated. This offers huge cost-savings for DH projects. But more importantly it increases the size of the text collections which can be processed and therefore the scale and scope of feasible research questions.
The DHI in Sheffield has been involved in many large scale projects over the past decades which involved extensive human markup work. We are now beginning to work on projects which perform markup using deep learning models. We have insights to share about which tasks are better suited to, for example, encoder-only models such as BERT, and which are better suited to the generative Large Language Models (“LLMs”) which have become synonymous in the popular consciousness with “AI”. We have developed strategies for producing the most accurate results from locally run smaller models rather than remotely hosted commercial services such as ChatGPT or Gemini. For example, the absolute minimum number of ground truth human examples required to fine tune a BERT model for good results in a particular domain, or using spans generated from a partially trained encoder to in turn formulate more effective LLM prompts.
This paper aims to demystify what is possible with deep learning in our domain, where material is highly specialised, markup requirements atypical and appropriate training data often scant or nonexistent. With respect to completed projects, the DHI has amassed a huge tranche of training and test data over the years against which we can evaluate deep learning approaches. For new projects, we are quickly learning how conventional wisdom about “AI”, often “throw everything at a commercial LLM”, can be a flawed strategy.
Deep learning should allow digital humanities scholars to allocate more time to research and less to data processing and markup. This paper aims to help researchers fulfil the promise of “AI”; that we do less of the repetitive work we want to avoid and more of the higher thinking and creating that defines us.
1. A. Vaswani et al., "Attention is all you need," in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 5998–600