Meaning

How is meaning in Chaucer’s works generated through human-AI interaction and open data?

1. Introduction

Generating meaning is an important endeavour when creating scholarly editions, which are bespoke and layered critical frameworks that weave issues related to historical and contemporary textual traditions.  The main aim of producing a digital pedagogical edition of Chaucer’s Pardoner’s Prologue and Tale (PPT) should therefore be to engage students in issues of the medieval period, empowering them to come up with their own interpretations of the text regardless of the prominence of the author. One purpose of scholarly editions more generally might be to represent and/or reimagine medieval authors’ works in ways that highlight the multiple possible perspectives of historical texts, and their enduring relevance to modern readers and users. The textual and societal questions arising from the analysis of medieval texts bear on the present, and Geoffrey Chaucer’s Canterbury Tales is an example of a frame narrative that incorporates different people from all walks of life, leading to further (re)interpretations as time goes on.

For a digital edition that primarily centres the academic skills development and knowledge of the medieval period of today’s university students, conveying meaning within Chaucer’s texts is particularly crucial if students are to cultivate their authentic voice on the text and its author, alongside the encouragement of their tutors. What happens when LLMs are utilised in editorial practices, and by students for independent study? Can we consider these models as meaning generators when probed and utilised by humans?

We wanted to understand whether newly available technologies such as generative AI, currently at the disposal of editors and researchers, could a) increase the scope to which critical scholarly editions cover themes and issues of the medieval past, potentially reducing the workload of the editor in assessing all avenues of the editorial practice, and b) increase the access to which students with entry-level knowledge of medieval literature and language can conjure their own ideas about Chaucer’s text. The latter point will require further iterations of the edition, as driven by user community collaborations, to fully establish the role of Large Language Models (LLMs) in guiding (or hindering) students’ critical thinking and creativity. On the whole, this chapter presents the opportunities and challenges within AI-assisted editions for students, as raised by an investigation into whether LLMs can generate text that is transformed into meaning (and authority) by editors, regarding Chaucer’s narratives, themes and concepts.

We drew on generative AI interfaces that are currently accessible to the public, and their responses to different prompts in producing critical-creative aspects of the edition. The methodological frameworks as presented in this source book, and our analysis of the technologies used, have been produced to be replicated by other editors and researchers. We therefore ensure a large number of the technologies for producing  AI-assisted editions are available without the need to subscribe to any services. Even if access to free AI technologies is limited in the future, the questions we raise and solutions we propose will be appropriate for investigating any source, and are especially relevant for those that are new to editing or learning about medieval texts.

We also highlight the reverse; that the problems relevant to the medieval period and Chaucer’s literature, as identified during the creation of an edition, affect assessments of generative AI outputs, their benefits and limitations. For instance, core issues affecting medieval public life, including:  religious authority and opinions and attitudes of ‘sin’; how medieval texts and analogues were, and still are, enriched by texts across the globe; the othering of identities; linguistic variation across different communities through time; and the complex process of textual transmission that takes us to the present, are packed into Chaucer’s literature and its reception. The insight gained from analysing medieval challenges, for secondary commentary in the present, can feed into investigations of generative AI’s treatment of the same topics.

An intriguing question might therefore be: Given Chaucer’s literature and legacy has led to new interpretations and adaptations of his work in the 21^st century, can Chaucer therefore influence the development of technologies in the present-day?

1.1. Generative Pre-trained Transformers

The way in which we critique and reflect on different sources is crucial for decision-making, and understanding whether certain technologies should be utilised in research, learning and teaching. As shown when probing AI-generated translations of Chaucer’s text, one-to-one mapping between word forms in the past and present is not always possible; additional information is often required to ascertain the meaning of a particular translation, and therefore annotation practices are key.

Generative AI (genAI), and Generative Pre-trained Transformers such as GPT-3.5 and -4, are systems that draw on generalised patterns in large datasets, to produce content and perform ‘intelligent’ tasks through statistical  predictions of this data. LLMs such as OpenAI’s ChatGPT are considered Artificial Narrow Intelligence (ANI) as its capabilities are limited to a specific set of tasks; they do not truly mirror  our understanding or interpretation of different sources. It is argued that Artificial General Intelligence (AGI) does not yet exist because it promises replication of this complex human cognition, “enabling machines to learn and execute a vast array of tasks, from driving cars to making medical diagnoses” (Marr 2024; see this Forbes article about the differences between genAI and AGI).

As Bender et al. (2021: 616-617) note, LLMs are ‘stochastic parrots’. These large models scrape information from their training datasets and combine sequences of data probabilistically, without ascribing any human meaning to the data when generating outputs. This data scoping can lead to outputs built on hegemonic views of society; the probable nature of the model can result in ‘hallucinations’, where the answers provided may be unrecognisable from the initial prompt or inaccurate without the user ever knowing. These hallucinations often happened when attempting to generate meaning and understanding through AI-assisted translation of medieval text; LLMs do not have the capacity to consistently choose appropriate words based on reasoning around etymology, semantic change, and phonology, alongside the voices of multiple communities.

There is widespread concern within education, law, healthcare, creative industries, and research on the climate crisis, about how generative AI might directly harm workers and members of society through responses lacking in accuracy and authority. The human capacity should thus be emphasised, as more investment is placed in the growth of these technologies, to ensure they are not negatively impacting its users and their dependents.

In education contexts, institutions are adopting balanced approaches that communicate the implications of generative AI on society. At the same time, empowering students to make decisions about the use of AI can help them to recognise their role in the development of these technologies, and weigh up whether its use as an educational tool is appropriate for a particular task. For students (and in this context, editors creating editions) who wish to use generative AI models to gain an insight into the traditions and cultures of medieval society, and the texts arising from this period, we address a number of challenges and opportunities in this chapter. Our investigation into the creation of student-friendly digital editions for learning about the medieval period was based around the aforementioned problems associated with the use of LLMs, and made specific to questions on the communication of Chaucer’s literature and the Middle Ages.

West et al.’s (2024) introduce the ‘Generative AI Paradox’, showing that while LLMs may be trained to produce expertise on a wide range of topics, the capability to generate this expertise may exceed its ability to acknowledge the core issue or sentiment behind that piece of knowledge. A hypothetical case for this paradox might be that LLMs can retrieve knowledge on complex medieval textual traditions, recognising that medieval manuscripts existed and ceased to exist during the move to print from the late fifteenth century. However, the model might simultaneously hallucinate the intricacies of Chaucer’s writing in the fourteenth century author, by conflating his biography with other medieval authors. His works were often disruptive of the prominence of the Catholic Church, and presented a combination of more ‘traditional’ and ‘modern’ linguistic and literary devices. Thus, the ability to generate information about a complex tradition outweighs the ability to distinguish Chaucer from other authors. We highlight cases where LLMs do not provide sufficient detail for students to do well on their course, and pinpoint where human editorial knowledge is key to rectifying these issues. In doing so, we assess the role of the human editor alongside AI and ascertain whether its use speeds up the process, alleviating some of the responsibilities of critical thinking and reflection, and show where the editor must be more vigilant about LLM responses on medieval texts.

1.2. Research questions

The Pardoner, like the other pilgrims, is described in the General Prologue with statements such as ‘hair as yellow as wax […] And he spread them over his shoulders’, ‘to make an attractive appearance, he wore no hood’, and ‘voice as small as a goat has’ (translation from the Harvard Geoffrey Chaucer Website). The scrutiny of his character by the narrator and other pilgrims arises from the false relics he carries, which he uses to convince innocent civilians that they should absolve their sin. The narrator also believes him to be a geldyng or a mare, which has been translated in most modern works as a ‘eunuch’ or ‘homosexual’ (but note that these labels are outdated today, e.g. see Homosaurus). The Pardoner is therefore said to be riddled in hypocrisy, because he represents the image of ‘sin’, subverting medieval ideals specific ideas of greed (and by extension, sodomy) despite his status in the Catholic Church. His tale is also pitched as a moral fable about three taverners who meet a gruesome end, warning others against the same greed which he is accused of being guilty of. There are therefore multiple discussion points for representations of the Pardoner that reflect  medieval society as a whole.

The ‘meaning’ element of the project investigated two key areas of AI edition-making for Chaucer’s Pardoner’s Prologue and Tale (PPT). Given what we know about the probabilistic nature of LLMs, we investigate whether they can a) produce annotations for specific words in the tale (linked to, for example, the etymologies of words, meaning change, biographies of referenced medieval/ancient philosophers, and places in the Middle Ages), and b) identify topics or themes in the text requiring additional summaries or guides for users/students. To ensure the retrieval and analysis of the above themes by LLMs, the following sub-questions were addressed:

1. Can LLMs identify the relevant themes in PPT; if so, can these themes be formatted into annotations and topic guides as part of the critical apparatus of the digital teaching edition?
2. Can LLMs retrieve relevant source material for the identified themes?
3. Can LLMs undertake both these tasks, while making links to specific examples in the text, to reduce the workload of the editor in finding elements of the text to encode?

These questions were then applied to the following tasks, which were later transformed into prompts for LLMs:

• Identify the Middle English terminology in the tale requiring a definition for the student to understand aspects of the tale. This includes words loaned from other languages, words that have changed their meaning over time, specific religious concepts relevant to the medieval period, etc.
• Summarise the relevant themes across the whole tale and within specific sections, and produce annotations and/or student guides to accompany the edition.

The human editor plays a crucial role in predetermining key areas that must be covered by the LLM, and monitoring the outputs produced by the LLM for accuracy, depth, and academic integrity. This latter point is particularly important for students who may not recognise the presence of hallucinations for a topic which is new to them, and could reflect future teaching practices which probe the accuracy and appropriateness of LLM responses.

2. Focus groups

The main themes of analysis for the Pardoner’s Tale relate closely to those brought to our attention in user focus groups and interviews (see our ‘Understanding’ chapter for detailed information on their insights). Students and teachers noted several areas where in-text annotations would be useful for modern readers, particularly those new to studying Chaucer. Some students drew on their experiences of engaging with digital editions and historical text editing to explain how useful annotated information was for contextualisation and interpretation of the text, and for showing any changes made in the editorial process. For those with limited experience in digital editing, students referred to issues of accessibility, as modern readers with little familiarity of Middle English language. They also expressed their historical (socio)linguistic interests regarding etymology and semantic change, showing where a word may be loaned from another language or has changed in meaning significantly since the medieval period.

Teachers confirmed the views of students, and explained that annotations can provide an enhanced reading experience and interactivity, which in turn increases engagement with different sociocultural issues. Annotations describing a word’s position within the text can provide additional information about why Chaucer (or the scribe) decided to use particular language for a character or narrative. For instance, Chaucer’s language to describe the Pardoner is interesting in that the character exploits lower social classes, yet he himself is prejudiced in the context of ‘anti-sodomy’ and the othering of identities outside medieval social norms. We therefore decided to focus on key religious and biblical words (including people and places mentioned by the Pardoner), cases where the medieval and modern language form differed in order to highlight significant semantic changes and variation, and words borrowed from other languages to highlight the European and global textual cultures in which Chaucer operated.

The learning and teaching community also appreciated the ability to contextualise the Chaucerian tale they were reading within the wider symbolism of the medieval period, along with the literary devices Chaucer used, the biography of Chaucer and the entire narrative of The Canterbury Tales. From a linguistic perspective, they appreciated details about the Middle English language and its status during periods of language change and variation.

One of the main themes underpinning student and teacher discussion, regarding why Chaucer studies is important to research today, was the human element of his works. They mentioned the idea of pilgrimage as both a religious and rebellious act; the portrayal of desire and hatred within the pilgrim’s tales and the dialogue between characters; social hierarchy and division in the medieval period; representations of gender, sexuality, race and culture; Chaucer’s writing within a global context; and the ability to uncover hidden identities and stories from the medieval period, whether that be through Chaucer’s writing, or through criticism of his works.

Five main themes were chosen as a result of student and teacher insights, to reflect the issues brought to light above. The following issues were also relevant to the study of the Pardoner, which intersect across many facets of humanity in both the past and present:

1. Themes of corruption, hypocrisy and satire in The Pardoner’s Prologue and Tale, related to the sin of the Pardoner and his tale, and links to religious morality and acceptability.
2. The scrutiny of the Pardoner’s gender and sexuality and how he was ridiculed by other characters and the narrator, and later explored by academic scholars. The fluidity of experience and identity was raised in the user focus groups and interviews, which captures the complexity of the Pardoner’s character and how others respond to him, including perceptions of medieval sin, as above.
3. The figure of the immortal Old Man, who stops the taverners in their tracks in the tale, raised important questions about where Chaucer gathered inspiration for his works and theories about where mortality comes from (e.g. links with religion and philosophy).
4. The Pardoner’s tale, which covers lessons on the dangers of greed and hypermasculinity, is a story that is reused in popular culture, but was also inspired by ancient and global literature. This theme therefore analyses the extension of stories (told centuries prior) to new contexts. The Pardoner’s prologue has also been linked to a medieval sermon, with the Pardoner drawing on ancient biblical and philosophical thought to construct lessons on desire, excess and self-destruction for his audiences, while he was considered to be sinning himself.
5. From a linguistic perspective, the Middle English language represents a point in time of considerable variation in structures, sounds, and poetic tendencies. These facts are crucial for understanding why Chaucer constructed The Canterbury Tales using particular language forms and literary devices.

In terms of background information and guides, students and teachers discussed many different topics which led to six main areas to be covered in the digital edition. These include:

1. A summary of The Pardoner’s Prologue and Tale and its sociohistorical context, format (e.g. as a ‘sermon’ or ‘literary confession’) and original form as folios in a medieval manuscript.
2. A summary of The Canterbury Tales and the frame narrative, along with details of its completeness and influences on medieval and early modern literature.
3. The specific manuscript covered in the edition (the Ellesmere manuscript), along with details of its potential scribe and its transmission and reception through time.
4. Details about the main character of the Pardoner, including the role of pardoners in the Catholic Church.
5. Details of the historical context, particularly surrounding the Black Death and how it affected people’s lived experiences and their stories.
6. A biography of the author, so students can infer how his language, experiences and social standing affected the works he developed. This also includes information on how his legacy affected literature across different cultures in England and beyond.

Each of these insights formed the focus of our investigation into LLMs’ capabilities in retrieving and linking elements of the tale for student users.

3. Prompt engineering for LLMs

Prompt engineering is pitched as having the potential to contribute to generated responses so they a) provide enough in-depth information to analyse a topic within its medieval/Chaucerian context and b) are accessible to beginners to the field. In most cases, we found that the more detailed and layered the prompt (in that there was a step-by-step process built into the prompt, or specific language to outline what was required), the more appropriate the response. However, there still remained inaccuracies and falsities in several responses, meaning that higher levels of detail did not automatically equal accuracy. There is thus increased workload for the editor in checking responses for inaccuracies.

Walter (2024) references two main ways in which language models can be prompted, i.e. by using ‘Zero-Shot prompts’ and ‘Few-Shot prompts’. Zero-Shot prompts result in non-specific responses which might be most useful when a user wishes to read a broad overview of a topic. Few-Shot prompts involve layers of tasks for the LLM to complete, and can be bolstered by identifying examples or data for the LLM to work with. In addition, Walter (2024: 14-16) outlines ‘Chain-of-Thought (CoT)’ prompting, which asks the AI to consider step-by-step reasoning in its response, and Narrative or Expert ‘role play’ prompting, where the LLM is asked to produce a response by ‘stepping into the shoes’ of someone with particular expertise. Self-Consistency (SC) prompting also results in step-by-step reasoning, but tasks the LLM with considering different viewpoints, yet this kind of task is prompted by the user and the LLM does not automatically reduce the homogeneity of its responses. Due to the timing of our Chaucer LLM investigation, in that it coincided with the rapid growth in public use of generative AI chatbots, we reserve in-depth investigation, of whether prompt engineering has an effect on response, for future research endeavours. We primarily focused on generating knowledge of Chaucer’s literature through a combination of Zero-Shot, Few-Shot, and Chain-of-Thought LLM prompts.

Regardless of the efforts made to engineer prompts, LLMs are not trained to ‘listen’ to their requests, and they  do not interpret and act on user prompts in the same way humans can. Stone, Goodlad and Sammons (2024) emphasise that the language produced by the LLM is formulaic, resembling encyclopaedia entries like Wikipedia, and often overlook experiences of underrepresented communities, reinforcing “the dominant discourse” by default. The model cannot be confident and open about its limitations, despite the language of the output suggesting otherwise, and they often resort to prompting the user to consider contextual factors around the topic themselves. This latter point may be useful for prompting a student to delve further into their research, but LLMs do not automatically retrieve data on these different topics and experiences. Instead, data is retrieved based on statistical probabilities, and a response is shaped which appears to answer the user. The main focus of the testing was thus on information-retrieval and suggestions of topics/themes, as opposed to assessing the LLM solely on meaning, which can only be generated by the human in response to LLM outputs.

3.1. testing phase

The testing phase commenced between September and December 2023. We predominantly used ChatGPT through OpenAI Playground (focusing on version gpt-4, now gpt-4o-mini¹), as it provides more control over the output generated. This included setting the ‘Temperature’ to 0, which reduces the randomness of the response for reproducibility purposes. We also provided instructions for the LLM via ‘System’, specifically ‘You are a university academic’, to mimic the ‘Expert’ style of prompting mentioned above.² Google Gemini (named ‘Bard’ in late 2023) and Microsoft Copilot (named ‘Bing Chat’ in late 2023) were also tested to provide alternatives to the ChatGPT responses, given their links to popular search engines. We recognise the technology is ever-changing, so we primarily focused on editorial questions which would be relevant for large statistical models in years to come.

To begin with, we tested whether LLMs could identify relevant words, phrases and themes for the specific tale, as well as source material which might support the user’s further reading, using Zero Shot responses that lacked any detail about the text and author. Prompts (1-3) provide examples of responses received from ChatGPT.

These responses led us to consider whether more detail in prompting, such as referencing the specific manuscript, tale, or providing examples, would support LLMs to identify relevant medieval language.

4. Wikimedia, APIs and supporting LLM prompts

As a result of the above challenges, we wanted to understand whether structured, open and crowdsourced encyclopaedic data, such as that of Wikimedia, could support AI’s generation of material. The acronym RAG (retrieval-augmented generation) was coined by Lewis et al. to show how LLMs and natural language processing (NLP) tasks can be fine-tuned using open data from Wikipedia to produce more “specific, diverse, and factual language” (2020: 1), which we aimed to replicate when inputting prompts into chatbots.

We explored the outputs of multifaceted prompts incorporating Wikidata/Wikipedia, and tested whether its addition could improve LLM’s identification of relevant glosses in the text, summarisation of the topic, issue or theme, and generation of student support guides. One of the texts involved a comparison of LLM outputs where there was a presence or absence of Wikimedia support in the prompt. These routes allowed us to explore the overall editorial intervention required at each stage of the process, and how much prior knowledge would be required to engineer prompts as an editor.

Eijkman (2010) investigated attitudes in higher education about the use of Wikipedia by students. He referenced academics’ views that students must critically evaluate sources regardless of their origin or whether they arose from crowdsourced knowledge. There were also suggestions that Wikipedia may not have the same authority as sources that underwent rigorous peer review, and that if it were subjected to the same traditional reviewing process, it may hold some academic integrity. Yet, academics also noted that Wikipedia serves as a new “regime of power in academic knowledge construction” (Eijkman 2010: 181), providing access to vaster pools of knowledge for students and members of the public which is often hidden behind institutional paywalls. This article from 2015, written by a member of the Digital Data Design Institute at Harvard, also emphasises the advantages of crowdsourcing. In the case of Wikipedia, there are regular community debates and engagement built into each article. With the addition of Wikimedia data, we could thus test whether LLM responses improve based on support by open data monitored by an online community of users.

We used Wikidata to retrieve structured data on specific people, places and concepts raised in the tale, such as this entry for Seneca the Younger, a Roman Stoic philosopher (4 BCE-65 CE). Entries such as the title of the page, religion, date of birth, place of birth, date of death, etc. formed key biographical information for the LLM to produce annotations within the text. Since only small amounts of data were required, the MediaWiki Action API was used to collect data entities in plain text and JSON format (a format for storing and transmitting structured data), which were then fed into LLMs.

For lengthier thematic guides, for instance, on the religion, social class and reception of The Canterbury Tales, entire Wikipedia page entries were retrieved using the ‘action=parse’ query within the MediaWiki API. These API methods allowed for data in plain text or JSON to be inputted into different LLM interfaces, without the distraction of hyperlinks. We layered prompts for the LLM using ‘Few-Shot’ or ‘Chain-of-Thought’ techniques, requesting that the LLM considers Wikidata in its response while also producing an output for a particular type of audience, i.e. university students.

As noted by Henrickson and Meroño-Peñuela, crowdsourced opportunities such as Wikipedia might forge “collaborative meaning-making”, leading to “joint discussions towards common understandings of texts written by others” (2023: 14). They argue that prompt engineering is a way to mimic these complex layers of meaning (using hermeneutic interpretation as their site of investigation). These types of initiatives, alongside in-depth analysis of LLM responses, may therefore be welcomed as academia moves to open research approaches involving multiple voices and viewpoints. While Wikipedia has the potential to bolster the seemingly homogenous responses arising from LLMs, Wikipedia is also a privileged resource that often centres the academic insight of one kind of researcher. Schneider notes that: 

“Standard-oriented text, as found, for example, on Wikipedia, is treated as an objective “ground truth” or “gold standard,” and this particular datafied content is conceived as if it were the essence of a given language (Schneider 2022). Bender et al. (2021: 614) have discussed how such assumptions reproduce social biases. Nonetheless, NLP researchers commonly refer to privileged sites such as Wikipedia as if its contents constituted “raw, ungrounded text” (Bommasani et al. 2021: 25).”

Mitchell (2021) also highlights the gender and racial bias within Wikipedia topics, with those articles of interest to women likely to be left out, few topics on Black history, and negative images of African people. Over 80% of the self-reported ‘occasional’ and ‘regular’ contributors were also likely to be male, many of which were under 30 with some sort of degree, and no partners or children. Much like genAI cannot retrieve much nuance from the study of the medieval period and its wide range of identities, the use of an LLM prompt that has been fine-tuned by Wikipedia may parallel tasks whereby large amounts of training data are scoped for generalised patterns, resulting in hegemonic worldviews. At the end of this chapter, I consider how crowdsourced knowledge might be retrieved in equitable ways from a wider range of people. Edition-making that incorporates the voices of many different communities is required to counteract hegemonic LLM responses, and for feeding into the development of AI technologies overall.

4.1. Prompting LLMs to annotate Chaucer’s tales, with support from Wikidata

To understand fully the merits and downfalls of Wikipedia, both as an encyclopaedia of knowledge, and as a stakeholder in LLM knowledge retrieval, we created a structure to be used consistently across prompts. The following structure investigated the creation of different types of annotation by LLMs. This included the annotation of people, places, medieval religious concepts, words arising from the medieval period, and loanwords: 

1. Your task is to identify which part of the following text from Chaucer’s Pardoner’s Prologue and Tale mentions the term [word, as identified by the editor and found from Wikidata]: [extract from tale, pre-determined by the editor].
2. Then, write a summary for students, to a university level standard, of the meaning of the term [word] in the context of the Pardoner’s Prologue and Tale, using the following Wikidata API output: [data].
3. Finally, write a guide for university level students about how they might analyse the term [word] within the context of the Pardoner’s Prologue and Tale.

Here we present some of the tests we conducted to determine whether LLMs would utilise the data from the API output, or draw on its own database of knowledge. Prompts (4-5) were inputted into different LLMs using the Wikidata structured entries for the word ‘Flanders’, which is referenced in the text to set the tavern scene for the Pardoner’s tale. 

4.1.1. Summarisation of Wikidata

Commentary on responses

ChatGPT-4 provided an overall summary of elements of the API output, grouping together relevant pieces of information about the history of the region, languages, economy, and where information on the place can be found. It also flags the ‘different from’ entry of the Wikidata entity, explaining that Flanders is “distinct from the Flemish Region and Flemish Community”. There appears to be a clear focus on paraphrasing the Wikidata content in a readable manner. 

Gemini (at the time, Bard) summarises the term ‘Flanders’, before providing a condensed set of entries from Wikidata underneath. It brings in knowledge about Flanders’ position in the continent, bordering countries, and the industries making up the economic activities in Flanders. Gemini thus appears to fill in gaps in knowledge from the Wikidata output, while also providing a summary readable to the end-user.

Copilot (at the time, Bing Chat) began by grouping the languages Belgian French, Flemish Sign Language and  highlighting the location of the region. Following this, the chronology of the response matches that of the API output, resulting in repetition that may not be readable to an end-user. It does, however, reference important topics requiring additional detail (e.g. the economy of Flanders), as well as the lack of English labels in some cases. Copilot also provides sources for Flanders, including the original Wikipedia page, along with the publicly available encyclopaedia, Britannica. These references suggest that Copilot is drawing on a wider bank of knowledge, potentially using the Bing search engine in the absence of relevant data describing the term. 

Each LLM appears to produce a different response, based on its training data and any other information to which it has access (e.g. search engine data). The LLMs may also be trained to provide responses using different styles, such as conversational summaries or to analyse the prompt closely. These priorities also continue to evolve in real time as the technologies develop.

4.1.2. Definitions, contextualisation, annotation

Commentary on responses

In both responses, ChatGPT accurately identifies the relevant part of the text featuring ‘Flanders’, i.e. the second line, In Flaundres whilom was a compaignye. ChatGPT also draws on elements of the API output to highlight issues for student consideration, such as linguistic diversity and the economic conditions, yet the former is potentially not relevant to the overarching theme of sin in the Flanders tavern. The focuses are different in the response that does not utilise Wikidata. This includes information about: the textile industry and its contribution to wealth and commercial activity, morality and the symbolism of Flanders as materialism, and guides for students to consider the location of Flanders within the plot, as well as the literary devices used to convey the overall theme of sin. Both responses also accurately contextualise Flanders within the geographical and cultural context, which depend on the use of the Wikidata output in terms of the issues it raises. Indeed, the editor must also check whether the response remains on track in terms of the original intentions of the prompt.

Different responses were provided by Gemini. The response supported by Wikidata summarised the term in the context of the tale, focusing on culture and religion, the dangers of sin, and the Pardoner’s role in telling the tale. This is supported by an additional guide on ‘symbolism’, whereby students could interpret the taverners of Flanders as a representation of sin, which was not raised by ChatGPT. There are also further questions about the Pardoner’s ability to create fear in his audience, links to the ‘human condition’ and the choice of setting. The response unsupported by Wikidata provided similar avenues for exploration, with fewer details about Flanders as a location. However, both Gemini responses provided the incorrect line number for Flanders’ mention, yet repeated the correct extract of the tale. Rather than focusing on the irrelevant details of the Wikidata output, Gemini thus appeared to identify relevant contextual information through the use of the structured data entries, which may guide the role of the human editor in producing the LLM prompt to begin with, or provide more ‘thoughtful’ links between the term, the tale, and the historical context.

We concluded that LLMs use Wikidata by either incorporating the API output in its entirety (including potentially irrelevant information), or, if the prompt more closely matches the information of the structured data entry, the LLM may paraphrase the relevant aspects of the API output while drawing on its own bank of knowledge to fill in the gaps. If Wikidata is not used, the LLM may focus more on the wider sociohistorical context and symbolism within the Chaucerian canon to produce student guides, the latter of which may draw on contemporary literary interpretation. The role of the editor is therefore to assess the response for, not only accuracy, but also relevance to the tale and the target user community. For some, the lack of detail may not be enough to generate ideas for a classroom task or activity, and more (human-generated) information would guide students to unpick the tale for themselves. There are also benefits and drawbacks to LLM prompting with and without Wikidata, which depend on the tasks of the human editor (resulting in varied response reliability), but also the foundations on which Wikipedia is built.

4.2. Prompting LLMs to generate Chaucer topic summaries and guides, with support from Wikipedia

For lengthier topic guides and summaries, we used ChatGPT and Gemini (given Copilot did not offer enough tokens to utilise Wikidata), based on the following prompting structure:

1. Your task is to identify which part of the following text from Chaucer’s Pardoner’s Prologue and Tale is [description of a theme or topic]: [extract from the tale, also pre-determined by the editor].
2. Then, write a summary for students, to a university level standard, of [theme], and how it links to [wider context of tale/medieval period, if relevant], using the following Wikipedia API output: [data].
3. Finally, write a guide for university level students about how they might analyse [theme] within the context of the Pardoner’s Prologue and Tale.

4.2.1. Summarisation of Wikipedia

Commentary on responses

ChatGPT and Gemini summarised the main points of the output, referring to aspects such as the sermon’s theme, the Pardoner’s ‘false’ position in the church and hypocrisy, and the literary confession of Faus Semblaunt in the Roman de la Rose. Both responses accurately identify the text as coming from the Pardoner’s Prologue and Tale, as opposed to any other pilgrim’s tale, despite there being no reference to the tale or author in the API output. Whereas ChatGPT summarised the API output in less detail, replicating the structure/ordering of the output, Gemini provided more detail, adopting similar language to the Wikipedia data (e.g. ‘duping of his victims’ and ‘false credentials’) where ChatGPT may have paraphrased. Gemini does offer alternative drafts of the output which may contain more paraphrasing or different language to the Wikipedia data. The issue relates to whether LLM considers the definition of the word ‘summarise’ as LLMs do not routinely provide their sources or citations. The academic definition of the verb, where authorship and copyright is taken into account, may not transfer to LLMs when they are tasked to draw on data provided in the prompt, as well as its own bank of knowledge.

4.2.1. Topic guide generation

Commentary on responses

When testing LLMs and their ability to produce topic guides of Chaucer, one of the vital aspects we wanted to ensure was whether the model could identify the theme from a long extract of text. The API output is not likely to be able to support the LLM in completing this task, given it is up to the editor as to which extract is chosen, and the Wikipedia article may not have identified the same examples. In this case, the article has provided some of the Pardoner’s words, such as Radix malorum est cupiditas (the sermon theme of ‘Greed is the root of [all] evils’), as well as discussion around the Pardoner’s false relics, his self-professed quick wit and viciousness, and lack of care for his audience and correcting their sin. Interestingly, when the API output was not used, ChatGPT provided an example from the specific extract included in the initial prompt (e.g. For myn entente is nat but for to wynne / And no thyng for correccioū of synne / I rekke neuere whan they been beryed / Though þt hir soules goon a blakeberyed). When the Wikipedia data was included, the only reference to the Pardoner’s words are those provided in the API output and not from the specific extract (i.e. the Latinate phrase mentioned above). Gemini does not provide any specific Middle English examples in either of the responses, simply providing auto-summarisation and commentary.

The summaries and guides from both LLMs also differed in content and format. ChatGPT’s response paraphrased the Wikipedia API output and provided an overall introduction to the literary confession, telling a detailed story of the Pardoner’s deceit and sin. There are additional links to the existence of corruption in the church, which do not occur in the specific API output but appear in the Wikipedia article of The Pardoner’s Tale. ChatGPT’s response may have linked the Apostolic Authority of the priesthood to discussion of the church’s corruption in the entire article (however, this was provided as an example in the prompt). The questions/guide provided by ChatGPT centre different themes, such as the Pardoner’s character, the irony of the sermon theme, the role of the confession, and links to hypocrisy and manipulation, all of which can be unpacked by the prologue provided in the Wikipedia output. ChatGPT, without support from the API output, also provided a relevant summary, with a different description of the Pardoner (e.g. the Pardoner as ‘self-serving’, his focus on ‘accumulating wealth’, and the contradictions with the church’s teachings of ‘humility, charity, and selflessness’). The response also focused on the theological debate raised by God’s involvement in sin through the Apostolic Authority, as provided as an example in the prompt. However, the guide for students’ analysis instead focused on identifying the confession (reflecting the beginning of the response where an explicit example is provided), and how the language can be analysed, contextualised, linked to the aforementioned religious climate, and evaluated in terms of the audience. The editor might choose which of the two approaches to use for a student activity or task (i.e. a thematic analysis, or analysis of language and its link to the time period), and factor this into the initial prompt, or compare responses with and without support from Wikipedia. 

For Gemini, the format of the response was split into headings which arise depending on the LLM’s interpretation of the prompt. In the case of the response supported by Wikipedia, there was a summarised literary confession which follows the structure of the API output, links to the religious climate of the church, and theories related to God’s involvement (again, bringing in corruption which was not necessarily mentioned in the API output). The response without Wikipedia support followed the same structure, suggesting the API was not utilised entirely and the LLM drew on its bank of knowledge for this topic (which may often be similar Wikipedia data). The guides in both responses were very similar, with no difference in approach despite the difference in support for the prompt. The API output seems to have impacted whether Gemini made links between the tale and the wider frame narrative; for example, Gemini considers the Wife of Bath’s confession, as raised by the API output. 

Interestingly, none of the prompts from both LLMs reference the Roman de la Rose as an influence on the Pardoner’s confession, or the analogues in Buddhist, Persian and African literature, all of which are relevant to students today in analysing the global context of Chaucer’s works. There is also no mention of the Pardoner’s gender and sexuality, which were a major factor in considering the ‘sin’ of the Pardoner, and link to the hypocrisy discussed in the LLM responses. There are therefore some occasions where LLMs select relevant information, drawing on its own bank of knowledge to link the story to the wider context to provide a readable summary, but equally, there are times when vital insight is missed from the API output, or the training data is lacking and cannot address topics of interest to Chaucer students today.

Overall, we summarise the editorial considerations when engineering prompts to annotate medieval texts and provide contextualisation or avenues for further analysis:

1. The amount of prior knowledge required by the editor, including the external resources that should be added to the initial prompt to support the LLM’s response. For example, will you adopt a layered task/Few-Shot prompting method, or use Zero-Shot prompts for initial knowledge generation? How much primary textual material will you draw on, and how will you be explicit about how the LLM should use this data? Do you think any of these approaches will result in the response you are expecting? Will they speed up or hinder the editorial process?
2. The number of tasks to factor into the prompt, which may increase or reduce the level of detail in the LLM’s response.
3. The audience of the edition or resource, which would affect the content and format of the LLM’s response and what you deem suitable for that audience (the type of expertise the LLM should assume might be written into the prompt). 
4. The amount of information to retrieve from the response, to feature in the digital edition or resource. How much of this information needs to be edited to fulfil your own criteria (which ideally should be established before deciding whether LLMs can be useful tools for a specific task)? How will you be transparent about the use of the LLM data in your edition? Does the use of LLM increase or decrease the amount of time spent gathering data for the edition or building its critical apparatus?

5. ‘Interpretations’ of Chaucer’s language: Example of LLM response analysis

The tests above highlighted the editor’s involvement in building prompts to receive different types of responses from LLMs, especially when they may not have been trained extensively on medieval data. Here, we analyse a response we received across different areas of the edition, which were intended to add meaning to the text for students who are new to studying Chaucer’s literature. We put ‘interpretation’ in quotation marks, given LLMs do not accomplish any human-level interpretation of the text, but provide probabilistic responses to the user’s prompts based on the data it has scraped. 

From interviewing our user community, we acknowledged several areas which LLMs must draw on, in order for their responses to constitute authoritative knowledge on Chaucer’s literature. For instance, the responses must provide nuanced descriptions and explanations of evolving etymologies from the medieval period to the present-day, links between language and literature and the wider medieval context, and information which is suitable for student audiences to sustain engagement with the period, all while appreciating the importance of literature from the Middle Ages and its impact on modern society. Many of these key tasks underpin the work currently being completed by scholars in the field, and the original aims were to improve access to Chaucer’s literature for beginners, as well as the process for editors. With the inclusion of crowdsourced data in an edition, we hoped to increase the human element of automated editing, particularly those that encourage online community debate and discussion about world matters, from people of all walks of life. 

One example where we tested the use of a prompt without input from Wikidata was for the annotation of the word ‘wench’ and its etymology. The explanation of this example is also provided in a submitted paper by Sophie Whittle (Whittle 2024).

Use of ‘wench’ by Chaucer in the Pardoner’s prologue

By cause I wol nat beggen ydelly
Because I will not beg idly
I wol noon of the Apostles countrefete
I will not imitate any of the Apostles
I wol haue moneie wolle chese and whete
I will have money, wool, cheese, and wheat
Al were it yeuen of the pouereste page
Even if given by the poorest page
Or of the pouereste wydwe in a village
Or by the poorest widow in a village
Al sholde hir children sterue for famyne
Even if her children should starve for famine
Nay I wol drynke licour of the vyne
Nay, I will drink liquor of the vine
And haue a ioly wenche in euery toun
And have a jolly wench in every town

(Lines 120-127 of our edition)

Carissa Harris has written extensively on the use of ‘wench’ in the medieval period. She recognises the complex history of the word, whose meaning encompasses the control and domination of many marginalised groups through time. The word’s meaning extended to a number of groups in its earliest attestation, with Old English wenchel meaning “a child (of either sex)” or “a servant or slave; also, a common woman” (OED, ‘wenchel (n.)’, 2023). As noted here, its sense developed to primarily refer to adult women, yet it also developed a sense from the 17th century whereby ‘wench’ was used against Black enslaved women and embedded within colonialism. In the OED, sense 3.b. of ‘wench’ comes from the US, with little explanation about its use in the context, other than four attestations from 1765 to 1891 which reference Black enslaved women (OED, ‘wench (n.)’, 2023). While these meanings arose following its use in the medieval and early modern periods, it is important to address how the language has evolved for students who may only know of its limited modern usage. As Harris (2023) states, “the label of wench is capacious enough to include many, yet binding enough to compel to coerce and to constrain”. Its use by Chaucer in this extract therefore reflects the objectification and marginalisation of those outside the elite few in the medieval period, particularly women, which led to the later weaponisation of ‘wench’ in order to degrade Black women.

Given the important etymological history of the term, any explanation of its use at any point in time, whether through adaptations and representations of earlier work in the form of editions, or by a machine, must consider how it was used to push individuals of lower social standing to the margins. LLMs might not produce this knowledge given its wide generalisation of patterns in large datasets, which is problematic if there is little knowledge of the term from an editorial or student perspective. The following prompt was inputted into ChatGPT in October 2023 (version gpt-4, now available to the public via gpt-4o) as part of a long list of words forming an activity for students to compare different types of translations (human or AI): 

Prompt: 

Produce an activity for university-level undergraduate students which allows them to investigate the use of the following words from the Chaucer's Pardoner's Prologue and Tale. Each of the words and their translations relate to referents in the text with different perceived characteristics or roles, all with unique etymologies. Please also include a brief description of each word in relation to data in the Oxford English Dictionary and Middle English Dictionary. Finally, include a summary for each translation which serves as a prompt for the students, explaining some of the reasons behind the choices of these translations: ME ‘ioly wenche’ > ModE ‘jolly wench’ or ‘pretty wench’ [among other words/translations].

ChatGPT’s response: 

In Middle English, a phrase for a pretty or cheerful young woman. Modern English retains the positive associations of this phrase even as 'wench' has taken on more negative or derogatory connotations. 

First, there is little detail about the sense of ‘wench’ that is steeped in coercion, only that its use in this context can be translated to ‘pretty or cheerful young woman’. The prompting also resulted in a hallucination, suggesting Modern English retains “the positive associations of this phrase even as ‘wench’ has taken on more negative or derogatory connotations.” Other than the sense 1.c. in the OED from the early modern period, where ‘wench’ was used “as a familiar or endearing form of address; used chiefly in addressing a daughter, wife or sweetheart” (OED, ‘wench (n.)’, 2023) – a now dialectal or archaic use of the word – there are no positive connotations associated with its use in the present-day (as the modern use of ‘wench’ is likely to have been reclaimed or used for ironic purposes). The prompt has likely recognised one of the translations as ‘pretty’ and the other as ‘jolly’ (the former of which is part of Harvard Geoffrey Chaucer Website’s interlinear translations, and the latter produced by ChatGPT in an earlier prompt), and provided an explanation of its etymology based on these translated descriptors.

Commentary on response

The main aspect to note is that ChatGPT-4o only references the ‘young girl or woman’ use in early Middle English, and does not acknowledge that ‘wench’ was shortened from wenchel, which is shrouded in the coercion and control of children, slaves and servants. The response refers to women of ‘lower social status’ and the pejoration of the term as it came to be linked to ‘sexual immorality’, yet there is no mention of its use by people in positions of power against women, and its later colonial use.

The only place where ‘wench’ is discussed in relation to objectification is in the specific context of the Pardoner and his ‘attitude towards women as objects of pleasures and tools for manipulation’. Students are also invited to analyse whether the translated term ‘pretty wench’ captures its intended meaning by the author, how it might be received by a reader, and what these social attitudes reveal about “medieval gender roles and expectations” and their depictions in medieval works. The response insinuates that there is further meaning to uncover, through the statement ‘Many modern translations render “ioly wenche” as “pretty wench,” a sanitized and perhaps oversimplified version of the original phrase.’ Yet, this statement is likely not enough to uncover a nuanced understanding of the term and its development across the history of English by newcomers to Chaucer’s works.

Lastly, a transcription is provided by ChatGPT to contextualise the use of ‘wench’, yet this is not the correct extract from the text and only references women from a widespread medieval perspective about infidelity and sin: “And nevere shal he moore his wyf mystriste / Though he the soothe of hir defaute wiste / Al hadde she taken preestes two or thre.” The response also may not be drawing on legitimate data from the OED and MED, as the information in these resources differs to the explanations provided by ChatGPT.

With human input, students may be more encouraged to uncover the racialised use of the term and make links to its origins where it was initially used to assert hierarchical structures of power. LLMs may produce an appropriate response if prompted to explore the 17th century use, suggesting it is heavily influenced by the human in the prompt (and potentially rewarded for later iterations of the technology). With lacking knowledge about the development of this meaning, editors and/or students may not have the tools to begin researching the topic.

6. Transparency documents

The following three transparency documents demonstrate the level of editorial input required when including LLM responses to produce a university resource (specifically, annotations, topic/thematic guides, and background guides on the text and author). 

Many of the responses were edited by merging different responses together, so it is the responsibility of the editor to evaluate these responses and determine which match their own established criteria. Because of this, the transparency documents may not include a large amount of edits, yet the final material will differ depending on what the editor feels is relevant. The documents also do not show the time spent discarding irrelevant information from multiple AI sources. Please note, there may also be some final tweaks to the interface of the edition that are not displayed in the below documents.

Annotations and topic summaries

Topic guides

Background guides

7. Manuscript image retrieval, recognition and generation

A crucial element to representing and learning the meaning of medieval texts, and the context surrounding their production and transmission, is through access to the ‘original’ text (as far as is possible, noting the existence of multiple manuscripts and scribes). 

Our edition provides access to digitised versions of the Ellesmere manuscript showing The Pardoner’s Prologue and Tale (information on the scribal hand can be found on the Late Medieval English Scribes project website). These were retrieved from The Huntington Library (San Marino, California) website using the International Image Interoperability Framework, or IIIF, a set of open standards to deliver high quality images for use in online resources such as digital editions. These are displayed on our edition alongside transcriptions of the tale, and each folio of the tale is available to inspect more closely, so students can zoom in on areas such as marginalia, decoration and borders, illuminations of pilgrims, and explore medieval scribal palaeography. 

The folios also feature in a Manuscript activity we produced to compare the process of textual transmission through time. This activity included annotated images, or image mapping, to prompt questions about features of medieval manuscript folios. Students can also explore representations of Chaucer’s works and characters through time via manuscripts, print editions, and modern adaptations of his works. The activity occurs alongside a Manuscript guide to take students through the process of text dissemination at various points in the history of English. Each of these features of the edition contribute to enriching students’ understanding of the medieval textual tradition, and recognise how different the process is today.

7.1. Image-to-Text generation

Prior to including IIIF and links to the digitised manuscript in our edition, we briefly investigated the ability of generative AI to recognise folio features, analyse them with regard for the historical context, and provide a useful summary for a student user. In November 2023 to January 2024, we began with Google’s genAI chatbot Gemini (at the time, Bard), to discover whether medieval manuscripts featured as part of the training data for these technologies, despite this being unlikely. Our below discussion highlights one test we conducted, involving the attachment of publicly available images of the Ellesmere manuscript. Here, we queried the use of Anglicana handwriting, decoration and illumination and how this information can be digested to students. The prompts were again bolstered by open Wikidata:

Gemini prompts (Bard version), 29 November 2023

Prompt 1

Describe what you can see on the following image of a manuscript folio, to a university level standard [attached image of folio 138r from the Ellesmere manuscript, which displays the beginning of The Pardoner’s Tale alongside an illumination of the Pardoner].

Response 1

The manuscript folio you have provided depicts the Wife of Bath’s Tale from Geoffrey Chaucer’s Canterbury Tales. The image is a full-page illustration, with text on the verso (back) of the folio. The image is divided into two halves, with the text on the left and the illustration on the right.

The illustration shows a group of eight people seated around a table. In the center of the table is a man with a long beard and a bald head. This is the Wife of Bath’s fifth husband, Jankyn. He is wearing a red robe and a white turban. To his left is a woman with long blonde hair and a green dress. This is the Wife of Bath. She is gesturing with her right hand, and she is holding a cup in her left hand.

Around the table are six other people. They are all dressed in different colors and styles, and they are all engaged in conversation. It is likely that these are the other pilgrims on the Canterbury pilgrimage.

The text on the left-hand side of the folio is the beginning of the Wife of Bath’s Tale. The text begins with the following lines:

In th’olde dayes of the king Arthour,
Of which that Britons speken great honour,
All was this land fulfilled of fairy;
The Elf-queen, with her jolly company,
Daunced full oft in many a greene mead;
This was the old opinion, as I rede.

The Wife of Bath’s Tale is a humorous and bawdy tale about a woman who has been married five times. In the tale, the Wife of Bath argues that women should have the same rights as men, and that they should be free to choose their own husbands. The tale is also a critique of the double standards that existed in medieval society.

The illustration on the right-hand side of the folio is a lively and colorful depiction of the Wife of Bath’s Tale. The artist has captured the energy and humor of the tale, and he has created a vivid and memorable image of the Wife of Bath.

University-level analysis

The illustration of the Wife of Bath’s Tale is a significant work of medieval art. It is one of the few surviving full-page illustrations from the Canterbury Tales, and it is a valuable example of medieval painting. The illustration is also significant because it provides a rare glimpse into the world of medieval women. The Wife of Bath is a complex and contradictory figure, and the illustration captures her strength, humor, and independence.

The illustration can be analyzed on a number of levels. On one level, it is a straightforward depiction of the Wife of Bath’s Tale. The artist has captured the main characters and events of the tale, and he has created a visually appealing image.

On another level, the illustration can be seen as a commentary on medieval society. The Wife of Bath is a subversive figure, and her tale challenges many of the social and religious norms of the time. The illustration reflects the Wife of Bath’s subversive nature, and it can be seen as a challenge to the authority of the Church and the aristocracy.

The illustration can also be analyzed on a formal level. The artist uses a variety of techniques to create a dynamic and engaging image. The composition is carefully balanced, and the use of color is bold and vibrant. The artist also uses a variety of brushstrokes to create different textures and effects.

Overall, the illustration of the Wife of Bath’s Tale is a sophisticated and accomplished work of art. It is a valuable example of medieval painting, and it provides a unique insight into the world of medieval women.

Prompt 2

Describe the use of Anglicana handwriting, illuminations, and marginalia, in the following folio from the Ellesmere Chaucer [attached image of folio 138r], with reference to the following Wikipedia API output [from the Ellesmere Chaucer Wikipedia article]: 

The Ellesmere Chaucer, or Ellesmere Manuscript of the Canterbury Tales, is an early 15th-century illuminated manuscript of Geoffrey Chaucer's Canterbury Tales, owned by the Huntington Library, in San Marino, California (EL 26 C 9). It is considered one of the most significant copies of the Tales.

History
Written most likely in the first or second decade of the fifteenth century, the early history of the manuscript is uncertain, but it seems to have been owned by John de Vere, 12th Earl of Oxford (1408–1462). The manuscript takes its popular name from the fact that it later belonged to Sir Thomas Egerton (1540–1617), Baron Ellesmere and Viscount Brackley, who apparently obtained it from Roger North, 2nd Baron North (1530/31-1600). The library of manuscripts, known as the Bridgewater Library, remained at the Egerton house, Ashridge, Hertfordshire, until 1802 when it was removed to London. Francis Egerton, created Earl of Ellesmere in 1846, inherited the library, and it remained in the family until its sale to Henry Huntington by John Francis Granville Scrope Egerton (1872–1944), 4th Earl of Ellesmere. Huntington purchased the Bridgewater library privately in 1917 through Sotheby's. The manuscript is now in the collection of the Huntington Library in San Marino, California (EL 26 C 9).

Description
The Ellesmere manuscript is a highly polished example of scribal workmanship, with a great deal of elaborate illumination and, notably, a series of illustrations of the various narrators of the Tales (including a famous one of Chaucer himself, mounted on a horse). As such, it was clearly a de luxe product, commissioned by a very wealthy patron.
The manuscript is written on fine vellum and the leaves are approximately 400mm by 284mm in size; there are 240 leaves, of which 232 contain the text of the Tales.

Scribe and its relation to other manuscripts
The Ellesmere manuscript is thought to be very early in date, being written shortly after Chaucer's death. It is seen as an important source for efforts to reconstruct Chaucer's original text and intentions, though John M. Manly and Edith Rickert in their Text of the Canterbury Tales (1940) noted that whoever edited the manuscript probably made substantial revisions, tried to regularise spelling, and put the individual Tales into a smoothly running order. Up until this point the Ellesmere manuscript had been used as the 'base text' by several editions, such as that of W. W. Skeat, with variants checked against British Library, Harley MS 7334.

Linne Mooney identified Ellesmere's scribe as Adam Pinkhurst, a man employed by Chaucer himself, but recent scholarship has claimed that this identification is without merit. The same scribe appears to have been responsible for writing the Hengwrt Manuscript of the Tales, now considered the earliest, most authoritative, and closest to Chaucer's holograph. This would also imply, however, that the revisions seen in the Ellesmere manuscript would have been carried out by someone who had worked with Chaucer, knew his intentions for the Tales, and had access to draft materials.

The Ellesmere manuscript is conventionally referred to as El in studies of the Tales and their textual history. A facsimile edition is available.

Response 2

The Ellesmere Chaucer is one of the most important manuscripts of the Canterbury Tales. It is a highly polished example of scribal workmanship, with a great deal of elaborate illumination and, notably, a series of illustrations of the various narrators of the Tales. As such, it was clearly a de luxe product, commissioned by a very wealthy patron.
The manuscript is written on fine vellum and the leaves are approximately 400mm by 284mm in size; there are 240 leaves, of which 232 contain the text of the Tales.

The scribe of the Ellesmere manuscript is believed to be the same one who wrote the Hengwrt Manuscript of the Tales, now considered the earliest, most authoritative, and closest to Chaucer’s holograph. This would also imply, however, that the revisions seen in the Ellesmere manuscript would have been carried out by someone who had worked with Chaucer, knew his intentions for the Tales, and had access to draft materials. The Ellesmere manuscript is conventionally referred to as El in studies of the Tales and their textual history. A facsimile edition is available.

The use of Anglicana handwriting, illuminations, and marginalia, in the following folio from the Ellesmere Chaucer is described below:

Anglicana handwriting: The text of the Ellesmere Chaucer is written in a style of handwriting called Anglicana, which was popular in England during the 15th century. Anglicana is a highly stylized script with a lot of flourishes and curves. The scribe of the Ellesmere Chaucer was a skilled calligrapher, and the handwriting is clear, elegant, and easy to read.

Illuminations: The Ellesmere Chaucer is also notable for its beautiful illuminations. These are small illustrations that are often found in the margins of the manuscript. The illuminations in the Ellesmere Chaucer are some of the best surviving examples of early 15th-century English illumination. They are full of color and detail, and they often depict scenes from the Canterbury Tales.

Marginalia: The Ellesmere Chaucer also contains a large amount of marginalia. This is text that is written in the margins of the manuscript, and it often includes commentary on the text, notes about the illustrations, and other additional information. The marginalia in the Ellesmere Chaucer is a valuable source of information about how the text was read and understood in the 15th century.

Here are some specific examples of the use of Anglicana handwriting, illuminations, and marginalia, in the following folio from the Ellesmere Chaucer:

Anglicana handwriting: The text of the folio is written in a clear and elegant style of Anglicana handwriting. The scribe uses a variety of different pen strokes, and the letters are well-formed and proportional. The handwriting is easy to read, and it contributes to the overall beauty of the manuscript.

Illuminations: The folio contains a small illumination of the Knight, one of the narrators of the Canterbury Tales. The illumination is well-drawn and colored, and it shows the Knight riding on his horse. The illumination is a beautiful addition to the manuscript, and it helps to evoke the atmosphere of the Canterbury Tales.

Marginalia: The folio also contains a number of glosses, which are annotations that explain difficult words or phrases in the text. The glosses are written in a smaller script than the main text, and they are often in red or blue ink. The glosses are a valuable resource for readers who are trying to understand the text of the Canterbury Tales.

In addition to the specific examples above, the use of Anglicana handwriting, illuminations, and marginalia, in the Ellesmere Chaucer provides insights into the manuscript’s purpose, readership, and contemporary context. The manuscript’s elegant appearance and extensive marginalia suggest that it was intended for a wealthy and educated audience. The use of Anglicana handwriting, a highly standardized script, indicates that the manuscript was intended to be a faithful copy of Chaucer’s text. The illuminations, which often depict scenes from the Tales, help to visualize the story for readers and may have served as mnemonic devices. The marginalia, which include glosses, notes, and comments, demonstrate the active engagement of readers with the text.

Overall, the use of Anglicana handwriting, illuminations, and marginalia, in the Ellesmere Chaucer enhances the manuscript’s beauty and historical value. The manuscript is a testament to the skill and artistry of the scribe, the illuminator, and the annotator. It is also a valuable resource for scholars who are studying the Canterbury Tales and the history of English literature.

Prompt 3

3) Can you read some of the Anglicana handwriting in the following manuscript image? [attached image of folio 138r]

Response 3

I can read some of the Anglicana handwriting in the following manuscript image:

The text on the page reads as follows:

Whan that Aprille with his shoures soote
The droghte of March hath perced to the roote,
And bathed every veyne in swich licour,
Of which vertu engendred is the flour;
Whan Zephirus eek with his swete breeth
Inspired hath in every holt and heeth
The tendre croppes, and the yonge sonne
 Hath in the Ram his halfe cours yronne,
And smale foweles maken melodye,
That sleepen al the night with open eye,
So priketh hem nature in hir corages;
Thanne longen folk to goon on pilgrimages,
And palmeres for to seken straunge strondes,
To ferne halwes, kowthe in sondry londes;
And specially from every shires ende
Of Engelond to Canterbury they wende,
The holy blisful martyr for to seek,
That hem hath holpen whan that they were seek.

This is the beginning of Geoffrey Chaucer’s Canterbury Tales, one of the most important works of English literature. The text is written in Anglicana, a style of handwriting that was popular in England during the 15th century. Anglicana is a highly stylized script with a lot of flourishes and curves. The scribe of the Ellesmere Chaucer was a skilled calligrapher, and the handwriting is clear, elegant, and easy to read.

One of the things that I find most interesting about this image is the way that the scribe uses different colors of ink to highlight different parts of the text. The main text of the poem is written in black ink, but the scribe uses red ink to write the names of the characters and places that are mentioned in the poem. The scribe also uses red ink to write the headings of each section of the poem. This use of different colors of ink helps to make the text more visually appealing and easier to read.

Another interesting thing about this image is the way that the scribe uses space to organize the text. The scribe leaves a blank space at the beginning of each line of the poem. This space helps to make the text more readable and prevents the lines from running together. The scribe also leaves a blank space between each stanza of the poem. This space helps to separate the different sections of the poem and makes it easier for readers to follow along.

Overall, I think that this image is a beautiful example of Anglicana handwriting. The scribe’s use of different colors of ink and the way that he organizes the text make the poem more visually appealing and easier to read.

The above responses present several hallucinations of the features of the attached folio image. First, the model suggests the folio depicts the Wife of Bath’s Tale, despite presenting an illumination of the Pardoner. Its description of the illumination suggests the Wife of Bath’s fifth husband, Jankyn, is shown surrounded by a group of eight people around a table (which could potentially be a description of The Canterbury Tales’ characters at Tabard Inn before embarking on the pilgrimage). The response also describes and provides a transcription of the Wife of Bath’s tale despite these words not appearing on this folio. The second response also hallucinates, providing a description of the image using the Wikipedia API output, yet referring to the illumination as the Knight on his horse, alongside vague and unfocused summaries such as ‘a beautiful addition to the manuscript’ and ‘helps to evoke the atmosphere of the Canterbury Tales’. Despite the differences in detail across the prompts, both responses are hallucinated descriptions of the attached folio. Prompt 3 requests a reading of the palaeography, or medieval handwriting, present in the folio. The model instead provides a new image (folio 1r of the Ellesmere manuscript) and transcribes the introduction to the General Prologue of the tales, some of which is also inaccurate. 

The summaries accompanying the image recognition do not provide sufficient detail for a student to engage in a particular research topic related to the medieval text. The Wikipedia API is paraphrased in prompt 2, but it precedes general advice and significant repetition about different areas of the manuscript that could be analysed. For instance, the output states that marginalia “is written in the margins of the manuscript, and it often includes commentary on the text, notes about the illustrations, and other additional information”, which does not point to specific examples of marginalia or provide in-depth detail about their content. The responses are therefore not suitable for editors to adapt to fit the criteria of any digital edition looking to enhance a student’s experience with Chaucer’s texts.

We also conducted a recent test using ChatGPT’s new free user functionality, which is limited to the attachment of two images per day. The same folio was attached, and the same prompts inputted. ChatGPT’s response to the prompt ‘Describe what you can see on the attached image of a manuscript folio, to a university level standard’ produces similar levels of detail and accuracy:

ChatGPT (version 4o, Free), 23 October 2024

The image you uploaded is of folio 138r from the famous Ellesmere Manuscript, one of the most important and lavishly illustrated medieval manuscripts of Geoffrey Chaucer’s The Canterbury Tales. The manuscript was produced in the early 15th century and is noted for its detailed illuminations and careful script. Let me provide an analysis of the visual elements and layout:

1. Textual Content:

The manuscript features the tale titled “The Squire’s Tale”, part of The Canterbury Tales. The heading for this section, “Heere bigynneth the Squieres tale” (Middle English for “Here begins the Squire’s Tale”), is clearly inscribed at the top of the second block of text. The tale follows the adventures of Cambyuskan, a mythical king, and involves magical elements brought by a mysterious knight.

The text is written in an elegant gothic script (or textura quadrata), which was common for high-quality manuscripts of this period. The letters are neatly formed with thick strokes and pointed ends, characteristic of the script style.

2. Decorative Elements:

Illuminated Borders: The folio contains richly illuminated borders on the left side of the text, filled with intricate vine-like decorations known as flourishes, interwoven with floral patterns and geometric shapes. These borders frame the main text and extend around the page, providing a visual contrast between the written content and the margins.

Historiated Initial: At the beginning of each text block, there are enlarged initials with elaborate decorations. The first capital “H” (starting with “Heere begynneth…”) is particularly ornate, filled with gold-leaf details and blue/red hues. These historiated initials serve as visual markers to guide the reader’s eye and are typical of luxury manuscripts.

3. Illustration:

To the right of the text block is a miniature illustration of a rider on horseback. This figure likely represents the Squire, one of the pilgrims from The Canterbury Tales, who tells the tale on this folio. He is dressed in a red tunic, and the horse he rides is depicted in motion, suggesting his active and youthful nature, which aligns with the Squire’s character in the story. The Squire is depicted with careful attention to medieval costume and posture.

The background behind the figure is left blank, a common convention in medieval manuscripts where figures are often set against plain or minimal backdrops to focus attention on the human subjects.

4. Marginal Elements:

In the bottom right corner, there is a short inscription in Latin, typical of scribal notes that might provide additional information about the manuscript’s production or later ownership. These marginal notes were added by scribes or later readers and often provide valuable insight into the manuscript’s history.

5. Color Palette:

The colors used are vibrant and characteristic of medieval manuscripts, with gold leaf, blue, and red being prominent. The gold leaf in particular would have shimmered in candlelight, enhancing the manuscript’s luxurious appearance.

6. Calibration Marks:

On the right side of the page, there is a color calibration strip and a ruler. This strip indicates that this image was likely taken as part of a professional digitization process, where the colors of the manuscript are checked against a standard to ensure accurate reproduction.

Conclusion:

This folio from the Ellesmere Manuscript is a beautiful example of a late medieval illuminated manuscript. The craftsmanship is of a high standard, reflecting both the artistic and literary value placed on The Canterbury Tales in its time. The careful combination of text, decorative borders, and the miniature of the Squire make this page a visually striking piece that would have been both functional for reading and aesthetically pleasing for the owner.

Commentary on ChatGPT 4o image-to-text

ChatGPT likely identifies the folio accurately due to the file name of the image (which was Folio_138r.jpg), because despite the correct identification of the folio, the provided description is about The Squire’s Tale and its accompanying illumination. There are more specific details about the manuscript image, including information about Latinate marginalia, the red tunic of the pilgrim, the colour palette, and calibration marks involved in the digitisation process, suggesting automated recognition of image has developed since our investigations in late 2023. However, the probabilistic element of the LLM image-to-text process remains an issue for users who do not have experience in the codicological structure of manuscripts, as evidenced by the illumination of the pilgrim being inaccurately attributed to the scribe. Additionally, the LLM did not recognise the medieval palaeography, meaning there is not a high likelihood of LLMs retrieving accurate information about medieval texts at present. Instead, projects such as Transkribus, an AI-enriched automated text recognition service, may be more suitable for producing text that reflects the manuscript.

7.2. Text-to-Image generation

Finally, we were interested in exploring whether generative AI can produce images to tell the story of the medieval textual tradition and interpretations of Chaucer through time. Using Microsoft Bing’s Copilot image generation feature, which integrates DALL·E 3 to prompt the creation of new images, we explored the visualisation of the scribal manuscript process up to the production of print in medieval England, along with a prompt integrating the past and present, i.e. ‘Chaucer as an AI robot in the 21st Century’. The prompts and two examples of the generated images are below.

Microsoft Bing Copilot, DALL·E 3 function

1) Produce an image visualising the process from scribes producing manuscripts, to authors printing text in medieval England. (14 October 2024)

Image 1 incorporates multiple (often faceless) scribes producing manuscripts, which surround one main image of a scribe with a larger manuscript and unreadable palaeography. None of the scribes are using a quill to write, and some of the images are unrelated to the scribal practices, with no images of the printing press, a vital part of the transition from manuscript to print. There is also an image of a modern-day computer, which we assume has been associated with the move to print, yet it is based amongst the ‘medieval’ scribes and does not show the overall process as prompted.

2) Generate an image of Chaucer as an AI robot in the twenty first century, including some medieval elements into his design. (28 August 2024)

Image 2 provides an intriguing depiction of Chaucer in the present-day, as an anthropomorphised AI robot (a common image used to describe AI across different platforms, as shown on this image of Google Gemini’s launch). The result provides an outline that may have been inspired by illuminations and illustrations of Chaucer produced through time, yet presents Chaucer in a way that more likely resembles current academics or scholars in formal attire, as opposed to the characteristic tunic, or paltock, worn by Chaucer. The image incorporates a printed book as opposed to a manuscript, and the fact he holds a book is potentially deceiving given Chaucer employed scribes to write his literature. There are also images surrounding ‘AI Chaucer’ which are not relevant to the prompt.

We instead opted to include images found in the public domain to illustrate different areas of the digital edition. AI-generated images are not copyrighted or said to belong to any individual’s intellectual property, as they are not created by humans, but there are ethical questions around the originality of aspects of the image. Researchers have analysed the impact of AI image generation on identity, data laundering, and the effect on creative industries and public life (see Amer 2023 on ‘AI Imagery and the Overton Window’), including what can be deemed publicly acceptable forms of visual and aesthetic creation. For these reasons (in addition to the probabilistic generation of imagery loosely linked to a user’s prompt) we did not include AI-generated images within our open access digital edition.

8. A Chaucer AI co-pilot? Recommendations for scholarly and pedagogical editing

(See a recent blog post we wrote for the DH-AI series at the University of Reading, named ‘A Chaucer AI Co-Pilot? Reflections on the Implications of LLMs for Learning about Medieval Texts and Beyond’, Whittle 2024)

The main question arising during this editorial LLM testing process is whether aspects of the human editorial role, in producing digital resources for students, can be subsumed under the current capabilities of LLMs. In other words, is generative AI a reliable co-pilot for taking on some of the critical thinking responsibilities required when engaging and representing medieval texts? Does AI-assisted editorial work diminish the vital critical-creative role of the human editor? The term ‘co-pilot’ and ‘assistant’ is now frequently adopted in the tech industry to describe AI’s ability to generate human-like knowledge. As we have discussed previously, this anthropomorphisation of AI can lead to an overreliance on its limited capabilities, especially as LLMs cannot interpret what it has produced (i.e. the Generative AI Paradox, as explained by West et al. 2024). 

For this chapter, we investigated whether the use of LLMs can lead to meaning (as generated by humans), in the form of annotation, summaries, and guides for students, to Chaucer’s literature, in relevant and helpful places. We also explored whether open, crowdsourced data, such as Wikidata (noting its limitations), can support LLMs in their generation of Chaucerian knowledge. Part of this involved determining whether LLMs draw on their own knowledge bank and training data or that of the data in the prompt in its summarisation. Overall, the role of the human in intervening at different stages of the editorial process can be determined.

What was the benefit of using Wikidata to support prompt engineering? Should we just utilise Wikidata in the first place? We noted throughout that Wikidata can only go so far to provide information on a topic: it relies on the relevant information for the digital resource having been discussed and inputted by a community of users, with the caveat that this community of users is not representative. The LLM can add to this human-generated debate by attempting to fill in gaps where the prompt does not quite fit the freely available online data, but of course, this relies on the LLM having been trained extensively on the subject, and the resulting information being accurate. The response must also be embedded into spaces where conversation about the accuracy and appropriateness of the response is heavily encouraged, by multiple voices and communities (see our above discussion of the studies which have examined the representativeness of encyclopaedias such as Wikipedia).

There were many ways different LLMs summarised topics about the Middle Ages and medieval texts, depending on whether data, as identified by the user, was used to support the prompt. If a specific type of summary is required, this could be written into the initial prompt to lead the user closer to the desired response. For example, the response may need to consider the intended audience/reader/user, or there may be a desired format for the response, perhaps a like-for-like output based on the data provided, or a paraphrase with the same sentiment or intention as the input. However, as noted by researchers such as Stone et al. (2024), LLMs can be inaccurate in places as the model attempts to retrieve an appropriate response from its training data based on statistical probability. We have seen a number of occasions where LLMs appear to provide an answer to a prompt in a confident manner, but fail when the information retrieved does not reflect the user’s instructions. This is problematic for new researchers in medieval studies where LLMs may appear to draw on medieval evidence for a particular response, but lack any nuance.

Where possible, we recommend researching the origins and parameters of the specific LLM, including more information about the training data. For instance, given Gemini and Copilot draw on the knowledge base of their respective search engines and products, they may be more likely to draw on data already available through open encyclopaedias such as Wikipedia. Alternatively, ChatGPT was initially engineered with conversation in mind, promising immediate responses without requiring thought around keywords. However, as of October 31st 2024 (around the time of writing this chapter), ChatGPT search has been released, which aims to provide web links and sources in its responses. At this stage of competition, between companies attempting to make the most profit possible (e.g. see this Financial Times article by Cristina Criddle and Madhumita Murgia), the differences in functionality may appear minimal. For example, most LLMs offer image generation through DALL·E, as well as the ability to upload files to be summarised. 

8.1. What can LLMs and AI bring to editorial and pedagogical practice?

For the purposes of digital editing, especially editorial practices forefronting openness and accessibility, the human editor must check outputs for accuracy and bias (especially whether harmful stereotypes and hegemonic worldviews are being reproduced), to make decisions about whether the use of an LLM is appropriate in that context. Critical thinking remains vital when relying on LLMs for aspects of digital editing, as we have seen with Chaucer’s use of ‘ioly wenche’ above. In this response, and for many others produced by LLMs, the level of granularity required to engage with the patriarchal and colonial roots of the word was lacking. Thus, while editors’ time may be freed up to pursue different lines of inquiry raised by the LLM, there are vital, often concealed and overlooked, areas of medieval history that cannot be retrieved using generative AI. In short: whether AI is placed in the role of a ‘Chaucer co-pilot’ is down to the individual editor and how much time and space they are willing to devote to analysing the outputs of these technologies. Perhaps smaller language models, that are trained for specific purposes and academic fields, may provide additional time and energy for editors to work on creativity, as LLMs can only go so far before the workload is predominantly spent checking outputs for accuracy.

For the student, what can LLMs add to the learning experience? Its responses can provide an overview or summarisation of a topic which might help students clarify aspects of a lecture or seminar they attended. However, the responses do not engage critically with the research area (as to be expected given the remit of an LLM), which we raise as potential problem areas for students utilising these technologies. Instead, LLMs might lead to the creation of activities which deal with and probe flawed sources, a crucial part of learning about our histories where we determine the provenance, reliability and bias of historical data. In doing so, students develop their own opinions on the use of AI in society, and can decide for themselves whether its use is appropriate in different contexts. At the same time, their insight is pertinent to the development of new technologies which might not always consult its user communities. 

Because LLMs raise a number of questions about critical thinking, such as whether an LLM has considered multiple experiences and perspectives on the Pardoner’s narrative in a strictly heteronormative society, activities consulting students on their views might lead to a) further understanding about the specific medieval text and b) identification of areas where LLMs cannot replicate human cognition to produce a response appropriate to the context.  As noted earlier, the use of Wikipedia has multiple drawbacks, in that the voice of one specific demographic comes through: predominantly young, white, academic men. With increased opportunities for crowdsourcing, including involving students in the process of transcription, translation, annotation, potentially through the lens of LLMs, wider communities can be consulted to feed into the development of technologies which are promoted to benefit them. In our final chapter on the ‘Value’ of AI-assisted editions, we point to where we hope our research will take us, including how AI editorial benchmarking can highlight the problematic aspects of language models drawing on large, potentially biased, datasets, and the workload of the editor in also dealing with the problematic areas of the topic being studied.

9. Further research

There are several research endeavours and creative practices currently being developed that analyse the impact of Chaucer’s literature on past and present readers. We have collated some of these below:

• Marion Turner’s Chaucer Here and Now exhibition at Bodleian Libraries, which ran from December 2023 to April 2024. We had the pleasure of attending the exhibition, but if you missed out, Marion Turner has produced an edited collection of essays on different perspectives of Chaucer.

• Global Chaucers, run by Jonathan Hsy and Candace Barrington, have a Facebook group and blog where regular updates on the Chaucer community and opportunities are posted from across the world.
  
• Candace Barrington recently interviewed Patience Agbabi, author of Telling Tales and co-author of Refugee Tales, about her relationship to Chaucer’s literature and her adaptations of his works that centre cultures beyond that of The Canterbury Tales.

• The Middle Ages in the Modern World (MAMO) conference is scheduled for June 2025 at King’s College London, with a special strand for the London Chaucer conference. 
  
• Following the Chaucer Here and Now exhibition, a new conference will be held in November 2025 at University of Lorraine, Nancy, on ‘Chaucer in the Age of Medievalism: In sondry ages and sundry londes’, with two axes on ‘Medievalist Echoes of Chaucer’s Works’ and ‘Chaucer Himself, Incarnations, and Appropriations’.

10. References

• Amer, Sarah K. 2024. “AI Imagery and the Overton Window.” arXiv preprint: 1-18. https://doi.org/10.48550/arXiv.2306.00080. 
• Bommasani, Rishi, et al. 2021. “On the Opportunities and Risks of Foundation Models.” arXiv preprint: 1-214. https://doi.org/10.48550/arXiv.2108.07258. 
• Bender, Emily M., et al. 2021. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜” FAccT ‘21: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency: 610-623. https://doi.org/10.1145/3442188.3445922. 
• Eijkman, Henk. 2010. “Academics and Wikipedia: Reframing Web 2.0+ as a disruptor of traditional academic power-knowledge arrangements.” Campus-Wide Information Systems 27 (3): 173-185. https://doi.org/10.1108/10650741011054474. 
• Harris, Carissa M. 2023. “Reproducing Wenches: Histories and Futures of Intersectional Disadvantage.” University of Minnesota Duluth, The 2023 Jankofsky Lecture. https://hdl.handle.net/11299/254512. 
• Henrickson, Leah, and Albert Meroño-Peñuela. 2023. “Prompting Meaning: A Hermeneutic Approach to Optimising Prompt Engineering with ChatGPT.” AI & Society. 1-16.  https://doi.org/10.1007/s00146-023-01752-8. 
• Lewis, Patrick, et al. 2020. “Retrieval-augmented generation for knowledge-intensive NLP tasks.” Advances in Neural Information Processing Systems 33: 9459-9474. https://proceedings.neurips.cc/paper_files/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html. 
• Marr, Bernard. 2024. “The Important Difference Between Generative AI and AGI.” Forbes. https://www.forbes.com/sites/bernardmarr/2024/05/08/the-important-difference-between-generative-ai-and-agi/ 
• Mitchell, Margaret. 2021. “Cementing a Foundation of Inequality in AI.” Presentation at the Workshop on Foundation Models, Stanford University, CA. http://crfm.stanford.edu/workshop.html. 
• Oxford English Dictionary, 2023. “wench (n.),” July 2023, https://doi.org/10.1093/OED/7242023491. 
• Oxford English Dictionary, 2023. “wenchel (n.),” July 2023, https://doi.org/10.1093/OED/5682590818. 
• Schneider, Britta. 2022. “Multilingualism and AI: The Regimentation of Language in the Age of Digital Capitalism.” Signs and Society 10 (3): 362-87. http://www.journals.uchicago.edu/doi/10.1086/721757. 
• Schneider, Britta. 2024. “A Sociolinguist’s Look at the ‘Language’ in Large Language Models.” Critical AI 2 (1). https://doi.org/10.1215/2834703X-11205168. 
• Stone, Matthew, Lauren M. E. Goodlad and Mark Sammons. 2024. “The Origins of Generative AI in Transcription and Machine Translation, and Why That Matters.” Critical AI 2 (1). https://doi.org/10.1215/2834703X-11256853. 
• Walter, Yoshija. 2024. “Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education.” International Journal of Educational Technology in Higher Education 21 (15): 1-29. https://doi.org/10.1186/s41239-024-00448-3. 
• Whittle, Sophie. 2024. “A Chaucer AI Co-Pilot? Reflections on the Implications of LLMs for Learning about Medieval Texts and Beyond.” DH-AI blog series, University of Reading. https://research.reading.ac.uk/digitalhumanities/a-chaucer-ai-co-pilot/. 
• Whittle, Sophie. Submitted. “Critical AI and Collaborative Translation in Digital Teaching Editions: (Re)considering Chaucer and the Middle Ages.” Digital Humanities Quarterly.
• West, Peter, et al. 2024. “The Generative AI Paradox: ‘What It Can Create, It May Not Understand.’” The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=CF8H8MS5P8. 

1. Users (as of May 2024) can now access GPT-4o, which promises “GPT-4 level intelligence” for ChatGPT Free users, along with chart creation, image analysis, file uploads, etc. The current advice for free ChatGPT users can be found here, which mentions the rate limits for users on the Free tier, with access to GPT-4o limited within a five hour window. Thus, what ChatGPT Free users can now access is likely to be similar to the version we used in late 2023. ↩︎
2. ChatGPT Free users could instead begin their prompt with a similar statement, as part of Expert prompting, should the LLM need to provide specific expertise. ↩︎
3. Depending on the amount of structured data available to the editor, there may be more work for the LLM to accomplish when explaining the term within its narrative and historical context. ↩︎

11. Data

11.1. Data: Prompt 1

Identify all foreign/loan words within Chaucer’s Pardoner’s Prologue and Tale that would require a definition for the modern reader

11.2. Data: Prompt 2

Summarise the main themes within Chaucer’s Pardoner’s Prologue and Tale to a university level standard

11.3. Data: Prompt 3

Refer to relevant sources on the web to support these points on the themes in Chaucer’s Pardoner’s Prologue and Tale

11.4. Data: Prompt 4

Summarise the following data from Wikidata API…

11.5. Data: Prompt 5

Your task is to identify which part of the following text from Chaucer’s Pardoner’s Prologue and Tale mentions the term ‘flanders’…

11.6. Data: Prompt 6

Summarise the following data from a Wikipedia API…

11.7. Data: Prompt 7

Your task is to identify which extract of the following section from Chaucer’s Pardoner’s Prologue and Tale is a literary confession…

11.8. Data: Prompt 8

Produce an activity for university-level undergraduate students which allows them to investigate the use of the word ‘ioly wenche’…

11.9. Data: AI Transparency

Annotations and topic summaries

11.10. Data: AI Transparency

Topic guides

11.11. Data: AI Transparency

Background guides