Translating

A modern English machine translation of Chaucer’s Pardoner’s Prologue and Tale using ChatGPT

1. Introduction: Issues in translation

One of the challenges that scholars, researchers, and editors face in the process of translation, particularly from ancient to modern languages, is the decision between preserving what could be deemed the author’s ‘original’ intentions, and presenting a text to modern audiences that is readable and understandable in the target language. This complex process for translators is not immediately visible to readers when dealing with the text in the target language, yet more scholars are advocating for further transparency about decisions behind word choice, structure, considering prose versus verse, etc. The upcoming sections detail the different issues involved in translation.

First, we explore the challenges of translation as an editorial practice, including the possibility of non-equivalence, issues of accessibility and representativeness, opaque translation processes and the move towards community translation, and why students should be aware of these issues before translating Middle English. We then dive into the additional layer of Machine Learning and the challenges already associated with machine translation, looking specifically at the problems encountered when using Large Language Models (LLMs) to translate medieval text for pedagogical contexts. Finally, we summarise the usefulness of generative AI translations for students learning about Chaucer’s texts, and for immersing students in the language, sociocultural context and issues of the medieval period.¹

1.1. Non-equivalent translations

A challenge in the process of translation is the possibility of non-equivalence, when there are no suitably equivalent words available when translating from a source language to the target language. Baker (2011: 18-23) summarised the issues related to non-equivalence, including (but not limited to):

• Culture-specific concepts that are not (yet) lexicalised in the target language, making it difficult to find a one-word equivalent in the target language;
• Loan words in the source text, which might not have been introduced into the target language;
• Differences in physical [and] interpersonal perspectives, including differences in the form and meaning of words and phrases, as well as the frequency and purpose of using specific forms, between both languages.

Semantic variation and change through time is a significant factor when examining medieval language, with the view to translating it into the English of the present-day. Specific political, cultural and social concepts may have been represented with one word in Middle English, but they may require additional explanation within a modern context. For instance, in Chaucer’s Pardoner’s Prologue and Tale (henceforth, PPT), the plural noun relikes might initially appear to have an equivalent translation in the present-day (‘relics’), and therefore an equivalent meaning, yet there is a wider etymological context and semantic change process to consider, and one that is specific to the tale. The word ‘relic’ was borrowed into Middle English from Anglo-Norman (originating in Old and Middle French relique), with the following definition as per the OED:

“In the Christian Church, esp. the Roman Catholic and Orthodox churches: the physical remains (as the body or a part of it) of a saint, martyr, or other deceased holy person, or a thing believed to be sanctified by contact with him or her (such as a personal possession or piece of clothing), preserved as an object of veneration and often enshrined in some ornate receptacle. Also figurative.”

(“relic (n.),” 1.a, OED Online 2024)

In the context of the Pardoner, the relics are false and are used to deceive his audiences for capital gain. The relics are not holy and cannot be attributed to any kind of saint or martyr. Furthermore, ‘relics’ has different meanings in different contexts in the present-day. For instance, the sense attested in 1624, “a physical reminder or surviving trace of some occurrence, period, people, etc.” (meaning 4.c) is a widening of its original meaning, referring to a trace of something or someone, with no reference to religion or spirituality. Meaning 4.b from 1605 also does not reference religion: “an object vested with interest because of its age or historical associations; an artefact”. The specific meaning in its medieval context must be established for modern day readers, despite the assumed one-to-one mapping between the Middle English form and its translation. This makes the translation process, and subsequent annotation processes, highly nuanced and specific to the text, resulting in a labour-intensive set of tasks for a translator and/or editor.

1.2. Invisible translations and translators

The translation process is complex and often invisible to readers. Lawrence Venuti (2017: 1) refers to this invisibility as an “illusion of transparency”. Translators, in their pursuit of fluency for the reader in the target language, often do not disclose the different conditions under which the translator was required to make a decision:

“A translated text, whether prose or poetry, fiction or nonfiction, is judged acceptable by most publishers, reviewers, and readers when it reads fluently, when the absence of any linguistic or stylistic peculiarities makes it seem transparent, giving the appearance that it reflects the foreign writer’s personality or intention or the essential meaning of the foreign text—the appearance, in other words, that the translation is not in fact a translation, but the ‘original.’ The illusion of transparency is an effect of fluent discourse, of the translator’s effort to insure easy readability by adhering to current usage, maintaining continuous syntax, fixing a precise meaning. What is so remarkable here is that this illusory effect conceals the numerous conditions under which the translation is made, starting with the translator’s crucial intervention in the foreign text. The more fluent the translation, the more invisible the translator, and, presumably, the more visible the writer or meaning of the foreign text.”

(Venuti 2017: 1)

As Rachel Linn (2023: 75) explains, one of the issues a user might encounter within Middle English editions is the lack of transparency surrounding broad and minute decisions and translations. The translation process includes selecting a form closest to the medieval ‘original’, while ensuring a parsable form or sentence for the modern reader.

Annotations of a word or phrase might enlighten the reader, or user of an edition, as to any meaning differences across contexts, or difficult decisions made in the process of translation. These annotations might incorporate: whether the word is a loan from another language, whether there have been any semantic changes between the forms, or other information about the word, such as if it is a named entity and relates to a real-world person or place. These annotations are particularly helpful for students or anyone who has not engaged with the text previously, who might require more background knowledge before delving further into specific topics or surrounding context.

1.3. Accessibility and representativeness

Linn (2023: 75) refers to two “troubling” aspects of modern-day editions. First, the analytical work, including the accumulation of decisions made by transcribers and editors, is not often available to readers. Second, the decision to select a word that closely matches the manuscript of the text is often considered “the best and most representative of the work”, despite the numerous changes that might have occurred since the word was adopted into the language. The possibility of multiple interpretations of the word also affects representation, not only in terms of what the author intended, but also for readers of different communities and heritages who may not closely relate to a ‘standard’ English translation (thus, not increasing access to the medieval text).

Student activities are vital for maintaining the openness and visibility of the translation process. The aim of our translation activity is to invite students (and their tutors) to contribute to multiple possible interpretations of a medieval word or phrase via a Padlet discussion board, where they are also encouraged to take a critical approach to AI-generated translations. As Linn shows, there are a number of ways translators can translate, that incorporate different voices. In her translations of Middle English lyrical poems, Linn has decided not to decide, but rather, adopt multiple processes:

“I am working to make the processing (and lack of an identifiable original version) of manuscript texts more visible to the modern reader, to capture a wilder range of possibilities when it comes to word choice and meaning (rather than allowing it to appear that these choices are uninterpreted), and to translate multiple poetic features rather than focusing simply on definitional problems—acknowledging that in some cases it might be most meaningful to recreate rhyme scheme, for example, even if that requires a significant shift in content. I have decided to not decide. The point that I am trying to make in my own translation work is that a certain degree of float—not allowing one version to simply stand solidly in place, even if this makes the reader feel adrift—is a good thing.”

(Linn 2023: 75)

The translation activity and discussion board arise from ideas in our focus groups and interviews about community or crowdsourced translations, where experienced or aspiring translators can adapt or retell aspects of the tale from their lived experience. One of the first community translations to exist – Beowulf By All, by Jean Abbott, Elaine Treharne, and Mateusz Fafinski (2021) – provides a space for anyone to submit “non-hierarchical, radical contribution[s] to a more representative Old English Studies”, to counter the “elitist, exclusionary, misogynistic, often racist, and anti-feminist” agenda of academia (Treharne 2021: 3), where communities’ histories of marginalisation are further marginalised through erasure from the record. The community translation invited people to provide a translation of Beowulf that might represent their own communities, languages, dialects, and experiences. 

Like Beowulf By All, we encourage students to post their own translations of PPT. Part of this process also involves critiquing the translations produced by editors and AI, including whether students would edit any of the translations further, or whether they would incorporate different analytical processes to guide readers in their interpretation of the word within its historical context. These endeavours also promote the use of multiple interpretations based on students’ own understanding of the tale, or experience within their respective communities. The Pardoner has been interpreted as a queer figure in medieval society, and students may wish to anonymously comment on this experience using their own understanding of queer theory (similar to students at UCLA and their Chaucer Today resource).

In the main text, we purposefully left some of the translations produced by AI unedited, so students could find routes into their analysis of the translated text. Overall, the main aim is to increase students’ knowledge and understanding of the text, improve their ability to interpret medieval texts, and encourage them to critically engage with processes of AI, all while making visible the processes of translation in a way that is active and collaborative.

1.4. Possible benefits and limitations of AI and machine translation

Generative AI models have received a mixed response following their introduction to the public, particularly in higher education. Since the launch of ChatGPT in late 2022, there has been increased wariness surrounding plagiarism at UK universities, with concerns over research integrity and methodological rigour. However, more scholars are realising the potential to reverse such narratives by asking students to critique ChatGPT outputs. Wierdak and Sheridan (2023) label ChatGPT as “friend not foe”—students are still required to work alongside AI to “meticulously defin[e] their research question, craft precise prompts, critically assess generated content and integrat[e] it with their original thoughts.” In the case of AI-generated translation, students might utilise the academic literacies developed in their degree programme (as linked to some of the discussion points in the previous sections) to critically analyse the translation choices arising from different sources.

A student in one of our focus groups (conducted in the latter half of 2023) identified that generative AI models might offer “learning in reverse engineering”—a way to interrogate, ‘mark’, and find the faults in what AI creates, if they are initially made aware of its flaws, lack of specific task-based training in their field of study, and its detachment from human cognition (ID: Student 3; Fourth year undergraduate, English Literature). In addition, two students commented on the benefits of student critique in AI translation contexts. One confirmed the possibility of improving students’ critical thinking skills via the analysis of AI-generated translation:

“I definitely could see, especially a translation environment based on what everyone else has said, how that could be really illuminating for the student […]. I think it would foster a lot of critical thinking skills about this type of text.”

ID: Student 13; Senior, undergraduate, English, US

Another student mentioned that their teacher could see the benefit of using generative AI responsibly, critically, and in an experimental manner, ensuring that they are transparent about their use and include a statement on their assignment:

“… one of our top academics at [redacted] has told our seminar that he’s okay with us using ChatGPT as long as we tell him that we’ve used it, which I thought was really interesting. So you have to admit that you used AI-assisted writing […]. I think he’s really curious about the implications of AI and how that’s going to change how we are able to convey information with one another, how we process information […]. I think it’s really interesting that some professors aren’t necessarily shutting down the use of AI-assisted writing, but are in fact encouraging it on an experimental basis.”

ID: Student 12; Second year, Masters, English Literature, US

It appears that an exercise in assessing the AI translation of Chaucer’s PPT, given the well-known flaws of LLMs (e.g. see Hanlon’s 2024 paper on LLM outputs as ‘fictional’), could offer more insight into producing translations and critical commentaries of specific language choices, as well as a lesson on whether the use of AI could ever be considered responsible and ethical. Given this latter point, the use of AI in university classrooms should be approached with caution, particularly with regard to academic authority, integrity and potential for harm on members of society. Students must also consult the guidance of their teachers to determine whether generative AI use is permitted for study.²

There are also a number of issues to consider for machine translation more generally. Koehn and Knowles (2017) explored several challenges for neural machine translation (NMT). In particular, they noted the poor performance of NMTs when trained outside of the domain of the language of study (in this case, the source language of Middle English), especially when translating words of low frequency, which resulted from the small vocabularies on which the NMT was trained. The historical semantic and pragmatic changes over the course of the English language may therefore prevent NMTs from selecting the most appropriate word in the present-day, instead looking for literal and/or equivalent translations regardless of changes in meaning over time. NMTs also perform poorly when translating long sentences of 60 tokens or higher. Of course, these issues are specific to NMT, as opposed to translation by a generative AI model, where the training data is less transparent and unpredictable. While sentence length may not be an issue when AI generates translations of Middle English verse, there remain challenges that accompany verse – rhyme and metre – which cannot be preserved in modern English translations due to the syntactic, phonological and prosodic changes that occurred over the history of English. Even though these challenges were identified in the specific environment of neural machine translation, the same issues must be considered for translation by generative AI chatbots. This is because the priorities of companies such as OpenAI are to produce ‘conversational’ user interfaces rather than act as cobots for more niche tasks.

There are therefore two necessary endeavours for the creation of digital pedagogical editions which incorporate AI translation in the current age of technology. First, there should be an assessment of whether such technologies can produce a resource for students that is interactive, critical and engaging, while preserving some of the textual practices of the past. This process must therefore include an exploration of semantic and conceptual meaning in the medieval period (through the lens of translation), how this has changed, and how texts were produced, disseminated and received (thus, directing students to resources on book history). Second, editors should explore whether students can feasibly develop their skills in interrogating, and by extension, offering improvements for, the translation process, to increase the accessibility of medieval text interpretation. As noted above, there are also challenges specific to translation, without the additional layer of AI. How can editions make an invisible process more transparent, alongside the use of AI? How might students contribute to the discussion of (AI-generated) translation choices at the word and sentence level? 

In the following sections, we analyse the findings of the generative AI approach to machine translation, in producing an accessible and engaging modern English translation of The Pardoner’s Prologue and Tale. Here, we investigate examples from the AI decision-making process and the level of editorial intervention required.

2. Translation prompts for ChatGPT

In this analysis, we explore the translations made by OpenAI’s ChatGPT, on the level of the word, sentence, and verse (i.e. issues of prosody, such as rhyme and metre). In total, 644 lines of verse were inputted into ChatGPT, folio-by-folio. Diplomatic transcriptions came from the Ellesmere Manuscript, which were produced by researchers on the Canterbury Tales Project and The Norman Blake Editions of The Canterbury Tales at the Digital Humanities Institute.

We tested two different types of prompts for ChatGPT version gpt-3.5-turbo (via OpenAI’s API, with a JSON input and output for storing the data across individual lines of the text). Version 3.5 was the newest update for ChatGPT at the time (in late 2023) and was made freely available to the public via their main interface.

One of our prompts requested a translation from Middle English (ME) into a modern British English (ModE) translation, and the other requested a translation with additional consideration of medieval structure, rhyme and metre. This latter prompt was added to see whether ChatGPT would preserve aspects of the original text to show a more ‘traditional’ modern English version, potentially reflecting earlier structures, as opposed to forming an entirely new structure and selecting word choices more appropriate to modern contexts. The Temperature of the prompt was also set to ‘0.1’ within ChatGPT to avoid the randomness associated with a more ‘natural’ and ‘flowing’ conversational AI. OpenAI states that different controls in Temperature lead to different types of output, with a setting of 0.2 or lower making the output more ‘focused’ and ‘deterministic’. Instead, our main requirement for the outputs was to ensure a consistent and replicable response from ChatGPT, if at all possible. The prompts are provided below in (1-3), and examples of the differences in Table 1.

(1) Baseline prompt for translation:

Your task is to process The Pardoner's Tale from Chaucer's The Canterbury Tales derived from the Ellesmere Manuscript. This is in the original language of Middle English. \
The text has been split into lines and is provided in JSON format, delimited by triple backticks. \
for each line: \
[Part of prompt focusing on the specific type of translation]

(2) Prompt with preservation of the original rhyming structure and metre:

Create a modern British English translation of the text, keep the rhyming structure and metre of the original across the lines. \
Provide output in JSON format with the following keys: ln, en_gb_rhyme. \
Only process the supplied lines that have a 'ln' key. \

(3) Prompt without preservation of the original rhyming structure and metre:

Create a modern British English translation of the text. \
Provide output in JSON format with the following keys: ln, en_gb_modern. \
Only process the supplied lines that have a 'ln' key. \

The way in which these prompts were crafted were a result of specific research questions, namely, whether it is possible to balance the accessibility of a modern English translation while maintaining some aspects of medieval poetry. We felt a closer replication of medieval poetry was necessary for immersing students in the textual tradition of the period, as they begin to learn Middle English language and grammar. We identified two possible preferences for editors, and acknowledge caveats to these prompts below:

a. Prompt en_gb_rhyme: A modern translation that is closer to the structure and metre of the original Middle English verse. Equivalent rhymes might not be available given the phonological changes that occurred from the late Middle Ages (e.g. the Great Vowel Shift). Partial edits might be required by a human, especially where the structure of the translation cannot be parsed by a modern user. Equivalent word-for-word translations might need to be supplemented with annotations to explain any semantic change that occurred between the past and present.

b. Prompt en_gb_modern: A modern translation that is closer to modern day structure, and further away from medieval metre (e.g. iambic pentameter). This type of translation might lead to more edits by a human, given ChatGPT would have more freedom and flexibility to choose between different word forms, including meanings that may not be appropriate to a modern context. Yet, if an appropriate word is selected by ChatGPT or the editor, fewer annotations might be required, since the word might be more self-explanatory to a modern reader.

Table 1 highlights the outcomes of these different choices, and the final edits made. The example is from lines 621-625 towards the end of the tale. Here, the Host challenges the Pardoner following the remarks he makes about the Host being one of the most sinful pilgrims, prompting the Pardoner to sell his false relics to him.

Line number	Middle English transcription (Ellesmere Ms.)	Prompt A (en_gb_rhyme)	Prompt B (en_gb_modern)	Final choice made by a human editor3
621	Nay nay quod he thanne haue I cristes curs	Nay nay, said he, then have I Christ’s curse	“No, no,” he said, “then I have Christ’s curse.”	“Nay, nay!” he said, “then I will have Christ’s curse!
622	Lat be quod he it shal nat be so theech	Let be, said he, it shall not be so, thee	“Let it be,” said he, “it shall not be so, thou fool.”	Let it be,” he said, “it shall not be, so I may thrive!
623	Thou woldest make me kisse thyn olde breech	Thou wouldst make me kiss thine old breech	“You would make me kiss your old breeches.”	Thou would make me kiss thine old underpants,
624	And swere it were a relyk of a Seint	And swear it were a relic of a saint	And swear it was a relic of a saint.	And swear it were a relic of a saint,
625	Though it were wt thy fundement depeint	Though it were with thy fundament depaint	Though it were painted on your fundament.	Though it were stained with thy fundament!

We have highlighted differences between each of the ChatGPT outputs and the choices that were made by the editor. While we settled on Prompt A, since more flexibility could be adopted on the part of the editor (including subsequent users who may wish to contribute to this process of critical AI), outputs from both prompts were considered in the final edits.

There were also occasions when a modern prosodic structure was required if it meant the user would be able to parse the translation more easily. Verbal structures such as haue I on line 621 were changed to a non-inverted structure of ‘I have’, and syllables were added where required for the modern reader (e.g. lat be to ‘let it be’ on line 622).

In addition, errors in translation were corrected, such as line 622’s ‘thee’ (prompt A output) and ‘thou fool’ (prompt B output), from the ME word theech, meaning “to flourish, thrive or prosper”. Translations were also edited depending on factors such as archaism, semantic change, and other contextual factors. The translation of breech (line 623) was changed from ‘breeches’ to ‘underpants’, as this sense of ‘breech’ has declined in frequency from its first attestation in Old English, compared to the more modern ‘underpants’. Additionally, while the word depeint (line 625) has multiple possible senses, the use of ‘stained’ over ‘painted’ appears more appropriate in this context. Here, the Host is implying the Pardoner’s false relics can be likened to stained underwear, given their lack of holiness—a reaction following the Pardoner’s preposterous proposition. The word fundement is maintained in the translation for the purposes of metre (i.e. the syllabic structure maintains a similar rhythm to the original ME verse) and would be accompanied by an annotation to explain the meaning of the term in its medieval context (see meaning 7a of the MED). 

Archaic words such as ‘Nay, Nay!’ and ‘thine’ are kept in the translation from the original to preserve some of the medieval context. Students might also consider the pragmatics of politeness, different greeting styles, rejections, etc. In linguistics courses, students may already be learning about Chaucer’s use of ‘th-’ pronominal forms, potentially due to his connections to Anglo-Norman, a choice that soon became archaic in the late Middle Ages. The use of ‘nay’ is also archaic, regional and humorous according to the OED. These original forms in the modern English translation are essential for immersing students in the humour of the tale, but also for highlighting the differences between pragmatic strategies in the past and present, so they must be accompanied by annotations or supplementary information.

3. Word choice and semantic change

Some challenges concern the selection of specific modern English words by ChatGPT. These challenges are in relation to hallucinations made by genAI, for example, minor or major variations between what was intended by the use of the word in its context and the output, as well as issues relating to semantic change. We have provided three examples of these issues below, with context surrounding why the output is not suitable for the modern reader, and what the solution might be for such issues—e.g. the use of a human editor to sift through outputs, or annotation occurring alongside the Middle English word or modern translation. We have also outlined the differences between the ‘rhyme’ (b) translation prompt (asking ChatGPT to maintain some of the rhyming structure of the original), and the ‘modern’ (c) prompt (without these constraints), in case of any variation in word choice. The final line refers to any changes made by the editor to rectify these challenges (d).

(4) Error in translation linked to (non)-equivalence and training data

a. ME (Ellesmere): That shewe I first my body to warente
b. ModE (GPT, rhyme prompt): That I show first to warrant my body
c. ModE (GPT, modern prompt): I first show my body to guarantee
d. ModE (Editor): That I show first, to protect my body

(Folio 136v, Line 12)

Example (4) represents one of the errors made by ChatGPT in the process of translation, with the use of the verb ‘to warrant’ for the Middle English verb to warente. In this case, the genAI chatbot has directly substituted the verb warente for what it perceives to be a modern-day equivalent, the sense which describes a “guarantee as true” or “to give (a person) assurance of a fact”, attested from c.1400. In fact, the ‘modern’ GPT prompt (4c) uses the verb “guarantee”. The intended sense of warente, “to keep safe from danger, to protect,” is now obsolete, and the MED confirms that the verb in its Middle English sense means “to act as protector, afford protection; protect (sb. or sth., oneself), shield; also, guarantee the safety of (sth.), guard; save (sb. from death), rescue.” Thus, the editor settled on ‘protect’ as an appropriate translation, with a possible solution being an annotation on the Middle English word to highlight the etymology and semantic shift described here.

This particular challenge makes evident that GPT is not trained in data and evidence arising from a long time ago, especially not the medieval period. There is no authority and insight about the intended meaning of passages in medieval texts, as well as the shifts in meaning through time. The GenAI bot is not able to select between multiple meanings, or reflect these changes, thus opting for the most ‘literal’ translation based on the form of the word.

(5) Error in translation linked to (non)-equivalence and editorial preference

a. ME (Ellesmere): Haue heer my trouthe as thou art his espye
b. ModE (GPT, rhyme prompt): Have here my truth, as thou art his spy
c. ModE (GPT, modern prompt): Have here my truth as you are his spy
d. ModE (Editor): Have here my truth, as thou art his witness

(Folio 141r, Line 430)

The example in (5) highlights interesting possible translations of the noun espye, which, in the medieval period, meant “stealthy investigation, observation, or watching; scouting, spying; also, information obtained by scouting.” GPT makes a literal translation, with both the ‘rhyme’ and ‘modern’ prompt (5b-c), selecting ‘spy’. Here, the use of espye is used by one of the rioters in the tale after coming across the old man in the forest. They ask whether the Old Man has seen Death, as they are looking to avenge their friend who was killed by Death. 

As a modern reader, the use of the translation ‘spy’ has two possibilities, meaning there is potentially some ambiguity at play, made more challenging by the use of the possessive pronoun ‘his’. Does the Old Man act on behalf of Death, or is Death unknowingly being spied upon by the Old Man? Since the Old Man is acting on his own accord (in fact, he does not come into contact with Death in the tale itself), the editor resolves this ambiguity by using the word ‘witness’. In Middle English, the noun ‘witness’ meant “knowledge, understanding, wisdom” (now obsolete), which might encompass the Old Man’s role in the tale, whereas now the word reflects a ‘presence’ (“one who is or was present and is able to testify from personal observation”). This presence also accurately reflects the role of the Old Man in this scenario in observing Death. The issues here come down to personal preference, in particular, whether the translator/editor believes there to be ambiguity for modern audiences and whether there may be some disruption to the flow of reading. Again, GPT is opting for the most literal translation, no matter the nuance present in the medieval text—the model does not take into account the discursive context about the Old Man’s character and his role.

(6) Translations where further annotation is required to reflect semantic change

a. ME (Ellesmere): For lewed peple louen tales olde
b. ModE (GPT, rhyme prompt): For lewd people love tales old
c. ModE (GPT, modern prompt): For uneducated people love old stories
d. ModE (Editor): For unlearned people love old stories

(Folio 137v, Line 111)

Example (6) provides translations of the noun phrase lewed peple, in relation to the Pardoner telling stories to his audiences and the tales they prefer. The rhyme prompt (6b) produced a literal translation, ‘lewd people’, which, given changes in meaning to the word ‘lewd’ over time, is potentially not an appropriate translation for modern readers and their current understanding of the word, without further annotation or explanation. The sense of the earliest attestation of the word ‘lewd’ meant “of a person: not in holy orders, not clerical; lay”, which is used to describe the Pardoner’s audiences, and is now an obsolete meaning. In the present, the meaning ascribed to ‘lewd’ is “lascivious, lecherous; (also) involving or relating to sexual activity”, and is the only surviving sense. By using this same form in the translation, the user might ascribe an incorrect meaning to Middle English lewed. The prompt in (6c) uses the word ‘uneducated’, and the editor settles on ‘unlearned’. The latter choice is attested much earlier than the former, with ‘unlearned’ in c.1384 meaning “lacking knowledge, expertise, or experience with regard to a particular subject or skill”, and may therefore be closest in meaning to “not clerical, or lay.”. This is likely a matter of editorial preference, but since education was not available to the majority of the population in the time of Chaucer, it makes sense to describe the ‘lack’ of knowledge afforded to them. The phrase ‘unlearned people’ is annotated in the edition to highlight the semantic change described here, and the edition builds in activities to explore other possibilities for this word in the translation process (e.g. see our ‘Learning’ chapter). 

It is evident that ChatGPT is not able to provide the same level of critical thinking and authority over the selection between different translations and their efficacy for modern readers. In particular, students require additional notes to explain some of the changes that have occurred between the medieval period and the present, depending on their module of instruction. Even with explicit prompting about what is required of the translation output, the envisioned ‘productivity’ of automated editorial processes through LLMs may fall short. The model may also struggle to produce all the necessary information, because it does not register shifts in language and sociocultural climates across the two time periods. Even with careful step-by-step instructions for the model, it is up to the human editor to select between or merge different outputs.

While genAI is certainly not authoritative in word-for-word translation, it does open up conversation about the flexibility of the process, and what does or does not constitute a suitable translation for a modern audience. In doing so, it provides an opportunity for further investigation and collaboration on issues of access to academic and scholarly material on the Middle English language (see Section 5).

4. Issues with replicating structure, rhyme and metre

The ‘rhyme’ prompt was used to attempt to preserve some of the rhyme and metre of the ‘original’ text, i.e. the pairs of rhymes and the five syllable, five stress metre of each line, also known as iambic pentameter (although it is unlikely Chaucer followed this rule strictly across each line of verse). This partial preservation of the structure and prosody of the text was maintained in order to engage students in the form of medieval poetry while also improving access through translation. The below examples investigate this prompt further in terms of phonological, semantic and syntactic change, comparing the outputs with the ‘modern’ prompt which was not restricted by medieval structure and metre. The examples provide a window into the choices that the human editor must make, in terms of weighing up medieval structure and prosody with the readability of medieval texts. These are issues that genAI models struggle to contend with, without the medieval training data and capacity for creativity and critical thinking.

(7) Sound change

a. ME (Ellesmere): That no man be so boold ne preest ne clerk / Me to destourbe of Cristes hooly werk
b. ModE (GPT, rhyme prompt): That no man be so bold nor priest nor clerk / To disturb me of Christ’s holy work
c. ModE (GPT, modern prompt): That no man be so bold nor priest nor clerk / To disturb me from Christ’s holy work
d. ModE (Editor): That no man be so bold, not a priest nor a clerk / To disturb me from Christ’s holy work

(Folio 136v, Lines 13-14)

The first challenge relates to the differences in pronunciation between Middle and Modern English. From noting their spelling, the words clerk and werk in (7a) were likely to have been pronounced with the vowel [ɛ:] (or similar), which is close to the vowel in the word ‘there’ in the present-day. Across all lines of translation in (7), the structure of the sentence is similar to the original, but the rhyme can no longer be preserved in modern English due to the Great Vowel Shift having occurred between c. 1400 and 1700. This means that vowels shifted in their quality, and ‘clerk’ and ‘work’ are now pronounced with the vowels [ɑ:] and [ɜ:] respectively. Caveats surrounding the lack of rhyme at the end of some lines should be provided for students if they are expecting something similar to Middle English verse.

(8) Semantic change

a. ME (Ellesmere): Agayns an oold man hoor vpon his heed / Ye sholde arise wherfore I yeue yow reed
b. ModE (GPT, rhyme prompt): Against an old man, hoar upon his head / You should arise, wherefore I give you read
c. ModE (GPT, modern prompt): Against an old man, gray-haired upon his head / You should arise, therefore I give you advice
d. ModE (Editor): Against an old man, hoar upon his head / You should arise wherefore I give you advice

(Folio 141r, Lines 418-19)

In the case of (8), the Middle English words heed and reed rhyme with the vowel [i:] (like the modern word ‘see’), yet the change in form and meaning of the word reed means that the lines of verse can no longer rhyme in modern English. In ME, reed meant “counsel or advice given by one person to another”, and therefore the translations with the word ‘advice’ (7b-c) are most accurate. The attempt to maintain the rhyme in (7b) is therefore not accurate in this case. While the model did follow instructions to maintain some of the rhyming structure of the original, it did not acknowledge that the translation to ‘read’ is not appropriate in this case. Perhaps further prompting is required which ensures the rhyming structure is only maintained in cases where it makes sense in the modern context. However, this nuanced prompting slows down the editorial process, counteracting any original intentions that genAI will speed up translation.

(9) Structural change

a. ME (Ellesmere): Hadde filled with wyn hise grete botels thre / To hise felawes agayn repaireth he
b. ModE (GPT, rhyme prompt): He had filled with wine his great bottles three / To his fellows again repairs he
c. ModE (GPT, modern prompt): He had filled his three large bottles with wine / He returns to his companions again
d. ModE (Editor): He had filled with wine his three great bottles / To his fellows again he returns

(Folio 142v, Lines 552-53)

The final challenge in maintaining some of the prosodic aspects of medieval verse are the inevitable structural differences between the late Middle Ages and the present. In (9a), the pairs thre and he rhyme at the end of the lines, both pronounced with the vowel [e:] (there is no modern English equivalent, but the vowel is close to and a longer version of the modern word ‘bed’). The ‘rhyme’ prompt in (9b) attempts to preserve some of the medieval structure, however, the phrases ‘great bottles three’ and ‘repairs [i.e. returns] he’ (representing ‘subject-verb inversion’) are ungrammatical for modern readers. In modern English, the numerical adjective ‘three’ must appear before the phrase it is modifying, ‘great bottles’, and the subject pronoun ‘he’ must appear before the verb ‘returns’. The ‘modern’ prompt recognises this difference, and produces the grammatical version. 

The editor settled on ‘He had filled with wine his three great bottles / To his fellows again he returns’; the only differences being the aforementioned structural changes and the addition of a subject in ‘he’ before ‘had’ in the first line. The use of the prepositional phrase, ‘to his fellows’, at the beginning of line 553 is preserved to maintain some of the medieval structure. This kind of structural preservation is appropriate because modern readers can parse topicalization, a process where the main thrust of the sentence is placed at the beginning. For instance, contemporary films such as Star Wars uses this type of syntax, with Yoda placing topics to the beginning of the sentence: e.g. ‘to be Jedi is to face the truth and choose’. Overall, there is a change to the general five stress, five syllable line that occurred in medieval verse, but this tendency was not always adhered to by Chaucer anyway, meaning it is not an issue worth considering in-depth for the modern translation.

There is therefore a balance to be struck between preserving the original rhyme, structure, and metre of the ME sentence, to maintain the intended flow of poetry by the scribe, and ensuring the text is readable for those encountering Chaucer for the first time. Some of the structures might be understandable to modern-day readers, yet the meaning is likely masked by the overuse of archaic structures throughout the entire tale. The editor therefore needs to consider a number of issues related to grammaticality, rhyme/metre, structure, and flow, when reviewing translation outputs from genAI.

5. Conclusions on the usefulness of machine translation

As noted throughout this chapter, there are a number of challenges, for both the editor and student, in using generative AI to create and study translations of Chaucer’s text. These challenges relate to making decisions about highlighting the medieval context through language, to familiarise students with some of the features and constraints on medieval verse, and making the translated text understandable, interactive and engaging. The challenges arise because of the differences in sound, prosody and (rhyming) structure, all of which have changed between the medieval period and present-day, and continue to evolve in the English language today. The differences are made more complex when using genAI to interpret them and produce a translation in line with modern expectations for learning about Chaucer’s texts, because the model does not have the human capacity or training on medieval data for making authoritative decisions. On top of this, there is no transparency as to where the data on medieval translation is coming from. It may be that genAI models are drawing on freely available translation data on the web (e.g. the regularly used Harvard translations of Chaucer’s texts), in which case the model fails to give due credit to the original authors. Or, the model is hallucinating based on what it knows about current language, as we have seen with translation errors where the exact same form as the medieval word is used, yet there is now a completely different meaning, making the translation unsuitable for the modern audience. 

The main question we are left with, is whether there is any value in using these AI translations of Chaucer for contemporary students. Do they adequately convey the issues students are interested in learning about in the present day? Do they engage students in medieval studies? Is it even worth using AI translations if it is just as quick (and more authoritative) for editors to translate themselves? 

One of the intriguing aspects of genAI translation is the discussions the use of the model opens up. Going back to crowdsourced and collaborative translations such as Beowulf By All, genAI might present an opportunity to discuss the many choices behind translation, and which choice is the most accurate or reflective of the individual and the wider community (whether that be a pedagogical community, or related to a particular culture). Thus, AI editions are not just for ‘productivity’ and ‘automation’ purposes, which, as we have seen, may be hindered in the process of using these models.

Instead, what arises from AI-generated translation process is the human ability to identify key problems in society (past and present) and find solutions going forward. The Pardoner’s Prologue and Tale is a text questioning what constituted moral corruption and decay in the medieval period, and its modern translations highlight the societal similarities and differences between both periods. Students can explore how identities are formed and maintained through society and literature, with more flexibility and openness when it comes to Chaucer’s satirical writing. Collaborative translation both informs students about the (in)visibility and (un)representativeness of the process. In cases where students have had more time to engage in Chaucer studies, this cooperative process centres the expertise they have gained through their course, allowing them to draw on their own experiences from the classroom, assignments and the community around them. In particular, we explore an example of gendered and racialised Middle English words, a type of analysis we encourage you to consider for your classroom environments, especially as we enter a period of uncertainty about what generative AI can (and should) do.

6. References

• Abbott, Jean, Elaine Treharne & Mateusz Fafinski. 2021. Beowulf by All: Translation and Workbook. Leeds: Arc Humanities Press.
• Baker, Mona. 2011. In Other Words: A Coursebook on Translation, 2nd ed. Oxon: Routledge.
• Hanlon, Aaron R. “LLM Outputs are Fictions.” Critical AI 2 (1). https://doi.org/10.1215/2834703X-11205210.
• Koehn, Philipp & Rebecca Knowles. 2017. “Six Challenges for Neural Machine Translation.” In Proceedings of the First Workshop on Neural Machine Translation, ed. by Luong, Thang, Alexandra Birch, Graham Neubig & Andrew Finch, 28-39. Vancouver: Association for Computational Linguistics.
• Linn, Rachel. 2023. “False Fiends: Middle English Lyric Poems in Translation.” Subtropics 33: 73-90, 129.
• Oxford English Dictionary. 2024. “relic (n.).” Oxford UP. https://doi.org/10.1093/OED/5133618035 [accessed 2 July 2024]. 
• Treharne, Elaine. 2021. “Introduction.” In Beowulf by All: Translation and Workbook, ed. by Abbott, Jean, Elaine Treharne & Mateusz Fafinski, 1-4. Leeds: Arc Humanities Press.
• Venuti, Lawrence. 2017. The Translator’s Invisibility: A History of Translation London: Routledge. 
• Wierdak, Nathalie & Lynnaire Sheridan. 2023. “ChatGPT could have an upside for universities – helping bust ‘contract cheating; by ghostwriters.” The Conversation. https://theconversation.com/chatgpt-could-have-an-upside-for-universities-helping-bust-contract-cheating-by-ghostwriters-205004 [accessed 2 July 2024].

7. Data

Data: ChatGPT translation output with editorial changes

Number of edits made to the AI translation, per line of verse: 2.8
Edits made, over the number of words in the AI translation, as a percentage: 36.1%

1. Some of the issues discussed in this chapter is also included in the Translation Guide on our edition. ↩︎
2. We have included some resources for students on the edition itself, regarding generative AI use in higher education, but they must also consult resources within their own department, and speak with their tutors to understand whether its use is permitted on their course. ↩︎
3. Punctuation was modelled on Benson’s (2008) edition of The Canterbury Tales in The Riverside Chaucer. ↩︎