Our work has shown that many resources ignore the subtlety of research and tend not to accommodate a wider range of research methodologies and practices.
One of the major innovations has been to bring conceptual thinking into the minutiae of using a resource; seeing the resource as part of a search pattern and keeping search managed and coherent in one place, not just improving search but designing for actual search practice.
The following diagram illustrates a background structure of search, detailing the main areas of user requirements we have observed over the course of the project and particularly during the Design Groups. It shows that users value simple and intuitive presentation: a resource that provides the “raw” data but at the same time does not just throw everything at you in an incoherent way. Results should be trustworthy, citable and presented in such a way that you can see how they have been ordered and then reorder and refine according to your own preferences and ways of working. Users are keen to save search results lists and individual result pages and add comments and labels to these to keep for their own purposes and/or to share selectively with a wider, affiliated research community.
1. Key Findings from Phase One
1.1. Landscape Survey
Some of the key suggestions from our survey are reflected in the following open response quotations:
- “more intuitive advanced search”
- “more systematic”
- “the ability to be able to take the metadata from online databases and structure it in different ways. To be able to make sub-sets within a search. So that a sub-set could be compared to a larger set of search results clearly”
- “More methodical record keeping; maybe software to do this?”
- “Better strategies for key word searches, resources for beginning searches”
- “A good online reference/search result manager – one that can save results…... Being able to record, sort, comment and save searches and sources, transforms the experience”
The survey results yielded three key themes:
1. Context, Quality and Validity
There was a concern that the navigation of search using search engines or search functions within specific resources needs to be transparent and intuitive. The key message was that researchers want search to be flexible in that it enables them to follow lines of inquiry and that these lines of inquiry are able to be saved so that the search methodology can be validated. Consistency and quality are key factors in search. When claims are made based on the results of a search, researchers need to be able to see the structuring of the search so that they can trust it and understand how it underpins the authority of the claims.
2. Online Versus Offline Practices
Most of the researchers worked from online to offline in search, with many saying that they conduct a search using a search engine or the search function within a particular resource to identify resources and then verify the reliability of results with archival or library work. There was a general concern that online sources may not provide the full picture or may be flawed in some way. For example, online sources were often felt to be incomplete or unreliable compared to printed sources which are checked by researchers ensure reliability and to fill in any missing gaps. However, the accessibility of online sources and search was highly valued by researchers and the community now recognises that most scholars work online to get access to digitised versions of primary sources. There were some concerns about the use of keywords for searching catalogues, in that keyword classifications may have been poorly assigned and as a result return many irrelevant results and miss relevant ones. There was a need to balance the openness of online search with a targeted approach. Researchers felt it was important that search might uncover a wide range of relevant and important insights. However, there still needs to be a systematic framework for managing results. The ability to conduct full text search on resources appears to be increasingly needed and expected by all disciplines.
There is a need to improve awareness of what tools and resources are available and how to access, navigate and use them effectively. There was also a strong sense that the research community could unite more to share knowledge about new and existing resources, research practices and methodologies within and across disciplines.
1.2. Focus Groups
The findings from the landscape survey and focus groups have echoed one another and show that researchers engage with search in different ways. Three key themes were distilled from analysis of the focus groups:
This includes: (1) transparency of content and the extent of the resource; (2) transparency of resource definition and contextualisation; and (3) transparency of the expectations and limitations of a resource and search.
This includes recognising the advantages of online research; but also that the pathways and layering of search are well structured, flexible and traceable. The resource needs to be open to browse at different times in a search process and it needs to support a more systematic search methodology. There is a need for access to be inclusive to a wide range of researchers and disciplines.
3. Profile and Recognition
Researchers have a strong sense that they are participating in a wider research community. Researchers are happy to share their knowledge about sources within the framework of online resources but want to do so in a way that leaves traces of their contributions and acknowledges that contribution. There is a common ideal that there should be appropriate mechanisms for researchers to contribute to, improve and thereby assist with the overall sustainability of online resources.
The ways in which researchers relate to search highlighted three main observations that needed to be fed into improving search design.
- Transparency of a resource, in terms of contextual information and definition of the dataset, is important in terms of both search and overall site functionality. Researchers will struggle along with difficult resources but that is not an ideal situation. Many participants expressed how important it is to preserve as much of the integrity of original artefacts as possible and include some notion of the extent of the resource. In terms of developing some kind of coherent web resource this means finding ways to contextualise so that people can search intelligently.
- Access to different types of search is important, i.e. from basic keyword to semantic or linguistic search, as is the ability to search in different ways, i.e. from entering keyword(s) into a search bar to browsing categorised lists, not to make search necessarily easy but to make it more accessible to more people at different levels of interest and with different kinds of questions.
- Engaging with the research community through a digital resource was found to be an interesting way of sharing search methodologies, approaches and findings. It could also be a way to make a resource more sustainable by continuing to add value through expert contribution.
1.3. Content Provider Interviews
The interviews with eight content providers generated some understanding of how search design is approached by a range of organisations concerned with research infrastructure or research products. We interviewed representatives from three national libraries/archives, a digital historical library, an online genealogy resource, a major search engine, a large-scale digital content provider and two strands of a national funding programme which champions the use of digital technologies in UK education. The content providers are governed by funding structures, partner institutions and the information market, which impact on how they select what to digitise and what digitisation processes they adopt, including how they design and integrate search into their sources. Control is sometimes also relinquished by integrating outsourced background software and design work as opposed to keeping these in-house. This understanding goes some way to addressing the second research question: How is the design of search conditioned by technical considerations or the viewpoint of content creators?
Levels of user involvement in the design of their products varied considerably, from limited consultation that only takes place when a resource is launched, if at all, to fairly well developed user testing processes and implementation of user community feedback. What was clear was that the content providers involve users to a greater or lesser extent in the design and development of digital resources, but they are not driving design decisions from the beginning through to the end of a project. Users are generally brought in to evaluate work that has already been done and it seems that, while their feedback and any feedback gathered from the wider user community might well be collected, it is not always acted upon because of time and funding restrictions. Sometimes the resources that researchers have the most problems with are old -in digital terms- and have not been updated because funding from subscriptions or otherwise is being directed towards new digitisation projects directed by market demand. All of this means that researchers may be forced to persevere with resources they do not find intuitive and user-friendly because of the desire to tap into the material they contain, developing ‘work-arounds’ to address hindrances caused by issues with search and interface design.
The content providers are each at different stages of implementing more user-centred approaches as they begin to realise the positive impact these can have on the uptake and sustainability of digitisation projects. The importance of engaging users in the design process is gaining recognition amongst the funders of academic research as a consideration for helping with the ongoing impact of funded resources. An infoKit called ‘Planning a Participatory workshop’ 11. Jisc infoNet (2012) Planning a participatory workshop, [Online] Available: http://www.jiscinfonet.ac.uk/infokits/participatory/. is available to download from another strand of the national funding programme, which gives ideas for a variety of techniques that could be used in isolation or combination to elicit ideas from a group. However, while participatory workshops are, of course, a way to involve users to a greater extent in design and development, they are still not being placed in the driving seat.
2. Key Findings from Phase Two
2.1. Design Group Workshops
The Design Group workshops have produced some innovative search functionality as well as improving existing search processes. Four main themes emerged from the Design Groups, which were: searching; personalising, citing, and sharing.
The key issues that were criticised in search were: frustration with trying many variations of search that do not yield results; annoyance with having to scroll back to the top of a screen to check search terms or parameters; mistrust of tools if different results are returned for the same search. Some of the suggestions for design included: un-ticking/swiping away unwanted results; using tags to discount results; case sensitive searching, distinguishing proper nouns, spelling variation, word classes and so on. There was also a community focus, which involved a feedback loop to aid other researchers with search and making user comments searchable. In general terms researchers wanted more control about how they searched and how they saved their search process.
Theme 1: Searching
The key aspects of searching are as follows:
- Context and help providing background description and detail.
- Browsability allowing researchers to move through a resource in more flexible ways.
- Layering/ordering of search results in ways that suit particular research questions.
- Self-refining options giving researchers the flexibility to select what they need and what they don’t need.
- A ‘third’ page? A possible further step in the search process that could reflect the thought process of a researcher and provide a trace of that process. This has the potential to record notions of ‘voice’ and ‘authority’ in search.
- Visualising a search network so that the search process is represented and understood.
Theme 2: Personalising Search Within a Researcher’s Workflow
The key aspects of personalising search are as follows:
- User account to support the personalisation of search and the mapping, managing and manipulating of search results, which involves saving searches and creating subsets, setting search preferences, annotating results and having ownership over recorded search paths and findings.
Theme 3: Citing Search
The key aspects of citing search are as follows:
- Demonstrating the reliability of results by making search consistent and providing enough contextual information.
- Showing validity through institutional affiliation attached to public comments and the ability to illustrate search methodology effectively.
- Improving trust in digital tools, resources and methods by internalising browser-like features such as bookmarks and favourites, having feedback evaluated and acted upon and a simple interface.
- Supporting open access to data with persistent URLs and ways of recognising and recording that data may have been mediated in some way.
Theme 4: Sharing Search
The key aspects of sharing search are as follows:
- Publishing annotations and comments within the resource to inform future users.
- Institutional affiliation of users should be indicated to increase trust in user comments and other contributions.
- Reporting errors in the transcribed text; marking these for review by resource managers or having appointed “super-users” who have extended account privileges allowing them to make direct changes.
- Future proofing the resource by enabling the user community to evolve and enhance it through annotation and error correction.
These findings and design suggestions are incorporated into the design of the Newsbooks online resource, which can be accessed here: http://www.hrionline.ac.uk/newsbooks. They point towards a more personalised and flexible search approach that is more accessible for a wider variety of humanities scholars and for a wide range of research questions. Many aspects of the site and search are innovative, with one aspect – ‘refining’ – being particularly innovative. These characteristics give more control to the researchers to shape their search strategy and process, which enables them greater scope. It means that they do not have to work around the constraints of search developed by the commercial and large developers.
2.2. Key User Requirements
The following is a list of key requirements for search identified by users throughout the project:
- Both transcribed text and image should be made available, to preserve the integrity and context of original source material.
- Resource context: how records are stored, bibliographic data etc.
- Flexible navigation of a resource.
- Different pathways through a resource and layers of complexity to accommodate different research approaches.
- Different ways of ordering and refining results.
- A way of saving results.
- Trust in both the validity of search results and the search tools themselves.
- Simple interface.
- Clear citation guidelines.
- Data export options.
- Feedback function.
2.3. Key Innovations
The following is a list of key innovations made by the Newsbooks resource prototype:
- Advanced refining tools for search results: show or discard hits from a result list for a specific title, year, month + year or issue.
- “Published around this time” feature: titles published during the same week as the result being viewed.
- Extensive navigation options from within a search result: go to title page/next issue/previous issue; go back to results; new search; save page.
- A user correct function allowing “super-users” to make changes to the transcription.
- Annotation facility with option to make notes public.
- A personal workspace (for registered users) to save search results lists and results pages into folders and subfolders.
- A shopping cart icon to show how many items have been saved.
3. How Useful is Participatory Design for the Development of Online Resources in Digital Humanities?
The main focus of the Newsbooks Project was to ascertain whether Participatory Design (PD) is useful in the development of online resources in Digital Humanities. If it is indeed found to be useful, then to what degree and in what way can PD improve the design of search. The PD process has created the first digitised, searchable resource of the George Thomason’s Newsbooks (1649-1653), comprising all titles for the year 1649 and the full run of the title Mercurius Politicus. It has developed search in an innovative way that meets the needs of researchers in the academic disciplines of History, English Literature, English Language and Linguistics, Journalism, and Politics.
The use of PD has enabled the Newsbooks resource to be designed around the practice of working researchers and the ways in which they conceptualise and operationalise research. This has resulted in a highly intuitive and functional resource that is flexible and can be adapted for a wide range of humanities research processes and disciplines. Some key innovative features of the search include: an advanced tool for refining a search, a user-correct function, an account facility to save search results lists and pages into folders, and a way of recording annotations for personal or public use.
The PD approach has enabled the design process to address the diversity of practice from within and across, different disciplines. This has widened the usability of the resource because the search function allows for a wide range of search practice, and in doing so it has widened access to academics from a range of different disciplines.
The PD process has also identified emerging requirements of search in two main areas. In the first area, our research identified the development of new types of processes within search such as the ability to network in the search process. This includes networking sources, concepts, key terms, and research communities in order to address research questions from different perspectives, validate findings, identify relations between research concepts, and use the community’s collective knowledge to improve the source and increase knowledge about the source. Secondly, our research identified some emerging expectations that researchers have, such as personalising search, the development of a third page and visualising a search network, and developing ways to cite searches.
Our study identified that senses of research community and sharing knowledge across a research community are part of scholarly culture. Part of sharing is of course citing authors and acknowledging contributions, but there is a growing consensus that sharing knowledge about a source and about search is important for supporting, sustaining and developing digital resources. Part of that sharing involves researchers posting comments related to the source, which ensures comments are acknowledged by others and it makes researchers responsible for what they post.
The PD process was found to be beneficial from a development perspective. There were some very practical benefits including that the audio-visual recorded data helped to inform the developers about how researchers actually undertook search. The developers found that the knowledge of what researchers are actually doing helped them to create the technical specifications in developing search functionality and its look and feel. What was also helpful to the developers, with the caveat that the developers met with the PD social design researchers, was that the discursive approach to identifying requirements created many insights.
The developers found that the requirements that were constructed through debate in the design workshops and in the meetings with the PD researchers led to a richer understanding of the requirements. A key aspect in the success of the project was the way developers were integrated into the PD process; this improved understanding and enabled translation between user perspectives (and how they articulated their needs) and technical specification.
A key observation is that the integration of experienced digital humanities developers with PD researchers enables design processes to bridge the gap between rich user based understanding of their own needs with system and site requirements. This approach allows for the possibility of creating and supporting an emerging understanding of new kinds of functionality. The participants in the design groups are very capable of visualising and articulating what they want from search so that search could support them in addressing their research questions.
The openness of the developers is very important in supporting a user driven approach. One area, the development of an account system that mimics existing aspects of web browser functionality, was felt to be counter-intuitive by the developers. However, the developers created the account, which can now be tested as the source goes live by the broader research community.
The original contribution PD has made in developing search for the humanities is that it has developed the notion of search as a narrowly defined process and found one that is a conceptual practice-based approach. What this means is that the design of search has been shaped by the way in which researchers conceptualise research and then undertake search in relation to the conceptualisation of a particular research problematic. This has resulted in a list of key aspects that could apply more broadly. For example, both transcribed text and image should be available to provide the contextual information of original source material.
The design group participants, the developers and the PD researchers all comment that the PD process involves learning. There is consensus that the Newsbooks Project has been a learning process for all involved: each participant, researcher and developer contributes different types of insights and knowledge to the process. The PD process itself facilitates bringing together the different insights, knowledge and expertise in the development of more open and usable resources. The general process of critique, discuss and create supports the integration of different expertise required in developing search. This is supported by the use of specific methods, such as a landscape survey, focus groups, interviews, design groups, and developer and PD researcher discussion. Furthermore, PD is a reflective and reflexive process and any project undertaking PD needs to allow for such reflection and understand that a design will emerge in an iterative and reflexive manner. In the Newsbooks project this resulted in search that enables different humanities disciplines to undertake research into the early newsbooks.
4. Diagrammatic Overview of Participatory Design Stakeholders
This diagram illustrates the different contributors to the Participatory Design process. There is a range, from the initial scoping review into research cultures and methodologies in the humanities, a landscape survey, focus groups and interviews with content providers, through to consultation with field experts and a pilot Design Group. These aspects have all fed into the knowledge base from which the project Design Groups and development group (team meetings in which the observations, critique and suggestions of the Design Group were discussed and technical specifications for the prototype were drawn up) took their lead.
5. Limitations of a Participatory Design Approach
There are some limitations of the PD process in that it is time, cost, and resource intensive. There are therefore questions about how best to use PD in digital humanities development. Thought needs to be given on how to embed PD more routinely into the usual design and development process so that this knowledge becomes integral to design, which can then be funded in design process. A further area that might address the cost of PD is to consider how humanities scholars can contribute to the development of resources through community based contributions. Once PD becomes established in the humanities, scalability can be addressed by project RAs undertaking PD within their usual workflow. This could be supported by providing some steps to conducting PD, which would also include a basic workshop structure that could be adapted to different design contexts, such as how to address different site aspects in a series of workshops using the ‘critique-discuss-create’ method; recording workshop interactions and project team meetings. There would also need to be some guidance about analysis such as how to pick out key insights from the critique, the discussions and from creative suggestions from the various design processes. Certainly, RAs with awareness and experience of digital humanities are a key resource in this process. A template for PD workshops is provided in Appendix 4.
6. Resource Impact and Sustainability
‘A guide to Sustainability’ by JISC Digital Media explains the technical, cultural and strategic aspects of digital sustainability as well as revenue generation. Much depends on the specific characteristics, objectives and purpose of a project; as such there is no precise formula for achieving sustainability and every project is different. Questions of cultural sustainability, which is the only type we can legitimately comment on within this paper, revolve around how digital resources can prove their value when there is no monetary transaction involved, since resources are increasingly open access. One of the keys to understanding value is being able to measure impact. On a micro level:
“Digitisation can: streamline workflows; enhance the teaching and learning experience by facilitating access to and innovative use of resources; provide new research tools, resources and processes… A key driver for digital sustainability therefore lies in creating digital resources that make a measurable impact, and in being able to articulate and leverage this impact. An integral tool in this process is to develop a deep understanding of user needs and behaviours.” 22. Jisc Digital Media (2014) A guide to Sustainability, [Online] Available: http://www.jiscdigitalmedia.ac.uk/guide/a-guide-to-sustainability
Various quantitative and qualitative measures are suggested to help digitisers understand how users interact with resources including web analytics, surveys, focus groups and interviews. Taking this a step further, the successful implementation of PD relies on strong participant engagement. If projects can gain a thorough understanding of the intellectual and technological objectives, and the requirements of users, then they are more likely to succeed and create impact.
This notion of the cultural sustainability of digital resources is reflected in certain aspects of the functionality of the Newsbooks resource, achieved through a PD approach. The built-in user feedback mechanisms have the potential to sustain the resource and improve knowledge of how to use it and make corrections. An important caveat for building sustainability in this way requires that users’ comments must be citable and verifiable. By designing in these features of citation, trust and trustworthiness is assured.
Our work has already proved that a strong sense of community cohesion exists among researchers, and incorporating that as a feature of resources is recommended. Appointing ‘super users’ (users with certain privileges to make direct changes to data and metadata etc.) who can manage community feedback and review comments and corrections (submitted by other users) supports the idea that the resource ‘belongs’ to everyone and is not just something created by an elite group holding the exclusive rights to make changes. In line with the principles of PD, this democratisation of the resource empowers users, increases their engagement and ensures a high uptake of the technology.
Cultural sustainability can also be achieved by extending the availability of a resource to a wider audience, which has been addressed in the Newsbooks resource by having a variety of academic disciplines represented in the Design Group and designing in search functionality to suit a variety of research methods and questions.
7. Further Research Questions
- How have some of the new developments such as the use of an account been used by the wider research community?
- How has the refining function of search contributed to research processes in the humanities?
- How can PD be embedded into digital humanities
- How can search be improved by the research community?
- In what ways can the translation process between PD researchers and developers be enhanced?
- Raise awareness of developments in search to a wider range of researchers.
- Embed PD into design and development processes.
- Ensure that time and space is made available for the translation work of PD and technical development.
- Undertake evaluation of new search developments in order to assess new functionality .
- Generate multi-disciplinary teams to develop search; teams should involve subject specialists, developers and PD researchers.
- Foster a learning community amongst scholars to inform the design of search and other digital sources.
- Support a community approach to digital search and sources to support the cultural sustainability of sources and to improve search knowledge.