logo

Innovations in Technology

Supporting language use and research through transformative technologies

hero_image

Introduction

CoEDL rode a tide of unprecedented growth in language science technologies. A mandate to explore the intersection between language and computer engineering was embedded in the Centre through its Technology program. Led by CI Janet Wiles, the Technology program worked across the Centre’s nodes and other programs to facilitate the development of tools and systems that, for example, would support transcription, language learning or people living with special communication needs.

This work harnessed the transformative power of new technologies in interdisciplinary and collaborative ways. A lasting contribution to how CoEDL members and engineers develop and understand technologies was the program’s emphasis on co-design methods and workshops. Bringing engineers together with linguists, speech pathologists and living experience experts ensured that the tools and technologies produced are effective and useful, as well as innovative.

The highlights below introduce three projects at the heart of the Technology program: a pipeline to assist the transcription of language recordings; an ecosystem of language technologies to support the lives of people living with dementia and their carers; and software that helps researchers visualise, analyse and interpret human communication.

background
Transcription Acceleration Project

Linguists and language workers have long engaged in the key yet laborious task of transcribing language recordings. CoEDL's Transcription Acceleration Project (TAP), led by CI Janet Wiles and project manager Ben Foley, sought to understand the pains and gains of transcription processes and develop language technologies to accelerate these ways of working.

A survey led by CoEDL Research Fellow Gautier Durantin found that people take 40 minutes to transcribe one minute of recorded audio on average. For some respondents, the ratio was as high as 250 to one.

CoEDL linguists were keen to investigate the potential of language technologies such as speech recognition to reduce the transcription burden. When TAP started, speech recognition was becoming more readily available for some of the world’s languages through commercial platforms. However, few of the languages which CoEDL researchers were working with were supported by these services, because they needed to be trained on thousands of hours of existing, transcribed data.

Other speech recognition tools, such as Kaldi, could be "trained" to learn the languages not supported by commercial providers, but these were notoriously difficult to use. TAP focussed on making these existing tools more accessible by developing a user-friendly interface for speech recognition. Elpis was built so that people could make speech recognition systems for their own needs, without having the technical experience that was traditionally required to use these tools.

Elpis was designed not to fully automate the transcription process, but to generate a best guess as a basis for correction. Elpis has been used for linguistic analysis, speeding up transcription and as a key component in the Pintupi-Luritja health translation app "Wangka Kutju", developed by Purple House. In Elpis workshops, linguists built speech recognisers for languages including Arrernte, Bininj Kunwok, Cook Islands Māori, Ende, Indonesian, Mangarrayi, Nafsan, Pitjantjatjara and Warlpiri.

TAP led to several fruitful collaborations. As a co-design project, Elpis was created with language workers, linguists, designers and software developers from across CoEDL. A collaboration with Languages and Cultures of Oral Tradition (LACITO) in France involved an international group of linguists and programmers. In 2019, Advisory Committee member Daan van Esch of Google delivered a keynote address on Elpis at the International Conference on Language Technologies for All at UNESCO in Paris. A partnership with Boeing software engineers aimed to reduce the size of the Elpis software, to make it easier to distribute. A CoEDL crossover with the Sydney Speaks Project also contributed to TAP by confirming that forced-aligner technology to facilitate the production of acoustic models can be applied to languages with small quantities of recorded material; these models provided new information that TAP researchers could feed into Elpis.

potrait
Florence

The Florence Project develops technologies to meaningfully support the lives of people living with dementia and their carers. Initially led by CI Helen Chenery, the Florence team evolved and expanded to include CIs Tony Angwin and Janet Wiles, AI Dan Angus, Research Fellow Jacki Liddle, Project Manager Peter Worthy, several research assistants and an expert reference group of people living with dementia who share their experiences and provide feedback.

“It was extremely important to embed the perspectives of people who live with dementia in this project from the beginning,” Janet said. “We wanted to be guided by them and their families and friends.”

Reference group member Dennis Frost emphasised the importance of living experience experts being involved in technology design. He feels he has a positive effect on making sure the process is efficient and that the tools are helpful.

“There is a lot of technological innovation that could be vastly improved if a few simple changes were made early in the design process,” he said.

“It is easy to design tech that seems useful,” Janet echoed. “It is much harder to design language tech devices that are actually wanted and used by people with communication difficulties.”

As well as championing the co-design model, the interdisciplinary team — covering computer science, interaction design, speech pathology, psychology, cognitive science and occupational therapy and including living experience experts — have developed several prototypes. These include a device that looks and functions like a simple radio but streams music shared by family members and a calendar that displays photos and other information. The resulting ‘ecosystem’ of devices can be used and combined in different ways to meet different needs.

Connecting this ecosystem is an underlying language bank creating a web of connections between significant information, images and other data. By learning about the person’s world, the language bank can provide context for the words they use.

“Key to the technology that we have been developing [for Florence] is an ability to ‘understand’ speech,” Peter explains. “We have been developing a technological system that processes speech in everyday settings to capture everything from the words and features of speech to identifying what the person is talking about and storing that information in a format so that it can be ‘used’ by other technologies.”

The project has produced a range of outputs including software that supports the use of language by technology. This technology and the methods and insights of the Florence team have informed the Australian Text Analytics Project (part of the Australian Research Data Commons project) and influenced the development of other tools to support people living with different conditions that cause language or communication impairment. For example, the LifeCHAT project to develop an app to assist people living with aphasia — a language impairment affecting communication, experienced by one third of stroke survivors — commenced in 2021, led by CoEDL Affiliate Sarah Wallace.

potrait
Discursis

Led by Associate Investigator Dan Angus, Discursis is a computer-based tool that visualises the structure, information content and inter-speaker relationships of human conversation. It relies on conceptual recurrence plotting, a methodology first introduced by Dan with CI Janet Wiles and Andrew Smith before CoEDL started [1]. From 2014, CoEDL supported the development of Discursis through collaborations across the Centre.

Discursis extracts key data from conversation transcripts using a natural language processing algorithm. It then tags conversational turns, utterances, pauses and concepts to reveal patterns across the conversation. The software also offers a variety of options for visualising this data to assist researchers with analysing and interpreting human communication.

Breaking down, visualising and understanding conversation in this way can assist language science researchers in investigating, for example, the relationship between language and cognitive development. Working with CoEDL Research Fellow Christina Atay, Dan used Discursis to analyse conversations between people living with dementia and their carers and identify points of communication breakdown [2, 3]. CoEDL PhD student Noëlie Creaghe also applied Discursis in her research with the Canberra Longitudinal Child Language lab on the effect of different types of play on language acquisition.

The Discursis team facilitated these collaborations through several workshops for the CoEDL community. Discursis also provided a foundation for the development of the cALPY language processing software library, developed at the Centre’s University of Queensland node and applied in the Florence Project. In addition to visualising conversation data, Discursis can be used to model collaborations in different settings (see image).

With the conclusion of CoEDL, Discursis has taken a place in the Australian Text Analytic Platform (ATAP), an initiative co-established by CoEDL Affiliate Michael Haugh. Several other CoEDL members and projects, including aspects of CI Catherine Travis’ Sydney Speaks Project, feed into ATAP, which will continue to innovate ways in which researchers can use computational methods to study language data.

image

Further information

To learn more about other CoEDL research, explore the Research Projects subset of Connections data in map or list form.

Captions

Hero image: Advisory Committee member Craig Cornelius interacts with Opie, a robot designed by CoEDL members at the Centre’s University of Queensland node to facilitate the study of interaction between robots and children. Image: CoEDL.

Image 1: Advisory Committee member Daan Van Esch (L) with TAP Program Manager Ben Foley (R) at a conference where they presented on Elpis. Image: CoEDL

Image 2: CoEDL Research Fellow Andrew Back explains features of the Florence Project during a showcase on people’s views of emerging technology. Image: CoEDL/University of Queensland.

Image 3: A sociogram of collaborations across CoEDL. Image: CoEDL.

References

[1] Angus, Daniel, Smith, Andrew, & Wiles, Janet (2012) Conceptual recurrence plots: Revealing patterns in human discourse. IEEE Transactions on Visualization and Computer Graphics, 18(6), pp. 988-997. DOI: 10.1109/TVCG.2011.100

[2] Christina Atay, Erin Conway, Daniel Angus, Janet Wiles, Rosemary Baker, and Helen Chenery. December 10, 2015. "An Automated Approach to Examining Conversational Dynamics between People with Dementia and Their Carers." PLoS ONE. 10 (12): e0144327. doi: 10.1371/journal.pone.0144327.

[3] Rosemary Baker, Daniel Angus, Erin Smith-Conway, Katherine Baker, Cindy Gallois, Andrew Smith, Janet Wiles, and Helen Chenery. 2015. "Visualising conversations between care home staff and residents with dementia." Ageing and Society. 35 (2): 270-297. doi: 10.1017/S0144686X13000640.

Ben Foley, Josh Arnold, Rolando Coto-Solano, Gautier Durantin, T. Mark Ellison, Daan van Esch, Scott Heath, František Kratochvíl, Zara Maxwell-Smith, David Nash, Ola Olsson, Mark Richards, Nay San, Hywel Stoakes, Nick Thieberger, and Janet Wiles. 2018. "Building Speech Recognition Systems for Language Documentation: The CoEDL Endangered Language Pipeline and Inference System (ELPIS)". In Proceedings of the 6th International Workshop on Spoken Language Technologies for Under-Resourced Languages, 205-209. Gurugram, India.

David Ireland, Christina Atay, Jacki Liddle, Dana Bradford, Helen Lee, Olivia Rushin, Thomas Mullins, Daniel Angus, Janet Wiles, Simon McBride, and Adam Vogel. 2016. "Hello harlie: Enabling speech monitoring through chat-bot conversations". In Digital Health Innovation for Consumers, Clinicians, Connectivity and Community - Selected Papers from the 24th Australian National Health Informatics Conference, HIC 2016, 55-60. Melbourne, Australia.

Lydia Byrne, Daniel Angus, and Janet Wiles. 2015. "Acquired Codes of Meaning in Data Visualization and Infographics: Beyond Perceptual Primitives."  IEEE Transactions on Visualization and Computer Graphics. 22 (1): 509-518. doi: 10.1109/TVCG.2015.2467321.

Rosemary Baker, Daniel Angus, Erin Smith-Conway, Katherine Baker, Cindy Gallois, Andrew Smith, Janet Wiles, and Helen Chenery. 2015. "Visualising conversations between care home staff and residents with dementia." Ageing and Society. 35 (2): 270-297. doi: 10.1017/S0144686X13000640.

Liddle, Jacki, Worthy, Peter, Frost, Dennis, Taylor, Eileen, Taylor, Dubhglas, Beleno, Ron, Angus, Daniel, Wiles, Janet, Angwin, Anthony, and The Florence Project Living Experience Expert Reference Group (2022). “Personal and complex: The needs and experiences related to technology use for people living with dementia”. Dementia 21 (5) 1-21.https://doi.org/10.1177/14713012221084521