Talk for me application

Supervisors

• Prof Derek Abbott
• Dr Mathew Berryman

Honours students

• 2024: Keefe Dela Cruz, Addy Dhingra and Matthew Fowler, see Talk for me application 2024

Project guidelines

• Project Handbook

Project description

The Talk For Me app provides a fully customisable yet easy to use grid to point to communicate for people who can't talk, and supports text-to-speech output and image-recognition suggested text. This project is about extending the capability further, from words to sentences, possibly involving the use of ChatGPT. You will require a Mac computer since this is an iOS app, and familiarity with Swift and SwiftUI is essential.

Aims

Talk For Me is a text-to-speech (TTS) iOS application that processes uploaded images to generate contextual text suggestions, which can t hen be spoken aloud by the device. The application is currently in alpha, offering only basic text suggestions. The aim is to enhance this feature and incorporate additional elements to improve the application. Improving these features and subsequently integrating the application into the neurodivergent community ensures future contributions to the advancement of the health caregiving market.

Background

Nonverbal individuals often face significant challenges with communication. While tools like sign language have provided solutions, they are not universally effective as it is not widely known. Consequently, TTS applications have been developed to enable nonverbal individuals to communicate in real-time with others using a common language.

Artificial Intelligence (AI), particularly through the use of Large Language Models (LLMs), is the key tool driving the improvements for Talk For Me. LLMs can perform language generation and natural language processing, enabling users to input keywords in the form of images and generate complex sentences. With numerous LLM options available, this project will also evaluate their performance to enhance communication speed. Additionally, the project will explore LLMs capable of operating locally on the user’s mobile phone.

Motivation

The Dr Matthew Berryman, of Across the Cloud, inspired the idea for Talk For Me after experiencing a stroke and facing communication struggles. As a results, the application development project is driven by the need to help neurodivergent individuals nonverbally specify their needs to the public. In the past, TTS applications have been slow and cumbersome because users had to manually type and edit every word they wanted to be spoken. However, with the integration of AI, these interactions are expected to become more natural and seamless, eliminating the pauses caused by manual input. AI will achieve this by predicting the most likely sentences based on various factors such as location, keywords, and previous behaviour or chosen sentences. It will the give contextual suggestions to the users allowing them to communicate easier.

Objectives

To ensure success within the allotted time frame and considering the team's expertise, the project focuses solely on enhancing the application's user interface (UI), functionality, performance, and analytics. By narrowing the scope, the purpose is to develop a high-quality, scalable application. Specifically, the objectives are to make UI improvements, integrate location-based systems, and LLM exploration. UI improvements will enhance both the readability and marketability of the product, particularly for neurodivergent audiences. Integrating location-based systems will create a more personalized user experience, significantly improving the user's quality of life. Exploring and experimenting with Large Language Models (LLMs) will ensure the generation of consistently appropriate sentences, reducing the frequency of errors.

Project management

Regular meetings with the team will be held to ensure tasks are complete. Furthermore, regular bi-weekly meetings with Dr Berryman will also ensure that the project remains aligned with the company's needs and values. To adhere to the time constraints, the schedules outlined in Appendix A: Gantt Chart will be strictly followed.

Report overview

This report provides a review of relevant literature pertaining to the project objectives. It offers a comprehensive overview of the objectives, outcomes, and strategies implemented to achieve them. Additionally, it provides details on the current progress of each approach and the plan to complete them in the future.

Useful notes

As useful bits of information come to light, just list them here:

Approach and methodology

List dot points: SS

Expectations

• To get good marks we expect you to show a logical approach.

• In your conclusion, you need to come up with a list of to do items for future improvement.

• We expect all the written work to be place on this wiki. No paper reports are to be handed up.

• It is expected that you fill out a short progress report on the wiki each week, every Friday evening, to briefly state what you did that week and what the goals are for the following week.

• It is important to regularly see your main supervisors. Don't let more than 2 week go by without them seeing your face briefly.

• You should be making at least one formal progress meeting with supervisors per month. It does not strictly have to be exactly a month, but roughly each month you should be in a position to show some progress and have some problems and difficulties to discuss.

• The onus is on you to drive the meetings, make the appointments and set them up.

• You are expected to make a YouTube presentation of your whole project.

Relationship to possible career path

This will enhance your software skills and project management skills and prepare you for a wide range of job opportunities.

See also

List any pages you create here:

References and useful resources

If you find any useful external links, list them here:

Back

• Back to MyUni
• Back to Derek Abbott's homepage
• Back to EEE Department page
• Back to the University of Adelaide homepage