Difference between revisions of "Talk for me application"

From Derek
Jump to: navigation, search
(→‎Useful notes)
Line 3: Line 3:
 
*[[Matthew J. Berryman|Dr Mathew Berryman]]
 
*[[Matthew J. Berryman|Dr Mathew Berryman]]
  
−
==Honours students==
+
== Honours students ==
−
*'''2024:''' [[Keefe Dela Cruz]], [[Addy Dhingra]] and [[Matthew Fowler]], see [[Talk for me application 2024]]
+
*[[Keefe Dela Cruz]]
 +
*[[Addy Dhingra]]
 +
*[[Matthew Fowler]]
  
−
==Project guidelines==
+
==General project description==
−
*[http://www.eleceng.adelaide.edu.au/personal/dabbott/project_handbook_2009.pdf Project Handbook]
+
−
 
+
−
==Project description==
+
 
The Talk For Me app provides a fully customisable yet easy to use grid to point to communicate for people who can't talk, and supports text-to-speech output and image-recognition suggested text. This project is about extending the capability further, from words to sentences, possibly involving the use of ChatGPT. You will require a Mac computer since this is an iOS app, and familiarity with Swift and SwiftUI is essential.
 
The Talk For Me app provides a fully customisable yet easy to use grid to point to communicate for people who can't talk, and supports text-to-speech output and image-recognition suggested text. This project is about extending the capability further, from words to sentences, possibly involving the use of ChatGPT. You will require a Mac computer since this is an iOS app, and familiarity with Swift and SwiftUI is essential.
  
−
== Aims ==
+
==Specific tasks==
  
 +
The tasks identified by the group to complete for this project focus around improving three areas, the UI, the functionality of the App, and the performance of the app through LLM optimisation.
  
−
Talk For Me is a text-to-speech (TTS) iOS application that processes uploaded images to generate contextual text suggestions, which can t hen be spoken aloud by the device.  The application is currently in alpha, offering only basic text suggestions. The aim is to enhance this feature and incorporate additional elements to improve the application. Improving these features and subsequently integrating the application into the neurodivergent community ensures future contributions to the advancement of the health caregiving market.
 
  
−
== Background ==
+
UI
 +
* Update the UI to fit with similar apps on the market
  
  
−
Nonverbal individuals often face significant challenges with communication. While tools like sign language have provided solutions, they are not universally effective as it is not widely known. Consequently, TTS applications have been developed to enable nonverbal individuals to communicate in real-time with others using a common language.
+
Functionality of the App
 +
* Receive Location/Time information to make informed suggestions
 +
* Suggest Menu Items based off the users location
 +
* Suggest previously used sentences given a selection of words
 +
* Generate 1+ messages from the LLM
 +
* Sort the keywords presented to the users based off the acquired Location/Time information gained above.  
  
−
Artificial Intelligence (AI), particularly through the use of Large Language Models (LLMs), is the key tool driving the improvements for Talk For Me. LLMs can perform language generation and natural language processing, enabling users to input keywords in the form of images and generate complex sentences. With numerous LLM options available, this project will also evaluate their performance to enhance communication speed. Additionally, the project will explore LLMs capable of operating locally on the user’s mobile phone.
 
  
−
== Motivation ==
+
Large Language Models
 +
* Run and compare the time take to respond over a number of datasets reflecting 1-3 word combinations
 +
* Compare the number of outliers in each performance
 +
* Compare the models "reliability" as in how well it follows the provided instructions to respond with a response that contains the expected words (and a sentence that can be used for TTS).
  
 +
==Deliverables==
  
−
The Dr Matthew Berryman, of Across the Cloud, inspired the idea for Talk For Me after experiencing a stroke and facing communication struggles. As a results, the application development project is driven by the need to help neurodivergent individuals nonverbally specify their needs to the public. In the past, TTS applications have been slow and cumbersome because users had to manually type and edit every word they wanted to be spoken. However, with the integration of AI, these interactions are expected to become more natural and seamless, eliminating the pauses caused by manual input. AI will achieve this by predicting the most likely sentences based on various factors such as location, keywords, and previous behaviour or chosen sentences. It will the give contextual suggestions to the users allowing them to communicate easier.
+
===Semester 1===
 +
* Project Proposal [[File:TalkForMe-ProjectPlan.pdf]]
 +
* Seminar [https://adelaide.zoom.us/rec/share/ZmSdxd6Drx1VFdIszz-0N71q885vowWd4ge_iUgfHhZevbzT__buvo9mldA4GLDO.AkJ5dqBwcs8UOFh0?startTime=1718062694000] password: 3LC!+USl
 +
* Progress Report: [[File:TalkForMe-ProgressReport.pdf]]
  
−
== Objectives ==
+
===Semester 2===
  
 +
* Ingenuity Poster [[File:Talk For Me Ingenuity Poster.pdf]]
 +
* Ingenuity Video
 +
* Final Report [[File:TalkForMe-FinalReport.pdf]]
  
−
To ensure success within the allotted time frame and considering the team's expertise, the project focuses solely on enhancing the application's user interface (UI), functionality, performance, and analytics. By narrowing the scope, the purpose is to develop a high-quality, scalable application. Specifically, the objectives are to make UI improvements, integrate location-based systems, and LLM exploration. UI improvements will enhance both the readability and marketability of the product, particularly for neurodivergent audiences. Integrating location-based systems will create a more personalized user experience, significantly improving the user's quality of life. Exploring and experimenting with Large Language Models (LLMs) will ensure the generation of consistently appropriate sentences, reducing the frequency of errors.
+
== Weekly progress and questions ==
−
 
+
This is where you record your progress and ask questions. Make sure you update this every week.
−
== Project management ==
+
*[[Talk for me application 2024 weekly progress]]
−
+
−
 
+
−
Regular meetings with the team will be held to ensure tasks are complete. Furthermore, regular bi-weekly meetings with Dr Berryman will also ensure that the project remains aligned with the company's needs and values. To adhere to the time constraints, the schedules outlined in Appendix A: Gantt Chart will be strictly followed.
+
−
 
+
−
== Report overview ==
+
−
 
+
−
+
−
This report provides a review of relevant literature pertaining to the project objectives. It offers a comprehensive overview of the objectives, outcomes, and strategies implemented to achieve them. Additionally, it provides details on the current progress of each approach and the plan to complete them in the future.
+
−
 
+
−
==Useful notes==
+
−
As useful bits of information come to light, just list them here:
+
  
 
==Approach and methodology==
 
==Approach and methodology==
−
List dot points:
+
The approach involved a number of methods listed in our Progress and Final Reports.
−
SS
+
Due to our NDA, strict methods are not able to be provided, but can be summarised as accessing the existing code, and modifying/adding to it to gain the desired functions/changes.
  
 
==Possible extension==  
 
==Possible extension==  
−
 
+
TBA
−
 
+
  
 
== Expectations ==  
 
== Expectations ==  
 +
* To get good marks we expect you to show a logical approach to coding.
  
−
* To get good marks we expect you to show a logical approach.
+
* In your conclusion, you need to come up with a short list of likely possibilities and a list of things you can definitely eliminate.
−
 
+
−
* In your conclusion, you need to come up with a list of to do items for future improvement.
+
  
 
* We expect all the written work to be place on this wiki. No paper reports are to be handed up.   
 
* We expect all the written work to be place on this wiki. No paper reports are to be handed up.   
Line 74: Line 75:
  
 
== Relationship to possible career path==
 
== Relationship to possible career path==
−
This will enhance your software skills and project management skills and prepare you for a wide range of job opportunities.
+
The the coding and project management skills you pick up in the project will serve you for a wide range of careers.
−
 
+
−
==See also==
+
−
List any pages you create here:
+
−
 
+
  
 
== References and useful resources==
 
== References and useful resources==
 
If you find any useful external links, list them here:
 
If you find any useful external links, list them here:
−
 
+
*
−
 
+
*
  
 
==Back==
 
==Back==
Line 90: Line 87:
 
*[http://www.eleceng.adelaide.edu.au Back to EEE Department page]
 
*[http://www.eleceng.adelaide.edu.au Back to EEE Department page]
 
*[http://www.adelaide.edu.au Back to the University of Adelaide homepage]
 
*[http://www.adelaide.edu.au Back to the University of Adelaide homepage]
 +
*[https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/Cipher_Cracking_2012 Back to Cipher Cracking 2012 homepage]

Revision as of 13:17, 5 November 2024

Supervisors

Honours students

General project description

The Talk For Me app provides a fully customisable yet easy to use grid to point to communicate for people who can't talk, and supports text-to-speech output and image-recognition suggested text. This project is about extending the capability further, from words to sentences, possibly involving the use of ChatGPT. You will require a Mac computer since this is an iOS app, and familiarity with Swift and SwiftUI is essential.

Specific tasks

The tasks identified by the group to complete for this project focus around improving three areas, the UI, the functionality of the App, and the performance of the app through LLM optimisation.


UI

  • Update the UI to fit with similar apps on the market


Functionality of the App

  • Receive Location/Time information to make informed suggestions
  • Suggest Menu Items based off the users location
  • Suggest previously used sentences given a selection of words
  • Generate 1+ messages from the LLM
  • Sort the keywords presented to the users based off the acquired Location/Time information gained above.


Large Language Models

  • Run and compare the time take to respond over a number of datasets reflecting 1-3 word combinations
  • Compare the number of outliers in each performance
  • Compare the models "reliability" as in how well it follows the provided instructions to respond with a response that contains the expected words (and a sentence that can be used for TTS).

Deliverables

Semester 1

Semester 2

Weekly progress and questions

This is where you record your progress and ask questions. Make sure you update this every week.

Approach and methodology

The approach involved a number of methods listed in our Progress and Final Reports. Due to our NDA, strict methods are not able to be provided, but can be summarised as accessing the existing code, and modifying/adding to it to gain the desired functions/changes.

Possible extension

TBA

Expectations

  • To get good marks we expect you to show a logical approach to coding.
  • In your conclusion, you need to come up with a short list of likely possibilities and a list of things you can definitely eliminate.
  • We expect all the written work to be place on this wiki. No paper reports are to be handed up.
  • It is expected that you fill out a short progress report on the wiki each week, every Friday evening, to briefly state what you did that week and what the goals are for the following week.
  • It is important to regularly see your main supervisors. Don't let more than 2 week go by without them seeing your face briefly.
  • You should be making at least one formal progress meeting with supervisors per month. It does not strictly have to be exactly a month, but roughly each month you should be in a position to show some progress and have some problems and difficulties to discuss.
  • The onus is on you to drive the meetings, make the appointments and set them up.
  • You are expected to make a YouTube presentation of your whole project.

Relationship to possible career path

The the coding and project management skills you pick up in the project will serve you for a wide range of careers.

References and useful resources

If you find any useful external links, list them here:

Back

Retrieved from "https://www1.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php?title=Talk_for_me_application&oldid=23489"