TEXT2TASTE: A Versatile Egocentric Vision System for Intelligent Reading Assistance Using Large Language Model
The ability to read, understand and find important information from written text is a critical skill in our daily lives for our independence, comfort and safety. However, a significant part of our society is affected by partial vision impairment, which leads to discomfort and dependency in daily act...
Saved in:
Main Authors: | , , , , |
---|---|
Format: | Journal Article |
Language: | English |
Published: |
14-04-2024
|
Subjects: | |
Online Access: | Get full text |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Summary: | The ability to read, understand and find important information from written
text is a critical skill in our daily lives for our independence, comfort and
safety. However, a significant part of our society is affected by partial
vision impairment, which leads to discomfort and dependency in daily
activities. To address the limitations of this part of society, we propose an
intelligent reading assistant based on smart glasses with embedded RGB cameras
and a Large Language Model (LLM), whose functionality goes beyond corrective
lenses. The video recorded from the egocentric perspective of a person wearing
the glasses is processed to localise text information using object detection
and optical character recognition methods. The LLM processes the data and
allows the user to interact with the text and responds to a given query, thus
extending the functionality of corrective lenses with the ability to find and
summarize knowledge from the text. To evaluate our method, we create a
chat-based application that allows the user to interact with the system. The
evaluation is conducted in a real-world setting, such as reading menus in a
restaurant, and involves four participants. The results show robust accuracy in
text retrieval. The system not only provides accurate meal suggestions but also
achieves high user satisfaction, highlighting the potential of smart glasses
and LLMs in assisting people with special needs. |
---|---|
DOI: | 10.48550/arxiv.2404.09254 |