HandSight: A Touch-Based Wearable System to Increase Information Accessibility for People With Visual Impairments

HandSight augments the sense of touch in order to help people with visual impairments more easily access the physical and digital information they encounter throughout their daily lives. It is still in an early stage, but the envisioned system will consist of tiny CMOS cameras and micro-haptic actuators mounted on one or more fingers, computer vision and machine learning algorithms to support fingertip-based sensing, and a smartwatch for processing, power, and speech output. Potential use-cases include reading or exploring the layout of a newspaper article or other physical document, identifying colors and visual textures when getting dressed in the morning, or even performing taps or gestures on the palm or other surfaces to control a mobile phone.

Publications

teaser image of Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

ACM Transactions on Accessible Computing (TACCESS), 2016.
Keywords: accessibility, real-time OCR, visual impairments, wearables, XR interaction

teaser image of Supporting Everyday Activities for Persons With Visual Impairments Through Computer Vision

Supporting Everyday Activities for Persons With Visual Impairments Through Computer Vision

Proceedings of the 17th International ACM SIGACCESS Conference on Computers Accessibility (ASSETS), 2015.
Keywords: Blind; visually impaired; wearable computing; computer vision; vision-augmented touch


Videos

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind


Talks

Cited By

  • Understanding the Two-Step Nonvisual Omnidirectional Guidance for Target Acquisition in 3D Spaces. 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).SeungA Chung, Kyungyeon Lee, and Uran Oh. source | cite | search
  • Compact Microscopy Systems With Non-conventional Optical Techniques. 9.Tahseen Kamal. source | cite | search
  • Improving the Accessibility of Mobile OCR Apps Via Interactive Modalities. ACM Transactions on Accessible Computing.Michael Cutter and Roberto Manduchi. source | cite | search
  • Imagining Artificial Intelligence Applications With People With Visual Disabilities Using Tactile Ideation. Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility.Cecily Morrison, Edward Cutrell, Anupama Dhareshwar, Kevin Doherty, Anja Thieme, and Alex Taylor. source | cite | search
  • FingerReader: A Wearable Device to Explore Printed Text on the Go. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems.Roy Shilkrot, Jochen Huber, Wong Meng Ee, Pattie Maes, and Suranga Nanayakkara. source | cite | search
  • FingerReader: A Finger-Worn Assistive Augmentation. Assistive Augmentation.Roy Shilkrot, Jochen Huber, Roger Boldu, Pattie Maes, and Suranga Nanayakkara. source | cite | search
  • Elastic Documents: Coupling Text and Tables Through Contextual Visualizations for Enhanced Document Reading. IEEE Transactions on Visualization and Computer Graphics.Sriram Karthik Badam, Zhicheng Liu, and Niklas Elmqvist. source | cite | search
  • Towards Automatic Diagram Description for the Blind. Proceedings of the 11th International Convention on Rehabilitation Engineering and Assistive Technology.Seehait Chockthanyawat, Ekapol Chuangsuwanich, Atiwong Suchato, and Proadpran Punyabukkana. source | cite | search
  • Computer Vision Methods for Visually Impaired People. Fifteenth International Conference on Correlation Optics.Rostyslav Zatserkovnyi, Volodymyr Maik, and Roksoliana Zatserkovna. source | cite | search
  • Opto-electrotactile Feedback Enabled Text-line Tracking Control for a Finger-wearable Reading Aid for the Blind. 2022 International Conference on Robotics and Automation (ICRA).Mehdi Rahimi, Yantao Shen, Cong Peng, Zhiming Liu, and Fang Jiang. source | cite | search
  • Computational Glasses: Repurposing Augmented Reality Glasses for Vision Assistance. University of Otago.Jonathan Sutton. source | cite | search
  • Stimulation of Numerical Skills in Children With Visual Impairments Using Image Recognition. Procedia Computer Science.Jos{\'{e}} Antonio Taqu{\'{\i}}a Gutiérreza. source | cite | search
  • Evaluating Angular Accuracy of Wrist-based Haptic Directional Guidance for Hand Movement. GI '16: Proceedings of the 42nd Graphics Interface Conference.Jonggi Hong, Lee Stearns, Jon E. Froehlich, David Ross, and Leah Findlater. source | cite | search
  • Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface. ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility.Jonggi Hong, Alisha Pradhan, Jon E. Froehlich, and Leah Findlater. source | cite | search
  • Recognizing Clothing Colors and Visual Textures Using a Finger-Mounted Camera. 5.Alexander J. Medeiros, Lee Stearns, Leah Findlater, Chuan Chen, and Jon E. Froehlich. source | cite | search
  • Towards a Natural User Interface to Support People With Visual Impairments in Detecting Colors. 2016 23rd International Conference on Pattern Recognition (ICPR).Sergio Mascetti, Chiara Rossetti, Andrea Gerino, Cristian Bernareggi, Lorenzo Picinali, and Alessandro Rizzi. source | cite | search
  • HapAR: Handy Intelligent Multimodal Haptic and Audio-Based Mobile AR Navigation for the Visually Impaired. Technological Trends in Improved Mobility of the Visually Impaired. EAI/Springer Innovations in Communication and Computing.Ahmad Hoirul Basori. source | cite | search
  • Investigating Microinteractions for People With Visual Impairments and the Potential Role of On-Body Interaction. ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility.Uran Oh, Lee Stearns, Alisha Pradhan, Jon E. Froehlich, and Leah Findlater. source | cite | search
  • Augmented Reality Magnification for Low Vision Users With the Microsoft Hololens and a Finger-Worn Camera. ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility.Lee Stearns, Victor DeSouza, Jessica Yin, Leah Findlater, and Jon E. Froehlich. source | cite | search
  • Designing to Support and Extend the Competencies of People With Visual Impairments. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems.Gisela Reyes Cruz. source | cite | search
  • Three-dimensional Nonvisual Directional Guidance for People With Visual Impairments. 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops).Seung A Chung, Kyungyeon Lee, Sohyeon Park, and Uran Oh. source | cite | search
  • Social Sensemaking With AI: Designing an Open-ended AI Experience With a Blind Child. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.Cecily Morrison, Edward Cutrell, Martin Grayson, Anja Thieme, Alex Taylor, Geert Roumen, Camilla Longden, Sebastian Tschiatschek, Rita Faia Marques, and Abigail Sellen. source | cite | search
  • Digital Digits: Designing Assistive Finger Augmentation Devices. Massachusetts Institute of Technology.Roy Shilkrot. source | cite | search
  • Digital Digits: A Comprehensive Survey of Finger Augmentation Devices. ACM Computing Surveys.Roy Shilkrot, Jochen Huber, Jürgen Steimle, Suranga Nanayakkara, and Pattie Maes. source | cite | search
  • GestuRING: A Web-based Tool for Designing Gesture Input With Rings, Ring-Like, and Ring-Ready Devices. 0.Radu-Daniel Vatavu and Laura-Bianca Bilius. source | cite | search
  • TouchCam: Realtime Recognition of Location-Specific On-Body Gestures to Support Users With Visual Impairments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.Lee Stearns, Uran Oh, Leah Findlater, and Jon E. Froehlich. source | cite | search
  • Word Based Text Extraction Algorithm Implementation in Wearable Assistive Device for the Blind. 2019 IEEE International Conference on Intelligent Techniques in Control, Optimization and Signal Processing (INCOS).Arunima B Krishna, Meghana Hari, and Dr. Sudheer A.P. source | cite | search
  • Understanding Needs, Identifying Opportunities: ICT in the View of Universal Design. International Conference on Smart Objects and Technologies for Social Good.Ilaria Garofolo, Eric Medvet, Fulvio Babich, and Giovanni Ramponi. source | cite | search
  • E-mail Management System for Blind People in Spanish Language. Interacción '17: Proceedings of the XVIII International Conference on Human Computer Interaction.Guillermo Arturo Hernández Tapia and Ana Lilia Reyes-Herrera. source | cite | search
  • AiSee: An Assistive Wearable Device to Support Visually Impaired Grocery Shoppers. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.Roger Boldu, Denys J.C. Matthies, Haimo Zhang, and Suranga Nanayakkara. source | cite | search
  • HearIt: Auditory-Cue-Based Audio Playback Control to Facilitate Information Browsing in Lecture Audio. Applied Sciences.Jeongmin Hong, Hyesoo Jeon, Hana Lee, Daehyun Kim, and Minsam Ko. source | cite | search
  • Printed Texts Tracking and Following for a Finger-Wearable Electro-Braille System Through Opto-electrotactile Feedback. https://arxiv.org/abs/2109.02385.pdf.Mehdi Rahimi, Yantao Shen, Zhiming Liu, and Fang Jiang. source | cite | search
  • FingerReader2.0: Designing and Evaluating a Wearable Finger-Worn Camera to Assist People With Visual Impairments While Shopping. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.Roger Boldu, Alexandru Dancu, Denys J.C. Matthies, Thisum Buddhika, Shamane Siriwardhana, and Suranga Nanayakkara. source | cite | search
  • Computer Vision Based Assistive Technology for Blind and Visually Impaired People. ICCCNT '16: Proceedings of the 7th International Conference on Computing Communication and Networking Technologies.Shankar Sivan and Gopu Darsan. source | cite | search
  • Localization of Skin Features on the Hand and Wrist From Small Image Patches. 2016 23rd International Conference on Pattern Recognition (ICPR).Lee Stearns, Uran Oh, Bridget J. Cheng, Leah Findlater, David Ross, Rama Chellappa, and Jon E. Froehlich. source | cite | search
  • Terminal Operation Learning Application for the Screen Reader Users. 2015 IEEE International Conference on Systems, Man, and Cybernetics.Junji Onishi, Masatsugu Sakajiri, Takahiro Miura, and Tsukasa Ono. source | cite | search
  • Interactive Technology for People With Colour Vision Deficiency. Uppsala Universitet.Alexandra Kandler. source | cite | search
  • Markit and Talkit. Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology.Lei Shi, Yuhang Zhao, and Shiri Azenkot. source | cite | search
  • Handsight: A Touch-Based Wearable System to Increase Information Accessibility for People With Visual Impairments. University of Maryland, College Park.Lee Stearns. source | cite | search
  • Η Συμβολή Της Τεχνητής Νοημοσύνης Στην Εξ Αποστάσεως Εκπαίδευση Ατόμων Με Οπτική Αναπηρία. Πανεπιστήμιο Μακεδονίας .Αναστάσιος Γκότσης. source | cite | search
  • Diseño E Implementación De Un Sistema Electrónico De Lectura Para Estudiantes De Educación General Básica Con Discapacidad Visual. UNIVERSIDAD NACIONAL DE CHIMBORAZO.Mireya Daniela Armendariz. source | cite | search
  • HandSight: Supporting Everyday Activities Through Touch-Vision. Annual Report, U.S. Army Medical Research and Materiel Command.Jon E. Froehlich, Leah Findlater, David Ross, and Rama Chellappa. source | cite | search
  • FLight: A Low-Cost Reading and Writing System for Economically Less-Privileged Visually-Impaired People Exploiting Ink-based Braille System. 6.Tusher Chakraborty, Taslim Arefin Khan, and A. B. M. Alim Al Islam. source | cite | search
  • Towards Enabling Blind People to Independently Write on Printed Forms. Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology.Shirin Feiz, Syed Masum Billah, Vikas Ashok, Roy Shilkrot, and IV Ramakrishnan. source | cite | search
  • Stay In Touch