A Webcam-Based Gaze-Aware Human–Computer Interaction Framework for Hands-Free Digital Reading and Context-Aware LLM-Assisted Text Understanding

Reading from digital text sources such as PDF documents, word processor documents, and web e-books involves extensive use of manual scrolling and pointing, which can be slow, physically demanding, and difficult for users with reduced dexterity in their hands. In this paper, the design and development of a cost-effective, calibration-based, webcam-only eye tracking technology that translates normal eye movements into a hands-free reading experience is presented. The system employs Google’s MediaPipe Face Mesh approach to detect the face, eyes, and irises of the user, estimate the distance of the user from the screen based on the number of pixels between the eyes, and construct a custom-made screen map with nine points (4 corner points plus the center) through a calibration process done for each user. As reading takes place, the system uses the computed position of the gaze to automatically highlight the line being read and scroll down when the last line of view is reached. An easily calibrated Eye Aspect Ratio (EAR) double-blink detector triggers a deliberate double-blink in relation to the fixated word or sentence; in response, the system opens a temporary side panel, where a locally hosted or API-based large language model (LLM) creates a brief explanation for the selected piece of text, and after that closes the side panel when detecting the second double-blink. Contrary to previously developed eyetyping, eye-scrolling, and magnification technologies that considered eye movement only as a pointing device, the novel system combines the function of eye-based reading position tracking with the blink-gated, on-demand AI-based explanation generator in a closed feedback loop. We provide a detailed description of the system architecture, calibration and gaze mapping algorithm, blink gesture state machine, and LLM integration process. We further explain the novelty of this approach in the context of eye-typing, gaze-contingent display, and assistive reading techniques. We next provide an overview of the evaluation procedure followed during the pilot study along with the measures (fixation mapping error, scrolling accuracy, blink detection error rate, and usability scores) that are being used in our main study.