The aim of this project is to use artificial intelligence (AI) to extract valuable information from unstructured eye movements of highly-skilled domain experts, in particular those of expert clinicians as they perform complex diagnostic decision-making tasks. Such eye-movement data is rich in patterns that can be deciphered using the power of unsupervised machine learning algorithms (such as k-nearest neighbor/hierarchical clustering and principal components analysis) or unsupervised deep learning algorithms (such as deep generative models, autoencoders, and long short-term memory autoencoders for sequence data). Furthermore, as novices transform into experts, patterns embedded in their eye movements (time spent on regions of interest vs. time spent on surgical equipment) may offer a valuable tool for extracting features that pinpoint the critical mechanisms (’eureka moments’) behind expert decision-making. The primary objectives of this project are (1) to collect eye movements of novice and expert ophthalmologists as they view medical images during eye-disease diagnoses using benchtop-based, head-mounted, or Virtual Reality embedded eye trackers (Eyelink 1000, Pupil Labs Core, or HTC Vive Pro, respectively) and (2) to apply unsupervised machine learning/deep learning approaches to extract meaningful information from this data. Features to be extracted from this data include but are not limited to: fixation duration and fixation count in regions of interest, fixation order, saccade velocity, and pupil diameter. This data collection and data analytics project will enable extraction of the most relevant features for task-oriented training of future AI-based disease diagnosis systems. Capturing eye movements, and thereby the underlying visual decision-making mechanisms behind an expert’s knowledge that are not otherwise quantifiable, will allow us to mimic these mechanisms in AI systems, potentially improving their diagnostic accuracy and interpretability for future clinical applications.

This project is eligible for a matching fund stipend from the Data Science Institute. This is not a guarantee of payment, and the total amount is subject to available funding.

Faculty Advisor

  • Professor: Kaveri Thakoor
  • Center/Lab: Ophthalmology
  • Location: Artificial Intelligence for Vision Science (AI4VS) Lab, Harkness Eye Institute
  • Dr. Kaveri Thakoor’s Artificial Intelligence for Vision Science (AI4VS) laboratory is focused on transforming AI/deep learning systems into teammates for ophthalmologists by tackling key challenges currently inhibiting the translation of AI to the clinic, such as robustness, interpretability, and portability. The goal of the AI4VS Lab is to develop human-vision-inspired AI to automate the analysis of biomedical images to augment and work in tandem with the abilities of medical experts to expedite the detection of eye diseases, such as glaucoma, age-related macular degeneration, and beyond.

Project Timeline

  • Earliest starting date: 11/1/2022
  • End date: 8/31/2023
  • Number of hours per week of research expected during Fall 2022: ~10

Candidate requirements

  • Skill sets: Fluency in Python, past experience with eye tracking/VR headsets is a plus
  • Student eligibility: freshman, sophomore, junior, senior, master’s
  • International students on F1 or J1 visa: eligible
  • Academic Credit Possible: Yes
  • Additional comments: Some experience in deep learning is a plus!