Advancing STEM Online Learning by Augmenting Accessibility with Interactive Companionship

Full title: Collaborative Research: Advancing STEM Online Learning by Augmenting Accessibility with Interactive Companionship

Videos are a popular option for online learning, and captions are essential for accessibility. Two types of captions exist: typical closed captions and explanatory captions. Closed captions are a text representation of the words spoken in a video, while explanatory captions are created to give students insight into the visual, textual, and audio content of a video.

Current technologies have focused on closed captions alone. For STEM subjects, explanatory captions have the potential to play a new role in learning. This project will work to devise mechanisms and interactions that enable students and instructors to generate explanatory captions for STEM videos in a collaborate manner. The proposed technologies will augment accessibility and learning experiences for under-served populations, including the Deaf and Hard-of-Hearing (DHH) community, made up of 48 million Americans, while also improving comprehension for non-native English speakers, even those without hearing impairments.

This interdisciplinary approach draws from and contributes to computer science and learning science while also strengthening accessibility practices. The first step is discovering new information about how accessibility-enabled videos broaden the participation of under-served populations in STEM learning. This information will provide the foundation for developing a theory of how explanatory captions can contribute to learning, as well as providing a foundation for effective mechanisms to create these explanatory captions for STEM videos at different learning stages. The investigators will then use the theory to create a novel chatbot that enables knowledge sharing for students with diverse backgrounds. Theoretical frameworks will guide the evaluation of how explanatory captions and chatbots can contribute to learning. Finally, the team will study how accessibility achieved with AI agents like the chatbot impacts students’ and instructors’ practices.