New study shows LLMs respond differently based on user’s motivation

Chaewon Bak
Michelle Bak
Jessie Chin
Jessie Chin, Assistant Professor

A new study conducted by PhD student Michelle Bak and Assistant Professor Jessie Chin, which was recently published in the Journal of the American Medical Informatics Association (JAMIA), reveals how large language models (LLMs) respond to different motivational states. In their evaluation of three LLM-based generative conversational agents (GAs)—ChatGPT, Google Bard, and Llama 2—the researchers found that while GAs are able to identify users' motivation states and provide relevant information when individuals have established goals, they are less likely to provide guidance when the users are hesitant or ambivalent about changing their behavior.

Bak provides the example of an individual with diabetes who is resistant to changing their sedentary lifestyle.  

"If they were advised by a doctor that exercising would be necessary to manage their diabetes, it would be important to provide information through GAs that helps them increase an awareness about healthy behaviors, become emotionally engaged with the changes, and realize how their unhealthy habits might affect people around them. This kind of information can help them take the next steps toward making positive changes," said Bak.

Current GAs lack specific information about these processes, which puts the individual at a health disadvantage. Conversely, for individuals who are committed to changing their physical activity levels (e.g., have joined personal fitness training to manage chronic depression), GAs are able to provide relevant information and support. 

"This major gap of LLMs in responding to certain states of motivation suggests future directions of LLMs research for health promotion," said Chin.

Bak's research goal is to develop a digital health solution based on using natural language processing and psychological theories to promote preventive health behaviors. She earned her bachelor's degree in sociology from the University of California Los Angeles.

Chin’s research aims to translate social and behavioral sciences theories to design technologies and interactive experiences to promote health communication and behavior across the lifespan. She leads the Adaptive Cognition and Interaction Design (ACTION) Lab at the University of Illinois. Chin holds a BS in psychology from National Taiwan University, an MS in human factors, and a PhD in educational psychology with a focus on cognitive science in teaching and learning from the University of Illinois.

Updated on
Backto the news archive

Related News

Trainor receives the Karen Wold Level the Learning Field Award

Senior Lecturer Kevin Trainor has been selected by the Division of Disability Resources and Educational Services (DRES) to receive the 2024 Karen Wold Level the Learning Field Award. This award honors exemplary members of faculty and staff for advocating and/or implementing instructional strategies, technologies, and disability-related accommodations that afford students with disabilities equal access to academic resources and curricula. 

Kevin Trainor

Seo coauthors chapter on data science and accessibility

Assistant Professor JooYoung Seo and Mine Dogucu, professor of statistics in the Donald Bren School of Information and Computer Sciences at the University of California Irvine, have coauthored a chapter in the new book Teaching Accessible Computing. The goal of the book, which is edited by Alannah Oleson, Amy J. Ko and Richard Ladner, is to help educators feel confident in introducing topics related to disability and accessible computing and integrating accessibility into their courses.

JooYoung Seo

iSchool instructors ranked as excellent

Fifty-five iSchool instructors were named in the University's List of Teachers Ranked as Excellent for Fall 2023. The rankings are released every semester, and results are based on the Instructor and Course Evaluation System (ICES) questionnaire forms maintained by Measurement and Evaluation in the Center for Innovation in Teaching and Learning. 

iSchool Building

ConnectED: Tech for All podcast launched by Community Data Clinic

The Community Data Clinic (CDC), a mixed methods data studies and interdisciplinary community research lab led by Associate Professor Anita Say Chan, has released the first episode of its new podcast, ConnectED: Tech for All. Community partners on the podcast include the Housing Authority of Champaign County, Champaign-Urbana Public Health District, Project Success of Vermilion County, and Cunningham Township Supervisor’s Office.

Community Data Clinic podcast logo

Santos honored by Illinois State Comptroller

Professor and Dean Eunice E. Santos was named a 2024 Women's History Month Honoree by the Illinois Office of Comptroller. She was recognized at a ceremony hosted by Illinois Comptroller Susana Mendoza on March 21 in Springfield. At the Women's History Month celebration, Santos and three other women trailblazers were honored for their achievements and contributions to the community.

Eunice Santos