New study shows LLMs respond differently based on user’s motivation

Chaewon Bak
Michelle Bak
Jessie Chin
Jessie Chin, Assistant Professor

A new study conducted by PhD student Michelle Bak and Assistant Professor Jessie Chin, which was recently published in the Journal of the American Medical Informatics Association (JAMIA), reveals how large language models (LLMs) respond to different motivational states. In their evaluation of three LLM-based generative conversational agents (GAs)—ChatGPT, Google Bard, and Llama 2—the researchers found that while GAs are able to identify users' motivation states and provide relevant information when individuals have established goals, they are less likely to provide guidance when the users are hesitant or ambivalent about changing their behavior.

Bak provides the example of an individual with diabetes who is resistant to changing their sedentary lifestyle.  

"If they were advised by a doctor that exercising would be necessary to manage their diabetes, it would be important to provide information through GAs that helps them increase an awareness about healthy behaviors, become emotionally engaged with the changes, and realize how their unhealthy habits might affect people around them. This kind of information can help them take the next steps toward making positive changes," said Bak.

Current GAs lack specific information about these processes, which puts the individual at a health disadvantage. Conversely, for individuals who are committed to changing their physical activity levels (e.g., have joined personal fitness training to manage chronic depression), GAs are able to provide relevant information and support. 

"This major gap of LLMs in responding to certain states of motivation suggests future directions of LLMs research for health promotion," said Chin.

Bak's research goal is to develop a digital health solution based on using natural language processing and psychological theories to promote preventive health behaviors. She earned her bachelor's degree in sociology from the University of California Los Angeles.

Chin’s research aims to translate social and behavioral sciences theories to design technologies and interactive experiences to promote health communication and behavior across the lifespan. She leads the Adaptive Cognition and Interaction Design (ACTION) Lab at the University of Illinois. Chin holds a BS in psychology from National Taiwan University, an MS in human factors, and a PhD in educational psychology with a focus on cognitive science in teaching and learning from the University of Illinois.

Updated on
Backto the news archive

Related News

Desai defends dissertation

Doctoral candidate Smit Desai successfully defended his dissertation, "Designing Metaphor-fluid Voice User Interfaces," on June 10.

Smit Desai

Student says ‘thank you’ with a helicopter ride

Last month, Michael Ferrer showed his appreciation for one of his MSIM instructors in a unique way—by inviting him for an insider’s look at his work as a reservist in the Illinois Army National Guard. For the ILARNG BOSS Lift, which took place on June 18 at Camp Atterbury, Indiana, Ferrer selected Michael Wonderlich, iSchool adjunct lecturer and senior associate director of business intelligence and enterprise architecture for Administrative Information Technology Services (AITS) at the University of Illinois.

Michael Wonderlich and Michael Ferrer hold a U of I flag in front of a military helicopter

Project helps librarians use data storytelling to advocate for public libraries

A toolkit for public librarians can help them use data to communicate the value of their services and justify their funding needs. The Data Storytelling for Librarians Toolkit helps librarians present data in story form using narrative strategies. It was developed by University of Illinois Urbana-Champaign information sciences professors.

Kate McDowell

Chan to deliver keynote at SIGCIS 2024

Associate Professor Anita Say Chan will deliver the keynote at the 15th annual conference of the SHOT (Society for the History of Technology) Special Interest Group for Computing, Information, and Society (SIGCIS), which will be held on July 14 in Viña del Mar, Chile. SIGCIS is the leading international group for historians with an interest in the history of information technology and its applications. The theme for SIGCIS 2024 is "System Update: Patches, Tactics, Responses."

Anita Say Chan

Mattson receives ISTE Making It Happen Award

Adjunct Lecturer Kristen Mattson has received the 2024 International Society for Technology in Education (ISTE) Making It Happen Award. The award honors educators and leaders who demonstrate outstanding commitment, leadership, courage, and persistence in improving digital learning opportunities for students.

Kristen Mattson