Willis defends dissertation

Craig Willis
Craig Willis, Teaching Assistant Professor

Doctoral candidate Craig Willis successfully defended his dissertation, "Trust, But Verify: An Investigation of Methods of Verification and Dissemination of Computational Research Artifacts for Transparency and Reproducibility," on June 15.

His committee included Associate Professor Victoria Stodden (chair); Assistant Professor Peter Darch; Professor Bertram Ludäscher; and Michela Taufer, professor of electrical engineering and computer science at the University of Tennessee, Knoxville.

From the abstract: In this study, I investigate how research communities are addressing concerns about the quality and rigor of computational research. I focus specifically on initiatives to expand the peer review and publication process to include new requirements for the assessment and dissemination of computational research artifacts. I report the results of a multiple-case analysis of two primary (American Journal of Political Science, ACM/IEEE Supercomputing) and five supplemental cases in political science, computer science, economics, mathematics, and statistics. Cases were developed through qualitative analysis based on interviews with key stakeholders (n= 17) including editors, reviewers, and verifiers; a sample of (n= 27) verified artifacts; and documentary evidence including policies, guidelines, and workflows. Based on the cases, I identify key factors that influence the operationalization of policies and workflows; the elements that each community considers important to the assessment of computational transparency and reproducibility; as well as the tools and infrastructure that they leverage to aid in the creation, assessment and dissemination of reproducible research artifacts. I develop a framework to analyze the reproducibility initiatives and a conceptual model of reproducible research artifacts. I relate my findings to recommendations from the recent National Academies of Sciences, Engineering and Medicine (NASEM) report on Reproducibility and Replicability in Science and provide a set of normative guidelines for communities interested in pursuing similar initiatives with implications for journal and conference leadership; tool and infrastructure developers; and funding bodies. I conclude that, while promising, further efforts should be made to increase our understanding of the effect of initiative policies and technological advancements on research quality.

Research Areas:
Tags:
Updated on
Backto the news archive

Related News

Spectrum Scholar Spotlight: Dalia Ortiz Pon

Twelve iSchool master's students were named 2024–2025 Spectrum Scholars by the American Library Association (ALA) Office for Diversity, Literacy, and Outreach Services. This "Spectrum Scholar Spotlight" series highlights the School's scholars. MSLIS student Dalia Ortiz Pon earned her bachelor's degree in Latina/Latino studies from San Francisco State University. 

Dalia Ortiz Pon

Debnath datafies "The Bulletin"

MSIM student Tan Debnath, whose interests span data mining, statistical modeling, text mining, and digital humanities, joined the Center for Children's books as a research assistant. He was tasked with building curation processes that would datafy seventy-five years' worth of archival issues of The Bulletin of the Center for Children's Books, one of the nation's leading children's book review journals.

Tan Debnath stands casually with his hands in his pockets and smiles broadly at the camera. It's a sunny day

iSchool undergraduates selected as 2025 Community-Academic Scholars

The Interdisciplinary Health Sciences Institute (IHSI) has selected BSIS student Dhanvi Puttur and BSIS+DS student Lara Terpetschnig as 2025 Community-Academic Scholars. Representing nineteen majors and nine minors in eight colleges and schools at the University of Illinois Urbana-Champaign and two additional universities, the eighteen scholars in this cohort encompass diverse fields of study, from community health to graphic design to statistics. 

BSIS+DS student Lara Terpetschnig and BSIS student Dhanvi Puttur

Guan successfully defends dissertation

Doctoral candidate Yingjun Guan successfully defended his dissertation, "Disambiguating Academic Institution Names: A Comprehensive Study of Authority Files, Linguistic Variations, and Computational Evaluation in PubMed Affiliations," on April 28. 

Yingjun Guan

Scholarship provides validation, motivation for Martinez

BSIS+DS student Fabian Martinez chose his major because he wanted to learn how to help people understand and interpret data and information. While his immediate plans include finding a job in data analytics, business analytics, consulting, or product management, his ultimate goal is "to create meaningful relationships and help make a meaningful impact in the world" in whatever way he can.

Fabian Martinez graduation