Willis defends dissertation

Craig Willis
Craig Willis, Teaching Assistant Professor

Doctoral candidate Craig Willis successfully defended his dissertation, "Trust, But Verify: An Investigation of Methods of Verification and Dissemination of Computational Research Artifacts for Transparency and Reproducibility," on June 15.

His committee included Associate Professor Victoria Stodden (chair); Assistant Professor Peter Darch; Professor Bertram Ludäscher; and Michela Taufer, professor of electrical engineering and computer science at the University of Tennessee, Knoxville.

From the abstract: In this study, I investigate how research communities are addressing concerns about the quality and rigor of computational research. I focus specifically on initiatives to expand the peer review and publication process to include new requirements for the assessment and dissemination of computational research artifacts. I report the results of a multiple-case analysis of two primary (American Journal of Political Science, ACM/IEEE Supercomputing) and five supplemental cases in political science, computer science, economics, mathematics, and statistics. Cases were developed through qualitative analysis based on interviews with key stakeholders (n= 17) including editors, reviewers, and verifiers; a sample of (n= 27) verified artifacts; and documentary evidence including policies, guidelines, and workflows. Based on the cases, I identify key factors that influence the operationalization of policies and workflows; the elements that each community considers important to the assessment of computational transparency and reproducibility; as well as the tools and infrastructure that they leverage to aid in the creation, assessment and dissemination of reproducible research artifacts. I develop a framework to analyze the reproducibility initiatives and a conceptual model of reproducible research artifacts. I relate my findings to recommendations from the recent National Academies of Sciences, Engineering and Medicine (NASEM) report on Reproducibility and Replicability in Science and provide a set of normative guidelines for communities interested in pursuing similar initiatives with implications for journal and conference leadership; tool and infrastructure developers; and funding bodies. I conclude that, while promising, further efforts should be made to increase our understanding of the effect of initiative policies and technological advancements on research quality.

Research Areas:
Updated on
Backto the news archive

Related News

iSchool researchers to present at NeurIPS

iSchool faculty, staff, and students will present their research at the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), which will be held from December 10-16 in New Orleans, Louisiana. NeurIPS is one of the most prestigious and competitive international conferences in machine learning and computational neuroscience.

Petrella defends dissertation

Doctoral candidate Julia Burns Petrella successfully defended her dissertation, "Educating Pre-Service School Librarians about Race, Racism, and Whiteness," on December 4.

Julia Burns Petrella

Guo defends dissertation

Doctoral candidate Qiuyan Guo successfully defended her dissertation, "Exploring Chinese Celebrity Fans’ Online Information Behaviors and Understandings of Their Practices," on December 6.

Qiuyan Guo

BIG projects span the globe

This fall, students in the Business Intelligence Group (BIG), the student consultancy group associated with Associate Professor Yoo-Seong Song's Applied Business Research class (IS 514), worked on projects related to supply chain management, digital health, biotechnology, and higher education for companies located at the University of Illinois Research Park and overseas in East Asia and Africa. 

Internship Spotlight: State Farm

BSIS student Evan Chen discusses his summer internship at State Farm, where he developed an interest in computer vision.

Evan Chen