Willis defends dissertation

Craig Willis
Craig Willis

Doctoral candidate Craig Willis successfully defended his dissertation, "Trust, But Verify: An Investigation of Methods of Verification and Dissemination of Computational Research Artifacts for Transparency and Reproducibility," on June 15.

His committee included Associate Professor Victoria Stodden (chair); Assistant Professor Peter Darch; Professor Bertram Ludäscher; and Michela Taufer, professor of electrical engineering and computer science at the University of Tennessee, Knoxville.

From the abstract: In this study, I investigate how research communities are addressing concerns about the quality and rigor of computational research. I focus specifically on initiatives to expand the peer review and publication process to include new requirements for the assessment and dissemination of computational research artifacts. I report the results of a multiple-case analysis of two primary (American Journal of Political Science, ACM/IEEE Supercomputing) and five supplemental cases in political science, computer science, economics, mathematics, and statistics. Cases were developed through qualitative analysis based on interviews with key stakeholders (n= 17) including editors, reviewers, and verifiers; a sample of (n= 27) verified artifacts; and documentary evidence including policies, guidelines, and workflows. Based on the cases, I identify key factors that influence the operationalization of policies and workflows; the elements that each community considers important to the assessment of computational transparency and reproducibility; as well as the tools and infrastructure that they leverage to aid in the creation, assessment and dissemination of reproducible research artifacts. I develop a framework to analyze the reproducibility initiatives and a conceptual model of reproducible research artifacts. I relate my findings to recommendations from the recent National Academies of Sciences, Engineering and Medicine (NASEM) report on Reproducibility and Replicability in Science and provide a set of normative guidelines for communities interested in pursuing similar initiatives with implications for journal and conference leadership; tool and infrastructure developers; and funding bodies. I conclude that, while promising, further efforts should be made to increase our understanding of the effect of initiative policies and technological advancements on research quality.

Research Areas:
Updated on
Backto the news archive

Related News

iSchool researchers present at virtual CIRN conference

iSchool researchers presented their work at the 19th annual Community Informatics Research Network (CIRN) Conference on November 8-12. The theme of this year's conference was "Communities, Technology and This Moment." CIRN 2021 explored how researchers and practitioners ethically collect information, including what happens when community information is deliberately not collected and how information systems can be designed "in harmony with communities."

Spectrum Scholar Spotlight: Cordiah Hayes

Eight iSchool master's students were named 2021-2022 Spectrum Scholars by the American Library Association (ALA) Office for Diversity, Literacy, and Outreach Services. This "Spectrum Scholar Spotlight" series highlights the School's scholars. MS/LIS student Cordiah Hayes earned her BS degree in communications studies with an emphasis in media studies from Northern Illinois University.

Cordiah Hayes

Chin and Desai discuss conversational agents at TMS Conference

Assistant Professor Jessie Chin and PhD student Smit Desai will present their research at the Technology, Mind and Society (TMS) Conference, which will be held virtually November 3-5. Hosted by the American Psychological Association, TMS brings together scientists, industry leaders, practitioners, students, and policymakers to explore the critical role that psychology plays in the design, use, adoption, and impact of technology and the artificial intelligence that powers it.

iSchool well represented at ASIS&T 2021

iSchool faculty and students will participate in the Association for Information Science and Technology (ASIS&T) Annual Meeting, which will be held in a hybrid format—in Salt Lake City, Utah, and online—from October 30-November 2. The theme of this year's conference is "Information: Equity, Diversity, Inclusion, Justice, and Relevance." The meeting, now in its 84th year, is the premier international conference dedicated to the study of information, people, and technology in contemporary society.

Huang and students present at CSCW 2021

Assistant Professor Yun Huang and students will present their research at the 24th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2021), which will be held virtually on October 23-27. CSCW is the premier venue for experts from industry and academia to explore the technical, social, material, and theoretical challenges of designing technology to support collaborative work and life activities.

Yun Huang