CAREER: Using Network Analysis to Assess Confidence in Research Synthesis

Policy in areas such as conservation, energy, healthcare, and sustainable development is informed by a variety of factors, including the best available science. Determining the best available science requires synthesizing multiple scientific results to gauge both the level of scientific consensus and the reliability of the research. However, on some policy-relevant topics, syntheses continue to come to incompatible conclusions over a decade or more. Such inconsistency in the synthesis of evidence wastes money, generates misleading results, and can lead to poor decisions impacting large numbers of people. Through research, education, and outreach, this CAREER project aims to iteratively develop and test a novel framework of tools and workflows that will show stakeholders potential sources of bias in expert literature. The framework will enable evidence-seekers to quickly understand which individuals, institutions and funders contributed to the creation of the evidence along with other factors that create risk of bias, as well as the degree of confidence an expert community has in the evidence. Research outcomes could facilitate data-driven decision-making in a broad range of areas—for example, vital topics in energy and environmental sciences, such as the carbon footprint of various forms of food production, and in health sciences, including COVID-19 related topics such as herd immunity and the effectiveness of vaccines. This project will also diversify the science workforce, directly by employing student assistants from underserved populations; and indirectly by developing two policy-relevant STEM university courses and a middle school career video that attract underrepresented students, meeting the National Science Board priority to make science more representative of the U.S. population.

This project explores how to improve the assessment of confidence in research at scale, in order to enable evidence-seekers to quickly understand the level of consensus within a body of literature along with risk factors that might impact reliability of research, providing a key resource for robustness and reproducibility. 

This framework can be applied to any bibliography, from manuscripts under peer review, published articles, and database search results. Project outputs will be beneficial for identifying risks in literature reviews, such as sponsor bias or the avoidance of citation of contradictory evidence, which will help reduce the spread of misinformation. This project is made possible by recent advances in network science and text mining methods, as well as the availability of abstracts, affiliation, citations, and funding data under suitable licenses for data science. The approach is novel in bringing together complementary approaches that have not previously been combined: argumentation theory and the study of controversies; approaches for synthesizing evidence; and bibliometrics and scientometrics approaches for looking structurally at a field.

Read more