Written by Gwennie Ogilby (Associate Systematic Review Analyst) and Tom Metcalf (Senior Systematic Review Analyst)
Systematic literature reviews (SLRs) are considered one of the most powerful forms of evidence to guide healthcare policy and inform future research. Because high standards are maintained through rigorous methodology and adherence to strict principles of research integrity, SLR can be time-consuming and resource-intensive. Automation through artificial intelligence (AI) has the potential to increase speed and efficiency, but can it simultaneously reduce accuracy, reproducibility, and transparency? There is currently a relative lack of published guidance on this topic.
In the first international collaborative attempt to establish a set of standards and framework for the safe use of AI in evidence synthesis, International Collaboration to Automate Systematic Reviews (ICASR)Cochrane and Campbell produced “Responsible Use of AI in Evidence Synthesis.” (RAISE) Instructions. It is divided into three documents covering the following focus areas:
-
- raise 1: Recommendations for 8 specific individuals/groups, including organizations such as AI tool developers, SLR analysts, Source, etc.
- Raise 2: Guidelines for responsibly developing and evaluating AI tools for use in SLR
- raise 3: Selecting and using appropriate tools at various stages of the SLR process
Below we summarize the key themes of the RAISE guidelines and explore their immediate implications for SLR performance.
Responsibility and Supervision
How can reviewers assume ultimate responsibility for every step of the SLR process?
-
- Maintaining human oversight throughout the process ensures accountability for the output of each AI tool (referred to as ‘human in the loop’).
- We thoroughly audit AI tool output to protect against data hallucinations and potential bias.
Disclosure and Transparency
How can we keep our processes transparent about the impact of AI tools?
-
- Report publicly when, why, and how tools are used through clearly written protocols and publications.
- Leverage AI tools that are developed with transparency about their limitations and biases.
Methodological rigor and validation
How can we ensure the performance of AI tools without introducing bias?
-
- Consider which steps in SLR are appropriate for the current capabilities of AI (e.g. screening) and which steps may require further development (e.g. critical evaluation)
- Use tools that have been rigorously evaluated for the specific task at hand with appropriate performance metrics.
- We utilize a proven framework to assess the adequacy of tools based on transparent reporting of their strengths, limitations, and compliance with regulatory standards.
- AI tools must be trained on large, high-quality data sets to optimize accuracy.
Ethical, legal and regulatory compliance
How can we respect data privacy concerns and comply with regulatory requirements?
-
- Integrate guidelines for AI use into internal plagiarism and copyright policies
- Use tools that transparently describe training data sources and incorporate privacy protections around data collection.
Continuous learning and collaboration
In such a rapidly evolving field, how can we keep up with developments and optimize our use of AI?
-
- Engage in continuous learning as the field evolves to stay informed and compliant with new regulatory policies.
- We provide AI tool developers with real data to help them create high-quality tools that produce accurate output.
conclusion
The guidance outlined here is an important first step toward standardized and responsible use of AI in evidence synthesis. This is essential to address the complex legal and ethical issues that may arise from the use of AI, along with the potential for bias and inaccurate results. As new processes such as the Joint European Clinical Assessment (JCA) place more time pressure on SLR analysts, increased efficiency through responsible integration of AI technologies is likely to transform the industry. At Source, the RAISE guidance lays the foundation for updating your internal processes as you prepare for this exciting transition.
To learn more about SLR, contact Source Health Economics, a HEOR consultancy specializing in evidence generation, health economics and communications.
