To update or not to update systematic literature searches? Deciding what constitutes stable evidence

25/11/21

The Belgian Red Cross has been developing evidence-based first aid guidelines and training materials for use in Flanders (Belgium) and Sub-Saharan Africa for over 10 years. To provide a basis for these evidence-based materials, CEBaP collects the best available evidence through systematic literature searches in electronic databases of medical literature and synthesizes the evidence in structured evidence summaries (ES). Because science evolves with the accumulation of the research, the guidelines are updated every five years, to maintain the validity of recommendations. However, because of the volume of research questions, the process of updating and revisiting the systematic literature searches is time consuming. 

Update classification system

We here first aimed to develop a classification model to decide whether an ES needs to be updated primarily based on estimating the likelihood that the current conclusions will maintain their relevance in the foreseeable future. Situations in which an evidence summary may be declared to be stable include the following:

1. the conclusion is so certain that the addition of new information will not change it, and there are no foreseeable adverse effects of the intervention (conclusive evidence);
2. the intervention can be considered common sense and the available evidence supports the recommendation.

First, what defines ‘conclusive evidence’? We here assess whether future studies are likely to change the findings of the evidence summaries. According to the GRADE Working Group, certainty of evidence reflects the extent of our confidence that the estimates of the effect are correct. The GRADE approach results in the assessment of the certainty of a body of evidence as high, moderate, low, or very low. In some cases, the decision can be very straightforward, for example when the existing findings of the summary are based on high certainty evidence, no further update might be needed. Although, it is equally possible that an evidence summary can (temporarily) be declared as stable if the certainty of the evidence supporting it is only, at best, low or very low. The primary reason is that further research is very unlikely to change our confidence in the effect estimate given the inherent and anticipated limitations in the research domain.

Secondly, when interventions or risk factors can be considered common sense and the conclusions of an evidence summary confirm the effectiveness or harm, the evidence can be categorized as being stable. At the same time, it should be reasonable that future studies would not change the recommendation. This can be anticipated by controlling for inconsistent results and by paying attention to potential adverse events or harm associated with the intervention or risk factor.

A pilot study

The Belgian Red Cross has updated its first aid guidelines for Flanders (Belgium) and Sub-Saharan Africa for the third time in 2021. During this process, ES for 454 research questions were newly developed or updated and, after completion, the update status of each individual ES was assessed by two reviewers independently. Based on the in-house developed classification system, 95 ES (21% of total) were considered to be stable. 27 ES (28%) were stable due to conclusive evidence and 68 (72%) were considered common sense supported by the best available evidence. In only 7% of the ES, a conflict between reviewers was noted. A final decision was then made by a third reviewer. The update status should be revaluated every five years. Based on the total time investment for updating the guideline ES (412 working days), one can estimate that implementing the update classification system could save about 87 working days. These results provide proof-of-concept that the update classification system will help us to increase time-efficiency of future updates of evidence-based guidelines. 

Curious about the details of our update classification model and further results? Our findings will be published in 2022, stay tuned!