Guest blog: A tsunami of overlapping reviews in surgery

As systematic review papers on the same topic continue to be published, a massive amount of information pours down on the readers, a trend that gradually amplifies like a tsunami every year¹. This phenomenon is still not well unknown in our field for surgeons. A meta-epidemiological study conducted by our research team showed that considerable variability in results and methodological quality can be found among multiple systematic reviews and meta-analyses (SRs/MAs) on the same topic. Additionally, high methodological quality in the systematic review evaluation is independently associated with increased citation counts in the subsequent five years after adjusting for journal impact factors and other covariates. These results highlight the current situation wherein surgeons seeking high-quality “evidence-based surgery“ do not know which review results to believe in the clinical setting, even though SRs/MAs are central to “evidence-based medicine“. To reduce the burden on busy surgeons in finding the most relevant evidence, this study suggests that quality evaluations should play a complementary role, and researchers and journal editors should concentrate their efforts on reviews of more limited quantity but higher quality. 

“These results highlight the current situation wherein surgeons seeking high-quality “evidence-based surgery” do not know which review results to believe in the clinical setting“

Many clinicians and researchers have felt the wastefulness of the recent explosion of similar review articles^1,2. Several descriptive studies regarding duplicate review articles on pharmacological intervention have been reported^3,4 but particularly in surgery, this is rarely discussed. Therefore, we hypothesized that there are many redundant overlapping SR/MA publications with conflicting results and variable qualities, even in the field of surgery⁵. We first identified all SRs/MAs focusing on surgical interventions published in 2015 via PubMed. We then searched for SRs/MAs of similar RCTs published within the preceding five years (between 2011 and 2015). As the research progressed, an interesting idea arose among our research team to examine the number of citations as a measure of scientific impact after publication; therefore, we decided to investigate factors associated with the number of five-year citations through 2020 (Figure 1). This research was conducted in collaboration with the Research Group on Meta-epidemiology at Kyoto University School of Public Health.The results of this study were published online in the British Journal of Surgery on 17^th November 2021.

Omission and commission phenomenon

Ideally, all randomized controlled trials (RCTs) included in previous SRs/MAs should also be included in subsequently published SRs/MAs. In the real world, however, some of them have been omitted (Figure 2). We attempted to visualize this omission and commission phenomenon and compared the coverage probabilities of all RCTs that were published until the publication year of each SR/MA (one example is shown in Table 2). As a result, we found that there was considerable variability in coverage probability for relevant RCTs, and this was observed on almost every topic except the Cochrane update review.

Discrepant results among overlapping systematic reviews

We found discrepancies among overlapping SRs/MAs on the same topic, such as different effect sizes, different statistical precisions, and even different directions of effect for many topics. Of course, although this is only the result of a sample within a wide surgical field, we believe that it is highly representative.

We show some examples of the discrepancies of pooled effect estimates among overlapping SRs/MAs in Table 2. For the topic ‘Appendectomy vs. Antibiotic treatment for acute appendicitis’, while the largest odds ratio (OR) was 8.13, the smallest was 0.54, a difference by a factor of 15.06 (Table 2), which was the largest difference among the 29 surgical topics which we reviewed in this study. We hypothesize that the major reasons why the pooled effect estimates are so different among overlapping SRs/Mas are related to differences in coverage probability for relevant RCTs (Table 1) and differences among the study subjects. Of course, there could be many other reasons, so we expect further research to shed light on some of them. Interestingly, overlapping publications have continued to increase since 2015, according to an updated PubMed search, particularly for topics with a large discrepancy in the study results.

How do you select a systematic review paper for citation?

When we write a scientific paper, what important factors are considered when picking systematic review articles to cite for our research? Many researchers may answer that it is the journal’s name recognition or impact factor. We used a mixed-effects regression model with a random intercept for surgical topics, which has been used as an analytical method in previous meta-epidemiological studies⁶, to investigate factors associated with the five-year citation counts of each SR/MA. As expected, a high journal impact factor (5.50 counts more per 1-point increment; 95% CI, 3.97 to 7.03; P <.001) was associated with a greater 5-year citation count. In addition, after adjusting for journal impact factors and other covariates, a high quality score (AMSTAR score 8-11) was independently associated with greater 5-year citation counts with a low quality score (AMSTAR score of 0-3) used as a reference (33.36 counts more; 95% CI, 16.20 to 50.51; P <.001). It is natural for researchers to evaluate the quality of the papers they would cite.

Implication for research and practice

Surgical SRs/MAs rarely include RCTs but instead focus on many nonrandomized trial designs. In this study, we evaluated SRs/MAs with only RCTs, which means that most SRs/MAs were excluded, and therefore, the situation is expected to be more serious in the real world. Surgical SRs/MAs may also be conducted predominantly by surgeons, lacking the involvement of methodologists or evidence synthesis experts, which could limit the quality of the research. There may be a significant risk of accepted wisdom concerning the hierarchy of evidence levels if numerous SRs/MAs on the same topic show high variability in the results and methodological quality. To prevent unnecessary duplication, the international Prospective Register for Systematic Reviews (PROSPERO) was launched in 2011; today, however, it may rarely work properly in practice and may only be considered as a courtesy⁷. Another novel approach, the “Living systematic review“, has been proposed: SRs are continually updated, incorporating relevant new evidence as it becomes available. It may be time to seek such new and more efficient methods⁸. Prior to initiating a project in surgical SRs/MAs, adequate communication and team building between surgeons and methodologists are critical in preventing duplication of research efforts and ensuring methodological rigor and currency. While further study is needed to discuss the problems of overlapping publications in surgical SRs/MAs, findings from our study may help guide clinicians, researchers and journal editors.

“Researchers and journal editors should concentrate their efforts on reviews of more limited quantity but higher quality.“

The illustration was created by the Tsuchimochi design office. Image source: hand drawing/Shutterstock.com under license, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.