Systemic review and meta-analysis of diagnostic imaging technologies

“As long as the centuries continue to unfold, the number of books will grow continually, and one can predict that a time will come when it will be almost as difficult to learn anything from books as from the direct study of the whole universe. It will be almost as convenient to search for some bit of truth concealed in nature as it will be to find it hidden away in an immense multitude of bound volumes.”—Denis Diderot, “Encyclopédie” [1755].

In the recent paper by John P. A. Ioannidis (1), it was pointed out that currently there is a massive production of unnecessary and misleading reviews and meta-analyses. These publications often serve mostly as easily publishable units as primary research is not required, or sometimes as biased and selective tools for marketing. Another problem of the numerical competition for publishing more papers is that there are too many more or less similar publications, thus leading to information overload. It is not necessarily one paper is exactly the same as another, but the difference is small and very incremental. It is time-consuming for researchers to read all related papers, but not wise not to read them as in case there is a gem there. Then, important publications become less visible in the sea of published papers (2). However, this should not lead to the misconception that reviews are less needed. Indeed, high quality reviews authored by seasoned researchers are highly desired, even more so with the current ever-increasing output of scientific publications (3,4). In 2010 Bastian noted that there are 75 trials, and 11 systematic reviews of trials, per day and a plateau in growth has not yet been reached (3). With a median of perhaps 80 participants per trial, in 2005, Chan and Altman estimated the number of people being enrolled in trials is likely to be more than 2,000,000 per year (5). Given such mountains of papers and studies, scientists cannot be expected to examine in detail every single new paper relevant to their interests, it necessary to rely on regular summaries of the recent literature, and avoid unnecessary repetitive studies (6,7). As mountains of unsynthesised research evidences accumulate, we need to keep effectively gathering, filtering, and synthesising them. Once a new study is completed, it is necessary to update the cumulative evidence, as nicely shown by a number of papers in this issue of QIMS (8,9). Timely literature reviews can lead to new insights (10). Researchers who have spent their career working on certain research areas are in a natural position to review, and summarize, and update the related literatures.

Review articles are broadly classified into expert review, systemic review, and educational review which commonly includes pictorial review in diagnostic imaging. Good systematic literature reviews can validate hypotheses and opinions, offer more conclusive results than a single primary research. However, it is important to remember that reviewing literature is not stamp-collecting. Some researchers suggest that a proper review should critique each study included, and suggest that it is usually necessary to systemically search at least two databases appropriate for the area of study in order to provide a reasonable breadth and depth on a topic. During the search process, the authors need to keep track of the databases searched and the terms used, a starting year, and the ending year and month of the search. Some authors also like to keep track of how many ‘hits’ or article citations that are found with each search. It is important that the search methods can be replicated by the readers (11,12).

A good review also discusses literature critically, identifies methodological problems, and points out research gaps (4). One important criterion for a good review is that it should stay neutral and balanced. Particularly in narrative reviews, some scientists may be overly enthusiastic about what they have published, and thus risk giving too much importance to their own findings in the review. Scientific neutrality is not often completely achievable, since authors are humans with their own experiences, but neutrality should be considered an ultimate goal (13). Stephen Jay Gould [1941–2002] once argued that “unconscious manipulation of data may be a scientific norm” because “scientists are human beings rooted in cultural contexts, not automatons directed toward external truth” (14).

In 1986 and 1987, Goldschmidt and Mulrow showed the potential for errors in reviews of health literature that were not conducted systematically (15,16). Without a requirement that reviewers clearly specify inclusion criteria and then exhaustively include all studies that fit these criteria, reviewers may consciously or unconsciously decide to include studies that favor their own biases and ignore those that do not. Systematic reviews attempt to test a hypothesis based on the published evidence, which is gathered using a predefined protocol to reduce bias (17). Meta-analyses is especially useful when clinical trials exist in the literature but possess low sample sizes, which was more so in the past, that prevent the authors from making conclusions that can be generalized to the population at large (18). Adequate meta-analyses, combining data from many studies and thousands of patients, can enhance the precision of treatment effects and reduce the risk of false-negative results (19). Conclusions from meta-analyses are susceptible to reporting biases and to choices of study eligibility criteria. The pooling of data that can be analyzed statistically, which is the strength of the meta-analysis, can also be a drawback because it is difficult to find studies that are similar enough to one another to draw valid comparisons (19).

Important factors that can threaten the validity of clinical trials are selection bias (biased allocation to comparison group), observer bias (unblinded outcome assessment), and attrition bias (unbalanced dropout rates). Furthermore, authors tend to report the outcomes with the most favorable results and suppress other outcomes with non-significant or negative results. Not all relevant studies are published in indexed journals. According to one analysis, even for randomised clinical trials (RCTs), barely half of all abstracts presented at scientific meetings are subsequently published (20). This implies that potentially important information may be inaccessible to reviewers. It has been noted that a ‘negative’ study, with no statistically significant treatment effect, is less likely to be presented at scientific meetings, to be subsequently published, to be published promptly, to be published in English and to be cited (19). On the other hand, positive studies are more likely to be published more than once (multiple publication bias) (21). Missing ‘negative’ studies might cause an overestimation of the true treatment effect. Meta-analyses based exclusively on published literature can, lead to overoptimistic results (19). Therefore, ideal meta-analyses should be primary research efforts where investigators collaborate preemptively in consortia.

The rationale given of meta-analysis to include all studies regardless of quality rather than identifying the methodologically adequate ones is primarily to avoid the reviewer’s own biases. While it is difficult to justify the haphazard study selection of narrative reviews, it is also difficult to accept the meta-analysts’ exhaustive inclusion strategy. In actual practice all-inclusive meta-analysis can also produce serious errors (22). The problem of the reviewer’s bias entering into inclusion decisions is hardly solved by exhaustive inclusion followed by statistical tests. Meta-analyses can only be as valid as the studies selected for the systematic review. Due to the potential problems of meta-analysis, particularly those based on scattered small studies, Slavin proposed the concept of best-evidence synthesis (23). Slavin argues that all other things being equal, far more information can be extracted from a large literature by clearly describing the best evidences on a topic as determined by objective standards. For best-evidence synthesis, studies of lower methodological rigor are not included, and thus differ from the exhaustive inclusion principle suggested by Glass et al. (24) and others. Additionally, it should be noted that systemic reviews should primarily be based on original work, not to re-use older review’s suggestion or conclusion. Importantly, recently published suggestions and guidelines to standardize study design and publication of all data, including the inclusion of some raw data maybe as supplement documents, will allow more efficient synthetic analysis, such as ‘the ARRIVE guidelines for reporting pre-clinical research’ (7,25-28).

In the current increasingly complex scientific fields, expert review, educational reviews and tutorials are also of great helps for scientists to scan information outside their specific area of expertise and for researchers and students starting to work on new topics. Expert review is usually a narrative review where the author retrieves and synthesizes information about a particular topic for the readers. It is expected that the author possesses expertise in the content area, and is expected to present the author’s own experiences, insights, and predictions. However, it should be noted that author’s synthesis of the articles is likely to contain bias.

In the field of diagnostic imaging technologies, systemic review and meta-analysis are both highly needed, but in the meantime difficult to perform. It can be considered irrational that not to systematically review what is already known before deciding to perform any new study. Taking the case of validating of new MRI technique for application in various organs and diseases, it is desirable that the interim progresses are periodically analyzed and summarized, so to make the next study design more rational and more scientifically valid. However, in the meantime, to synthesize results is difficult and sometimes impossible due to the highly heterogeneity of data acquisition techniques, and also the rapid updating and releasing of newer versions of the technologies. To report and validate work-in-progress technologies is probably necessary, though it may be a frustration that finally only the end-product will be popularized, and only the results involving the end-product will be broadly recognized and better cited.

Naturally review authors should focus on areas they are most familiar with. To write reviews may also provide the opportunities to bring together the end-users of imaging technologies (experts of clinical specialties), image interpreters (radiologists), and imaging technology developers (medical physicists and engineers). A gap and misunderstanding exist and will continue to exist among these three groups of experts (29-31). Sometimes an addressable clinical need remain not being tackled; while other times imaging technology developers might be working on problems which do not exist clinically; or imaging technology developers might have not taken into account of heterogeneity of patient pathologies, and an over-simple version of technology will not work in clinical practice (32-35). Truly multi-disciplinary research efforts will more likely be to be executed in the private research sector such as in industries, but remain an idealistic goal consistently to be pursued in publically funded academia (36). In this connection, reading review papers written by other disciplines will be helpful. It is also better that authors for a review paper include expertise from different disciplines, review papers written by authors from a single narrow research field run the high risk that the importance of areas the authors are working are exaggerated. Rosy pictures may be described for immature techniques. This is unfair to junior readers as they may not be able to recognize errors in a review.

Acknowledgements

None.

Footnote

Conflicts of Interest: The author has no conflicts of interest to declare.

References

1. Ioannidis JP. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q 2016;94:485-514. [Crossref] [PubMed]
2. Wáng YX. Why so many Chinese clinical doctors are competing for basic research grants? Quant Imaging Med Surg 2015;5:489-90. [PubMed]
3. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 2010;7:e1000326. [Crossref] [PubMed]
4. Pautasso M. Ten simple rules for writing a literature review. PLoS Comput Biol 2013;9:e1003149. [Crossref] [PubMed]
5. Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 2005;365:1159-62. [Crossref] [PubMed]
6. Ioannidis JP. Why Most Clinical Research Is Not Useful. PLoS Med 2016;13:e1002049. [Crossref] [PubMed]
7. Ioannidis JP. How to make more published research true. PLoS Med 2014;11:e1001747. [Crossref] [PubMed]
8. Ruggieri M, Polizzi A, Strano S, Schepis C, Morano M, Belfiore G, Palmucci S, Foti PV, Pirrone C, Sofia V, David E, Salpietro V, Mankad K, Milone P. Mixed vascular nevus syndrome: a report of four new cases and a literature review. Quant Imaging Med Surg 2016;5:515-24.
9. Ruggieri M, Polizzi A, Schepis C, Morano M, Strano S, Belfiore G, Palmucci S, Foti PV, Pirrone C, Roggini M, David E, Salpietro V, Milone P. Cutis tricolor: a literature review and report of five new cases. Quant Imaging Med Surg 2016;5:525-34.
10. Wáng YX. Advance modern medicine with clinical case reports. Quant Imaging Med Surg 2014;4:439-43. [PubMed]
11. Hampton SE, Parker JN. Collaboration and Productivity in Scientific Synthesis. BioScience 2011;61:900-10. [Crossref]
12. Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Acad Med 2011;86:1049-54. [Crossref] [PubMed]
13. Lewis JE, Degusta D, Meyer MR, Monge JM, Mann AE, Holloway RL. The mismeasure of science: Stephen Jay Gould versus Samuel George Morton on skulls and bias. PLoS Biol 2011;9:e1001071. [Crossref] [PubMed]
14. Gould SJ. Morton's ranking of races by cranial capacity. Unconscious manipulation of data may be a scientific norm. Science 1978;200:503-9. [Crossref] [PubMed]
15. Goldschmidt PG. Information synthesis: a practical guide. Health Serv Res 1986;21:215-37. [PubMed]
16. Mulrow CD. The medical review article: state of the science. Ann Intern Med 1987;106:485-8. [Crossref] [PubMed]
17. Cook DA, West CP. Conducting systematic reviews in medical education: a stepwise approach. Med Educ 2012;46:943-52. [Crossref] [PubMed]
18. Verhulst SJ, Colliver JA. Basic meta-analysis: conceptualization and computation. J Dev Behav Pediatr 2009;30:75-80. [Crossref] [PubMed]
19. Finckh A, Tramèr MR. Primer: strengths and weaknesses of meta-analysis. Nat Clin Pract Rheumatol 2008;4:146-52. [Crossref] [PubMed]
20. Scherer RW, Langenberg P, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev 2007.MR000005. [PubMed]
21. Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ 1997;315:635-40. [Crossref] [PubMed]
22. Slavin RE. Meta-analysis in education. How has it been used? Educ Res 1984;13:24-7.
23. Slavin RE. Best evidence synthesis: an intelligent alternative to meta-analysis. J Clin Epidemiol 1995;48:9-18. [Crossref] [PubMed]
24. Glass GV, McGaw B, Smith ML. Meta-analysis in social research. Newbury Park: Sage Publications, 1981.
25. Bastian H. A stronger post-publication culture is needed for better science. PLoS Med 2014;11:e1001772. [Crossref] [PubMed]
26. Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol 2010;8:e1000412. [Crossref] [PubMed]
27. Hooijmans CR, Leenaars M, Ritskes-Hoitinga M. A gold standard publication checklist to improve the quality of animal studies, to fully integrate the Three Rs, and to make systematic reviews more feasible. Altern Lab Anim 2010;38:167-82. [PubMed]
28. Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 2014;383:166-75. [Crossref] [PubMed]
29. Wang YX, Ng CK. The impact of quantitative imaging in medicine and surgery: Charting our course for the future. Quant Imaging Med Surg 2011;1:1-3. [PubMed]
30. McGowan JC. On the use of quantitative MR imaging. AJNR Am J Neuroradiol 2001;22:1451-2. [PubMed]
31. Wáng YX. Physical scientists research biomedicine: a call for caution. Chin J Cancer Res 2015;27:94-5. [PubMed]
32. Wáng YX, Idée JM, Corot C. Scientific and industrial challenges of developing nanoparticle-based theranostics and multiple-modality contrast agents for clinical application. Nanoscale 2015;7:16146-50. [Crossref] [PubMed]
33. Wang J, Choi HS, Wáng YX. Exponential growth of publications on carbon nanodots by Chinese authors. J Thorac Dis 2015;7:E201-5. [PubMed]
34. Belavý DL, Albracht K, Bruggemann GP, Vergroesen PP, van Dieën JH. Can Exercise Positively Influence the Intervertebral Disc? Sports Med 2016;46:473-85. [Crossref] [PubMed]
35. Nakamura K, Kimura I, Fujita K, Takano Y, Saga T, Ohta K, Yamaki K. Invitation to be basic medical research doctors; current status and efforts at Kurume University School of Medicine. Kaibogaku Zasshi 2013;88:21-3. [PubMed]
36. Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov 2011;10:712. [Crossref] [PubMed]