Rank reversal in indirect comparisons

OBJECTIVE: To describe rank reversal as a source of inconsistent interpretation intrinsic to indirect comparison (Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epi 1997;50:683-91) of treatments and to propose best practice. METHODS: We prove our main points with intuition, examples, graphs, and mathematical proofs. We also provide software and discuss implications for research and policy. RESULTS: When comparing treatments by indirect means and sorting them by effect size, three common measures of comparison (risk ratio, risk difference, and odds ratio) may lead to vastly different rankings. CONCLUSIONS: The choice of risk measure matters when making indirect comparisons of treatments. The choice should depend primarily on the study design and the conceptual framework for that study.