Many high quality RCTs in sports physical therapy are making false positive claims of treatment effect: a systematic survey

C M Bleakley, Jonathan Reijgers, James Smoliga

Research output: Contribution to journalArticle

Abstract

Objective: To examine the risk of false positive reporting within high quality randomized controlled trials (RCTs) in the sports physical therapy field.Design: Cross-sectionalMethods: We searched the PEDro database for parallel design 2-arm RCTs reporting positive treatment effects based on null hypothesis significance testing, and scoring >6/10 on the PEDro scale. No restrictions were made on pathology, intervention or outcome variables. Sixty-two of 212 RCTs reported positive effects in at least one outcome variable. We estimated False Positive Risk (FPR) using the FPR Web Calculator (version 1.5) based data on: n of participants, p-value, and effect size. For each study, FPR was estimated using a range of prior probability assumptions: 0.2 (skeptical hypothesis), 0.5 and 0.8 (optimistic hypothesis). Results: We calculated the FPR associated with 189 statistically significant findings (p<0.05) reported across 44 trials. The median FPR was 9% (25th-75th PCTL: 2-22%). 59% of statistically significant results (102/174) had FPR >5%, and 16% (28/174) had FPR >50%. Changing the prior probability from skeptical to optimistic reduced the median FPR from 30% (25th-75th PCTL: 9-54%) to 2% (25th-75th PCTL: 0.5-7%). Conclusion: High quality RCTs using null hypothesis significance testing often overestimated treatment effects. The median false positive risk (FPR) was 9% -- in one in 10 trials, the researchers falsely concluded there was a treatment effect. Future RCTs in sports physical therapy should be informed by pre study odds and a minimum FPR estimation.
LanguageEnglish
Pages104-109
Number of pages6
JournalJournal of Orthopaedic and Sports Physical Therapy
Volume50
Issue number2
DOIs
Publication statusPublished - 1 Feb 2020

Fingerprint

Sports
Randomized Controlled Trials
Therapeutics
Surveys and Questionnaires
Research Personnel
Databases
Pathology

Cite this

@article{90f3b3d10329470a82a58b1b04ca073d,
title = "Many high quality RCTs in sports physical therapy are making false positive claims of treatment effect: a systematic survey",
abstract = "Objective: To examine the risk of false positive reporting within high quality randomized controlled trials (RCTs) in the sports physical therapy field.Design: Cross-sectionalMethods: We searched the PEDro database for parallel design 2-arm RCTs reporting positive treatment effects based on null hypothesis significance testing, and scoring >6/10 on the PEDro scale. No restrictions were made on pathology, intervention or outcome variables. Sixty-two of 212 RCTs reported positive effects in at least one outcome variable. We estimated False Positive Risk (FPR) using the FPR Web Calculator (version 1.5) based data on: n of participants, p-value, and effect size. For each study, FPR was estimated using a range of prior probability assumptions: 0.2 (skeptical hypothesis), 0.5 and 0.8 (optimistic hypothesis). Results: We calculated the FPR associated with 189 statistically significant findings (p<0.05) reported across 44 trials. The median FPR was 9{\%} (25th-75th PCTL: 2-22{\%}). 59{\%} of statistically significant results (102/174) had FPR >5{\%}, and 16{\%} (28/174) had FPR >50{\%}. Changing the prior probability from skeptical to optimistic reduced the median FPR from 30{\%} (25th-75th PCTL: 9-54{\%}) to 2{\%} (25th-75th PCTL: 0.5-7{\%}). Conclusion: High quality RCTs using null hypothesis significance testing often overestimated treatment effects. The median false positive risk (FPR) was 9{\%} -- in one in 10 trials, the researchers falsely concluded there was a treatment effect. Future RCTs in sports physical therapy should be informed by pre study odds and a minimum FPR estimation.",
author = "Bleakley, {C M} and Jonathan Reijgers and James Smoliga",
year = "2020",
month = "2",
day = "1",
doi = "10.2519/jospt.2020.9264",
language = "English",
volume = "50",
pages = "104--109",
journal = "Journal of Orthopaedic and Sports Physical Therapy",
issn = "0190-6011",
publisher = "JOSPT",
number = "2",

}

Many high quality RCTs in sports physical therapy are making false positive claims of treatment effect: a systematic survey. / Bleakley, C M; Reijgers, Jonathan; Smoliga, James.

In: Journal of Orthopaedic and Sports Physical Therapy, Vol. 50, No. 2, 01.02.2020, p. 104-109.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Many high quality RCTs in sports physical therapy are making false positive claims of treatment effect: a systematic survey

AU - Bleakley, C M

AU - Reijgers, Jonathan

AU - Smoliga, James

PY - 2020/2/1

Y1 - 2020/2/1

N2 - Objective: To examine the risk of false positive reporting within high quality randomized controlled trials (RCTs) in the sports physical therapy field.Design: Cross-sectionalMethods: We searched the PEDro database for parallel design 2-arm RCTs reporting positive treatment effects based on null hypothesis significance testing, and scoring >6/10 on the PEDro scale. No restrictions were made on pathology, intervention or outcome variables. Sixty-two of 212 RCTs reported positive effects in at least one outcome variable. We estimated False Positive Risk (FPR) using the FPR Web Calculator (version 1.5) based data on: n of participants, p-value, and effect size. For each study, FPR was estimated using a range of prior probability assumptions: 0.2 (skeptical hypothesis), 0.5 and 0.8 (optimistic hypothesis). Results: We calculated the FPR associated with 189 statistically significant findings (p<0.05) reported across 44 trials. The median FPR was 9% (25th-75th PCTL: 2-22%). 59% of statistically significant results (102/174) had FPR >5%, and 16% (28/174) had FPR >50%. Changing the prior probability from skeptical to optimistic reduced the median FPR from 30% (25th-75th PCTL: 9-54%) to 2% (25th-75th PCTL: 0.5-7%). Conclusion: High quality RCTs using null hypothesis significance testing often overestimated treatment effects. The median false positive risk (FPR) was 9% -- in one in 10 trials, the researchers falsely concluded there was a treatment effect. Future RCTs in sports physical therapy should be informed by pre study odds and a minimum FPR estimation.

AB - Objective: To examine the risk of false positive reporting within high quality randomized controlled trials (RCTs) in the sports physical therapy field.Design: Cross-sectionalMethods: We searched the PEDro database for parallel design 2-arm RCTs reporting positive treatment effects based on null hypothesis significance testing, and scoring >6/10 on the PEDro scale. No restrictions were made on pathology, intervention or outcome variables. Sixty-two of 212 RCTs reported positive effects in at least one outcome variable. We estimated False Positive Risk (FPR) using the FPR Web Calculator (version 1.5) based data on: n of participants, p-value, and effect size. For each study, FPR was estimated using a range of prior probability assumptions: 0.2 (skeptical hypothesis), 0.5 and 0.8 (optimistic hypothesis). Results: We calculated the FPR associated with 189 statistically significant findings (p<0.05) reported across 44 trials. The median FPR was 9% (25th-75th PCTL: 2-22%). 59% of statistically significant results (102/174) had FPR >5%, and 16% (28/174) had FPR >50%. Changing the prior probability from skeptical to optimistic reduced the median FPR from 30% (25th-75th PCTL: 9-54%) to 2% (25th-75th PCTL: 0.5-7%). Conclusion: High quality RCTs using null hypothesis significance testing often overestimated treatment effects. The median false positive risk (FPR) was 9% -- in one in 10 trials, the researchers falsely concluded there was a treatment effect. Future RCTs in sports physical therapy should be informed by pre study odds and a minimum FPR estimation.

U2 - 10.2519/jospt.2020.9264

DO - 10.2519/jospt.2020.9264

M3 - Article

VL - 50

SP - 104

EP - 109

JO - Journal of Orthopaedic and Sports Physical Therapy

T2 - Journal of Orthopaedic and Sports Physical Therapy

JF - Journal of Orthopaedic and Sports Physical Therapy

SN - 0190-6011

IS - 2

ER -