TY - JOUR
T1 - A Decade On: Reflecting on the Limitations of the First Meta-analysis of the Implicit Relational Assessment Procedure’s (IRAP) Criterion Validity in the Clinical Domain
AU - Vahey, Nigel
AU - Nicholson, Emma
AU - Barnes-Holmes, Dermot
PY - 2025/1/15
Y1 - 2025/1/15
N2 - Hussey (in press) recently conducted a detailed critical reanalysis of Vahey, Nicholson and Barnes-Holmes’ (2015) meta-analysis. Its stated purpose was to (a) examine the extent to which Vahey et al.’s (2015) meta-analysis contains errors; and (b) to test how computationally reproducible it is by current standards of best practice. Hussey identified a small number of minor numerical errors, but crucially was unable to exactly replicate the original meta-effect of 𝑟̅ = .45. Six different variations of the meta-analysis reported by Vahey et al. were used and obtained meta-effects that deviated from the original by Δ𝑟̅ = .01-.02. Hussey also reported corresponding 95% credibility intervals that were all of zero width. These discrepancies prompted the present authors to conduct a detailed audit of the original meta-analysis. This revealed one minor transposing error in addition to three identified by Hussey. Once corrected this resulted in a marginally increased Hunter and Schmidt meta-analytic effect of 𝑟̅ = .46 without a credibility interval, and a Hedges-Vevea meta-effect of 𝑟̅ = .47 with 95% confidence interval (.40, .54). This correction was too small to have any bearing on Vahey et al.’s supplementary analyses regarding publication bias or statistical power. Vahey et al. contained a much lower proportion of transposing errors than is typical of meta-analyses even still (cf. Kadlec et al., 2023; Lakens, 2016; Lakens et al., 2017). Nonetheless, Hussey highlighted important ambiguities about the theoretical and practical meaning of the meta-effect reported by Vahey et al. We clarify our position on these matters in summary, and in so doing explain why we believe that the wider IRAP literature would undoubtedly benefit from increased adoption of contemporary open science standards.
AB - Hussey (in press) recently conducted a detailed critical reanalysis of Vahey, Nicholson and Barnes-Holmes’ (2015) meta-analysis. Its stated purpose was to (a) examine the extent to which Vahey et al.’s (2015) meta-analysis contains errors; and (b) to test how computationally reproducible it is by current standards of best practice. Hussey identified a small number of minor numerical errors, but crucially was unable to exactly replicate the original meta-effect of 𝑟̅ = .45. Six different variations of the meta-analysis reported by Vahey et al. were used and obtained meta-effects that deviated from the original by Δ𝑟̅ = .01-.02. Hussey also reported corresponding 95% credibility intervals that were all of zero width. These discrepancies prompted the present authors to conduct a detailed audit of the original meta-analysis. This revealed one minor transposing error in addition to three identified by Hussey. Once corrected this resulted in a marginally increased Hunter and Schmidt meta-analytic effect of 𝑟̅ = .46 without a credibility interval, and a Hedges-Vevea meta-effect of 𝑟̅ = .47 with 95% confidence interval (.40, .54). This correction was too small to have any bearing on Vahey et al.’s supplementary analyses regarding publication bias or statistical power. Vahey et al. contained a much lower proportion of transposing errors than is typical of meta-analyses even still (cf. Kadlec et al., 2023; Lakens, 2016; Lakens et al., 2017). Nonetheless, Hussey highlighted important ambiguities about the theoretical and practical meaning of the meta-effect reported by Vahey et al. We clarify our position on these matters in summary, and in so doing explain why we believe that the wider IRAP literature would undoubtedly benefit from increased adoption of contemporary open science standards.
U2 - 10.1016/j.jbtep.2024.102016
DO - 10.1016/j.jbtep.2024.102016
M3 - Article
SN - 0005-7916
SP - 102016
JO - Journal of Behavior Therapy and Experimental Psychiatry
JF - Journal of Behavior Therapy and Experimental Psychiatry
ER -