A Decade On: Reflecting on the Limitations of the First Meta-analysis of the Implicit Relational Assessment Procedure’s (IRAP) Criterion Validity in the Clinical Domain

Nigel Vahey, Emma Nicholson, Dermot Barnes-Holmes

Research output: Contribution to journalArticle

Abstract

Hussey (in press) recently conducted a detailed critical reanalysis of Vahey, Nicholson and Barnes-Holmes’ (2015) meta-analysis. Its stated purpose was to (a) examine the extent to which Vahey et al.’s (2015) meta-analysis contains errors; and (b) to test how computationally reproducible it is by current standards of best practice. Hussey identified a small number of minor numerical errors, but crucially was unable to exactly replicate the original meta-effect of 𝑟̅ = .45. Six different variations of the meta-analysis reported by Vahey et al. were used and obtained meta-effects that deviated from the original by Δ𝑟̅ = .01-.02. Hussey also reported corresponding 95% credibility intervals that were all of zero width. These discrepancies prompted the present authors to conduct a detailed audit of the original meta-analysis. This revealed one minor transposing error in addition to three identified by Hussey. Once corrected this resulted in a marginally increased Hunter and Schmidt meta-analytic effect of 𝑟̅ = .46 without a credibility interval, and a Hedges-Vevea meta-effect of 𝑟̅ = .47 with 95% confidence interval (.40, .54). This correction was too small to have any bearing on Vahey et al.’s supplementary analyses regarding publication bias or statistical power. Vahey et al. contained a much lower proportion of transposing errors than is typical of meta-analyses even still (cf. Kadlec et al., 2023; Lakens, 2016; Lakens et al., 2017). Nonetheless, Hussey highlighted important ambiguities about the theoretical and practical meaning of the meta-effect reported by Vahey et al. We clarify our position on these matters in summary, and in so doing explain why we believe that the wider IRAP literature would undoubtedly benefit from increased adoption of contemporary open science standards.
Original languageEnglish
Pages (from-to)102016
JournalJournal of Behavior Therapy and Experimental Psychiatry
Early online date15 Jan 2025
DOIs
Publication statusPublished online - 15 Jan 2025

Fingerprint

Dive into the research topics of 'A Decade On: Reflecting on the Limitations of the First Meta-analysis of the Implicit Relational Assessment Procedure’s (IRAP) Criterion Validity in the Clinical Domain'. Together they form a unique fingerprint.

Cite this