Assessing Mental Skill and Technique Use in Applied Interventions: Recognizing and Minimizing Threats to the Psychometric Properties of the TOPS

Charlotte Woodcock, Joan L. Duda, Jennifer Cumming, Lee-Ann Sharp, Mark J.G. Holland

    Research output: Contribution to journalArticle

    5 Citations (Scopus)

    Abstract

    Drawing from the experiences of the authors in developing, conducting, and evaluating sport psychology interventions, several considerations are highlighted and recommendations offered for effective psychometric assessment. Using the Test of Performance Strategies (TOPS; Thomas, Murphy, & Hardy, 1999) as a working example, opportunities for bias to undermine a measure’s validity and reliability are discussed with reference to a respondent’s four cognitive processes: (a) comprehension, (b) retrieval, (c) decision-making, and (d) response generation. Further threats to an instrument’s psychometric properties are highlighted in the form of demand characteristics athletes perceive in the environment. With these concerns in mind, several recommendations are made relating to the process of questionnaire administration and how possible compromises to the psychometric soundness of measures used in applied interventions can be minimized.
    LanguageEnglish
    Pages1-15
    JournalThe Sport Psychologist
    Volume26
    Publication statusPublished - 2012

    Fingerprint

    Psychometrics
    Reproducibility of Results
    Athletes
    Decision Making
    TOPS
    Surveys and Questionnaires

    Cite this

    @article{dc68753c01854bf881583b4166b996eb,
    title = "Assessing Mental Skill and Technique Use in Applied Interventions: Recognizing and Minimizing Threats to the Psychometric Properties of the TOPS",
    abstract = "Drawing from the experiences of the authors in developing, conducting, and evaluating sport psychology interventions, several considerations are highlighted and recommendations offered for effective psychometric assessment. Using the Test of Performance Strategies (TOPS; Thomas, Murphy, & Hardy, 1999) as a working example, opportunities for bias to undermine a measure’s validity and reliability are discussed with reference to a respondent’s four cognitive processes: (a) comprehension, (b) retrieval, (c) decision-making, and (d) response generation. Further threats to an instrument’s psychometric properties are highlighted in the form of demand characteristics athletes perceive in the environment. With these concerns in mind, several recommendations are made relating to the process of questionnaire administration and how possible compromises to the psychometric soundness of measures used in applied interventions can be minimized.",
    author = "Charlotte Woodcock and Duda, {Joan L.} and Jennifer Cumming and Lee-Ann Sharp and Holland, {Mark J.G.}",
    note = "Reference text: Anderson, A.G., Miles, A., Mahoney, C., & Robinson, P. (2002). Evaluating the effectiveness of applied sport psychology practice: Making the case for the case study approach. The Sport Psychologist, 16, 432–453. Baumeister, R.F. (2010). The self. In R.F. Baumeister & E.J. Finkel (Eds.), Advanced social psychology: The state of science (pp. 139–175). New York, NY: Oxford University Press. Beckmann, J., & Kellmann, M. (2003). Procedures and principles of sport psychological assessment. The Sport Psychologist, 17, 338–350. Brener, N.D., Billy, J.O.G., & Grady, W.R. (2003). Assessment of factors affecting the validity of self-reported health-risk behavior among adolescents: Evidence from the scientific literature. The Journal of Adolescent Health, 33, 436–457. Brewer, B.W., & Shillinglaw, R. (1992). Evaluation of a psychological skills training workshop for male intercollegiate lacrosse players. The Sport Psychologist, 6, 139–147. Choi, B.C.K., & Pak, A.W.P. (2005). A catalogue of biases in questionnaires. Preventing Chronic Disease, 2, 1–13. Cohen, A.B., Tenenbaum, G., & English, R.W. (2006). Emotions and golf performance: An IZOF-based applied sport psychology case study. Behavior Modification, 30, 259–280. Conroy, D.E., Kaye, M.P., & Schantz, L.H. (2008). Quantitative research methodology. In T.S. Horn (Ed.), Advances in sport psychology (3rd ed., pp. 15–30). Leeds: Human Kinetics. Dietrich, H., & Ehrlenspiel, F. (2010). Cognitive interviewing: A qualitative tool for improving questionnaires in sport science. Measurement in Physical Education and Exercise Science, 14, 51–60. Duda, J.L., Cumming, J., & Balaguer, I. (2005). Enhancing athletes’ self regulation, task involvement, and self determination via psychological skills training. In D. Hackfort, J.L. Duda, & R. Lidor (Eds.), Handbook of applied sport psychology research (pp. 143–165). Morgantown, WV: Fitness Information Technology. Filby, W.C.D., Maynard, I.W., & Graydon, J.K. (1999). The effect of multiple-goal strategies on performance outcomes in training and competition. Journal of Applied Sport Psychology, 11, 230–246. Fletcher, D., & Hanton, S. (2001). The relationship between psychological skills usage and competitive anxiety responses. Psychology of Sport and Exercise, 2, 89–101. Gravetter, F.J., & Forzano, L.B. (2008). Research methods for the behavioral sciences (Vol. 3, 3rd ed.). Belmont, CA: Wadsworth Cengage Learning. Hagger, M.S., & Chatzisarantis, N.L.D. (2009). Assumptions in research in sport and exercise psychology. Psychology of Sport and Exercise, 10, 511–519. Hardy, J. (2006). Speaking clearly: A critical review of the self-talk literature. Psychology of Sport and Exercise, 7, 81–97. Hardy, L., Roberts, R., Thomas, P.R., & Murphy, S.M. (2010). Test of performance strategies (TOPS): Instrument refinement using confirmatory factor analysis. Psychology of Sport and Exercise, 11, 27–35. Holland, M.J.G., Sharp, L., Woodcock, C., Cumming, J., & Duda, J.L. (2010a, October). Evaluating MST research procedures: Observations, reflections, and progressions. In J. L. Duda (Discussant) An action research approach to mental skills training: Experience, evaluation and evolution. Symposium conducted at the meeting of the Association for Applied Sport Psychology, Providence, RI. Holland, M.J.G., Sharp, L., Woodcock, C., Cumming, J., & Duda, J.L. (2010b). Validity and reliability of the Behavioral Regulation in Sport Questionnaire (BRSQ) with youth athletes. Journal of Sport & Exercise Psychology, 32, s175–s176. Howorka, K., Pumprla, J., Schlusche, C., Wagner-Nosiska, D., Schabmann, A., & Bradley, C. (2000). Dealing with ceiling baseline treatment satisfaction level in patients with diabetes under flexible, functional insulin treatment: Assessment of improvements in treatment satisfaction with a new insulin analogue. Quality of Life Research, 9, 915–930. Huang, C., Liao, H., & Chang, S. (1998). Social desirability and the Clinical Self-Report Inventory: methodological reconsideration. Journal of Clinical Psychology, 54, 517–528. Jobe, J.B. (2000). Cognitive processes in self-report. In A. Stone, C.A. Bachrach, J.S. Turkkan, J.B. Jobe, H.S. Kurtzman, & V.S. Cain (Eds.), The science of self-report: Implications for research and practice (pp. 25–28). Mahwah, NJ: Lawrence Erlbaum Associates. Kirschenbaum, D.S., Owens, D., & O’Connor, E.A. (1998). Smart golf: Preliminary evaluation of a simple, yet comprehensive, approach to improving and scoring the mental game. The Sport Psychologist, 12, 271–282. Lonsdale, C., Hodge, K., & Rose, E.A. (2008). The Behavioural Regulation in Sport Questionnaire (BRSQ): Instrument development and initial validity evidence. Journal of Sport & Exercise Psychology, 30, 323–355. Markus, H. (1977). Self-schemata and processing information about the self. Journal of Personality and Social Psychology, 35, 63–78. McCarthy, P.J., Jones, M.V., Harwood, C.G., & Olivier, S. (2010). What do athletes implicitly understand about psychological skills? Journal of Clinical Sports Psychology, 4, 158–172. Morsbach, S.K., & Prinz, R.J. (2006). Understanding and improving the validity of self-report of parenting. Clinical Child and Family Psychology Review, 9, 1–21. Podsakoff, P.M., MacKenzie, S.B., Lee, J.Y., & Podsakoff, N.P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Sport Psychology, 89, 879–903. Ravizza, K. (2006). Increasing awareness for sport performance. In J.M. Williams (Ed.), Applied sport psychology: Personal growth to peak performance (5th ed., pp. 228–239). New York: McGraw-Hill. Raedeke, T.D., & Smith, A.L. (2001). Development and preliminary validation of an athlete burnout measure. Journal of Sport & Exercise Psychology, 23, 281–306. Schober, M.F., Conrad, F.G., & Fricker, S.S. (2004). Misunderstanding standardized language in research interviews. Applied Cognitive Psychology, 18, 169–188. Schutz, R.W. (1998). Assessing the stability of psychological traits and measures. In J.L. Duda (Ed.), Advances in sport and exercise psychology measurement (pp. 393–408). Morgantown, WV: Fitness Information Technology. Sharp, L., Woodcock, C., Holland, M.J.G., Duda, J.L., & Cumming, J. (2010a, October). Youth athletes responses to a mental skills training program. In J. L. Duda (Discussant), An action research approach to mental skills training: Experience, evaluation, and evolution. Symposium conducted at the meeting of the Association for Applied Sport Psychology, Providence, RI. Sharp, L., Woodcock, C., Holland, M.J.G., Duda, J.L., & Cumming, J. (2010b). Validation of the Athlete Burnout Questionnaire with youth athletes. Journal of Sport & Exercise Psychology, 32, s218–s219. Tenenbaum, G., Kamata, A., & Hayashi, K. (2007). Measurement in sport and exercise psychology: A new outlook on selected issues of reliability and validity. In G. Tenenbaum & R.C. Eklund (Eds.), Handbook of Sport Psychology (3rd ed., pp. 757–773). Hoboken, NJ: John Wiley & Sons. Thomas, P.R., Murphy, S.M., & Hardy, L. (1999). Test of performance strategies: Development and preliminary validation of a comprehensive measure of athletes’ psychological skills. Journal of Sports Sciences, 17, 697–711. Vealey, R.S. (1988). Future directions in psychological skills training. The Sport Psychologist, 2, 318–336. Vealey, R.S., & Garner-Holman, M. (1998). Applied sport psychology: Measurement issues.In J.L. Duda (Ed.), Advances in sport and exercise psychology measurement (pp. 433–446). Morgantown, WV: Fitness Information Technology. Willis, G.B. (2005). Cognitive interviewing: A tool for improving questionnaire design. London, UK: Sage. Woodcock, C., Holland, M.J.G., Sharp, L., Duda, J.L., & Cumming, J. (2010a). Psychological skill use in adolescents: Exploring the structural and temporal validity of the TOPS. Journal of Sport & Exercise Psychology, 32, s233. Woodcock, C., Holland, M.J.G., Sharp, L., Duda, J.L., & Cumming, J. (2010b, October). Origins and evolution: Coming full circle with mental skills training. In J. L. Duda (Discussant), An action research approach to mental skills training: Experience, evaluation, and evolution. Symposium conducted at the meeting of the Association for Applied Sport Psychology, Providence, RI. Young, B.W., & Starkes, J.L. (2006). Measuring outcomes of swimmers’ non-regulation during practice: Relationships between self-report, coaches’ judgments, and video observation. International Journal of Sports Science & Coaching, 1, 131–148.",
    year = "2012",
    language = "English",
    volume = "26",
    pages = "1--15",
    journal = "Sport Psychologist",
    issn = "0888-4781",

    }

    Assessing Mental Skill and Technique Use in Applied Interventions: Recognizing and Minimizing Threats to the Psychometric Properties of the TOPS. / Woodcock, Charlotte; Duda, Joan L.; Cumming, Jennifer; Sharp, Lee-Ann; Holland, Mark J.G.

    In: The Sport Psychologist, Vol. 26, 2012, p. 1-15.

    Research output: Contribution to journalArticle

    TY - JOUR

    T1 - Assessing Mental Skill and Technique Use in Applied Interventions: Recognizing and Minimizing Threats to the Psychometric Properties of the TOPS

    AU - Woodcock, Charlotte

    AU - Duda, Joan L.

    AU - Cumming, Jennifer

    AU - Sharp, Lee-Ann

    AU - Holland, Mark J.G.

    N1 - Reference text: Anderson, A.G., Miles, A., Mahoney, C., & Robinson, P. (2002). Evaluating the effectiveness of applied sport psychology practice: Making the case for the case study approach. The Sport Psychologist, 16, 432–453. Baumeister, R.F. (2010). The self. In R.F. Baumeister & E.J. Finkel (Eds.), Advanced social psychology: The state of science (pp. 139–175). New York, NY: Oxford University Press. Beckmann, J., & Kellmann, M. (2003). Procedures and principles of sport psychological assessment. The Sport Psychologist, 17, 338–350. Brener, N.D., Billy, J.O.G., & Grady, W.R. (2003). Assessment of factors affecting the validity of self-reported health-risk behavior among adolescents: Evidence from the scientific literature. The Journal of Adolescent Health, 33, 436–457. Brewer, B.W., & Shillinglaw, R. (1992). Evaluation of a psychological skills training workshop for male intercollegiate lacrosse players. The Sport Psychologist, 6, 139–147. Choi, B.C.K., & Pak, A.W.P. (2005). A catalogue of biases in questionnaires. Preventing Chronic Disease, 2, 1–13. Cohen, A.B., Tenenbaum, G., & English, R.W. (2006). Emotions and golf performance: An IZOF-based applied sport psychology case study. Behavior Modification, 30, 259–280. Conroy, D.E., Kaye, M.P., & Schantz, L.H. (2008). Quantitative research methodology. In T.S. Horn (Ed.), Advances in sport psychology (3rd ed., pp. 15–30). Leeds: Human Kinetics. Dietrich, H., & Ehrlenspiel, F. (2010). Cognitive interviewing: A qualitative tool for improving questionnaires in sport science. Measurement in Physical Education and Exercise Science, 14, 51–60. Duda, J.L., Cumming, J., & Balaguer, I. (2005). Enhancing athletes’ self regulation, task involvement, and self determination via psychological skills training. In D. Hackfort, J.L. Duda, & R. Lidor (Eds.), Handbook of applied sport psychology research (pp. 143–165). Morgantown, WV: Fitness Information Technology. Filby, W.C.D., Maynard, I.W., & Graydon, J.K. (1999). The effect of multiple-goal strategies on performance outcomes in training and competition. Journal of Applied Sport Psychology, 11, 230–246. Fletcher, D., & Hanton, S. (2001). The relationship between psychological skills usage and competitive anxiety responses. Psychology of Sport and Exercise, 2, 89–101. Gravetter, F.J., & Forzano, L.B. (2008). Research methods for the behavioral sciences (Vol. 3, 3rd ed.). Belmont, CA: Wadsworth Cengage Learning. Hagger, M.S., & Chatzisarantis, N.L.D. (2009). Assumptions in research in sport and exercise psychology. Psychology of Sport and Exercise, 10, 511–519. Hardy, J. (2006). Speaking clearly: A critical review of the self-talk literature. Psychology of Sport and Exercise, 7, 81–97. Hardy, L., Roberts, R., Thomas, P.R., & Murphy, S.M. (2010). Test of performance strategies (TOPS): Instrument refinement using confirmatory factor analysis. Psychology of Sport and Exercise, 11, 27–35. Holland, M.J.G., Sharp, L., Woodcock, C., Cumming, J., & Duda, J.L. (2010a, October). Evaluating MST research procedures: Observations, reflections, and progressions. In J. L. Duda (Discussant) An action research approach to mental skills training: Experience, evaluation and evolution. Symposium conducted at the meeting of the Association for Applied Sport Psychology, Providence, RI. Holland, M.J.G., Sharp, L., Woodcock, C., Cumming, J., & Duda, J.L. (2010b). Validity and reliability of the Behavioral Regulation in Sport Questionnaire (BRSQ) with youth athletes. Journal of Sport & Exercise Psychology, 32, s175–s176. Howorka, K., Pumprla, J., Schlusche, C., Wagner-Nosiska, D., Schabmann, A., & Bradley, C. (2000). Dealing with ceiling baseline treatment satisfaction level in patients with diabetes under flexible, functional insulin treatment: Assessment of improvements in treatment satisfaction with a new insulin analogue. Quality of Life Research, 9, 915–930. Huang, C., Liao, H., & Chang, S. (1998). Social desirability and the Clinical Self-Report Inventory: methodological reconsideration. Journal of Clinical Psychology, 54, 517–528. Jobe, J.B. (2000). Cognitive processes in self-report. In A. Stone, C.A. Bachrach, J.S. Turkkan, J.B. Jobe, H.S. Kurtzman, & V.S. Cain (Eds.), The science of self-report: Implications for research and practice (pp. 25–28). Mahwah, NJ: Lawrence Erlbaum Associates. Kirschenbaum, D.S., Owens, D., & O’Connor, E.A. (1998). Smart golf: Preliminary evaluation of a simple, yet comprehensive, approach to improving and scoring the mental game. The Sport Psychologist, 12, 271–282. Lonsdale, C., Hodge, K., & Rose, E.A. (2008). The Behavioural Regulation in Sport Questionnaire (BRSQ): Instrument development and initial validity evidence. Journal of Sport & Exercise Psychology, 30, 323–355. Markus, H. (1977). Self-schemata and processing information about the self. Journal of Personality and Social Psychology, 35, 63–78. McCarthy, P.J., Jones, M.V., Harwood, C.G., & Olivier, S. (2010). What do athletes implicitly understand about psychological skills? Journal of Clinical Sports Psychology, 4, 158–172. Morsbach, S.K., & Prinz, R.J. (2006). Understanding and improving the validity of self-report of parenting. Clinical Child and Family Psychology Review, 9, 1–21. Podsakoff, P.M., MacKenzie, S.B., Lee, J.Y., & Podsakoff, N.P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Sport Psychology, 89, 879–903. Ravizza, K. (2006). Increasing awareness for sport performance. In J.M. Williams (Ed.), Applied sport psychology: Personal growth to peak performance (5th ed., pp. 228–239). New York: McGraw-Hill. Raedeke, T.D., & Smith, A.L. (2001). Development and preliminary validation of an athlete burnout measure. Journal of Sport & Exercise Psychology, 23, 281–306. Schober, M.F., Conrad, F.G., & Fricker, S.S. (2004). Misunderstanding standardized language in research interviews. Applied Cognitive Psychology, 18, 169–188. Schutz, R.W. (1998). Assessing the stability of psychological traits and measures. In J.L. Duda (Ed.), Advances in sport and exercise psychology measurement (pp. 393–408). Morgantown, WV: Fitness Information Technology. Sharp, L., Woodcock, C., Holland, M.J.G., Duda, J.L., & Cumming, J. (2010a, October). Youth athletes responses to a mental skills training program. In J. L. Duda (Discussant), An action research approach to mental skills training: Experience, evaluation, and evolution. Symposium conducted at the meeting of the Association for Applied Sport Psychology, Providence, RI. Sharp, L., Woodcock, C., Holland, M.J.G., Duda, J.L., & Cumming, J. (2010b). Validation of the Athlete Burnout Questionnaire with youth athletes. Journal of Sport & Exercise Psychology, 32, s218–s219. Tenenbaum, G., Kamata, A., & Hayashi, K. (2007). Measurement in sport and exercise psychology: A new outlook on selected issues of reliability and validity. In G. Tenenbaum & R.C. Eklund (Eds.), Handbook of Sport Psychology (3rd ed., pp. 757–773). Hoboken, NJ: John Wiley & Sons. Thomas, P.R., Murphy, S.M., & Hardy, L. (1999). Test of performance strategies: Development and preliminary validation of a comprehensive measure of athletes’ psychological skills. Journal of Sports Sciences, 17, 697–711. Vealey, R.S. (1988). Future directions in psychological skills training. The Sport Psychologist, 2, 318–336. Vealey, R.S., & Garner-Holman, M. (1998). Applied sport psychology: Measurement issues.In J.L. Duda (Ed.), Advances in sport and exercise psychology measurement (pp. 433–446). Morgantown, WV: Fitness Information Technology. Willis, G.B. (2005). Cognitive interviewing: A tool for improving questionnaire design. London, UK: Sage. Woodcock, C., Holland, M.J.G., Sharp, L., Duda, J.L., & Cumming, J. (2010a). Psychological skill use in adolescents: Exploring the structural and temporal validity of the TOPS. Journal of Sport & Exercise Psychology, 32, s233. Woodcock, C., Holland, M.J.G., Sharp, L., Duda, J.L., & Cumming, J. (2010b, October). Origins and evolution: Coming full circle with mental skills training. In J. L. Duda (Discussant), An action research approach to mental skills training: Experience, evaluation, and evolution. Symposium conducted at the meeting of the Association for Applied Sport Psychology, Providence, RI. Young, B.W., & Starkes, J.L. (2006). Measuring outcomes of swimmers’ non-regulation during practice: Relationships between self-report, coaches’ judgments, and video observation. International Journal of Sports Science & Coaching, 1, 131–148.

    PY - 2012

    Y1 - 2012

    N2 - Drawing from the experiences of the authors in developing, conducting, and evaluating sport psychology interventions, several considerations are highlighted and recommendations offered for effective psychometric assessment. Using the Test of Performance Strategies (TOPS; Thomas, Murphy, & Hardy, 1999) as a working example, opportunities for bias to undermine a measure’s validity and reliability are discussed with reference to a respondent’s four cognitive processes: (a) comprehension, (b) retrieval, (c) decision-making, and (d) response generation. Further threats to an instrument’s psychometric properties are highlighted in the form of demand characteristics athletes perceive in the environment. With these concerns in mind, several recommendations are made relating to the process of questionnaire administration and how possible compromises to the psychometric soundness of measures used in applied interventions can be minimized.

    AB - Drawing from the experiences of the authors in developing, conducting, and evaluating sport psychology interventions, several considerations are highlighted and recommendations offered for effective psychometric assessment. Using the Test of Performance Strategies (TOPS; Thomas, Murphy, & Hardy, 1999) as a working example, opportunities for bias to undermine a measure’s validity and reliability are discussed with reference to a respondent’s four cognitive processes: (a) comprehension, (b) retrieval, (c) decision-making, and (d) response generation. Further threats to an instrument’s psychometric properties are highlighted in the form of demand characteristics athletes perceive in the environment. With these concerns in mind, several recommendations are made relating to the process of questionnaire administration and how possible compromises to the psychometric soundness of measures used in applied interventions can be minimized.

    M3 - Article

    VL - 26

    SP - 1

    EP - 15

    JO - Sport Psychologist

    T2 - Sport Psychologist

    JF - Sport Psychologist

    SN - 0888-4781

    ER -