This week’s #OTalk is on the topic of developing outcome measures and will be hosted by Alison Laver-Fawcett (@alisonlaverfaw) from York St John University in the UK.
Here’s what Alison had to say:
In the early 1990s I embarked on developing and standardising an occupational therapy assessment. It was a daunting prospect as I had little idea of where to start! The process involved my undertaking several interrelated psychometric research studies and became the focus of my doctoral studies. Luckily I had a lot of excellent support, advise and mentorship on my test development journey and finally, around 5 years later, the test was published (the Structured Observational Test of Function, SOTOF, Laver and Powell, 1995). Since then I have been committed to supporting occupational therapists to use and develop standardised occupational therapy assessments and outcome measurement and so I am delighted to have been asked to host this #OTalk on ‘Developing Outcome measures’.
So why is this topic so important? Outcome measures are required to evaluate the effectiveness of occupational therapy services. Commissioners of services expect service evaluation evidence that draws upon the routine application of robust outcome measures. Outcome measurement is driven by both policy and professional standards. The College of Occupational Therapists began its 2013 Position Statement on ‘Occupational therapists’ use of standardised outcome measures’ with the following assertion:
‘The College of Occupational Therapists promotes the use of evidence-based outcome measures to demonstrate the delivery of high quality and effective occupational therapy services and to provide credible and reliable justification for the intervention that is delivered. Without accruing data from such sources the evidence-base to support the value of occupational therapy will fail to grow and the profession will be challenged to produce the robust information that will be essential to support future commissioning of occupational therapy services’ (p1).
There are quite few psychometric terms related to test development so here are a few definitions for students or as a reminder:
Reliability is ‘the extent to which the same measurements of individuals obtained under different conditions yield similar results,’ (Everitt, 2006, p.200).
Inter-rater reliability is the level of agreement between different raters administering the test (Bowers, 2014)
Test-retest reliability is the ‘correlation of scores obtained by the same person on two administrations of the same test and the consistency of this score over time,’ (Laver Fawcett, 2007, p.198).
Validity relates to whether the outcome measure assesses what it proposes to measure.
Content validity is ‘the degree to which the content of an … instrument is an adequate reflection of the construct to be measured’ (Mokkink et al, 2012: 9).
Face validity is the ‘degree to which (the items of) an … instrument indeed looks as though they are an adequate reflection of the construct to be measured’ (Mokkink et al, 2012: 9).
Whether you are an experienced researcher, a clinician or a student please join us on 2nd May for this #OTalk twitter chat and share your ideas and experience. It is never too early in your occupational therapy career to start engaging in the development of outcome measures. On the Occupational Therapy Programme at York St John University students have been collaborating with me to undertake psychometric studies for their final year dissertation (e.g. see Laver-Fawcett et al, 2016). You may have an idea for a new outcome measure, so where do you start with test development? You may be using an unstandardized outcome measure developed in your service and want to know how you would go about standardising it and checking it is a valid and reliable measure. Although you may not be embarking on developing a new outcome measure from scratch, you may still want or need to engage in related research and undertake psychometric studies. For example, you may identify an outcome measure developed in a different country and want to translate it or develop a culturally sensitive version or need to develop norms for your client population. Previous psychometric research may have had limitations that warrant replication of studies, for example existing studies of reliability or validity may have been conducted using small sample sizes and more robust evidence is required. You may want to evaluate the clinical utility of an outcome measure for your area of practice or understand your clients’ experience of undertaking the test.
Suggested talking points and discussion questions to focus our chat:
- If you were looking to develop an outcome measure what would it be and why?
- What factors should occupational therapists consider before deciding to develop an outcome measure?
- If you are using an unstandardized outcome measure, how would you go about standardising this?
- How can we check the reliability of a new or existing measure?
- How do we know if our outcome measure is really measuring what it was developed to measure?
- If occupational therapists are client centred why are there so few face validity studies of occupational therapy outcome measures?
Bowers D. (2014) Medical Statistics from Scratch: An Introduction for Health Professionals. 3rd ed. Chichester: Wiley.
College of Occupational Therapists (COT; 2013) Position statement: Occupational therapists use of standardized outcome measures. London: COT. Available from: https://www.cot.co.uk/sites/default/files/position_statements/public/COT-Position-Statement-measuring-outcomes.pdf (accessed 25 April 2017)
Everitt BS. (2006) Medical Statistics from A to Z: A Guide for Clinicians and Medical Students. Cambridge: Cambridge University Press.
Laver, A. J. & Powell, G. E. (1995). The Structured Observational Test of Function (SOTOF). Windsor: NFER-NELSON.
Laver-Fawcett AJ. (2007) Principles of Assessment and Outcome Measurement for Occupational Therapists and Physiotherapists: Theory, Skills and Application. Chichester: Wiley. Note: BAOT/COT members can access an electronic copy of this book at: http://lib.myilibrary.com.cot.idm.oclc.org/ProductDetail.aspx?id=83859
Laver-Fawcett A J, Brain L, Brodie C, Cardy L, Manaton L (2016) The Face Validity and Clinical Utility of the Activity Card Sort – United Kingdom (ACS-UK). British Journal of Occupational Therapy, 79(8) 492–504. doi:10.1177/0308022616629167. Available from: http://journals.sagepub.com/doi/abs/10.1177/0308022616629167 (accessed 25th April 2017).
Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM and de Vet HCW (2012) COSMIN checklist manual. Available at: http://www.cosmin.nl/images/upload/files/COSMIN%20checklist%20manual%20v9.pdf (accessed 25 April 2017).
Some related resources:
College of Occupational Therapists (nd). Resources to help you choose assessments and outcome measures. Available from: https://www.cot.co.uk/cot-library/assessments-and-outcome-measures (accessed 25 April 2017)
College of Occupational Therapists (nd). Developing an assessment tool or outcome measure. Available from: http://www.cot.co.uk/cot-library/developing-assessment-tool-or-outcome-measure (accessed 25 April 2017)
Laver-Fawcett, A J (2014) Routine standardised outcome measurement to evaluate the effectiveness of occupational therapy interventions: essential or optional? Ergoterapeuten, 4, 28-37. Available from: http://www.ergoterapeuten.no/Admin/Public/Download.aspx?file=Files%2fFiles%2fFagartikler%2foutcome.pdf (accessed 25th April 2017).
Laver-Fawcett AJ (2010). The importance of measuring outcomes, including patient reported outcome measures (PROMS). BAOT Lifelong learning event Slide Share. Available at: https://www.slideshare.net/baotcot/the-importance-ofmeasuringoutcomes (accessed 25 April 2017)
Link to a bibliography for Outcomes and Evaluation of Occupational Therapy: https://lsbu.rl.talis.com/lists/ABE8C0F1-30D2-60C4-137E-3F7348999C39/bibliography.html (accessed 25 April 2017)
Post chat transcript
285 Avg Tweets/Hour
9 Avg Tweets/Participant