#OTalk 2nd May 2017 – Developing Outcome Measures.

This week’s #OTalk is on the topic of developing outcome measures and will be hosted by Alison Laver-Fawcett (@alisonlaverfaw) from York St John University in the UK.

Here’s what Alison had to say:

In the early 1990s I embarked on developing and standardising an occupational therapy assessment. It was a daunting prospect as I had little idea of where to start! The process involved my undertaking several interrelated psychometric research studies and became the focus of my doctoral studies. Luckily I had a lot of excellent support, advise and mentorship on my test development journey and finally, around 5 years later, the test was published (the Structured Observational Test of Function, SOTOF, Laver and Powell, 1995). Since then I have been committed to supporting occupational therapists to use and develop standardised occupational therapy assessments and outcome measurement and so I am delighted to have been asked to host this #OTalk on ‘Developing Outcome measures’.

So why is this topic so important? Outcome measures are required to evaluate the effectiveness of occupational therapy services. Commissioners of services expect service evaluation evidence that draws upon the routine application of robust outcome measures. Outcome measurement is driven by both policy and professional standards. The College of Occupational Therapists began its 2013 Position Statement on ‘Occupational therapists’ use of standardised outcome measures’ with the following assertion:

‘The College of Occupational Therapists promotes the use of evidence-based outcome measures to demonstrate the delivery of high quality and effective occupational therapy services and to provide credible and reliable justification for the intervention that is delivered. Without accruing data from such sources the evidence-base to support the value of occupational therapy will fail to grow and the profession will be challenged to produce the robust information that will be essential to support future commissioning of occupational therapy services’ (p1).

There are quite few psychometric terms related to test development so here are a few definitions for students or as a reminder:

Reliability is ‘the extent to which the same measurements of individuals obtained under different conditions yield similar results,’ (Everitt, 2006, p.200).

Inter-rater reliability is the level of agreement between different raters administering the test (Bowers, 2014)

Test-retest reliability is the ‘correlation of scores obtained by the same person on two administrations of the same test and the consistency of this score over time,’ (Laver Fawcett, 2007, p.198).

Validity relates to whether the outcome measure assesses what it proposes to measure.

Content validity is ‘the degree to which the content of an … instrument is an adequate reflection of the construct to be measured’ (Mokkink et al, 2012: 9).

Face validity is the ‘degree to which (the items of) an … instrument indeed looks as though they are an adequate reflection of the construct to be measured’ (Mokkink et al, 2012: 9).

Whether you are an experienced researcher, a clinician or a student please join us on 2nd May for this #OTalk twitter chat and share your ideas and experience. It is never too early in your occupational therapy career to start engaging in the development of outcome measures. On the Occupational Therapy Programme at York St John University students have been collaborating with me to undertake psychometric studies for their final year dissertation (e.g. see Laver-Fawcett et al, 2016). You may have an idea for a new outcome measure, so where do you start with test development? You may be using an unstandardized outcome measure developed in your service and want to know how you would go about standardising it and checking it is a valid and reliable measure. Although you may not be embarking on developing a new outcome measure from scratch, you may still want or need to engage in related research and undertake psychometric studies. For example, you may identify an outcome measure developed in a different country and want to translate it or develop a culturally sensitive version or need to develop norms for your client population. Previous psychometric research may have had limitations that warrant replication of studies, for example existing studies of reliability or validity may have been conducted using small sample sizes and more robust evidence is required. You may want to evaluate the clinical utility of an outcome measure for your area of practice or understand your clients’ experience of undertaking the test.

Suggested talking points and discussion questions to focus our chat:

  1. If you were looking to develop an outcome measure what would it be and why?
  2. What factors should occupational therapists consider before deciding to develop an outcome measure?
  3. If you are using an unstandardized outcome measure, how would you go about standardising this?
  4. How can we check the reliability of a new or existing measure?
  5. How do we know if our outcome measure is really measuring what it was developed to measure?
  6. If occupational therapists are client centred why are there so few face validity studies of occupational therapy outcome measures?

References

Bowers D. (2014) Medical Statistics from Scratch: An Introduction for Health Professionals. 3rd ed. Chichester: Wiley.

College of Occupational Therapists (COT; 2013) Position statement: Occupational therapists use of standardized outcome measures. London: COT. Available from: https://www.cot.co.uk/sites/default/files/position_statements/public/COT-Position-Statement-measuring-outcomes.pdf (accessed 25 April 2017)

Everitt BS. (2006) Medical Statistics from A to Z: A Guide for Clinicians and Medical Students. Cambridge: Cambridge University Press.

Laver, A. J. & Powell, G. E. (1995). The Structured Observational Test of Function (SOTOF). Windsor: NFER-NELSON.

Laver-Fawcett AJ. (2007) Principles of Assessment and Outcome Measurement for Occupational Therapists and Physiotherapists: Theory, Skills and Application. Chichester: Wiley. Note: BAOT/COT members can access an electronic copy of this book at: http://lib.myilibrary.com.cot.idm.oclc.org/ProductDetail.aspx?id=83859

 Laver-Fawcett A J, Brain L, Brodie C, Cardy L, Manaton L (2016) The Face Validity and Clinical Utility of the Activity Card Sort – United Kingdom (ACS-UK). British Journal of Occupational Therapy, 79(8) 492–504. doi:10.1177/0308022616629167. Available from: http://journals.sagepub.com/doi/abs/10.1177/0308022616629167 (accessed 25th April 2017).

Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM and de Vet HCW (2012) COSMIN checklist manual. Available at: http://www.cosmin.nl/images/upload/files/COSMIN%20checklist%20manual%20v9.pdf (accessed 25 April 2017).

Some related resources:

College of Occupational Therapists (nd). Resources to help you choose assessments and outcome measures. Available from: https://www.cot.co.uk/cot-library/assessments-and-outcome-measures (accessed 25 April 2017)

College of Occupational Therapists (nd). Developing an assessment tool or outcome measure. Available from: http://www.cot.co.uk/cot-library/developing-assessment-tool-or-outcome-measure (accessed 25 April 2017)

Laver-Fawcett, A J (2014) Routine standardised outcome measurement to evaluate the effectiveness of occupational therapy interventions: essential or optional? Ergoterapeuten, 4, 28-37. Available from: http://www.ergoterapeuten.no/Admin/Public/Download.aspx?file=Files%2fFiles%2fFagartikler%2foutcome.pdf (accessed 25th April 2017).

Laver-Fawcett AJ (2010). The importance of measuring outcomes, including patient reported outcome measures (PROMS). BAOT Lifelong learning event Slide Share. Available at: https://www.slideshare.net/baotcot/the-importance-ofmeasuringoutcomes (accessed 25 April 2017)

Link to a bibliography for Outcomes and Evaluation of Occupational Therapy: https://lsbu.rl.talis.com/lists/ABE8C0F1-30D2-60C4-137E-3F7348999C39/bibliography.html (accessed 25 April 2017)

Post chat transcript

Online transcript

The Numbers

687,039 Impressions
356 Tweets
41 Participants
285 Avg Tweets/Hour
Avg Tweets/Participant

#OTalk Participants

Advertisements

#OTalk 25th April – Media Club- Being human on the internet.

This weeks #OTalk is a media club on the topic of being human on the internet by Bill Wong (@BillwongOT).

The link for this week’s media club can be found here. https://www.youtube.com/watch?v=nm0p2NQPB-M

Here’s what Bill had to say…

As a frequent attendee to occupational therapy conferences across the world since I was an OT student in 2009, I am aware that there have been constant discussions and activities on promoting the occupational therapy profession on the Internet. For example, major OT conferences have been tracking how their conference hashtags are doing on Twitter. Another example will be hashtags such as #ot365, #valueofot, #otdistinctvalue, #otphoto, #otmonth, #otweek.

Then, when occupational therapy made appearances on the Ellen show (a popular daytime afternoon show in the US) and Jeopardy (popular quiz show), it has gone viral on social media. Finally, the 2012 Eleanor Slagle lecture by Dr. Karen Jacobs is a notable scholarly speech on promoting occupational therapy. One of the byproducts is that there has been an increasing number of OT programs globally have introduced to their students the idea of Twitter as a professional development tool.

That said, despite all the buzz about promoting our great profession, I feel there have been two items that are not addressed as much. First, although many of us still feel we need to increase public awareness of our profession, it has been a challenge for many OT students and practitioners to constantly promote our profession on social media due to our busy schedules, social media competencies aside. Second, this is something I feel we have not addressed too much in our OT education, is to promote our profession with effectiveness. This represents the perfect transition of this week’s media club material and my reflection of it for the blog.

I started out my OT journey as a wallflower in OT school. I preferred to be in my own world more than in the company of others. Although I connected with majority of my classmates on Facebook, I did primarily for survival reasons, since I was below average for many of my OT classes in OT school. Meanwhile, since I knew OT was my second chance at a career, I also was trying to make network better than my undergrad days (which I did virtually no networking) so that I wouldn’t repeat my struggles of obtaining employment again. At that time, to the surprise of many people who know me now, I actually merely wanted to be a qualified occupational therapist. Although I had ambitions to achieve great things, I actually had not much idea on social media’s role in the profession. Also, not surprisingly, I still was searching for my professional identity as a member of the OT community.

As many people know, summer 2010 was a crucial point of my life personally and professionally. Making a long story short, finding out that I have Asperger’s Syndrome turned out to be a key foundation for my professional identity. After going through the grieving process through the help of some of my OT colleagues, I realized whatever I share about my lived experiences has immense value if I were courageous enough to share regularly. The kicker was that I must pass my placements and become a qualified occupational therapist. However, it was a difficult time for me because in the process of discovering my diagnosis, I walked out of my first hands-on placement. Because of that, I was in a position where I couldn’t afford to fail any more of such placements, as the consequence of walking out under most circumstances is a fail.

Then, when I became a qualified occupational therapist in fall 2012, I have decided to come full circle to support other OT peers while trying to expand my horizons professionally. At that time, I had received some positive feedback from the local OT community in California regarding my courage to share my lived experiences in autism. Also, I started to build my international connections in OT through participation of our great Twitter chat series. Finally, with the great Sarah Bodell as my preceptor for my OTD project, I was introduced to OT 24 Hour Virtual Exchange for the first time. All that added up to my boldness to share my personal insights from an OT perspective on social media.

Like many newly qualified occupational therapists, however, I had bouts of struggles of finding my voice. My primary struggles came from trying to balance out my two primary occupational roles- as an emerging autism self advocate and an occupational therapist. After all, although there were common grounds between the two roles on autism related issues, mastering the appropriate tones of what I say on social media was the most difficult challenge (which is not surprising given my autism diagnosis). The next most difficult challenge was about sharing my views on autism related subjects. I am thankful for my loyal peers in the OT profession for being patient through my trials and errors on finding my voice. Once I have progressed towards mastery, my social media presence exploded, which was part of the reasons why I was selected to do 2 TEDx Talks. Meanwhile, I have also set a continuous CPD goal since 2014 where I will try to master at least 1 new social media platform per year. Of course, I keep on breaking my Twitter impression records at OT conferences. That said, even though I understood the rationale of my social media related CPD goals, I have received mixed reviews from my peers when I mentioned these goals. The reactions I most often heard was “You are way up there with what you already do social media wise. I can’t even keep up with you. I don’t even understand how you do it, let alone you want to continue to innovate without losing your human touch, in spite you are working 40 or more hours a week.”

In researching for this topic, I also found this interesting fact. We all lie somewhere in the technology adaptation spectrum. 2.5% are innovators, 13.5% are early adopters, 34% are early majority, 34% are late majority, and 16% are laggards. (Source: http://www.ondigitalmarketing.com/learn/odm/foundations/5-customer-segments-technology-adoption/)

That said, watching this presentation has prompted me to create these discussion questions for our chat.

  • What social media platforms do you have? For each platform, what purposes are you using it for?
  • In regards to social media ideas, do you consider yourself an innovator, early adopter, early majority, late majority, or laggard? Why?
  • On a scale of 1-10, how important is online presence in having success in the OT profession? Why?
  • What qualities are you looking for on an influential OT student or practitioner online?
  • On a scale of 1-10, how good are you trying to build rapport with your social media/online followers? Why?
  • On a scale of 1-10, how well do you think you utilize your professional social media accounts (whether it’s for CPD or promoting OT)? Why?

Post Chat Information

online transcript

The Numbers

1,734,327 Impressions
640 Tweets
58 Participants
512 Avg Tweets/Hour
11 Avg Tweets/Participant

#OTalk Participants

#OTalk 18th April – When clients don’t get better.

This weeks #Otalk is on the topic of “When clients don’t get better” and will be hosted by Keir Harding (@Keirwales).

Here’s what Keir has to say….

“With patients who do not get better, or who even get worse in spite of long devoted care, major strain may arise.  Those who attend the patient are then pleased neither with him (the patient) nor themselves and the quality of their concern for him alters accordingly, with consequences that can be severe for both patient and attendants”  (Main 1957).

Our training tends to prepare us for people who will come to see us with a problem, take our advice, then go away and get better.  The reality is that some people who are referred won’t turn up to appointments, some will ignore our advice, some won’t improve despite our best intentions and some will do things that seem to make their situation significantly worse.  I have sat in offices cursing people for not turning up and making judgements in my head about whether they want to get better.  On the other side of the scale I’ve worked with people who have repeatedly self-harmed in ways that are potentially lethal and made repeated suicide attempts.  I often hear this described as ‘attention seeking’ or ‘sabotage’.  These explanations are seductively simple.  After reading in someone’s notes the other week that they were self-harming ‘due to their diagnosis’, I wanted to spend some time promoting a more nuanced way of thinking about the things people do that seem to cause harm. I wonder whether we all do something on the self-harm spectrum and whether there is an impact of people we are there to ‘make better’ apparently making themselves worse.  To set the scene here is a paragraph ripped from my blog.  It would also be worth checking out “The Dark Side of Occupation” (Twinley 2013) which you can get free if you’re in BAOT.

“Let’s start by saying that I self-harm.  I self-harm regularly in a way that society tends to approve of.  Most Saturdays  I strap on my rugby boots and on a good day,  for 80 minutes large, hairy men will charge at me while I try to knock them over.  On other days I am punched, stamped on, scraped with studs, or just hurt.  Over the years I have broken my nose, chipped my teeth, ripped the skin under my chin open, split my forehead and all last week, sported a big purple eye.  I play rugby every week, not seeking pain but knowing full well that it is inherent in this activity.  The pain and damage that it gives me is worth it in terms of the other benefits that I receive. Now obviously playing rugby isn’t the same as cutting lines in my thigh, but I’m arguing that that both activities are on a spectrum of things that damage you but come with some reward that makes it worthwhile.”

Suggested Talking points:

1 What do we do in our own lives that isn’t totally in our best interests?

2 How do we understand what our client’s do that seems to make things worse?

3 What is our role in working with these clients?  Do we help them stop or do we facilitate harm?

4 What is the impact on us of our clients not getting better? (Or seemingly making things worse?)

5 How do we protect ourselves from making simple explanations to complex occupational problems? (Often in systems that will tend to reinforce that)

References:

MAIN, T. F. (1957), THE AILMENT*. British Journal of Medical Psychology, 30: 129–145. doi:10.1111/j.2044-8341.1957.tb01193.x (Free online at http://www.ljaa.lv/download/dokumenti/the_ailment_by_t_main.pdf )

Twinley, R. (2013), The dark side of occupation: A concept for consideration. Aust Occup Ther J, 60: 301–303. doi:10.1111/1440-1630.12026

https://thediagnosisofexclusion.wordpress.com/2017/02/24/use-a-little-restraint-why-people-who-self-harm-must-be-forced-to-stop/

Post Chat

Online Transcript

The Numbers

996,404 Impressions
501 Tweets
60 Participants
401 Avg Tweets/Hour
Avg Tweets/Participant

OTalk Participants

#OTalk 11th April – Occupational Therapy and helping Seniors age in place.

This weeks #Otalk is on the topic of OT helping seniors to age in place and will be hosted by Julie Entwistle (@entwistlepower).

Here is what she has to say…

According to a report by the Canadian Medical Association, 63% of Canadians selected home and community care for older adults as a top priority.  Meaning that as baby boomers age and the over 65 population grows, home care is becoming an even larger industry that may put a strain on healthcare.

How can Occupational Therapists ease the strain on the healthcare system and how can we, as OT’s, capitalize on the senior market?

In a previous post on our blog, Occupational Therapy and Aging in Place, we discussed the top ways OT’s can assist older adults in their plan to Age well at Home which include:

  1. Proactivity
  2. Space Modifications
  3. Fall Prevention
  4. Preparing for and Being open to relocation
  5. Planning ahead for assistance
  6. Planning ahead for emergencies

Questions to discuss:

  1. What other aspects to you believe should be included in this list?
  2. What do you do to proactively meet the needs of older adults who want to age in place?
  3. Falls prevention:  beyond space modification, how are you educating to prevent falls in seniors?
  4. What fall prevention techniques do you use?
  5. How do you broach the difficult conversations with older clients including:  relocation to a safer home, long term care, ceasing of driving, etc
  6. How do you deal with adult children who have differing opinions to those of their aging parents?
  7. What tips and strategies do you provide to caregivers to help them learn about their role and to avoid burnout?
  8. Are there any apps/technology that you use with your senior clients to help them manage at home?

Post chat

online transcript

The Numbers

1,109,479 Impressions
338 Tweets
36 Participants
270 Avg Tweets/Hour
9Avg  Tweets/Participant

#OTalk Participants