MANITOBA SPEECH-LANGUAGE PATHOLOGY OUTCOMES MEASURE REPORT - CASLPA COMMITTEE REPORT

The following is a report dated January 29, 2003 prepared by an ad hoc committee convened to review the Manitoba Outcomes Measure. Mark Robertson, the head of the project team has thanked the committee for its review and input and has reported that investigations and reports will be forthcoming to address issues of validity and reliability. A partnership to establish a national outcomes measure is desired by both parities. CASLPA will follow the progress made by the project and report to members.

The CASLPA Board of Directors would like to thank the chairs and committee members for their contribution to this committee and report.

Introduction:

This report reviews the Manitoba Speech-Language Pathology Outcome Measure, an instrument developed for Speech-Language Pathology (SLP) clients from birth to twenty-one years of age in all settings across Manitoba. A project team under the auspices of The Manitoba Education, Training and Youth, Program and Student Services Branch spearheaded the initiative to develop this tool. Items from ASHA's Functional Communication Measures, the Alberta Priority Rating Scale and the New Brunswick Priority Rating Scale were incorporated as part of this new tool. The instrument was initially trialed in 1998 by 25 Speech-Language Pathologists (SLPs) across Manitoba. Currently, there are 86 SLPs using the instrument on a voluntary basis across various regions and sectors in Manitoba. Practice settings include education, health and family services and housing in both urban and rural areas. To date, data has been collected for 3,521 individuals.

A Committee comprised of SLP members of the Canadian Association of Speech Language Pathologists and Audiologists prepared this report. (Appendix 1). The SLPs from both education and health care practices across urban and rural areas responded to a call to volunteer their time and expertise because of their interest in outcome measurement. Collectively this committee represents experts with paediatric and school age populations and outcome measurement development. Not all identified members of the committee were able to participate in both teleconferences and all related activities.

The committee was struck by CASLPA to provide a recommendation regarding the readiness of the Manitoba SLP Outcomes Measure for use by CASLPA SLP members. The Committee met via teleconference on two occasions and communicated via email from August to December 2002. A representative from the Manitoba SLP Outcomes Measure Project Group, Mark Robertson, Provincial Consultant for SLP provided all of the materials from which this review is based and ensured temporary access to the tool's web-site which included data entry and report generation capability. (Appendix 2). Mark also participated in one teleconference call to address specific questions.

The criteria used to review the Manitoba SLP Outcomes Measure was established by the Committee chairs and circulated to all members to apply independently. (Appendix 3) Each member assessed the components of the Manitoba SLP Outcomes Measure and provided feedback to the group. This review reflects the Committee's conclusion on each of the components and the overall instrument.

Overall Strengths:

The Committee identified several strengths:

• The Manitoba SLP Outcomes Measure is easily accessible over the web for data input and generation of a variety of reports related to demographic, caseload and performance data
  
  
• Both the conceptual framework behind this tool and its utilization have already received support and positive feedback from the community of SLPs in the province of Manitoba who have used this tool.
  
  
• Part of the instrument is scored using a seven point ordinal scale that has been utilized successfully in other SLP measures.
  
  
• Acceptance throughout the provincial wide system has been established in a manner that once reliability and validity is determined, there will be good potential for generalization across regions and settings.

   

Review of the specific components of the Manitoba SLP Outcomes Measure:

A. Purpose, Population and Setting:

The basic premise of any scale is that it declares specific limits of applicability in relation to 'what', 'who' and 'where' it measures. The declared population for this tool is birth to twenty-one years of age in all settings including health, family services and education. The stated purpose of the Manitoba SLP Outcomes Measure is threefold:

• To develop a standardized classification of patients and thereby provide a profile (e.g.volume and type) of SLP clientele across the province
  
  
• To measure change in the individual's performance and level of severity across several parameters as a result of SLP intervention
  
  
• To determine caseload prioritization

In the Committee's opinion this tri-fold purpose may be overly ambitious for a single tool. It may be more feasible to develop three separate but related tools each with one of these specific purposes.

B. Feasibility:

To ensure that a scale has high utilization it must be easy to administer and its results easy to interpret. Consensus was reached among the Committee that this tool has been set up for easy access using a web interface for data entry. Both individual and group data can be generated along multiple dimensions in a relatively short period of time.

C. Reliability:

A scale with proven reliability will derive the same results, assuming that the client variables are the same, if administered at separate times and/or by separate raters. This is an important feature to ensure that the scale in itself is stable and therefore its results reflect the client status and not variations in the scale administration. To date there has been no specific reliability testing for the instrument in its current form.
There was reference to established reliability with the three scales from which this was adapted although specific citations were not noted in the materials provided. Regardless, this is now a new tool, which merits its own reliability assessment. Recently, there has been an initiative to establish research collaboration with the University of Brandon in order to assess the reliability between two different raters, that is, between the discharging SLP and receiving SLP, for clients who have been transferred between services.

D. Validity:

A scale with proven validity will derive only accurate results, therefore reflecting the true state of affairs of the features that it is measuring. It is possible that a scale has reliability but not accuracy. For this reason, the validity of a new scale must be shown independently. To date there has been insufficient validity testing for the instrument in its current form. The evidence provided for validity compared demographic profiles between the ASHA NOMS and the Manitoba SLP Outcomes Measure. This evidence is useful in detailing and comparing demographic information across countries but does not determine the accuracy of the Manitoba SLP Outcomes Measure.

As part of the process for developing this report, committee members with clinical expertise in the paediatric to school age clientele assessed the content validity of this scale. Content validity refers to the consensus opinion that scale items are relevant, not redundant and not ambiguous. These content experts were of the opinion that the items included in the Manitoba SLP Outcomes Measure were appropriate for the school aged population but queried applicability for the very young (i.e. birth to two years) and young adult (i.e. eighteen to twenty-one years) populations. The opinion from these experts was uniform and declared a need for further content testing.

E. Generalizability:

To ensure utility, a scale must have proven applicability to the population that it is being developed for. The tool was designed to apply to a heterogeneous population between the ages of birth and twenty-one years across all practice settings. Preliminary data confirms that this goal has been reached at least in reference to setting. Specifically, clinicians from a variety of settings across Manitoba have indicated that this tool applies to their practice. The next step would be to establish that the scale is applicable to all etiologies found within these settings.

F. Responsiveness:

This scale claims to be an 'evaluative' scale in that it can measure change in status over time. In order to be able to measure this change, the scale must demonstrate that it is able to detect this change if it indeed were present. This is a separate assessment from showing that a scale is accurate, but can only be executed after the scale is first proven to be reliable and accurate. The Committee concluded that it is too early in the development of this tool to prioritize this component.

 

Summary and Recommendations:

We applaud the initiative of the project team, the Manitoba government and the SLPs across Manitoba in the development of this instrument. The web based access for data input and report generation has facilitated data collection and analysis of the delivery of SLP services across the province.

It is the opinion of this committee that the recommendation for implementation of this tool nation wide to members of CASLPA is premature until further testing is completed. As an interim step the tool may be ready to launch across Canada for the specific purpose of collecting a client profile database. This would provide a wealth of data regarding demographic and caseload characteristics of clients from birth to twenty-one years of age served by Speech-Language Pathologists across Canada, thereby being the first resource to show trends in service demands and project future service needs.

In order to enhance development and utilization of this instrument it is recommended that:

• The multiple purposes of this tool be simplified and that three separate but related tools be considered
  
  
• Reliability testing be completed to show stability
  
  
• Validity testing be conducted to show accuracy
  
  
• Responsiveness testing be completed to ensure enough sensitivity to detect minimally significant clinical change

With further validation the Manitoba SLP Outcomes Measure would assist SLPs in the collection, evaluation and reporting of outcome data with similar clients. The committee looks forward to further development and enhancement of this instrument.

Respectfully submitted,

Debbie Barton, Co-chair
Rosemary Martino, Co-chair

 

Committee Members

Debbie Barton, Co Chair, Toronto, Ontario
Rosemary Martino, Co-Chair, Toronto, Ontario

Linda Keel-Hale, Charlottetown, Prince Edward Island

Margaret Melanson, Rothesay, New Brunswick

Sharon Fotheringham, Ottawa, Ontario

Heather Heaman, Uxbridge, Ontario

Nancy Thomas-Stonnell, Toronto, Ontario

Shelagh McKeown, Toronto, Ontario

Cameron Allen, Saskatoon, Saskatchewan

Barb Stoez, Edmonton, Alberta

Rachelle Viray, Prince George, British Columbia

 

Data provided to the Committee for review:

Executive Summary
Project Charter 1998
Project Plan 1998
Results, Statements and Performance Indicators
Risk Analysis
Budget
Executive Summary 1998-1999 Pilot Project
Project Plan 1999-2000
Meeting Minutes
Training Process and Data Base Development
"Focused" Review of Manitoba SLP Outcome Measure
Case Comparison of Alberta Severity and Priority Rating and Manitoba
SLP Outcome Measure
Preliminary 2000-2001 Provincial Information

Website address for temporary access(November 2002):

http://64.42.222.148/speech-oct2002/

 

Criteria by which each component of the new measure was rated

Purpose
Is the purpose clearly specified according to the following?

- Discriminative To measure extreme differences between people
- Predictive To predict future outcome
- Evaluative To measure differences over time

Population
Is the population clearly specified?

Setting
Is the setting clearly specified?

Feasibility
Is the scoring system easy to understand?
Is the scoring easy to perform?
Does it rely on clinical skill exam alone?
Are special skills, tools/or special training required?
How long does it take to administer?

Reliability
Does the scoring system provide consistent results when scoring the same conditions
(Test-retest)?
Is the intra-rater & inter-rater reliability satisfactory (>0.70)?

Validity:
Are all the irrelevant items excluded?
Are all the relevant items included
How were the items derived?
Does the scale differentiate between groups in a way appropriate for the purpose?
Does the scale perform satisfactorily when compared to other similar assessments?

Generalizability
Has the scale been assessed in other studies and/or settings?

Responsiveness
Is the scale able to detect clinically relevant change over time