Continuity of Mental Health Services Study of Alberta: A Research Program on Continuity of Mental Health Care

Transcription

1 Continuity of Mental Health Services Study of Alberta: A Research Program on Continuity of Mental Health Care April, 2004 Carol E. Adair T. Cameron Wild Anthony Joyce Gerald McDougall Alan Gordon Norman Costigan Craig Mitton Mingshan Lu Funding Provided by: The Canadian Health Services Research Foundation The Alberta Mental Health Board Eli Lilly (Canada) Inc. The Alberta Heritage Foundation for Medical Research The Institute of Health Economics The Capital Health Authority The Calgary Health Region The David Thompson Health Region The Canadian Mental Health Association (Alberta Chapter)

2 Principal Investigator: Carol E. Adair, MSc, PhD Assistant Professor Department of Community Health Sciences Faculty of Medicine University of Calgary 3330 Hospital Drive NW Calgary, Alberta, T2N 4N1 Telephone: (403) Fax: (403) This document is available on the Canadian Health Services Research Foundation web site ( For more information on the Canadian Health Services Research Foundation, contact the foundation at: 1565 Carling Avenue, Suite 700 Ottawa, Ontario K1Z 8R1 Telephone: (613) Fax: (613) Ce document est disponible sur le site Web de la Fondation canadienne de la recherche sur les services de santé ( Pour obtenir de plus amples renseignements sur la Fondation canadienne de la recherche sur les services de santé, communiquez avec la Fondation : 1565, avenue Carling, bureau 700 Ottawa (Ontario) K1Z 8R1 Courriel : communications@fcrss.ca Téléphone : (613) Télécopieur : (613)

3 Continuity of Mental Health Services Study of Alberta: A Research Program on Continuity of Mental Health Care Carol E. Adair, MSc, PhD 1,2,3 T. Cameron Wild 4 Anthony Joyce 5 Gerald McDougall 2,3 Alan Gordon 5 Norman Costigan 5 Craig Mitton 1 Mingshan Lu 1,6 1 Department of Community Health Sciences, University of Calgary 2 Department of Psychiatry, University of Calgary 3 Alberta Mental Health Board 4 Centre for Health Promotion Studies, University of Alberta 5 Department of Psychiatry, University of Alberta 6 Department of Economics, University of Calgary Acknowledgements: Our primary decision-maker partner and instigator of the study was the Alberta Mental Health Board. Don Schurman, Chief Executive Officer of the AMHB in 1999 provided the original inspiration for the Research Program, and Janet Davidson, Chief Operating Officer, served most enthusiastically as our first decision-maker partner from the Board. The decision-maker partner link was subsequently provided by Ray Block and Sharon Read, and most recently by Dr. Pierre Beausejour (Chief of Staff and Director of Provincial Programs). Our team is deeply grateful for the contributions of an estimated 840 individuals, 320 family members, and 400 mental healthcare professionals, who participated and/or assisted across all three phases of this study. In particular we thank Austin Mardon, Joan Baker, Neil Congo, Fay Herrick, and Trish Cameron for providing liaison with patient/clients and their families. Key decision makers in the regions Eleanor Grant, Di Vosburgh, Bruce Swan, Susan Todd and others continued to support the study through the long wait for results. Most of the credit for the success of the field work goes to Laura Kowalsky, Gloria Pasmeny, Sandy Polis, Corinne Dinnin-Davis, Fran Barnes, and Michael McKenzie. They were supported during recruitment and/or end-stage interviews by Olga Nelson, Cristelle Audet, Julie Henkelman, Barb Mallette, Debbie Renner, and Jeanne Vidacovich. Superb quality study coordination was provided by Anora Beckie and Kimberly Lewis. Doreen Ma developed and managed the databases throughout the study and ran most of the primary statistical analyses. Aliya Kassam provided conscientious data entry. Lynne Kostiuk and Gisele Marcoux did meticulous work reviewing charts and summarizing service utilization activity for costing. Outstanding collaborative support for the Phase III study came from the Calgary Health Region Mental Health Services Executive (Maureen Coleman, Susan Rawlings, Garth Mihalcheon, and Jim Merchant) and Toni Steer of the CMHA Calgary Chapter. Richard Alarie and his expert clinical team (Sara Butler, Suzanne Connelly, and Kathy Talon) developed and implemented the Phase III intervention. Helen Gardiner joined the research team as a Co-Investigator for Phase III, providing the crucial link between research and practice. Helen also conducted the preliminary Phase III analysis.

4 Table of Contents Key Implications for Decision Makers... i Executive Summary... ii Introduction...1 The Research Program...2 Phase I Instrument Development...2 Purpose...2 Design and Methods...2 Results...3 Phase II Prospective Cohort Study...6 Purpose...6 Design and Methods...6 Results...7 Methodologic Learnings...15 Planned or Possible Further Analysis...16 Phase III Intervention Study...16 Purpose...16 Design and Methods...16 Pilot Results...19 Methodologic Learnings...21 Planned Further Analysis...21 Communication, Dissemination, and Linkage Strategies...21 Communications...21 Linkage...22 Dissemination of Results to Researchers and Decision Makers...22 Conclusions...23 References...24 COMHS-Related Presentations/Publications...24 Manuscripts (Internals)...25 Manuscripts (External Collaboration & Validation Study)...25

5 Key Implications for Decision Makers Continuity of care is a complex phenomenon, comprised of events in the process of mental healthcare that occur both among diverse services at one point in time and within services over a longer period of time. Better continuity was associated with higher community costs (averaging $1,511 higher for the highest continuity category compared to the lowest) and lower hospital costs (averaging $4,790 lower for the highest continuity category compared to the lowest). This relationship held even after adjusting for age, household income, suicidal thoughts and behaviour, and duration of illness. This finding is new evidence that health service systems with good continuity of care can reduce hospital costs. The Alberta Continuity of Services Scale for Mental Health is an instrument to measure continuity from both the subjective client/patient perspective and the objective observer perspective. Initial indications are that the instrument has reasonable construct validity and reliability; however, it needs to be refined before it can be used widely or routinely to measure the performance of mental health services. Clients in different local programs or in different demographic subgroups had different levels of continuity. These differences should be analysed to inform local service improvements. Patients at risk of poor continuity of care are those with a diagnosis of mood disorders, current or a history of suicidal thoughts or behaviour, and those with addictions. There were no differences for patients already in treatment according to their gender, ethnic status, education, or employment. This indicates that care is relatively equitable. Continuity of care is associated with improved quality of life, community functioning, and satisfaction with services. The strong correlation between satisfaction with services and patient/client reports of continuity may be partly explained by a tendency to respond favourably or unfavourably to questions about services in general, rather than a real assessment of the very broad and abstract experience of continuity. An intervention team that helps patients receive continuous services in the community was associated with patients having lower symptom levels, better community functioning, better quality of life, and better self-reported health, compared with those who received usual care. i

6 Executive Summary Continuity of care is defined as a process involving the orderly, uninterrupted movement of patients among the diverse elements of the service delivery system. 1 In mental health services, continuity of care has long been considered to be essential for good patient/client outcomes for those with severe and persistent mental illness. However, there is little evidence to support either the basic premise that continuity of care leads to improved outcomes or the contention that service changes designed to improve continuity actually achieve either continuity itself or the intended clinical outcomes in a cost-effective manner. The lack of evidence is felt to be due to inadequate measures of continuity, as well as the lack of well-designed studies to test the set of related research questions at hand. Reasons for the lack of progress in measuring continuity are clear care continuity is, by its nature, a complex, multi-dimensional process that occurs over time and among multiple settings during an individual s care. The Continuity of Mental Health Services Study of Alberta was a three-phase multi-year research program, which operated between October 1999 and August 2003 in three health regions in Alberta. Its purposes by phase were: to develop and test a reliable and valid standard measure of continuity of care for mental health services by operationalizing components of continuity of care as expressed in the mental health literature and through interviews with patients/clients, families, and services providers in the field; to describe continuity of care in three parallel cohorts of patients/clients with severe and persistent mental illness and its relationship with clinical and economic outcomes; and to pilot an economic evaluation methodology which will examine the effectiveness of new initiatives designed to achieve continuity of care, and their efficiency in terms of costs and benefits. In Phase I, a systematic literature review and interviews with patient/clients and their families provided conceptual material to develop draft items for two components of an instrument to measure continuity a patient/client self-report component and a component containing items to be rated by a trained observer on health event information collected prospectively. Draft items 1 Bachrach, L., Continuity of care for chronic mental patients: a conceptual analysis. American Journal of Psychiatry, (11): p ii

7 for the patient/client report component were pre-tested with patients/clients and mental healthcare professionals and then piloted in a sample of 317 patients/clients in three regions. Draft items for the observer component were validity-tested by seven raters on five case scenarios developed from health records. Both components were found to have reasonable validity and reliability, although there was much variability among items in performance. The resulting instrument the Alberta Continuity of Services Scale for Mental Health was then used in a 17-month cohort study of adult service recipients with severe and persistent mental illness in three health regions (Phase II), along with measures of service utilization, outcomes, and costs. Follow-up rates for cohort participants were very high at 85 percent. The study samples were comparable by region on many variables, but significant differences were found on recruitment location, education, income, and duration of illness. These differences were adjusted for in multivariate analysis to make regional comparisons on continuity, costs, and outcomes appropriate for reporting to local stakeholders. In initial analyses differences among regions on the mean overall score for the patient/client scale were not significant, but the overall score masked interesting variations among regions on the 43 individual items. In contrast with the patient/client scale, mean total score differences were found to be statistically significant by region on the observer-rated scale. Ratings on nine of the 17 items were also significantly different by region, with regions alternating rank position according to the specific item. Correlation between the two scales was only moderate in magnitude, suggesting that they are only partially measuring the same concept or that the method of reporting introduces some bias. In the total sample, continuity of care as measured by either scale had statistically significant associations with some of the demographic variables and clinical variables. Higher levels of continuity were associated with older age, lower annual household income, psychotic diagnoses, and no suicidality or substance use. Continuity of care was also associated with health outcomes as measured at the end of the study period, including patterns in the expected direction across disease-specific quality of life and symptom severity and statistically significant associations with generic quality of life, self-reported health status, community functioning, service satisfaction, hospital costs, and community costs. In Phase III, we piloted a method to study the effectiveness of a new service designed to improve continuity of care (the community extension team) in the Calgary Health Region. This phase also used the study s measurement tool, the outcome measures, and the methods for iii

8 costing that were used in Phase II, and experimented with methods for randomized study designs in natural practice settings. Initial findings show that the patients who received the intervention had lower symptom levels and better functioning, quality of life, and self-reported health than those who received usual care. Suicide attempts, hospital admission, and inpatient days were also lower among the intervention group, but ER and mobile crisis team events were more prevalent for as-yet-unknown reasons. Given that the intent was to pilot methods, the sample size precluded the establishment of statistical significance for these findings. Data verification via chart review and cost comparisons are yet to be made. A larger, confirmatory study will be necessary to fully test the community extension team model. Recommendations The study s measurement tool should undergo further refinement, including a detailed review of which items were most strongly associated with outcomes and reduction to a smaller item set before it is ready for use in performance measurement. Scores on individual continuity items at the regional and program levels can be used to identify specific areas to target for general services improvement. The rich data set from the Phase II cohort study can now be used for in-depth studies of a set of mental health service issues that were identified by families and participants, including basic supports (housing and income); patterns of medication use and co-morbid physical illness; crisis and emergency services; work/vocational supports; psychiatrist services; use of complementary and alternative therapies and private providers; and addictions co-morbidities. New and existing services intended to improve continuity of care can be evaluated using the methods and measures that were piloted with the community extension team use of a standard approach will permit comparison of the performance of service innovations across settings and patient populations. The community extension team model should be further evaluated with a larger study sample once the intervention achieves more stable funding and greater consistency in delivery. iv

9 Introduction Continuity of care is defined as a process involving the orderly, uninterrupted movement of patients among the diverse elements of the service delivery system. 1 In mental health services, continuity of care has been touted in the literature as critical to patient/client outcomes for those with severe and persistent mental illness for more than 50 years. Many resources have been invested in attempts to achieve continuity from the individual provider through program levels, and more recently at the system level through services integration. Despite the almost universal belief in the value of continuity of care, there is little evidence to support either the basic premise that continuity leads to improved outcomes or the contention that service changes designed to improve continuity actually achieve either continuity itself or the intended clinical outcomes in a cost-effective manner. The paucity of evidence is largely attributable to the lack of rigorously tested measures of continuity of care, as well as the lack of well-designed studies to test the set of related research hypotheses at hand. Reasons for the lack of progress in the field are clear care continuity is, by its nature, a complex, multi-dimensional process that occurs over time and across multiple settings in the trajectory of an individual s care. In 1999, our team was invited by executive management of the Alberta Mental Health Board to submit a proposal to study continuity of care in mental health services. At that time it was clear to us that to make progress in this area we would need: to develop new measures for continuity of care in mental health services, since the few measures in existence were inadequate to date; to include a prospective longitudinal cohort study, since the longitudinal aspects of continuity of care could only be captured over time and no retrospective data set held the requisite variables of interest; to include a large enough sample to allow for patterns to emerge across the diversity of diagnoses, programs, and specific treatments; to include more than one healthcare system (that is, region), since decision makers were interested in differences in care patterns and costs across a range of service delivery models in the province; to use outcome measures including costs, since decision makers were interested in knowing whether improving continuity of care could save healthcare costs; and 1

10 to include an intervention study to more rigorously test the causal associations between continuity of care and other variables and to test methods for evaluating future continuity interventions for cost-effectiveness. The task at hand required a staged approach wherein later phases built upon the findings of earlier phases. The resulting three-phase research program * was initiated in October 1999 as the Continuity of Mental Health Services Study of Alberta. The research program was in operation for just less than four years, to the end of August This report provides a summary of achievements and results to the end of study operations, as well as some plans for further analysis and dissemination. The Research Program Phase I Instrument Development Purpose To develop and test a reliable and valid standard measure of continuity of care for mental health services by operationalizing components of continuity of care as expressed in the mental health literature and through interviews with patients/clients, families, and service providers in the field. Design and Methods Evidence-based development of a new measurement tool, especially for ill-defined concepts, requires sources of conceptual material that are broader than the investigators own notions about the topic. It is also essential to include those on whom the instrument will be used when developing it, not just as a source of conceptual material, but for better understanding of possible issues related to administration and response patterns. However, given that the level of agreement between client/patient perceptions of continuity and more objective indicators was completely unknown, we had to include both types of measurement in our instrument. Based on reports in the literature about previous attempts at operationalization, our working principles for conceptualizing continuity were that it should be: * The original scientific title of the research program was Service Delivery Patterns for Persons with Severe and Persistent Mental Illness: Continuity of Care, Clinical and Economic Outcomes. 2

11 multi-dimensional (having many aspects); considered as a characteristic of the process of care rather than the outcome of care; best measured at the individual patient/client level even though it occurs at all service levels; and measured using both the patient/clients own perceptions and observer-rated indicators. We also sought to develop an instrument that was psychometrically sound (that is, reliable, valid, and responsive) as well as simple to use, such that it might ultimately have application in routine performance measurement activities. The first tasks of Phase I were a systematic, validated literature review (which resulted in 305 articles selected using multiple raters on continuity in mental health services out of a total 1,604 articles on continuity in all health services) and semi-structured interviews with 36 patients/clients and their families. Each of these processes was intended to provide material for the patient/client report instrument items. Content analysis was used to extract continuity attributes which were grouped into themes and from which 121 initial items were drafted. Items were then pre-tested with 41 healthcare professionals and patients/clients and reduced to the 47 that had the highest endorsement for relevancy and coherence. The final step in developing the self-report scale was a field test with 317 patients/clients in mental health services in the three targeted regions. Factor analysis of the field test data provided item groupings which were characterized as sub-scales. We also included two test-re-test reliability studies of the patient/client report scale. For the observer-rated scale we drafted 17 items based on previously used operational definitions from the literature. These items were validated using five case scenarios developed from health record information in each region and the ratings of seven healthcare professionals that had no previous exposure to the continuity concepts derived from the literature. Results Each of the results sections in this report contain only highlights of the study findings. Detailed results have been or will be reported in published abstracts, manuscripts, and presentations to stakeholders. Abstracts, presentations, and manuscripts which have been prepared or published to date are listed at the end of the report. 3

12 A multi-dimensional instrument to measure continuity of care, now called the Alberta Continuity of Services Scale for Mental Health (ACSS-MH), was developed. The instrument has 43 patient/client-reported items and 17 items to be rated using health record information. Table 1 Examples of ACSS-MH items* Patient/Client Scale** (total items = 43) Observer Scale (total items = 17) My treatment fits my needs. Involvement in community-based interventions I am not treated like an individual in mental can be described as: health services. - No community-based services (continuous or My appointments can be more often if I am episodic hospital care only) doing worse. - Some community-based services but hospital I ve had to repeat my history every time I need resident > 12m help. - Irregular community services AND/OR most care I have been treated with dignity and respect. hospital-based My care doesn t change when my needs - Regular community-service involvement and only change. one or two hospital events (admission or ER) I don t know where I would go if I needed help. - Regular community service involvement plus full I have to deal with a confusing number of community tenure (no admission or ER for study agencies and programs. period) I have been unable to pay for my medications when out of hospital. Treatment gaps the number of approximate I m confident that my psychiatrist can admit me 30-day periods without primary provider whenever I need to be in hospital. contact when it was apparently needed have I ve been refused admission to certain been programs and am unable to understand why. - > 5 gaps gaps gaps - 0 gaps/care continuous for whole study period *complete scale available from the authors on request ** all item responses for the patient/client scale were given on a five-point Likert scale ranging from strongly agree to strongly disagree. This scale was administered via patient/client interview. Psychometric properties of the new scale were examined as follows: Item distributions on the patient/client scale were reasonably normal, with acceptable variances for all but six items. Content validity was examined through exploratory factor analysis on the pilot sample (N=319) as well as the 4

13 full sample (N=404) for the patient/client scale. Both analyses produced similar three factor solutions; that is, three groups of items that can be characterized as subscales. The three subscales were system fragmentation, relationship base, and responsive treatment. These accounted for a moderate proportion (39 percent) of the total variance in the dataset, suggesting that there are still aspects of continuity that are unexplained and unmeasured and/or there are many items that contribute only noise to the overall scale. Internal consistency is very good for each subscale, with Cronbach s alpha values ranging from.78 to.92. Two-week test-re-test reliability coefficients for N=49 pairs in the main study ranged from.24 to.64. However, test-re-test reliability is not completely relevant to this type of scale, because it is intended to measure a dynamic process rather than a stable psychological trait, and high test-re-test values may indicate lack of responsiveness to change over time. The more appropriate split-half reliability coefficients were very good at.77 to.88. The patient/client scale has also been externally validated in work by J. Durbin and P. Goering at the Centre for Addiction and Mental Health at the University of Toronto in a separate sample. The results of that work mirror our findings and have been submitted for publication. 2 There are now sufficient data and preliminary information on the scale to reduce the item pool and conduct the more robust confirmatory factor analysis on the instrument. This promises to produce a more concise and effective instrument. Psychometric properties of the observer scale were also examined. Construct validity was pre-tested via ratings of seven providers on five case scenarios. Intra-class correlation coefficients across judges ranged from.52 to.95 for 15 items, with only two falling below.50. Item-to-total-score correlations ranged from.60 to.95, indicating reasonably good internal consistency. Inter-rater reliability was also examined for the actual item ratings as applied to the prospectively collected data, including chart reviews in the Phase II study. Much variability in reliability across items was found, with percent agreement for exact rating agreement across two raters as low as 38 percent for one item and as high as 88 percent for others; the average percent agreement was 60 percent. Analysis of percent agreement 5

14 for close rating agreement (within one rating either direction) was better, with average percent agreement across two raters at 85 percent. Further work will also be necessary on the validity of the indicator items on the observer scale. The implication of these findings is that it would be as yet premature to use these items routinely for performance measurement purposes. Phase II Prospective Cohort Study Purpose To describe continuity of care in three parallel cohorts of severe and persistent mental illness patients/clients and its relationship with clinical and economic outcomes. Design and Methods Individuals presenting for care between March and July 2001 to 70 directly funded inpatient, outpatient, ER, and community mental health service sites in the three health regions were notified about the study by attending health professionals, and names of interested participants were forwarded to study staff. Inclusion criteria were a confirmed diagnosis of mental illness, ages 18 to 65, and not under guardianship or receiving involuntary or forensic care at the time of enrolment. Study staff administered informed consent and baseline study measures. These included the Mini International Neuropsychiatric Interview for diagnostic confirmation and basic demographics, and the Brief Psychiatric Rating Scale for symptoms severity. The Colorado Client Assessment Record was mailed to each participant s primary mental healthcare provider for ratings on baseline problem severity and needs. Participants were then prospectively followed and contacted at two-to-three-month intervals to collect additional information on social support and all health service utilization events, including all inpatient, outpatient, emergency room, and both directly and indirectly funded community-based care. Specially designed wallet cards (passports) for recording service events were provided to participants to assist with recall of events. For a subset of participants who provided consent for contact, telephone interviews were conducted with family members or significant others to measure the amounts of family burden, including informal supports provided to these individuals by informal caregivers. At the end of the follow-up period, outcome measures were administered. These were the Multnomah Community Ability Scale for community functioning, the Brief Psychiatric Rating Scale for 6

15 symptom severity, the Service Satisfaction Scale-10, the Wisconsin Quality of Life Inventory for disease-specific quality of life, and the Euroqol to measure generic quality of life on a scale that had the necessary utilities for economic analysis. At this point perceived continuity of care for the follow-up period was also measured using the study s client/patient scale. Approximately 700 charts were reviewed across multiple service settings to verify service event information and to collect key information needed for the study s observer scale ratings during and at the end of the follow-up period for the participants. Observer-rated continuity of care ratings were then applied by a single rater (the principal investigator) to the comprehensive set of information for each participant, including an inter-rater reliability check by two co-investigators for a 10 percent subset. Administrative data, which included service events and, in most cases, hospital-based inpatient and outpatient costs, was received from all three regions. Physician costs (grouped as general practitioner, psychiatrist, and other specialist) were compiled from Alberta Health and Wellness billing data. Community-based service listings (including indirect treatment agencies, residential, vocational, addictions, and housing services) were produced, and relevant agencies were surveyed for cost estimates for their services. Standard service and medication summaries were used to optimize consistency and accuracy in compilation of costs. Results Follow-up was very good, with at least one source of end-point information obtained on 439 of 486 (90.3 percent) participants; end-point interviews on the key variables (observer-rated continuity of care [411 of 486, 84.6 percent] and generic quality of life [401 of 486, 82.5 percent]); and cost data (430 of 486, 88.5 percent). The average follow-up period was 72 weeks (range 61 to 90) or just less than 17 months; administrative data was cut off according to individual follow-up dates except for physician billing data. Baseline demographic and clinical characteristics of the sample are shown in Table 2. 7

16 Table 2 Participant demographic and clinical characteristics Recruitment location Psychiatric hospital General hospital IP, OP or ER Community clinic Characteristic Percent N Sex (% female) Education (% any post-secondary) Currently working (%) Personal income (< 20,000. CAD annually) Household income (< 20,000. CAD annually) Receiving AISH* Ethnic status (% visible minority)** Length of follow-up (months) Age (years) Education (years) Previous hospitalizations (% >10) Primary current diagnosis Mood disorders Psychotic disorders Psychiatric co-morbidity (% >1 DSM-IV diagnosis) Suicidality (history + current) Substance use co-morbidity (% any) Medical co-morbidity (% any) Age first symptoms (years) Duration of illness (years) Baseline problem severity score** Mean Percent Mean * Assured Income for the Severely Handicapped ** Statistics Canada 2001 Census proportion of visible minorities in total urban Alberta populations is about 16 percent *** average score rated on a scale of nine across 21 problem areas by the healthcare provider on status at enrolment on the Colorado Client Assessment Record Significant differences between regional samples were found for recruitment location, education, household income, and duration of illness. These differences were adjusted in regional comparisons of overall continuity scores and costs. Table 3 displays the total sample continuity scores. The mean total score on the patient/client component of the study s instrument was on a total possible score of 185. The mean total score on the observer-rated scale (based on the pattern of care seen in prospective collection and chart reviews) was 39.2 on a total possible score of 59. 8

17 Table 3 Continuity of care scores Instrument component: Mean Range Total Possible Score SD N Patient/Client Scale Total scale, 37 item version* Observer Scale Total scale, 17 items * Items related to hospital admission and family involvement were removed in this version because of large numbers of not applicable responses In initial analyses, differences among regions on the patient/client rated scale score were not statistically significant (F = 1.25, p =.286). This finding did not change after adjusting for potential confounders (age, recruitment location, household income, education, employment, diagnosis, duration of illness, and problem severity) in multivariate analysis (logistic regression). However, grouped item scores also tended to mask interesting variations among regions on individual items. These represent indicators of service performance in very specific areas and will be reported to decision makers by region and, where sample size permits, by program. In contrast with the patient/client scale, mean total score differences were found to be statistically significant by region on the observer-rated scale (F=4.2, p=.014), although the actual magnitude of the differences was not large. These differences disappeared after adjusting for age, diagnosis, and recruitment location, in multiple logistic regression (OR (region one versus two) =1.30 (95% CI ) p=.480). Ratings on nine of the 17 observer-rated items were also significantly different by region, with regions alternating rank position according to the specific item. These differences were or will be reported to local decision makers at both the regional and program levels. Total scale scores were only moderately correlated between the client/patient scale and the observer scale (r=.36, p=.000). This finding provides some support for content validity, but the association is not of sufficient magnitude to create a total score across both scales. In contrast, the patient/client scale was highly correlated with the total score on the Service Satisfaction Scale-10 (r=.73, p=.000). We hope this means that those who have good continuity of care are more satisfied with that care, but there are other possible reasons. First, the patient/client scale on the study s instrument might simply be a substitute measure for 9

18 satisfaction with care, and the concept of continuity may be too broad and abstract to report on beyond simple service satisfaction. Second, the same individuals may respond similarly on both measures through response sets the tendency to respond positively or negatively on any set of questions. We suspect that all of these reasons may contribute to the phenomenon. Further analysis of our data as well as future research is needed to determine the relative contributions of these explanations. Relationships between continuity of care scores and demographic and clinical variables are shown in Tables 4a and 4b. Table 4a Relationships between continuity of care and demographic variables Demographic Variable: Age years >50 Patient/client Scale Total Score Statistical Significance.050 Observer Scale Total Score Statistical Significance.009 Sex F M Education 12 years or less Any postsecondary Employment Working for wages Not working for wages Personal income (annual) < $ 20,000 $ 20,000-59,000 $ 60, Household income (annual) < $ 20,000 $ 20,000-59,000 $ 60, Ethnic Status Caucasian Non-Caucasian using Student s t-tests or ANOVA. Differences that are statistically significant at the.001 level are in bold text. The significance level was set conservatively to adjust for multiple testing and to avoid identification of small non-clinically significant differences that result from large sample sizes 10

19 Among the demographic variables shown in Table 4a, higher levels of observer-rated continuity were associated with older age and lower annual household income but not with sex, education levels, current employment status, personal annual income, or ethnic status. This suggests that there is reasonable equity and response to need (as indicated by continuity of care) among patients on demographic characteristics. With respect to clinical variables (Table 4b) higher observer-rated continuity scores were associated with a current diagnosis of psychosis, no suicidal thoughts or behaviour or substance abuse co-morbidity, and lower problem severity scores. These findings point to patients who are at risk for poor continuity and might benefit from targeted interventions. In particular, baseline Colorado Client Assessment Record scores were very predictive of poor continuity, so this instrument might be useful as an eligibility screen for such interventions. Table 4b Relationships between continuity of care and clinical variables Clinical Variable: Duration of illness <5 years 5 years+ Patient/client Scale Total Score Statistical Significance.258 Observer Scale Total Score Statistical Significance.068 Previous hospitalizations None 1-10 > Primary Diagnosis Depression Bipolar disorder Psychotic disorder Psychiatric co-morbidity 1 diagnosis only >1 diagnosis Suicidality (history + current) No Yes Substance use co-morbidity No Yes Medical co-morbidity No Yes

20 Clinical Variable: Baseline problem severity (quartiles)* Patient/client Scale Total Score Statistical Significance Observer Scale Total Score Statistical Significance differences that are statistically significant at the.001 level are in bold text. The significance level was set conservatively to adjust for multiple testing and to avoid identification of small non-clinically significant differences that arise with large samples average score rated on a scale of nine across 21 problem areas by the healthcare provider on status at enrolment on the Colorado Client Assessment Record, where higher scores represent greater problem severity. Table 5 displays the results of the primary research questions for the cohort study: associations between continuity and health outcomes. Trends were consistent across all variables, but associations between the observer scale and generic quality of life (including ratings across five areas of quality of life and a total rating out of 100), both scales and service satisfaction, and the patient/client scale and community functioning were statistically significant at the.001 level. Table 5 Continuity of care and health outcomes Patient/ Client Scale Outcome Measure Quartile Generic Quality of Life Statistical Significance Observer Scale Quartile Statistical Significance Mean EQ-5D Index Score* Mean EQ-5D Health Status Rating** Disease-specific QL WQL weighted total scores*** Symptoms BPRS End-point Scores**** Functioning Mean MCAS Scores^

21 Outcome Measure Service Satisfaction Mean SSS-10 Total Scores# Patient/ Client Scale Quartile Statistical Significance Observer Scale Quartile Statistical Significance.022 * Total score on five items and three responses (total possible=15) where higher scores represent poorer quality of life after adjustments for utility weights ** Rating on a 100-point scale where higher scores represent better quality of life *** Total weighted scores on multiple items and seven subscales range from to +3.00, higher scores representing better quality of life **** Rating on a 24-item and seven-responses scale (total possible=168) where higher scores represent worse symptoms ^ Rating on a 17-item and five-response scale (total possible=68) where higher scores represent better functioning # Rating on a 10-item and five-response scale (total possible=50) where higher scores represent greater satisfaction The association between continuity and health outcomes held up in several multivariate statistical models (logistic regression) that accounted for potential confounders. For example, the relationship between patient continuity ratings and self-rated health status remained highly significant (OR=2.72 (95% CI ) p=.001) after adjusting for household income and baseline problem severity. The relationship between observer-rated continuity and generic quality of life (OR=2.06 (95% CI ) p=.044) remained significant after adjusting for diagnosis, age, suicidal thoughts or behaviour, and household income. The total costs (including income support) for our sample over the 17-month period were $13,658,701. The per patient mean total cost was $31, (range $192 to $227,353), for an annualized per patient mean cost of about $22,422. These are variable costs only (staff, supplies, medications, and diagnostics), not capital or facility overhead costs. Average per patient total health service costs (without income support or patient-paid medications) were $24,070 (range $36 to $212,968), with an annualized mean cost of $16,991. Hospital costs accounted for 51 percent of these health services costs, followed by publicly paid medications at 20 percent, physician billings at 19 percent, and home services at three percent. The remaining expenditures (all less than three percent of total health service costs) included community clinic services, non-physician health professionals, funded agencies, community lab test costs, housing-related services, and crisis calls/visits. Table 6 summarizes the bivariate findings for the association between total healthcare costs and continuity. There was no association between continuity and total healthcare costs, but

22 better continuity was associated with higher community costs (averaging $1,511 higher for the highest continuity category compared to the lowest, p<.001)) and lower hospital costs (averaging $4,790 lower for the highest continuity category compared to the lowest (p=.001)). Table 6 Continuity of care and healthcare costs Cost Variable (CAD)* Overall system costs (median)* Observer Scale Quartile 11, , , ,472. Bivariate Statistical Significance Multivariate** Statistical Significance Hospital costs (mean) 7,244. 7,943. 7,762. 2, Community costs (mean) 1,000. 1,122. 1,412. 2, * Because of non-normal distributions, cost variables were transformed for statistical tests. The values here represent medians after back-transformation to the original scale (Canadian dollars). ** After adjustment for household income, suicidality, age, duration of illness, in multiple linear regression models The relationships between continuity and costs held even after adjusting for age, household income, suicidal thoughts or behaviour, and duration of illness in multiple linear regression models. The findings represent new evidence that improving continuity for severe and persistent mental illness may produce better outcomes while averting system costs in the form of inpatient hospitalizations and emergency visits. In summary, positive health outcomes across several measures were found to be associated with good care continuity. However, since this was an observational study, it is hard to state unequivocally that continuity causes good health outcomes. Another possibility is that individuals with better health status and quality of life are more likely to receive continuous care or are more likely to rate their service experiences favourably. The findings are 14

23 encouraging, but experimental studies are needed for further confirmation. The Phase III study, reported in the next section, was designed to pilot methods for an intervention that could ultimately be used to test whether good continuity of care causes better quality of life among individuals with severe and persistent mental illness. Methodologic Learnings Despite extensive communications it is difficult to get clinical staff to track the status of eligible populations during recruitment. It was also not possible to obtain ethics approval to collect data from charts to describe characteristics of non-participants. These constraints make it difficult to estimate the amount of selection bias that may be present at the time of recruitment and ultimately to assess how representative the participants are. Methods need to be developed to overcome these methodologic constraints, including the judicious use of incentives. Multiple sources of data on service events are necessary for adequate capture, as success varies across methods. Of the four methods we used (service staff reports, client-held passports, client interviews, and chart reviews) chart reviews and client interviews were the most useful, followed by client-held passports (which were very useful but for a subset of participants only), and service staff reports, which were least useful due to minimal compliance. The current state of the science for data collection forms for tracking service events (we adapted materials from three large studies in the U.S.) which will ultimately be used for costing purposes is woefully inadequate we now know, after many iterations, what the necessary characteristics of such data forms should be for maximum efficiency in utilization data collection and compilation. Participant engagement and retention in follow-up studies of patients/clients with severe and persistent mental illness is resource-intensive and not completely predictable. Inflexible timelines are not realistic for this type of field-based research. 15

24 Planned or Possible Further Analysis Individual items on both of the study s scales will be examined to identify those which have the strongest associations with outcomes, as these will be the candidate items for a shorter scale which may be used for both outcomes research and routine performance measurement. Dr. Mingshan Lu will do further econometric analysis, including hierarchical models to examine program-level contributions to continuity and healthcare costs. The detailed care patterns of all participants in Phase II of the study provided us with a large dataset on many topics related to mental health services for further study. These include issues of income support, medication use, crisis and emergency services, costs of care, work/vocational issues, alternative therapies, addictions, and psychiatrist services. Phase III Intervention Study Purpose To pilot an economic evaluation methodology which will examine the effectiveness of new initiatives designed to achieve continuity of care, and their efficiency in terms of costs and benefits. Design and Methods In collaboration with the Calgary Health Region mental health services management team, a new program the community extension team was developed and piloted. The team of four two paraprofessionals (independent living skills staff) and two professionals (a nurse and a psychologist) provided individual assistance on an outreach basis to patients/clients who were deemed to be at high risk of poor continuity. The health region provided the funding for clinical operations, and the research program provided the funding for the research study as well as the design of a sustainable, ongoing evaluation component which would continue to inform program operations after the end of the study. Participants were referred to the community extension team from two groups of sources. First, those who participated in Phase II of the study who provided consent for future contact were telephoned and invited to participate, with priority placed on those who reported the lowest scores on the study s instrument at the final interview of the cohort follow-up. This group was 16

25 considered to be more chronic yet stable at the time of enrolment. Second, other individuals who were not previously known to the study but who were considered to be at high risk for poor continuity because of either imminent discharge from an inpatient unit, a recent crisis episode, or attendance at a high-needs inner-city mental health clinic were recruited in service settings. This second group was considered to be more acute in presentation. The intervention consisted of providing interim functional support while patients/clients were in transition between existing services. For example, the team would make outreach visits to the patient/client while he/she was waiting for regular ongoing services, ensure attendance at critical medical or psychiatric appointments, negotiate for expedited access if justified by level of need, ensure access and adherence to medication, and arrange for housing and income assistance. The intent of the new service was not to add a new program (wherein caseloads were held longterm) to an already complex set of services, but to bridge the more immediate gaps among existing services. Thus the intervention occurred at the system level in addition to the individual level. System-level interventions included increasing awareness among service personnel about the mandates and intake procedures of other services, identifying gaps between programs, and making suggestions for remedies. The team was conceptualized as a step down in intensity compared with assertive community treatment models, which typically provide long-term support with a client-to-staff ratio of about one-to-10. The reasoning was that, at a proposed client-staff ratio of about 30, a short-term intervention would be able to serve a much larger proportion of the clients with severe and persistent mental illness in the region, who would have difficulty navigating the service delivery system at critical points in their care trajectory but not be having a level of difficulty in functioning once they were connected with services that would warrant the intense support of the regional assertive community treatment program. The intent of the research component of the community extension team was to pilot methods for economic evaluation in applied settings, as it was acknowledged from the outset that the timeframe would likely not support a full trial of the intervention itself. Nevertheless, a two-group randomized intervention trial was designed that could theoretically test the hypothesis that the community extension team would be clinically effective and cost effective compared with usual care. Usual care was defined as any service that would ordinarily be offered and/or provided to these individuals at their enrolment point of care. Thus the standard course of discharge 17