Impacts of Evidence-Based Quality Improvement on Depression in Primary Care

Transcription

1 Imacts of Evidence-Based Quality Imrovement on Deression in Primary Care A Randomized Exeriment Lisa V. Rubenstein, MD, MSPH, 1,2,3 Lisa S. Meredith, PhD, 2 Louise E. Parker, PhD, 2 Nancy P. Gordon, PhD, 4 Scot C. Hickey, MA, 2 Carole Oken, MA, 2 Martin L. Lee, PhD 1,5 1 VA Greater Los Angeles, Los Angeles, CA, USA; 2 RAND Health Program, Santa Monica, CA, USA; 3 David Geffin School of Medicine and Deartment of Medicine at UCLA, Los Angeles, CA, USA; 4 Kaiser Permanente Division of Research, Oakland, CA, USA; 5 UCLA School of Public Health, Los Angeles, CA, USA. CONTEXT: Previous studies testing continuous quality imrovement (CQI) for deression showed no effects. Methods for ractices to selfimrove deression care erformance are needed. We assessed the imacts of evidence-based quality imrovement (EBQI), a modification of CQI, as carried out by 2 different health care systems, and collected qualitative data on the design and imlementation rocess. OBJECTIVE: Evaluate imacts of EBQI on ractice-wide deression care and outcomes. DESIGN: Practice-level randomized exeriment comaring EBQI with usual care. SETTING: Six Kaiser Permanente of Northern California and 3 Veterans Administration rimary care ractices randomly assigned to EBQI teams (6 ractices) or usual care (3 ractices). Practices included 245 rimary care clinicians and 250,000 atients. INTERVENTION: Researchers assisted system senior leaders to identify riorities for EBQI teams; initiated the manual-based EBQI rocess; and rovided references and tools. EVALUATION PARTICIPANTS: Five hundred and sixty-seven reresentative atients with major deression. MAIN OUTCOME MEASURES: Aroriate treatment, deression, functional status, and satisfaction. RESULTS: Deressed atients in EBQI ractices showed a trend toward more aroriate treatment comared with those in usual care (46.0% vs 39.9% at 6 months, P=.07), but no significant imrovement in 12-month deression symtom outcomes (27.0% vs 36.1% oor deression outcome, P=.18). Social functioning imroved significantly (mean score 65.0 vs 56.8 at 12 months, P=.02); hysical functioning did not. CONCLUSION: Evidence-based quality imrovement had ercetible, but modest, effects on ractice erformance for atients with deression. The modest imrovements, along with qualitative data, identify otential future directions for imroving CQI research and ractice. KEY WORDS: quality imrovement; deression; continuous quality management; social function. DOI: /j x J GEN INTERN MED 2006; 21: T he current quality crisis highlights the ga between what we know, based on research evidence, and what rimary care ractices actually deliver. 1,2 In the case of deression, aroriate treatment (antideressants or sychotheray) imroves outcomes. 3 5 Only a minority of rimary care atients, however, comlete a minimally adequate course of deression treatment, 6,7 desite a large national investment in deression care. 8 Dissemination of clinical guidelines with or without additional clinician education is ineffective in rectifying this situation. 4,5,9 11 New care models for deression that change ractice structure 18 to facilitate high-quality deression care are cost-effective and relatively affordable, but are difficult for health care systems and ractices to imlement, in art because they require significant organizational change. This study tests the imacts of a modified version of continuous quality imrovement (CQI) when used to hel health care systems design and imlement evidence-based care models for deression. Continuous quality imrovement and related methods are among the few systematic aroaches available to hel health care ractices lan and imlement organizational change Originally CQI focused on finding roblems and solutions within individual ractice settings rather than seeking outside evidence. 24 Later, researchers and others introduced an evidence focus on the rocess of care 18 into CQI by charging teams with imlementing evidence-based clinical treatment guidelines. Studies of deression guideline imlementation using this form of CQI, however, showed no effects on erformance of aroriate deression care at the ractice level or deression outcomes A ossible reason why CQI was not effective in these studies is that although teams accessed evidence on the rocess of care, they were left to their own devices in terms of modifying the structure of care 18 (e.g., by redesigning the care model). We modified CQI to encourage QI teams to focus on increasing aroriate deression treatment and on using evidencebased care models to do so. We included a focus on effective rovider behavior change strategies that QI teams could incororate into their models We termed the modification evidence-based quality imrovement (EBQI), and evaluated its imacts by assessing the erformance of exerimental and usual care ractices on measures of deression-related care and outcomes. Continuous quality imrovement trials are comlex. They test both the effectiveness of the CQI method in heling ractices achieve effective organizational changes, and the effectiveness of the changes themselves. Practices, not researchers, control the change rocess initiated by CQI. In this aer, in addition to assessing the effects of EBQI, we aim to suort future imrovements in CQI ractice and research by No conflicts of interest. Address corresondence and requests for rerints to Dr. Rubenstein: VA Greater Los Angeles, Plummer (152), North Hills, CA ( lisa_rubenstein@rand.org). Manuscrit received February 02, 2005 Initial editorial decision Aril 25, 2005 Final accetance Aril 18,

2 1028 Rubenstein et al., EBQI for Deression in Primary Care JGIM FIGURE 1. MHAP intervention and evaluation design and timeline as imlemented. z Zero indicates the start of the intervention. The overall timeline ranges from 6 months (start of the reintervention survey) to 30 months. w The callouts at the left hand side of the figure identify the design, Plan-Do-Study-Act (PDSA) and imlementation hases of evidence-based quality imrovement (EBQI). The EBQI design call-out reresents the study intervention (researchers initiated and structured the EBQI design rocess in the exerimental ractices). z The three-dimensional boxes in the diagram reresent atient oulations. Numbers (n) of atients for the re and ostintervention samles are listed on the aroriate boxes. In the exerimental ractices, the white box labeled deressed atients exosed to care model reresents atients exosed to the new deression care models develoed by QI teams (e.g., seen by a deression care manager), whether or not they were enrolled in the ostintervention survey samle, and contrasts with all unexosed deressed atients visiting exerimental ractices (vertical stries). Note: we do not know what roortion was exosed. The boxes labeled longitudinal survey samle reresent atients enrolled in the ostintervention evaluation. In exerimental ractices, the enrolled atients exosed to new care models (white area) contrast with the roortion not exosed (vertically stried area). The time eriod labeled exosure indicates the mean duration of the window (i.e., between 8 and 12 months) during which atients who visited an intervention ractice before their enrollment in the longitudinal survey samle might have been exosed to the new care models. roviding qualitative detail on our intervention and evaluation rocess. and method of accessing locally available mental health staff. No ractices or clinicians refused to articiate. Protocol METHODS Our study was a cluster randomized exeriment 31,32 comaring EBQI with usual care conducted both within a nonrofit community organization (Kaiser Permanente of Northern California, or Kaiser) and a government-funded health care system (Veterans Administration, or VA). Institutional review boards at Kaiser, VA, and the RAND Cororation aroved the study. We aroached 9 large managed rimary care ractices in California (6 from Kaiser and 3 from VA) identified by regional organizational leaders as not having reviously initiated or articiated in formal deression care imrovement. Each ractice had its own leadershi team, staffing, atients, Assignment and Practice Characteristics We matched the 9 enrolled ractices into 3 trilets (2 Kaiser trilets and 1 VA trilet) based on atient ethnicity and urban or suburban location. Within each trilet, the statistician randomly assigned 2 ractices as exerimental and 1 as usual care. Interventions In both usual care and exerimental grous, we mailed clinicians coies of clinical ractice guidelines for deression. 33 We reviously ublished qualitative information on the EBQI intervention. 34,35 Figure 1 summarizes the major hases and timeline, including the QI design rocess, care model start-u, and maintenance (care model in lace). The

3 JGIM Rubenstein et al., EBQI for Deression in Primary Care 1029 entire timeline as ictured in Figure 1 sans the eriod from 6/96 to 12/98. Only ractices in the exerimental grou articiated in EBQI. The major EBQI activities are summarized in Aendix 1, available on the web. Researchers rovided QI team leaders with (1) an EBQI manual outlining how teams should carry out the design rocess, including guidelines for 16 hours of meeting time; (2) a deression care tool kit 12,36,37 ; (3) articles on effective care models for imroving deression care or changing rovider behavior; and (4) the to 5 guideline-based goals organizational senior leaders identified through an exert anel rocess (Aendix 2, available on the web). 34,38 Researchers additionally rovided 2 brief orientation sessions to each QI team, but had no ongoing involvement in the QI rocess; articiating organizations and their QI teams, not researchers, were the decision makers. Researchers also rovided articiating organizations with limited funds (about $25,000 total) that could be used to ay for release time for QI articiants during the EBQI design hase. 34 Particiating organizations rovided all other resources that EBQI teams and ractices used. Existing quality imrovement committees at each organization reviewed and aroved each EBQI team s roosed deression care model. Trained qualitative observers documented EBQI team care models and meetings and conducted semi-structured interviews. 34,35 Assessment of Preintervention Equivalence of Grous We aroached consecutive atients in each clinic, asked for their consent, assessed demograhic and health status information by survey, and accessed their comuter medical records for the succeeding 6 months for mental health secialty visit codes or deression diagnosis codes. Outcomes We began outcome assessment after EBQI structural changes were in lace in each articiating ractice (Fig. 1). We enrolled a reresentative cross section of atients with current major deression by systematically aroaching consecutive atients attending rimary care aointments over a 5- to 8- month eriod in each clinic, screening them for deression symtoms, and testing them for major deression diagnosis. We administered follow-u surveys 6 and 12 months after enrollment. Imlemented care models served enrolled and nonenrolled atients alike; enrolled atients were not informed about their deression or deression care otions and clinicians did not know which atients were enrolled. Samle size calculations, accounting for cluster, showed 80% ower to detect a 20% difference in aroriate treatment (from 40% to 60%) with a samle of 56 atients er ractice, or 504 atients overall. The research team identified aroriate deression treatment and recovery from deression as rimary outcomes. We also targeted imacts on social functioning, which is critically imortant to emotional health 39 and more resonsive to rimary care deression interventions than hysical functioning. 14,40 In both study organizations, senior leaders identified imroved atient education and articiation in care as 1 of the 5 key goals for EBQI (Aendix 2); we evaluated atient exosure to deression education and atient satisfaction with articiation in care as corresonding measures. We exected maximum increases in comletion of aroriate treatment at 6 months and outcome effects at 12 months. During each survey wave, we queried atients about treatment during the revious 6 months. If EBQI ractices were more roactive, we exected they would find and begin to treat our enrolled deressed atients sooner. Treated cases should accrue raidly during the 6 months after enrollment, with the comleted atients beginning to discontinue treatment thereafter. Cases of remission following treatment should start accruing in the initial 6-month window and continue to accrue through 12 months after enrollment. We exected usual care ractices to be less roactive, and thus to find and treat enrolled atients at a delayed and relatively low but continuous rate, and to have relatively high rates of rolonged, artial treatment. Enrollment began with an initial 10-minute self-administered survey that included deression screening questions based on the Comosite International Diagnostic Interview (CIDI). 41 Patients with current, frequent lack of leasure, or deressed mood were eligible to consent to follow-u surveys and record review. After consent, we administered the full CIDI. 41 We followed only atients with a structured diagnosis of major deression (Aendix 3, available on the web, shows enrollment yield at each ste). As deendent variables, we measured aroriate treatment with antideressants or sychotheray 14 and atient exosure to deression education 36 based on survey resonses at baseline, 6, and 12 months, using reviously validated indicators. We measured atient satisfaction with articiation in care on a 0 to 100 scale constructed from 9 items on satisfaction with quality of communication with the atient s clinician, exlanation of tests and treatments, involvement in decisions about care, and ease of getting hel. Items are scored from 1 (low satisfaction) to 5 (high satisfaction). This scale was administered only at 6 months, was modified from a revious scale, 42 and had a Cronbach s a of A 20% imrovement on the single item involvement in decisions about care from this scale has been found to be associated with a 4% to 5% increased robability of receiving guideline-concordant care and a 2% to 3% increased robability of deression resolution over the following 18 months. 43 At baseline, we measured atient satisfaction with care for ersonal roblems using a single item scored from 1 to We measured oor deression outcome based on a reviously tested summary measure 44 of whether the atient remained deressed (scored below cutoffs) on all of the following: (1) current major deression symtoms with functional imairment based on CIDI items 41 ; (2) the Center for Eidemiological Studies Deression (CESD) Scale 44,45 ; and (3) the Mental-health Comosite Score (MCS) from the SF ,46 We assessed the effects of hysical health or emotional roblems on social functioning using a 0 to 100 scale (higher is better) from the SF We assessed each variable, other than satisfaction, at baseline, 6, and 12 months. As covariates, we measured age, sex, comletion of high school, ethnicity, count of chronic diseases, marriage, alcohol use (3 questions), 48 dysthymia, 14,41 and household wealth. 49 We also assessed the timing of enrollment relative to any rior visits that the atient had made while EBQI ractices new care models were already in lace, thus reflecting otential

4 1030 Rubenstein et al., EBQI for Deression in Primary Care JGIM Table 1. Evidence-Based Quality Imrovement (EBQI) Team Design and Imlementation of Deression Care Model Comonents Deression Care Model Comonents Major Care Model Comonents Planned or Imlemented by Each Practice KP Practices VA Practices #1 #2 #3 #4 #1 #2 Provider education and decision suort Presentations, seminars, written materials Face-to-face detailing on deression care Individual rovider feedback on erformance Patient education Patient education classes Written materials Screening/detection Nurses flag charts for susected deression P P P Annual screening olicy with comuter reminders Monitoring/enforcement activities carried out Assessment Provider deression assessment worksheet Provider assessment reminders Care management By a non-md clinician Collaboration with mental health secialists Imroved referral rocess to mental health secialty I I Psychiatrists give feedback to rimary care P P P P, lanned but not imlemented;, lanned and imlemented; I, imlemented but not lanned. rior exosure to imroved deression care among exerimental grou atients (see Fig. 1). Blinding Indeendent telehone interviewers blinded to treatment grou conducted outcome assessments of atients and ractices. Practices and clinicians were blinded as to which atients articiated in the evaluation, excet in the case of 7 atients deemed to be at substantial risk of suicide. Patients were not told whether they attended an exerimental or a usual care ractice. Data Analysis We comared atients in ractices assigned to the exerimental grou with those in ractices assigned to usual care. In our final models, we used analysis of covariance 50 including the baseline value of the tested outcome with logistic regression for categorical and ordinary least squares regression for continuous measures, using SAS software, Version We used the sandwich estimator, also known as the robust variance estimator, 51,52 to adjust standard errors for hierarchical samling with clustering of atients in ractices. We used regression arameters to generate a redicted ercentage or mean for each deendent variable at 6 or 12 months based on all covariates. We weighted data for the robability of enrollment and attrition at each ste. We carried out 1 set of subgrou regression analyses by classifying EBQI ractices into 2 subgrous based on the theoretical strength of their care models and comaring them with usual care ractices (as the reference grou). RESULTS We assessed equivalency of ractice characteristics across grous before any changes in care had been initiated based on a samle of 655 atients (432 in ractices assigned to EBQI and 223 in usual care) (Fig. 1), and on ractice characteristics. We found no significant differences (not shown) between exerimental and usual care ractices in mean selfreorted atient age or distribution across race, gender, marital status, education, or health status. There were also no significant differences in the ercent of atients having a mental health secialty visit or in the ercent diagnosed with deression. In terms of other ractice characteristics, 1 of the 3 usual care ractices was larger in size than the remaining 8 ractices, with 111 hysicians comared with between 22 and 45 clinicians for all remaining ractices (mean 33). Two exerimental and 2 usual care ractices included resident hysicians. Ratios of suort staff and of mental health secialists er rimary care clinician were similar for exerimental and usual care grous, but different across organizations with VA ractices having lower levels of suort staff and higher levels of mental health secialists. 35 Through our qualitative analyses, we examined the effect of EBQI on team imlementation of otentially effective lans. We found that teams imlemented most lanned elements as well as some elements that had not been lanned during the design hase (Table 1). Based on this information, and before any quantitative data analysis, we judged KP Teams #1, #2, and #4 and VA Team #1 to have imlemented care models that were sufficiently adherent to the deression care and rovider behavior literature to have the otential to affect deression outcomes. For examle, the adherent KP teams imlemented care management by a non-md, a key comonent of evidencebased models. Veterans Administration Team #1, in view of

5 JGIM Rubenstein et al., EBQI for Deression in Primary Care 1031 Table 2. Demograhic and Functional Status Characteristics of Exerimental and Usual Care Practices at Survey Baseline Characteristic Exerimental (n=369) Usual Care (n=198) P Value Raw No. Resonding Weighted Mean Weighted % (Raw No.) Raw No. Resonding Weighted Mean Weighted % (Raw No.) %(n) white vs not white w (288) (147).95 %(n) American Indian 1.8 (6) 1.6 (3) %(n) Asian 6.9 (21) 4.2 (8) %(n) African American 5.5 (18) 7.9 (16) %(n) hisanic 10.6 (33) 10.9 (21) % male (163) (71).90 % married (139) (99).21 % working (237) (118).26 % oor health status (26) (19).45 % 3 or more chronic diseases (166) (93).58 % less than high school education (103) (57).78 % oor deression outcome (218) (133).05 % baseline aroriate treatment (145) (89).14 % satisfied or very satisfied with care (150) (75).10 for ersonal roblems % with dysthymia (68) (28).12 % with no alcohol use (138) (79).92 Age (y), mean SD Highest level of education (y), mean SD Household income, mean SD , , , , Mental health comosite of the SF-12, mean SD z Physical health comosite of the SF-12, mean SD z Social functioning, mean SD z Alcohol use, mean SD Window before baseline survey and after date new care models were in lace in exerimental ractices, in days, mean SD Because means and ercents are weighted to reresent the full ractice oulations, ercents will not exactly reresent the raw number in arentheses over the analytic n (raw number resonding). w The distribution of ethnicities across white and all others is also not significant by w 2 (P=.08). z Higher scores indicate better health on these scales. Alcohol use is calculated by multilying the quantity times the frequency of drinking alcohol in the ast 12 mo. oosition to care management by VA leadershi, used screening, comuter clinical reminders that mandated a follow-u action before the visit could be closed, and imroved mental health secialty access. 53 The remaining 2 teams (KP Team #3 and VA Team #2) imlemented educational strategies known not to affect outcomes. 10,11 VA Team #2 imlemented reminders with no enforcement, a strategy reviously shown to be ineffective for major deression. 9,40 Table 3. Deression Treatment and Outcomes for Patients in Exerimental Versus Usual Care Practices Process or Outcome/Survey Point Analytic N Percentage or Mean w (95% CI) Significance: P Value Effect Size Exerimental Practices Usual Care Practices % comleting aroriate treatment 6 mo (41.3 to 50.6) 39.9 (34.7 to 45.2) mo (37.8 to 53.5) 47.0 (42.7 to 51.3) Mean satisfaction with articiation in care 6 mo (54.6 to 60.3) 54.3 (50.9 to 57.6) Mean social functioning (SF) 6 mo (55.9 to 66.7) 58.6 (54.7 to 62.5) mo (59.1 to 70.8) 56.8 (52.7 to 60.9) Percentage oor deression outcome 6 mo (25.9 to 41.4) 36.4 (30.5 to 42.2) mo (18.1 to 35.8) 36.1 (24.0 to 48.4) Po.05. w All regressions controlled for covariates (age, sex, comletion of high school, ethnicity, count of chronic diseases, marriage, alcohol use, dysthymia, household wealth, timing of enrollment) and baseline values of the deendent variable. Satisfaction with articiation in care is controlled for baseline satisfaction with care for ersonal roblems.

6 1032 Rubenstein et al., EBQI for Deression in Primary Care JGIM The evaluation samle, enrolled after new deression care models were in lace in exerimental ractices, included 567 atients (369 exerimental and 198 usual care) with major deression. As shown in Table 2, there were no significant differences between atients in EBQI and usual care ractices at baseline in demograhics, hysical or mental health-related quality of life, or in the timing of the baseline survey in relationshi with exerimental grou atients otential exosure to new care models. Evidence-based quality imrovement ractices had a 9% lower roortion of atients who met criteria for our deression outcome at baseline than did usual care ractices (P =.05). As shown in Table 3, controlling for baseline values, by 6 months after baseline 46.0% of atients in exerimental ractices had comleted aroriate deression treatment comared with 39.9% in usual care ractices. This difference of 6.0% (confidence interval [CI] 0.00, 0.12, P =.07, effect size 0.17) was at a trend level of significance. Also, by 6 months, atient satisfaction with articiation in care was 3 oints higher for atients in exerimental ractices comared with usual care, and this difference was statistically significant (CI 0.94, 5.40, P =.02, effect size 0.14). By 12 months, also shown in Table 3, deressed atients in EBQI ractices had significantly higher levels of social functioning, scoring 8.2 oints higher (better) on the social functioning scale of the SF-36 than did those in usual care ractices (CI 2.8, 13.6; P=.02, effect size 0.23). Also, by 12 months, 27.0% of atients in EBQI ractices were exeriencing a oor deression outcome, comared with 36.1% of atients in usual care ractices. This difference of 9.1% (CI 21.1, 2.6, P =.18, effect size 0.12) was not significant. Results for the mean number of deression symtoms on the CESD and the mean score on the mental health comonent of the SF-12, both of which are comonents of our measure of oor deression outcomes, similarly favored the exerimental grou but were not significant. There were also no significant differences in exosure to or helfulness of educational classes or materials. In our subgrou analysis, at 6-month follow-u 46.4% of 247 deressed atients in the 4 EBQI ractices we had judged as imlementing theoretically stronger care models had comleted aroriate deression treatment, a significant difference of 6.4% from usual care (CI 0.01, 0.12, P=.05, effect size 0.26). At 12 months, 21.9% of atients in ractices with more theory-based care models were exeriencing a oor deression outcome, comared with 36.1% of atients in usual care, a significant difference of 14.2% from usual care (CI 23.8, 4.7, P =.03, effect size 0.31). DISCUSSION As is the case for other chronic conditions, 1,2,54 research indicates that national erformance imrovement for deression is unlikely to occur without widesread imlementation of new deression care models. 55 Effective deression care models are relatively low cost once imlemented, but require organizational change. 56 Continuous quality imrovement-based methods hel organizations and ractices self-design and imlement imrovements. In this study, in contrast to revious CQI studies, ractice-based QI teams received EBQI manuals, 5 key senior leader goals for deression care, a deression care model tool kit, and relevant literature. Organizational quality imrovement leaders also reviewed and aroved team roosals before imlementation. Our results signal both encouragement and caution to those interested in using CQI-based methods for moving the abundant knowledge about effective deression care models into routine use in rimary care settings. Our study also oints toward ways to imrove current QI and QI evaluation aroaches. The study signals encouragement because it shows a ercetible effect of EBQI on ractice erformance and 2 outcomes, atient articiation and social functioning, whereas revious studies on CQI for deression have been entirely negative. These results suggest that the tyes of imrovements we made to CQI are on the right track. Conversely, these findings caution against assuming that current CQI-based methods will roduce large imacts at the ractice oulation level within short time frames. Effects on deression outcomes were absent or minimal. We consider the low-level effects we observed on social functioning and satisfaction with articiation in care to be early sign osts on the ath toward imroving deression outcomes through routine QI. These markers are relevant to imroving deression outcomes, as shown in other studies 39,43 but may be achievable through nonsecific care model changes such as atient and clinician education 40 in the absence of secific changes such as care management that directly suort comletion of major deression treatments. Our results are most alicable to managers in large ractices or health care systems who intend to imrove erformance across diverse clinicians and ractices. We aimed to test a quality imrovement method under routine conditions. By design, no ractices we aroached had articiated in deression quality imrovement, and although all agreed to articiate, they were not best-case examles. For examle, the mental health leadershi for the 2 VA ractices oenly oosed rimary care treatment of deression, 34,53 a realistic imediment likely to be encountered in some ractices. Some KP rimary care leaders were not suortive of deression care imrovement. 34 Nevertheless, all QI teams develoed and imlemented imrovement lans. Some team lans aeared aroriate for bringing a ractice from recontemlation into contemlation of deression care imrovement. 57,58 As we showed in revious work, the ractices imlementing theoretically weaker care models had lower levels of team leader exertise and/or leadershi suort. 34 A steed QI model that adjusts a ractice s goals to its readiness for change may imrove return on QI investment among low readiness ractices. 59 We rovided EBQI teams with senior organizational leader riorities, identified using exert anel methods 34,35,38 as guidance. Teams resonded to these riorities, with no KP teams imlementing screening and no VA teams imlementing rimary care-based treatment suort through care management. Both organizations rioritized imroving atient knowledge and engagement, and nearly all teams addressed this goal, ossibly accounting for the imrovements we saw in atient articiation. Both organizations also rioritized imroving clinician assessment of deression. A reviously ublished analysis of suicide assessment rates based on this and a comanion study suggests that our exerimental ractices imroved clinician assessment. 60 The results of this study, and our resentation of these results, should challenge researchers to consider which evaluation designs and reorting standards should be used for QI

7 JGIM Rubenstein et al., EBQI for Deression in Primary Care 1033 interventions. In our design, like other randomized evaluations of CQI, 22,25 27,61,62 enrolled atients were not told whether they were in the exerimental grou and study clinicians did not know which atients were enrolled, reflecting the goal of testing imacts on ractice erformance. Other cluster randomized studies of care models test outcomes on atients enrolled based on willingness to be referred to the study care model, which may only be available to research atients. 14,15 Some care model studies are randomized at the atient level. 12 Some are nonrandomized. 63 These tyes of studies serve different uroses; yet, researchers or managers may not distinguish between them in drawing conclusions. In addition, features often not reorted in QI studies may affect how the information should be used, such as the degree to which researchers control care model design, the quantity of research resources rovided to teams and organizations, and the characteristics of articiating ractices and organizations. Consort Criteria, 64 the current standard reorting criteria for randomized trials, address some but not all relevant design issues. More comrehensive theory and guidelines for using, reorting, and interreting intervention and evaluation designs for QI studies are needed. 65 Given our design, even the minimal changes we observed may be a cause for cautious otimism because the deression care models in our study ractices were no more available to the reresentative deressed atients enrolled in our evaluation than to any other deressed atient in the ractice. Exerimental ractices were visited by a total of about 165,000 atients during the year ost-imlementation. If we assume that about 9% of rimary care atients have major deression, a 9% lower rate of oor deression outcomes such as we observed in exerimental comared with usual care ractices reresents about 1,200 additional treatment successes in the exerimental grou. Estimating a 60% recovery rate from treatment, these additional treatment successes reresent about 2,000 treated atients, and as many as 3,500 identified for ossible treatment through screening or case-finding. Reaching a significant difference at a ractice level is challenging, yet is what we must aim for if we wish to show true imact on routine care. We carried out 1 hyothesis-generating subgrou analysis. Evidence-based quality imrovement s lack of effect on deression-secific outcomes could have been due to ineffectiveness in stimulating ractices to adot evidence-based imrovements or due to ineffectiveness of evidence-based imrovements as imlemented by ractices. Our results, showing that EBQI ractices imlementing theoretically stronger care models yielded better deression outcomes, rovide suort for future efforts to imrove the link between articiating in EBQI and imlementing evidence-based care model elements. For examle, initial and ossibly eriodic review of teams care models to assess fidelity to evidence shows romise. 34,59 More centralized, exert-suorted QI methods also aear romising for roducing models with higher fidelity to evidence. 15,34 Our study has limitations. Our ower to detect differences in outcomes was limited by the small number of ractices involved, and the conservative adjustment for hierarchical samling that we used to account for this. 51,52 The organizations in our study were large, not-for rofit, staff-model health maintenance-tye organizations, which have different deression-related characteristics than other tyes of managed care or feefor-service ractices. 66 Findings may not be generalizable across ractice tyes. Also, some QI aroaches urosely target ractices that are ready for change. In contrast, in this study organizations selected ractices for not having demonstrated rior interest in deression. Our subgrou analysis is hyothesis generating only, does not address causality, and is not relevant to conclusions regarding EBQI effectiveness. Finally, we reort only research suort dollars rovided for EBQI, not costs. In summary, this study is the first randomized exeriment showing that ractices can self-design and self-imlement imroved deression care models using a CQI-based aroach and gain a ercetible imact on deression-related ractice outcomes. Looking beyond its outcomes, our study shows the comlexity of both real-world QI and the evaluation designs needed for studying it. By exosing readers to this comlexity, we hoe to stimulate the develoment of new intervention aroaches and evaluation standards that more fully take into account the needs and challenges of translating research evidence on effective care models into routine care solutions. L.V.R. was the rincial investigator and is guarantor. She designed the study along with L.S.M. and L.E.P. C.O. led data collection. Data analysis was carried out by L.V.R., M.L.L., and S.C.H. with suort from all authors. All authors collaborated on interretation of the data and writing the manuscrit. The authors acknowledge the invaluable contributions of QI team leaders, as well as Christy Klein, BA, Mary Abdun-Nur, MA, Bernadette Benjamin, MS, Cathy Sherbourne, PhD, Becky Mazel, PhD, Chantal Avila, MA, and James Chiesa, MS. Robert Brook, MD, ScD, and Laurence Rubenstein, MD, MPH, reviewed and critiqued the manuscrit. The study could not have taken lace without the suort of Kaiser Permanente in Northern California and of the VA Greater Los Angeles Healthcare System, Los Angeles, California. Funding sources: The National Institute for Mental Health, the MacArthur Foundation, and the VA Health Services Research and Develoment Center of Excellence for the Study of Healthcare Provider Behavior funded the Mental Health Awareness Project uon which this aer is based. The study funders had no involvement in this ublication. REFERENCES 1. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; Institute for the Future. Health and Health Care 2010: The Forecast, the Challenge. 2nd edn. New York: John Wiley & Sons; Schulberg HC, Katon W, Simon GE, Rush AJ. Treating major deression in rimary care ractice: an udate of the agency for health care olicy and research ractice guidelines. Arch Gen Psychiatry. 1998;55: Deression Guideline Panel. Clinical Practice Guideline, Deression in Primary Care: Vol. 1. Detection and Diagnosis (AHCPR Publication No ). Rockville, MD: U.S. Deartment of Health and Human Services, Public Health Service Agency for Health Care Policy and Research; Deression Guideline Panel. Clinical Practice Guideline, Deression in Primary Care: Vol. 2. Treatment of Major Deression (AHCPR Publication No ). Rockville, MD: U.S. Deartment of Health and Human Services, Public Health Service Agency for Health Care Policy and Research; Charbonneau A, Rosen AK, Ash AS, et al. Measuring the quality of deression care in a large integrated health system. Med Care. 2003;41: Kerr EA, McGlynn EA, Van Vorst KA, Wickstrom SL. Measuring antideressant rescribing ractice in a health care system using administrative data: imlications for quality measurement and imrovement. Jt Comm J Qual Imrov. 2000;26:

8 1034 Rubenstein et al., EBQI for Deression in Primary Care JGIM 8. Olfson M, Marcus SC, Druss B, Elinson L, Tanielian T, Pincus HA. National trends in the outatient treatment of deression. JAMA. 2002;287: Rollman BL, Hanusa BH, Lowe HJ, Gilbert T, Kaoor WN, Schulberg HC. A randomized trial using comuterized decision suort to imrove treatment of major deression in rimary care. J Gen Intern Med. 2002;17: Dobscha SK, Gerrity MS, Ward MF. Effectiveness of an intervention to imrove rimary care rovider recognition of deression. Eff Clin Pract. 2001;4: Thomson C, Kinmonth AL, Stevens L, et al. Effects of a clinical-ractice guideline and ractice-based education on detection and outcome of deression in rimary care: hamshire deression roject randomised controlled trial. Lancet. 2000;355: Katon WJ, Von Korff M, Lin E, et al. Collaborative management to achieve treatment guidelines: imact on deression in rimary care. JAMA. 1995;273: Rost K, Nutting P, Smith J, Werner J, Duan N. Imroving deression outcomes in community rimary care ractice: a randomized trial of the quest intervention. Quality enhancement by strategic teaming. J Gen Intern Med. 2001;16: Wells KB, Sherbourne C, Schoenbaum M, et al. Imact of disseminating quality imrovement rograms for deression in managed rimary care: a randomized controlled trial. JAMA. 2000;283: Dietrich AJ, Oxman TE, Williams JW Jr., et al. Re-engineering systems for the treatment of deression in rimary care: cluster randomised controlled trial. BMJ. 2004;329: Unutzer J, Katon W, Callahan CM, et al. Collaborative care management of late-life deression in the rimary care setting: a randomized controlled trial. JAMA. 2002;288: Hunkeler EM, Meresman JF, Hargreaves WA, et al. Efficacy of nurse telehealth care and eer suort in augmenting treatment of deression in rimary care. Arch Fam Med. 2000;9: Donabedian A. Exlorations in Quality Assessment and Monitoring. Vol. I: The Definition of Quality and Aroaches to its Assessment. Ann Arbor, MI: Health Administration Press; Schoenbaum M, Unutzer J, Sherbourne C, et al. Cost-effectiveness of ractice-initiated quality imrovement for deression: results of a randomized controlled trial. JAMA. 2001;286: Liu CF, Hedrick SC, Chaney EF, et al. Cost-effectiveness of collaborative care for deression in a rimary care veteran oulation. Psychiatr Serv. 2003;54: Simon GE, Katon WJ, VonKorff M, et al. Cost-effectiveness of a collaborative care rogram for rimary care atients with ersistent deression. Am J Psychiatry. 2001;158: Shortell SM, Bennett CL, Byck GR. Assessing the imact of continuous quality imrovement on clinical ractice: what it will take to accelerate rogress. Milbank Q. 1998;76: McLaughlin CP, Kaluzny CA. Continuous Quality Imrovement in Health Care. 2nd edn. Gaithersburg, MD: Asen; Berwick DM. Continuous imrovement as an ideal in health care. N Engl J Med. 1989;320: Brown Jr. BW, Shye D, McFarland BH, Nicholas GA, Mullooly JP, Johnson R. Controlled trials of CQI and academic detailing to imlement a clinical ractice guideline for deression. Jt Comm J Qual Imrov. 2000;26: Horowitz CR, Goldberg HI, Martin DP, et al. Conducting a randomized controlled trial of CQI and academic detailing to imlement clinical guidelines. Jt Comm J Qual Imrov. 1996;22: Solberg LI, Fischer LR, Wei F, et al. A CQI intervention to change the care of deression: a controlled study. Eff Clin Pract. 2001;4: Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to imrove rofessional ractice. Can Med Assoc J. 1995;153: Davis DD, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME: a review of 50 randomized controlled trials. JAMA. 1992;268: Grimshaw JM, Shirran L, Thomas R, et al. Changing rovider behavior: an overview of systematic reviews of interventions. Med Care. 2001;39(Sul):II Cook TDCDT. Quasi-Exerimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin Comany; Grimshaw J, Cambell M, Eccles M, Steen N. Exerimental and quasiexerimental designs for evaluating guideline imlementation strategies. Fam Pract. 2000;(Sul 1): Agency for Health Care Policy and Research. Clinical Practice Guideline: Quick Reference Guide for Clinicians. AHCPR Publication No Rockville, MD: Agency for Health Care Policy and Research; Rubenstein LV, Parker LE, Meredith LS, et al. Understanding team-based quality imrovement for deression in rimary care. Health Serv Res. 2002;37: Rubenstein LV, Meredith L, Parker LE, et al. Design and Imlementation of Evidence-based Quality Imrovement for Deression in the Mental Health Awareness Project: Background, Methods, Evaluation, and Manuals, MR1699-NIMH. Santa Monica: RAND Cororation; Rubenstein LV, Jackson-Triche M, Unutzer J, et al. Evidence-based care for deression in managed rimary care ractices. Health Aff (Millwood). 1999;18: Deression Patient Outcomes Research Team (PORT-II). Partners in care (PIC) tool for an integrated aroach to imroving care for deression in rimary care. Available at: htt:// ic.roducts/. Accessed February 3, Rubenstein LV, Fink A, Yano EM, Simon B, Chernof B, Robbins AS. Increasing the imact of quality imrovement on health: an exert anel method for setting institutional riorities. Jt Comm J Qual Imrov. 1995;21: Sherbourne C, Schoenbaum M, Wells KB, Croghan TW. Characteristics, treatment atterns, and outcomes of ersistent deression desite treatment in rimary care. Gen Hos Psychiatry. 2004;26: Rubenstein LV, McCoy JM, Coe DW, et al. Imroving atient quality of life with feedback to hysicians about functional status. J Gen Intern Med. 1995;10: World Health Organization. Comosite International Diagnostic Interview (CIDI) Core Version 2.1 Interviewer s Manual. Geneva: World Health Organization; Meredith LS, Orlando M, Humhrey N, Cam P, Sherbourne CD. Are better ratings of the atient rovider relationshi associated with higher quality care for deression? Med Care. 2001;39: Clever S, Ford DE, Rubenstein LV, et al. Primary care atients involvement in decision making is associated with imrovements in deression. Med Care. 2006; 44: Sherbourne CD, Wells KB, Duan N, et al. Long-term effectiveness of disseminating quality imrovement for deression in rimary care. Arch Gen Psychiatry. 2001;58: Radloff L. The CES-D scale. A self-reort deression scale for research in the general oulation. Al Psychiatric Measures. 1977;1: Ware JE Jr, Kosinski M, Keller S. SF-12: How to Score the SF-12 Physical and Mental Health Summary Scales. Boston, MA: The Health Institute, New England Medical Center; Ware JE Jr, Sherbourne CD. The MOS 36-item short-form health survey (SF-36). I. Concetual framework and item selection. Med Care. 1992;30: Bohn MJ, Babor TF, Kranzler HR. The alcohol use disorders identification test (AUDIT): validation of a screening instrument for use in medical settings. J Stud Alcohol. 1995;56: Smith JP. Racial and ethnic differences in wealth in the health and retirement study. J Hum Resour. 1995;30(sul):S van Belle G, Fisher LD, Heagerty PJ, Lumley T. Biostatistics: A Methodology for the Health Sciences. 2nd edn. Hoboken, NJ: Wiley-Interscience; McCaffrey DF, Bell RM. Bias reduction in standard errors for linear regression with multi-stage samles. AT&T Technical Reort TD- 4S9H9T; Williams RL. A note on robust variance estimation for cluster-correlated data. Biometrics. 2000;56: Sherman SE, Chaman A, Garcia D, Braslow JT. Imroving recognition of deression in rimary care: a study of evidence-based quality imrovement. Jt Comm J Qual Safety. 2004;30: Wagner EH, Austin BT, Von Korff M. Organizing care for atients with chronic illness. Milbank Q. 1996;74: Katon WJ. The Institute of Medicine Chasm reort: imlications for deression collaborative care models. Gen Hos Psychiatry. 2003;25: Rosenheck RA. Organizational rocess: a missing link between research and ractice. Psychiatr Serv. 2001;52: Lehman WE, Greener JM, Simson DD. Assessing organizational readiness for change. J Subst Abuse Treat. 2002;22:

9 JGIM Rubenstein et al., EBQI for Deression in Primary Care Weeks B, Helms MM, Ettkin LP. A hysical examination of health care s readiness for a total quality management rogram: a case study 1. Hos Mater Manage Q. 1995;17: Gustafson DH, Sainfort F, Eichler M, Adams L, Bisognano M, Steudel H. Develoing and testing a model to redict outcomes of organizational change. Health Serv Res. 2003;38: Nutting PA, Dickinson LM, Rubenstein LV, Keeley RD, Smith JL, Elliot CE. Imroving detection of suicidal ideation among deressed atients in rimary care. Ann Fam Med. 2005;3: Solberg LI, Kottake TE, Brekke ML, et al. Failure of a continuous quality imrovement intervention to increase the delivery of reventive services: a randomised trial. Eff Clin Pract. 2000;3: Flottor S, Havelsrud K, Oxman AD. Process evaluation of a cluster randomized trial of tailored interventions to imlement guidelines in rimary care why is it so hard to change ractice? Fam Pract. 2003;20: Wagner EH, Glasgow RE, Davis C, et al. Quality imrovement in chronic illness care: a collaborative aroach. Jt Comm J Qual Imrov. 2001;27: Cambell MK, Elbourne DR, Altman DG. CONSORT statement: extension to cluster randomised trials. BMJ. 2004;328: Davidoff F, Batalden P. Toward stronger evidence on quality imrovement. Draft ublication guidelines: the beginning of a concensus roject. Qual Saf Health Care. 2005;14: Meredith LS, Rubenstein LV, Rost K, et al. Treating deression in staff-model versus network-model managed care organizations. J Gen Intern Med. 1999;14: Sulementary Material The following sulementary material is available for this article online at Aendix 1. Timing of Evidence-Based Quality Imrovement (EBQI) and Research Evaluation Activities. Aendix 2. The To Five Priorities for Imroving Deression Care at Kaiser and at the VA. Aendix 3. MHAP Patient Samle Flow.