Primary Care Mental Health 2005;3:00 00 # 2005 Radcliffe Publishing International research The effectiveness of a primary care mental health service delivering brief psychological interventions: a benchmarking study using the CORE system Naomi Gilbert BSc Research Officer Michael Barkham BEd MA MSc PhD Professor of Clinical and Counselling Psychology Psychological Therapies Research Centre, University of Leeds, UK Ann Richards BSc MPH RN Primary Care Mental Health Manager Ian Cameron MBChB MPH MA FFPHM Director of Public Health Leeds North West Primary Care Trust, Leeds, UK ABSTRACT High demand for psychological therapies in primary care has led to increased interest in the effectiveness of brief interventions. Our aims were to (1) profile patients and their presenting problems accessing a new Primary Care Mental Health (PCMH) service; (2) evaluate the effectiveness of the PCMH service in terms of patient outcomes; and (3) compare service parameters with those derived from national primary care counselling. The PCMH service was delivered in 54 general practitioner (GP) practices and provided mental health assessment, brief psychological interventions (usually up to six sessions), and onward referral to specialist services where required. Data were routinely collected over a 34-month period regarding (1) access to the PCMH service (number of patients referred, demographics, nonattendance rates, assessment outcome and completion of treatment) and (2) patient outcome using the CORE outcome measure. Outcome data were compared with a national dataset from 33 primary care counselling services. In total 6750 patients were referred to the PCMH service and 5539 had been discharged by the end of the evaluation. The non-attendance rate for assessment was 26.1%. One in five patients referred to the service completed therapy sessions. Outcomes of treatment were comparable with primary care counselling data. Patients choosing to complete treatment after three sessions achieved the greatest gains on average. Attrition is a key problem for efficient mental health service delivery in primary care. Evaluating this new service against national comparators of more established service delivery is a clinically useful method for monitoring outcomes of brief interventions in primary care. Keywords: to come?
2 N Gilbert, M Barkham, A Richards et al Introduction One in four people in the UK will experience a mental health problem in the course of a year. 1 This high prevalence of mental health problems has a large impact on primary care with up to 40% of general practitioner (GP) consultations relating to mental health. 2 Current government policy and primary care-led commissioning have led to an increase in the provision of psychological therapies in primary care settings. 3 5 The Department of Health has reviewed the organisation and delivery of psychological therapies. 6 Among the recommendations were that it would be helpful to prioritise the development of primary care-based psychological therapies for people with common disorders and co-ordinate provision so that access is equitable in that service delivery is not restricted by extraneous factors such as ethnicity, age, gender, and diagnosis. The recent clinical guidelines for depression produced by the National Institute for Clinical Excellence (NICE) recommend as follows (p. 7): In mild and moderate depression, consider psychological treatment specifically focused on depression (problemsolving therapy, brief CBT [cognitive behavioural therapy] and counselling) of 6 to 8 sessions over 10 to 12 weeks. 7 The trend towards the need for cost-effective and practical delivery of psychological treatment within primary care has prompted growing interest in brief intervention models (up to six sessions). 8,9 Access Any evaluation of the delivery and effectiveness of such services needs to take into account the minimum components of access (i.e. input) and outcomes (i.e. output). 10 Access relates to effective service delivery (e.g. referral rates, patient demographics, attendance and patient flow through the service). How mental health services should be configured in order to deliver to this continuing agenda has produced a number of contributions in the literature. 11,12 Other models focusing on the primary/secondary interface have also been espoused (Tomson D, personal communication, 2000). They range from individual link workers, a whole team approach based in primary care (Chester model) and tiered models such as those pioneered in Northumberland. 13 The key component in terms of the National Service Framework for Mental Health (NSF-MH) 3 is to provide accessible services for patients that are appropriate to their needs. Both the NICE guidelines for depression and a recent commissioning guide, advocate a stepped care approach with the former providing guidance of between six and eight sessions. Stepped care offers the least intensive treatment to meet the need of the patient, with a stepping up if they are not achieving significant health gain. 14 While setting a limit regarding the number of sessions is understandable in terms of limited NHS resources, there will be concerns as to whether such a limit is sufficient for patients within routine service settings. This concern arises from both empirical literature questioning whether effects of interventions including duration transpose from research settings to routine practice, 15 and also from questions regarding conceptual differences between research and practice paradigms. 16 Accordingly, there is both a pragmatic and an empirical argument for evaluating such a model in routine primary care services. Delivery by primary care mental health practitioners A primary care mental health (PCMH) service was developed in response to improving services at tier 2 level of the NSF-MH. Tier 2 services refer to services which allow people with common mental health problems that is, anxiety and depression to be cared for in their local community such as their GP surgery. For most primary care teams this meant enhanced training for all staff and the provision of services within primary care that could deliver brief interventions such as cognitive behavioural therapy and develop closer links with the voluntary sector. One means of enhancing the capacity and capability of primary care was through the development of primary care mental health practitioners. As such, these practitioners are clearly distinguished from graduate primary care workers. Outcomes A further key recommendation of the Department of Health report was the requirement for the effectiveness of such interventions to be routinely assessed by exploring the means to measure outcomes routinely, including quality of life and service user satisfaction. 3 Guidance on this agenda has been disseminated in a report from the National Institute for Mental Health in England entitled Outcome Measures Implementation: best practice guidance. 17 This report introduces four levels of outcomes rollout: level 1 relating to issues of measurement implementation; level 2 relating to the monitoring of data; level 3 referring to the use of data in the management of cases, and level 4 relating to benchmarking
A benchmarking study using the CORE system 3 against national data. Simply auditing access and outcomes of new services is of limited value without the ability to compare with other types of intervention. We therefore built upon this approach by comparing the data yielded by the present study against benchmark data using the same outcome measure. This component requires the routine use of a standardised outcome measure. However, it also requires the ability to group patient outcomes into clinically meaningful categories. Hence, rather than using simple pre post differences in the outcome measure as an indication of change, we adopted stringent criteria for determining reliable and clinically significant change. 18 This enabled us to report the percentage of patients meeting specific outcome thresholds. Accordingly, this paper examines the effectiveness of a new PCMH service and has three aims: (1) to examine the sample and presenting problems of people accessing the PCMH service; (2) to evaluate patient outcomes for people accessing the service; and (3) to compare those outcomes with data from national primary care counselling services. Method Service Five primary care trusts (PCTs) covering one large northern city developed the PCMH service between May 2000 and January 2002. The service provided mental health assessment and brief psychological interventions (usually up to six sessions using, for example cognitive behavioural approaches) for patients aged 16 to 65 years with common mental health problems. The service also provided liaison with specialist services and capacity building within the primary care team. It was provided to 54 GPs who made the majority of referrals (97.5%), with the remaining referrals coming from other members of the primary healthcare team. Referrals were directed to individual practitioners with patients seen in GP surgeries. The 33 PCMH practitioners were employed on the basis of core competencies including skills in assessment and brief psychological interventions. They came from a range of professional backgrounds including community psychiatric nursing, counselling, occupational therapy and social work. A short cognitive behavioural therapy-focused course was offered to all new practitioners. Data collection Integral to the development of this new service was the establishment of data collection systems for routine evaluation. All patients referred to the service received an information sheet about data collected, and returned a signed declaration to their practitioner if they did not wish to complete measures. Data were recorded routinely by the practitioners and collected monthly by the evaluation team for a period of 34 months. This included the number of patients referred, non-attendance rates, assessment outcomes and treatment completion rates. All data were anonymised and aggregated. Measures Patients completed the CORE outcome measure (CORE-OM), a 34-item scale measuring the domains of symptoms, functioning, wellbeing and risk. 19 21 The mean item score ranges from 0 to 4, and a high score represents increased problem severity. We followed procedures reported elsewhere for multiplying the mean item score by 10 and referring to this as a clinical score. 22 Using this format, cut-off scores differentiating clinical and non-clinical populations have been calculated as 11.9 (males) and 12.9 (females). A pre-treatment CORE-OM form was sent to each patient with their appointment letter, with instructions to complete and bring it to the first session. Occasionally, forms were completed at the subsequent session if patients did not bring them to the initial assessment. A post-treatment CORE-OM was completed on discharge. Patients who did not have a planned ending were sent a post-treatment CORE-OM to complete and return. PCMH practitioners completed the CORE therapy assessment form (CORE-A) comprising information on patient demographics and presenting problems. 21,23,24 Analysis CORE data were computer scanned using Formic 3 for Windows (Formic Design and Automatic Data Capture, 1996). Statistical analyses were carried out using SPSS for Windows version 11. Confidence intervals were calculated for percentages, 25 and chi-squared analysis was applied to test for differences. Given the large sample size and high statistical power the cut-off point for statistical significance was at P < 0.01. For CORE-OM outcome data, pre post effect sizes of treatment and confidence intervals were calculated using Microsoft Excel 2000. 26,27 The calculation used for effect sizes was the pretreatment mean score minus the post-treatment
4 N Gilbert, M Barkham, A Richards et al mean score, divided by the pooled standard deviation. Clinically significant changes in CORE-OM scores were calculated by determining parameters of reliable and clinically significant change, which were applied to an on-treatment sample. 18 Reliable and clinically significant change analyses concern the evaluation of change between two time points for each case, taking into account the relationship of the individual change scores to pre-treatment sample scores and the psychometric reliability of the outcome measure. For the CORE-OM, a change in pre post clinical scores greater than 4.8 is considered reliable (i.e. not attributable to measurement error). Change in which a patient s pre-treatment score was above the clinical cut-off before treatment and below the clinical cut-off after treatment was defined as clinically significant change. Data samples In total, 6750 patients were referred to the PCMH service during the 34-month evaluation. Three subsamples of these patients are reported on, as set out below. Sample 1: closed cases i.e. patients discharged from the service By the end of the evaluation period in March 2003, 5539 patients (82.1% referrals) had been discharged from the service. Data regarding access to the service and throughput are reported using this sample. Figure 1 illustrates the flow through the service per 100 discharged cases. The remaining 1211 referrals were still active cases at the end of the evaluation, i.e. on the waiting list for assessment or receiving ongoing treatment. Referred 100 Offered assessment 75 Not offered assessment 25 Attended assessment 55 Did not attend assessment 20 Offered therapy sessions 37 Not offered therapy 18 Attended therapy sessions 32 Did not attend sessions 5 Completed therapy 18 Referred onward 4 Dropped out of therapy 10 Figure 1 Illustrated service pathway per 100 closed cases (actual n = 5539)
A benchmarking study using the CORE system 5 Sample 2: patients for whom CORE-A demographic data were completed In total, 2425 patients who attended for assessment agreed to take part in the evaluation using the CORE system. Practitioners completed a CORE-A form for 2276 (93.9%) of these patients, and rated presenting problems on this form for 2132 (87.9%) patients. Demographic and problem-mix data are reported using this sample, and are compared with data from national counselling services. Sample 3: patients completing pre- and post-treatment outcome measures A total of 553 patients taking part in the evaluation completed the CORE-OM before and after intervention with the service. Outcomes of treatment are reported for this sample of patients, and benchmarked against national data. Following assessment, 2047 (66.7%) of the 3069 attending assessment were offered further therapy sessions. Those not offered therapy (n = 1022, 33.3%) were either discharged from the service with no further action, e.g. if the patient declined therapy (n = 386, 12.6%), or referred on to other agencies (n = 636, 20.7%). Of the 2047 patients offered therapy sessions, 1761 (86.0%) attended one or more therapy session, and 286 (14.0%) did not return to the service to attend further therapy appointments. Of the 1761 patients attending therapy sessions, 982 (55.8%) completed sessions with an agreed ending of treatment, 582 (33.3%) dropped out of therapy, and 197 (11.2%) were referred on to other services for further support after therapy sessions with the PCMH service. As illustrated in Figure 1, this equates to approximately one in five closed cases completing treatment with an agreed ending. Comparative data In order to draw comparisons with existing service delivery models, we utilised a national dataset from 33 UK primary care-based counselling or psychology services in which the CORE system was routinely used (n = 6610 CORE-A demographic data, n = 2494 CORE-OM outcome data). 28 Results Access Figure 1 illustrates the flow through the service per 100 discharged patients or closed cases. Of the 5539 closed cases, a total of 1387 (25.0%) were not offered an assessment appointment. The primary reason for discharge at this stage was patients failing to contact the service to opt in i.e. confirm that they would like to attend an assessment (n = 840, 15.2%). Other patients were referred directly to more suitable agencies (n = 333, 6.0%) or considered inappropriate and directed back to the referrer (e.g. outside the age range for the service) (n = 214, 3.8%). Of the 4152 (75.0%) closed cases who were offered an assessment, 1083 (26.1%) were discharged following non-attendance. In total 151 (3.6%) patients cancelled the appointment and 932 (22.4%) failed to attend without giving prior notice. Attrition rates at referral and assessment were high, with a total of 3069 (55.4%) closed cases actually attending the initial assessment appointment. Patient demographics Of the 2205 patients for whom practitioners completed a CORE-A, a total of 69% were women and 86.3% were from white European ethnic backgrounds. The mean age was 36.2 years (standard deviation (SD) = 12.2 years), with 91.3% of the sample falling within the age range of 20 59 years. A total of 58.5% of the sample were in full- or parttime employment and 11.0% were unemployed. In terms of living status, 47.9% were living with a partner, 24.8% were living alone and 45.3% were caring for children. The majority of patients were female in both the PCMH service and in the comparative data set (PCMH service 69.0% versus 71.5%, 2 = 5.1, P = 0.02). Over half of all patients were in full-time or part-time employment in both PCMH service and comparative data (PCMH 58.5% versus 54.3%, 2 = 5.2, P = 0.02). The PCMH service saw a significantly higher proportion of patients from non-white European ethnic backgrounds (13.7% versus 8.8%, 2 = 39.6, P < 0.001), and a significantly lower proportion were over 59 years of age (3.3% versus 6.5%, 2 = 30.3, P < 0.001). PCMH service patients were also significantly more likely to be taking prescribed psychotropic medication (52.5% versus 48.0%, 2 = 12.8, P < 0.001). Problem mix Table 1 illustrates the frequency of presenting problems as rated by practitioners for patients in the PCMH service (n = 2132) and the comparative dataset (n = 6610). Depression and anxiety were the most
6 N Gilbert, M Barkham, A Richards et al Table 1 Percentage of patients presenting with identified problems at assessment Problem PCMH service (n = 2132) Benchmark (n = 6610) Difference % 95% CI % 95% CI 2 P Anxiety 80.6 78.8 to 82.2 79.3 78.3 to 80.3 1.5 0.21 Depression 75.4 73.5 to 77.2 73.3 72.2 to 74.4 3.4 0.06 Interpersonal problems 45.1 43.0 to 47.2 57.8 56.5 to 59.0 101.0 <0.001 Self-esteem problems 43.9 41.8 to 46.0 51.7 50.5 to 53.0 38.8 <0.001 Bereavement 21.5 19.8 to 23.3 33.1 32.0 to 34.4 101.3 <0.001 Physical problems 20.5 18.8 to 22.3 20.9 19.9 to 22.0 0.2 0.68 Work/academic problems 19.7 18.0 to 21.4 21.0 19.9 to 22.0 1.6 0.20 Trauma/abuse 14.1 12.7 to 15.7 20.0 19.0 to 21.1 36.5 <0.001 Living/welfare problems 13.2 11.9 to 14.7 15.0 14.2 to 16.0 4.2 0.04 Addictions 6.3 5.4 to 7.4 5.6 5.1 to 6.3 1.4 0.24 Personality problems 3.3 2.6 to 4.2 5.5 4.9 to 6.1 15.4 <0.001 Eating disorder 3.0 2.4 to 3.8 3.0 2.6 to 3.5 0.0 0.98 Cognitive problems 1.8 1.3 to 2.5 1.9 1.6 to 2.3 0.1 0.77 Psychosis 0.6 0.4 to 1.0 0.8 0.6 to 1.0 0.6 0.43 common presenting problems, experienced by over 70% patients in both the PCMH service and benchmark data. However, some differences in the problem mix were identified. A significantly lower proportion of PCMH patients presented with interpersonal problems, self-esteem problems, bereavement, trauma, and personality problems in comparison with the benchmark data. Outcomes Of the 1761 patients attending therapy, 1365 (77.5%) agreed to take part in the evaluation and complete the CORE-OM at initial assessment. Of these patients, 760 subsequently had a planned ending of therapy, 153 were referred on, and 452 dropped out of therapy. Matched pre- and post-intervention CORE-OM forms were completed by 553 PCMH service patients in total that is, 40.5% of those who had agreed to take part in the evaluation. Return rates as percentages within each of the above three groups (i.e. proportion of patients who did return both a preand post-treatment CORE-OM) were as follows: completed therapy n = 441 (58.0%), referred on after therapy n = 66 (43.1%), dropped out of therapy n = 35 (7.7%). An additional 11 were completed after extended assessment. Comparative outcome data were available for 2494 patients. Treatment outcomes were recorded simply as planned or unplanned endings in comparative services, and return rates were as follows: unplanned endings (usually meaning dropped out of treatment) n = 136 (5.5%), planned endings n = 1956 (78.4%), unrecorded endings n = 402 (percentage of patients returning pre- and posttreatment measures cannot be calculated as type of treatment ending was not recorded). Treatment effect sizes Group change in mean scores was calculated using effect sizes: PCMH service (n = 553) mean (SD) clinical pre-treatment score, 17.7 (6.6); mean clinical post-treatment score, 8.8 (6.5); pre post effect size, 1.36, (95% confidence interval (CI), 1.23 to 1.49); comparative data (n = 2494) mean pretreatment clinical score, 17.9 (6.5); mean clinical post-treatment score, 8.3 (6.1); pre post effect size, 1.52 (95% CI, 1.46 to 1.59). In both datasets, females
A benchmarking study using the CORE system 7 achieved greater mean gains during treatment than males: females PCMH (n = 394) pre post effect size (95% CI) = 1.38 (1.18 to 1.58); females comparative data (n = 1823) pre post effect size = 1.59 (CI 1.52 to 1.67); males PCMH (n = 159) pre post effect size = 1.30 (1.06 to 1.54); males comparative data (n = 671) pre post effect size = 1.35 (1.23 to 1.46). Reliable and clinically significant change The Jacobson plot (see Figure 2) illustrates outcomes of treatment for PCMH service patients (n = 553). It shows that a high proportion of PCMH patients made reliable improvement following treatment (n = 384, 69.4%), and only a small proportion showed reliable deterioration (n = 7, 1.3%). Table 2 presents reliable and clinical change data for the PCMH service and comparative data. In both datasets, over 50% of patients made reliable and clinically significant improvement. Female patients were significantly more likely to report no reliable change in the PCMH service than in the comparative data, however this trend was not found for male patients. Impact of number of sessions The mean number of sessions attended by patients was 4.6 (SD 2.4) in the PCMH service compared with 5.9 (SD 3.0) in the national data (t = 8.8, P < 0.001). To assess the impact of treatment duration on outcome, patients who had an agreed ending of treatment (i.e. did not drop out of treatment) after 1 6 therapy sessions were selected; 397 PCMH patients and 1434 patients from the comparative dataset met these criteria (a proportion of patients in both the PCMH service and comparative data either dropped out of treatment, or attended more than six sessions, and outcomes for these patients were excluded from this analysis). In Table 3 the mean pre- and posttreatment scores and effect size analysis is presented by number of sessions attended. In both the PCMH service and comparative data, patients choosing to complete treatment after three sessions made the greatest gains. Figure 3 plots the effect sizes and 95% CIs for the PCMH service and comparative data, illustrating a negatively accelerating trend after three sessions. It also indicates that PCMH patients completing treatment after one to three sessions showed greater gains than comparative patients, however this trend was reversed for patients completing treatment after four to six sessions. Post-therapy CORE-OM clinical score 40 30 20 10 0 0 Clinical cut Reliable deterioration No reliable change Clinically significant improvement 10 20 30 40 Pre-therapy CORE-OM clinical score Figure 2 Jacobson Plot showing reliable and clinically significant change for PCMH patients (n = 553). Pretreatment scores are plotted on the x axis against post-treatment scores on the y axis. The clinical cut-off line is shown as a guide on each axis, however this is not exact as cut-offs are dependent on sex. The diagonal tramlines represent the boundaries of reliable change. Points above the diagonal tramlines represent deterioration, points within the tramlines represent unreliable change, and points below the tramlines represent improvement. Points below the horizontal clinical cut-off and to the right of the vertical cut-off represent reliable and clinically significant improvement
8 N Gilbert, M Barkham, A Richards et al Table 2 Percentage of patients meeting components of reliable and clinically significant change based on CORE-OM scores PCMH service (n = 553) Benchmark (n = 2494) Difference % 95% CI % 95% CI 2 P Reliable and clinically significant improvement All 51.7 47.6 to 55.9 56.7 54.7 to 58.6 4.47 0.03 Males 49.1 41.4 to 56.8 51.6 47.8 to 55.3 0.32 0.57 Females 52.8 47.9 to 57.7 58.5 56.3 to 60.8 4.37 0.04 Reliable improvement All 17.7 14.8 to 21.1 18.7 17.2 to 20.3 0.30 0.58 Males 19.5 14.1 to 26.3 19.1 16.3 to 22.2 0.01 0.90 Females 17.0 13.6 to 21.0 18.6 16.9 to 20.4 0.55 0.46 No reliable change All 29.3 25.7 to 33.2 23.2 21.6 to 24.9 9.22 0.002 Males 31.4 24.7 to 39.0 27.4 24.2 to 30.9 1.03 0.31 Females 28.4 24.2 to 33.1 21.6 19.8 to 23.6 8.54 0.003 Deterioration All 1.3 0.6 to 2.6 1.4 1.0 to 2.0 0.10 0.75 Males 0.0 0.0 to 2.4 1.9 1.1 to 3.3 3.13 0.08 Females 1.8 0.9 to 3.6 1.3 0.8 to 1.9 0.64 0.42 Discussion This was a practice-based study using data collected as part of routine evaluation. Shadish et al highlight the importance of such clinically representative studies in terms of generalisability of findings to therapy carried out in actual clinical conditions. 29 The difficulty in evaluating new service developments is that it is not always possible to compare outcomes with other treatment options. The availability of primary care counselling data as a comparator against which to benchmark outcomes enabled this new service to be compared quickly with an existing approach to service delivery. The study highlighted the fact that attrition is a key problem in the delivery of mental health services in primary care. However, the PCMH assessment non-attendance rate of 26% compares favourably with a recent study of a brief primary care psychiatric screening clinic for patients with mild to moderate mental health problems, in which 48% did not attend. 30 This may suggest that less stigmatising referrals (i.e. to mental health workers rather than psychiatry services) can encourage higher patient attendance. Patients accessing the PCMH service were qualitatively different from those accessing primary care counselling in terms of problem mix. A lower proportion of patients referred to the PCMH service had problems traditionally seen within counselling or psychology services such as trauma, bereavement and relationship problems. 31,32 Patient outcomes in the PCMH service were predominately positive and compared favourably with counselling. To further enhance the interpretation of these change scores, look-up tables have been developed for transforming male and female scores from CORE-OM to Beck Depression Inventory (BDI-I) and vice versa. 22 The CORE-OM change scores achieved in the present study equate with BDI-I change scores from approximately 22 to 11 (female) and 19 to 12 (male). Patients attended fewer sessions in the PCMH service than in the comparative service (mean 4.6, SD 2.4 versus mean 5.9, SD 3.0), suggesting that the PCMH service achieved similar outcomes to traditional counselling delivery within a shorter intervention period. However, a significantly higher proportion of female patients reported no change on the CORE-OM in the PCMH service than in the comparative data. This finding may suggest that, for some patients, the brief
A benchmarking study using the CORE system 9 Table 3 Pre post treatment effect sizes by number of sessions (for patients with agreed ending of treatment) Pretreatment CORE-OM clinical score Post-treatment CORE- OM clinical score Effect size (ES) n Mean SD Mean SD ES 95% CI ES PCMH service (n = 397) 1 30 16.8 7.2 7.0 5.8 1.49 0.90 to 2.04 2 57 15.3 7.4 4.9 3.8 1.76 1.31 to 2.17 3 43 17.3 5.6 6.4 4.5 2.16 1.61 to 2.67 4 69 17.2 7.4 8.4 7.0 1.24 0.86 to 1.59 5 70 18.6 5.9 9.5 6.8 1.42 1.04 to 1.78 6 128 17.8 6.7 10.3 6.3 1.15 0.88 to 1.41 Benchmark (n = 1434) 1 12 15.4 6.8 8.2 3.5 1.34 0.41 to 2.17 2 120 15.8 6.3 6.5 4.7 1.69 1.39 to 1.98 3 163 16.6 6.0 6.1 4.9 1.90 1.64 to 2.16 4 237 17.1 6.6 7.1 5.2 1.66 1.45 to 1.87 5 254 17.5 6.4 7.4 5.8 1.66 1.46 to 1.86 6 648 18.2 6.2 9.3 6.2 1.43 1.31 to 1.55 intervention approach of the PCMH service was not sufficient to bring about change. A key finding of this study was that the greatest gains were made in both the PCMH and comparative services by patients choosing to complete their treatment after three sessions. A similar finding has also been reported in a separate large-scale study investigating the relationship between effectiveness and the amount of psychological therapy received. 33 This clearly has implications for more traditional ways of delivering psychological approaches in primary care. A 2 + 1 model, 34 or a stepped care approach as outlined in the NICE guidance for depression may be potential options for delivering such ultrabrief interventions. Limitations This evaluation was limited by the substantial attrition at assessment, and by low response rates in outcome data, particularly for patients dropping out of therapy. In both the PCMH and comparative datasets, less than 8% of those who discontinued treatment returned a pre- and post-treatment outcome measure. This means that the outcome data may be based on a subset of well-motivated patients who engaged with treatment and completed therapy, and therefore findings may not be generalisable to all patients. In addition, over 50% of PCMH patients were prescribed psychotropic medication, and while this is representative of real life clinical practice, the outcomes of the intervention should be interpreted with consideration of the impact of medication. A further limitation is the lack of follow-up data monitoring patient outcomes after discharge from the service. Future practice-based studies could address this by delivering a follow-up session after 3 6 months, at which patients complete an outcome measure to monitor maintenance of gains made during treatment. Implications Attrition and service delivery The finding that one in five patients referred to the PCMH service actually completed treatment has implications for the way mental health services are delivered in primary care. A recently evaluated and successful method of increasing treatment compliance with psychotropic medication in primary care is telephone follow-up. 35 Further research could investigate whether such follow-up could increase attendance and engagement with psychological interventions. Comparisons audit and evaluation The application of a comparative approach illustrates the clinical utility of monitoring outcomes
10 N Gilbert, M Barkham, A Richards et al Pre post treatment effect size Pre post treatment effect size 3.00 2.50 2.00 1.50 1.00 0.50 0.00 3.00 2.50 2.00 1.50 1.00 0.50 0.00 1 1 PCMH service 2 3 4 5 6 n sessions Benchmark 2 3 4 5 6 n sessions Figure 3 Pre post treatment effect sizes (95% CI) by number of sessions against national datasets. Mental health services in primary care are increasingly required to evaluate their performance in response to the national outcomes agenda. Benchmarking routine evaluation of patient outcomes and audit of patient throughput against a referential database of other UK services will enable services to monitor their performance in a national context and provide a rapid evaluation of new and innovative service developments. Outcomes of ultrabrief interventions The findings of this evaluation suggest that significant improvements in patients mental health can be made within one to three sessions of psychological intervention. These findings have subsequently been replicated using a larger sample of patients. 35 Patients attending the PCMH service had outcomes that were comparable with national counselling services, while on average attending fewer sessions. Future research is needed to further investigate the impact of ultrabrief treatments in primary care. ACKNOWLEDGEMENTS We are grateful to the former Leeds Health Authority for funding. In particular, we thank the primary care mental health workers: Najma Allybocus, Sue Andrews, Julie Atkinson, Maria Barker, Bernie Bell, Nikki Bray, Claire Bulman, Lesley Butlin, Shahin Butt, Jen Chandler, Beth Chappell, Sally Downie, LisaHollingworth,BeeHoward,JeanHowarth,Heather Johnston, Naomi Kirkby, Chris Mason, Sue McAndrew, Kirsty McArthur, Germaine McClusky, Cath McCoy, Wendy Murray, Kevin Nicholas, Philippa Ostler, Ciara Payne, Sarah West, Jane Wood, Nick Wood, and Dawn Woollin. In addition, we thank the administrative staff from the five Leeds primary care trusts. We are grateful to all users of the PCMH service for their contribution to the service evaluation. Additional support was also provided by the R&D Priorities and Needs Levy to Leeds Community Mental Health Trust. REFERENCES 1 Bird L. The Fundamental Facts: all the latest facts and figures on mental health. London: Mental Health Foundation, 1999. 2 Goldberg D. The management of anxious depression in primary care. Journal of Clinical Psychiatry 1999;60:43 4. 3 Department of Health. National Service Framework for Mental Health: modern standards and service models. London: Stationery Office, 1999. 4 Department of Health. The NHS Plan. London: Stationery Office, 2000. 5 Lee J, Gask L, Roland M et al. Primary care led commissioning of mental health services: Lessons from total purchasing. Journal of Mental Health 2002;11:431 9. 6 Department of Health. Organising and Delivering Psychological Therapies. London: HMSO, 2004. 7 National Institute for Clinical Excellence. Depression: management of depression in primary and secondary care. Clinical Guideline no 23. London: NICE, 2004. www.nice.org.uk/cg023niceguideline (accessed 5 January 2006). 8 Harvey I, Nelson SJ, Lyons RA et al. A randomized controlled trial and economic evaluation of counselling in primary care. British Journal of General Practice 1998;48:1043 8. 9 Hudson-Allez G. Time-limited Therapy in a General Practice Setting. London: Sage, 1997. 10 Bower P. Primary care mental health workers: models of working and evidence of effectiveness. British Journal of General Practice 2002;52:926 33. 11 McEvoy P, Richards D and Owen J. Selective access: prioritising referrals at the primary care/cmht interface. Clinical Effectiveness in Nursing 2000;4:163 72. 12 Gask L, Sibbald B and Creed F. Evaluating models of working at the interface between mental health
A benchmarking study using the CORE system 11 services and primary care. British Journal of Psychiatry 1997;170:6 11. 13 Paxton R, Shrubb S, Griffiths H et al. Tiered approach: matching mental health services to needs. Journal of Mental Health 2000;9:137 44. 14 Fletcher J, Bower P, Richards D et al. Enhanced Service Specification for Depression under the New GP Contract: a commissioning guidebook. National Institute for Mental Health England, North West Development Centre, 2004. 15 Barkham M, Rees A, Shapiro DA et al. Outcomes of time-limited psychotherapy in applied settings: replicating the Second Sheffield Psychotherapy Project. Journal of Consulting and Clinical Psychology 1996;64:1079 85. 16 Barkham M and Mellor-Clark J. Bridging evidencebased practice and practice-based evidence: Developing a rigorous and relevant knowledge for the psychological therapies. Clinical Psychology and Psychotherapy 2003;10:319 27. 17 National Institute for Mental Health in England (NIMHE). Outcome Measures Implementation: best practice guidance. London: NHS-NIMHE, 2005. 18 Jacobson NS and Truax P. Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology 1991;59:12 19. 19 Barkham M, Margison F, Leach C et al. Service profiling and outcomes benchmarking using the CORE-OM: toward practice-based evidence in the psychological therapies. Journal of Consulting and Clinical Psychology 2001;69:184 96. 20 Evans C, Connell J, Barkham M et al. Towards a standardised brief outcome measure: psychometric properties and utility of the CORE-OM. British Journal of Psychiatry 2002;180:51 60. 21 Barkham M, Gilbert N, Connell J et al. The suitability and utility of the CORE-OM and CORE-A for assessing severity of presenting problems in primary and secondary care based psychological therapy services. British Journal of Psychiatry 2005;186: 239 46. 22 Leach C, Lucock M, Barkham M et al. Transforming between Beck Depression Inventory and CORE-OM scores in routine clinical practice. British Journal of Clinical Psychology 2006 (in press). 23 Mellor-Clark J, Barkham M, Connell J et al. Practicebased evidence and standardized evaluation: informing the design of the CORE system. European Journal of Psychotherapy, Counselling and Health 1999;2:357 74. 24 Mellor-Clark J and Barkham M. Quality evaluation: methods, measures and meaning. In: Feltham C and Horton I (eds). Handbook of Counselling and Psychotherapy (pp. 255 270). London: Sage Publications, 2000, pp. 255 70. 25 Wilson EB. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 1927;22:209 12. 26 Cohen J. Statistical Power Analysis for the Behavioural Sciences (2e). New Jersey: Lawrence Erlbaum, 1988. 27 Hedges LV and Olkin I. Statistical Methods for Metaanalysis. New York: Academic Press, 1985. 28 Evans C, Connell J, Barkham M et al. Practice-based evidence: benchmarking NHS primary care counselling services at national and local levels. Clinical Psychology and Psychotherapy 2003;10:374 88. 29 Shadish WR, Matt GE, Navarro AM et al. Evidence that therapy works in clinically representative conditions. Journal of Consulting and Clinical Psychology 1997;65:335 65. 30 Hamilton R, Gordon P and Naji S. Service innovation: The first year of a brief psychiatric screening clinic in primary care. Psychiatric Bulletin 2002;26:218 21. 31 Burton MV, Sadgrove J and Selwyn E. Do counsellors in general practice surgeries and clinical psychologists in the National Health Service see the same patients? Journal of the Royal Society of Medicine 1995;88:97 102. 32 Cape J and Parham A. Rated casemix of general practitioner referrals to practice counsellors and clinical psychologists: A retrospective survey of a year s caseload. British Journal of Medical Psychology 2001;74:237 46. 33 Barkham M, Connell J, Stiles WB et al. Dose effect relations and responsive regulation of treatment duration: The good enough level. Journal of Consulting and Clinical Psychology 2006 (in press). 34 Barkham M, Shapiro DA, Hardy GE et al. Psychotherapy in two-plus-one sessions: Outcomes of a randomized controlled trial of cognitive-behavioural and psychodynamic-interpersonal therapy for subsyndromal depression. Journal of Consulting and Clinical Psychology 1999;67:201 11. 35 Simon GE, Katon WJ, VonKorff M et al. Cost-effectiveness of a collaborative care program for primary care patients with persistent depression. American Journal of Psychiatry 2001;158:1638 44. CONFLICTS OF INTEREST None. ADDRESS FOR CORRESPONDENCE Professor Michael Barkham, Psychological Therapies Research Centre, University of Leeds, 17 Blenheim Terrace, Leeds LS2 9JT, UK. Tel: +44 (0)113 343 5699; email: m.barkham@leeds.ac.uk Received 28 September 2005 Accepted????????