An Evaluation of the Francophone Telemedicine Mental Health Service

Transcription

1 February, 2013 An Evaluation of the Francophone Telemedicine Mental Health Service Prepared for 147 Delhi St. Guelph, ON N1E 4J3 T: (519) ext By The Centre for Community Based Research

2 Table of Contents 1. Introduction Background The Current Evaluation Methodology... 5 a. Data gathering methods... 5 b. Participants... 5 c. Data analysis Findings... 6 a. Increased accessibility... 7 b. Value and relevance... 8 c. Appropriateness of service... 9 d. Other accomplishments of the program e. Challenges Conclusions Recommendations Appendix...14 a. Process evaluation report... 14

3 1. Introduction This is the outcome evaluation report for the Francophone Telemedicine Mental Health Service. The purpose of this evaluation was to assess the project s impact in terms of accessibility, relevance, value, and appropriateness of the service to Francophone residents of Waterloo-Wellington. This evaluation follows a process evaluation of the Francophone Telemedicine Mental Health Service, which was conducted in 2011 and covered the process of project development and implementation (Appendix A). 2. Background This initiative was proposed by the Waterloo-Wellington Local Health Integration Network (WWLHIN) to the Société Santé en français. The proposal was developed as a result of a need that was identified in the Francophone community of Waterloo-Wellington: According to the proposal, When seeking assessment and consultation in French, our residents must travel to assessment and consultation centres such as Hamilton (WWLHIN).The purpose of the proposed service was to fill this gap by linking Francophone mental health service users in Waterloo-Wellington to psychiatrists using technology. In addition, filling this gap falls in line with one of the WWLHIN s priorities, which is to improve access to mental health and addiction services, as identified in their Integrated Health Services Plan Telemedicine communication technology was proposed as the service delivery method and telemedicine suites would be chosen strategically throughout the WWLHIN. The Ontario Telemedicine Network (OTN) was named as a partner who would support in the implementation of the service. Hiring a bilingual coordinator who would be responsible for the implementation of the service was also part of the plan. When funding for this project was granted, Trellis Mental Health and Developmental Services was nominated to manage the implementation of the service. Subsequently, a bilingual coordinator was hired who also had a background in healthcare as a trained nurse. Being a new service, the role of the coordinator naturally expanded to include such project development activities as partnership building, promotion and leading the development of policies and procedures for the service. The main goal of this project is to increase access to quality French language mental health services in Waterloo-Wellington by providing telepsychiatry services in French. To this end, the project aims to eliminate travel costs and time required for clients to travel out of town to consult French-speaking psychiatrist by linking them with French-speaking psychiatrists via telemedicine. The Francophone Telemedicine Mental Health Service has two main components: project development and project implementation. Project development includes activities undertaken to develop the different aspects and components of the service, and lay the foundation for its implementation. These activities include partnership building, the establishment of an advisory group to guide the development and implementation of the service, development of policies and procedures, designing the intake and referral processes, and finding and contracting French-speaking psychiatrists to provide assessment and consultation to 3

4 clients. Project development activities also included evaluation planning to support the implementation of the service and assess its impact. Project implementation has two main sub-divisions: promotion and service delivery. Promotion involved, first, promoting the service to service providers in order to gain their buy-in so that they refer their patients, and second, promoting the service to the Francophone community to create awareness of the service. Through these promotion activities, the project aims to engage both service providers and Francophone community members and create a better understanding of Francophone mental health service needs. Service delivery activities include engaging referral sources, receiving referrals, and intake of clients leading to assessment and consultation by a psychiatrist via telemedicine. Assessment and consultation is followed by the development of a treatment plan for the client. Figure 1 is a logic model of the Francophone Telemedicine Mental Health Service showing its main components, activities and expected short-term, intermediate and long-term outcomes. Figure 1: Francophone Telemedicine Mental Health Service in Waterloo-Wellington 3. The Current Evaluation The purpose of this outcome evaluation was to assess the project s impact in terms of accessibility, relevance, value, and appropriateness of the service to Francophone residents of Waterloo- Wellington. At the time of this evaluation, 49 clients had had telepsychiatric consultations through the Francophone Telemedicine Mental Health Service. These clients included 33 adults and 16 children and youth. All of the children and youth were bilingual, but identified having conducted their consultations in English. Upon further investigation, it was clarified that, often both languages were used during the consultation and that the primary language used in the consultation many times 4

5 depended on the language abilities of the adults that accompanied the child or youth. Four of the 33 adults had their consultations in French and 29 in English. 4. Methodology A participatory, community based approach was adopted for the outcome evaluation. This approach, which combines research with action involved working closely with a multi-stakeholder Advisory Committee that provided guidance at each step of the evaluation. a. Data gathering methods Two data gathering methods were used in this evaluation, namely surveys and individual interviews. Evaluation form/client survey: An evaluation form was given to telemedicine clients following their consultations. For clients who did not complete the evaluation form, the form was converted into a web-based survey, which was administered by phone. The survey was administered in French and English by Trellis staff and volunteers. Individual interviews with clients: Qualitative interviews with clients of the telemedicine mental health service were conducted via telephone by the CCBR evaluation team. Survey for service providers: Two surveys were developed; one for referring physicians and one for psychiatrists. The survey for physicians was faxed to a number of doctors who had either referred patients to the telemedicine mental health service, or had agreed in principle to refer patients. The version of the survey for psychiatrists was created online. The survey link was ed to three psychiatrists who had conducted consultations through the telemedicine mental health service. b. Participants Altogether, 26 people participated in the evaluation. They included 22 clients of the telemedicine mental health service and four physicians. Clients: Twenty-two clients completed the evaluation form/client survey. They included three individuals who completed the evaluation form/survey in French, and 19 who completed it in English. Parents completed the survey on behalf of clients who were under 18 years of age. The majority of client survey respondents resided in Guelph (68.2%; n=15) with one each of the remaining clients coming from Cambridge, Clifford, Kitchener, Mount Forest, Orangeville, and Waterloo 1. They predominantly identified their ethnicity as white (n=21); one person did not indicate their ethnicity. The client survey respondents were 40.9% male and 59.1% female. Their age range was 12 years (or younger) to 55 years (or older). The distribution of client survey respondents by age is presented in Table 1. 1 One respondent did not indicate their city of residence. 5

6 Table 1: Distribution of client survey respondents by age Age range Frequency Percentage 12 years or younger years years years years years Older than 55 years Total * * rounded up As shown in Table 1, the majority of client survey respondents (59.1%) were aged 36 years or older. Also, the majority of them (90.9%; n=20) were first time users of telemedicine: only 2 individuals indicated that they had previous experience with telemedicine. Individual interviews: In addition to the client surveys, three Francophone clients participated in qualitative interviews. Survey for service providers: Four physicians returned completed surveys to the evaluators. They included two who had referred clients to the service and two who had not. Two of the four respondents were located in Guelph, one in Kitchener, and one in Mississauga. They all had over 11 years of experience as physicians. One of them works at a hospital and three work in doctors offices. Two of them heard about the telemedicine mental health service from patients, and one from Trellis. The fourth did not indicate how they heard about the service. None of the three psychiatrists who received the online survey link completed the survey. c. Data analysis The client evaluation forms and survey data was entered into SPSS and analyzed. The analysis involved calculating frequencies of responses. Qualitative data from open ended survey questions and key informant interviews with Francophone clients were analyzed using content analysis. Themes from the content analysis were used to support or illustrate the findings from the survey data. 5. Findings The findings of this evaluation indicated that the Francophone Telemedicine Mental Health Service is addressing an important gap in psychiatric services in Waterloo-Wellington. Although it was initiated primarily to serve the Francophone community, the telemedicine service is increasing access to psychiatric services, and providing quality, valued and appropriate services for people in both official languages. The findings of the evaluation are presented in detail below. 6

7 a. Increased accessibility The evaluation findings demonstrated that the Francophone Telemedicine Mental Health Service has increased access to psychiatric services for residents of Waterloo-Wellington. Increasing number of people using the service: Although it was set up mainly for Francophone service users who prefer to have psychiatric consultations in French, the service is open for consultation in both official languages. The number of clients has increased from four adults at the time of the process evaluation, to 49 adults, children and youth at the time of this evaluation. All clients who participated in the evaluation (n=22) indicated that the service was well organized. The majority of them (63.7%) said that they were given sufficient notice about their scheduled consultations. Also, the majority (68.2%) said their appointments were scheduled on time. The participants responses are summarized in Table 2. Table 2: Timely access to psychiatric consultation Item I received enough notice about when my telemedicine visit was scheduled My telemedicine appointment was scheduled quickly N=22 Strongly disagree Disagree Neither Agree Strongly agree 0% 0% 0% 36.4% 63.6% 9.1% 13.6% 9.1% 22.7% 45.5% Participants in the qualitative interviews echoed these sentiments: One individual interview participant said that the service was excellent compared to her previous experiences. According to her, she liked the setting and also felt secure. Others expressed great satisfaction with the set up and the process. Professionally done! Done in a nice room! I was satisfied. I felt the system was secure. (Client) It was a really nice conference room. Nice table, big screen TV. Technology all worked properly. (Client) The administrative process was great. They are very supportive and take care of you. (Client) Timely access to services: The Francophone Telemedicine Mental Health Service has made it possible for community members to access psychiatric services in a timely manner. More than one-fifth of respondents (22.47%) reported having their consultation within one week of referral, which is well within the Canadian Psychiatric Association s 2 recommended maximum wait time of four weeks for a psychiatric consultation. The majority of the clients (63.7%), however, reported waiting between one 2 Canadian Psychiatric Association (2006). Wait Time Benchmarks for Patients with Serious Psychiatric Illnesses: A series of recommendations made by Canada s psychiatrists. 7

8 and six months for their consultation. The remaining 13.6% said they experienced wait times of more than six months 3. They, however, provided no reasons for the delays. One client chose to access this service because it was a six month to one year wait to see her own psychiatrist in person. Through this service she only had to wait one to two months. One interview participant praised the staff person for ensuring that she had the appointment quickly. The two physicians who had referred clients to the service also expressed that the telemedicine mental health service has increased access to care for patients. According to them, their patients had their consultations on time through telemedicine. Reduced out-of-town travel: Besides reducing wait times, the service has reduced the need for people to travel out of town for psychiatric consultations. Both clients and physicians said the location of such a service within the Waterloo-Wellington region was very convenient. Clients who participated in the individual interviews expressed that the location made the service easily accessible to them. It removes the barrier of travel. (Client) Overall it was good. It is better than driving to London. (Client) Eliminated cost associated with out-of-town travel: The Francophone Telemedicine Mental Health Service has increased accessibility by eliminating the costs associated with traveling out of town for psychiatric consultation. As one client explained, the cost of traveling out of town is not only limited to mileage or transportation fare, it also includes loss of school or work hours for those who are accompanied by family members. In her words, My husband doesn't need to take time off work, and my son doesn't need to miss school. (Client) b. Value and relevance The evaluation findings indicated that the Francophone Telemedicine Mental Health Service is a valuable and relevant service for psychiatric service users in Waterloo-Wellington. The majority of clients who had used the Francophone Telemedicine Mental Health Service (72.8%) had a positive appraisal of it. According to them, using telemedicine is better than traveling out of town or not seeing a psychiatrist at all, and that they would use it again if they needed another consultation. Some of them appraised the service using the following words: This is a good [service], and I certainly prefer it to not seeing my doctor at all. (Client) 3 It appeared that some clients considered their wait times for telemedicine as a continuation of wait times experienced prior to being recommended for telemedicine. This is further discussed under Challenges. 8

9 I thought it was good. I d rather see a specialist "quickly" via telemedicine than waiting too long before getting an appointment. (Client) The two physicians who had referred patients to the service said the service was valuable as it has improved continuity of service for their patients. Combined, both of them had referred 11 patients to the service and had received reports for all of them. Three Francophone clients who had used the service described it as very valuable to their community: All three of them indicated that it was important for them to receive services in French rather than in English. It s better for Francophone to receive services in French. (Client) The Francophone clients also said that they felt more comfortable receiving services in French than they would have felt receiving it in English. One interview participant said it made a huge difference to be able to speak in French and that health services in French are indispensable. Communicating in one s mother tongue allows for better expression of one s issues as well as better understanding of what the psychiatrist is saying. It is difficult to express yourself [in a second language], especially about mental health and psychological problems. (Client) The language made a huge difference. Really, I felt better understood by the psychiatrist. (Client) It s better to be able to express yourself in your first language, especially for health issues. It s a lot more comfortable; you can express yourself more in depth. (Client) Expressing their satisfaction with the service, all the Francophone survey respondents said they would use the telemedicine service again if they needed another consultation. They also said they would recommend the telemedicine service to a family member or friend. One interview participant said that she would say, Don t hesitate! Go for it! to others considering telemedicine. In her opinion, telemedicine services in French are preferable to in-person consultations in English. c. Appropriateness of service Most clients who responded to the survey described the service as appropriate. All of them said their privacy was respected. One client described the efforts taken to ensure confidentiality. She explained that the consultation took place in a closed room with only the psychiatrist over video, and the support person in the room. 4 Furthermore, the majority of clients who responded to the client survey said they had a positive experience with the service; 66.7% of them reported a high level of overall satisfaction. The clients 4 Note: The support person stayed in the room to provide technical assistance, if the client gave consent. 9

10 responses to a series of questions about the process and appropriateness of the service are summarized in Table 3. Table 3: Participants' self-reports on the appropriateness of the telemedicine mental health service Item Strongly disagree 10 Disagree Neither Agree Strongly agree Overall, I was satisfied with my telemedicine visit 14.3% 9.5% 9.5% 14.3% 52.4% There was enough time to deal with everything that needed to be covered 9.1% 18.2% 9.1% 36.4% 27.3% I felt I was understood by the healthcare provider during the 4.5% 18.2% 9.1% 45.5% 22.7% telemedicine visit The next steps in my care were explained to me 9.1% 13.6% 9.1% 36.4% 31.8% I feel that my privacy was respected 0% 0% 0% 45.5% 54.5% I will use telemedicine again if I need another visit with a healthcare provider 13.6% 9.1% 4.5% 36.4% 36.4% Overall, I am satisfied with the arrangement of this service/appointment N=22 4.5% 9.1% 9.1% 36.4% 40.9% As shown in Table 3, the majority of the clients (77.5%) expressed satisfaction with the arrangements of the service; the majority (63.7%) also reported satisfaction with the amount of time they had with the psychiatrists; and the majority (68.2%) said they felt that the consultants understood them; and also that the next steps in their care were clearly explained to them. In the words of one client, It was as if [the psychiatrist] was in the room with us. (Client) d. Other accomplishments of the program Partnership building was a key component of the Francophone Telemedicine Mental Health Service. The main partners that were engaged in this project were Trellis Mental Health and Developmental Services, the Waterloo-Wellington Local Health Integration Network (WWLHIN), the Ontario Telemedicine Network (OTN), the Francophone associations of Waterloo and Wellington (AFKW and Centre Communautaire Francophone of Cambridge) as well as the Catholic Mission St-René Goupil of Guelph. The role of the WWLHIN was to provide support in the implementation, promotion and integration of the project. OTN provided technical and organizational support to implement this new service. Lastly, the Francophone associations facilitated the communication with the local francophone community. In addition, Trellis built formal and informal partnerships with local French school boards. Trellis has a formal partnership with Le Conseil scolaire Viamonde. This partnership facilitates the referral process

11 for students needing psychiatric consultation. Trellis also has an informal agreement with Conseil Scolaire de District Catholique Centre Sud (CSDCCS). Currently, Trellis is working with the CSDCCS on developing a protocol. To date, 16 students from both school boards have accessed the service. As reported in the process evaluation, the partners worked well together in providing support to project staff. e. Challenges Low numbers of Francophone clients: Low numbers of Francophone clients using the telemedicine mental health service remains a challenge for the service. As reported earlier, only four out of the 22 consultations (18%) so far were conducted in French. This percentage is, however, much higher than the percentage of the Francophone population of Waterloo-Wellington which is about 1.2%. Also, the ongoing publicity and partnership building in relation to the service is yielding good results. For example, 16 bilingual children and youth have accessed telepsychiatry consultations through the service. The impersonal nature of the service: The impersonal nature of telemedicine is recognized as one of its main disadvantages (Hjelm, 2005) 5. In this evaluation, a slight majority of client survey respondents (59.2%) noted that consultation by telemedicine does not feel personal. Some of them also said they did not feel very comfortable during the consultation. According to those clients, they would rather see a healthcare provider in-person rather than by telemedicine. Some of those who also participated in individual interviews said: I had only one telemedicine consultation... but if it is possible I would prefer to see my doctor face-to-face. (Client) Not so personal, I didn't feel so comfortable talking. I would prefer to see a doctor in person. (Client) It is, however, important to note that all those who shared this opinion were first time users of telemedicine. Also, while these individuals would prefer in-person visits for healthcare, most of them said telemedicine was an acceptable alternative. Perceived long wait times: As reported earlier in this report, 13.6% of respondents reported wait times of more than six months which was unexpected. Further investigation of the issue made it apparent that clients were referred to the telemedicine because they were experiencing long wait times to see a psychiatrist. It appears some of those individuals had added up their wait times before and after referral to telemedicine. Technical difficulties: A few respondents reported slight technical difficulties during the consultations. For example, one client said, [There was a] problem with volume on the side of the psychiatrist. Occasionally, I couldn't understand her very well. (Client) 5 Hjelm, N. M. (2005). Benefits and draw backs of telemedicine. J Telemed Telecare, 11(2),

12 Technical difficulties are not unexpected in telemedicine, and OTN has on-site support to handle such issues. 6. Conclusions The Francophone Telemedicine Mental Health Service is increasing access to psychiatric services in Waterloo-Wellington. According to clients, this service has reduced wait times to see a psychiatrist, the need to travel out-of-town for psychiatric services, and travel costs associated with out-of-town travel. Overall, clients were satisfied with the service. They reported satisfaction with the shorter wait times, the accessible locations, the service arrangements, and the amount of time spent with the psychiatrist. They also reported that the service was well organized and professional. Most clients had a positive appraisal of the service. The majority of them reported a good experience of the service, and felt that their privacy was respected during the process. Most clients also reported a high comfort level in using this service, and said they would use it again. Even those who said they would prefer to see a psychiatrist in-person still said telemedicine was an acceptable alternative to the long wait times. Clients also noted that this service was of value and relevance to them, especially Francophone clients. Francophone clients said it was important for them to receive services in French because they are better able to express themselves and feel understood by the consulting professional. 7. Recommendations Based on the findings of this evaluation we recommend the following: a) Continue the Francophone Telemedicine Mental Health Service The service is fulfilling an important mental health service need in the Waterloo-Wellington. While its main purpose of increasing access to French language psychiatric services is yet to be fully measurable, the service has generally increased access to psychiatric services. It has been a faster alternative for psychiatric consultation to both English and French-speaking clients who, otherwise, would have had to wait a long time to see a psychiatrist in-person. The ongoing publicity and partnership building in relation to the service is leading to increased utilization. Overall, the findings of the evaluation make it apparent that the efforts and resources so far invested into this service are beginning to yield results. b) Continue to implement the recommendations from the process evaluation: Continue with promotion and partnership building: Ongoing promotion and partnership building will increase awareness of the service among both referring physicians and clients, and with time, increase utilization by both Francophone and Anglophone clients. 12

13 Continue improving processes: It is important to continue reviewing and revising the scheduling and timing of appointments as well as the service process as it continues to grow. This will help to improve the efficiency and effectiveness of the service. Seek more funding and sustainability options: The project is fulfilling a need in the community and should therefore be continued beyond its developmental stages. For this reason, it is important to identify sustainable financing options for it. 13

14 8. Appendix a. Process evaluation report An Evaluation of the Francophone Telemedicine Mental Health Service Introduction This is a process evaluation report for the Francophone Telemedicine Mental Health Service. The purpose of this evaluation was to assess the implementation of the telemedicine initiative, and document its effectiveness as well as identify key facilitating and inhibiting factors affecting it. This initiative was proposed by the Waterloo-Wellington Local Health Integration Network (WWLHIN) to the Société Santé en français. The proposal was developed as the result of a need that was identified in the Francophone community of Waterloo-Wellington: According to the proposal, When seeking assessment and consultation in French, our residents must travel to assessment and consultation centres such as Hamilton (WWLHIN).The purpose of the proposed service was to fill this gap by linking francophone mental health service users in Waterloo-Wellington to psychiatrists using technology. In addition, filling this gap falls in line with one of the WWLHIN s priorities, which is to improve access to mental health and addiction services, as identified in their Integrated Health Services Plan Telemedicine communication technology was proposed as the service delivery method and telemedicine suites would be chosen strategically throughout the WWLHIN. The Ontario Telemedicine Network (OTN) was named as a partner who would support in the implementation of the service. Hiring a bilingual coordinator who would be responsible for the implementation of the service was also part of the plan. When funding for this project was granted, Trellis Mental Health and Developmental Services was nominated to manage the implementation of the service. Subsequently, a bilingual coordinator was hired who also had a background in healthcare as a trained nurse. Being a new service, the role of the coordinator naturally expanded to include such project development activities as partnership building, promotion and leading the development of policies and procedures for the service. The main goal of this project is to increase access to quality French language mental health services in Waterloo-Wellington by providing telepsychiatry services in French. To this end, the project aims to eliminate travel costs and time required for clients to travel out of town to consult French-speaking psychiatrist by linking them with French-speaking psychiatrists via telemedicine. The Francophone Telemedicine Mental Health Service has two main components: project development and project implementation. Project development includes activities undertaken to develop the different aspects and components of the service, and lay the foundation for its implementation. These activities include partnership building, the establishment of an advisory group to guide the development and implementation of the service, development of policies and procedures, designing the intake and referral processes, and finding and contracting French-speaking psychiatrists to provide assessment and consultation to 14

15 clients. Project development activities also included evaluation planning to support the implementation of the service and assess its impact. Project implementation has two main sub-divisions: promotion and service delivery. Promotion involved, first, promoting the service to service providers in order to gain their buy-in so that they refer their patients, and second, promoting the service to the francophone community to create awareness of the service. Through these promotion activities, the project aims to engage both service providers and Francophone community members and create a better understanding of Francophone mental health service needs. Service delivery activities include engaging referral sources, receiving referrals, and intake of clients leading to assessment and consultation by a psychiatrist via telemedicine. Assessment and consultation is followed by the development of a treatment plan for the client. Figure 1 is a logic model of the Francophone Telemedicine Mental Health Service showing its main components, activities and expected short-term, intermediate and long-term outcomes. Figure 1: Francophone Telemedicine Mental Health Service in Waterloo-Wellington Currently, the project is nearing the end of its first year with many of its planned activities executed. This evaluation covers the process of project development and project implementation. Methodology A participatory, community based approach was adopted for the process evaluation. This approach, which combines research with action involved working closely with a multi stakeholder Advisory Committee that provided guidance at each step of the evaluation. A logic model development forum and key informant interviews were used for qualitative data gathering. 15

16 Stakeholder Involvement Members of the Advisory Committee have met twice over the first phase of this evaluation. The first meeting was done with other stakeholders in a project logic model development session to review the activities and outcomes of the Francophone Telemedicine Mental Health Service. The second meeting was an Advisory Committee meeting for project updates, a review of data gathering tools, and to set a direction for the process evaluation. Besides those two meetings, there was ongoing feedback and communication primarily between the research team and the project coordinator and also with Advisory Committee members regarding the project. Data Gathering Data gathering at this phase of the project involved document review, logic model development forum, and key informant interviews. Document review The document review component of the process evaluation included review of program related documents. This was a first step for the evaluation team to orient itself to the program s history and processes to fully understand its activities and intended outcomes. This component was also necessary for an alignment of data collection strategies and tools with the appropriate elements of the project. Program logic model Document review was followed by a program logic model development session. A program logic model is a flowchart that depicts a program s main activities/processes and links them to its intended short-term, intermediate and long term outcomes. The session was a two-hour meeting with members of the Advisory Committee, staff and other stakeholders. Participants described and discussed the Francophone Telemedicine Mental Health Service project s main activities and intended outcomes. This included the strategies for development and promotion of the project as well as its implementation steps. Key informant interviews Key informant interviews were conducted with program stakeholders. The purpose of these in-depth qualitative interviews was to gather information about the program activities, processes and partnerships from individuals who could provide in-depth information about the program. Data Analysis Information gathered through document review was analyzed to develop a basic draft logic model for the Francophone Telemedicine Mental Health Service. This draft model served as the basis for discussion in the logic model development session. Discussion during this session helped to clarify all aspects of the program. Content analysis was performed on the qualitative data gathered through the key informant interviews. Through this process, the research team develop themes for answering the main evaluation questions. 16

17 Participants Eight people participated in the logic model development session, and 6 6 people participated in key informant interviews. In selecting participants in this phase of the evaluation, a list of potential key informants was developed in consultation with the Trellis team. This list included individuals who had a range of involvement with the program and could speak about it from the various stakeholder perspectives. Among them were, program staff, representatives of the local Francophone community, the LHIN, OTN, and practitioners. The interviews were done by telephone by the CCBR evaluation team. Six individuals were selected from the list and interviewed. The findings of the process evaluation are presented in the following sections. Findings The key informants described and assessed the implementation of the Francophone Telemedicine Mental Health Service. Overall, they were pleased with the project implementation thus far. They expressed satisfaction in being involved in the project, and recognized the efforts that have made this project what it is today. They believe that continuing with the strong coordination and outreach that has taken place so far will bring continued success to the project. Project Development Project development and coordination involved the undertaking of many different activities in order to implement the project. These activities included the following: Building a partnership Establishing an advisory group Developing policies and procedures Designing the intake and referral processes Confirming access to telemedicine sites Communication planning Finding and contracting French-speaking psychiatrists Evaluation planning Building a Partnership Building the partnership was a key component during the project development stage. The main partners that were engaged in this project were Trellis Mental Health and Developmental Services, the Waterloo-Wellington Local Health Integration Network (LHIN), the Ontario Telemedicine Network 6 Four of them also participated in the logic model development session 17

18 (OTN) and the Francophone communities of Waterloo and Wellington. The partnership is seen as fundamental in the development of this project. Key informants believed that the partners have worked well together and provided a lot of support in developing this new service. Precisely, it s working through the partnership, rather than working on our own, that enabled us to gain knowledge that, otherwise, would not have been possible to gain individually (Key Informant) The WWLHIN is the initiating partner of this project. In this role, the WWLHIN has provided support and resources for this initiative, including translation services and additional financial resources. For instance, the WWLHIN provided additional funding to increase the number of days of work for the coordinator from three to five from March to June The LHIN found some money somewhere and transferred it to this program so [the coordinator] could be fulltime until June, and now [the coordinator] is working four days a week. We decided that was the right amount of days. They provided support with the coordination of the project, and also with money. Also they helped with communication; a lot of documents [were] translated through the LHIN and it s so expensive, so that was also really helpful and supportive. (Key Informant) The LHIN has been very supportive, they ve been very open, they have been flexible and provided practical support and even financial support outside of the grant that was required (Key Informant) Trellis Mental Health and Developmental Services was the lead organization for implementing the project. As the implementers of this project, Trellis has played a direct role in the project s development, promotion and delivery. Trellis management has shown commitment to this project and has provided a great amount of support to the coordinator in executing the various aspects of this initiative. In particular, the executive director, the communications coordinator, and the risk, privacy and health information service manager of Trellis worked closely with the coordinator of the Francophone Telemedicine Mental Health Service providing her with various forms of support, including attending meetings with her, developing processes, facilitating communication, and problem-solving. Vital support has also come from OTN for the execution of this project. OTN provides support in various ways, including training staff with the communication equipment to providing input into to potential French speaking psychiatrists that could be contracted for this initiative. I would say they were really into it in the beginning, and they were really helpful and part of the process; going from getting the material to training and also helping us to try to find some psychiatrists, through their network (Key Informant) Partnerships within the community were also developed. These include the Kitchener-Waterloo Francophone Association (AFKW) and Centre Communautaire Francophone de Cambridge (CCFC). Currently, a partnership with the two local French school boards, Conseil scolaire Viamonde and Conseil scolaire de district catholique Centre-Sud, is in the development stages as the project expands 18

19 to include French telepsychiatry services for youth. The school boards have expressed their pleasure with the availability of these services for youth and are facilitating the access to Trellis services. [This] partnership is coming along. We re still developing that relationship and it s coming along; we re at that stage where we re trying to put some protocols in place. Advisory Committee The advisory committee is composed of five different community stakeholders. The committee that was established is seen as an effective group which brings much knowledge about healthcare issues as well as connections to the Francophone community of Waterloo-Wellington. The committee is comprised of both individuals who are Francophones and are linked to the healthcare system. The advisory committee is made up of different sections of the community, a good representation, both, in terms of geography and good understanding of the community s health service needs (Key Informant) Our role was to help [the coordinator] navigate the system regarding our knowledge of the health system and the actors in the health and how the health system functions (Key Informant) Key informants indicated that this committee has worked well together in developing and implementing this project, especially in providing information and support to the coordinator. In particular, members of the advisory committee were helpful in linking the coordinator with key individuals who needed to be contacted or engaged in this project. They are usually involved in different groups within the community, so that s a great group for [the coordinator because they have] knowledge about healthcare systems and the francophone component of it. [...] They give [the coordinator] feedback about those different groups [...] One member is [...] working with [the coordinator] to set up a meeting with the seniors [...] I think they are really useful in the sense that they are really community based (Key Informant) I think all those involved with the committee were opened to share their feedback and opinions, and did so in a constructive manner. I think the co-ordinator was excellent in organizing the process for the advisory board committee, facilitation and treatment of all participants equally (Key Informant) Developing Policies, Procedures and Processes A key part of the development phase was to develop the policies, procedures and processes that would guide the French Telemedicine Mental Health service. This involved drawing upon the resources available within Trellis as well as working together with the advisory committee to design the policies and procedures of this new service. The coordinator spent time meeting with staff of various departments within Trellis to understand the procedures and processes of other programs and inform the development of appropriate ones for this service. This learning process also helped in designing the intake and referral processes for the French Telemedicine Mental Health service. 19

20 Project Coordination Overall, interview participants were very positive in their comments regarding the project s coordination. In particular, they expressed that the coordinator has been very effective in her role of coordinating all aspects of the service, promotion and delivery. They also believe she has meaningfully engaged stakeholders and has made strong connections in relations to the service. I think that the coordinator has done an enormous amount of very good work. Actually, I have shared the processes with my colleagues; we are wondering whether we could probably use similar processes for other on-going programs. I think the work was well done (Key Informant) [The coordinator] has done an amazing job; she excels in terms of community engagement and has really made strong connections with the francophone community (Key Informant) [The coordinator] has worked effectively with stakeholders and that they ve felt engaged and included (Key Informant) Finding and contracting French-speaking psychiatrists Finding and contracting French-speaking psychiatrists was a main activity of the project development stage. Currently, three psychiatrists have been contracted to provide psychiatric consultations in French via telemedicine, and more are interested in becoming involved. Finding French-speaking psychiatrists involved making many phone calls and sending of letters across Ontario as well as researching online for any leads to French-speaking psychiatrists. This also involved working within the networks of the partner organizations as well as individual contacts in order to find Frenchspeaking psychiatrists that could be contacted. Contracting French-speaking psychiatrists was a vital component in ensuring that the service could be delivered to clients. Interview participants identified this component as the most challenging 7. Project Implementation Promotion Promotion was a key component in implementing this project. This included extensively promoting the service within Waterloo-Wellington to both service providers and Francophone community members. Promotion to service providers included promoting the service to both psychiatrists (to be contracted as psychiatrists for this service) and family physicians and other health professionals (to refer their patients to this service). Promotion to the Francophone population of Waterloo-Wellington consisted of promoting to community members via various outlets in order to create awareness of this service being offered. This component of the project was not able to be executed within the planned timelines due to the delay faced in finding and contracting French-speaking psychiatrists. Promotion to service providers Promotion to service providers was done in a number of ways. The service was promoted to physicians, various health organizations and various health networks. The purpose of promoting the 7 The challenge in finding and contracting French-speaking psychiatrists is further discussed under Challenges. 20

21 service to service providers is to gain buy-in and have them use and/or refer their patients to the French Telemedicine Mental Health Services. This was done by both the distribution of information including sending letters to physicians, to more direct contact such as giving presentations and having face-to-face meetings with individuals and organizations that could provide referrals. The family doctors, for example, have been aware and that has made possible to get the program started from this angle, there has been engagement at the school level, as well as the emergency psychiatric centres. This made it possible for referent doctors to know this service is available for the French community (Key Informant) The partners worked together in promoting the service to service providers. Both OTN and Trellis used their resources and networks to help promote the service. OTN [published] some articles about the service in their newsletter, and now [the service is] on their website, in the directory, detailing each service that is available. (Key Informant) Promotion to the Francophone community Promotion to the Francophone community was also carried out in order to create awareness of the French Telemedicine Mental Health Service. This involved outreaching to various organizations and cultural associations across Waterloo-Wellington to promote the service among their clients. Different strategies were executed, including: information sessions, presentations, and article disseminations. They ve always had that back and forth with the Francophone community, and media events, different forms of communication were used, so people were really made aware of the program and that kept it front and centre. Because of the communication that s happened, it s heightened the awareness in the community about it (Key Informant) One key informant noted that the outreach strategies that were used for the Francophone community were appropriate for the community, and that there was a commitment amongst the partners to increase awareness in the community about the service. There was a promotion campaign that was carried out. The plan, I think, resonates well with the French community in the region, and the partnership that was created to use the service, made sure that the community was aware of the project (Key Informant) Service delivery The final component of project implementation is service delivery. Service delivery includes continuing to engage the referral base as well as the process of delivering the service to clients, starting with the referral, intake, facilitating access (coordinator) and ending with the assessment and consultation and subsequent treatment plan. Regardless of who contacts the project coordinator, the process starts with a phone call where the coordinator learns a little bit about the patients and their needs. From there the appropriate paperwork begins. One key informant described the service delivery process as follows: Once the family physician or provider sends the referral the process [starts]. [...] From there [the coordinator] would contact the psychiatrist that did the appointment and then contact the client [to confirm the appointment] if it s okay [the coordinator] goes back to the psychiatrist 21

22 to confirm, and we continue the process and get ready for the consultation. The day of the consultation [the coordinator] is there with the client to introduce the system and go through the privacy things, and then according to the patient [the coordinator] stays with them in the room or not, depending if they feel comfortable or not. After the consultation [the coordinator] is there again if [she] didn t stay in the room for the follow up and coordination with other services, and follow up with the family physician or other organization in community if needed (Key Informant) The referral process varies depending on the psychiatrist as well as the client s age, etc. For example, the referral process for a child is more comprehensive in terms of the paperwork that needs to be completed as well as the parties involved. Parents and school staff (e.g. social worker) are all involved in the referral of a child for this service. Parents consent is a key requirement in the referral process for those under 18 years of age. In the case of older children, the consent may be given jointly by both the parent and the child. Although only two consultations had been completed at the time of the interviews for this evaluation, key informants were able to share their initial perceptions of the service delivery process. One key informant shared what the process was like for the first client: She was really happy to speak with a French-speaking psychiatrist, she felt comfortable with [telemedicine], she was saying that this kind of service should be provided more often (Key Informant) Other than having to wait for her appointment to be scheduled and a few technical glitches, the key informant shared that the client was emotional when expressing her happiness about being able to access this service in French. By the second consultation the technical issues had been resolved. The initial one had a few technical glitches; that being said, the feedback from the patient and provider were very positive. The second consult went off without any issues whatsoever. Like any program there s always growing pains and people getting used to technology, but for the most part I ve heard very positive feedback (Key Informant) It had been planned that the coordinator stays in the room with the client during the consultation with the psychiatrist. During the first consultation, the client was not comfortable with the coordinator in the room, so she made sure she was safe and left the room and the consultation took place. Since this was not the original plan, the coordinator sought the necessary clarification on the best practice when it comes to being in the room with the client during the consultation. Both OTN and Trellis were consulted on this matter in order to determine the best practice for this service, specifically. It was decided that for the safety of the client, the coordinator, who is part of the circle of care, will stay in the room during the consultation. After the first consultation, it was determined that timing of appointments needed to be discussed by the coordinator and the individual psychiatrists. This mainly had to do with the length of appointments and how there will be a need to be more flexible with timing depending on the psychiatrist s preferences. 22