Using the CDC framework for program evaluation in public health to assess tuberculosis contact investigation programs

INT J TUBERC LUNG DIS 7(12):S375 S383 2003 IUATLD Using the CDC framework for program evaluation in public health to assess tuberculosis contact investigation programs S. Logan,* J. Boutotte,* M. Wilce, S. Etkind* * Division of Tuberculosis Prevention and Control, Massachusetts Department of Public Health, Jamaica Plain, Massachusetts, Division of Tuberculosis Elimination, Centers for Disease Control and Prevention, Atlanta, Georgia, USA SUMMARY SETTING: In Massachusetts, despite the efforts of state and local health department tuberculosis (TB) programs, the rates of contact testing and follow-up remain below the state and national objectives. Changes in contact investigation practices are therefore needed to achieve these objectives. OBJECTIVE: To develop contact investigation selfevaluation tools in accordance with the Centers for Disease Control and Prevention s (CDC) Framework for Program Evaluation in Public Health. These tools will be used to assess state and local level contact investigation practices. DESIGN: The self-evaluation tools were developed using the CDC s framework and pilot-tested by public health nurse case managers in five city health departments. The tools were revised according to feedback received from the nurses. RESULTS: The Massachusetts TB Division conducted three of the six steps of the CDC s framework. Stakeholders of the evaluation were identified and engaged, logic models were created describing state and local TB program components, and self-evaluation tools were developed. CONCLUSION: The CDC s framework provided a useful methodology for beginning the assessment process for evaluating TB contact investigation programs. When the contact investigation self-evaluation tools are implemented statewide, the findings will be used to target areas in need of improvement and develop strategies to make noteworthy changes. KEY WORDS: program evaluation; tuberculosis; contact investigation ACCOUNTABILITY is a key issue for public health agencies, and one way to justify program operations and expenditures is through evaluation. Program evaluation identifies practices that are successful and worth funding, as well as areas in need of improvement. However, in many cases, proper program assessments are not done because staff may lack the necessary tools and expertise. The Centers for Disease Control and Prevention (CDC), in collaboration with evaluation experts, developed an organizational framework for program evaluation of public health practices in 1997. 1 According to the framework, well-designed, credible program evaluations adhere to the following standards: 1 3 1) utility ensuring that the user s information needs are satisfied; 2) feasibility ensuring that the evaluation is viable and pragmatic; 3) propriety ensuring that the evaluation is ethical; and 4) accuracy ensuring that the evaluation produces findings that are considered correct. In addition, program evaluations should be practical, done on a routine and ongoing basis, and involve all program staff. 1,2 Tuberculosis (TB) contact investigation procedures vary widely, 4 as do program operations, 5 7 sometimes leading to ineffective programs. Recent studies show that contact investigations may not be achieving optimal outcomes, 8,9 Contact investigations often fail to identify the most at-risk contacts in a timely manner, 10 and systematic failures in processes for conducting contact investigations result in missed opportunities for TB prevention. 11 Recently, the Advisory Council for the Elimination of Tuberculosis recommended enhancing the effectiveness of contact investigations. 12 Program enhancements, such as clarifying screening and monitoring policies 13 and improving training, 14 can positively impact on contact investigation outcomes. An evaluation of these program enhancements is needed to provide data on their effectiveness. This evaluation project focuses on contact investigation programs in the state and local health departments in Massachusetts. Although contact investigation outcomes have recently improved due to initiatives by the Massachusetts Department of Public Health Division of Tuberculosis Prevention and Control (TB Correspondence to: Janice Boutotte, Division of Tuberculosis Prevention and Control, Massachusetts Department of Public Health, 305 South Street, Jamaica Plain, MA 02130, USA. Tel: ( 1) 617-983-6986. Fax: ( 1) 617-983-6990. e-mail: janice. boutotte@ state.ma.us

S376 The International Journal of Tuberculosis and Lung Disease Division), they still fall short of state and national objectives. For example, the evaluation rate for contacts of acid-fast bacilli (AFB) smear-positive cases in 1999 was 60.5%, rising to 73.2% in 2000 (the national objective is 95%), 15 and the latent TB infection (LTBI) treatment completion rates in 1999 and 2000 were 60.0% and 59.2%, respectively (the national objective is 85%). Thus, further changes in the contact investigation program appear to be warranted. One of the purposes of this project was to use the CDC s framework for program evaluation and adapt it for application by local level program managers and staff. The process of working through the framework s six steps can result in a set of self-evaluation tools that meet the established standards for contact investigations. 16 Once the tools are designed, pilottested, and validated, the TB Division will provide training and guidance at the local level on the appropriate use of the tools. STUDY SAMPLE Public health nurse case managers in five city health departments in Massachusetts agreed to participate in this evaluation project. Nurse case managers manage TB cases and suspects who reside in their local communities, and conduct the contact investigations. These health departments were selected because they 1) were located in a higher-risk city, defined as those cities with: a) a 7-year average incidence of four or more cases and b) a 7-year mean TB case rate above the state average, 2) represented each of the five TB geographical regions of the state, and 3) had experienced staff and well-developed contact investigation policies and procedures in place. Each health department received $200 and signed a statement of commitment to the project. APPLYING THE CDC S FRAMEWORK FOR PROGRAM EVALUATION Contact investigation self-evaluation tools for use by local public health agencies were developed in accordance with the CDC s Framework for Program Evaluation in Public Health. 1 There are six main steps to the CDC s framework: 1) engage stakeholders; 2) describe the program; 3) focus the evaluation design; 4) gather credible evidence; 5) justify conclusions; and 6) ensure use and share lessons learned. The first three steps of the framework were conducted for this evaluation and are described as they were applied to the local and state contact investigation programs. Step one: engage stakeholders Step one of the framework engages stakeholders, persons or organizations having an investment in what will be learned [by the evaluation] and what will be done with the knowledge. 1 There are three principal groups of stakeholders: 1) those involved in program operations; 2) those served or affected by the program; and 3) primary users of the evaluation. Nurse case managers were identified as key stakeholders and primary users of the program evaluation. Other important stakeholders are the state public health nurses from the TB Division who advise local nurse case managers and the TB Division s contact program staff. TB cases, suspects, and their close contacts are stakeholders because the program serves them and they are affected by the findings. Others affected by the evaluation are workers in congregate settings (e.g., jails, shelters, and schools) or in private businesses. Community members served by the local TB programs are also stakeholders because they make decisions on funding for public health resources. Finally, the CDC Division of TB Elimination is an important stakeholder because not only is it a major funding source for a large portion of the TB Division s staff and activities, it also provides the TB Division with guidance, sets standards and objectives, and works closely with the TB Division in many research and program activities. For this project, the local and state level public health nurses were the only stakeholders invited to plan and pilot test the evaluation. In the planning stage, important values and standards about the contact investigation program were elicited. They discussed values such as building a trusting relationship with the TB case, counseling and educating contacts, and providing interpretation services for non-english speaking contacts. The nurses also indicated that interpersonal and communication skills are vital to success in contact investigations. The local and state public health nurses wanted the self-evaluation tools to exemplify high standards for contact investigation. It was important that the evaluation materials gathered data that would accurately assess the programs. The tools should also be useful for training purposes, especially for new or inexperienced nurse case managers. Other ideal values they placed on the self-evaluation tools were that they be kept as simple as possible and be universal, to fit almost every case/suspect investigation. Step two: describe the program Step two of the framework describes the state and local level contact investigation programs to convey the mission and objectives of the program being evaluated. 1 This includes explaining the need for the program and what it aims to accomplish, the activities of all persons involved, the resources, the program s stage of development, and the setting and environment in which the program operates. In step two, the project staff described the state and local contact investigation programs. Information about the state program was gathered from a variety of sources: state TB Division written policies,

Assessing TB contact investigation programs S377 procedures, reports, and recommendations; daily activities of staff involved in patient management and contact investigations; budget documents; and aggregate reports of contact follow-up and treatment of LTBI. Information about local TB programs was obtained by interviewing nurse case managers at the pilot sites. Within a week after the initial interviews, the research assistant made phone contact with the nurse case managers and, using a standardized data collection tool, gathered information on their resources and capacity to conduct contact investigations. They were asked about the health departments infrastructure and resources, such as staff time and experience, equipment, foreign language capabilities, methods for documenting contact information, and funding sources. Selected data on health department staff and resources were organized into a chart for comparison (see Table 1). Case rates, averaged over a 7-year period from 1994 to 2000, varied among the five health departments, ranging from 6.30 to 14.9 per 100000. Staff time was proportionate to the rate of cases: the cities with the higher case rates had more available staff. The availability of office equipment varied among the five health departments. Administrative support also varied among the five health departments, with the program with the greatest morbidity reporting no administrative support and only one full-time outreach worker. Finally, three of the five health departments collected contact investigation data only on the state forms, which record simple demographics, TB skin tests, chest X-rays, and treatment of LTBI; no additional data were collected that might help the nurse case managers to prioritize contacts and more effectively manage their cases. A major task of step two was the construction of logic models, graphic representations that illustrate and summarize the contact investigation program. In evaluation, logic models are used to provide a documented description of the rationale behind program activities and the cause-and-effect relationships between specific activities and expected outcomes. 17,18 In addition to their uses during an evaluation, logic models can also serve as valuable communication tools by visually summarizing a program for administrators, funders, and new staff members. 19 Table 1 TB case statistics and resources in five city health departments in Massachusetts 7-year mean caseload (case rate)* Staff time and experience in TB Office equipment HD 1 HD 2 HD 3 HD 4 HD 5 88 (14.9) 9 (9.2) 8 (8.76) 14 (13.3) 11 (6.30) 5 FT public health nurses; 1 nurse supervisor; 4 20 years in TB Voice mail, e-mail, fax machines, copiers, computers, printers and direct connection to the state network via high-speed modems 3.5 FT public health nurses; 1 nurse supervisor; 7 20 years in TB Voice mail, e-mail, fax machines, computers, printers, highspeed internet access, and access to laboratory and radiology reports 2 FT public health nurses; 1 nurse supervisor; 1 30 years in TB 1 centrally-located fax machine and an inadequate computer 3.5 FT public health nurses; 1 nurse supervisor; 20 25 years in TB E-mail, fax machines, copiers, computers, and printers, but no personal voice mail Administrative Inadequate Inadequate ND Excellent Excellent support Outreach 1 FT outreach Available from an outside source as needed Nursing staff language capability Access to interpreters Record-keeping other than using state forms Chinese, Vietnamese and Spanish Excellent; through hospital staff A separate chart is used Haitian Creole, Cape Verde Creole, Portuguese and Spanish Excellent No additional data collected; use MS Access database None Available as needed 2 FT public health nurses, but only 1 does CI; 3 FT nurses help with DOT as needed; 1 2 years in TB Fax machines, phones (direct lines and a long distance line), and copiers Available from an outside source as needed English only ND Office staff speak Spanish Adequate; one Asian interpreter once/week; also phone interviews No additional data collected Excellent Additional data collected Adequate; Albanian and Vietnamese interpreters through outside source No additional data collected * Data are averaged over a 7-year period, 1994 2000. Case rates are number of cases per 100 000 population, based on 2000 census information. Described on the scale: excellent, adequate, and inadequate. HD health department; TB tuberculosis; FT full time; DOT directly observed treatment; CI contact investigation; ND not determined.

S378 The International Journal of Tuberculosis and Lung Disease (continues) * Outreach educators are employed by MDPH TB Division and Refugee and Immigrant Health Program; they work on the local level with TB cases and their families. The TB clinics are funded by the MDPH TB Division, but clinic personnel (physicians, nurses, office administrators) are employed by the local hospital or health department. A systematic approach, recommended by the MDPH TB Division, to identify, prioritize, and test contacts (see ref 15). Figure 1 Logic model of the contact investigation process: state and local level interactions. Text in bold highlights necessary services/interventions provided by the Contact Program. Note that these activities are linked to resulting outputs and outcomes. If federal funding was no longer available, the capacity to maintain these activities would be greatly compromised. VNA Visiting Nurse Association; BOH Board of Health; LTBI latent tuberculosis infection; AFB acid-fast bacilli; TST tuberculin skin test.

Assessing TB contact investigation programs S379

S380 The International Journal of Tuberculosis and Lung Disease Figure 2 Summary logic model of the contact investigation process: state and local level interactions. BOH Board of Health; VNA Visiting Nurse Association; CI contact investigation; AFB acid-fast bacilli; LTBI latent tuberculosis infection. Three logic models were constructed: the first model depicts the interactions between the state and local level contact investigation programs (Figure 1), the second summarizes the previous model (Figure 2), and the third is an illustration of the local program only (not shown). The third model is a duplicate of the first model, except that it does not contain state-dependent resources and responsibilities. Each model lists current resources, activities, intermediate outputs, and expected outcomes. The activities and outputs are essential elements in Massachusetts TB case and contact management and are outlined in the state nursing protocols. The expected short-term outcomes are national and state contact investigation objectives. Finally, the expected long-term outcome, to prevent TB cases from occurring, depends on the fulfillment of the short-term outcomes. The complexity of the models necessitates some explanation. The models are arranged in sequential order, from notifying public health officials about the infectious case to generating contact reports and conducting a self-evaluation at the conclusion of the investigation. Arrows link each column with the next, portraying how the activities are involved in cause-and-effect relationships with the intermediate outputs. Also, certain boxes within the same column are grouped. For example, in Figure 1, the sixth and seventh activities are grouped under contact identification. These activities are related to the intermediate outputs of identifying contacts, such as contacts were identified within 3 working days. Finally, the intermediate outputs lead to the expected short-term outcome, which is the national objective that contacts are identified for at least 90% of highly infectious TB cases. Step three: focus the evaluation design Step three of the framework focuses the evaluation design to assess the issues of greatest concern to stakeholders while using time and resources as efficiently as possible. 1 Four basic elements of focusing the evaluation design were addressed: 1) the purpose of the evaluation plan (to change contact investigation program practices that were not working well and to motivate the nurse case managers to utilize program evaluation and build it into the framework of their contact investigation programs); 2) the intended users, particularly the state and local public health nurses; 3) the potential uses of the evaluation findings; and 4) evaluation questions to measure the contact investigation program s quality and effectiveness. Table 2 provides a sample list of questions developed by project staff and key stakeholders.

Assessing TB contact investigation programs S381 Table 2 Selected evaluation questions for step 3 of the CDC s framework: focusing on the evaluation design Sample questions Did the nurse case manager identify the initial contacts within 3 working days of TB case notification? Was the nurse case manager able to retrieve results from private providers? Did the contacts with newly acquired LTBI receive chest X-rays and were they evaluated for treatment? Did the nurse case manager send the contact report to the state nurse after the first round of TB skin testing? If so, within how many days was it sent? Did the nurse case manager use/distribute the following materials to other health care providers and contacts during this contact investigation, when appropriate: the TB Contact Investigation Fact Sheet and the contact educational pamphlet, Someone I Know Has TB? CDC Centers for Disease Control and Prevention; TB tuberculosis; LTBI latent tuberculosis infection. Developing and pilot-testing contact investigation self-assessment materials Information obtained from working through steps 1 3 of the CDC s framework laid the groundwork for creating self-assessment materials. Three tools were created: 1) the self-evaluation questionnaire, 2) the transmission risk assessment checklist, and 3) the contact investigation decision tree. Each tool was pilot-tested in the participating cities. The latter two tools were not designed to evaluate contact investigations, but to improve the investigators performance in conducting contact investigations. First, the self-evaluation questionnaire was created by formulating evaluation questions based on program activities and outputs depicted in the logic models. The questionnaire form was organized into four parts: 1) identification of contacts, 2) medical evaluation, 3) treatment of LTBI, and 4) administrative (e.g., timeliness of reporting and utilizing educational materials developed by the TB Division). Where appropriate, TB Division recommendations were inserted into the questionnaire as a reminder to the nurse case managers conducting the evaluation. The nurse case managers at the pilot sites were instructed to complete a minimum of two self-evaluation questionnaires on previously conducted contact investigations. Eleven questionnaires were received from four of the five pilot sites and entered into the database. The results were inspected for missing data, unanswered questions, inserted answers, and side notes that might indicate confusion about a question or that a question was unnecessary. In the post pilot test meetings with the nurse case managers, more was learned about the usefulness and feasibility of the questionnaire. One question was particularly problematic: Did the close contacts the nurse case manager identified have a higher than expected rate of infection? They had difficulty determining what was higher than expected, especially for non-us-born contacts who came from countries with a high TB incidence. Another question the nurse case managers had difficulty answering was, Were the contacts of the infectious case identified within 3 working days of TB case notification? They felt it was unrealistic to expect the TB case/suspect to disclose names of close family and friends at their first encounter; sometimes it takes 1 or 2 months to build a trusting relationship with the patients. Reasons for not cooperating were anger about the diagnosis, mistrust of public officials, and fear of being kicked out of the home. A nurse who used the questionnaire suggested that space be provided to explain the difficulty in identifying contacts. Other general comments made were that the questionnaire be universal to fit most types of cases, simplified, flexible enough to be used during and after the contact investigation, and should not include redundant information recorded on other forms. The second component of the self-assessment materials was the transmission risk assessment checklist, which is to be used at initial visits to the TB case/ suspect s home, work or school, and other places where they spent a lot of time. The checklist provides details centered on the TB case/suspect: the degree of infectiousness, type of housing, work environment, and amount of time spent with friends or in other social settings. The nurse case managers were instructed to complete a minimum of three risk assessment checklists on current contact investigations. However, no new cases/suspects occurred in the 1-month period given them to use the tools. Six checklists completed on previous cases were received from two pilot sites, two other sites did not complete the forms because they had no new cases/suspects, and the fifth site did not submit the forms. The nurse case managers felt that the checklist, which contains check-off items regarding the size of the room, ventilation, etc., would be helpful to inexperienced investigators. Finally, the third component of the self-evaluation tools was the decision tree. It illustrates the contact investigation process at the patient level, from determining the infectiousness of the TB case/suspect to screening close contacts for TB infection and disease. The nurse case managers were to use the decision tree as a guide in their current investigations, but due to a lack of current cases, they were not able to use it as instructed. However, most of the nurses said that they liked the decision tree and would use it in their investigations. LESSONS LEARNED The Framework for Program Evaluation in Public Health was found to be a useful methodology to plan a TB contact investigation program self-assessment. By working through the first three steps of the framework, a rather complex and detailed process was

S382 The International Journal of Tuberculosis and Lung Disease pulled apart and scaled down to focus on local programs and their interaction with the state program in conducting contact investigations. It was also found that keeping a project log and documenting lessons learnt along the way were important to maintain perspective, review procedures, and record unsuccessful strategies. The state and local public health nurses, as well as the CDC, were key stakeholders and played an essential part in laying the groundwork for the evaluation. They were involved in each step of the process: affirming their commitment to program evaluation, reviewing drafts of the program descriptions and logic models, and providing input into the development of the selfassessment materials. Using stakeholders to assist in planning the evaluation, as recommended by the CDC s framework, produced evaluation questions that were based on their value systems. More importantly, the product of this collaboration was an instrument they would use. For example, the self-evaluation questionnaire was designed to accommodate most of the investigators needs. It can be used as an analytic tool (to calculate rates and timeliness of their investigations) and a form to take notes on cases when necessary. The process of developing logic models, step two of the CDC s framework, required discussion and consensus among stakeholders, thus elucidating assumptions and expectations. 20 Multiple logic models are often needed to meet the needs of the different stakeholder groups. 21 For this evaluation project, three models were developed, each highlighting different aspects of the contact investigation program. The first, developed for staff directly involved in the program, showed the details of how the state and local programs coordinate activities related to contact investigations (see Figure 1). The summary model (see Figure 2) explained this complex network to managers, clients, and others who needed an overview of operations. The local model enabled nurse case managers and program managers to see the positive effects of their activities on program goals and objectives. This effort (creating logic models) was the first attempt to systematically document how contact investigation programs achieve patient and cohort level outcomes. As indicated previously, the local nurse case managers pilot-tested the self-evaluation materials for a 1-month period. This exercise provided valuable information and insight into each of the next steps of the CDC s framework. Regarding their future use, the Massachusetts TB Division recently started a 3-year project to study intervention strategies for contact evaluation, one of which is utilizing the transmission risk assessment checklist. The Division proposes to focus on the evaluation phase of contact investigations (initial and repeat skin testing and medical evaluations) in several higher risk cities and on large worksite investigations to observe whether interventions such as the checklist affect rates. In conclusion, the goals of contact investigation program evaluation, using the CDC s framework as a guide, are to improve contact outcomes and achieve state and national objectives. The means of reaching these goals include: 1) using the validated selfevaluation tools, as well as the logic models, as: i) guidelines for recommended contact investigation program activities and ii) training tools for new and inexperienced nurses, 2) assessing each contact investigation conducted, 3) using the evaluation findings to target problem areas for change, and 4) making recommendations based on valid and credible evidence, which will help garner political support and funding to sustain successful program activities and provide resources for new strategies. It is hoped that through the utilization of these evaluation instruments and findings in Massachusetts, and by conducting similar self-evaluation projects elsewhere, the rates of contact medical evaluation and treatment completion will improve. Acknowledgements The authors would like to thank Heather Miller, Kathryn Hendricks, MDPH Division of Tuberculosis Prevention and Control Patient Management Services Unit, and the public health nursing staff of the Boston, Cambridge, Lowell, Quincy, and Worcester, MA Health Departments. Supported by a grant from the Division of Tuberculosis Elimination, Centers for Disease Control and Prevention, Atlanta, GA. References 1 Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999; 48(RR- 11): 1 40. 2 Patton M Q. Utilization-focused evaluation: the new century text. 3rd ed. Thousand Oaks, CA: Sage Publications, 1997. 3 Joint Committee on Standards for Educational Evaluation. Program evaluation standards: how to assess evaluations of educational programs. 2nd ed. Thousand Oaks, CA: Sage Publications, 1994. 4 Wilce M, Shrestha-Kuwahara R, Taylor Z, Qualls N, Marks S. Tuberculosis contact investigation policies, practices, and challenges in 11 US communities. J Public Health Manag Pract 2002; 8: 69 78. 5 Glaser T, Simmons C. Contact investigation in New York City. Investigation of Contacts to Tuberculosis Cases Symposium Summary. New York, NY: New York City Department of Health, Bureau of Tuberculosis Control, 1996: 13 18. 6 Holcombe J M. Contact investigation in a rural setting: a state perspective. Investigation of Contacts to Tuberculosis Cases Symposium Summary. New York, NY: New York City Department of Health, Bureau of Tuberculosis Control, 1996: 20 25. 7 Ruben F L, Lynch D C. Tuberculosis control through contact investigation. PA Med 1996; 99: 22 23. 8 Marks S, Taylor Z, Qualls N, Shrestha-Kuwahara R, Wilce M, Nguyen C. Outcomes of contact investigations of infectious tuberculosis patients. Am J Respir Crit Care Med 2000; 162: 2033 2038. 9 Reichler M R, Reves R, Bur S, et al. Evaluation of investiga-

Assessing TB contact investigation programs S383 tions conducted to detect and prevent transmission of tuberculosis. JAMA 2002; 287: 991 995. 10 Chin D P, Crane C M, Ya D M, et al. Spread of Mycobacterium tuberculosis in a community implementing recommended elements of tuberculosis control. JAMA 2000; 283: 2968 2974. 11 McAnulty J M, Fleming D W, Hawley M A, Barron R C. Missed opportunities for tuberculosis prevention. Arch Intern Med 1995; 155: 713 716. 12 Centers for Disease Control and Prevention. Tuberculosis elimination revisited: obstacles, opportunities, and a renewed commitment. Advisory Council for the Elimination of Tuberculosis (ACET). MMWR 1999; 48(RR-9): 1 12. 13 MacIntyre C R, Plant A J. Impact of policy and practice on effectiveness of contact screening for tuberculosis. Prev Med 1998; 27: 830 837. 14 Jasmer R M, Hahn J, Small P, et al. A molecular epidemiologic analysis of tuberculosis trends in San Francisco, 1991 1997. Ann Intern Med 1999; 130: 971 978. 15 Massachusetts Department of Health, Division of Tuberculosis Prevention and Control. Aggregate Reports for Tuberculosis Program Evaluation: follow-up and treatment for contacts to tuberculosis cases. Jamaica Plain, MA: MA Dept of Health, 1999 2000 data reports. 16 Etkind S, Veen J. Contact follow-up in high- and low-prevalence countries. In: Reichman L, Hershfield E S, eds. TB: a comprehensive international approach. New York, NY: Marcel Dekker, 2000: pp 385 388. 17 United Way of America. Measuring program outcomes: a practical approach. Arlington, VA: United Way of America, 1996. 18 Weiss C. Evaluation. 2nd ed. Upper Saddle River, NJ: Prentice Hall, 1998. 19 Porteous N, Sheldrick B J, Stewart P J. A blueprint for public health management: a program evaluation tool kit. Ottawa: University of Ottawa, 1997. 20 Wholey J S. Evaluability assessment: developing program theory. In: Bickman L, ed. Using program theory in evaluation. new directions for program evaluation, 33. San Francisco, CA: Jossey-Bass, 1987. 21 Harris J. Logic model basics. The Evaluation Exchange, 2001; 2(VII) 2: 14.