National Grid. Upstate New York EnergyWise Program Process Evaluation (Final) October 9, 2012

Transcription

1 National Grid Upstate New York EnergyWise Program Process Evaluation (Final) October 9, 2012

2 National Grid Upstate New York EnergyWise Program Process Evaluation (Final) October 9, 2012 Copyright 2012 Tetra Tech, Inc. All Rights Reserved. Prepared for: National Grid Tetra Tech 6410 Enterprise Lane, Suite 300 Madison, WI Tel Fax ii

3 TABLE OF CONTENTS TABLE OF CONTENTS... iii APPENDICES... v TABLE OF TABLES... v 1. Executive Summary Program and Evaluation Overview Original Program Design Program Design Changes Evaluation Methodology Summary of Key Findings Areas that are Working Well Conclusion and Recommendations Introduction Program Background Original Program Design Program Design Changes Program Staffing and Operations Program Objectives and Achievements To-date Program Logic Model Evaluation Methodology Key Researchable Issues Data Collection Report Organization Detailed Process Evaluation Findings Summary of Key Findings Program Successes iii

4 3.1.2 Challenges to Program Success Program Recommendations Program Administration and Processes Program Team Structure and Collaboration Program Tracking and Information Quality Control Customer Awareness and Marketing Program Awareness Program Understanding Ease of Participation Barriers to Participation Decision-making Processes Equipment Installation Program Participation Program Satisfaction Overall Satisfaction Satisfaction with Specific Program Aspects Suggested Program Changes Customer Characteristics Recommendations iv

5 APPENDICES Appendix A: Program/Implementation Staff Interview Guide... A-1 Appendix B: Building Owner Guide... B-1 Appendix C: Tenant Survey Instrument... C-1 Appendix D: Tenant Survey Response Rate... D-1 Appendix E: Completion Certificate... E-1 TABLE OF TABLES Table 2-1. EnergyWise Program Rebates Table 2-2. EnergyWise Program Progress Table 2-3. Key Researchable Issues Table 3-1. Tenant s Reasons for Participation Table 3-2. Benefits of Participation Table 3-3. Installation Persistence Table 3-3. Tenant Satisfaction with Program Aspects Table 3-4. Tenant Characteristics v

6 1. EXECUTIVE SUMMARY This report presents the findings and recommendations resulting from the 2011 process evaluation of National Grid s EnergyWise Program in upstate New York, and includes feedback from participants and program activity between June 2010 and June 2011.This report is one of a series of energy efficiency portfolio standard (EEPS) process evaluation reports of National Grid s energy efficiency programs in New York. Preliminary results, from the initial in-depth interviews with program and implementation staff (including trade allies), were provided to National Grid in memo format in April The memo also included preliminary recommendations based on the early evaluation activities. 1.1 PROGRAM AND EVALUATION OVERVIEW The EnergyWise program was initially approved in July 2009, with the implementation plan resubmitted with new fixtures approved in April Final approval was received in May 2010 and the program officially launched in June Original Program Design The EnergyWise program in upstate New York targets individually metered electric and/or gas multifamily buildings with five to 50 dwelling units. Eligible customers include National Grid multifamily customers of record that are building owners and property managers or individual dwelling customers on residential rates. Single family home owners, residing in dwellings of one to four units, are not eligible. In the original program design, multifamily buildings that are master metered for both heating and water heating were not eligible for the program. The parties involved in the design and delivery of the EnergyWise program include the program manager at National Grid, managers at RISE Engineering (RISE) and ICF International (ICF), auditors at both RISE and ICF, and equipment installers at RISE or who are managed by RISE. The program is being promoted and delivered by implementation contractors, RISE Engineering and ICF. Promotion also occurs through National Grid s website and other marketing activities. RISE works with architects, builders, engineers, remodeling contractors, technicians, retailers, equipment suppliers, and related channel intermediaries. Originally the program was designed to provide participants with a free energy use assessment and low-cost measures at no direct cost. These low-cost efficiency measures include CFLs, low flow showerheads, aerators, domestic hot water pipe wrap, and domestic hot water tank wrap. In addition, the program was to provide rebates of 25 percent for the installation of a number of electric and gas building envelope measures such as insulation, air sealing, attic ventilation, ductwork and air infiltration testing. Lighting fixtures in common areas were offered to building owners for $20 per fixture. Lastly, the program offered $300 toward the cost of each new early replacement refrigerator and up to $75 for a normal replacement refrigerator. 1 A preliminary results memo was delivered to National Grid in April 2011 based on feedback from program and implementation staff. The draft final report was delivered to National Grid on March 23, Final comments on the draft report were received by the evaluation team in May 2012 from National Grid and August 2012 from DPS. 1-1

7 1.1.2 Program Design Changes Since the program launch in June 2010, there have been a number of adjustments made to program design. While these adjustments were not included in the evaluation because they had not yet been made, these do warrant mention. Initially, only multifamily buildings that are individually metered for electric and/or gas service were eligible for the program. While National Grid was progressing well against their electric savings targets, they found that it would be nearly impossible to meet the original gas savings goals given the stock of individually metered gas units in Upstate New York. National Grid petitioned the PSC regarding the original forecasted goals and the likely potential of meeting those goals. In June 2011, the PSC approved the eligibility of master metered gas buildings for the program. Once the program was launched and the energy assessments were proceeding, National Grid determined that the insulation and air sealing efforts were generally not cost effective and decided to focus more attention on the direct install measures (CFLs, showerheads, aerators, pipe wrap, and tank wrap). It has also been the case that virtually none of the refrigerators examined during the energy assessments were found to meet the guidelines for the refrigerator replacement rebate. In researching what other measures could be offered to replace the measures that were discontinued, National Grid found verbiage in the filing that allowed for measure additions if a similar program exists that is also rebating that measure. National Grid found another program rebating programmable thermostats which were cost effective, as well as boiler reset controls. Both of these measures were approved for the program in July As a result of the preliminary findings from the staff interviews and drafting of the logic model conducted as part of this process evaluation, the program staff have implemented quality assurance and quality control (QA/QC) inspections that were originally intended as part of the program but had not occurred early on. ICF is in charge of targeting ten percent of the participating properties to review what has been installed and to conduct a short survey with property managers. Program implementation staff are now trying to determine the most appropriate way to track the QA/QC information using the InDemand system. The most recently added benefit to building owners and managers from the program is the availability of EnergyWise Choice logo window decals that property managers can use for advertising participation in the program and upgrades to individual units Evaluation Methodology Process evaluation activities for the EnergyWise Program consisted of qualitative interviews with program and implementation staff, qualitative interviews with participating and dropout building owners and managers, and structured quantitative phone surveys with tenants. The process evaluation covered audit and equipment installations through the program between June 2010 and June

8 a. Program and Implementation Staff Interviews The first evaluation activity for the EnergyWise program consisted of in-depth interviews with program and implementation staff involved with the design and/or delivery of National Grid s EnergyWise Program in upstate New York State. Tetra Tech conducted one interview with the upstate EnergyWise program manager and six interviews with implementation staff. National Grid provided a list of implementation staff with key responsibilities with the program in a variety of different roles. The six implementation staff interviewed were from RISE Engineering and ICF and included managers as well as auditors. Interviews covered a variety of topics depending on each interviewee s role within the program. Topics covered included 2 : Roles and responsibilities Program design and marketing Program operations Internal and external communication Customer and trade ally outreach and interaction Quality assurance Program database and tracking Barriers to program uptake and meeting goals and objectives Satisfaction with program Areas of interest to investigate during subsequent data collection. A copy of the in-depth interview guide can be found in Appendix A of this report. b. Participating and Dropout Building Owner/Manager Interviews Overall, 89 unique records at the local level and 25 unique records at the decision-maker level were provided by National Grid for the purpose of interviewing participating and dropout building owners and managers. We completed a total of 18 interviews, 14 with property managers at the local level, and four interviews with corporate-level decision-makers. Interviews were scheduled with six more local property managers, but they missed their appointments. Rescheduling efforts were unsuccessful and were discontinued to focus on reaching more corporate decisionmakers. Initially, we attempted to reach the contact listed in the sample file at the local level, which turned out in most cases to be the local property manager for the property. Since the majority of 2 Tetra Tech conducted interviews between March 8, 2011 and March 25, Interviews ranged in duration from thirty to sixty minutes, depending on the interviewee s role. We do not identify interviewees by name, nor do we identify them by role when discussing specific findings as a means of protecting interviewees confidentiality. 1-3

9 local property managers were not involved in the decision to participate in the program, we attempted to obtain information about the decision-making process by identifying and following up with the corporate-level contact for each property. After attempting to reach the corporatelevel contacts by phone and having little success (only two completed phone interviews), we tried to obtain addresses in order to send out a shortened questionnaire that contained key decision-making questions. This effort resulted in two additional completes. Calls to building owners and managers were attempted from late July to the end of August. At least two attempts were made to determine working numbers, leave voic messages, and identify the best way to reach the most knowledgeable contact. However, many cases were attempted four or five times in order to find a convenient time to schedule the interview. The interview guide can be found in Appendix B. c. Participating tenant phone surveys The last phase of evaluation activities consisted of CATI (computer assisted telephone interview) surveys with tenants served by the program. Files containing participating building owners and their tenants were provided by National Grid. Due to staffing changes at National Grid, the tenant survey was approved in November 2011 and calling occurred in December. A goal of 70 completes was met for the survey, which explored tenant characteristics, program awareness and satisfaction, measure persistence, and benefits of participation. The tenant phone survey can be found in Appendix C, and the detailed response rate can be found in Appendix D. 1.2 SUMMARY OF KEY FINDINGS The following summarizes what we have identified as the most important and recurring themes based on all of the evaluation activities for the EnergyWise program Areas that are Working Well There is a good team working relationship. The staff report that the working relationship among National Grid, RISE engineering, and ICF International is very supportive and collegial. Meetings are held regularly along with regular reporting of current progress and barriers. This leads to quick resolution of concerns and questions fielded by the staff in a timely manner. Initial indications are that the program team is united in addressing the challenge of identifying and engaging eligible gas customers to achieve the targeted gas savings for the program. Customer satisfaction is high. Several staff mentioned that program participants are satisfied with the program process. Feedback they have received from the completion certificates indicates that participants find the audit process easy and 90 to 95 percent are estimated to be satisfied with the program overall. This perception of high satisfaction is supported through interviews with property owners and managers, as well as with tenants receiving the measures. The program is meeting goals for electricity savings. The interviewees reported that the electricity savings side of the EnergyWise program is performing well, and has actually exceeded the current electricity savings goal. Staff also indicated that CFL replacement made up the bulk of electricity measures installed, and that the rebate levels for these measures are competitive in the current service area. 1-4

10 Competition from other programs for customers has been minor to date. Whereas other commercial program customers may be eligible for incentives from multiple sources, there is a mandated split in the multifamily market by size. National Grid can approach customers with five to 50 units per building while NYSERDA can approach those with more than 50 units. In addition, National Grid has been working collaboratively with all of the administrators delivering EEPS programs. The only concern will be possible confusion when the National Grid team approaches management firms that manage buildings in both categories, but that was not an issue with the management we spoke with. In fact, one large building manager feels the utilities are much better at serving customers with energy efficiency programs than NYSERDA and that there is little action from NYSERDA. The Program Manager is receiving necessary data. The team meets weekly by conference call and the implementers provide frequent reports to the program manager. The program database kept by the implementer is separate from the InDemand tracking and is more current. The program is appropriately staffed. The implementer closely monitors the level of activity and has adequate staff available for audits, installation, and data entry. RISE has also added a marketing person to support the marketing effort for EnergyWise, as it has proven to be more work than originally anticipated. Original issues with QA/QC checks have been resolved quickly. QA/QC was not occurring as intended when the program was initially rolled out. At first, RISE was checking a portion of their own installations, but ICF was responsible for QA/QC as a second set of eyes. Program staff reported there is no QA/QC requirement in the implementation plan and no associated budget for it. Interviewees also report that the focus on meeting the gas savings goal was distracting from the QA/QC part of the program process. National Grid has already clarified the QA/QC process for projects since the delivery of the preliminary staff interview results by confirming ICF s role as the QA/QC lead for the program and ICF has begun the process of selecting and verifying installations. Dropouts are not has high as expected. Dropouts are defined as those properties that had an energy assessment but had not yet installed any energy efficient equipment. Most projects listed as dropouts are not actually dropping out of the program, but are rather still in the process of determining what they will install at locations not already served by the program. Therefore, these cases would be more accurately categorized as in-progress rather than dropouts. Program satisfaction is high. Participants report a high level of satisfaction with the program experience and National Grid as a result of participation. Over half of the tenants (54 percent) reported that their experience with the EnergyWise Program increased their satisfaction with National Grid as their energy provider. The program is promoting energy efficiency upgrades. The group of building owners and managers we spoke with typically only implement upgrades when equipment fails or if there is turnover in a unit. The program is motivating them to replace equipment earlier than planned. a. Challenges to Program Success Because of the newness of this program, there were several challenges identified through the evaluation activities for the first year of the program. The challenges mentioned in the interviews are highlighted below, with suggestions for addressing them in the Recommendation Section. 1-5

11 The gas savings side of the Energy Wise program struggled early on. Staff indicated that the initial individual metering requirements drastically limited the pool of eligible gas customers. Several staff also indicated that the current gas incentive levels do not seem to be high enough to influence participation of the property owners, which has compounded the problem of the small eligible customer pool. Some staff have heard property owners requiring at least 50 percent of the project cost covered before they can justify the expenditure. Refrigerator cost-effectiveness requirements are unrealistic for the current service area. Staff report customers interest in the refrigerator rebate has driven overall interest in the program. However, several staff stated that it is nearly impossible for customers to qualify for a refrigerator rebate if the unit is less than 17 years old (early replacement). Implementation staff reported that almost none of the tested refrigerators ended up cost-effective based on the current calculation. Property managers confirm that this has led to frustration and confusion on their part as many were originally interested in the program for the refrigerator rebates. There has been difficulty in selling the program to property owners. The interviewees report that the current economic situation is causing property owners to overlook measures that benefit tenants, but do not directly benefit themselves. In addition, insulation and weatherization improvements are high-cost measures that require advance budgeting. Interviewees have found that the 25 percent program contribution is not enough to motivate customers to undertake this advance budgeting. They also indicate that gaining contact information and access to out-ofstate property owners has been a challenge, which evaluators confirmed to be the case when they attempted to contact these owners to complete an interview. Current marketing efforts have not been sufficient. Numerous members of the staff mentioned that more marketing materials are needed. They also indicate that the lack of marketing has been a factor in the current slow generation of energy audit leads. Mass marketing efforts have not reached the targeted customers and have created little activity on the program website. Cost-effectiveness screening at the measure level has impacted customer participation. Several staff mentioned that it is not uncommon for projects that achieve an overall positive cost ratio on the project level to be passed over due to the requirement that each measure be costeffective. They have also reported that measure level screening is causing confusion with customers, as well as burdening property owners with tracking measures installed on a buildingby-building basis. Staff feels that this leads to only a fraction of the measures a facility could receive being installed. Some property owners and managers confirmed that they did not understand why they were not eligible for the refrigerator or insulation rebates and why not all units are treated the same. The InDemand database has little support for report generation. Staff mentioned that the lack of a reporting feature in the InDemand database does not allow for an up-to-date picture of the current progress of the program. This has contributed to redundancy as other databases with reporting features are being used to track the same information. Program tracking data from InDemand is not 100 percent accurate. During the evaluation, we found that there were inconsistencies concerning how measures were recorded or pulled for the participating tenant interviews. For example, interviews with property managers revealed that they often had water saving measures installed while the database records received showed only lighting measures. Review of some of the applications from RISE revealed that the 1-6

12 full range of measures installed did not match what we received. It is not clear if the issues arise from data entry errors or how the queries pull data from the InDemand system. There is a lack of understanding regarding who offers services and eligibility criteria. Building owners and managers find it very difficult to determine who provides programs to assist the different types of multifamily properties with energy efficiency upgrades. There are varying eligibility criteria depending on the location of the property, whether buildings are individually or master metered, the type of equipment they are interested in replacing, and the number of units per building. Currently, many of the building owners and managers said that they are not completing energy efficiency projects because they are not able to identify utility or other programs to assist them. Multiple visits to properties lead to building owner and manager frustration. Some building owners and managers have gone through multiple visits from the program to serve their properties for electric and then gas measures. Others received one visit for tenant units, then another for common areas. These respondents did not understand why the efforts could not be coordinated and that it was a waste of time to go through the properties twice. The inability to serve the entire property removes some of the benefits to building owners and managers. In particular, the restriction that National Grid can serve the five to 50 unit properties diminishes the ability of building owners and managers with multi-unit properties to use the program and energy efficient equipment for marketing the units, especially if buildings in the complex do not fall within the five to 50 range. 1.3 CONCLUSION AND RECOMMENDATIONS Overall, feedback from program staff, implementers, building owners and managers, and tenants involved with the EnergyWise Program indicate that after a late launch and some midstream program adjustments, the program is on the right track and achieving estimated savings for gas. Participants are highly satisfied with their program experience and participation has improved their perception of National Grid. At the same time, the program faces several internal challenges, most specifically those associated with effective outreach and cost effectiveness testing. Current marketing efforts are not reaching multifamily customers efficiently and multifamily customers are uncertain of where to turn for energy efficiency upgrades. In addition, the inability for any refrigerators to pass the cost effectiveness testing, the requirement for individual measure cost effectiveness testing, and multiple property visits are testing the patience of some building owners and managers. Below are recommendations to consider that address some of the opportunities identified. Pursue the opportunity with regulators to expand to units and continue to target mid-sized property management firms that handle multiple properties. The implementation team feels they have been more successful in targeting some of the mid-sized property management firms and working with them to serve multiple facilities than targeting individual properties. The program should continue to make an effort to identify and target those firms that handle multiple properties. Another approach to consider is utilizing regional property manager or landlord associations. Increase the coordination between National Grid marketing and the program manager and implementers. Program implementers have been cold-calling multifamily customers to 1-7

13 create awareness and interest in the EnergyWise Program. This practice is very time intensive. Consider selecting customer records based on metering so the program can target appropriate customers with detailed marketing information. Clearly define program benefits for customers. In order to address both economic barriers and the potential competition from other programs, National Grid will need to fully understand what other programs are offered to customers within their territory and position National Grid programs to offer the best product to customers. Benefits may be a simple application process, payment for their portion of the project costs only (eliminating the wait for a rebate check), knowledgeable contractors offering technical assistance, and more responsive program staff. As suggested by the new RISE marketing staff, a Frequently Asked Question (FAQ) document for customers would also be useful. Review program incentive levels for gas and weatherization measures. All program staff and implementation interviewees pointed to difficulties gaining interest from building owners and managers given what they feel are low incentive levels. The EnergyWise program has had no trouble meeting electric savings goals. That portion of the program is designed as a direct install service. However, the program pays 25 percent of gas and weatherization measure costs, which the program team feels is too low to overcome the long lead time for decision making and promote interest from a group historically unwilling to spend money given split incentives with tenants. Implementer experience with other multifamily programs, and comments implementers have received from customers, suggest that offering 40 to 50 percent would be much more successful in encouraging customer participation in the program. These measures may now be more attractive with the eligibility of master metered gas customers. There is interest from them in decreasing their gas bills, and the weatherization measures could be marketed as a way to achieve savings. Explore the possibility of offering bill payment or loan options for major measures similar to other programs. Program staff mentioned that other programs for commercial customers often provide the option of paying off the amount on their monthly bills. Some customers have told staff the reason they are not participating has to do with the lower proportion paid by the utility for the projects. Including a financing option would provide implementation staff with another sales tool to help overcome the initial cost barrier mentioned by some customers for major measures such as insulation that are not fully covered by the program. One building manager suggested that option during our interviews and at least onequarter of upstate Commercial and Industrial customers (participating and nonparticipating) interviewed through the upstate commercial and industrial process evaluation confirmed interest in financing offered by National Grid for energy efficiency upgrades. Another 25 percent were open to the option depending on factors such as management decisions, the size of the project, and the interest rate. Investigate how to better serve an entire property in a more comprehensive manner. In conjunction with more targeted marketing, program staff may want to consider incentivizing RISE to work more closely with multifamily customers in order to get them directed to the appropriate programs. Awareness of other programs and energy efficiency offerings appears to be a major gap for many of the multifamily building owners and managers. RISE is in the position to assist multifamily customers with a variety of energy efficiency needs by routing them to Residential High Efficiency Heating and Water Heating and Controls Program for heating and water heating measures, Enhanced Home Sealing for two to four unit properties, and even NYSERDA if needed for the more extensive renovations and retrofits that National Grid cannot cover. RISE and ICF should have very clear and distributable information on what other 1-8

14 programs are offered that multifamily customers can utilize. The ability to offer a more cross cutting service to multifamily managers would help alleviate some of the frustration and maximize savings potential. Review the tracking database inputs for accuracy and completeness. There were mislabeled measures in the program database download we received that were identified as we conducted interviews with building owners and managers. In addition, only one-third of the records had a correct phone number attached to the record. It would be worthwhile for tracking purposes and evaluation purposes to check a percentage of the database records against paper documents to ensure program information is accurate and recorded as expected by all parties using the data. For multifamily programs it is also important to capture the differences between local property managers and corporate decision makers to facilitate future communication and evaluation. Consider presenting a case to the PSC regarding the requirement that each measure pass the cost effectiveness test. Interviewees have found that it is very confusing to customers to go through the process of an audit, only to find that several of the recommended measures are not eligible for program incentives because they are not individually cost effective. This becomes even more of an issue when program staff deal with a property owner who has multiple buildings and a measure may be cost-effective at one building but not at others. Interviewed parties would like to see more of a whole-building or whole-project approach if the overall project level cost ratio is positive to increase participation and building-wide efficiency. 1-9

15 2. INTRODUCTION This report presents the results of the 2011 process evaluation of National Grid s EnergyWise Program in upstate New York, and includes feedback from participants and program activity between June 2010 and June This report is one of a series of EEPS process evaluation reports of National Grid s energy efficiency programs in New York. Preliminary results, from the initial in-depth interviews with program and implementation staff (including trade allies), were provided to National Grid in memo format in April The memo also included preliminary recommendations based on the early evaluation activities. 2.1 PROGRAM BACKGROUND The EnergyWise Program was initially approved in July 2009, with the implementation plan resubmitted with new fixtures approved in April Final approval was received in May 2010 and the program officially launched in June Original Program Design The EnergyWise program in upstate New York targets individually metered electric and/or gas multifamily buildings with five to 50 dwelling units. Eligible customers include National Grid multifamily customers of record that are building owners and property managers or individual dwelling customers on residential rates. Single family home owners, residing in dwellings of one to four units, are not eligible. In the original program design, multifamily buildings that are master metered for both heating and water heating were not eligible for the program. The parties involved in the design and delivery of the EnergyWise Program include the program manager at National Grid, managers at RISE Engineering (RISE) and ICF International (ICF), auditors at both RISE and ICF, and equipment installers from RISE or managed by RISE. The program is being promoted and delivered by implementation contractors, RISE Engineering, and ICF. Promotion also occurs through National Grid s website and other marketing activities. RISE works with architects, builders, engineers, remodeling contractors, technicians, retailers, equipment suppliers, and related channel intermediaries. Originally the program was designed to provide participants with a free energy use assessment and low-cost measures at no direct cost. These low-cost efficiency measures include CFLs, low flow showerheads, aerators, domestic hot water pipe wrap, and domestic hot water tank wrap. In addition, the program was to provide rebates of 25 percent for the installation of a number of electric and gas building envelope measures such as insulation, air sealing, attic ventilation, ductwork, and air infiltration testing (Table 2-1. Original EnergyWise Program RebatesTable 2-1). Lighting fixtures in common areas were offered to building owners for $20 per fixture. Lastly, the program offered $300 toward the cost of each new early replacement refrigerator and up to $75 for a normal replacement refrigerator. 3 A preliminary results memo was delivered to National Grid in April 2011 based on feedback from program and implementation staff. The draft final report was delivered to National Grid on March 23, Final comments on the draft report were received by the evaluation team in May 2012 from National Grid and August 2012 from DPS. 2-1

16 Measure Insulation (Attic, wall, basement/crawl space, rim joist, duct, heating system pipes) Air infiltration sealing for gas heated property where National Grid is the gas utility Refrigerators Table 2-1. Original EnergyWise Program Rebates 25% of measure cost 25% of measure cost Lighting fixtures Participant pays $20 Rebate $300 for each new early replacement and up to $75 for a normal replacement Compact fluorescent bulbs Low flow showerheads, Aerators Minimal domestic hot water pipe wrap Domestic hot water tank wrap 100% of measure cost 100% of measure cost 100% of measure cost 100% of measure cost Program Design Changes Since the program launch in June 2010, there have been a number of adjustments made to program design. While these adjustments were not included in the evaluation because they had not yet been made, these do warrant mention. Initially, only multifamily buildings that are individually metered for electric and/or gas service were eligible for the program. While National Grid was progressing well against their electric savings targets, they found that it would be nearly impossible to meet the original gas savings goals given the stock of individually metered gas units in Upstate New York. National Grid petitioned the PSC regarding the original forecasted goals and the likely potential of meeting these goals. In June 2011, the PSC approved the eligibility of master metered gas buildings for the program. Once the program was launched and the energy assessments were proceeding, National Grid determined that the insulation and air sealing efforts were generally not cost effective and decided to focus attention on the direct install measures (CFLs, showerheads, aerators, pipe wrap, and tank wrap). It has also been the case that virtually none of the refrigerators examined during the energy assessments were found to meet the guidelines for the refrigerator replacement rebate. In researching what other measures could be offered to replace the measures that were discontinued, National Grid found verbiage in the filing that allowed for measure additions if a similar program exists that is also rebating that measure. National Grid found another program rebating programmable thermostats which were cost effective, as well as boiler reset controls. Both of these measures were approved for the program in July As a result of the preliminary findings from the staff interviews and drafting of the logic model conducted as part of this process evaluation, program staff have also implemented quality assurance and quality control (QA/QC) inspections that were originally intended as part of the program but had not occurred early on. ICF is in charge of targeting 10 percent of the 2-2

17 participating properties to review what has been installed and to conduct a short survey with property managers. Program implementation staff are now trying to determine the most appropriate way to track the QA/QC information using the InDemand system. The most recently added benefit to building owners and managers from the program is the availability of EnergyWise Choice logo window decals that property managers can use for advertising participation in the program and upgrades to individual units Program Staffing and Operations Initial marketing for the program consisted of mass mailings to customers thought to be multifamily customers. The staff we interviewed felt the mass mailings were not effective in reaching the intended targets or customers who are eligible for the program. Therefore, National Grid has supplied lists of customers with meter type to the implementation contractors who have turned to one-on-one communications where they cold-call customers and talk with them about the program. While RISE and ICF have focused on personal communication with condominium owners, RISE also hired a marketing staff person who is responsible for targeted marketing and outreach efforts to eligible gas customers. Once customers are aware of the program, they can contact RISE for more information. RISE sends any leads to ICF who schedules and conducts the free energy audits. ICF forwards all resulting audit information to RISE to calculate the cost-effectiveness of each measure. Based on the data, RISE drafts a contract for the customer outlining what equipment is eligible for installation through the program. ICF then presents that contract to the customer, answers their questions, and obtains the customer s consent to continue. If the customer accepts the contract, RISE or their subcontractor installs the equipment. ICF staff will then conduct quality assurance checks on a portion of the installations. RISE requires that a certificate of completion be returned by the customer that reports what was installed, the location of installation, five survey questions, and signatures of the customer and program representative. RISE then invoices the customer for their portion of the project cost and National Grid for the remaining portion Program Objectives and Achievements To-date After a late launch in 2010 and some recent program adjustments, the program has gained momentum in reaching gas goals and has far exceeded its electric goals. The electric and gas savings goals for were set when the program was approved. These goals are outlined in the EnergyWise Program Revised Implementation Plan 4. At the end of 2010, the program achieved 92 percent of its electric goals. On the gas side, the program achieved three percent of its goals. 4 Niagara Mohawk Power Corporation d/b/a National Grid, Case 08-E-1133 and Case 09-G-0363, EnergyWise Program Revised Implementation Plan, June 18,

18 While the achieved gas savings were very low, it was not for lack of conducting energy use assessments, but rather due to a smaller than anticipated potential market. Four hundred thirty gas assessments were performed, with less than one-half (205) of these projects being eligible for the program. The remaining 225 projects were ineligible since they did not have either gas heating or gas domestic hot water heaters that were individually metered. As mentioned earlier, a few post-launch adjustments to the program, particularly the ability to serve master metered gas customers, resulted in a much more successful second year. Table EnergyWise Program Progress 5 Upstate Electric Upstate Gas Annual participation goals for (units) 3,150 2,325 Actual participation for (units) 1,566 0 Annual savings goals for ,620 MWh 235,940 therms Actual savings for ,490 MWh 7,078 therms Percent of savings goal in % 3% Annual participation goals for 2011 (units) 2,800 1,800 Actual participation for 2011 (units) Annual savings goals for ,303 MWh 158,760 therms Actual savings for ,916 MWh 134,946 therms Percent of savings goal in % 85% Program Logic Model A program logic model is a visual representation of the program s theory that illustrates a set of interrelated program activities that combine to produce a variety of outputs that lead to key short-, mid- and long-term outcomes. A program logic model can lead to a cost-effective determination of program effectiveness. Logic models can be linked to performance indicators in order to provide on-going feedback to program managers. The models flow top to bottom and are typically organized according to five basic categories: Program resources: financial, staffing, and infrastructure resources that support the activity Program activities: overarching activities that describe what the program is doing. Examples include marketing, rebate processing. Outputs: metrics resulting from the activities. These tend to be measurable bean counting results (e.g., provide outreach events at five community fairs) 5 Some figures were pulled from the 2010 Energy Efficiency Programs Annual Report, March 15, 2011 and others were provided by the EnergyWise program manager. The percent saved against goal for the year may be due to program adjustments that were not included in the timeframe evaluated. 2-4

19 Short- to medium-term outcomes: expected outcomes resulting from program activities, with goals attached to those outcomes when possible. Examples include target energy savings, recruitment into the program. Long-term outcomes: ideal, sustainable outcomes resulting from program activities, such as all eligible customers participate in program and increase customer awareness of program offerings. Stepping across the activities enumerated in the logic model indicates an approximate flow in the sequence of activities. For example, the logic model begins with the program infrastructure and ends with the activity which results in direct install and/or rebated measures. In each column, the resources needed are specified above each activity, and then the direct outputs of the activity appear. The outcomes are causally linked to the various outputs in each column of the logic model. In other words, it is expected that the specified output (e.g., audit recommendations) will result in the specified outcome (e.g., equipment installation). The logic model in Figure 2-1 was developed from program documentation and interviews with program and implementation staff, and has been reviewed by the program manager. 2-5

20 Figure 2-1. National Grid Upstate New York EnergyWise Original Program Logic Model Comment [PAH1]: Pagination at the bottom of this page needs adjusting. Inputs/ Resources Sufficient budget is allocated National Grid Staff Marketing materials and customer lists National Grid call center and web site (powerofaction.com), RISE, ICF National Grid program team (internal staff, implementers, etc) Audit forms National Grid program team (internal staff, implementers, etc) Signed customer agreement Customer Agreement Completion Certificate Activities Develop Program Infrastructure Outreach to Customers Perform Audits Install Measures Invoice Projects Outputs Energy Wise program implementation plan is developed and approved Target direct communications and other outreach such as newsletters, website to multifamily customers ICF performs audits for all interested customers Equipment is installed by either RISE or electrical contractor for common area lighting RISE validates customer applications and invoices customer for their portion of project cost Implementer identified and program is launched to National Grid customers in 2010 In-person communication with targeted customers by ICF and RISE RISE uses audit data to calculate cost effectiveness of measures Completion certificates are sent to RISE RISE uploads project data into InDemand and invoices National Grid monthly Program measures, rebates, marketing strategy and technical assumptions developed, refined and documented ICF presents customers with customized agreement as result of audit and cost effectiveness testing Post inspections are completed by ICF on 10% of projects (QA/QC) Tracking system developed and appropriate information is requested, captured and entered Short to medium term outcomes Implementation team is ready to assist customers through the program process Customers' awareness of and participation in the program increases Auditors identify areas where customers can save money by making energy-saving improvements Customers gain savings benefits and property enhancement from installing high-efficiency equipment Customer copay is handled accurately and efficiently. Program progress is tracked and reviewed for accuracy The program team is adjusting periodically to meet program needs Customers go to Power of Action website for energy efficiency information Customers sign agreement to proceed with equipment installation Customers experience an efficient and educational process and are encouraged to participate in other National Grid programs Long term outcomes Program passes total resource cost test Budget is secured The demand for energy-efficient equipment increases Customers understand areas of inefficiency and look for other opportunities to make improvements Ensure that incentivized equipment meets program requirements Increased penetration of energy efficient equipment among National Grid multifamily customers Energy saving goals of the program are achieved within budgetary constraints Achieve 2,923 MWh 394,700 Therms savings from

21 The first activity in the above program logic model is developing a program infrastructure. National Grid selected two vendors, RISE and ICF, to assist with the implementation of the program. Once the infrastructure was in place and the budget was secured, outreach to customers occurred. While National Grid provided widespread communication for building owners and managers in the form of newsletters and website information, RISE and ICF visited customers in person to generate project leads. The third program activity, as shown in the logic model, is performing the audits or energy assessments for the properties. During this phase, ICF conducts energy assessments and RISE calculates the cost effectiveness of measures. ICF can then present the customers with their customized agreements. After the building owner or manager accepts the agreement, RISE visits the property to install the equipment and deliver the completion certificates to tenants. ICF then targets approximately ten percent for post-installation inspections or QA/QC. The last phase of the program is for RISE to validate the customer application and invoice the customer for their portion of the project cost if applicable. RISE also enters the project information into InDemand and invoices National Grid monthly. 2.2 EVALUATION METHODOLOGY Several data collection activities occurred in order to address the key researchable issues identified in the evaluation planning stage. We talked with program staff and implementers, building owners and managers, and tenants who received energy efficient equipment. First we outline the key researchable issues and then describe the data collection activities employed to gather data in attempt to address the key researchable issues Key Researchable Issues The researchable issues identified for the process evaluation are in Table 2-3 below, and are organized around the main research topics. 2-7

22 Table 2-3. Key Researchable Issues Research Topic Customer Awareness and Marketing Program Administration, Processes and Resources Ease of Customer Participation Program Satisfaction Customer Characteristics and Decision Making Processes Program Performance Indicators Researchable Question How effective is the program marketing? What activities are most effective in providing program information? How does the EnergyWise program distinguish itself from other multifamily programs? How do participants most commonly hear about, and become involved in, the program? What marketing and outreach efforts are most successful in generating customer leads? Does the overlap with other internal and external programs help or hinder the success of these programs? How effective is the collaboration between National Grid and RISE? Do program manager and RISE feel they have sufficient staffing resources to deliver the program? What additional information or resources are needed? What are the characteristics of the participating landlord population and how does that compare to the eligible population? Are there any groups not reached by the program that also have financial and efficiency needs? What barriers exist for landlord s participation in the program? Is competition from other multifamily programs inhibiting participation in the National Grid program? How is the program working? How could it be improved? What enhancements are needed in the design and delivery of the program? Are customers satisfied with the program and the recommended or installed measures? What do they believe could be offered to improve program services? Which measures have been installed and what type of equipment did it replace? How are they accepted and valued by the customer? What additional measures would they be interested in? Do measures remain installed and, if not, why not? Did the technical assessment or free low-cost measures provide information which prompted important energy savings projects? How important was the technical assistance in their decision to participate? The program incentive? Does participation affect participants perception of the utility and, if so, how? Why do customers decide not to install measures after receiving technical assistance? Is the program delivering the intended benefits to participants and are they achieving planned energy impacts? Is the referrals process working effectively in identifying appropriate customers for other programs? Are all applicable/cost-effective electric and gas measures being offered to participants? Why aren t some being taken? Is the appropriate information being collected to support future evaluation activities (i.e., impact evaluation)? Are program goals set appropriately? Will the program be on target to reach its savings and spending goals? Why or why not? Data Collection a. Program and implementation staff interviews The first evaluation activity for the EnergyWise program consisted of in-depth interviews in March 2011 with program and implementation staff involved with the design and/or delivery of National Grid s EnergyWise Program in upstate New York State. 2-8

23 Tetra Tech conducted one interview with the upstate EnergyWise program manager and six interviews with implementation staff. National Grid provided a list of implementation staff with key responsibilities with the program in a variety of different roles. The six implementation staff interviewed were from RISE Engineering and ICF and included managers as well as auditors. Interviews covered a variety of topics depending on each interviewee s role within the program. Topics covered included 6 : Roles and responsibilities Program design and marketing Program operations Internal and external communication Customer and trade ally outreach and interaction Quality assurance Program database and tracking Barriers to program uptake and meeting goals and objectives Satisfaction with program Areas of interest to investigate during subsequent data collection. b. Participating Building Owner Interviews Overall, 89 unique records at the local level and 25 unique records at the decision-maker level were provided by National Grid for the purpose of interviewing participating and dropout building owners. Of these, we completed 14 interviews with property managers at the local level and four interviews with corporate-level decision-makers in August Initially, we attempted to reach the contact listed in the sample file, which turned out in most cases to be the local property manager for the property. Since the majority of local property managers were not involved in the decision to participate in the program, we attempted to obtain information about the decision-making process by identifying and following up with the corporate-level contact for the property. After attempting to reach the corporate-level contacts by phone and having little success (only two completed phone interviews), we tried to obtain addresses in order to send out a shortened questionnaire that contained key decisionmaking questions. This effort resulted in two additional completes. Throughout the calling process we experienced difficulty in determining the role the contact had in the decision-making process based on sample information. In most cases, the corporate-level contact was identified by either looking for contacts that spanned multiple properties or being redirected by the local property manager. In addition, approximately one-third of the records had 6 Tetra Tech conducted interviews between March 8, 2011 and March 25, Interviews ranged in duration from thirty to sixty minutes, depending on the interviewee s role. We do not identify interviewees by name, nor do we identify them by role when discussing specific findings as a means of protecting interviewees confidentiality. 2-9

24 the correct phone number attached to the record, which contributed to the difficulty in scheduling interviews. c. Participating Tenant Phone Surveys The last phase of evaluation activities consisted of CATI (computer assisted telephone interview) surveys in December 2011 with tenants served by the program. Files containing participating building owners and their tenants were provided by National Grid. A goal of 70 completes was met for the survey, which explored tenant characteristics, program awareness and satisfaction, measure persistence, and benefits of participation. The detailed response rate can be found in Appendix D. While conducting in-depth interviewers with property owners and managers, it became apparent that there were inconsistencies between the tenant level data and the measures that the property managers indicated had been installed. After a review of 16 out of the 253 applications, 11 were chosen at random. Three issues were identified that had to be resolved with before proceeding with sampling. These included a mismatch between total measure cost and the measure, duplicate tenant cases, and extra cases associated with the property owner. The first issue was resolved by compiling a list of measure pricing information and then comparing this with the measure listed for each tenant case. The second issue was corrected by removing cases with the same application and tenant information. No correction was found for the third problem. However, out of the 16 applications reviewed, this was only found in one application. In addition to these corrections, a confirmation of the measures tenants received was placed in the survey. During fielding, it was determined that the majority of the contact phone numbers were attached to cellphones. Due to the nature of cellphone sample the following actions were taken: a maximum of six attempts was set with only one contact per night allowed, no refusal conversion was attempted, and no bad number lookups were performed. 2.3 REPORT ORGANIZATION The next section of this report presents the results of the process evaluation. Section 4 discusses key conclusions and recommendations. The technical appendices contain the evaluation data collection instruments and detailed survey response rates. 2-10

25 3. DETAILED PROCESS EVALUATION FINDINGS This section presents the findings of the process evaluation of National Grid s EnergyWise Program in upstate New York. First, we highlight key findings from the process evaluation. These are followed by detailed findings in the following categories: Program administration and processes Customer awareness and marketing Ease of participation Customer characteristics and decision-making processes Program satisfaction. 3.1 SUMMARY OF KEY FINDINGS The following summarizes what we have identified as the most important and recurring themes based on all of the evaluation activities for the EnergyWise program Program Successes There is a good team working relationship. The staff report that the working relationship among National Grid, RISE Engineering, and ICF International is very supportive and collegial. Meetings are held regularly along with reporting of current progress and barriers. This leads to quick resolution of concerns and questions fielded by the staff. Initial indications are that the program team is united in addressing the challenge of identifying and engaging eligible gas customers to achieve the targeted gas savings for the program. Customer satisfaction is high. Several staff mentioned that program participants are satisfied with the program process. Feedback they have received from the completion certificates indicates that participants find the audit process easy and 90 to 95 percent are estimated to be satisfied with the program overall. This is supported through interviews with property owners and managers, as well as with tenants receiving the measures. The program is meeting goals for electricity savings. The interviewees reported that the electricity savings side of the EnergyWise program is performing well and has exceeded the current electricity savings goal. Staff also indicated that CFL replacement made up the bulk of electricity measures installed and that the rebate levels for these measures are competitive in the current service area. Competition from other programs for customers has been minor to date. Whereas other commercial program customers may be eligible for incentives from multiple sources, there is a mandated split in the multifamily market by size. National Grid can approach customers with five to 50 units per building while NYSERDA can approach those with more than 50 units. In addition, National Grid has been working collaboratively with all of the administrators delivering EEPS programs. The only concern will be possible confusion when the National Grid team approaches management firms that manage buildings in both categories, but that was not an issue with the management we spoke with. In fact, one large building manager feels the utilities are much better at serving customers with energy efficiency programs than NYSERDA and that there is little action from NYSERDA. 3-1

26 3. Detailed Process Evaluation Findings The Program Manager is receiving necessary data. The team meets weekly by conference call and the implementers provide frequent reports to the program manager. The program database kept by the implementer is separate from the InDemand tracking and more current. The program is appropriately staffed. The implementer closely monitors the level of activity and has adequate staff available for audits, installation, and data entry. RISE has also added a marketing person to support the marketing effort for EnergyWise, as it has proven to be more work than originally anticipated. Original issues with QA/QC checks have been resolved quickly. QA/QC was not occurring as intended when the program was initially rolled out. At first, RISE was checking a portion of their own installations, but ICF was responsible for QA/QC as a second set of eyes. Program staff reported there is no QA/QC requirement in the implementation plan and no associated budget for it. Interviewees also report that the focus on meeting the gas savings goal was distracting from the QA/QC part of the program process. National Grid has already clarified the QA/QC process for projects since the delivery of the preliminary staff interview results by confirming ICF s role as the QA/QC lead for the program and ICF has begun the process of selecting and verifying installations. Dropouts are not has high as expected. Dropouts are defined as those properties that had an energy assessment but had not yet installed energy efficient equipment. Most projects listed as dropouts are not actually dropping out of the program, but are rather still in the process of determining what they will install at locations not already served by the program. Therefore, these cases would be more accurately categorized as in-progress rather than dropouts. Participant satisfaction is high. Participants report a high level of satisfaction with the program experience and National Grid as a result of participation. Over half of the tenants (54 percent) reported that their experience with the EnergyWise Program increased their satisfaction with National Grid as their energy provider. The program is promoting energy efficiency upgrades. The group of building owners and managers we spoke with typically only implement upgrades when equipment fails or if there is turnover in a unit. The program is motivating them to replace equipment earlier than planned Challenges to Program Success Because of the newness of this program, there are several challenges identified through the evaluation activities for the first year of the program. The challenges mentioned in the interviews are highlighted below, with suggestions for addressing them in the Recommendation Section. The gas savings side of the Energy Wise program struggled early on. Staff indicated that the initial individual metering requirements drastically limited the pool of eligible gas customers. Several staff also indicated that the current gas incentive levels do not seem to be high enough to influence participation of the property owners, which has compounded the problem of the small eligible customer pool. Some staff have heard property owners requiring at least 50 percent of the project cost covered before they can justify the expenditure. 3-2

27 3. Detailed Process Evaluation Findings Refrigerator cost-effectiveness requirements are unrealistic for the current service area. Staff report customers interest in the refrigerator rebate has driven overall interest in the program. However, several staff stated that it is nearly impossible for customers to qualify for a refrigerator rebate if the unit is less than 17 years old (early replacement). Implementation staff reported that almost none of the tested refrigerators ended up costeffective based on the current calculation. Property managers confirmed that this has led to frustration and confusion on their part as many were originally interested in the program for the refrigerator rebates. There has been difficulty in selling the program to property owners. The interviewees reported that the current economic situation is causing property owners to overlook measures that benefit tenants, but do not directly benefit themselves. In addition, insulation and weatherization improvements are high-cost measures that require advance budgeting. Interviewees found that the 25 percent program contribution was not enough to motivate customers to undertake this advance budgeting. They also indicated that gaining contact information and access to out-of-state property owners has been a challenge, which evaluators confirmed to be the case when they attempted to contact these owners to complete an interview. Current marketing efforts have not been sufficient. Numerous members of the staff mentioned that more marketing materials are needed. They also indicated that the lack of marketing has been a factor in the current slow generation of energy audit leads. Mass marketing efforts have not reached the targeted customers and have created little activity on the program website. Cost-effectiveness screening at the measure level has impacted customer participation. Several staff mentioned that it is not uncommon for projects with an overall positive cost ratio on the project level to be passed over due to the requirement that each measure be cost-effective. They also reported that measure level screening is causing confusion with customers, as well as burdening property owners with tracking measures installed on a building-by-building basis. Staff feels that this leads to only a fraction of the measures a facility could receive being installed. Some property owners and managers confirmed that they did not understand why they were not eligible for the refrigerator or insulation rebates and why not all units are treated the same. The InDemand database has little support for report generation. Staff mentioned that the lack of a reporting feature in the InDemand database does not allow for an up-to-date picture of the current progress of the program. This has contributed to redundancy as other databases with reporting features are being used to track the same information. Program tracking data from InDemand is not 100 percent accurate. During the evaluation, we found that there were inconsistencies around how measures were recorded or pulled for the participating tenant interviews. For example, interviews with property managers revealed that they often had water saving measures installed while the database records we received showed only lighting measures. Review of some of the applications from RISE showed the full range of measures installed which did not match what we received. It is not clear if the issues arise from data entry errors or how the queries pull data from the InDemand system. There is a lack of understanding regarding who offers services and eligibility criteria. Building owners and managers find it very difficult to determine who provides programs to 3-3

28 3. Detailed Process Evaluation Findings assist the different types of multifamily properties with energy efficiency upgrades. There are varying eligibility criteria depending on the location of the property, whether buildings are individually or master metered, the type of equipment they are interested in replacing, and the number of units per building. Right now, many of the building owners and managers are not completing energy efficiency projects because they are not able to identify utility or other programs to assist them. Multiple visits to properties lead to building owner and manager frustration. Some building owners and managers have gone through multiple visits from the program to serve their properties for electric and then gas measures. Others received one visit for tenant units, then another for common areas. These respondents did not understand why the efforts could not be coordinated and that it is a waste of time to go through the properties twice. The inability to serve the entire property removes some of the benefits to building owners and managers. In particular, the restriction that National Grid can serve the five to 50 unit properties takes away the ability for many building owners and managers with multiunit properties to use the program and energy efficient equipment for marketing the units, especially if buildings in the complex do not fall within the five to 50 range Program Recommendations The following section summarizes the most important and recurring themes we identified from all the partied interviewed along with our recommendations to address these issues. Pursue the opportunity with regulators to expand to units and continue to target mid-sized property management firms that handle multiple properties. The implementation team feels they have been more successful in targeting some of the midsized property management firms and working with them to serve multiple facilities than targeting individual properties. The program should continue to make an effort to identify and target those firms that handle multiple properties. Another approach to consider is utilizing regional property manager or landlord associations. Increase the coordination between National Grid marketing and the program manager and implementers. Program implementers have been cold-calling multifamily customers to create awareness and interest in the EnergyWise Program. This is very time intensive. Look into pulling customer records based on metering so the program can target appropriate customers with detailed marketing information. Clearly define program benefits for customers. In order to address both economic barriers and the potential competition from other programs, National Grid will need to fully understand what other programs are offered to customers within their territory and position National Grid programs to offer the best product to customers. Benefits may be a simple application process, payment for their portion of the project costs only (eliminating the wait for a rebate check), knowledgeable contractors offering technical assistance, and more responsive program staff. As suggested by the new RISE marketing staff, a Frequently Asked Questions (FAQ) document for customers would also be useful. Review program incentive levels for gas and weatherization measures. All program staff and implementation interviewees pointed to difficulties gaining interest from building owners and managers given what they feel are low incentive levels. The EnergyWise program has had no trouble meeting electric savings goals. That portion of the program is designed as a 3-4

29 3. Detailed Process Evaluation Findings direct install service. However, the program pays 25 percent of gas and weatherization measure costs, which the program team feels is too low to overcome the long lead time for decision making and promote interest from a group historically unwilling to spend money, given split incentives with tenants. Implementer experience with other multifamily programs, and comments implementers have received from customers, suggest that offering 40 to 50 percent would be much more successful. These measures may now be more attractive with the eligibility of master metered gas customers. There is interest from them in decreasing their gas bills, and the weatherization measures could be marketed as a way to achieve savings. Explore the possibility of offering bill payment or loan options for major measures similar to other programs. Program staff mentioned that other programs for commercial customers often provide the option of paying off the amount on their monthly bills. Some customers informed staff the reason they are not participating has to do with the lower proportion paid by the utility for the projects. Including an additional financing option would provide implementation staff with another sales tool to help overcome the initial cost barrier mentioned by some customers for major measures such as insulation that are not fully covered by the program. One building manager suggested that option during our interviews and at least one-quarter of upstate Commercial and Industrial customer (participating and nonparticipating) interviewed through the upstate commercial and industrial process evaluation confirmed interest in financing offered by National Grid for energy efficiency upgrades. Another 25 percent were open to the option depending on factors such as management decisions, the size of the project, and the interest rate. Investigate how to better serve an entire property in a more comprehensive manner. In conjunction with more targeted marketing program staff may want to consider incentivizing RISE to work more closely with multifamily customers in order to get them directed to the appropriate programs. Awareness of other programs and energy efficiency offerings appears to be a major gap for many of the multifamily building owners and managers. RISE is in the position to assist multifamily customers with a variety of energy efficiency needs by routing them to the Residential High-Efficiency Heating and Water Heating and Controls Program for heating and water heating measures, Enhanced Home Sealing for two to four unit properties, and even NYSERDA if needed for the more extensive renovations and retrofits that National Grid would not cover. RISE and ICF should have very clear and distributable information on what other programs are offered that multifamily customers can utilize. The ability to offer a more cross cutting service to multifamily managers would help alleviate some of the frustration and maximize savings potential. Review the tracking database inputs for accuracy and completeness. There were mislabeled measures in the program database download we received that were identified as we conducted interviews with building owners and managers. In addition, only one-third of the records had a correct phone number attached to the record. It would be worthwhile for tracking purposes and evaluation purposes to check a percentage of the database records against paper documents to ensure program information is accurate and recorded as expected by all parties using the data. For multifamily programs it is also important to capture the differences between local property managers and corporate decision makers to facilitate future communication and evaluation. Consider presenting a case to the PSC regarding the requirement that each measure pass the cost effectiveness test. Interviewees have found that it is very confusing to customers to go through the process of an audit, only to find that several of the recommended 3-5

30 3. Detailed Process Evaluation Findings measures are not eligible for program incentives because they are not individually cost effective. This becomes even more of an issue when program staff deal with a property owner who has multiple buildings and a measure may be cost-effective at one building but not at others. Interviewed parties would like to see more of a whole-building or whole-project approach if the overall project level cost ratio is positive to increase participation and buildingwide efficiency. 3.2 PROGRAM ADMINISTRATION AND PROCESSES This section presents the detailed findings from the interviews and surveys for topics related to program team structure and collaboration, program tracking and information, training, and tools and resources for program delivery Program Team Structure and Collaboration The delivery responsibilities bounce back and forth between RISE and ICF, which could potentially be a cause for concern or confusion. There are multiple checks and balances throughout this process that could be prone to miscommunication or delay, but all feedback suggests that the process is actually working very smoothly and customers are being served in a timely manner and project work completed as expected. Once customers are aware of the program, they can contact RISE for more information. RISE sends any leads to ICF who schedules and conducts the free energy audits. ICF sends all resulting audit information to RISE to calculate cost-effectiveness of each measure. Based on the data, RISE drafts a contract for the customer outlining what equipment is eligible for installation through the program. ICF then presents that contract to the customer, answers any questions they may have, and obtains the customer s consent to continue. If the customer accepts the contract, RISE or their subcontractor installs the equipment. ICF staff will then conduct quality assurance checks on a portion of the installations. RISE requires that a certificate of completion be returned by the customer that reports what was installed, the location of installation, asks five survey questions, and obtains signatures of the customer and program representative. RISE then invoices the customer for their portion of the project cost and National Grid for the remaining portion. Additional assistance from within National Grid could decrease the amount of time spent locating eligible customers. While all delivery and implementation staff felt that they could manage the program workload with their current staffing levels, the only concern was the amount of time it was taking with cold-calling customers to find the ones eligible for the program and get them started with energy assessments. Implementers suggested having National Grid s marketing department provide more assistance to reach out to building owners and managers, as well as specific targeting of those with individually metered electric or gas buildings Program Tracking and Information Most of the project tracking information is detailed enough to provide progress against goals. Project information is recorded in the InDemand system for projects in process, but the implementers are keeping more detailed records on their own and tracking leads, which InDemand is not designed to do. 3-6

31 3. Detailed Process Evaluation Findings There may be some discrepancy in what is entered into the database or how data are pulled. As building owners and managers were called, we found that some of the measure information did not correspond with what was provided by the owners and managers when we talked with them. In many cases, the water saving devices were not accurately reflected in the sample file. It was also difficult to determine from the contact information if the contacts listed were the decision-maker or local contacts. Program tracking and communications, either initial or follow-up, as well as evaluation efforts may be more effective if the contacts are more clearly tracked. For instance, two separate contact fields (with telephone numbers) could be included that identify the contact as a local property manager with whom the program may be communicating to schedule the energy assessment and installation of measure, and identify who is the person responsible for deciding if the property will participate and to whom the energy assessment report should be sent Quality Control At the time of the program staff and implementation interviews, quality assurance and quality control (QA/QC) procedures had not yet started, but a QA/QC process is now in place and functioning. We identified early in the evaluation as we worked through the program logic model with the program staff and implementers that the one phase of the program that had not fully launched was the QA/QC task. The program structure and contracts accounted for it; however, it had not actually been implemented for any sites. It was initially speculated that the push to reach gas savings targets was distracting resources. Early after program launch the installation contractor checked a portion of their own installations, which was not the intent of the program structure. ICF, who is responsible for conducting the energy assessments, was also tasked with QA/QC, which included going back and visiting ten percent of the properties to verify installation, look for any health or safety issues, and review a series of questions with the property manager. This procedure is now being followed. 3.3 CUSTOMER AWARENESS AND MARKETING This section provides findings on the effectiveness of program marketing by examining program awareness levels and sources of program awareness Program Awareness Outreach is still a major challenge for the program. Corporate property managers and owners were typically cold-called by RISE Engineering, although a few may have learned about the program from the newspaper or on the National Grid website. During interviews with program staff and implementers, it was mentioned that the most effective way of reaching customers that resulted in actions had been through cold calls. However, this is a very time intensive method for reaching prospective participants. Interviews with property managers and reviews of QA/QC forms confirmed that the vast majority of local property managers did not hear about the program until their corporate office notified them of the decision to participate. One local property manager submitted a proposal to corporate suggesting they participate in the program. If more information can be 3-7

32 3. Detailed Process Evaluation Findings disseminated to local property managers, they may be able to be of assistance in motivating corporate managers to choose to do work through the program. One landlord, who is very active in his local landlord association, recommended more targeting of landlord associations, either through presentations to the association groups or articles in the association newsletters as a way to reach a larger audience. Two of the property managers with multiple properties relied heavily on information from manufacturers, suppliers, and vendors for new equipment and energy efficient options. Building owners and managers are typically looking for communications regarding costs, potential savings, and deadlines. There is a general lack of awareness of where to go for any energy efficiency assistance. Building owners and managers find it difficult to determine who provides programs to assist their different types of multifamily properties with energy efficiency upgrades. There are varying eligibility criteria depending on the location of the property, whether buildings are individually or master metered, the type of equipment they are interested in replacing, and the number of units per building. Property managers with multiple properties first have to determine who can serve each property and research what utilities may have available for the particular type of equipment they are interested in replacing or installing. Current multifamily programs, including National Grid s EnergyWise program, tend toward more straightforward direct install items. This still leaves the property managers looking into options for installing larger equipment or building envelope measures. Right now, many of the building owners and managers are not completing energy efficiency projects because they are not able to identify utility or other programs to assist them. The allocation of five to 50 unit buildings to National Grid further complicates the move toward efficiency for building owners and managers that have varying building types per complex. NYSERDA programs are available for buildings with more than 50 units. Those building owners and managers who are aware that NYSERDA offers energy efficiency programs for buildings over 50 units often prefer the utility-run programs over NYSERDA s, citing that the utilities seem to be better at getting the work done. Tenants have little knowledge of the program until notified by their property managers. Seventy-five percent of tenants learned about the EnergyWise program through their property manager or management company. A variety of methods were used by property managers to alert tenants about what would happen as part of their participation in the program, including letters, fliers taped to doors, and phone calls. The amount of detail and number of notices varied depending on how long property managers had to notify their tenants. Of those who received information from their property manager or management company, 60 percent said they received the time and date of installation, 29 percent received information about the equipment that would be installed, and 13 percent received an explanation of the EnergyWise program. After information from property managers, another 13 percent of tenants learned about the program through RISE Engineering, likely during the installation visit. A few others learned of the program from sources such as literature or websites. 3-8

33 3. Detailed Process Evaluation Findings Seventy-six percent of the tenants receiving information indicated that it was sufficient to determine the benefits, to them, from participating in the EnergyWise program. One-third of tenants felt they had a choice of whether or not to participate Program Understanding Considering that tenants are not the first point of contact with the program, a respectable proportion understands the purpose of the program and why National Grid offers it. At the beginning of the survey, the 73 respondents were asked about their understanding of the EnergyWise program. Forty percent of respondents indicated that they thought the purpose of the program was to save energy or be more energy efficient and another 21 percent felt the program would save them money on their energy bills. Twenty-five percent knew that someone was coming to install energy efficient equipment. However, one-quarter of the tenants surveyed had no knowledge of the program before the survey call. Tenants were also asked why they believed National Grid offers the EnergyWise program to customers. Forty-five percent believe the utility offers the EnergyWise program so they can lower demand, save energy, and save money. Almost 30 percent felt National Grid was trying to help customers and save them money. Less common reasons mentioned included improving the environment (17 percent), to improve their reputation (12 percent), and because National Grid received government funds (six percent). Sixteen percent were not able to provide a reason. It does not appear that the Completion Certificates are having the intended impact. To increase tenant knowledge of what was done in their unit, given that the decision to participate is most often made by the property owner or manager, the program leaves a completion certificate with each unit served by the program. A template for the completion certificate can be found in Appendix E. The main components of the Completion Certificate are: Client name, location, contact information Installation date Measures installed, location, and quantity Five-question survey Check box to approve use of electricity consumption data to evaluate energy savings. Just over half (52 percent) of all tenants recall receiving the completion certificate and another 27 percent could not recall if they had received it. Twenty-one percent confirmed they did not receive it. For the half of tenants who remember receiving their completion certificates, 42 percent do not recall what information was included on the certificate. The rest remembered at least a few items, with over half recalling the list of equipment that was installed in their unit. Others remembered seeing the date of installation and other information about the program. 3-9

34 3. Detailed Process Evaluation Findings Figure 3-1. Recall of Information on Completion Form for Those Who Recalled Form Many of the comments from tenants show that they paid little attention to the information they received on the completion form and after installation. I just looked at it fast and signed it. I threw it out, I don t know, I only glanced at it. I can t remember. 3.4 EASE OF PARTICIPATION Building owners and managers find the EnergyWise program easy to use. RISE Engineering was mentioned favorably during interviews with building owners and managers who feel they are providing high-quality service on a reasonable schedule and are flexible to tenant needs. RISE did a test run for one of the large management companies at one property. That worked well so they rolled it out to other properties RISE made it understandable for everybody, we knew exactly what we were going to get and their breakdowns were very comprehensive. RISE had a payback worksheet that showed what was an honest and realistic guesstimate of what we were looking at for savings. One property manager specifically mentioned appreciating that National Grid had contractors lined up to do the work through the program and they did not have to spend the time identifying contractors on their own. 3-10

35 3. Detailed Process Evaluation Findings Tenants found the program accommodating. Ninety-four percent of all tenants indicated they had no difficulties participating in the program and the rest did not know about the program. A few mentioned they liked that the program installed the equipment while they were at work or that they accommodated their odd schedule. As can be seen in Table 3-1, the majority of surveyed tenants (60 percent) indicated they did not have a choice to participate in the program. In addition, about one third of respondents (34 percent) indicated that they were able to choose whether or not measures were installed in their unit. Table 3-1. Tenant s Reasons for Participation Percent responding I had no choice, the management company decided to participate 60% Interested in lowering utility costs 23% The right thing to do for the environment 5% It was free 4% Other 4% Don't know 4% It took no effort 3% Total N=73 In general, 34 percent said they benefitted by saving money on their energy bill and 12 percent said they saved energy. Thirty-five percent specifically saw a decrease in their electric bills along with 20 percent who saw a reduction in their gas bills. Another 13 percent said the new equipment was an improvement over what was previously installed. Thirty-four percent said there was no particular benefit they could name, while 17 percent said they did not know of any benefits they had realized. Table 3-2. Benefits of Participation Percent responding No benefits 34% Saved money on energy bills 34% Don t know 17% The equipment installed is an improvement over the old equipment 13% I saved energy 12% Other 3% I learned how to change my energy use behavior 1% Total N= Barriers to Participation The current economic situation has made the EnergyWise Program even more important to multifamily properties. Program staff and implementation contractors reported that the current economic situation is causing property owners to overlook measures that benefit tenants, but do not directly benefit themselves. In addition, insulation and weatherization improvements are high-cost measures that require advance budgeting. Program staff and implementer interviewees have found that the 25 percent program contribution is not enough to motivate 3-11

36 3. Detailed Process Evaluation Findings customers to undertake this advance budgeting. They also indicate that gaining contact information and access to out-of-state property owners has been a challenge. The cost to participate was also a factor that most property owners and corporate managers closely considered. Multifamily properties in general operate on a very tight margin, which makes these programs very attractive. One property manager mentioned that their property was 80 percent low income, and therefore, the property contribution to the program was even more of an issue. In addition to costs, a few property managers also had to take into account the age of their properties and how that affected the upgrades they were able to make. Although property managers typically do not have much funding to work with, which can be a major barrier to installing any energy efficiency improvements, they benefit from the energy assessment to determine property needs which allows them to prioritize work. Building owners and managers experience difficulty in participation resulting from the logistics of notifying and disrupting residents. For many of the property owners and managers, primary considerations before program participation are the amount of their time that will be required to work with program staff, notify tenants and respond to their questions, the unfamiliar contractors who will be on-site performing the work, and the inconvenience to tenants. Tracking multiple programs and the number of units served is too much work for some property managers. The more properties a manager handles, the more tracking is needed. A further complication is when the manager oversees properties with a variety of buildings or a wide variety of properties. With the differences in programs offered and service providers by territory, tracking energy efficiency assistance is not a minor task. While property owners and managers had positive experiences with the services offered by the EnergyWise program, the limitation of serving five to 50 units created issues for property managers whose complexes had buildings that were eligible as well as buildings that were not. Some of the tenants benefited and at the same time my other tenants didn't, seemed like an overly rigid criteria. They told me that the buildings weren't eligible because of number of units. For the buildings that didn't qualify for the National Grid program, I wasn't informed about other programs to get those buildings covered. Tenants talk amongst themselves and I got a lot of questions about why so-and-so got them and they didn't. It put me in an awkward position of explaining why their building didn't qualify for program. Multiple trips to properties created frustration for some property managers. Through interviews we found that it was common for the implementers to make several visits to a property. Sometimes it would be one pass through for CFLs and then a later round to install gas saving measures. Other times it was one visit to install measures in the tenant units, then another to handle improvements to common areas and management offices. It seems like a big waste of time for three RISE guys to go through the properties and only install CFLs. I really wanted to have some gas measures installed for savings. Some of the property managers are familiar with rebates and services offered by NYSERDA, but prefer the services offered by the National Grid EnergyWise Program. 3-12

37 3. Detailed Process Evaluation Findings Property managers are always actively looking for programs that will provide assistance, either technical or financial, for installing new equipment. A few have had experience with other utility programs and services through NYSERDA, but several prefer the services that their utilities, particularly National Grid, can offer. One property manager was in the process of working with NYSERDA partners for the Multifamily Performance Portfolio (MPP) program. Two other comments are highlighted below. I heard about program from RISE Engineering, they seemed to be more competent than the NYSERDA and Energetic people who approached us individually. The others didn't have the whole package put together. We had talked with someone from NYSERDA about assistance, but decided to pass due to the financing requirements. My dealings with NYSERDA are not good. I would prefer to see all of the utility companies handling that themselves. It seems like you guys have the ability to get it done, while NYSERDA has been sitting on their hands for several years, which is why you're getting your money back. 3.5 DECISION-MAKING PROCESSES Both the building owners and managers and the building tenants made decisions about installing energy efficient equipment through the program. However, the tenant decisions were restricted to allowing the implementation staff in and accepting program equipment once the building owner or manager had made the call to participate at the property Equipment Installation The majority of energy efficient equipment installed through the program in 2010 consisted of CFLs. Ninety-six percent of tenants said they had CFLs installed in their units through the program. Just over half also had low flow showerheads installed. Few tenants had faucet aerators (13 percent), hot water pipe wrap (eight percent), or tank wrap (seven percent) installed. Only five percent thought they had a programmable thermostat installed through the program, which is likely due to the recent addition of the measure to the program offerings. Installation persistence was high for most equipment. The proportion of program-installed equipment that was still installed at the time of the tenant survey ranged from 100 percent for tank wrap to 83 percent for CFLs. Table 3-3. Installation Persistence Percent with All Equipment Still Installed Tank Wrap 100% Low flow showerheads 98% Faucet aerators 87% CFLs 84% Hot water pipe wrap 82% 3-13

38 3. Detailed Process Evaluation Findings The reasons for equipment removal were usually failure or dislike. It was mentioned by tenants that CFLs frequently failed and building owners and managers confirmed that issue. Of the 17 percent of tenants who removed CFLs, 58 percent cited the CFLs burning out as the reason for their removal and they typically only removed one bulb. Ninety-five percent of the CFLs still installed are working properly. A high percentage (84-93 percent) of the other equipment still installed is reported as working properly Program Participation The program was extremely influential in motivating building owners and managers to install energy efficient equipment. In some cases, building owners and managers had CFL, low flow showerhead, and faucet aerator improvements on their lists to do and would have done them as units turned over. However, none of them would have done these full scale changes at the time they did without the program. We ve been switching to T8 and CFLs, low flow showerheads and faucet aerators as units turn over. We change out water heating as we need to work on them. This program definitely helped us get work done sooner. I ve been trying to drive down operating costs of the building and business with the economy so bad, so this fit with what I was trying to do anyway. We had not considered any improvements and would have done nothing without the program. Building owners and managers also benefitted from the energy assessment and learned what improvements to prioritize within their buildings. Building owners and managers are happy with the direct install equipment, but many were initially interested in the program for the refrigerator rebates. Although satisfaction with the program is high overall, there are some concerns from building owners and managers that the program has not offered all that it said it would. Many of them were initially drawn to the EnergyWise program by the possibility of replacing the old, inefficient refrigerators located in many of the units. Refrigerator standards were not realistic. We had old not frost free 15+ year old refrigerators that did not meet the standard for replacement with your program. The program is also motivating individual tenants to become more energy efficient. Fifty-nine percent of tenants said they would not have installed any of the energy-saving equipment without the program. Seven percent of tenants have purchased a similar piece of energy efficient equipment for their unit since participating in the EnergyWise program, none of which received a rebate. Two tenants purchased more light bulbs and one purchased another showerhead with a low flow feature or rinse mode. There is very little spillover associated with the EnergyWise Program to date for building owners and managers. A couple of the building owners and managers interviewed had looked into energy efficiency improvements outside of the EnergyWise Program. One mentioned air sealing and another was looking into furnace and boiler replacement, but only when it was needed since he has not been able to find a program to cover it. 3-14

39 3. Detailed Process Evaluation Findings 3.6 PROGRAM SATISFACTION Overall Satisfaction Overall, building owners and managers are happy with the EnergyWise Program. For most program aspects, at least eight of the building owners and managers were able to provide ratings on a scale of zero to ten, where zero was not at all satisfied and ten was extremely satisfied. All aspects were rated higher than 7.5. A few individual property managers had minor issues with the auditor and audit report, the installer, and the equipment performance. Tenant satisfaction with the program overall is also high. All tenants were asked to rate their overall satisfaction with service provided by National Grid and various aspects of the EnergyWise Program on a scale of zero to ten, where zero was not at all satisfied and ten was extremely satisfied. Seventy-nine percent were satisfied with National Grid s service overall (rating of eight to ten). All ratings were above a five, except for four tenants who gave National Grid s service a zero rating (very dissatisfied). Two of these respondents mentioned that National Grid's phone service was the reason for their dissatisfaction with National Grid, while the other two respondents gave the prices they pay for service as the reason for their dissatisfaction. Figure 3-2. Tenant Overall Satisfaction with the EnergyWise Program Satisfaction with Specific Program Aspects Specific aspects of the programs were also highly rated by tenants. The 45 percent of tenants who were at home during the installation of the energy-saving equipment were asked to rate their level of satisfaction with the installation process, interaction with program staff, and the time it took for program staff to install the equipment on the same scale of zero to ten, where 3-15

40 3. Detailed Process Evaluation Findings zero was not at all satisfied and ten was extremely satisfied. All three of these program aspects received very high ratings, showing that tenants who interacted with the program had very good experiences. The aspects with the lowest mean rating were information related. Consistent with the fact that many of the tenants were unaware that their unit was being served by the program, information about the program and new equipment received the lowest satisfaction ratings of all aspects rated. Program Aspect Table 3-4. Tenant Satisfaction with Program Aspects Number of applicable respondents Mean Percent Satisfied (8-10 rating) Percent Extremely Satisfied (10 rating) Time it took to install equipment % 74% Installation process % 70% Interactions with program staff % 74% Type of equipment % 49% Performance of equipment % 49% Information provided about new equipment % 40% Information provided about program % 36% Program participation increases tenant satisfaction with National Grid. Satisfaction with National Grid as an energy provider increased for 54 percent of the tenants as a result of participation in the EnergyWise Program. For another 45 percent of tenants, the program participation made no difference in their satisfaction. Figure 3-3. Tenant Change in Satisfaction as a Result of the EnergyWise Program 3-16