Residential Energy Assistance: Effectiveness of Demonstration
Program as Yet Undetermined (17-AUG-01, GAO-01-723).
Paying for home heating and cooling can be difficult for
low-income families. With rising prices for natural gas,
electricity, and other fuels meeting utility bills has become
even harder for low-income families. By the end of the fiscal
year 2000, the Office of Community Services had awarded a $30
million in Residential Energy Assistance Challenge Option (REACH)
program grants to 24 states and 12 tribal organizations to fund
54 separate projects intended to help meet the home energy
(heating and cooling) needs of low-income households in a variety
of ways. These grants ranged in amount from $50,000 to $1.6
million. Most of the 54 REACH projects have educated low-income
clients about home energy efficiency through group workshops or
in individual homes. Many REACH projects have involved
energy-related repairs to home and budget counseling, and three
state REACH projects are developing consumer cooperatives to
purchase electricity or bulk fuels, such as heating oil. However,
some REACH projects have included social services not directly
related to meeting home energy needs. The legislation authorizing
REACH identifies three performance goals to be included in
individual REACH projects. The three goals are to (1) reduce
energy costs of participating households, (2) increase the
regularity of home energy bill payments, and (3) increase energy
suppliers' contribution to reduce eligible households' energy
burden. Of the six project evaluations completed by states as of
May 2001, only one project's design and implementation allowed
statistically valid conclusions to be made about the effect of
project services on participant energy use. The five other state
project evaluations had analytical problems and other
shortcomings that limited their usefulness in assessing project
results. The Department of Health and Human Services has not yet
developed a comprehensive plan for how it can best communicate
summary information on best practices and lessons learned from
the REACH program.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-01-723
ACCNO: A01618
TITLE: Residential Energy Assistance: Effectiveness of
Demonstration Program as Yet Undetermined
DATE: 08/17/2001
SUBJECT: Energy
Energy costs
Disadvantaged persons
Federal/state relations
Energy conservation
State-administered programs
Grants to states
Public assistance programs
HHS Low-Income Home Energy Assistance
Program
OCS Residential Energy Assistance
Challenge Option
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-01-723
Report to Congressional Committees
United States General Accounting Office
GAO
August 2001 RESIDENTIAL ENERGY ASSISTANCE
Effectiveness of Demonstration Program as Yet Undetermined
GAO- 01- 723
Page i GAO- 01- 723 REACH Program Effectiveness Letter 1
Appendix I REACH Project Activities 29
Appendix II Review of California?s REACH Project Evaluation Report 30
Appendix III Review of Maryland?s REACH Project Evaluation Report 34
Appendix IV Review of Massachusetts? REACH Project Evaluation Report 39
Appendix V Review of Michigan?s REACH Project Evaluation Report 44
Appendix VI Review of Nebraska?s REACH Project Evaluation Report 49
Appendix VII Review of Oregon?s REACH Project Evaluation Report 53
Appendix VIII Comments From the Department of Health and Human Services 59
Appendix IX GAO Contact and Staff Acknowledgments 72 Contents
Page ii GAO- 01- 723 REACH Program Effectiveness Tables
Table 1: REACH Grant Funding by Fiscal Year and Type of Grant 8 Table 2:
Grant Awards to States, by Fiscal Year 8 Table 3: Grant Awards to Tribes,
Tribal Organizations, and Insular
Areas, by Fiscal Year 9 Table 4: Activities Planned by REACH Grantees 29
Abbreviations
DOE Department of Energy HHS Department of Health and Human Services LIHEAP
Low- Income Home Energy Assistance Program OCS Office of Community Services
PRISM Princeton Scorekeeping Method REACH Residential Energy Assistance
Challenge Option
Page 1 GAO- 01- 723 REACH Program Effectiveness
August 17, 2001 The Honorable Edward M. Kennedy, Chairman The Honorable Judd
Gregg, Ranking Minority Member Committee on Health, Education, Labor and
Pensions United States Senate
The Honorable John A. Boehner, Chairman The Honorable George Miller, Senior
Democratic Member Committee on Education and Workforce House of
Representatives
Paying for home heating and cooling can be difficult for low- income
families. With this year?s rising prices for natural gas, electricity, and
other fuels, meeting utility bills has become even harder for the nation?s
neediest citizens. The federal government has spent about $1.8 billion in
each of the past 2 years to help low- income households pay for home heating
and cooling through the Low- Income Home Energy Assistance Program (LIHEAP).
Within this larger program, the Congress established the Residential Energy
Assistance Challenge Option (REACH) program, which provides grants that fund
demonstration projects to test various approaches to help low- income
families reduce their energy usage and become more self- sufficient in
meeting their home energy needs. In a sense, the REACH program serves as a
?laboratory? for identifying better ways to ensure that low- income families
can afford home heating and cooling.
Through the REACH program, the Office of Community Services (OCS)- part of
the Administration for Children and Families in the Department of Health and
Human Services (HHS)- provides grants to states and tribal organizations to
conduct demonstration projects. The states must conduct these projects
through community- based organizations; tribal organizations may conduct
projects themselves or through other community- based organizations. OCS has
funded 54 REACH grants since the program began in fiscal year 1996, using an
annual budget of about $6 million, which is about one- half of 1 percent, on
average, of the total funding for LIHEAP. REACH projects have provided
weatherization materials, workshops on energy efficiency measures for homes,
and budget counseling, and have formed consumer cooperatives to purchase
home energy. State projects run for 3 years, and state grantees are required
to contract for third- party evaluations and to report after the
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 01- 723 REACH Program Effectiveness
conclusion of the projects on the effectiveness of the approaches that they
have tried.
In the Low- Income Home Energy Assistance Amendments of 1998, the Congress
required that we evaluate the REACH program and include in our report
information on states? evaluations of their projects. (Tribal projects are
not required to do evaluations). As agreed with your staff, our report
addresses (1) grant recipients, amounts, and project activities; (2) the
goals of the REACH program; (3) the methodologies and results of states?
project evaluations; and (4) plans to communicate lessons learned from REACH
projects and to foster the further use of successful approaches.
By the end of fiscal year 2000, OCS had awarded $30 million in REACH grants
to 24 states and 12 tribal organizations to fund 54 separate projects
intended to help meet the home energy (heating and cooling) needs of low-
income households in a variety of ways. These grants ranged in amount from
$50, 000 to $1.6 million. Most of the 54 REACH projects have educated low-
income clients about home energy efficiency through group workshops or in
individual homes. Many REACH projects have involved energy- related repairs
to homes and budget counseling, and 3 state REACH projects are developing
consumer cooperatives to purchase electricity or bulk fuels, such as heating
oil. However, some REACH projects have included social services not directly
related to meeting home energy needs. For example, six projects included job
skill or employment development services, and one project provided funds to
help clients pay past- due rent or mortgage payments. One of the REACH
program?s criteria for grant selection- which is used to assess whether
grant proposals address the home energy needs of low- income families within
a ?holistic?
context of economic, social, and other barriers to self- sufficiency- may
have been misconstrued to allow projects to spend REACH funds on nonenergy-
related activities.
The legislation authorizing REACH identifies three performance goals to be
included in individual REACH projects: reduce energy costs of participating
households, increase the regularity of home energy bill payments, and
increase energy suppliers? contributions to reduce eligible households?
energy burden. The legislation provides broad purposes for the program, but
HHS has not developed performance goals that define the results that it
expects to achieve. Without program performance goals that are objective and
quantifiable, HHS cannot effectively evaluate the program?s overall
performance and report to the Congress on what was Results in Brief
Page 3 GAO- 01- 723 REACH Program Effectiveness
accomplished for the resources expended. Such goals would also provide a
clearer basis for selecting individual projects to fund.
Of the six project evaluations completed by states as of May 2001, only one
project?s design and implementation allowed statistically valid conclusions
to be made about the effect of project services on participant energy use.
This project focused on changing the behavior of low- income households
through teaching energy- efficient practices, and its evaluation report
indicated a statistically significant decrease in natural gas use by REACH
project participants compared with those in a control group. The five other
state project evaluations had analytical problems and other shortcomings
that limited their usefulness in assessing project results. Some evaluations
did not collect sufficient data on energy use by project participants, while
others did not adjust energy use data to account for variations in the
weather. Because of these and other shortcomings, the evaluation reports
could not be used to determine whether each REACH project met its project-
level performance goals. In addition, none of the six reports evaluated
whether the project met one of the project performance goals stated in the
authorizing legislation- increasing energy suppliers? contributions to
reduce the energy burdens of eligible households. None of the project
evaluation reports attempted to compare the costs of the services provided
through REACH with the costs of providing energy assistance payments alone,
yet REACH program guidance states that a cost- effectiveness analysis should
be an objective of each state project. As additional state projects complete
their 3- year terms, HHS will receive additional evaluation reports. Program
officials have been taking steps to improve future projects, including
developing guidance and hiring a consulting firm to advise grantees on
better project design and evaluation planning.
HHS has not yet developed a comprehensive plan for how it can best
communicate summary information on best practices and lessons learned from
the REACH program. Such information will be important in identifying- and
fostering the further use of- any methods of meeting the heating and cooling
needs of low- income families that are more effective than current methods.
As part of its communications efforts, the REACH program is designing a Web
site about REACH program requirements and projects. In addition, REACH
program officials hold annual conferences at which grantees report on their
projects? results.
We are making recommendations to HHS to address the lack of program
performance goals for REACH and to ensure that the results of demonstration
projects are communicated to organizations involved in
Page 4 GAO- 01- 723 REACH Program Effectiveness
assisting low- income households with their home energy needs. Given the
limited information available at this time, the Congress may want to
consider requiring HHS to report on REACH project results and program
performance after the projects funded in the first 3 years of the program
(fiscal years 1996, 1997, and 1998) have completed their evaluations, which
will probably occur late in 2002. At that time, the Congress may also wish
to consider whether the REACH program should continue indefinitely or
whether the program should have an end date after the completion of a
sufficient number of demonstration projects.
The Congress established the REACH program to
minimize the health and safety risks that result from high energy burdens
on low- income Americans,
prevent homelessness as a result of inability to pay energy bills,
increase the efficiency of energy usage by low- income families, and
target energy assistance to individuals who are most in need. The REACH
legislation requires that project plans provide a variety of services and
benefits, which may include energy efficiency education, residential energy
demand management services, counseling related to energy budget management
and payment plans, and negotiations with home energy suppliers on behalf of
eligible households. The legislation further requires each state?s plan to
describe performance goals for its projects and the indicators the state
will use to measure whether each project has achieved its performance goals.
OCS? stated purpose for the REACH projects is to demonstrate the longterm
cost- effectiveness of supplementing energy assistance payments with
nonmonetary benefits that can increase the ability of eligible households to
meet energy costs and achieve energy self- sufficiency. REACH is part of a
much larger program, LIHEAP, which primarily provides financial assistance
to low- income households for home heating and cooling. REACH projects may
target their services to a portion of the population eligible for LIHEAP,
such as a geographic area or type of client (households with elderly people,
for example).
Since fiscal year 1996, REACH funding has ranged from $5.5 million to $6.8
million annually, while the total funding for LIHEAP has ranged from $1.2
billion to $2.1 billion. Over the course of the REACH program, its funding
has averaged about one- half of 1 percent of total LIHEAP funds. The
legislation establishing REACH provided for funding it from LIHEAP?s
Background
Page 5 GAO- 01- 723 REACH Program Effectiveness
incentive program for leveraging nonfederal resources. This incentive
program provides additional monies to states that, in the previous year,
obtained additional assistance for low- income households? energy needs from
such sources as state funds, utility companies, and private charities. The
REACH legislation provides that, for each fiscal year, the Secretary of HHS
may allocate up to 25 percent of the funding for this incentive program to
the REACH program, and HHS has allocated approximately this amount each year
since the REACH program was first funded in fiscal year 1996.
Through the REACH program, OCS awards grants to states 1 and to tribes,
tribal organizations, and insular areas (generally referred to in this
report as ?tribal organizations?). OCS issues annual requests for grant
proposals and then awards grants on a competitive basis. On average, about
one- third of the applicants from fiscal year 1996 through fiscal year 2000
received a REACH grant. OCS? Division of Community Demonstration Programs
manages the REACH program.
State and tribal organization grants differ in a few ways. OCS has
established a 3- year time frame for states to complete their projects, in
part to allow time for states to contract with the community- based
organizations that carry out the states? REACH projects. Tribal projects
have a 17- month time frame if the tribal organization carries out the
project itself, or a 3- year time frame if the tribal organization chooses
to use a community- based organization. Unlike states, tribal organizations
are not required to conduct evaluations on the effectiveness of their
project approaches, but tribal organizations do submit final reports on
their projects to OCS. In addition, OCS has established differing maximum
grant amounts for state and tribal grants. According to the director of OCS?
Division of Community Demonstration Programs, these caps on grant amounts
reflect the differing amounts that states and tribal organizations receive
in their LIHEAP allotments and resulted from focus meetings with tribal
representatives held by OCS? LIHEAP staff prior to the first REACH request
for grant proposals.
OCS? annual requests for grant proposals, referred to as program
announcements, provide criteria for program eligibility, proposals,
1 The District of Columbia and Puerto Rico may apply for REACH grants under
OCS? provisions for state grants.
Page 6 GAO- 01- 723 REACH Program Effectiveness
selection of projects to fund, and states? evaluations of their projects. 2
Only states and tribal organizations that receive LIHEAP grants are eligible
to participate in REACH, and the households that may receive REACH project
services are those eligible for LIHEAP assistance. In its program
announcements for fiscal years 1996 through 2000, OCS required states? and
tribal organizations? proposals for REACH grants to address the following
elements: 3
the organizational experience and capability of the community- based or
tribal organization( s) conducting the project;
project staff skills and responsibilities;
state- level management and organization of the project;
project theory and design, including the target population and needs to be
addressed, activities, expected outcomes, goals, and work plan;
budget appropriateness and justification;
expected beneficial impact;
?holistic? strategies addressing the economic and social barriers to
selfsufficiency, and project innovations (required for state proposals
only);
community empowerment of areas characterized by severe poverty, high
unemployment, or other indicators of socioeconomic distress (required for
state proposals only); and
evaluation of the project. OCS uses the above proposal elements in its
criteria for selecting which projects to fund. OCS convenes panels to review
and score the grant proposals against the criteria; state and tribal
organization proposals are assessed separately. Factors in addition to the
scoring may be used in making selections, such as geographic distribution
and OCS? past experiences with the applicants.
States? evaluations of their projects are to address how effectively each
project was implemented, and whether and why the expected results and goals
were or were not achieved. OCS requires states to use third- party
evaluators to conduct the evaluations. Such evaluators are individuals or
2 Consistent with a preference stated in the congressional committee report
on the authorizing legislation, OCS has not developed and published
regulations for the REACH program.
3 OCS made some changes in its fiscal year 2001 program announcement,
including regrouping some of the elements listed and requiring tribal
organization applicants, as well as state applicants, to address innovative
approaches.
Page 7 GAO- 01- 723 REACH Program Effectiveness
firms that are organizationally distinct from the state agency or community-
based organizations involved in the REACH project.
Through the end of fiscal year 2000, OCS had awarded $30 million in REACH
grants to states and tribal organizations for 54 projects 4 (29 state and 25
tribal projects) to address the home heating and cooling needs of low-
income households using a variety of approaches. The most commonly used
approaches have been energy efficiency education, home energy audits, home
weatherization, and budget counseling, while innovative approaches have
included forming consumer cooperatives for energy purchasing and using solar
and/ or wind power. Several projects have included activities not directly
related to home energy- for example, job skill or employment development
services and financial assistance toward overdue rent or mortgage payments.
After its REACH grant period is over, a state or tribal organization may
wish to continue using the approaches tested in the REACH project. Some
activities could be replicated in a state?s or tribal organization?s LIHEAP
program, if the state or tribal organization chose to do so.
OCS has awarded REACH grants for 29 state and 25 tribal projects. Table 1
below shows REACH grant funding and numbers of grants by year and type of
grant (state or tribal organization). In addition to the grant funding shown
in the table, HHS has provided a total of $1.1 million to states for their
costs in administering the grants, overseeing the community- based
organizations that carry out REACH projects, and contracting for the
thirdparty evaluations. (Tribal organizations have not received such funding
because, to date, the tribal organizations have chosen to carry out their
REACH projects themselves and are not required to contract for thirdparty
evaluations.) Including both the grant funding and administrative funding
for states, HHS provided $31 million in REACH program funds in fiscal years
1996 through 2000.
4 The REACH program considers the fiscal year 1996 award to Michigan as
three separate grants for three separate projects and therefore considers
the total number of grants to be 56. Because Michigan addressed the projects
in one evaluation report, we have counted this as one grant award.
Demonstration
Projects Have Tried Various Approaches and Included Some Activities Not
Directly Related to Home Energy
Grant Recipients and Amounts
Page 8 GAO- 01- 723 REACH Program Effectiveness
Table 1: REACH Grant Funding by Fiscal Year and Type of Grant
Dollars in thousands
State grants Tribal organization grants All grants
Fiscal year Funding Number Funding Number Funding Number
1996 $5,102 6 $379 4 $5,482 10 1997 5,658 6 362 3 6, 020 9 1998 5,201 7 436
4 5, 638 11 1999 5,558 5 805 7 6, 363 12 2000 5,482 5 969 7 6, 451 12
Total $27,002 29 $2,951 25 $29,953 54
Note: The funding figures above do not include the administrative funding
for states. Source: GAO compilation of HHS data.
Tables 2 and 3 below list the grant awards made to states and tribal
organizations, respectively, in fiscal years 1996 through 2000. Four states-
Michigan, Nebraska, Oregon, and Pennsylvania- have received more than one
grant, for separate projects. Similarly, five tribal organizations and one
insular area have received more than one grant. State grants ranged in
amount from $166,667 to $1.6 million and averaged $931,108. Tribal
organization grants ranged in amount from $50, 000 to $199,276 and averaged
$118,051.
Table 2: Grant Awards to States, by Fiscal Year State Grant amount 1996
California $1,492,500 Maryland 169,178 Massachusetts 479,611 Michigan
761,111 Nebraska 600,000 Oregon 1,600,000
1997
Arkansas 210,000 Indiana 1, 501,840 Kentucky 266,002 New York 1,600,000
Pennsylvania 1, 200,000 Vermont 880,497
Page 9 GAO- 01- 723 REACH Program Effectiveness
State Grant amount 1998
Alaska 1,328,723 Arizona 1, 000,000 Illinois 166,667 Iowa 372,391 Maine
1,562,050 Michigan 337,437 Utah 433,853
1999
Connecticut 1,499,708 Michigan 341,635 Nebraska 1,136,042 North Carolina 1,
000,000 Oregon 1,580,600
2000
Alabama 1, 000,000 Montana 1,500,000 Nevada 1,100,000 Pennsylvania 882,277
Washington 1,000,000
Total $27,002,122
Source: GAO compilation of HHS data.
Table 3: Grant Awards to Tribes, Tribal Organizations, and Insular Areas, by
Fiscal Year Tribe, Tribal Organization, or Insular Area Location Grant
amount 1996
Central Council of the Tlingit and Haida Alaska $129,234 Grand Traverse Band
of Ottawa and Chippewa Indians Michigan 50,103 Sault Ste. Marie Tribe of
Chippewa Indians Michigan 150,000 South Puget Intertribal Planning Agency
Washington 50,000
1997
Cherokee Nation Oklahoma 199,276 Grand Traverse Band of Ottawa and Chippewa
Indians Michigan 50,103 South Puget Intertribal Planning Agency Washington
112,134
1998
Central Council of the Tlingit and Haida Alaska 175,000 Grand Traverse Band
of Ottawa and Chippewa Indians Michigan 70,738 Lumbee Regional Development
Association North Carolina 140,711 United Tribes of Kansas and Southeast
Nebraska Kansas and Nebraska 50,000
Page 10 GAO- 01- 723 REACH Program Effectiveness
Tribe, Tribal Organization, or Insular Area Location Grant amount 1999
American Samoa Government Insular area in the Pacific 123,866 Central
Council of the Tlingit and Haida Alaska 175,000 Fort Belknap Indian
Community Montana 70,819 Grand Traverse Band of Ottawa and Chippewa Indians
Michigan 65,109 Lumbee Regional Development Association North Carolina
149,995 Northern Cheyenne Tribe Montana 150,000 United Tribes of Kansas and
Southeast Nebraska Kansas and Nebraska 70,000
2000
American Samoa Government Insular area in the Pacific 124,304 Blackfeet
Tribe Montana 150,000 Central Council of the Tlingit and Haida Alaska
175,000 Grand Traverse Band of Ottawa and Chippewa Indians Michigan 149,895
Little River Band of Ottawa Indians Michigan 150,000 Lumbee Regional
Development Association North Carolina 150,000 United Tribes of Kansas and
Southeast Nebraska Kansas and Nebraska 70,000
Total $2,951,287
Source: GAO compilation of HHS data.
The activities conducted under the demonstration grants have varied. Almost
all of the REACH projects included multiple activities and addressed more
than one of the following areas:
reducing energy use for home heating and cooling by increasing efficiency;
helping clients pay for energy bills through budget planning, consumer
education, consumer cooperatives, and other methods;
reducing the use of other utilities (water and electric lighting); and
providing social services not directly related to home heating and cooling
needs.
Appendix I provides further details about the types of activities included
in REACH project plans and the number of projects conducting each type of
activity.
The most common project activity has been energy efficiency education, which
was included in 45 of 53 projects. 5 This education has been provided
through group workshops for the projects? clients or through in- home
counseling of individual households. Some grantees have also developed
5 One of the 54 projects, North Carolina?s, is not included in our
description of grant activities because it was being redesigned at the time
of our review. Project Activities
Page 11 GAO- 01- 723 REACH Program Effectiveness
energy educational materials, such as pamphlets or videos, some of which
were designed for children. Other activities aimed at reducing energy use
have included home energy audits and weatherization. Home energy audits were
used by 25 REACH projects, often in conjunction with weatherization of the
home or education in how to reduce energy use. In a home energy audit, a
detailed inspection of the home is carried out to identify repair and
weatherization needs and practices of the household that result in
inefficient energy use. For example, doors and windows are inspected for
drafts, attics for insulation needs, and furnaces for maintenance needs.
Families may be advised of energy- saving practices such as turning down
their thermostats at night or using fans to reduce the need for air
conditioning.
Twenty REACH projects have provided home weatherization work by construction
contractors. While the Department of Energy?s (DOE) Weatherization
Assistance Program helps many low- income families, some homes need repairs
that exceed the per- house dollar limit of the DOE program. The REACH
projects that included weatherization typically went beyond the services
provided by the DOE program or served clients who had not been addressed by
the DOE program. Some REACH projects worked in conjunction with DOE?s
program. In addition, 20 REACH projects provided clients with do- it-
yourself weatherization kits and training in how to install the
weatherization measures. The kits included items such as caulk, weather-
stripping, and plastic coating for windows.
Budget counseling was provided in 30 of the REACH grant projects. Such
counseling helps clients to plan ahead for paying for their essential needs,
including energy bills; to identify areas where they could cut costs; and to
become better informed about credit practices. Budget planning can help low-
income households avoid utility cutoffs due to unpaid bills, which are
followed by the need to pay reconnection fees and past- due bills. Utility
cutoffs are not uncommon for low- income households. According to OCS data,
during the 1992- 93 heating season, 1 million LIHEAP- eligible households
(3.3 percent of such households) reported that they were unable to use their
main source of heat for 2 hours or more because they were unable to pay for
their main heating fuel. 6 Sixteen REACH projects also negotiated with
energy vendors on behalf of their clients to obtain
6 These are the most recent data available; see Low Income Home Energy
Assistance Program: Report to Congress for Fiscal Year 1996, U. S.
Department of Health and Human Services, Office of Community Services,
Division of Energy Assistance, Sept. 2000.
Page 12 GAO- 01- 723 REACH Program Effectiveness
payment plans or forgiveness of past- due bills, and four projects provided
funds to help pay past- due bills and/ or reconnection fees. Seven projects
provided consumer education on utility deregulation and the consumer choices
provided by deregulation.
Some REACH projects addressed the use of utilities for purposes other than
heating and cooling. Thirteen projects provided energy- efficient light
bulbs. Energy- efficient lighting was the only focus of the two grants
received by American Samoa because the Samoan government determined that
this was the one measure most likely to reduce the high cost of electricity
for their low- income households. Six projects addressed water conservation
in the home by providing devices such as low- flow showerheads or repairing
plumbing. Such plumbing measures also reduce energy usage for hot water
heaters.
Some REACH projects included social services not directly related to home
energy needs. Seventeen of the REACH projects provided social services
through case management. For example, Nebraska?s fiscal year 1996 grant
project included caseworker visits to help families identify and address
their concerns in areas such as improving job prospects, eating nutritional
food, and obtaining health services. Only Indiana?s project provided funds
to help clients pay past- due rent or mortgage payments. Six projects
included job skill or employment development services. For example,
Indiana?s fiscal year 1997 grant included the use of REACH funds for job
skill training, transportation to work, and day care; $258,000 of Indiana?s
planned project budget addressed such services to help clients move from
welfare to work. In many states, services that support the transition of
welfare recipients to the workforce- such as job training, transportation,
and child care- are provided by the much larger Temporary Assistance to
Needy Families program ($ 22.6 billion in federal and state funding in
fiscal year 1999).
The REACH program?s criteria regarding ?holistic approaches? may have been
misconstrued to allow projects to fund social services that are not directly
related to home energy needs. Specifically, the REACH program announcement
for fiscal year 2000 stated:
?OCS is interested in having Applicants approach the energy needs of low-
income families within a holistic context of the economic, social, physical,
and environmental barriers to self- sufficiency. Thus applicants should
include in their REACH Plan an explanation of how the proposed projects( s)
will be integrated with and support other anti- poverty or development
strategies within the target community or communities.? Thus REACH
initiatives are expected to be closely coordinated with other public and
private sector
Page 13 GAO- 01- 723 REACH Program Effectiveness
programs involved with community revitalization, housing rehabilitation and
weatherization, and family development.?
Similar language has been included in each program announcement requesting
REACH grant proposals. Furthermore, the criteria used to select grant
proposals to fund have included holistic program strategies and project
innovations. REACH grant proposals are scored on a number of criteria- such
as the organization?s experience and the project strategy and design- and
each criterion has a maximum number of possible points. Out of a total of
100 possible points, state proposals for fiscal years 1996 through 2000
could receive up to 10 points in the selection process for the criterion
?holistic program strategies and project innovations.? When asked about this
criterion and the non- energy- related activities, REACH program officials
said that the criterion might need to be clarified and that they also had
some concerns about the emphasis on nonenergy activities in a few projects.
However, they also noted that the legislation sets broad purposes for REACH,
including preventing homelessness and minimizing health and safety risks
resulting from high energy burdens on low- income households. Because energy
burden is defined as home energy expenditures divided by household income,
the officials stated that some grantees had chosen to address increasing
clients? incomes.
Some REACH projects have involved innovative activities. Examples of
innovative activities include the installation of solar and/ or wind power
for low- income households in two tribal projects: the Grand Traverse Band
of Ottawa and Chippewa Indians (fiscal year 2000 grant) and the Cherokee
Nation (fiscal year 1997 grant). Another innovative activity used in three
state projects was the formation of consumer cooperatives to reduce the cost
of energy to low- income households. Vermont?s fiscal year 1997 project
formed a cooperative and purchased a home heating oil company. The other two
state projects- New York?s and Connecticut?s- are forming cooperatives to
purchase energy from deregulated utilities, allowing lowincome households to
aggregate their purchasing power and negotiate lower prices. Pennsylvania?s
fiscal year 2000 project is addressing cooling needs in urban areas,
following a number of heat- related deaths among elderly residents,
particularly in urban row houses. The project is both introducing an
innovative use of heat- reflective coatings on roofs of inner city row
houses and providing fans and safety devices that permit windows to be
locked in both open and closed positions.
REACH grantees have also developed innovative ways of working with other
organizations and delivering services. The project of the Central
Page 14 GAO- 01- 723 REACH Program Effectiveness
Council of the Tlingit and Haida tribes in Alaska trained AmeriCorps
personnel 7 to provide energy efficiency education. Because these personnel
remain in the Alaskan villages, the project proposal stated that the impact
of the services would continue beyond the completion of the grant period.
Nevada?s project for its fiscal year 2000 grant is using the concept of
individual development accounts to provide clients with matching funds for
their savings. The REACH clients? savings accounts are to be used for
replacing the furnace or appliances, changing to a lower cost fuel, or
paying higher winter utility bills. Kentucky?s project used volunteers from
local organizations to install weatherization kit materials in the homes of
disabled and elderly clients who could not do the work themselves.
After REACH projects are completed, some project activities may be
replicated in a state?s or tribal organization?s LIHEAP program, if the
state or tribal organization chooses to do so. (The state or tribal
organization may also choose to use its own funding to continue the
activities.) The legislation authorizing LIHEAP allows states to spend up to
5 percent of their allotted LIHEAP funds for services that encourage and
enable households to reduce their home energy needs and therefore their need
for energy assistance. According to LIHEAP program officials, such services
may include energy efficiency education, budget counseling, and assistance
in obtaining discounts or payment plans from energy vendors. According to
states? fiscal year 2001 plans, 25 of 50 states and the District of Columbia
planned to use a portion of their LIHEAP funds for energy use reduction
services, and their planned services included energy education, energy needs
assessment, liaison with energy vendors, budget counseling, and case worker
services for clients. The legislation also allows states to spend up to 15
percent of their allotted LIHEAP funds for low- cost weatherization or other
energy- related home repair for low- income households. States may also
apply to HHS for waivers to use up to 25 percent of their allotment for this
purpose. Forty- four of 50 states and the District of Columbia planned to
provide weatherization in fiscal year 2001.
Several types of activities used in REACH projects may not be covered by
these provisions of LIHEAP, such as forming or expanding energy consumer
cooperatives, installing solar and wind power units, providing
7 AmeriCorps, known as the domestic Peace Corps, engages more than 40, 000
Americans a year in community service. Replication of REACH
Project Activities
Page 15 GAO- 01- 723 REACH Program Effectiveness
efficient light bulbs, and providing matching funds for clients? savings to
be used for energy purposes. It is uncertain whether states may use LIHEAP
funds if they wish to continue or expand such activities after the
conclusion of their REACH project. LIHEAP officials noted that because
LIHEAP is a block grant program, states decide how to design their programs
and interpret the statutory provisions regarding energy use reduction
services and weatherization. Furthermore, LIHEAP?s funding caps in these
areas, as well as the need among low- income households for direct
assistance with energy bills, would determine the extent to which the
activities tested in a REACH project could be provided more broadly to a
state?s or tribal organization?s LIHEAP clients.
The legislation authorizing REACH identifies performance goals to be used by
individual REACH projects. For the program as a whole, however, HHS has not
developed performance goals and measurable indicators that define the
results that it expects to achieve. Furthermore, HHS? performance plans do
not address how the REACH program relates to the larger LIHEAP program of
which it is a part.
The legislation authorizing the REACH program requires that REACH project
plans describe performance goals for each project, which are to include
a reduction in the energy costs of participating households over 1 or more
fiscal years,
an increase in the regularity of home energy bill payments by eligible
households, and
an increase in energy vendor (such as utility companies) contributions
towards reducing energy burdens of eligible households. 8
The legislation further requires that project plans include a description of
the indicators that each state will use to measure whether the performance
goals have been achieved. For example, in addressing the performance goal of
more regular energy bill payments, Oregon?s proposal for its fiscal year
1999 grant specified the following indicator: 50 percent of households with
past- due energy bills will reduce their arrearages by
8 The REACH grant proposals did not always address all three of the
performance goals, and REACH program officials indicated that their
understanding of the legislation was that each state grantee was not
necessarily required to address all of them. REACH Program
Lacks Performance Goals and Measurable Indicators
Page 16 GAO- 01- 723 REACH Program Effectiveness
40 percent after 12 months of participation. In addition, OCS requires
states to address how results will be measured in their plans for evaluating
their projects.
Despite the program?s use of project- level performance goals and
indicators, the REACH program as a whole lacks performance goals and
measurable indicators. According to a REACH official, the Administration for
Children and Families decided not to address the REACH program in its
performance plan, which was developed to meet the requirements of the
Government Performance and Results Act of 1993. While agencies? performance
plans do not have to include the complete array of goals and measures used
in managing individual programs, the development and use of performance
goals and indicators can be beneficial to any federal program. Specifically,
performance goals and measurable indicators can
help a program to focus its efforts on achieving results and on those
activities most closely linked to program goals,
provide a clearer basis for selecting projects to fund,
provide a basis for determining how well the program is performing and
what has been achieved in return for the resources invested in it, and
facilitate reporting to the Congress and the public on the program?s
performance.
For the REACH program in particular, performance goals and measurable
indicators could address two needs: They could both provide a basis for
grant selection and enhance effectiveness in carrying out federal roles.
First, as described above, OCS has selected several grant proposals that
used a substantial portion of their grant funding for activities that were
not directly related to home energy needs, such as job skill or employment
development services. The use of performance goals could help OCS to better
target grant selection to projects that are closely linked to program goals.
Second, the federal roles in the REACH program are not fully addressed by
the project- level goals. For example, the federal role in the REACH program
currently includes selecting grantees, providing program guidance, helping
to strengthen project design and states? evaluations of their projects, and
providing information to grantees and others involved in providing energy
assistance. The effectiveness of federal efforts in the program could be
enhanced by results- oriented performance measures addressing these roles.
Furthermore, the performance plan for the Administration for Children and
Families (which includes OCS) does not address the relationship between
REACH and LIHEAP, or between REACH and other related
Page 17 GAO- 01- 723 REACH Program Effectiveness
federal programs, such as DOE?s Weatherization Assistance Program.
Performance plans can be useful tools for identifying the need for
coordination among programs, and for ensuring that the goals of related
programs are congruent and that crosscutting efforts are mutually
reinforcing. In the case of REACH, HHS could use this performance plan to
articulate REACH?s purpose in demonstrating and providing incentives for
LIHEAP grantees to try new approaches. Without HHS? addressing the
relationship between REACH and LIHEAP, in particular, it is unclear whether
HHS expects the REACH program to have a broader impact on the activities of
LIHEAP grantees, beyond a particular REACH grant project and its limited
time frame.
The six states that received funding in fiscal year 1996 have prepared
evaluation reports: California, Maryland, Massachusetts, Michigan, Nebraska,
and Oregon. However, only Nebraska?s evaluation report, with some
qualification, fairly reflects the REACH project?s effects on energy use.
The remaining five evaluation reports have substantial design and
implementation shortcomings that compromise the validity of the reports?
findings. In addition, all six evaluation reports have other shortcomings
that preclude an overall assessment of the projects? effectiveness.
(Appendixes II through VII summarize each state?s evaluation report,
including its goals and measures, project assumptions, evaluation design,
services provided, participant and control group selection, evaluation
findings, reported limitations, and GAO?s observations on analytical
problems and other shortcomings.) OCS is aware of the shortcomings in these
initial evaluations and has been taking several steps to improve future
project evaluations.
The design and implementation of Nebraska?s fiscal year 1996 project allowed
statistically valid conclusions to be made in the state?s evaluation report
about the effects of project services on participant energy use. The
project, which was carried out by a community- based organization, focused
on changing the behavior of low- income households to achieve economic self-
sufficiency through decreased energy use. This approach was based on the
assumption that most of the target population were renters who frequently
moved and that the housing available to them was generally substandard. It
was assumed that the only way to reduce energy use was to change household
behavior through teaching energy- efficient practices. Initial Project
Evaluations Have Limitations, but the Program Is Taking Action to Improve
Future Evaluations
Nebraska?s Evaluation Report Fairly Reflected Results
Page 18 GAO- 01- 723 REACH Program Effectiveness
The evaluation report noted that REACH participants significantly decreased
their use of natural gas from preproject to postproject compared with a
control group of nonparticipants, whose consumption showed no statistical
change. The outcome for electricity use was that while REACH participants
showed, on average, no change in electricity use, nonparticipants increased
electricity use significantly over the course of the project.
The evaluation report attributed the project?s successful implementation to
the project design and the collection of sufficient data to the community-
based organization?s experience, expertise, and knowledge of the target
population, as well as to the early involvement of the evaluator in the
project?s design. Cash incentives provided to the test and control groups
also appeared to help limit participant attrition (leaving the project
before its completion). In addition, administrative controls designed to
ensure access to utility bills seem to have played a role in obtaining
sufficient data for analysis.
The other five states? REACH projects experienced design and implementation
constraints that decreased the confidence that can be placed on the findings
contained in their evaluation reports. For example, Massachusetts and
Michigan did not use control groups as a means of assessing whether project
participants fared differently from similar households that did not receive
project benefits. California and Maryland experienced difficulties in
collecting complete income or utility data and experienced client attrition
at rates that call into question the likelihood that project effects could
be assessed.
To some extent, these state REACH projects faced common challenges: (1)
forming an adequate control/ comparison group against which to compare
project outcomes; (2) maintaining client participation in project
activities; (3) collecting complete, accurate, and reliable data (such as
client income and utility bills); and (4) adjusting (? normalizing?) energy
consumption data for changes in weather.
A properly formed control group, composed of people who did not receive
REACH project services but who are similar in other respects, allows a
comparison of what might have happened in the absence of the project.
Project designs that lack a control group can demonstrate that changes occur
in such factors as energy use, but the changes cannot be attributed to the
project since other factors might be responsible for the changes. Any
preexisting differences between a test group and a control group Other
Projects Faced
Evaluation Challenges
Page 19 GAO- 01- 723 REACH Program Effectiveness
could also be the cause of any observed project effects and, therefore,
should be addressed in the project?s design.
Attrition of participants can reduce the amount of data available for
analysis and introduce biased results. Attrition bias can occur if those
remaining in the project systematically differ from those who dropped out in
ways that are likely to affect the outcome of the project.
Lack of complete, reliable, and accurate data will result in imprecise or
biased results. Data that are not collected for an entire heating or cooling
season, for example, can lead to faulty assessments of typical energy use.
Failure to normalize energy use data makes it difficult to determine the
extent to which changes in energy use result from project activities or from
changes in weather that affect the need to heat or cool a residence. For
example, if the postproject data on energy use reflect a warmer winter than
the preproject data, a valid comparison of energy use for these two periods
should adjust the data to help determine if perceived energy savings were
due to warmer winter temperatures or to project activities.
In addition, other shortcomings precluded an overall assessment of the
projects? effectiveness. None of the evaluation reports addressed one of the
three project- level performance goals stated in REACH legislation and
program announcements: increasing energy suppliers? contributions to reduce
the energy burdens of eligible households. In addition, although the REACH
program announcements have stated that an objective of every state project
plan should be to measure whether its activities are more cost- effective in
the long term than energy assistance payments alone, none of the project
evaluation reports provided such an analysis. Finally, most of the
evaluation reports did not report lessons learned or best practices that
could be valuable to other projects. For example, if the report had
discussed the strategies used to successfully collect energy use data, this
information could have been useful to other projects. This lack of data was
often cited in evaluation reports as a principal reason for not being able
to measure project effectiveness.
The REACH program is aware of the challenges noted above and has been taking
several steps to improve future project evaluations, including providing
guidance and technical assistance. First, OCS has developed guidance for
project planning and evaluation in its demonstration programs, which include
the REACH program. According to REACH officials, this guidance was first
given to REACH grantees in 1997 or 1998. While this was too late to help the
fiscal year 1996 grantees plan their evaluations and related data needs, the
guidance addresses many of the evaluation limitations noted above and, if
followed by state grantees, REACH Program Is Taking
Steps to Improve Evaluations
Page 20 GAO- 01- 723 REACH Program Effectiveness
could help to improve future evaluations. For example, the guidance
discusses
selecting an evaluator- indicators of a good evaluator and questions to
ask in an interview;
using a logic model (described below) to design and evaluate a project;
addressing challenges to gathering data on project participants, such as
their reluctance to provide income data;
using comparison groups that are as similar as possible and overrecruiting
for comparison groups to allow for attrition; and
developing evaluation reports that are clear and complete. The guidance
was revised in 2000 to increase its emphasis on the logic model, according
to REACH program officials. A logic model identifies underlying assumptions
of the project, project activities (such as energy efficiency education),
immediate and intermediate outcomes expected from the activities (such as
improved understanding of behaviors that can affect energy use in the home),
and final project goals (such as a reduction in energy use). The REACH
program encourages grant applicants to use a logic model in their
applications and project planning.
Second, the REACH program provides technical assistance to grantees in their
project and evaluation planning. States that receive grants are required to
submit evaluation plans, which are reviewed by REACH program officials and
the consulting firm that is providing technical assistance to the program.
The REACH officials and consultant then discuss with the project team any
improvements that are needed, and the grantee submits a revised evaluation
plan. Once an acceptable evaluation plan is completed, the REACH program
sends a letter to indicate approval of the plan. According to officials of
the REACH program?s consulting firm, they emphasize improving the logic
model to strengthen the projects? designs and their ability to measure
whether expected outcomes and goals are achieved by project activities.
REACH officials also review draft evaluation reports and may ask states to
revise and improve them.
Third, guidance and assistance with evaluations have also been provided
through program conferences. The REACH program held conferences for
evaluators in July 2000 and July 2001. At the July 2000 evaluators?
conference, the participants discussed logic models, common issues in
gathering and analyzing data, evaluation methods, and lessons learned.
Developing quality evaluations has also been discussed at the annual REACH
program conferences, and evaluators as well as representatives of states,
tribal organizations, and community- based organizations may
Page 21 GAO- 01- 723 REACH Program Effectiveness
participate in the conference. For example, the REACH conference held in
January 2001 included a presentation and discussion on developing logic
models, indicators, and evaluation plans. In addition, as the state grantees
reported on their completed fiscal year 1997 projects, the conference
participants discussed many specific evaluation issues.
While the REACH program and its consultants can assist grantees in planning
their evaluations, conducting the evaluation is a responsibility shared
among the states, the community- based organizations with whom the states
contract, and third- party evaluators. State officials monitor and oversee
the projects and the evaluation reports. Staff of the communitybased
organizations are responsible for collecting the data needed to complete the
evaluation. The staff of the community- based organizations may need
training to understand the importance of data gathering and to carry out
this task effectively. Third- party evaluators help to design the
evaluation, analyze data, and prepare the evaluation report. Some of the
problems we noted in the six completed evaluations related to reporting,
such as not clearly explaining methods of analysis or data limitations. For
future REACH evaluations to improve, the states, community- based
organizations, and evaluators- as well as the REACH program and its
technical assistance consultants- must carry out their functions well.
OCS has not yet planned how it can best communicate information to state
officials and others about the results of REACH projects, such as what
approaches prove to be the most successful in meeting the home energy needs
of low- income households. As the REACH program proceeds and state grantees
complete additional project evaluations, more information will become
available about which approaches demonstrated in REACH projects were
successful and which were less successful. In addition to evaluating the
results of their projects, grantees report on the processes and procedures
they used to carry them out. Completed REACH projects will provide
information and tools such as
pamphlets, videos, and other materials for energy efficiency education;
forms for collecting data from clients; and
experiences and lessons learned in such areas as how to leverage
contributions from energy vendors and how to coordinate among various
organizations involved in energy assistance.
A comprehensive communications plan would help ensure that this information
is put to good use by identifying what information to communicate, to what
audiences, and by what methods. A HHS Has Not Yet
Planned How to Communicate REACH Project Successes and Lessons Learned
Page 22 GAO- 01- 723 REACH Program Effectiveness
communications plan would also estimate the amount of funding needed for
communications. REACH program officials realize that publications such as
summaries of successful approaches could provide information in a form more
readily accessible than the individual evaluation reports. Other OCS
programs have developed summary publications of best practices and have
found them to be frequently requested, according to the director of OCS?
Division of Community Demonstration Programs. For instance, one publication
provided lessons learned from 8 years of OCS demonstration programs (not
including the REACH program, which had not yet begun at the time). As the
REACH program matures and more information becomes available, OCS will have
a better basis for identifying and summarizing best practices. For instance,
information will become available from six project evaluations that are due
during the fall of 2001 (from the state grants awarded in fiscal year 1997),
and by the end of 2002, a total of 19 state projects will probably have
completed their evaluations.
To date, the REACH program?s communications efforts have included developing
a Web site and providing information through conferences. According to
program officials, the REACH Web site, located within OCS? Web site for its
demonstration programs, is expected to become available in the summer of
2001. Currently, a pilot version of the REACH Web site is available at the
Web site of the REACH program?s consulting firm. According to REACH program
officials, the OCS Web site will provide the same types of information as
the pilot Web site. The pilot REACH Web site includes
the current OCS program announcement requesting proposals for REACH
grants,
summaries of past and ongoing REACH projects,
summaries of the REACH program conferences, and
listings of contact points for grants and evaluations. Conferences are
also used to communicate project results and lessons learned. The REACH
program hosts an annual conference for certain people involved with REACH
projects. According to the REACH program announcements, state grantees are
expected to fund travel by the state project directors, community- based
organization project directors, and chief evaluators to the annual
conference in each of the 3 years of the project. Tribal organizations,
which have a shorter project term, are to fund travel to one conference. At
the conferences, grantees that are finishing their projects make
presentations about their project approaches and results. REACH program
officials also provide information geared to new grantees about program and
reporting requirements. The conference
Page 23 GAO- 01- 723 REACH Program Effectiveness
has also included topical presentations; for example, the conference in
January 2001 included presentations on the characteristics of good project
evaluation methods, solar power, and designs for low- energy- usage homes.
According to REACH officials, they and their consulting firm have also
arranged for presentations on REACH projects at the annual LIHEAP conference
and other energy- related conferences.
Grantees also have responsibilities for communicating about their REACH
projects. OCS requires that states? REACH project plans address
disseminating results of the individual projects among LIHEAP grantees,
utility companies, and others interested in increasing the self- sufficiency
of the poor. States are allowed to budget up to $5,000 of each grant for
dissemination purposes. Tribal organizations are allowed to budget up to
$1,000 for dissemination purposes. Such state and tribal efforts are
important, but they address only individual projects, not the broader
compilation of learning from a number of projects over a period of years.
Because the REACH program lacks performance goals and measurable indicators,
HHS has not defined the relationship between REACH and its parent LIHEAP
program and cannot assess the program?s overall effectiveness. Considering
the recent rise in home heating and cooling costs, the REACH program?s role
in testing approaches to help low- income families to meet their home energy
needs is an important one and should be clearly articulated. We believe that
the development of performance goals and measurable indicators could provide
the Congress with better information about what has been accomplished for
the resources expended. Furthermore, performance goals could provide HHS?
Office of Community Services with a clearer basis for selecting grant
proposals to fund. In addition, by addressing the relationship between
LIHEAP and REACH in its performance plan, HHS? Administration for Children
and Families could clarify the role of the REACH program and whether it
expects REACH to have an effect on the activities of LIHEAP grantees, beyond
a particular REACH grant and its limited time frame.
Some of the REACH projects funded to date have included activities other
than addressing clients? home heating, home cooling, and energy payment
needs, such as job skill or employment development services. The REACH
program?s requests for proposals with ?holistic approaches? may have been
misconstrued by grant applicants and by those reviewing proposals to allow
projects to use REACH funds for non- energy- related activities. However,
with only about $6 million in funding annually, we question whether non-
energy- related activities- including some activities typically Conclusions
Page 24 GAO- 01- 723 REACH Program Effectiveness
addressed through other, much larger social service programs- are an
effective use of limited REACH program funds.
While the evaluations conducted on the first year of state grants have many
shortcomings that limit their usefulness in assessing project effectiveness,
the REACH program recognizes the problems and has been taking steps to help
improve future project evaluations. It is too soon to gauge the
effectiveness of these efforts, but we noted some improvements in states?
plans for the next set of evaluations. We encourage the REACH program to
continue its efforts to improve the design and methodology of evaluations
because valid evaluations are vital to realizing the potential of the REACH
program in testing new approaches.
With only six project evaluations completed, there is currently not enough
information to reach a conclusion about the effectiveness of the REACH
program. By the end of 2002, a total of 19 state projects will probably have
completed their evaluations, and OCS will have awarded a total of about 80
grants, roughly half of them to states, which will eventually report on
project results. While the legislation authorizing the REACH program
requires this GAO review, it does not require HHS to report to the Congress
as more information on project results becomes available. Developing program
performance goals and measurable indicators and obtaining better data on the
REACH program?s effectiveness would enable HHS to provide the Congress the
information that it needs to assess the program. Furthermore, the
legislation did not specify an ending date for the REACH program, as is
sometimes the case for demonstration programs. It may be appropriate to
reassess the REACH program during 2003 and consider whether and how long it
should continue.
Finally, the lack of a comprehensive plan for communicating the results of
REACH projects and fostering the further use of effective approaches could
limit the impact of the REACH program. Without well- planned and adequately
funded communications, the results of REACH projects may fail to have an
impact on LIHEAP and other programs that provide energy assistance to low-
income families. Summaries reporting on best practices could be more
effective as communication tools than the state evaluation reports
themselves for reaching the state, federal, and community officials involved
in addressing the home energy needs of low- income households. To the extent
that OCS shares such information among the state, tribal, and community-
based organizations that provide energy assistance, the organizations can
make better- informed decisions about replicating successful approaches and
avoiding problematic ones. Furthermore, wellinformed organizations can avoid
?reinventing the wheel? by, for instance,
Page 25 GAO- 01- 723 REACH Program Effectiveness
not investing in developing energy education materials that may already be
available. In light of additional project evaluations and other information
that will become available over the next several years, OCS needs to plan
for its communications efforts.
To better target the use of the limited resources of the REACH program and
provide for reporting on program performance, we recommend that the
Secretary of HHS direct the Administration for Children and Families and its
Office of Community Services to
develop program performance goals for REACH that are objective,
measurable, and quantifiable;
address the relationship between the REACH and LIHEAP programs in its
performance plan; and
ensure that REACH funds are used for activities directly related to the
home energy (heating and cooling) needs of low- income households.
To ensure that the results of REACH projects are effectively communicated to
the government agencies and private organizations involved in addressing
low- income households? energy needs, we recommend that the Secretary direct
the Office of Community Services to develop a communication plan for the
REACH program describing intended audiences, types of information to be
communicated, communication methods appropriate to the intended audiences,
and the funding needed.
With HHS developing information for performance reporting and obtaining
additional project evaluations, the Congress may want to consider requiring
HHS to report on REACH program effectiveness and project results in several
years, after the projects funded in the first 3 years of the program have
completed their evaluations by the end of 2002. Once HHS has reported, the
Congress may also wish to consider whether the REACH program should continue
indefinitely or whether the program should have an end date after a
sufficient number of demonstration projects.
The Department of Health and Human Services provided written comments on a
draft of this report. These comments are reprinted in appendix VIII, along
with our responses. HHS generally agreed with our recommendations.
Recommendations for
Executive Action Matters for Congressional Consideration
Agency Comments and Our Response
Page 26 GAO- 01- 723 REACH Program Effectiveness
However, while HHS agreed with our third recommendation that it ensure that
REACH funds are used for activities directly related to the home energy
needs of low- income households, it made two comments related to this
recommendation. First, HHS disagreed with our assessment that language in
its program announcements on holistic program strategies may have been
misconstrued to encourage non- energy- related activities. We continue to
believe that HHS should review the language of its program announcement, as
it plans to do, to ensure that the REACH program does not fund activities
that are not directly related to the home energy needs of low- income
households. The language in the program announcement is used as criteria for
reviewing and selecting grant proposals, as well as for providing guidance
to applicants for grants. Our recommendation concerns only REACH program
funds; grantees would not be precluded from using other sources of funds for
such activities or from coordinating with other social service efforts.
Second, HHS stated that the term ?residential energy? is understood to
include all household energy use, not just home heating and cooling.
However, we note that the authorizing legislation does not define the term
?residential energy.? Therefore, we believe that HHS should apply the
definition of home energy in the authorizing legislation- namely, a source
of heating or cooling in residential dwellings. We have changed the wording
of our recommendation to make this more clear.
HHS suggested that our matters for congressional consideration include the
possibility of expanding the REACH program, as well as the options of
continuing it or setting an end date for the program. We have not
incorporated this comment because the effectiveness of the REACH program has
not yet been determined and because we believe that HHS should be required
to report to the Congress in 2003 on the REACH program?s effectiveness.
HHS also made a number of technical comments, which we have incorporated as
appropriate.
We reviewed all of the REACH grants that have been awarded since the program
was first funded in fiscal year 1996. However, in reviewing project
evaluations, we focused only on states? projects, because tribal
organizations are not required to do project evaluations. We also reviewed
documents about the LIHEAP program and interviewed LIHEAP officials in order
to better understand the context and purpose of the REACH program. Specific
actions that we took to accomplish each of our objectives are listed as
follows: Scope and
Methodology
Page 27 GAO- 01- 723 REACH Program Effectiveness
To obtain information about grant amounts, recipients, and project
activities, we reviewed REACH program summary documents and grant proposals;
interviewed state officials responsible for most of the state grants awarded
in the first 2 years of the program; and attended the REACH program?s annual
conference, where we heard presentations by grantees.
To obtain information about REACH program goals, objectives, and
performance measures, we reviewed the authorizing legislation, HHS
performance plans, requirements and guidance related to the Government
Performance and Results Act of 1993, and REACH program announcements that
request grant proposals. We also interviewed program officials.
To analyze the results of state evaluations, we reviewed the project
design and implementation of the six completed evaluations. We assessed the
adequacy of key aspects of project design and implementation since these
factors determine the confidence that can be placed on an evaluation?s
findings. By determining whether the evaluation reports contained critical
methodological flaws, we ascertained whether the reported findings were so
qualified as to preclude their use in assessing the REACH project?s
effectiveness. In addition, we summarized these evaluations, contacted the
state officials responsible for these six projects to discuss the
evaluations, and discussed efforts to improve evaluations with REACH program
officials and the program?s consultants.
To identify the REACH program?s communications efforts, we interviewed
program officials and reviewed the material made available on the REACH
program?s pilot Web site and through REACH program conferences. We did not
assess the communications efforts of REACH grantees.
We conducted our review from December 2000 through July 2001 in accordance
with generally accepted government auditing standards.
We are sending copies of this report to the Secretary of HHS, the Director
of the Office of Management and Budget, appropriate congressional
committees, and other interested parties. We will also make copies available
to others upon request.
Page 28 GAO- 01- 723 REACH Program Effectiveness
If you or your staff have any questions about this report, please call me on
(202) 512- 3841. Other key contributors to this report are listed in
appendix IX.
Jim Wells Director, Natural Resources
and Environment
Appendix I: REACH Project Activities Page 29 GAO- 01- 723 REACH Program
Effectiveness
Table 4 summarizes the activities planned for the 53 Residential Energy
Assistance Challenge Option (REACH) projects that have been funded since the
beginning of the program: 28 state and 25 tribal organization projects. (The
fiscal year 1999 North Carolina grant is not included in the table because
it was being redesigned at the time of our review because the state had not
deregulated utilities as had been expected when the project was originally
being designed.) Most project plans included multiple activities. Several
activities used by only one or a few projects are not included below.
Table 4: Activities Planned by REACH Grantees Activity Number of state
projects
providing the activity Number of tribal organization projects providing the
activity Total projects
providing the activity
Providing energy education through workshops or in- home counseling 26 19 45
Conducting home energy audits 16 9 25 Providing home weatherization repairs
12 8 20 Providing weatherization kits 8 12 20 Providing other energy-
related home repairs, such as fixing windows or roofs 8 412 Replacing
inefficient furnaces or appliances 15 5 20 Providing energy- efficiency
devices, such as programmable thermostats 7 310 Providing energy safety
devices, such as carbon monoxide detectors 4 48 Installing solar or wind
power systems 0 2 2 Providing budget counseling 17 13 30 Negotiating with
energy vendors to obtain payment plans or forgiveness of past- due bills
11 5 16 Providing financial incentives for reducing energy use or paying
utility bills regularly 4 913 Educating consumers about utility deregulation
7 07 Paying past- due utility bills or connection fees 4 04 Forming consumer
cooperatives for energy purchasing 3 03 Providing water conservation devices
or plumbing repairs 3 36 Providing energy- efficient light bulbs 9 4 13
Providing case management social services 16 1 17 Providing job skill or
employment development services 4 26 Making payments toward past- due rent
or mortgages 1 01
Source: GAO analysis of the grantees? project proposals.
Appendix I: REACH Project Activities
Appendix II: Review of California?s REACH Project Evaluation Report
Page 30 GAO- 01- 723 REACH Program Effectiveness
The primary goal of the California REACH project was to help low- income
households reduce their energy use by providing energy conservation and
other services. Specific project goals follow:
Dwellings would become more energy- efficient.
Health and safety risks would be mitigated.
Families would become knowledgeable about energy use and conservation.
Families would make and keep energy conservation goals.
Families would reach a stable level of energy consumption.
Families would reduce their demand for Low- Income Home Energy Assistance
Program (LIHEAP) payment assistance.
Eligible recipients would enroll in utility- supported rate discount
programs.
Families that needed additional help in other areas would receive
assistance.
Families would become stable and develop problem solving skills and coping
abilities.
The evaluation report did not clearly identify major project assumptions,
other than recognizing an expected participant attrition rate of 20 to 25
percent. To compensate for expected attrition, the project design reflected
plans for increasing the participant base.
The project was designed to determine which combinations of services
(described below) were the most effective in reducing energy use of
lowincome households. The project was designed to use test groups and
control group comparisons. It was to use six test groups (one for each of
six combinations of services), with corresponding subgroups located at three
separate locations. Services were to be provided to REACH participants by
four different community- based organizations. A seventh group- the control
group- was not to receive services other than energy assistance payments.
Utility bill data and other information were to be collected for all seven
groups before and after project implementation. The results of the REACH
groups were to be compared and evaluated. Appendix II: Review of
California?s REACH
Project Evaluation Report Goals and Measures
Project Design and Implementation
Project Assumptions Evaluation Design
Appendix II: Review of California?s REACH Project Evaluation Report
Page 31 GAO- 01- 723 REACH Program Effectiveness
The evaluation report identified five services to be provided by community-
based organizations through case management: (1) basic and (2) enhanced
energy conservation education, (3) basic and (4) enhanced home weatherizing,
and (5) family intervention (including referral to other agencies when
needed). The control group was to receive only energy assistance payments.
In addition to the above services that would be evaluated, the
communitybased organization was to provide a variety of other services, such
as outreach, eligibility determinations, residential assessments, and family
assessments.
Eligible project participants were households that received energy
assistance payments, had lived at the same residence for the past year, and
were not expected to move in the current year. In addition, households were
required to have metered energy- electricity, gas, or propane- for their
individual residence and could not have received previous home weatherizing
services. Project participation was further restricted to households that
spoke limited English, had a high energy burden, 1 received public
assistance for children, and/ or were unable to pay current utility bills.
The evaluation report stated that participant eligibility was verified using
the standard process for determining eligibility for energy assistance
payments.
The evaluation report indicated that households were to have been randomly
selected and assigned to the various test and control groups on the basis of
the services they were to receive.
According to the evaluation report, there was no statistically significant
reduction in the use or cost of energy among the test groups. The report
found that the average monthly household energy use and cost actually
1 Energy burden is defined as expenditures of a household for home energy
(heating and cooling) divided by the income of the household. Services to Be
Provided
REACH Participant and Control Group Selection
Evaluation Findings and Reported Limitations Findings
Appendix II: Review of California?s REACH Project Evaluation Report
Page 32 GAO- 01- 723 REACH Program Effectiveness
increased slightly during the project. Overall, average monthly electrical
and natural gas use was also found to have increased. The results of the
individual test groups, however, were reported to be mixed. For example, 6
groups reduced their electrical use, 7 groups reduced their natural gas use,
and 10 groups reduced their total energy cost.
The evaluation report also concluded that data were insufficient to
determine whether the project had helped increase the regularity of utility
bill payments. In addition, the evaluation report noted that according to
the results of an energy awareness survey, participant knowledge of energy
conservation measures had increased slightly. The report did not
specifically assess the project?s impact on the other eight project goals.
Data collection problems were identified by the evaluation report as a major
problem that could affect conclusions about the project?s outcomes. None of
the community- based organizations provided all of the data needed for the
evaluation. One community- based organization?s work was not included in the
evaluation because it did not collect any follow- up data. The information
provided by the other three community- based organizations was incomplete
because they either lost or did not completely collect the data. The
evaluation report also noted that the project did not collect data on the
regularity of utility bill payments, and the report provided no reason for
this.
The evaluation report attributed these data collection problems to a lack of
controls over the data collection activities of the service providers-
noting that procedures for data collection were limited, and the service
providers had no incentives to collect follow- up data. The report also said
that community- based organization staff turnover and the lack of training
in data collection requirements contributed to the problem and recommended
that in the future final award payments be withheld until all the data are
provided.
In addition, the evaluation report noted that energy use data provided by
the community- based organization was not adjusted for changes in weather
and concluded that this could have significantly limited the evaluation?s
findings and conclusions.
The effectiveness of the California REACH project cannot be determined on
the basis of the information presented in the project evaluation report
because of limitations in the evaluation?s implementation and analysis. The
Reported Limitations
Our Observations
Appendix II: Review of California?s REACH Project Evaluation Report
Page 33 GAO- 01- 723 REACH Program Effectiveness
report also provided insufficient information to determine the validity of
the findings and did not provide other information needed to assess the
project?s overall effectiveness.
Several key issues compromised this REACH evaluation effort. For example,
the evaluation did not compare the test groups with the control group as
called for in the project design. Without a control group there is no
assurance that changes found in the test groups did not result from factors
other than the combination of services provided. Moreover, while the report
noted that the project?s impact on utility bill payments could not be
assessed because of insufficient data, it did not identify or explain the
effect of attrition and the loss of data on the statistical validity of its
findings. The substantial household attrition that occurred and the
resulting loss of data could have further compromised the validity of the
project results because households that completed the project may differ
from those that dropped out in ways likely to affect the outcome.
In addition, as noted in the evaluation report, energy use data were not
adjusted for weather conditions, thereby further compromising the
statistical validity of the findings on energy use. For example, if
temperatures were significantly colder in winter or warmer in summer, a
corresponding increase in energy use would also be expected. If this
occurred during the period after energy conservation services were provided,
these services could have been responsible for stabilizing household energy
use or reducing the amount of energy that might otherwise have been
consumed. Such a finding could not have been reported because the effects of
changes in weather had not been taken into account when measuring results.
In addition to not including all of information necessary to assess the
validity of its findings, the evaluation report did not provide other
important information needed to assess the overall performance of the
project. For example, the report did not address the project?s performance
regarding one of the project performance goals identified in the authorizing
legislation and the Department of Health and Human Services (HHS) program
announcement- increasing the contributions energy suppliers make to reducing
the energy burden of eligible households. Finally, the evaluation report did
not provide a complete discussion of the project design, including
information on project assumptions, hypotheses to be tested in terms of
measurable objectives, or lessons learned. Analytical Problems
Other Shortcomings
Appendix III: Review of Maryland?s REACH Project Evaluation Report
Page 34 GAO- 01- 723 REACH Program Effectiveness
The goals of the Maryland REACH project were to help participants (1) reduce
their electric bills, (2) improve their budgeting skills, and (3) increase
their education or work opportunities with the ultimate goal of preparing
them for self- sufficiency once the project was completed.
The evaluation report did not clearly identify major project assumptions.
The project was designed to use two groups of low- income households- a test
group and a control group- to assess of services provided. Data on
participants in the test group were to be collected (1) from participant
surveys- before and after the project- and (2) from utility statements for
the year before the project, when available, and for the year during project
implementation. Participant data before project implementation were to be
compared with participant data collected during project implementation so
that the effect of the services could be evaluated. Utility bill statements
were also to be obtained for the control group receiving financial
assistance for energy bills, so that a comparison could be made between the
test group and the control group. The evaluation report also noted that
energy use data were to be compared by ?seasons?
to control for the effect of weather. Services were to be provided by a
community- based organization using a case management approach. Each
household was to receive four caseworker contacts. During the first contact
the caseworker would perform an initial assessment of household energy use,
income, education, and occupation. During the second contact, the family
assessment was to be updated. The caseworker would also obtain an
application for energy assistance from the household and provide energy
conservation counseling. During additional contacts, the family assessment
would continue to be updated and energy conservation counseling provided.
Referrals to public assistance programs for help with rent or mortgage
payments, utility bills, home weatherizing, or job training would be made
when necessary. Other services would include a newsletter and workshops on
energy conservation. (The newsletter was eventually cancelled because it was
not being read, and the workshops were cancelled because of low attendance.)
Appendix III: Review of Maryland?s REACH
Project Evaluation Report Goals and Measures Project Design and
Implementation
Project Assumptions Evaluation Design
Services to Be Provided
Appendix III: Review of Maryland?s REACH Project Evaluation Report
Page 35 GAO- 01- 723 REACH Program Effectiveness
In addition, program participants were often referred to other agencies
providing various kinds of assistance, and more than one referral could be
made for each participant. There were 152 referrals made for a variety of
social assistance services, including employment and education. Other
referrals were made for home weatherization assistance (86), such as
instruction on energy conservation; energy assistance payments (73); and
mortgage or rent assistance (20).
The local energy assistance office initially identified eligible households,
and later participants were recruited from energy assistance applicants and
referrals from other agencies. A questionnaire was used to establish
eligibility. Over a 25- month period, a total of 124 participants entered
the program and 92 participants were dropped from the program, primarily
because they could not be contacted.
The control group was selected from households receiving energy assistance
payments. Its members did not receive the REACH project services noted
above. The evaluation report noted that the size of the control group was
adjusted monthly to compensate for changes in the number of project
participants.
The evaluation report compared participant pretest and posttest data on
energy use and found that while households reduced their use of electricity,
the overall reduction was not statistically significant. However, the report
noted that a statistically significant reduction was seen in winter. The
average monthly energy use was reported to have decreased by 61 kilowatt
hours, from 1,275 kilowatt hours to 1, 214 kilowatt hours. The average cost
of household monthly utility bills was reported to have decreased by $16,
from $137 to $121.
The evaluation report also found that the use of electricity by the control
group had increased by 55 kilowatt hours, and that the difference between
the two groups was statistically significant. As a result, the evaluation
report concluded that participating households were able to stabilize their
use of electricity while the control group households were not. The report
did not assign a dollar cost savings to the difference between the groups.
REACH Participant and
Control Group Selection Evaluation Findings and Reported Limitations
Findings
Appendix III: Review of Maryland?s REACH Project Evaluation Report
Page 36 GAO- 01- 723 REACH Program Effectiveness
Utility bills were also used to assess the impact of program services on
arrearages. According to the evaluation report, the number of late payments
and termination notices decreased for households participating in the
project, but neither decrease was statistically significant. Although the
number of termination notices significantly increased for the control group,
the number of actual terminations decreased, while the number of actual
terminations increased for participating households. According to the
evaluation report, this difference was statistically significant and
suggested that the project goal of improving budgeting skills was not
accomplished.
The evaluation report also analyzed monthly utility bill payments and
balances carried over from month to month to assess the ability of
households to manage their personal finances. The evaluation report found a
slight increase in electricity bill payments that was not statistically
significant but found a reduction in monthly balances that was statistically
significant. Neither difference was found to be significant when compared
with the control group.
On the basis of these findings, the evaluation report concluded that the
project achieved the following:
Households received $60,000 in assistance for energy use and rent or
mortgage assistance.
A total of 363 referrals were made to agencies providing assistance.
Indications implied that households reduced or stabilized their energy
use.
Households received fewer termination notices.
Households reported less difficulty in paying utility bills and in
conserving energy.
The report, however, also concluded that the project did not meet some
expectations:
Over half of the households entering the project were dropped because they
could not be contacted later in the project.
Newsletters and workshops were discontinued because of a lack of interest.
Participants did not decrease the amount of arrearages in their utility
bills.
A decrease was not achieved in the termination of utility services.
Appendix III: Review of Maryland?s REACH Project Evaluation Report
Page 37 GAO- 01- 723 REACH Program Effectiveness
According to the evaluation report, the attrition rate of households
resulted in difficulty obtaining adequate information for the project
evaluation. In addition, survey responses were obtained in two ways: 29
households completed surveys, and 10 additional responses were obtained
through telephone interviews.
The effectiveness of the Maryland REACH project cannot be determined on the
basis of information presented in the project evaluation report. The
assessment has limitations that affect the validity of conclusions that can
be made regarding reducing energy use and other goals. For example, the
report does not present information on the comparability of the test and
control groups. In addition, there was a lack of information on the number
of months of utility data obtained for REACH and control group members. The
report did not address the possible effects of participant attrition on the
comparability of these groups. Moreover, energy use data were not adjusted
for the effects of weather. The report also had other shortcomings that
precluded an overall assessment of the project?s effectiveness.
The report did not provide sufficient information on the test and control
groups to allow an assessment of their comparability. Discussion of factors
that could affect comparability of the test and control group was incomplete
in several respects. For example, the report did not provide sufficient
information on the number of months of utility bills available for both
groups. Estimates based on data for a few months can be more easily
influenced by factors unrelated to the project services than those based on
data for an entire year. Although the report stated there was a
statistically significant difference in energy use between the groups for
the year, it did not discuss the possible reasons a statistically
significant difference was not observed in winter when the control group
also showed a decrease in energy use.
The validity of the conclusions regarding energy use could also be
compromised by attrition in the test and control groups because households
completing the project may not be representative of the target population as
a whole. The report, however, did not identify the effect of attrition and
the resulting loss of data on the validity of its findings.
In addition, energy use data were not adjusted for possible changes in the
weather from preproject implementation to postproject completion. The
decrease in winter energy use noted in the report could have occurred
Reported Limitations
Our Observations Analytical Problems
Appendix III: Review of Maryland?s REACH Project Evaluation Report
Page 38 GAO- 01- 723 REACH Program Effectiveness
because of a relatively milder winter during the project. The seasonal
comparison of the test and control groups actually showed no significant
statistical difference between the groups during the project.
The evaluation report did not provide other important information needed to
assess the overall effectiveness of the project. For example, the report did
not address the project?s performance in terms of the performance goal
stated in the authorizing legislation and the HHS program announcement-
increasing energy supplier contributions to reducing the energy burden of
eligible households. The evaluation report also did not provide a complete
discussion of the project design, including information on the project?s
assumptions, hypotheses to be tested in terms of measurable objectives, or
lessons learned. Other Shortcomings
Appendix IV: Review of Massachusetts? REACH Project Evaluation Report
Page 39 GAO- 01- 723 REACH Program Effectiveness
The overall goal of the Massachusetts REACH project was to reduce
homelessness by fostering self- sufficiency. In that context, the grant
application identified three main project goals:
reducing household energy costs,
increasing the regularity of household energy bill payments, and
increasing contributions by energy suppliers toward reducing households?
energy burden.
The evaluation report also identified five objectives for the REACH project
participants:
Increase awareness of energy consumption.
Increase awareness of the ability to shop for the least expensive energy
supplier.
Learn budgeting.
Make regular payments to energy suppliers.
Negotiate arrearage forgiveness with energy service providers. The
evaluation report did not- except in the calculation of energy burden-
clearly identify major project assumptions. Assumptions related to the
calculation of household energy burden were discussed in some detail (see
below).
The project was designed as a single group of REACH participants receiving a
variety of services. The study stated that financial constraints prevented
using a control group. Participant data were to be collected both before and
after the project using a standard data collection instrument,
questionnaires, or both. Utility bills were to be obtained from energy
suppliers. The information collected would then be used to assess changes in
household self- sufficiency, knowledge of energy use, the effect of specific
interventions, and changes in energy burden by comparing preREACH data with
post- REACH data.
The project focused on assisting households through case management
services. Caseworkers were to provide financial counseling and energy
Appendix IV: Review of Massachusetts?
REACH Project Evaluation Report Goals and Measures
Project Design and Implementation
Project Assumptions Evaluation Design
Services to Be Provided
Appendix IV: Review of Massachusetts? REACH Project Evaluation Report
Page 40 GAO- 01- 723 REACH Program Effectiveness
education and make referrals to other public assistance agencies for
services dealing with home energy conservation, employment, training, day
care, and improving language skills. Energy assistance payments were to be
the only immediate financial assistance provided- although 148 of 164
households (over 90 percent) reported receiving other public assistance.
Caseworkers also were to facilitate household access to an arrearage
forgiveness program offered by a major utility company. The evaluation
report noted that, during implementation, caseworkers provided intake to
over 460 households that were either homeless or at risk of being homeless,
and maintained contact with over 350 of these households.
Project participants were selected from low- income households eligible to
receive energy assistance payments, and caseworkers recruited clients from
assistance programs for the homeless. The report did not describe the
selection procedure. As noted above, the project did not use a control
group.
According to the evaluation report, project results were difficult to
quantify, and the true measure of success was the range of services provided
in response to participants? needs. However, information gathered by case
managers showed no change in self- sufficiency by project participants. The
evaluation report addressed project performance in terms of the three main
project goals.
Reducing household energy costs The evaluation report calculated the
energy burden to households using a formula developed to measure energy
costs, or debt, as a portion of income. On the basis of the results of the
calculation, the evaluation report found that the energy burden- the
proportion of household income represented by energy- related debt-
decreased during the project from $345 (105 percent) to $252 (64 percent).
However, the evaluation report noted that complete data- both pre- and
postproject data for the same households- were available for only five
project participants. The report also stressed that the calculation was
based on the unlikely assumption REACH Participant and
Control Group Selection Evaluation Findings and Reported Limitations
Findings
Appendix IV: Review of Massachusetts? REACH Project Evaluation Report
Page 41 GAO- 01- 723 REACH Program Effectiveness
that income would remain the same over time and on unreliable energy bill
data.
In addition, the evaluation report stated that, although the goal of
reducing energy costs appeared to have been achieved, the lack of posttest
followup data prevented a determination of whether the objectives of
increased awareness of energy use and energy supplier options were
accomplished.
Increasing the regularity of household energy bill payments The evaluation
report assumed that participant utility bills would be available to provide
a record of payments. However, only electric bills were available, limiting
payment analysis to the 22 households using electricity to heat their homes.
Of these households, follow- up data existed for only two, according to the
evaluation report, making assessment of the outcome impossible.
Increasing contributions by energy providers toward reducing the
households? energy burden
The evaluation report included information provided by the project staff on
the arrearage forgiveness program offered by a major public utility company
to households receiving energy assistance payments and indicated that
caseworkers processed 45 applications from project participants for this
program.
The evaluation report also identified lessons learned from experience with
the project and from interviews with case managers. Lessons learned included
the need for
additional guidance from HHS on the conduct of evaluations,
involving evaluators in project design,
obtaining access to utility bills from utility companies, and
thorough training for project staff involved in data collection. The
evaluation report noted that in many cases data were either unreliable or
missing, making an assessment of project outcomes impossible. The report
attributed this data problem to several factors, including the transient
nature of participating households, lack of training for caseworkers, and
failure to involve the evaluator early in the program design. Reported
Limitations
Appendix IV: Review of Massachusetts? REACH Project Evaluation Report
Page 42 GAO- 01- 723 REACH Program Effectiveness
The report also cited shortcomings in the assumptions used to calculate and
analyze household energy burden that adversely affected the assessment of
project performance. In addition, the report stated that data records were
sometimes constructed by case managers according to their memory of events
occurring weeks or even months earlier. The evaluation report also noted
that a scale used to measure self- sufficiency was never tested for validity
or reliability.
The effectiveness of the Massachusetts REACH project cannot be determined
using the information presented in the project evaluation report.
Performance could not be assessed and reported because of the lack of data.
Data were missing primarily because of attrition- participants leaving the
project before completion. In addition, the project used a pretest/ posttest
project design that attempted to measure the impact of delivering a variety
of services but did not include a control group. A comparison group is
necessary to help ensure that the effects observed resulted from services
provided and not other variables and to evaluate the effectiveness of the
services compared with energy assistance payments alone. The report had
additional shortcomings that would preclude an overall assessment of the
project?s effectiveness.
On the plus side, the evaluation report discussed project performance in
terms of the performance goals set out in the authorizing legislation and
the REACH program announcement. The report also discussed the difficulty
encountered in measuring project performance because of data collection
problems and described the limits on statistical analysis due to
insufficient data. The report also identified issues relating to the
calculation of energy burden that could adversely affect the analysis. In
addition, the report provided a description of lessons learned that could
benefit the design of future projects.
Statistical estimates of project performance could not be made because of
the lack of data. Problems with households leaving the project before
completion, accompanied by the inability to obtain utility bills from all
energy providers, resulted in insufficient information to assess the effect
of services provided on either household energy use or the regularity of
utility payments. Attrition (leaving the project before completion) raises
the potential for biased results if those remaining in the project
systematically differ from those who drop out in ways that are likely to
affect the outcome. Concerns regarding data quality also arise from the Our
Observations
Analytical Problems
Appendix IV: Review of Massachusetts? REACH Project Evaluation Report
Page 43 GAO- 01- 723 REACH Program Effectiveness
construction of records by case managers on the basis of memory months after
events occurred.
However, even if sufficient data had been available for statistical
analysis, failure to include a control group in the project design would
have limited conclusions regarding the effect of project services on
household energy use and changes in the regularity of utility bill payments.
When a control group is used, the credibility of the identified cause of
observed outcomes is generally greater because external factors have been
taken into account.
Additional analytical problems identified in the evaluation report include
assumptions implicit in the energy burden calculation and the use of a scale
to measure changes in self- sufficiency that was not tested for either
reliability or validity. These shortcomings further limit findings or
conclusions that can be made about the project.
While the evaluation report clearly identified certain problems encountered
in assessing performance and the resulting limitations of its findings, it
did not provide or address the lack of other information needed to assess
overall project performance. For example, the report did not address the
project?s performance in terms of the performance goal stated in the
authorizing legislation and the HHS program announcement- increasing energy
suppliers? contributions to reducing the energy burden of eligible
households. The report noted that the information needed to assess this goal
would be provided by the project staff in a separate report. Other
Shortcomings
Appendix V: Review of Michigan?s REACH Project Evaluation Report
Page 44 GAO- 01- 723 REACH Program Effectiveness
The Michigan REACH project was comprised of two subprojects with different
goals. One subproject?s primary goal was the reduction of household energy
use. The other subproject?s primary goal was to educate households on energy
deregulation, provide consumer advocacy, and explore the feasibility of bulk
energy purchases.
The evaluation report identified eight performance measures that were to be
used to evaluate the impact of services on households:
reducing utility bills,
increasing the regularity of utility bill payments,
increasing earned income,
reducing reliance on energy assistance programs,
increasing knowledge of energy conservation methods,
increasing understanding of utility bills,
reducing energy- related safety problems, and
maintaining or increasing the availability of affordable housing. The
evaluation report did not clearly identify major project assumptions. The
project was divided into two subprojects with different goals for different
providers of education services. One subproject was to focus on improving
energy conservation and developing life skills, such as budgeting. The other
subproject was to focus on increasing knowledge about energy deregulation.
Both subprojects, however, would use a preand posttest evaluation design-
collecting data before and after project implementation- to evaluate the
effectiveness of the services provided. A control group that did not receive
services was to be created from eligible households to permit comparison
with households that received services.
In the energy conservation and life skills education subproject, household
energy use data would be collected by seven service providers- located in
different areas- for the year before and the year after services were
provided. The evaluation report indicated that these data were to come from
utility bills obtained from energy suppliers. In addition, participants
would also be mailed a questionnaire after project completion to obtain
their views on the project. Appendix V: Review of Michigan?s REACH
Project Evaluation Report Goals and Measures
Project Design and Implementation
Project Assumptions Evaluation Design
Appendix V: Review of Michigan?s REACH Project Evaluation Report
Page 45 GAO- 01- 723 REACH Program Effectiveness
In the energy deregulation education subproject, households would complete a
questionnaire before and after project implementation to determine changes
in their knowledge about energy deregulation.
The energy conservation and life skills education subproject was designed to
help low- income households develop plans identifying changes directed at
achieving an immediate reduction in energy use. Education would be provided
through workshops. Other workshops would also be provided that addressed
broader issues, such as preparing household budgets and finding employment.
This subproject initially targeted 1,100 households at various locations
throughout the state.
The other subproject was designed to prepare low- income households for the
effect of utility deregulation through education and to serve as a consumer
advocate. This subproject was to consist of placing informational articles
on energy deregulation in newsletters and in pamphlets that were delivered
to participating households.
The evaluation report stated that the subprojects had a target population of
households at or below 150 percent of the federal poverty level. The report
also stated that households in the deregulation education subproject were
participants in the Head Start Program. The report did not further explain
the selection criteria or process other than to note that the control group
would be selected from eligible households.
According to the evaluation report, there was no significant difference
before and after the project in natural gas use by participants at the three
locations for which data were available. However, on the basis of the
results of two locations reporting pre- and postproject data, the report
found the project effect on electricity use mixed: a significant reduction
was reported at one location but not at the other.
On the basis of participant responses to the postproject survey, the
evaluation assessed the results, or outcomes, of four performance measures
in qualitative terms: Services to Be Provided
REACH Participant and Control Group Selection
Evaluation Findings and Reported Limitations Findings
Appendix V: Review of Michigan?s REACH Project Evaluation Report
Page 46 GAO- 01- 723 REACH Program Effectiveness
Increased regularity of utility bill payments- most participants described
this workshop education component as helpful.
Reduced reliance on energy assistance programs- about one- third of the
participants indicated that their use of emergency energy assistance had
decreased.
Increased knowledge of energy conservation methods- the majority of the
participants felt that the energy workshop was helpful.
Increased understanding of utility bills- most participants found the
educational information helpful.
Also, on the basis of participant responses to the surveys, before and after
the project, the evaluation report concluded that there was no change in
household understanding of utility deregulation.
The evaluation report noted that the data results cited on natural gas use
should be used with caution because of the small number of households for
which data were available. Three locations provided data on natural gas use,
while two locations provided data on electricity use. Data on gas use were
adjusted for weather, but data on electricity use were not adjusted. The
report also noted that the control group was not valid because some
households may have also received weatherization services. In addition, the
evaluation report stated that changes were made to the project in the second
year, but no postproject data were available to include in the final
evaluation report.
Regarding the survey of participants, the evaluation report noted that the
assessment of perceptions of energy conservation and life skills workshops
was based on 299 surveys returned out of 740 surveys mailed- a response rate
of 40 percent.
According to the evaluation report, it was not possible to measure the
effect of the project on earned income, reductions in housing safety
problems, and availability of affordable housing.
The effectiveness of the Michigan REACH project cannot be determined because
of limitations in design and implementation. For example, the lack of data
on participant energy use, along with other analytical problems, such as the
lack of a control group and failure to adjust energy use data for weather
conditions, compromised the validity of the findings on energy use. As a
result, a valid assessment of the effect of project activities on energy use
or regularity of participant utility bill payments Reported Limitations
Our Observations
Appendix V: Review of Michigan?s REACH Project Evaluation Report
Page 47 GAO- 01- 723 REACH Program Effectiveness
cannot be made. In addition, there were limited months available on which to
base pre- and postintervention estimates, ranging from 2 to 4 months.
Moreover, the indication of a decrease in electric usage for one location
was based on data that were not adjusted for effects of changes in weather.
In addition, the report had other shortcomings that precluded an assessment
of the project?s overall effectiveness.
The first Michigan subproject focused on changing participant behavior
through education. The inability to obtain quality data on participants?
energy use from the service providers and the local energy supplier-
especially data on postintervention energy use- meant that conclusions were
based on relatively few records. For example, postintervention data were not
available on gas use for four of the seven locations and not available for
electricity use for five locations. Moreover, the postintervention data did
not cover an entire year- data were available for only 2 to 4 months.
Overall attrition was also an issue. The evaluation report states that 1,028
participants were served by seven locations from October 1997 to September
1999 but that pre- or postintervention fuel consumption data were usable for
570 participants. Matched pre- and postintervention fuel data were available
for 93 participants using natural gas and 116 participants using
electricity. These data come from three of the seven locations. These
limited data restrict inferences that could be drawn from the originally
intended locations.
Additionally, attrition occurred within the locations for which pre- and
postintervention data were available. One of the three locations served 200
participants and obtained 34 useable natural gas records and 43 electric
records. The second location served 156 participants and obtained 40 useable
natural gas records and 73 electric records. The third location served 150
participants and obtained 19 useable natural gas records and no usable
electricity data. This level of attrition raises the issue of bias because
participants that remain in the project may differ from those dropping out
in ways that are likely to affect the outcome. The high attrition rate also
reduces the precision of any statistical estimate of change in fuel usage.
Even if sufficient data had been available for statistical analysis, not
having a valid control group for comparison greatly limited the ability to
make a valid assessment of project effects. Similarly, failure to adjust
Analytical Problems
Appendix V: Review of Michigan?s REACH Project Evaluation Report
Page 48 GAO- 01- 723 REACH Program Effectiveness
energy use data for weather conditions prevented a valid assessment of
participant energy use.
In addition to not including all of the information needed to assess the
statistical analysis and validity of its findings, the evaluation report did
not provide other important information needed to assess the project?s
overall effectiveness. For example, the report did not address the project?s
performance in terms of the project performance goal provided in the
authorizing legislation and the HHS program announcement: increasing energy
suppliers? contributions to reducing the energy burden of eligible
households. The report also did not provide a complete discussion of the
project design, including critical project assumptions, hypotheses to be
tested in terms of measurable objectives, or lessons learned from
implementing the design. Other Shortcomings
Appendix VI: Review of Nebraska?s REACH Project Evaluation Report
Page 49 GAO- 01- 723 REACH Program Effectiveness
The primary goal of the Nebraska REACH project was to increase the economic
self- sufficiency of low- income families by increasing household energy
efficiency. The project sought to decrease participants? household utility
bills by reducing energy use. A second project goal was to increase
participants? knowledge of energy conservation practices and personal
finance. Additional project goals included increasing the environmental
comfort and health and safety of the participating households.
The Nebraska REACH project was to focus on changing the behavior of the
target population- low- income households- as the best way of achieving
project goals. This approach was based on the assumptions that low- income
households were most often renters who frequently moved and that the housing
available to them was generally substandard. Given these premises, it was
assumed that the only way to reduce energy use in this population would be
to change household behavior through teaching energy- efficient practices.
It was also assumed that it would have been too expensive to weatherize
substandard housing, and upgrading the structure would have benefited the
household only temporarily, until its next move.
According to the evaluation report, the Nebraska REACH project provided a
unique opportunity to evaluate a social services project using a ?true
experimental design.? The project was designed to use two groups of
lowincome households- a test group and a control group- to evaluate the
primary hypothesis that changing behavior could reduce energy use. Data
collected on the two groups, both before and after project implementation,
would be analyzed using a standard computer program- the Princeton
Scorekeeping Method (PRISM) software- to determine the statistical
significance of the results. The PRISM software uses data from monthly
utility bills to produce a weather- adjusted index of energy use for both
the test and control groups. The program also compares the differences in
energy use among households in the test group and between households in the
test group and control group. Members of both the test group and the control
group would be required to sign forms authorizing the release of their
utility bills to the project.
In addition, knowledge of energy conservation practices and personal
financial management obtained from the workshops would be measured Appendix
VI: Review of Nebraska?s REACH
Project Evaluation Report Goals and Measures
Project Design and Implementation
Project Assumptions Evaluation Design
Appendix VI: Review of Nebraska?s REACH Project Evaluation Report
Page 50 GAO- 01- 723 REACH Program Effectiveness
using pre- and postknowledge tests and a third test given 6 months after
project completion.
Households were to receive four related services: (1) a home energy audit,
(2) instruction in energy conservation practices and personal financial
management, (3) case management, and (4) provision of basic home
weatherizing materials. Home energy audits would include an interview and
inspection by a certified energy specialist to assess the household?s energy
efficiency and form the basis of the action plan. Workshops providing
instruction on energy conservation and personal finance would be available
to participants each month, and training manuals would be distributed
providing detailed instruction on these issues. Case management would
include developing a household action plan, preparing family assessments,
and conducting monthly progress reviews during home visits. Materials such
as weather stripping, along with instruction on installation, would also be
provided. In addition to these services, project participants would receive
between $200 and $350 in vouchers to pay utility bills. Control group
members would receive $50 in vouchers for utility bills.
Potential applicants were recruited through advertisements and social
service agency referrals. Case managers determined project eligibility using
four criteria: participants had to (1) have income at or below the poverty
level, (2) have resided at their current address for at least 1 year, (3)
not be planning to move, and (4) need assistance in paying utility bills.
The control group was randomly selected from every third eligible applicant.
There were no significant differences between the two groups in either
education or other demographic characteristics. During implementation, the
project provided services and cash assistance to 439 participants and cash
assistance and goods to 202 control group members.
Separate analyses were done for natural gas and electricity use using the
PRISM computer program. PRISM provided normalized annual consumption data
(pre- and postproject) and normalized annual savings separately for both gas
and electricity. Services to Be Provided
REACH Participant and Control Group Selection
Evaluation Findings and Reported Limitations Findings
Appendix VI: Review of Nebraska?s REACH Project Evaluation Report
Page 51 GAO- 01- 723 REACH Program Effectiveness
The PRISM analysis indicated a statistically significant decreased use of
natural gas by REACH project participants compared with those in the control
group. Program participants had normalized annual savings of $48.12 per
household more than the control group- or a savings of 122 hundred cubic
feet.
The result of the PRISM analysis for electricity use revealed a different
use pattern. Although the analysis showed no change in electricity use by
project participants, the analysis revealed an annual increase in
electricity use of 2,716 kilowatt hours by the control group. The evaluation
report stated that this effect was consistent with the hypothesis that
electricity use would be lower for project participants- an annual savings
of $149.38 per household.
On the basis of these findings, the evaluation report concluded that project
participants had achieved greater economic self- sufficiency through a
reduction in utility costs- an average of $197.50 each year- compared with
the nonparticipant control group. However, the report also noted that an
actual reduction in utility use for project participants was confirmed only
for natural gas and not for electricity.
In addition, given the results of the knowledge tests administered to
participants, the report concluded that the instruction provided during the
project resulted in a significant increase in knowledge about energy
conservation practices and personal financial management.
The evaluation report attributed the project?s ability to implement the
project design as planned and to obtain sufficient data to perform the
statistical analysis to the community- based organization?s experience,
expertise, and knowledge of the target population and to the early
involvement of the evaluator in the project?s design. Cash incentives
provided to the test and control groups also appeared to have reduced
participant attrition. In addition, administrative controls designed to
ensure access to utility bills seemed to have played a role in obtaining
sufficient data for analysis.
The evaluation report noted several factors that greatly reduced the number
of households whose energy use data could be used in the PRISM analysis. For
example, frequent changes in residence eliminated many households from the
analysis. In addition, frequent changes in household size and composition,
along with utility shutoffs, made much of the data Reported Limitations
Appendix VI: Review of Nebraska?s REACH Project Evaluation Report
Page 52 GAO- 01- 723 REACH Program Effectiveness
too unreliable for use in the PRISM analyses; only a small percentage of the
database was considered appropriate for analysis.
According to the evaluation report, analysis was limited to determining the
overall effect of the project, not the effect of specific services or
combination of services (i. e., identifying which services were most
effective in achieving a reduction in energy use.)
The effectiveness of the Nebraska REACH project can be assessed with some
qualification, given the information presented in the evaluation report. The
project?s design and implementation allowed statistically valid conclusions
to be made about the effect of project services on participant energy use.
The report did not address the two other performance goals of reducing
utility bill arrearages and increasing contributions by energy suppliers and
did not provide other information needed to fully assess the project?s
overall effectiveness.
The evaluation report did not provide an assessment of potential bias due to
attrition, which may result if those remaining in the project systematically
differ from those who dropped out, and its possible effect on the analyses
and conclusions.
Although the evaluation report provided a statistical analysis regarding the
project?s main hypothesis, the report did not address two of the project
performance goals identified in the authorizing legislation and the HHS
program announcement. Specifically, the report did not address the
performance goal of increasing the regularity of utility bill payments or
the performance goal of increasing contributions by energy suppliers.
Finally, the evaluation report also did not state the hypotheses in terms of
measurable objectives. The report did not provide a discussion of lessons
learned or best practices that could be valuable to other projects. For
example, if the report had discussed the strategies used to successfully
collect energy use data, this information could have been useful to other
projects. This lack of information is often cited as a principle reason for
not being able to measure project performance. Our Observations
Analytical Problems Other Shortcomings
Appendix VII: Review of Oregon?s REACH Project Evaluation Report
Page 53 GAO- 01- 723 REACH Program Effectiveness
The primary goal of the Oregon REACH project was to help low- income
households manage their energy costs more effectively. The evaluation report
identifies four related project goals:
reducing household energy use;
reducing household energy cost;
increasing the regularity of utility payments, reducing arrearages in
utility bills, or both; and
eliminating health and safety risks related to energy use. The long- term
objectives of the project were described as sustained reduced energy use,
overall improvement in economic self- sufficiency, and eventual elimination
of reliance on energy assistance payments.
Three primary measures were used in the evaluation report to assess the
impact of the services provided by the project:
Reducing household energy use- 75 percent of participating households will
reduce energy use by 15 percent.
Reducing household energy burden- no operational objective stated.
Reducing arrearages in utility bills- 75 percent of participating
households will reduce arrearages in utility bills, and 50 percent of
households will not incur new arrearages for 6 months.
Other measures used in the evaluation report addressed program activities,
such as completing action plans and enrolling participants in social service
programs.
The Oregon REACH project was based on the premise that providing services to
low- income households- such as information on energy conservation practices
and personal financial management- would result in changes in their behavior
that would reduce their energy costs and utility bill arrearages. The
evaluation report identified three general assumptions that guided
development of the project:
Coordinating services provided by organizations within the community would
be more effective than uncoordinated assistance. Appendix VII: Review of
Oregon?s REACH
Project Evaluation Report Goals and Measures
Project Design and Implementation
Project Assumptions
Appendix VII: Review of Oregon?s REACH Project Evaluation Report
Page 54 GAO- 01- 723 REACH Program Effectiveness
Household knowledge of energy conservation, as well as assistance in
weatherizing homes, is necessary to reduce reliance on energy assistance
payments.
Households that participated in the project would be willing and able to
make changes in their behavior and personal financial management that could
affect their energy costs and the regularity of their utility payments.
According to the evaluation report, the Oregon REACH project was conceived
as a ?quasi- experimental? design. The project was designed as an experiment
to use two test groups and a control group to determine the effect of the
services provided. The report noted that the inclusion of a control group
was one of the distinctive features of the design.
A different combination of services would be provided to the two test
groups. One test group would receive a complete set of services- including
home weatherizing and heating system repairs- as needed. The second test
group would receive all of the project services, as needed, with the
exception of home weatherizing and heating system repairs. Households within
each group would receive different combinations of services on the basis of
need and availability of the service in the community. The control group
would not receive any of the project services.
Data on energy use would be collected for all three groups, when possible,
for the year preceding the project, the year of project implementation, and
the year after project completion. Members of each group would be required
to sign forms authorizing the release of their utility bills to the project.
These data would be used to evaluate energy use, utility arrearages, and
energy burden by performing a statistical analysis of the differences among
groups.
Both participants and caseworkers would be surveyed after project
implementation to obtain their views on the project?s implementation and
usefulness. An incentive payment of $20 would be given to households to
complete the survey.
Services were to be provided to participating households by 13 community-
based organizations across the state. Services would be provided through
case management that focused on working with households to develop an action
plan for reducing their energy use according to an assessment of their
needs. Case workers would also Evaluation Design
Services to Be Provided
Appendix VII: Review of Oregon?s REACH Project Evaluation Report
Page 55 GAO- 01- 723 REACH Program Effectiveness
provide instruction to households in energy conservation practices and
personal financial management, facilitate negotiation with energy suppliers
to develop payment plans for reducing arrearages, and make referrals to
other organizations providing social services.
Other services to be provided would include financial assistance to help pay
for utility bills and reduce arrearages, home energy audits, and home
weatherizing assistance. Project funds were also to be used to replace water
heaters, furnaces, and thermostats and to provide carbon monoxide detectors
and heating repairs, as needed. An emergency payment of $200 was to be
allowed for especially needy households.
Households were selected from those already receiving energy assistance.
Households also had to have utility bill arrearages equal to one- half the
energy assistance payment, an energy burden greater than 15 percent of
income, and an energy- related health or safety risk. In addition,
participants were selected on the basis of their motivation and the
priorities of local communities. According to the report, the control group
was selected from energy assistance recipients at each participating
community- based organization. The sampling procedure was not discussed.
According to the evaluation report, the Oregon REACH project was largely
successful in achieving two of its primary goals: reducing energy use and
reducing arrearages in household energy bills while increasing the
regularity of payments to energy providers. The report noted that the
project strongly supported the assumption that coordinated services
effectively reduce energy use, energy costs, and energy burden for lowincome
households. In addition, the report stated that its analysis confirmed that
the services helped households reduce arrearages and increase the regularity
of utility bill payments. The evaluation report assessed the effect of
services provided to households completing the project. The evaluation
report stated that participant attrition was about 8 percent in the last
half of the first year and 16 percent in the second year of the project.
REACH Participant and
Control Group Selection Evaluation Findings and Reported Limitations
Findings
Appendix VII: Review of Oregon?s REACH Project Evaluation Report
Page 56 GAO- 01- 723 REACH Program Effectiveness
The evaluation report found that both test groups reduced their electricity
use by 11 percent for the year after project implementation. Of the 173
households receiving services, 58 (or 33.5 percent) used less energy, and 40
(or 23 percent) achieved a reduction of at least 15 percent. This was 52
percent short of the project goal of 75 percent of households achieving the
15- percent reduction. In addition, households were found to have reduced
their energy burden by 2.5 percent.
The evaluation report also found that both test groups reduced arrearages in
their utility bills as a result of participation in the project and that
this difference was statistically significant. According to the report, the
project goal that 50 percent of households not incur new arrearages for 6
months was met, and the number of households with arrearages decreased from
59 percent (102 participants) to 36 percent (63 participants). Moreover, the
evaluation report found that households receiving the additional services of
weatherizing and repairs achieved a slightly larger reduction in their
energy bill arrearages than the test group not receiving those services- $77
compared with $55- but noted that the difference was not statistically
significant.
In the exit surveys, most households indicated that the project had helped
make their homes healthier, safer, and more comfortable and energyefficient.
Similarly, in the staff survey, case workers indicated satisfaction with the
project.
As a result of all these findings, the evaluation report concluded that the
Oregon REACH project greatly assisted low- income households in achieving a
greater degree of energy self- sufficiency.
The evaluation report did not specifically identify limitations other than
to note that the difference in the energy use among the groups before
project implementation was statistically significant, suggesting that the
households may not have been assigned to the test and control groups in a
random fashion. The control group had the lowest level of energy use both
before and after project implementation.
The evaluation report also noted that not all households received the full
benefits because they did not complete the project and that it was difficult
to contact some participants because they did not keep appointments or did
not have telephone service. Two hundred and twenty four households were
listed as members of the test groups, and fewer households were Reported
Limitations
Appendix VII: Review of Oregon?s REACH Project Evaluation Report
Page 57 GAO- 01- 723 REACH Program Effectiveness
used in the various analyses. Data were not available for participants
entering the project in the second year.
The effectiveness of the Oregon REACH project cannot be determined on the
basis of the information presented in the project evaluation report.
Although the study did have a control group, the omission of a comparison of
the test groups with the control group is a shortcoming in the analyses:
Only the energy consumption analysis compared the control group directly
with the REACH groups.
The assessments of both energy burden and change in arrearages used two
REACH test groups but did not compare changes between the two groups.
Instead, they assessed changes from pre- REACH to post- REACH project points
within each of the groups separately.
The study did not assess the impact of the project by comparing the change
in energy use of the two REACH groups to the change in energy use of the
control group.
The report also noted that participants might not have been assigned to
groups in a random manner, leaving the possibility that the changes observed
could be the result of factors other than the services provided. Finally,
the report had other shortcomings that precluded an overall assessment of
the project?s effectiveness.
The evaluation report did not identify important limitations of the analysis
or their effect on the conclusions. As noted, failure to compare the test
groups and the control groups for assessment of changes in energy use and
arrearages limits the usefulness of the findings. The report also did not
address the effect of the criteria and procedure for selecting both the test
and control groups, as well as the attrition rates from these groups. These
factors also adversely affect the report?s analysis and conclusions. For
example, assignment to the test and control groups appears not to have been
on a random basis. Test group selection resulted in a group of households
with the highest energy use, whereas control group selection resulted in a
group of households with the lowest energy use. Such assignment issues
weaken the conclusions that can be made about the effect of REACH services
on energy use. In addition, the report did not state whether energy use data
were adjusted for weather. Our Observations
Analytical Problems
Appendix VII: Review of Oregon?s REACH Project Evaluation Report
Page 58 GAO- 01- 723 REACH Program Effectiveness
In addition to not including all of the information necessary to assess the
statistical analysis and validity of the project?s results, the evaluation
report did not provide other important information needed to assess the
project?s overall effectiveness. For example, the report did not address one
of the project performance goals stated in the authorizing legislation and
the HHS program announcement- increasing energy suppliers? contributions to
reducing the energy burden of eligible households. In addition, the report
did not provide information on the dollar amount of the expected reduction
in energy use or the amount of financial assistance given to households.
Finally, the evaluation report did not provide a discussion of lessons
learned. A discussion of best practices- the techniques, procedures, and
controls- used to ensure sufficient data for analysis might have been useful
to future projects. Other Shortcomings
Appendix VIII: Comments From the Department of Health and Human Services
Page 59 GAO- 01- 723 REACH Program Effectiveness
Appendix VIII: Comments From the Department of Health and Human Services
Note: GAO comments supplementing those in the report text appear at the end
of this appendix.
Appendix VIII: Comments From the Department of Health and Human Services
Page 60 GAO- 01- 723 REACH Program Effectiveness
See comment 1. Page numbers in the draft report may differ from those in
this report.
Appendix VIII: Comments From the Department of Health and Human Services
Page 61 GAO- 01- 723 REACH Program Effectiveness
See comment 2.
Appendix VIII: Comments From the Department of Health and Human Services
Page 62 GAO- 01- 723 REACH Program Effectiveness
See comment 3.
Appendix VIII: Comments From the Department of Health and Human Services
Page 63 GAO- 01- 723 REACH Program Effectiveness
Appendix VIII: Comments From the Department of Health and Human Services
Page 64 GAO- 01- 723 REACH Program Effectiveness
See comment 2. See comment 2.
Appendix VIII: Comments From the Department of Health and Human Services
Page 65 GAO- 01- 723 REACH Program Effectiveness
Appendix VIII: Comments From the Department of Health and Human Services
Page 66 GAO- 01- 723 REACH Program Effectiveness
See comment 6. See comment 5. Now on pp. 16- 17.
See comment 4.
Appendix VIII: Comments From the Department of Health and Human Services
Page 67 GAO- 01- 723 REACH Program Effectiveness
Now on p. 17. See comment 7.
Appendix VIII: Comments From the Department of Health and Human Services
Page 68 GAO- 01- 723 REACH Program Effectiveness
See comment 8.
Appendix VIII: Comments From the Department of Health and Human Services
Page 69 GAO- 01- 723 REACH Program Effectiveness
Appendix VIII: Comments From the Department of Health and Human Services
Page 70 GAO- 01- 723 REACH Program Effectiveness
The following are GAO?s comments on the Department of Health and Human
Services? letter dated July 23, 2001.
1. The technical comments have been incorporated as appropriate. Several
more substantive technical comments are addressed individually below.
2. The authorizing legislation defines home energy as ?a source of heating
or cooling in residential dwellings.? While HHS argues that residential
energy commonly refers to all household energy use, for the purposes of this
review, we used the narrower definition stated in the legislation. We have
also changed the wording of our third recommendation, concerning the use of
REACH program funds, to clarify that REACH program funds should be used for
activities directly related to the home heating and cooling needs of low-
income households. We do not dispute that a broad range of services is
authorized in the legislation, but these services should relate to home
heating and cooling.
3. We are not including the possibility of expanding the program in our
matters for congressional consideration because the effectiveness of the
REACH program has not yet been determined and because we believe that HHS
should be required to report to the Congress in 2003 on the REACH program?s
effectiveness.
4. While coordination among programs is addressed in a general way in the
REACH program announcements, we believe that HHS should more specifically
articulate its view of the REACH program?s purpose and relationship to
LIHEAP in the Administration for Children and Families? performance plan. We
also note that a performance plan is more readily accessible to the
Congress, the public, and other agencies than the program announcements
intended for potential grant applicants.
5. Our draft report did not state that the use of randomized control groups
is the only acceptable design, so we have not made any wording changes.
Although there are legitimate questions concerning using randomized control
group designs in social service programs, such designs have sometimes been
effectively used. For instance, when resources are not sufficient to provide
services to all who are eligible, random assignment to control groups can be
feasible and ethical and provide convincing results. When randomized control
groups are not feasible, evaluations can be designed to use GAO Comments
Appendix VIII: Comments From the Department of Health and Human Services
Page 71 GAO- 01- 723 REACH Program Effectiveness
nonequivalent control groups, and, with suitable caveats, such designs may
yield defensible results. Regardless of the design used, evaluations of
project results must be able to distinguish between results that are most
likely due to the project services and results that may be due to other,
external factors.
6. We have added language indicating that the Office of Community Services
is aware of the evaluation challenges that we cite.
7. Pages 14 and 15 of our report describe the legislative authority and
restrictions on the use of LIHEAP funds, as well as the need to use much of
LIHEAP?s funding for direct assistance with energy bills. We also report
that, as allowed by law, 25 states plan to use a portion of their fiscal
year 2001 LIHEAP funds for activities such as energy education and budget
counseling and 44 states plan to provide weatherization services. As HHS?
comments note, opportunity exists to continue some of the activities tested
under REACH grants through states? LIHEAP funds. We have made minor wording
changes to more clearly recognize that not all REACH activities may fall
within the legislative and funding constraints of the LIHEAP program.
8. Coordinating with other programs is not the same as spending REACH grant
funds for non- energy- related activities. We believe that HHS? planned
efforts to clarify language in its program announcement should help ensure
that REACH funds are used only for activities directly related to the home
energy needs of low- income households and should provide a clearer basis
for reviewing and selecting proposals.
Appendix IX: GAO Contact and Staff Acknowledgments
Page 72 GAO- 01- 723 REACH Program Effectiveness
Jim Wells (202) 512- 3841 In addition, Julie Gerkens, Kathleen Gilhooly,
Curtis Groves, Rachel Hesselink, Judy Pagano, and Don Pless made key
contributions to this report. Appendix IX: GAO Contact and Staff
Acknowledgments GAO Contact Acknowledgments
(360030)
The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting:
Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013
Orders by phone:
(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet
For information on how to access GAO reports on the Internet, send an e-
mail message with ?info? in the body to:
Info@ www. gao. gov or visit GAO?s World Wide Web home page at: http:// www.
gao. gov
Contact one:
Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov
1- 800- 424- 5454 (automated answering system) Ordering Information
To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document. ***