Healthy Start: Preliminary Results From National Evaluation Are Not
Conclusive (Letter Report, 06/15/98, GAO/HEHS-98-167).

Pursuant to a congressional request, GAO reviewed the preliminary
results of a national evaluation of the Healthy Start program, focusing
on: (1) the plan for national evaluation; (2) what Mathematica Policy
Research, Inc.'s (MPR) preliminary evaluation results indicate; and (3)
what is expected from the final evaluation.

GAO noted that: (1) MPR's preliminary reports from the national
evaluation of Healthy Start do not provide a complete assessment of the
program and, therefore, should not be used to judge the program success;
(2) even the final report will not contain all the data expected to be
analyzed for the national evaluation; (3) if the evaluation plan were
expanded to include data from the sixth and final year of the
demonstration, conclusions about whether the program has met its goals
of reducing infant mortality could be strengthened; (4) the national
evaluation of the Healthy Start program had two major components: (a) an
impact evaluation, to determine whether the infant mortality rates in
Healthy Start communities have declined; and (b) a process evaluation to
describe how the program actually operates; (5) once these evaluations
are completed, MPR plans to link outcomes with processes in its final
report to determine why Healthy Start has or has not succeeded and what
would be required for a similar intervention elsewhere; (6) while MPR's
draft report on its impact evaluation suggests that Healthy Start has
little effect in reducing infant morality in targeted communities,
drawing such a conclusion at this time would be premature for several
reasons; (7) the process evaluation is also incomplete; (8) only some of
the reports that it comprises are available; (9) eventually, MPR plans
to cover program implementation at all sites, the characteristics of
program participants, and details about some of the most important
strategies used by the program; (10) with these two major components of
the evaluation in preliminary stages or incomplete, MPR cannot yet
relate process to impact; (11) the final evaluation is expected to
include an analysis of infant morality data from the original 5 years of
the demonstration for all 15 sites; (12) however, the final report on
the evaluation, now planned for early 1999, will include data from only
the first 4 years; and (13) further, because implementation of the
program was slower than anticipated, and the program was mature for
fewer years of the original demonstration period than planned for in the
evaluation, even results from the final report are likely to be
inconclusive and should be considered preliminary.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  HEHS-98-167
     TITLE:  Healthy Start: Preliminary Results From National Evaluation 
             Are Not Conclusive
      DATE:  06/15/98
   SUBJECT:  Infants
             Program evaluation
             Health care programs
             Community health services
             Pregnancy
             Prenatal care
             Statistical data
IDENTIFIER:  Healthy Start Program
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to Congressional Requesters

June 1998

HEALTHY START - PRELIMINARY
RESULTS FROM NATIONAL EVALUATION
ARE NOT CONCLUSIVE

GAO/HEHS-98-167

Preliminary Evaluation of Healthy Start

(108354)


Abbreviations
=============================================================== ABBREV

  HHS - Department of Health and Human Services
  HRSA - Health Resources and Services Administration
  MPR - Mathematica Policy Research

Letter
=============================================================== LETTER


B-279764

June 15, 1998

The Honorable Arlen Specter
Chairman
The Honorable Tom Harkin
Ranking Minority Member
Subcommittee on Labor, Health
 and Human Services, and Education
Committee on Appropriations
United States Senate

Over the past decade, the rate of infant mortality in the United
States has steadily declined.  In 1987, there were 10.1 deaths per
1,000 live births; by 1996, the rate had dropped to 7.2 deaths.  Yet,
U.S.  infant mortality rates have consistently been higher than those
of many other developed countries.  In addition, there are large
racial differences in infant mortality rates in this country--in
1996, for example, the mortality rate for black infants was more than
twice that for whites.  Medical interventions can potentially address
some of the leading causes of infant death.  However, it is thought
that poverty, inadequate community services, and educational factors
prevent some women from gaining access to appropriate medical care. 

In an effort to reduce the nation's infant mortality rate, in 1991
the Health Resources and Services Administration (HRSA) of the
Department of Health and Human Services (HHS) initiated the Healthy
Start program, which encourages community-based strategies for
reducing infant mortality.  The program was planned to be a 5-year
demonstration project, and initially 15 communities were awarded
Healthy Start grants.\1 In 1996, the demonstration phase was
continued for a sixth year.  Although, the demonstration phase is
considered to have been concluded, Healthy Start has continued into
its seventh year.  Over the life of the program, 48 communities have
been added to the original 15.  Under the Healthy Start initiative,
HRSA planned for an evaluation of the program to determine whether it
had reduced infant mortality.  In 1993, HRSA contracted with
Mathematica Policy Research (MPR), Inc., to conduct a national
evaluation of the 5-year demonstration in the original 15 sites. 
Although the evaluation is not complete, recent press reports have
presented conflicting stories on what MPR's preliminary evaluation
results indicate about Healthy Start. 

In light of these events, you asked us to (1) describe the plan for
the national evaluation, (2) determine what MPR's preliminary
evaluation results indicate, and (3) describe what is expected from
the final evaluation.  To conduct our work, we reviewed the national
evaluation's preliminary results, available evaluation reports, and
plans for future evaluations.  We also interviewed program officials
and MPR's principal investigators and visited Philadelphia and
Baltimore--2 of the original 15 Healthy Start communities recommended
by HRSA as exemplifying alternative approaches to project
organization.  Other elements of the evaluation of Healthy Start,
such as the local evaluations, are beyond the scope of our work.  We
conducted our study from January to May 1998 in accordance with
generally accepted government auditing standards. 


--------------------
\1 These sites were all or parts of Baltimore, Birmingham, Boston,
Chicago, Cleveland, Detroit, the District of Columbia, New Orleans,
New York, Northern Plains (19 Native American tribal organizations in
four states), Northwest Indiana, Oakland, the Pee Dee Region of South
Carolina, Philadelphia, and Pittsburgh. 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :1

MPR's preliminary reports from the national evaluation of Healthy
Start do not provide a complete assessment of the program and,
therefore, should not be used to judge program success.  Even the
final report, likely to be delayed until early 1999 because of
difficulties in acquiring vital record data from the states, will not
contain all the data expected to be analyzed for the national
evaluation.  If the evaluation plan were expanded to include data
from the sixth and final year of the demonstration, conclusions about
whether the program has met its goal of reducing infant mortality
could be strengthened. 

The national evaluation of the Healthy Start program has two major
components:  an impact evaluation, to determine whether infant
mortality rates in Healthy Start communities have declined, and a
process evaluation to describe how the program actually operates. 
Once these evaluations are completed, MPR plans to link outcomes with
processes in its final report to determine why Healthy Start has or
has not succeeded and what would be required for a similar
intervention elsewhere. 

While MPR's draft report on its impact evaluation suggests that
Healthy Start has had little, if any, effect in reducing infant
mortality in targeted communities, drawing such a conclusion at this
time would be premature for several reasons.  First, the results of
the impact evaluation reflect the experience at only 9 of the 15
program sites to be evaluated.  Second, the data are for only the
first 3 of the 6 years of the demonstration.  Third, since full
implementation of the program took longer than anticipated at most
sites, the analyses reflect the earliest years of the program, before
it was fully operational. 

The process evaluation is also incomplete.  Only some of the reports
that it comprises are available.  Eventually, MPR plans to cover
program implementation at all sites, the characteristics of program
participants, and details about some of the most important strategies
used by the program.  With these two major components of the
evaluation in preliminary stages or incomplete, MPR cannot yet relate
process to impact. 

The final evaluation is expected to include an analysis of infant
mortality data from the original 5 years of the demonstration for all
15 sites.  However, the final report on the evaluation, now planned
for early 1999, will include data from only the first 4 years.  The
fifth year will be included in an addendum to this report, to be
submitted by MPR a year later.  Further, because implementation of
the program was slower than anticipated, and the program was mature
for fewer years of the original demonstration period than planned for
in the evaluation, even results from the "final report" are likely to
be inconclusive and should be considered preliminary.  While the
fifth year of data to be reported in the addendum will be helpful, we
are recommending that a sixth year of data be used to make the
results more conclusive. 


   BACKGROUND
------------------------------------------------------------ Letter :2

In 1991, HRSA announced that it would fund 10 Healthy Start sites and
issued guidance on how communities could obtain a grant.  By July
1991, HRSA had received 40 applications, and in September of that
year, it began funding 15 communities for a 5-year demonstration
project.  In 1996, funding for these communities was extended for a
sixth year.  In 1994, HRSA began funding seven new
communities--called special projects--and funding for these was also
extended in 1996 for an additional year.  Forty-one additional
communities have been awarded grants since 1997, and these now share
funding with the 15 original sites and 5 of the special projects
judged by HRSA to have been successful. 

To be eligible for the original grants, a community had to have an
average annual infant mortality rate of at least 1.5 times the
national average between 1984 and 1988--that is, 15.7 deaths per
1,000 live births--and at least 50 but no more than 200 infant deaths
per year.  Applicants had to be local or state health departments,
other publicly supported provider organizations, tribal
organizations, private nonprofit organizations, or consortia of these
organizations.  HRSA required only a few specific activities of all
sites to provide grantees flexibility to make their projects relevant
to local circumstances.\2

Healthy Start's principal goal to reduce infant mortality has usually
been stated as a 50-percent reduction in infant mortality,
attributable to the program, over 5 years.\3 Healthy Start also aims
to achieve improvements in other outcomes--such as reductions in low
birthweight, improved maternal health, and increased community
awareness of threats to infant health--that are expected to help
reduce infant mortality.  In addition, Healthy Start was designed to
demonstrate how a program based on innovation, community commitment
and involvement, increased access to care, service integration, and
personal responsibility could work in a variety of locations with
high infant mortality. 

From fiscal year 1991 (a planning year that preceded the 5-year
demonstration), through fiscal year 1998, program funding for Healthy
Start has totaled more than $600 million.  Healthy Start's fiscal
year 1992 funding was less than half of what was initially proposed,
and the number of grantees was greater.  Instead of $171 million
being spread over 10 sites, funding for the first year of the
demonstration was $64 million spread over 15 sites. 

In 1997, HRSA concluded the demonstration phase of Healthy Start and
began the "replication phase" in 40 (now 41) new sites.  In addition
to providing Healthy Start services in their own communities, the
established Healthy Start communities--the original 15 sites and 5 of
the special projects--are mentoring several of the new sites.  While
the new sites receive, on average, somewhat less funding than the
established sites, funding is shared among all sites. 


--------------------
\2 Grantees were to focus on reducing infant mortality, include the
community in planning, assess local needs, increase public awareness
of issues related to infant mortality, conduct a review of infant
mortality in their area, develop innovative services for pregnant
women and infants, and monitor their progress. 

\3 HRSA officials acknowledge that the goal of 50-percent reduction
was intended to be "motivational." Other experts have asserted that
this goal is not realistic.  See D.  Strobino and others, "A
Strategic Framework for Infant Mortality Reduction:  Implications of
'Healthy Start,'" The Milbank Quarterly, Vol.  73 (1995), pp. 
507-33, in which it is demonstrated that the maximum reduction in
infant mortality that possibly could be achieved as a result of a
number of specific interventions, not all of which were implemented
in all Healthy Start sites, is 32 percent. 


   THE NATIONAL EVALUATION
   CONSISTS OF BOTH IMPACT AND
   PROCESS EVALUATIONS
------------------------------------------------------------ Letter :3

In September 1993, HRSA contracted with MPR to conduct the national
evaluation of the Healthy Start program.  This is currently funded
with about $4.8 million, paid from the 1-percent set-aside for
evaluation of health programs.\4 The original contract called for MPR
to evaluate the first 4 years of the 5-year demonstration program and
contained an option for HRSA to request evaluation of the fifth year. 
In 1995, HRSA exercised that option, and the contract now requires
that the evaluation cover all 5 years of the originally planned
demonstration.  Although the demonstration was extended for a year,
HRSA currently has no plans to request that MPR evaluate the sixth
and final year of the demonstration phase.\5

The national evaluation, focused only on the original 15 sites, is
designed to determine whether Healthy Start changed the rate of
infant mortality and related outcomes, what factors contributed to
any effects the program may have had, and how successful approaches
to lessening infant mortality can be replicated in other
communities.\6 Although each Healthy Start community is unique and
the details of the delivery of any one service may differ across
communities, many of the services are common to all sites:  outreach
and case management; support services, such as transportation and
nutrition education; enhancements to clinical services; and public
information campaigns. 

The national evaluation has two major components:  an impact
evaluation and a process evaluation.  The impact evaluation is used
to determine whether the infant mortality rates in Healthy Start
communities have declined and whether related outcomes have improved. 
The process evaluation describes how the program actually operates. 
In its final evaluation report, MPR intends to synthesize these two
components, linking outcomes with processes to determine why Healthy
Start has or has not succeeded in communities and which strategies
are likely to be successful elsewhere. 


--------------------
\4 P.L.  91-296 allows the Secretary of HHS to use up to 1 percent of
the appropriations for programs authorized under the Public Health
Service Act. 

\5 In addition to the efforts of MPR and its subcontractors, a
15-member Technical Advisory Group, consisting of experts in the
fields of maternal and child health and program evaluation, provides
consultation to MPR on all aspects of the design and conduct of the
national evaluation. 

\6 The original design for the national evaluation was prepared in
1992 by Lewin-ICF, Inc., and MDS Associates, Inc.  It was reviewed by
the 15 Healthy Start projects included in the demonstration and
subsequently revised prior to issuance of the Request for Proposals. 
Early in the implementation of the national evaluation, the
methodology was modified to accommodate changes in the availability
of vital record data and the processes for which these and client
data were obtained. 


   MPR'S PRELIMINARY EVALUATION
   RESULTS WERE NOT CONCLUSIVE
------------------------------------------------------------ Letter :4

In the fall of 1997, MPR reported some preliminary evaluation
results, including a draft interim report on its impact evaluation,
which led to press accounts suggesting a variety of interpretations
about the success of the Healthy Start program.  We believe these
preliminary evaluation results were not conclusive.  Although the
impact evaluation suggested that Healthy Start has not reduced infant
mortality, such conclusions about the program would be premature
because the impact evaluation does not include data from all the
program sites or data from all the years of the program.  Moreover,
the process evaluation indicates that program implementation in many
communities was slow and, therefore, that the impact data analysis
may not be representative of a mature Healthy Start program. 


      PRELIMINARY ANALYSIS OF
      PROGRAM IMPACT MAY NOT BE
      MEANINGFUL BECAUSE OF THE
      LIMITED DATA AVAILABLE
---------------------------------------------------------- Letter :4.1

The national evaluation's analysis of Healthy Start's effect on
infant mortality and related outcomes is preliminary--MPR
characterized its October 1997 report as a draft.  Because of
problems obtaining data from some of the states' departments of
health, only 9 of the 15 program sites to be evaluated were
represented in the analysis.  In addition, the analysis is related to
only the first 3 of the 6 years of program operation.  Moreover, for
illustrative purposes, MPR has limited its principal impact analysis
to data from only the last of those 3 years, 1994.  However, if, as
HRSA believes, fiscal year 1995 was the first fully operational year,
even 1994 data may not reflect the communities' mature programs. 

To determine program impact, MPR is conducting two types of analysis: 
availability and participation.  The availability analysis compares a
Healthy Start community and two similar communities without Healthy
Start to determine if the presence of the program in a community has
an effect on infant mortality and related outcomes.  The
participation analysis compares, within a Healthy Start community,
mothers who were clients of the program and mothers who were not. 
Both analyses can be used to study infant mortality; however, the
availability analysis directly addresses the issue of reducing infant
mortality in entire communities, while the participation analysis is
restricted to outcomes for program participants. 

The national evaluation's availability analysis found that for 1994,
the overall infant mortality rate in Healthy Start communities was
about the same as that in comparison communities.  Applied to the
individual sites, the analysis found that of the nine Healthy Start
communities analyzed, only one experienced a significant reduction in
infant mortality relative to its comparison sites.  MPR similarly
found that the neonatal and postneonatal mortality rates--two
components of infant mortality--were not significantly reduced in the
Healthy Start communities relative to the comparison sites.\7 In its
analysis of birth outcomes considered to be risk factors for infant
mortality at eight of the Healthy Start communities, MPR found that
in 1994, the low birthweight rate was reduced in only one community,
the preterm birth rate was reduced in two other communities, and the
rate at which women received adequate or better prenatal care was
improved in five communities.\8 None of the analyses of data pooled
from all sites yielded significant differences between sites with and
without Healthy Start. 

The national evaluation's participation analysis of Healthy Start's
effect on infant mortality has not been completed because of problems
of data availability.  The participation analysis of related
outcomes, like the availability analysis, yielded little evidence of
program effect in the eight communities analyzed.  Participation in
Healthy Start was not associated with reductions in low or very low
birthweight rates or preterm birth rates.\9 In postpartum interviews
with participants and nonparticipants, conducted in 1996, after all
sites became fully operational, MPR found that participants were more
likely to rate their prenatal care experience more highly and to be
using birth control.  However, no significant differences between
participants and nonparticipants were reported for the receipt of
services or health behaviors during pregnancy. 


--------------------
\7 The neonatal mortality rate is the number of deaths of infants
under 28 days of age per 1,000 live births, and the postneonatal
mortality rate is the number of deaths of infants between 28 days and
1 year of age per 1,000 live births. 

\8 The low birthweight rate is the number of live births weighing
under 2,500 grams at birth per 100 live births.  The preterm birth
rate is the number of live births born at less than 37 weeks of
gestation per 100 live births.  The receipt of adequate or better
prenatal care rate is the number of women who received "adequate" or
"adequate plus" prenatal care, as defined by the Kotelchuck index of
prenatal care, per 100 live births.  See M.  Kotelchuck, "An
Evaluation of the Kessner Adequacy of Prenatal Care Index and a
Proposed Adequacy of Prenatal Care Utilization Index," American
Journal of Public Health, Vol.  84 (1994), pp.  1414-20. 

\9 The very low birthweight rate is the number of live births
weighing under 1,500 grams at birth per 100 live births. 


      PROCESS EVALUATION INDICATES
      THAT PROGRAM IMPLEMENTATION
      WAS MORE GRADUAL THAN
      ANTICIPATED
---------------------------------------------------------- Letter :4.2

The national evaluation's process evaluation is intended to provide a
detailed picture of what happened over time when Healthy Start was
implemented at the various sites and assess its success in meeting
its process objectives, such as hiring and retaining staff and
putting the planned program in place.  The evaluation, which,
according to HRSA's project officer for the evaluation, is to result
in a series of reports, indicates thus far that the Healthy Start
program was implemented largely as originally envisioned but more
gradually than expected. 

MPR's implementation report, a major portion of the process
evaluation, provides an overview of program implementation in 14 of
the 15 original sites and draws conclusions about these projects.\10
The report includes detailed information on the development of the
projects, the barriers to successful implementation, and the gaps
between what was planned and what resulted.  In addition, the report
presents perceptions of variations across projects with respect to a
variety of criteria, such as staff stability and consumer
participation in the process.  It also contains timelines indicating
for each site when specific program components became operative. 
These timelines demonstrate, for example, that only 4 of the 14 sites
had all their planned services operational by October 1994.  The
implementation report concludes with lessons learned, which are
organized into four categories:  community context, organization and
administration, community involvement, and service delivery. 

MPR has also completed a report on the infant mortality review
process at the various Healthy Start sites.  It indicates that, in
general, the review programs are operational but with varying degrees
of success in identifying the factors leading to infant mortality in
their communities. 

Two detailed reports on specific interventions are available only in
draft form.  One describes, in greater detail than the implementation
report, program participants, including comparisons of participants
and nonparticipants with respect to the use of health and social
services, satisfaction with services, and health-related behaviors,
such as birth control and breast feeding.  The other describes how
outreach and case management were delivered, with consideration given
to both similarities and differences across sites. 


--------------------
\10 E.  M.  Howell and others, The Implementation of Healthy Start: 
Lessons for the Future (Washington:  Mathematica Policy Research,
Nov.  1997).  A report on the project in the Northern Plains is to be
published separately because of the grantee's special circumstance as
the only grantee serving a Native American population and its complex
geographic and organizational structure, with 19 separate sites in
four states. 


      THE DEGREE TO WHICH SPECIFIC
      PROGRAM CHARACTERISTICS
      CONTRIBUTE TO REDUCTIONS IN
      INFANT MORTALITY IS NOT YET
      KNOWN
---------------------------------------------------------- Letter :4.3

Because the impact and process evaluations are not finished, their
synthesis has not yet begun.  MPR expects to be able to draw
conclusions about the program characteristics that are most effective
in improving maternal and child health outcomes and the circumstances
under which they are most likely to succeed when it integrates the
impact and process evaluations.  The synthesis of the impact and
process components of the evaluation to be presented in the final
report will be based on impact data from years one through four of
the demonstration.  This synthesis may have to be revised when more
impact data are available. 


   MPR'S FINAL REPORT WILL BE
   BASED ON FEWER YEARS OF IMPACT
   DATA AND PROGRAM OPERATION THAN
   ORIGINALLY PLANNED
------------------------------------------------------------ Letter :5

The final report as currently planned will not be the final
evaluation of Healthy Start.  The final report will contain an
analysis of outcomes through 1995 and a synthesis of this with the
findings of the process evaluation reports.  Thus, it will assess the
program's impact on infant mortality through the fourth year of the
demonstration, not the fifth year as planned.  Further, these data
will reflect the impact of only 1 or 2 years during which the program
was fully operational.  An addendum to the report, planned to follow
a year later, will contain an updated analysis of outcomes through
1996.  The addendum will assess impact on infant mortality through
the fifth year of the demonstration and thus will reflect the impact
of only 2 or 3 years during which the program was fully
operational.\11 However, by evaluating the sixth year of the
demonstration, it would be possible to obtain an analysis of 3 or 4
years of impact data from the mature program. 

Evaluating the sixth year of the demonstration would likely enhance
the value of the investment in MPR's evaluation for several reasons. 
First, including data from the sixth year of the demonstration would
allow evaluation of the years in which all 15 sites have been fully
operational.  Second, additional data would represent the effects of
a more mature and potentially more effective program, which would
likely provide more definitive answers about Healthy Start's success. 
Third, having more years of data would increase the likelihood of
detecting small but real effects of the program.  Further, it is
possible that data from the more mature years of the program will
reflect program impact on the wider Healthy Start community, not just
direct participants in program services and activities.  In addition
to hoped-for effects on the pregnant clients of Healthy Start, there
may be effects of program services and education on those same women
at other times, such as before or early in their next pregnancy;
indirect effects on their social network, such as their male
partners, friends, and sisters; and indirect effects on the community
in general. 

HRSA's project officer for the national evaluation notes that the
cost associated with analyzing results for an additional year would
be about $100,000; this would be inexpensive relative to the total
national evaluation cost of about $5 million.  Since states collect
vital records on births and deaths routinely, funds would be needed
only to obtain, analyze, and report on the data and to revise the
synthesis of these data with the process evaluation. 


--------------------
\11 According to HRSA officials, the agency anticipated that the data
to assess the program's impact on infant mortality might not be
available within the period of the national evaluation. 
Consequently, in its Request for Proposals for the evaluation, HRSA
asked for recommendations on how final year data might be acquired
and analyzed. 


   CONCLUSIONS
------------------------------------------------------------ Letter :6

Since the national evaluation of the Healthy Start program has yet to
be completed, preliminary results should not be used to conclude that
the program has or has not achieved its goals.  HRSA and MPR plan for
the "final" report of the national evaluation to include an extensive
description of the program, indicate whether it has reduced infant
mortality rates at Healthy Start sites, and provide an analysis of
how program characteristics have influenced outcomes.  However, the
final report will analyze infant mortality data from only 4 years of
the demonstration.  Primarily because implementation of the projects
was slower than anticipated, data from the first 4 years of the
demonstration may be insufficient for judging the success of Healthy
Start in lowering infant mortality.  Thus, even the final report will
be inconclusive.  Analysis of the fifth year of the demonstration, as
planned, will help strengthen the evaluation, but this analysis will
not reflect as many years of mature program operation as possible. 
Thus, at a relatively modest cost, MPR's evaluation would be further
strengthened by including data from the sixth and final year of the
demonstration. 


   RECOMMENDATION
------------------------------------------------------------ Letter :7

To increase the value of the investment in the national evaluation of
Healthy Start, we recommend that HRSA contract with MPR to expand the
evaluation to include impact data from the sixth year. 


   AGENCY COMMENTS
------------------------------------------------------------ Letter :8

In commenting on a draft of this report, HRSA agreed with our
findings and indicated that it intends to add funds to the MPR
contract to include impact data from the sixth year of the
demonstration.  HRSA and MPR provided a number of technical comments
that we incorporated as appropriate. 


---------------------------------------------------------- Letter :8.1

As arranged with your office, unless you announce its contents
earlier, we plan no further distribution of this report until 30 days
after the date of this letter.  We will then send copies to the
Secretary of Health and Human Services, to the Administrator of HRSA,
and to others who are interested.  We will also make copies available
to others on request. 

Please contact me at (202) 512-7119 if you or your staff have any
questions.  You may also contact Michele Orza, Assistant Director, at
512-9228, or Donald Keller, Senior Evaluator, at 512-2932. 

Bernice Steinhardt
Director, Health Service Quality
 and Public Health Issues


MPR'S METHODOLOGY AND ITS
LIMITATIONS
==================================================== Appendix Appendix


   THE AVAILABILITY ANALYSIS
-------------------------------------------------- Appendix Appendix:1

The availability analysis of infant mortality and related birth
outcomes is part of Mathematica Policy Research's (MPR) attempt to
determine if Healthy Start has, as intended, reduced infant mortality
at program sites, looking at the vital statistics for entire program
and comparison areas where, respectively, the program is or is not
available.  It does this without concern about the participation in
the program of specific persons.  It attempts to separate any change
that may occur in outcomes at program sites that is attributable to
Healthy Start from change in outcomes at those sites that would have
occurred without the program--for example, changes stemming from the
national trends not related to health and social interventions, such
as the persisting decline in infant mortality experienced almost
everywhere in the United States.  It does this for each outcome of
interest, obtained from the state health department's vital records
of linked births and deaths for the sites of interest, by (1)
comparing each program site with two comparison sites without Healthy
Start, selected (matched) for similarity to the program site with
respect to race and ethnicity, infant mortality rate, and trend in
infant mortality over the pre-Healthy Start period and (2)
statistically adjusting the data for differences between program and
comparison site mothers on variables, also obtained from vital
records, believed to affect the outcome.  To the extent that, as a
result of site selection and statistical adjustment, the program and
comparison sites do not differ in expected infant mortality rate,
then the comparison of the program and comparison site adjusted
outcomes should be a valid indication of the effectiveness of the
program. 

MPR's approach involves accepted statistical methods with known
limitations.  One limitation stems from the possibility that program
and comparison site mothers will systematically differ in ways, such
as poverty level, that affect outcomes but are not taken into account
in the selection of comparison sites and are not available for use in
statistically adjusting the data.  Such a difference could bias the
estimation of the difference between program and comparison sites in
outcomes.  Nevertheless, MPR appears to have taken reasonable
precautions to minimize the likelihood of bias.  MPR did this, for
example, by using two comparison sites, not just one, for each
program site and by avoiding the selection of comparison areas known
to have interventions similar to Healthy Start.  Further, MPR shared
information on its site selections with each of the 15 sites and
sought their comments and agreement on the choices. 

Another potential limitation of the analysis concerns its statistical
power, the ability to detect differences between program and
comparison site reductions when such differences, in fact, exist. 
Roughly speaking, power depends on the number of observations--in
this case, live births--in the analysis, and it is therefore often a
problem when that number is not controlled by the design of the
study.  In the case of Healthy Start, this implies that the ability
to detect a real difference between program and comparison
communities depends upon whether the comparison involves, for
example, communities relatively small in population, large
communities, or all communities pooled.  With respect to infant
mortality, MPR reports that, using all data from 1984 to 1994, the
minimal detectable difference in infant mortality is computed to be
31 percent, 7 percent, and 6 percent for small, large, and all
communities, respectively.  This means that if there are real
differences between Healthy Start and comparison communities, but
these differences are smaller than we have the power to detect (that
is, smaller than the percentages listed above), then we will
mistakenly conclude that the program has no effect on infant
mortality.  Since power depends on the number of observations,
increasing the number of years of data included in the analysis will
increase the ability to detect any difference that may exist. 

A third potential limitation concerns the number of statistical tests
performed in the complete impact analysis.  If 15 sites and seven
different birth-related outcome variables are considered, then at
least 105 statistical tests will be done.  With as large a number of
tests as this being done, it is likely that a portion of them will
yield statistically significant results by chance alone.  This means
that even if Healthy Start has no effect on infant mortality, we will
mistakenly conclude that it does have one in a certain percentage of
the statistical tests conducted.  There are statistical methods of
dealing with this problem.  If they are not employed in the final
analysis, then differences that are statistically significant by
chance alone will occur more often than is considered acceptable by
statistical convention. 


   THE PARTICIPATION ANALYSIS
-------------------------------------------------- Appendix Appendix:2

MPR's participation analysis of birth outcomes is part of MPR's
attempt to determine the effect of participation in Healthy Start
within program sites.  It compares 1995 birth outcomes between
participants and nonparticipants in each project area.  Participants
in the program's prenatal activities are identified from program
files, the Minimum Data Set required of all Healthy Start sites, and
their birth certificates are flagged.  Their birth outcomes are then
compared with those of nonparticipants or participants with limited
prenatal program involvement. 

This kind of analysis is limited by possible preexisting differences
between participants and nonparticipants and by the very definition
of participant.  Since participation in Healthy Start is voluntary,
it is possible that participants systematically differ from
nonparticipants.  Program providers may, for example, tend to attract
persons who are especially knowledgeable about services or already
well connected to the health care system.  Under these circumstances,
program participants would be expected to have better outcomes even
without Healthy Start.  Alternatively, participants might be
especially needy and at high risk for poor outcomes, in which case
they would be expected to have relatively poor outcomes.  MPR deals
with this by statistically adjusting the data on the basis of
information that may reflect these preexisting differences,
background information from birth certificates, and any other
available sources.  Although it is difficult to be certain that
outcomes have been adjusted for all possible systematic differences
between the groups being compared, MPR has stated that participants
tend to be at high risk for poor birth outcomes, thereby making any
potential finding of better outcomes for them than for
nonparticipants more convincing of the program's value. 

The question of who is a participant must be answered in order to
conduct the participation analysis.  It turns out not to be easily
answered because (1) the Minimum Data Sets of many sites have been
slow in developing into accurate record systems, (2) it is not always
clear whether a participant's involvement has been intense enough to
classify that person as a participant, and (3) when supplementary
information has been sought from new mothers about their involvement
in the program it is not always clear what criteria they use for
judging whether or not to claim to be participants.  Moreover, these
problems vary somewhat from site to site, making it difficult to be
sure that all participation analyses are comparable. 

In addition to the participation analysis of infant mortality and
birth outcomes, MPR analyzes the results of its survey of postpartum
mothers in Healthy Start areas by comparing program participants with
nonparticipants.  Most of the survey items concern service use,
satisfaction with care, behaviors like practicing birth control and
breast feeding, and ratings of infant health.  Some of these
"intermediate outcomes" may be associated with infant mortality and
others may be more closely related to topics from the process
evaluation.  Either way, however, the limitations that apply to the
participation analysis of infant mortality and birth outcomes also
apply to these participation analyses.  The results will be
meaningful to the extent that the preexisting differences between
participants and nonparticipants can be taken into account. 


   THE PROCESS EVALUATION
-------------------------------------------------- Appendix Appendix:3

MPR's process evaluation is an effort to use both qualitative and
quantitative information to assess the degree to which Healthy Start
has implemented its program as conceived of, how it serves its target
population, and how these processes developed over time.  This
description of Healthy Start can be considered the documentation of
the program's second goal--to demonstrate what happens when this kind
of effort is mounted.  Its methods are varied, including making site
visits with and telephone calls to project staff, examining the
client records of the Minimum Data Set, the postpartum survey of
participants and nonparticipants, running focus groups with service
providers and with members of the communities, as well as using
documents and vital records.  Many aspects of the process evaluation
are not complete and will not therefore be described further, but one
major document--the implementation report--is complete. 

The implementation report is based mainly on site visits, expenditure
reports from each project, and the client records of the Minimum Data
Set.  Further, two independent teams of site visitors rated certain
dimensions of administrative success using a modified Delphi
consensus reaching process.  Although they may not be avoidable, the
limitations of this report are those common to most process
evaluations that are heavily qualitative.  The methods employed
provide a wealth of information suitable to inform those who would
develop similar programs about what to expect if different options of
organization, administration, and mode of service delivery are
attempted.  However, the essential subjectivity of interview methods
makes it difficult to know how closely other evaluators would agree
with the conclusions drawn. 


*** End of document. ***