CDC's National Immunization Survey: Methodological Problems Limit
Survey's Utility (Letter Report, 09/19/96, GAO/PEMD-96-16).
Pursuant to a congressional request, GAO assessed the Centers for
Disease Control and Prevention's (CDC) National Immunization Survey
(NIS), focusing on: (1) survey costs; (2) survey methods; and (3) use in
identifying groups of children in need of more timely immunization.
GAO found that: (1) CDC designed NIS for monitoring state progress in
achieving child immunization objectives, comparing coverage rates across
states, and awarding incentive funds; (2) CDC estimates and contractor
invoices indicate that NIS costs for fiscal year (FY) 1995, including
extraordinary expenses incurred when 1994 survey participants were
reinterviewed, were about $13 million; (3) although CDC anticipates that
survey costs will decrease in the future, it has requested $16 million
for NIS administration for FY 1997; (4) the two-phase survey
methodology, which gathers information by telephone from households and
immunization providers, excludes households that lack a telephone, may
not accurately represent the overall population, and is limited by
response accuracy; (5) NIS has not achieved sufficient precision in its
survey estimates to detect modest changes that occur in most coverage
levels; (6) CDC considers the identification of groups of children in
need of more timely immunization as a state rather than a federal
responsibility and has not designed and does not use NIS to make such
identifications; and (7) interviews with state officials indicate that
NIS is not useful in helping states to diagnose problems in immunization
activities, target efforts, or design interventions.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: PEMD-96-16
TITLE: CDC's National Immunization Survey: Methodological Problems
Limit Survey's Utility
DATE: 09/19/96
SUBJECT: Health surveys
Preschoolers
Immunization programs
Statistical methods
Administrative costs
Health research programs
IDENTIFIER: National Health Interview Survey
CDC Childhood Immunization Program
CDC National Immunization Survey
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Honorable
Dale Bumpers, U.S. Senate
September 1996
CDC'S NATIONAL IMMUNIZATION SURVEY
- METHODOLOGICAL PROBLEMS LIMIT
SURVEY'S UTILITY
GAO/PEMD-96-16
Methodological Problems Limit Utility of NIS
(973441)
Abbreviations
=============================================================== ABBREV
ASTHO - Association of State and Territorial Health Officials
CDC - Centers for Disease Control and Prevention
HHS - Department of Health and Human Services
IAP - Immunization action plan
NCHS - National Center for Health Statistics
NHIS - National Health Interview Survey
NIP - National Immunization Program
NIS - National Immunization Survey
WIC - Special Supplemental Food Program for Women, Infants, and
Children
Letter
=============================================================== LETTER
B-272491
September 19, 1996
The Honorable Dale Bumpers
United States Senate
Dear Senator Bumpers:
Recommendations for childhood immunization indicate that children
should receive most of their immunizations before they are 19 months
old. (See appendix I.) To monitor the extent to which this goal is
met at the national level, the Centers for Disease Control and
Prevention (CDC) has, since 1991, administered a supplement to the
National Health Interview Survey (NHIS). States have also monitored
this goal with their own methods.
In 1994, CDC decided to centralize the collection of immunization
coverage data for preschoolers in each state and certain urban areas.
The purposes CDC cited for this effort included monitoring state
progress in achieving childhood immunization objectives, permitting
comparison of current coverage rates across states, and awarding
incentive funds available to CDC grantees based on their immunization
of certain percentages of preschool children.
Thus, to augment the national estimates provided by the NHIS, in
1994, CDC initiated the National Immunization Survey (NIS) to produce
current and comparable immunization coverage estimates for children
aged 19 to 35 months in each state and 28 urban areas receiving CDC
funds to implement immunization action plans (IAPs).\1 (See appendix
II.) Unlike the NHIS, which is a face-to-face household survey, the
NIS is conducted by telephone. Unlike the other state immunization
surveys, the NIS is conducted under a federal contract jointly
managed by CDC's National Immunization Program (NIP) and National
Center for Health Statistics (NCHS) and financed directly by the CDC
rather than through the grant funds received by each state and urban
area. CDC has noted that the NIS is the largest survey ever
conducted to assess vaccination coverage levels of children in the
United States.
Although national, antigen-specific immunization rates are generally
high, there are still areas and communities at continued risk of
disease outbreaks (sometimes called "pockets of need") because they
have concentrations of children who have not received timely
immunizations.\2 This report responds to your request that we assess
(1) the cost of the NIS, (2) the methods used by CDC to conduct the
survey, and (3) the utility of the survey in identifying "pockets" of
children in need of more timely immunization.
--------------------
\1 The decision to develop coverage estimates for children up to 35
months is grounded in methodological realities rather than disease
prevention objectives. The NHIS sample reaches only a limited number
of households with children; thus, to allow a sufficient sample size,
the immunization supplement measures coverage in children up to 35
months. Other age ranges might have been chosen in designing the
NIS; however, this age range was carried over from the design for the
NHIS.
\2 See U.S. General Accounting Office, Vaccines for Children:
Reexamination of Program Goals and Implementation Needed to Ensure
Vaccination (GAO/PEMD-95-22; June 15, 1995).
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
The cost of the NIS for the most recently completed fiscal
year--1995--was about $13 million, using estimates provided by the
Department of Health and Human Services (HHS) on the costs of the
data collection contract and other survey-related expenditures.
These expenditures are large relative to the incentive awards the
survey results are used to distribute, for which approximately $33
million was available in fiscal 1996. For fiscal 1997, CDC has
requested $16 million for the survey and its administration, as it
requested and received in fiscal years 1995 and 1996. However, even
with the extraordinary expenses incurred in 1995, survey costs did
not reach $16 million.\3 CDC officials indicated that the balance of
funds received for the NIS (about $3 million) was spent on other
assessment activities. However, we did not independently verify this
statement.
With respect to survey methodology, a number of difficulties are
inherent in applying telephone survey methods to the estimation of
preschool immunization coverage. Although the random-number
telephone survey permits rapid centralized data collection, it also
has some widely recognized weaknesses, including the exclusion of
households that lack a telephone, lower response rates than typify
other survey methods, and inefficiencies in identifying particular
types of households. Recognizing the biases inherent in the survey
technique and problems with respondents' recall of children's
immunization history, CDC has made various adjustments to the results
of the NIS. The accuracy of the coverage estimates, consequently,
depends to some extent on the validity of a highly complex set of
assumptions undergirding these adjustments.
The precision of survey estimates and their narrow range raise
additional concerns about the utility of this approach. The
precision of survey estimates (or their "margin of error") governs
the survey's capacity to detect differences in immunization coverage
levels over time and across immunization projects. For a variety of
reasons, coverage estimates did not achieve planned levels of
precision, and the survey can detect only very large changes from
quarter to quarter. Moreover, although large differences were
anticipated across states, the range of actual results is
comparatively narrow. It is doubtful that the precision of survey
estimates is adequate for detecting the modest changes that occur
from year to year in most coverage levels. Yet CDC uses the survey
to measure, quarterly, states' progress toward early childhood
immunization objectives and to reward, annually, those that have
reached the highest levels of coverage.
Finally, CDC officials told us that the NIS was not designed to
identify pockets of children in need of more timely immunization, and
consequently, it does not do so. HHS officials indicated that their
previous statement to the contrary was in error.\4 Furthermore, the
NIS was not designed to suggest specific programs or interventions to
boost immunization rates, and our survey of state immunization
program managers confirmed that it did not.
The survey was never intended to identify pockets of children in need
of more timely immunization, and state estimates are higher and
closer together than originally anticipated. Moreover, in view of
the survey's cost and its methodological limitations for the purposes
of comparing state performance, monitoring progress toward
immunization objectives, and distributing incentive funds, we have
serious concerns about its utility and efficiency. At your request,
we are continuing to study the various means of identifying pockets
of need.
--------------------
\3 In fiscal 1995, CDC had to reinterview fiscal 1994 survey
respondents in order to obtain their consent to contact their
children's immunization providers.
\4 See Office of the Assistant Secretary for Management and Budget,
"Department of Health and Human Services: The Fiscal Year 1997
Budget," March 19, 1996, p. 28.
BACKGROUND
------------------------------------------------------------ Letter :2
CDC'S IMMUNIZATION PROGRAM
---------------------------------------------------------- Letter :2.1
The CDC's National Immunization Program provides grants to states and
28 urban IAP areas for the purpose of controlling vaccine-preventable
diseases. The Congress made available at least $142 million for
these grants in fiscal 1995. The portion of these funds received by
a particular grantee is based largely upon the amount received the
previous year. In addition to these funds, consistent with
statements of the Senate Appropriations Committee, CDC has awarded
annual incentive grants to states since fiscal 1994 to improve the
immunization levels of 2-year-olds.\5 For awards in fiscal 1994, CDC
allocated incentive grants based on state-supplied estimates of the
percentage of fully immunized 2-year-olds. To establish a common
basis for awarding subsequent grants and to monitor progress toward
early childhood immunization objectives, the CDC designed, and
starting in fiscal 1994 began to conduct, the National Immunization
Survey. In fiscal 1996, $33 million was allocated for such incentive
grants.
--------------------
\5 S. Rept. 143, 103d Cong., 1st Sess. 63 (1993), accompanying the
fiscal 1994 HHS appropriation; S. Rept. 318, 103d Cong., 2d Sess.
56 (1994), accompanying the fiscal 1995 HHS appropriation; and S.
Rept. 145, 104th Cong., 1st Sess. 53 (1995), accompanying the
fiscal 1996 HHS appropriation.
STATES' PREVIOUS METHODS FOR
MEASURING IMMUNIZATION AMONG
PRESCHOOLERS
---------------------------------------------------------- Letter :2.2
With the advent of the NIS, states had no further obligation to
produce statewide coverage estimates and were able to use the grant
funds formerly devoted to such measurement for other activities.
However, most states' former methods for estimating immunization
coverage were much less expensive than the NIS, which CDC has
heretofore financed at no cost to the states. Lately, CDC has made
inquiries of state health officers regarding their willingness to
devote certain percentages of grant funds to support the NIS (see p.
9).
To meet CDC's former requirement for measuring preschoolers'
immunization coverage, all states used either school retrospective
surveys or other population-based methods to estimate immunization
coverage. Most states estimated immunization coverage among
preschoolers through reviewing the immunization records of children
entering first grade or kindergarten to determine whether their
immunizations were up-to-date when the children were younger,
typically when they were 2 years old.
This method has both disadvantages and advantages. It produces
estimates that are about 3 years old by the time the data are
gathered, and immunizations may be selectively collected on school
records relative to the minimum state requirements for school entry,
which vary to some extent across states and may not include the newer
vaccines.\6 Because the retrospective method uses data that are
already collected for the purpose of verifying immunization at school
entry, it is fairly inexpensive and enables some states to develop
estimates of immunization coverage at substate levels for the use of
counties or state health districts.\7 Records for those immunizations
required for school entry should provide more accurate dates of
immunization than can be obtained in interviews with parents, who
frequently do not have ready access to immunization information.
Those states that did not use the retrospective method used others,
such as birth certificate surveys and registry-based methods, that
required more original data collection than the retrospective survey,
but produced more current coverage estimates while providing the
states with other benefits or additional information about their
specific activities. In 1995, CDC dropped its requirement that
grantees produce an independent assessment of preschool immunization
coverage with the view that the estimates from its new NIS would
supplant the data that had formerly been gathered by grantees.\8
--------------------
\6 See T.V. Murphy, P. Pastor, S.B. Turner, et al., "Estimating
immunization coverage from school-based childhood immunization
records," Pediatric Infectious Disease Journal, 14:7 (1995), 561-67.
\7 See L.E. Rodewald, K.J. Roghmann, P.G. Szilagyi, et al., "The
school-based immunization survey: An inexpensive tool for measuring
vaccine coverage," American Journal of Public Health, 83:12 (1993),
1749-51.
\8 Some states have continued to conduct the retrospective survey to
preserve the capacity to compare current coverage levels against
older data derived from retrospective methods. The retrospective
surveys generally measure immunization coverage on the 2nd birthday
(24 months), while the NIS measures immunization coverage between 19
and 35 months of age. Since the measured immunizations are due by 18
months, the estimates provided by the NIS are based on an age range
allowing a median of 9 (and as many as 17) additional months to
obtain immunizations, while the retrospective surveys allow only 6
additional months to reach up-to-date status. Consequently, the
estimates of up-to-date rates derived from the NIS will tend to be
higher owing to methodological differences between the surveys rather
than changes in the immunization status of preschoolers. It should
also be noted that the NIS measures immunization coverage in areas
(states and cities) defined by political boundaries. Very different
coverage estimates may emerge from surveys of particular areas or
populations incorporated in these states and cities.
SOURCES OF SURVEY ERROR
---------------------------------------------------------- Letter :2.3
In general, in assessing the quality of survey findings, analysts
should consider a variety of types of error that may affect a survey
result.\9 These include errors that arise because (1) surveys only
involve a sample of the population of interest, (2) some of the
sampled individuals may not respond to the survey, and (3) some of
the population of interest may not be covered by the group from which
the sample was chosen. In addition, there are problems associated
with interviewers, the respondent, or the questionnaire, such as
unclear questions or respondents' difficulty in recalling the
answers. What is commonly quoted in the reporting of poll results as
the "margin of error," typically plus or minus 3 percent for a random
sample of 1,500, represents only the error attributable to the first
factor named above. Assessing the quality of survey results also
requires considering the extent to which the other sources of error
may have affected the accuracy of survey findings.
--------------------
\9 See P.E. Converse and M.W. Traugott, "Assessing the accuracy of
polls and surveys," Science, 234 (1986), 1094-98.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3
To respond to your request, we met with officials of the National
Immunization Program and the National Center for Health Statistics
and with staff of the CDC contractor conducting the National
Immunization Survey. We reviewed documents describing the structure,
performance, and results of the survey. We also reviewed literature
on telephone survey methodology and parental recall of children's
immunization status. The methodology report for the 1995 survey was
not available as of June 18, 1996, when we conducted our exit
conference with CDC, and thus, our review of survey methodology was
limited to the procedures employed in the 1994 survey and reports of
NIS findings issued through June 1996. We understand from NCHS
officials that, since issuance of the 1994 methodology report,
procedures for using provider data to adjust survey results have been
documented and sensitivity analyses have been conducted to measure
the impact of changes in various assumptions inherent in the
adjustment of survey results.
To provide information on survey costs, we requested that agency
officials provide data on total payments under the survey contract
and estimates of the costs of related agency activities. We also
reviewed the survey contract and trends in the costs billed under the
contract. We did not independently verify the payments for the
survey or the CDC cost estimates, though we did review the invoices
from the survey contractor and assess the agency's cost estimates for
their consistency with the activities the agencies conducted. With
this exception, our work was conducted in accordance with generally
accepted government auditing standards between March 25 and June 18,
1996.
Finally, we surveyed state immunization program managers regarding
how they had used the results of the NIS and their costs for previous
survey approaches. In addition, CDC provided a list of six former
CDC contractors, officials, and current grantees that they
recommended we contact. We contacted some of these individuals and
asked them to provide comments consistent with their familiarity with
the survey's cost and methodology.
PRINCIPAL FINDINGS
------------------------------------------------------------ Letter :4
SURVEY COSTS
---------------------------------------------------------- Letter :4.1
The cost of the survey includes three major components-- expenditures
under the contract issued to conduct the survey and the costs of
survey-related activities conducted by NCHS and NIP. Both NCHS and
NIP were involved in managing the data collection contract and
providing statistical analysis of survey data. In addition to these
roles, NIP gathered and reviewed data from the survey respondents'
immunization providers. When problems with survey software created a
need for a larger interviewing staff, some work was done by the
Bureau of the Census, but costs for this work are included in
estimates provided by NCHS. Table 1 shows the costs of the NIS
contract, survey assistance provided by NCHS and the Bureau of the
Census, and NIP's survey-related activities. Only two quarters of
data were collected in fiscal 1994; 1995 was the first fiscal year in
which the survey operated in all four quarters. Extraordinary
expenses were incurred in fiscal 1995 when the agency discovered it
needed to reinterview 1994 survey participants in order to identify
their immunization providers.
Table 1
Estimates of Total NIS-related
Expenditures\a
NCHS and
Survey Census
Fiscal year contract\b Bureau\c NIP\d Total
------------------------------ ---------- ---------- ------ ======
1994 (2 quarters)\e $4.0 $1.8 $0.1 $5.9
1995 (4 quarters) 11.3 1.2 0.4 12.9
1996 (2 quarters)\f 4.9 0.7 0.2 5.8
======================================================================
Total (8 quarters) $20.2 $3.7 $0.7 $24.6
----------------------------------------------------------------------
\a In millions.
\b Based on an NCHS summary provided to us in April 1996. These
amounts cover contractor billings through March 30, 1996.
\c Estimates taken from an NCHS summary memo on administrative costs
provided to us in April 1996. For purposes of consistency, the
estimate provided for fiscal 1996 has been halved in order to
approximate the costs for the first two quarters.
\d Revised estimates provided to us by NIP on June 17, 1996.
\e January 1, 1994 to September 30, 1994; in fiscal 1994, only two
quarters of survey data were collected (between April and September).
\f First two quarters of fiscal 1996.
Source: Data from NCHS and NIP.
The contract to conduct the NIS provides the recipient with a fixed
fee and all reasonable costs for conducting the survey. Expenditures
under the survey contract have risen at twice the rate anticipated at
its signing, reaching nearly the full face amount of the contract
halfway through the 54-month performance period. Contractor and
agency representatives attribute the higher rate of expenditures to
difficulties arising from the need to replace survey management
software; the higher-than-expected number of calls required to
identify households in the sampling frame; and the addition of a
study to check parents' responses against provider records, which
increased the complexity of estimating survey results because of the
need to adjust them with provider-derived information. The number of
calls required to identify eligible households will continue to be an
important determinant of survey costs.
According to estimates from CDC and invoices from the contractor,
costs for the NIS have been roughly $25 million through March 30,
1996, including the $13 million for fiscal 1995, the first complete
year of data collection. Insofar as a number of extraordinary
expenses were incurred in fiscal 1995, CDC officials anticipate that
final survey costs will decrease in fiscal 1996 and future years.
However, for fiscal 1997, the agency has requested $16 million for
the survey and its administration, as it requested and received in
fiscal years 1995 and 1996 based on expenditures in the early
implementation stage of the survey. CDC officials indicated that the
balance of funds received in 1995 for the NIS (about $3 million) was
spent on other assessment activities, such as the NHIS and its
provider record check study, the Clinic Assessment Software
Application, and the provision of technical assistance to the states.
However, we have not independently verified this information.
In its report accompanying the fiscal 1996 appropriations, the Senate
Appropriations Committee noted its concern that the national findings
of the NIS duplicate the findings from the NHIS and that the annual
cost of the survey cannot be justified by its utility.\10 The
Committee noted particularly that the survey does not provide
significant information on high-risk communities for targeting
purposes, and in some respects, it duplicates surveys conducted by
each state. In the justification for its fiscal 1997 budget request,
CDC acknowledged these concerns and noted that it was holding ongoing
discussions with, among others, the Association of State and
Territorial Health Officials (ASTHO) and the Council of State and
Territorial Epidemiologists in which "various options related to the
NIS" were being considered.
For example, CDC explored with ASTHO the level of willingness among
state health officers to finance the survey through state grant funds
distributed by CDC rather than directly through CDC appropriations.
However, ASTHO surveys of its members found that many of the larger
states and urban areas were not prepared to devote 6-10 percent of
their immunization infrastructure grants to support of the survey.\11
This is consistent with the findings of our survey of state
immunization program managers, which indicated that while the NIS
findings were widely used to communicate with the news media and
respond to legislative inquiries, they were not used by most states
for targeting their activities or designing interventions.\12
--------------------
\10 S. Rept. 145, 104th Cong., 1st Sess. 55 (1995).
\11 The total financing available from those who indicated a
willingness to support the NIS through the donation of 10 percent of
infrastructure funds was $4.6 million.
\12 Thirty-four states told us they had not used the NIS results to
target program activities, and 36 told us they had not used them to
design programs or interventions.
METHODOLOGICAL CHALLENGES
---------------------------------------------------------- Letter :4.2
SURVEY DESIGN
INCORPORATES
INEFFICIENCIES
-------------------------------------------------------- Letter :4.2.1
NIS surveyors identify households with children between 19 and 35
months old by dialing random telephone numbers and asking a short set
of screening questions to assess the presence of children in the
correct age range. Surveyors ask for the number of doses of various
vaccines the child has received and a variety of demographic
information. Even with sampling refinements implemented by the
contractor, only a small proportion of randomly generated telephone
numbers results in contacting a residence that includes children
between 19 and 35 months old. CDC reported that roughly 1.2 million
telephone numbers were called to complete 25,247 interviews during
the first three quarters of data collection (47 numbers per
respondent, with an average of 4-5 calls per number required to reach
a respondent). Thus, roughly 200 calls are initiated per completed
interview. In view of the size of this undertaking, there was some
thought at the time the survey was planned of using it to gather
additional health data, but these plans never came to fruition and
the final survey addressed only immunization issues.
With the view that the data provided by parents and other household
contacts would be sufficient to produce accurate immunization
coverage estimates, the NIS was initially designed as a telephone
survey of households using a modified random digit-dialing technique.
When it later became clear that the data derived from household
interviews would not, on their own, be sufficient to produce accurate
estimates of immunization coverage, a second phase of the survey was
added--the Provider Record Check Study. In this second phase,
household contacts are asked for the names of medical providers, who
are then approached for independent information on the dates of
immunization for children identified through the household survey.
Thus, to some extent, the NIS was reconceptualized as a survey of
providers using a two-phase design. However, two-phase designs are
usually most efficient
"only when the first-phase element survey costs are smaller than
those for the second phase by a large factor . . . [as when]
the first-phase sample identifies the members of the rare
population inexpensively, and the survey items are then
collected from them in the second phase."\13
For the NIS, the reverse is true. It appears that CDC is spending a
large sum of money on the first phase of the survey, which provides
low-quality immunization data but identifies the sample for the
second phase, which provides high-quality immunization data from
provider records, albeit for a smaller number of children. Although
the provider-supplied data improves the accuracy of survey results,
earlier recognition of the problems with relying solely on household
data might have led to consideration of more efficient data
collection methods.
As of June 1996, summary coverage estimates had been published for
the first five quarters of NIS data collection (April 1994-June
1995).\14 CDC shares the survey results with state programs shortly
before their publication in Morbidity and Mortality Weekly Report.
Thus, the survey findings are available to states and the general
public about a year after data are collected.
--------------------
\13 Graham Kalton, Introduction to Survey Sampling (Beverly Hills,
Calif: Sage, 1983), p. 48.
\14 See CDC, "State and national vaccination coverage levels among
children aged 19-35 months--United States, April-December 1994,"
MMWR, 44 (August 25, 1995), 613, 619-23; "National, state, and urban
area vaccination coverage levels among children aged 19-35
months--United States, April 1994-March 1995," MMWR 45 (February 23,
1996), 145-50; and "National, State, and Urban Area Vaccination
Coverage Levels Among Children Aged 19-35 Months--United States, July
1994-June 1995," MMWR, 45:24 (June 21, 1996), 508- 13.
HOUSEHOLDS WITHOUT A
TELEPHONE ARE EXCLUDED
FROM THE SURVEY
-------------------------------------------------------- Letter :4.2.2
The telephone survey technique that CDC adopted for the NIS has
certain inherent biases, notably exclusion of any household without a
working telephone. The Department of Commerce's National
Telecommunications and Information Administration has recently noted
that more than 6 million households still lack a telephone and that
low-income and minority communities are substantially below the
national average for telephone penetration.\15 The methodology report
prepared for NCHS by the survey contractor notes that
"The high nontelephone noncoverage rates in many of the IAP
areas and the large differences between telephone and
nontelephone children's vaccination rates indicate that the
potential for noncoverage bias is considerable in several IAP
areas. Any candidate estimation technique for the NIS must
recognize this potentially large bias, and attempt to adjust for
differences between the telephone and nontelephone groups."\16
Appendix III shows the estimated percentage of households with a
2-year-old child that lack a telephone in each of the IAP areas, and
table 2 provides national data from the 1992 and 1993 National Health
Interview Surveys detailing the difference in reported immunization
rates between children in households with and without telephones.
Table 2
NHIS Estimates of Children With Up-to-
Date Immunizations in Households With
and Without Telephones\a
Without Without
Respondents have a With a a With a a
shot telepho telepho Tota telepho telepho Tota
card? ne ne l ne ne l
---------------------- ------- ------- ---- ------- ------- ----
Yes 64.8% 47.8% 62.9 65.2% 49.1% 63.6
% %
No 53.3 38.3 52.3 71.3 58.1 70.2
======================================================================
Total 58.2 42.3 56.6 68.6 53.9 67.3
----------------------------------------------------------------------
\a Up-to-date immunization is defined as the receipt of at least 4
doses of diphtheria and tetanus toxoids and pertussis
vaccine/diphtheria and tetanus toxoids, 3 doses of oral polio virus
vaccine, and 1 dose of measles-mumps-rubella vaccine.
Source: Abt Associates, Inc., The National Immunization Survey (NIS)
1994 Annual Methodology Report, chapter VIII, table 1. (Submitted to
NCHS under contract no. 200-94-7009.)
Although only about 5 percent of all U.S. households lack a
telephone, the absence of one is more than twice as common in
households with children under 2 years old (11.7 percent). However,
these national data mask the wide variation among IAP areas in the
percentages of households with children under 2 lacking telephones,
which ranges from 2 to 25 percent across the 50 states and 28 urban
IAP areas. Exclusion of households without a telephone requires that
the survey results be adjusted to account for the positive bias that
may result. However, there is no consistent source of information on
the immunization rates among children in households without
telephones in each area where the NIS is conducted. Consequently,
the adjustment for noncoverage of children without telephones is
based on a complex procedure involving the application of a
statistical model of the probability that a fully vaccinated child in
a related national survey resides in a household with a telephone.
It is not possible to know whether these adjustments are accurate in
each of the states and urban areas covered by the NIS.
--------------------
\15 See National Telecommunications and Information Administration.
Falling Through the Net: A Survey of the "Have Nots" in Rural and
Urban America (Washington, D.C.: U.S. Department of Commerce, July
1995).
\16 See Abt Associates, Inc., The National Immunization Survey (NIS)
1994 Annual Methodology Report, prepared for the National Center for
Health Statistics and the National Immunization Program of the
Centers for Disease Control and Prevention under contract no.
200-94-7009, p. 159.
RESPONSE RATES VARY
WIDELY
-------------------------------------------------------- Letter :4.2.3
The response rate is the estimated proportion of the target group (in
this case, households with telephones and age-eligible children) that
actually provided data.\17 This rate is important in evaluating
survey findings because, to the extent that nonrespondents might have
answered differently from those who completed the survey, a large
nonresponse rate indicates that survey findings will incorporate bias
and require adjustment. For example, CDC analyses of NIS respondents
indicated that, as a group, they differed in some respects from
census and vital statistics estimates for the population; they
slightly overrepresented mothers with more than 12 years of education
and in some areas were more likely to report household incomes
exceeding $50,000 and less likely to report income below $10,000.
Thus, answers from those types of respondents who tended to be
underrepresented were weighed more heavily in adjusting survey
results to arrive at final coverage estimates. Such adjustments will
remove bias to the extent that immunization coverage is similar
between respondents and demographically similar nonrespondents.
However, there is no clear way to test this assumption in the various
areas surveyed.
For the calendar year 1994 survey, contractors estimated that the
overall response rate was 69.5 percent. Appendix III identifies the
overall response rates reported for each surveyed area. Although
households determined to be eligible through their completion of the
screening questions had high rates of cooperation with the full
interview, they represented a smaller portion of the potential
households than would have been expected based on census data,
indicating that some 17.3 percent of eligible households with
telephones were never reached, refused cooperation during the
screening phase, or inaccurately responded to the questions about
age-eligible children. Although a response rate in this range is not
atypical of telephone surveys, nonresponse rates tend to run higher
for telephone interviewing than for personal visitation.\18 Also,
while overall response rates varied tremendously across states and
urban areas, nonresponse to particular questions ranged as high as 26
percent. When combined, these factors sometimes reduce to below 50
percent the effective response rates for key questions (for example,
how many times has your child received a polio vaccine?), raising
concerns about the accuracy of resulting estimates.
--------------------
\17 Assessing the extent of error introduced by nonresponse is
somewhat complicated in a random digit-dialing telephone survey of
the type employed for the NIS. This is true because surveyors are
not simply calling a list of houses with children in the appropriate
age range, but a list of random telephone numbers. If every
telephone number drawn into the sample could be categorized according
to its business or residential status and, if residential, according
to whether the household included a child in the appropriate age
range, calculation of response rates would be a simple matter.
However, nonresponse may occur well before this is accomplished, as
when a sampled number results in reaching an individual who hangs up
before any information can be gathered. Some of these instances may
represent eligible households who refused participation before it
could be determined that they contained children in the correct age
range. Others may represent numbers that did not belong to
households, but to businesses or numbers that were not assigned.
\18 See, for example, NCHS, An Experimental Comparison of Telephone
and Personal Health Interview Surveys, Data Evaluation and Methods
Research, Series 2, No. 106 (Washington, D.C.: U.S. Government
Printing Office, August 1987).
HOUSEHOLD RESPONDENTS
HAVE DIFFICULTY PROVIDING
ACCURATE DATA
-------------------------------------------------------- Letter :4.2.4
The potential to use household surveys for the collection of
childhood immunization data is limited by the accuracy with which
household respondents can supply information on children's
immunization status.\19 Data available to CDC before the initiation
of the NIS, including a report commissioned by the agency in 1975 to
review the United States Immunization Survey, questioned the
assumption that parents could accurately recall immunization
history.\20 Even as the NIS was initiated in 1994, NCHS had a study
in progress to assess the accuracy of responses to the immunization
supplement of the NHIS.\21 It is well documented that survey
respondents have trouble accurately recalling the occurrences of,
distinctions among, and number of, events that are not particularly
salient, or that are similar in nature, or that are repeated more
than a few times over a long time period. As a result, when
surveyed, they sometimes forget when the events occurred and are
confused as to how many of which types of events occurred. As a
rule, if events are socially desirable, respondents tend to
overreport them.
The NIS asks about the receipt of 14 different immunizations, given
in repeated sets, varying in number, over a 1- to 3-year period.
Respondents may not understand the differences among the various
types of shots and probably consider getting shots socially
desirable. As noted, these elements are among the factors associated
with inaccurate reporting. To the extent that a parent is able to
answer from an up-to-date vaccination record, few of these errors
would occur, but significant portions of NIS respondents did not have
a shot card and consequently reported from memory.\22 Others
apparently used shot cards that were not up-to-date.
In December 1994, after the first two quarters of NIS data
collection, CDC acknowledged the need to check parents' responses
against provider records. At that time, NCHS had determined from its
surveys assessing the accuracy of parental responses to immunization
questions in the NHIS that household respondent reports of
vaccinations contain a number of errors that result in
underestimation of the "true" vaccination coverage levels. NCHS
concluded that, although respondent information was necessary for
estimation and demographic analysis, household respondent records of
immunizations are often not sufficiently up-to-date to provide
accurate information, errors in reports from recall exist, and the
household information must be adjusted using provider data. Using
the findings from the NHIS substudy, NCHS and NIP attempted to adjust
the NIS estimates. However, these adjustments resulted in estimates
that did not differentiate the IAP areas. Therefore, CDC determined
that a provider substudy similar to the one being conducted in
connection with the NHIS was needed to produce accurate vaccination
coverage level estimates from the NIS. We reviewed the level of
agreement between household reports and physician records from the
NIS substudy and confirmed that it was generally only "poor" or
"fair" based on the application of recognized statistical
criteria.\23 Earlier recognition of this problem might have led to
more serious consideration of other survey methods.
--------------------
\19 See, for example, K.P. Goldstein, F.J. Kviz, and R.S. Daum,
"Accuracy of immunization histories provided by adults accompanying
preschool children to a pediatric emergency department," Journal of
the American Medical Association, 270:18 (1993), 2190-94. This study
indicated that only 8 percent of adults questioned in an inner-city
emergency room could accurately recall how many of each vaccine their
preschool child had received and that many did not know whether the
child had received a measles vaccination despite a recent epidemic in
the city. See also M.D. Joffe and A. Luberti, "Effect of emergency
department immunization on compliance with primary care," Pediatric
Emergency Care, 10:6 (1994), 317- 19.
Reports from other countries include: M.A. Soljak, "How many
children are fully immunised?" New Zealand Medical Journal (January
25, 1984), 37-39; P. Hawe, A. Wilson, P. Fahey, et al., "The
validity of parental report of vaccination as a measure of a child's
immunisation status," The Medical Journal of Australia, 155 (1991),
681-86; P.A. McKinney, F.E. Alexander, C. Nicholson, et al.,
"Mothers' reports of childhood vaccinations and infections and their
concordance with general practitioner records," Journal of Public
Health Medicine, 13:1 (1991), 13-22; and J.J. Valadez and L.H.
Weld, "Maternal recall error of child vaccination status in a
developing nation," Journal of the American Public Health
Association, 82:1 (1992), 120-22.
\20 See J. Bean, L. Burmesister, P. Isaacson, et al., "Estimation
of Community Immunization Levels," report based on CDC Grant No.
21-74-545 (PIC ID number 0436), July 1, 1975. The specific
recommendations of this study of the methods used in the United
States Immunization Survey included, "To institute as soon as
practicable additional studies of validity of immunization history by
both record search and comparison with serum antibody levels."
\21 Elizabeth R. Zell, James T. Massey, and Trena M. Ezzati-Rice,
"An Overview of the National Health Interview Survey and the State
and Local Area Immunization Coverage and Health Survey," presented at
the National Immunization Conference, Los Angeles, May 1995.
\22 As combination vaccines are introduced and children receive
immunizations on an increased variety of schedules, the task of
obtaining accurate data on receipt of particular antigens from
records or recall may increase in complexity.
\23 See D.G. Altman, Practical Statistics for Medical Research (New
York: Chapman and Hall, 1991), p. 404.
COVERAGE ESTIMATES FAILED
TO MEET DESIRED LEVELS OF
PRECISION
-------------------------------------------------------- Letter :4.2.5
The survey plan called for precision of plus or minus 5 percent for a
coverage estimate of 50 percent, meaning that the margin of error
would have been narrower for more extreme coverage estimates. Owing
to various factors, the actual estimates produced by the survey in
its first year had margins of error that were often larger. As these
margins of error increase, the survey's capacity to detect changes in
immunization coverage decreases: it becomes more difficult to
distinguish a change of a particular size from simple error in the
estimates.
CDC officials have indicated that the survey is useful in that it
permits them to rank states and helps to motivate the lower ranking
states to take positive action to improve immunization coverage.
However, partly because survey estimates did not meet planned levels
of precision, there appear to be remarkably few differences across
states. For example, for the most recently published four quarters
of NIS data (quarter 3 of 1994 through quarter 2 of 1995), in 31
states, the estimated percentage of children up-to-date in their
immunizations could not be statistically distinguished from the
national percentage of children up-to-date. (See figure 1.)
Figure 1: Estimated
Percentages of Children in Each
State Whose Immunizations Are
Up-to-Date and 95-Percent
Confidence Boundaries for Each
Estimate\a
(See figure in printed
edition.)
\a From July 1994 through June 1995 for children 19-35 months old.
Up-to-date immunization is defined here as having received at least 4
doses of diphtheria and tetanus toxoids and pertussis
vaccine/diphtheria and tetanus toxoids, 3 doses of oral polio virus
vaccine, and 1 dose of a measles-mumps-rubella vaccine.
Source: Based on NIS data published in "National, State, and Urban
Area Vaccination Coverage Levels Among Children Aged 19- 35
Months--United States, July 1994-June 1995," MMWR, 45:24 (June 21,
1996), 508-13.
Moreover, the survey is unlikely to show change from quarter to
quarter. The Final Sampling Plan for the survey notes, "it will only
be possible to detect very large changes between adjacent annualized
estimates." For example, a move from 50- to 70-percent coverage would
have been the smallest detectable change had the planned level of
precision been achieved.\24 As a result, there are no statistically
significant changes in full coverage across the first three sets of
survey results published by CDC for any of the 78 states or urban
areas surveyed.
The smallest change that the survey is likely to detect between
successive years for a particular IAP area (for example, quarters
1-4, 1995, versus quarters 1-4, 1996) may in some areas approach the
size of the largest change observed between successive years in
recent years' data from the NHIS for antigens that had been
recommended before every child in the survey cohort was born.\25
Thus, even if changes of a typical size were occurring, the survey
results might create the false impression of a lack of progress. At
a minimum, the survey's broad margins of error indicate that
reporting such statistics each quarter is neither necessary nor
advisable. Moreover, the imprecision of the survey estimates
combined with their narrow range raises questions about whether the
survey provides an improved basis for distribution of incentive funds
across states.
NCHS officials acknowledged that they had considered reporting the
results only semi-annually. However, even this may be too frequent.
For those vaccines that have been recommended for a number of
years--measles, polio, 3 doses of diphtheria, tetanus, and
pertussis--coverage is 80 percent or higher, limiting the size of any
increases that might occur.
--------------------
\24 See Abt Associates, Inc., State and Local Area Immunization
Coverage and Health Survey: Final Sampling Plan (March 7, 1994), p.
19. (Submitted to NCHS under contract no. 200-94-7009.)
\25 For more newly recommended vaccines, such as Hib and Hep-B, the
NHIS has estimated that coverage only increased 15 and 14 percent,
respectively, from the third quarter of 1993 to the second quarter of
1994. Sales or distribution reports might also be used to monitor
uptake of newer vaccines.
IDENTIFICATION OF POCKETS OF
NEED
---------------------------------------------------------- Letter :4.3
CDC officials have indicated that they view identification of pockets
of children in need of more timely immunization as a state
responsibility rather than a federal one. Although a departmental
statement accompanying the fiscal 1997 budget request had indicated
the NIS would be useful in identifying pockets of need, HHS officials
told us that the statement was in error. CDC has indicated that the
National Immunization Survey was not designed to identify such
"pockets of need," and consequently, it does not do so. Our survey
of state immunization program managers confirmed that they generally
drew upon other data for this purpose.
Instead, the primary objectives CDC has for the NIS have been
monitoring state progress in achieving childhood immunization
objectives, permitting comparison of current coverage rates across
states, and awarding incentive funds available to CDC grantees based
on their immunization of certain percentages of preschool children.
In this connection, we note that the accomplishment of national
immunization goals is simultaneously tracked through supplements to
the NHIS and that the cost of mounting the NIS (roughly $13 million
in fiscal 1995) has been large relative to the total amount of
incentive funds it is used to distribute ($33 million in fiscal
1996). We have noted above the survey's limitations for monitoring
changes in immunization coverage.
Although the NIS can produce national statistics for some
nongeographically defined subgroups, the sample size of the NIS is
not large enough to provide subgroup statistics for each state or
urban area. On a national basis, the NHIS provides these same
subgroup statistics with the exception of immunization coverage
estimates for persons of Hispanic and Asian origin. CDC has
suggested that the NIS can be used to evaluate immunization
activities; however, the NIS does not currently collect information
that could link immunization coverage to specific programs. For
example, CDC has encouraged immunization among in participants in the
Special Supplemental Food Program for Women Infants, and Children
(WIC). However, state estimates of immunization coverage by WIC
participation derived from the NIS would have unacceptably large
sampling error unless the survey sample size were increased at
substantial expense.
CONCLUSION AND MATTERS FOR
CONSIDERATION
------------------------------------------------------------ Letter :5
We have not had the opportunity to assess the NIS in light of the
list of additional purposes for the survey provided to us by HHS
after our exit conference on this study. Further, our survey of
state immunization directors turned up anecdotal evidence that a few
states view the NIS favorably even though they are unable to use it
to target pockets of underimmunized children. However, while the NIS
has provided estimates of current state-specific immunization levels
for awarding incentive grants and monitoring progress toward early
childhood immunization objectives, it has significant limitations
when used for these purposes.
First, of the appropriation that it has requested for fiscal 1997,
CDC has requested $16 million for the survey and its administration.
However, the actual costs of the NIS are now expected to be between
$12 and $13 million, and even these amounts would render it an
inefficient method of allocating incentive grants expected to total
$33 million. Second, the NIS does not provide useful quarterly
measurements of statewide immunization levels, and even annual
estimates may not be suitable for monitoring the level of annual
change that is likely to occur in immunization coverage. Third, the
NIS does not assist in the systematic targeting of underimmunized
children, a particular concern if HHS is to achieve levels of disease
reduction and elimination established as goals for the end of this
decade. To follow up on this report, we intend to continue to study
the various means of identifying pockets of children in need of
immunization.
State officials did make use of the NIS findings in communicating
with their legislators and the press; however, these objectives could
be met by previous methods at markedly lower cost. Moreover, the
survey provides only a statewide or citywide indicator of
immunization coverage. Insofar as this indicator is not linked to
any specific component of the unique set of immunization initiatives
pursued by a particular CDC grantee, it is not surprising that it is
not useful in helping states to diagnose problems in their ongoing
activities, target their efforts, or design interventions.
CDC has also stressed the motivational benefit of ranking states.
Apart from the concerns we have raised about the survey's capacity to
rank states, it is difficult to quantify the benefits of this
ranking.
In view of these limitations, the Congress may wish to reconsider the
NIS's benefits relative to its cost. At a minimum, the Congress may
want to ensure that the CDC appropriation reflects a more accurate
estimate of the survey's cost.
AGENCY COMMENTS
------------------------------------------------------------ Letter :6
We provided a draft of this report to CDC officials for their
comments, which are reprinted in appendix IV. CDC does not dispute
the cost we reported for the NIS or that CDC's fiscal 1997 budget
request for the survey exceeds by at least $3 million the survey
costs the agency anticipates in fiscal 1997. CDC disagrees with some
of our findings regarding the survey's methodology and our suggestion
that the Congress may wish to consider NIS' benefits relative to its
costs. However, the agency bases some of its objections on
statements that incorporate inaccurate representations of our
findings regarding the validity of survey estimates and factual and
technical errors, which we have identified in appendix IV.
CDC indicates that following our presentation of our findings to the
agency in late June, we failed to assess all the benefits of the
survey that they had identified. However, the additional benefits
asserted by CDC after our work was completed break no new ground.
Each of these putative benefits stems from the use of the survey
findings to compare state performance, monitor changes in
immunization coverage across time, or evaluate intervention efforts.
However, with few exceptions, our findings cast doubt on the
appropriateness or practicality of such uses of survey results in
view of the survey's broad margins of error for particular states and
urban areas, the generally high level of coverage for individual
vaccines, and the difficulty of attributing changes across time or
place to any particular causal factor.
CDC asserts that the survey provides an early warning of precipitous
changes in immunization coverage; however, we are concerned that the
survey may lend a false sense of security by obscuring the existence
of substantial pockets of underimmunized children. For example, a
recent household survey of central and southeast Seattle found an
immunization coverage rate of 57 percent, in contrast to the 79
percent reported by the NIS for the King County area incorporating
Seattle. Further, NIS data are not generally analyzed and released
until a year after data collection. We agree with the CDC that the
survey is technically capable of detecting changes in use of newly
introduced vaccines, but CDC already monitors these changes on a
national basis through its NHIS. Other means, such as sales and
distribution reports, may be available for monitoring the initial
uptake of newer vaccines at less expense.
Some data from the late 1980s indicated that immunization coverage
levels in the preschool population were quite low and highly variable
across areas. While the NIS might have been more useful under those
circumstances, it appears the situation has changed. Coverage for
particular diseases is now quite high, and coverage for
long-recommended vaccines has not been highly variable across states.
While the survey does provide more timely immunization coverage data
than the retrospective surveys that were formerly used for such data
collection, it does so at much higher cost. Thus, in the interest of
using immunization resources most efficiently, we have suggested that
the cost of collecting and analyzing these data be weighed against
their continued utility.
As we agreed with your office, we are sending copies of this report
to other interested congressional committees, the Secretary of HHS,
the Director of CDC, and other federal and state officials. We will
also make copies available to others upon request.
If you have any questions or would like additional information,
please contact me, at (202) 512-3092, or Sushil K. Sharma, Assistant
Director, at (202) 512-3460. Other major contributors to this report
are listed in appendix V.
Sincerely yours,
Kwai-Cheung Chan
Director of Program Evaluation
in Physical Systems Areas
RECOMMENDED CHILDHOOD IMMUNIZATION
SCHEDULE
=========================================================== Appendix I
This schedule was approved by the Advisory Committee on Immunization
Practices, the American Academy of Pediatrics, and the American
Academy of Family Physicians.
Vaccines are listed under the routinely recommended ages. Bars
indicate the range of acceptable ages for vaccination. Shaded bars
indicate catch-up vaccination: at 11-12 years of age, hepatitis B
vaccine should be administered to children not previously vaccinated,
and varicella zoster virus vaccine should be administered to children
not previously vaccinated who lack a reliable history of chicken pox.
Figure 1.1: Recommended
Childhood Immunization Schedule
for the United States,
January-June 1996
(See figure in printed
edition.)
\a Infants born to HBsAg-negative mothers should receive 2.5
g of Merck vaccine (Recombivax HB) or 10 g of
SmithKline Beecham (SB) vaccine (Engerix-B). The 2nd dose should be
administered 1 month after the 1st dose.
Infants born to HBsAg-positive mothers should receive 0.5 mL
hepatitis B immune globulin (HBIG) within 12 hours of birth, and
either 5 g of Merck vaccine (Recombivax HB) or 10 g
of SB vaccine (Engerix-B) at a separate site. The 2nd dose is
recommended at 1-2 months of age and the 3rd dose at 6 months of age.
Infants born to mothers whose HBsAg status is unknown should receive
either 5 g of Merck vaccine (Recombivax HB) or 10 g
of SB vaccine (Engerix-B) within 12 hours of birth. The 2nd dose of
vaccine is recommended at 1 month of age and the 3rd dose at 6 months
of age.
\b Adolescents who have not previously received 3 doses of hepatitis
B vaccine should initiate or complete the series at the 11- or
12-year-old visit. The 2nd dose should be administered at least 1
month after the 1st dose, and the 3rd dose should be administered at
least 4 months after the 1st dose and at least 2 months after the 2nd
dose.
\c DTP4 may be administered at 12 months of age, if at least 6 months
have elapsed since DTP3. DTaP (diphtheria and tetanus toxoids and
acellular pertussis vaccine) is licensed for the 4th and/or 5th
vaccine dose(s) for children aged 15 months and may be preferred for
these doses in this age group. Td (tetanus and diphtheria toxoids,
absorbed, for adult use) is recommended at 11-12 years of age if at
least 5 years have elapsed since the last dose of DTP, DTaP, or DT.
\d Three H. influenzae type b (Hib) conjugate vaccines are licensed
for infant use. If PRP-OMP (PedvaxHIB [Merck]) is administered at 2
and 4 months of age, a dose of 6 months is not required. After
completing the primary series, any Hib conjugate vaccine may be used
as a booster.
\e Oral poliovirus vaccine (OPV) is recommended for routine infant
vaccination. Inactivated poliovirus vaccine (IPV) is recommended for
persons with a congenital or acquired immune deficiency disease or an
altered immune status as a result of disease or immunosuppressive
therapy, as well as their household contacts, and is an acceptable
alternative for other persons. The primary 3-dose series for IPV
should be given with a minimum interval of 4 weeks between the 1st
and 2nd doses and 6 months between the 2nd and 3rd doses.
\f The 2nd dose of MMR is routinely recommended at 4-6 years of age
or at 11-12 years of age, but may be administered at any visit,
provided at least 1 month has elapsed since receipt of the 1st dose.
\g Varicella zoster virus vaccine (Var) can be administered to
susceptible children any time after 12 months of age. Unvaccinated
children who lack a reliable history of chicken pox should be
vaccinated at the 11- or 12-year-old visit.
Source: Adapted from American Academy of Pediatrics, Committee on
Infectious Diseases, "Recommended Childhood Immunization Schedule,"
Pediatrics, 97(1), 143 and 145-a.
THE URBAN IAP AREAS FOR WHICH THE
NIS PRODUCES IMMUNIZATION COVERAGE
ESTIMATES
========================================================== Appendix II
The urban IAP project names are listed below in roman type; the names
in bold identify the largest city in the IAP county project.
Atlanta, Georgia (Fulton/DeKalb Counties)
Baltimore, Maryland
Birmingham, Alabama (Jefferson County)
Boston, Massachusetts
Chicago, Illinois
Cleveland, Ohio (Cuyahoga County)
Columbus, Ohio (Franklin County)
Dallas, Texas (Dallas County)
Detroit, Michigan
El Paso, Texas (El Paso County)
Houston, Texas
Indianapolis, Indiana (Marion County)
Jacksonville, Florida (Duval County)
Los Angeles, California
Memphis, Tennessee (Shelby County)
Miami, Florida (Dade County)
Milwaukee, Wisconsin (Milwaukee County)
Nashville, Tennessee (Davidson County)
New Orleans, Louisiana
New York City, New York
Newark, New Jersey
Philadelphia, Pennsylvania (Philadelphia County)
Phoenix, Arizona (Maricopa County)
San Antonio, Texas (Bexar County)
San Diego, California (San Diego County)
San Jose, California (Santa Clara County)
Seattle, Washington (King County)
Washington, DC (District of Columbia)
HOUSEHOLDS WITH 2-YEAR-OLD
CHILDREN THAT LACK A TELEPHONE AND
OVERALL RESPONSE RATES FOR
HOUSEHOLDS WITH TELEPHONES\A
========================================================= Appendix III
Households with 2- Overall response
year-old children rate among
that lack a households with
Area telephone telephone\b
------------------------------ ------------------ ------------------
U.S. total 11.72% 69.5%
Alabama 20.30
Jefferson County (Birmingham) 9.03 64.5
Rest of state 66.3
Alaska 9.73 100.0
Arizona 25.41
Maricopa County (Phoenix) 11.79 69.8
Rest of state 67.5
Arkansas 22.11 62.2
California 6.09
Los Angeles 6.94 78.2
Santa Clara 2.21 85.4
San Diego 3.86 73.7
Rest of state 79.5
Colorado 8.21 68.5
Connecticut 5.92 60.2
Delaware 7.49 67.3
District of Columbia 7.36 52.1
Florida 11.91
Duval County (Jacksonville) 10.68 71.8
Dade County (Miami) 9.05 58.8
Rest of state 51.6
Georgia 16.96
Fulton/DeKalb County (Atlanta) 8.31 64.2
Rest of state 75.0
Hawaii 3.24 81.8
Idaho 11.81 85.1
Illinois 7.16
Chicago 17.14 69.2
Rest of state 71.0
Indiana 13.53
Marion County (Indianapolis) 12.63 76.0
Rest of state 71.6
Iowa 7.25 65.8
Kansas 8.91% 73.4%
Kentucky 19.94 57.7
Louisiana 14.97
Orleans Parish 15.31 60.6
Rest of state 70.0
Maine 7.09 62.4
Maryland 4.00
Baltimore City 16.34 62.9
Rest of state 77.8
Massachusetts 3.42
Boston 8.89 53.5
Rest of state 66.5
Michigan 7.38
City of Detroit 12.75 80.2
Rest of state 72.4
Minnesota 3.98 71.4
Mississippi 23.25 71.2
Missouri 11.15 71.7
Montana 12.13 71.9
Nebraska 7.75 76.3
Nevada 8.31 67.6
New Hampshire 6.33 67.3
New Jersey 6.50
City of Newark 23.32 68.3
Rest of state 56.0
New Mexico 23.65 73.3
New York 6.63
New York City (5 counties) 13.93 61.2
Rest of state 68.8
North Carolina 14.44 67.5
North Dakota 5.95 77.8
Ohio 10.99
Cuyahoga County 7.59 64.8
Franklin County 4.67 65.1
Rest of state 69.1
Oklahoma 18.39 60.1
Oregon 9.34 65.1
Pennsylvania 5.02%
Philadelphia 10.25 62.0%
Rest of state 68.3
Rhode Island 7.24 61.0
South Carolina 15.86 63.9
South Dakota 11.37 78.3
Tennessee 17.33
Shelby County (Memphis) 9.43 80.0
Davidson County (Nashville) 9.67 60.8
Rest of state 66.6
Texas 18.46
Dallas County 13.10 67.1
City of Houston 17.73 71.9
El Paso County 17.22 100.0
Bexar County (San Antonio) 14.49 71.8
Rest of state 83.3
Utah 4.93 100.0
Vermont 7.64 69.0
Virginia 11.07 68.2
Washington 7.64
King County (Seattle) 2.09 69.2
Rest of state 76.7
West Virginia 23.61 56.9
Wisconsin 4.45
Milwaukee County 9.47 69.0
Rest of state 73.0
Wyoming 9.32 85.4
----------------------------------------------------------------------
\a By area surveyed, for quarters 2 through 4, 1994.
\b The overall response rate is defined as ratio between the number
of households completing the interview for at least one child and
0.0508 times the total number of households identified in the sample.
(Census data indicate that 5.8 percent of telephone households
contain 2-year-old children.) The total number of households
identified in the sample is defined as the total of (1) households
identified as ineligible; (2) households determined to be eligible;
(3) households for which eligibility was not determined; (4) 90
percent of the numbers that reached an answering machine or service,
but did not result in determination of eligibility; and (5) 90
percent of the numbers that were categorized as likely households but
for which household status was not clearly determined. The overall
response rate calculated in this fashion is slightly higher than the
response rate derived when all possible households are used in the
calculation.
Source: Abt Associates, Inc., State and Local Area Immunization
Coverage Health Survey: Final Sampling Plan (March 7, 1994), Exhibit
I pp. 4-6, submitted to the National Center for Health Statistics
under contract number 200-94-7009; and The National Immunization
Survey (NIS) 1994 Annual Methodology Report, Table 12 Appendix,
Chapter V, submitted to the National Center for Health Statistics
under contract no. 200-94-7009.
(See figure in printed edition.)Appendix IV
COMMENTS FROM THE DEPARTMENT OF
HEALTH AND HUMAN SERVICES
========================================================= Appendix III
See comments 1 and 2.
See comment 3.
See comment 4.
See comment 1.
See comment 1.
See comment 5.
(See figure in printed edition.)
See comment 6.
See comment 2.
See comment 7.
See comment 8.
See comment 9.
See comment 10.
(See figure in printed edition.)
See comment 11.
See comment 12.
See comment 11.
See comment 13.
See comment 14.
See comment 15.
See comment 8.
See comment 11.
See comment 16.
(See figure in printed edition.)
See comment 17.
See comment 3.
See comment 1.
(See figure in printed edition.)
See comment 18.
See comment 19.
See comment 20.
See comment 21.
(See figure in printed edition.)
See comment 21.
See comment 22.
See comment 23.
(See figure in printed edition.)
See comment 24.
See comment 25.
See comment 26.
See comment 27.
See comment 28.
See comment 29.
The following are GAO's comments on the Department of Health and
Human Services' letter dated July 22, 1996.
GAO COMMENTS
1. CDC has mischaracterized our findings. Although we have
identified several issues that raise questions about accuracy,
neither we nor CDC can validate the accuracy of survey results. The
accuracy of the NIS results depends on the accuracy of the
assumptions inherent in CDC's adjustment of the survey results, some
of which are untestable. The results of the NHIS are used to adjust
the results of the NIS. Thus, while the similarity of the two
surveys is reassuring, the NHIS cannot provide an independent
assessment of the NIS' accuracy. In any event, the agreement of the
national estimates does not ensure that the local estimates are
accurate.
2. The various benefits asserted by CDC derive from the application
of the NIS to monitoring immunization rates and to comparing them
across states. We acknowledged both of these objectives in the
second paragraph of our report.
Many potential benefits or purposes could be asserted for the survey,
but its use in any of these capacities is limited by the low
precision, narrow range, and unverified accuracy of the survey
estimates.
3. It is true that surveys, to varying degrees, customarily require
the types of adjustments applied to the NIS to correct for biases
introduced by nonresponse and limitations in survey coverage.
However, the adjustment of NIS results for exclusion of households
without telephone service required a somewhat greater leap of faith
than customary adjustments for telephone noncoverage.
The success of such adjustments usually depends on the extent to
which the variable being measured can be accurately predicted by
demographic characteristics that are available or can be inferred for
both nontelephone and telephone households. As we have noted in the
report, based on data from the NHIS, which is an in-person survey,
there are large differences in immunization coverage between children
in households with and without telephones. These differences are not
completely explained by demographic differences between telephone and
nontelephone households.
Furthermore, although telephone ownership varies substantially across
the surveyed areas, there are no consistent sources of state and
local data on differences in immunization coverage between telephone
and nontelephone households. Consequently, the extent to which this
adjustment improved the accuracy of state and local survey results is
unclear.
4. CDC has acknowledged that the NIS does not identify pockets of
children in need of more timely immunization, and most state
immunization program managers have told us that the NIS does not help
them in targeting their efforts or designing interventions, although
it does relieve them of CDC's previous requirement that they collect
statewide coverage data on their own. We are studying alternative
means for identifying pockets of need.
Although there is currently no other means of comparing statewide
immunization coverage data, the NHIS, as we have noted, tracks
coverage changes at the national level. In addition, other methods
were used in the past to collect statewide coverage information,
albeit through a variety of methods across states.
5. It is true that the sample size of the NIS should afford the
calculation of rates for such subgroups on a national basis. The
NHIS is not currently large enough to provide childhood immunization
coverage information on these two groups.
6. CDC states that the NIS is an "important public health management
tool" and notes that Missouri, Arizona, and Idaho have taken steps
intended to improve immunization coverage in the wake of NIS results.
However, we have some concern that the NIS provides no guidance on
the type of action that is appropriate or where it is appropriate.
It is not necessarily clear that placing special emphasis on the
states with the lowest survey estimates for coverage with a
combination of four vaccines is the most appropriate way to prevent a
disease outbreak. States with high estimates may nonetheless include
significant pockets of underimmunized children.
7. CDC provides no evidence that the NIS is cost-effective. As we
note in our conclusion, it is markedly more expensive than the
retrospective surveys previously used to generate statewide coverage
data. Presuming that the capacity to measure differences between
states is an important objective, the NIS' capacity to meet this
objective is limited by the broad margins of error in survey
estimates and variations in survey participation and coverage. It is
similarly limited with respect to monitoring changes in immunization
coverage across time. As with previous state surveys, there is no
guarantee that the NIS provides unbiased estimates of immunization
coverage.
8. The NIS can detect small changes on a quarterly basis only at the
national level. Survey results are not released until roughly a year
after data collection, and it is doubtful that a 1-percent change in
national coverage should or would be construed as an early warning in
the context of very high vaccine-specific rates. In any case,
national coverage statistics are also available from the NHIS.
Availability of the NIS results did not prevent the recent outbreak
of measles in Utah.
Sudden drops in immunization levels for a particular disease in other
countries have been associated with problems, such as sudden concerns
about vaccine safety, that were evident apart from immunization
measurement. There was concern and widespread publicity in the
mid-1970s in both the United Kingdom and Japan about reports of
encephalitis following the receipt of pertussis vaccine. The reduced
utilization of this vaccine was precipitous and observable from
sources other than national survey data.
9. While states with lower immunization estimates may be motivated
by the NIS findings to improve coverage, the findings do not indicate
where the problem lies within these states or what corrective actions
are needed. We remain concerned that they may provide a false sense
of security to other areas that actually face significant problems
(for example, specific pockets of low immunization within states with
generally high coverage rates). In addition, the motivational
effects of such quarterly ranking may diminish over time. Finally,
CDC's argument presumes that states will be more motivated to act by
data collected through the NIS than they would have been by data
collected locally or through other means. We disagree.
10. The CDC has indicated that the NIS was not intended to identify
pockets of need and consequently does not do so. The NIS may
actually deflect attention from some serious problem areas because
they are incorporated in larger areas for survey purposes. For
example, the Seattle-King County Department of Public Health and the
University of Washington conducted a separate household survey of
Central and Southeast Seattle using the same age group and reference
dates as the NIS, but finding that 57 percent of children in this
part of the city were fully immunized, in contrast to the NIS rate of
79 percent up-to-date for all of King County in the same time
period.\1
11. Because of the wide margins of error of survey estimates, the
NIS is probably not sufficiently sensitive to permit evaluation of
interventions or policy changes in particular areas or subgroups.
Although national changes in immunization coverage may be monitored
with greater precision, changes in national or local immunization
coverage might be attributable to factors other than policy changes
(for example, trends in the demographic characteristics of children
to be immunized). Moreover, policy changes typically occur in groups
and are implemented gradually, which would make it quite difficult to
attribute any observed movements in immunization coverage to a single
change or a combination of changes.
In this context, it seems inadvisable to draw conclusions about
particular state activities based solely on the results of the NIS.
Similarly, with cross-state comparisons, multiple interventions are
linked to each area and subgroup, as well as variations in
demographic and other factors, making it difficult to disentangle the
reasons for any differences observed across states and cities in the
NIS findings.
12. We agree that the NIS is technically capable of detecting the
rapid and dramatic changes in coverage that typically accompany the
recommendation of new vaccines. However, on a national level, the
NHIS also reports on the uptake of newly recommended vaccines. Sales
and distribution reports may provide a less expensive means of
monitoring the uptake of such vaccines in particular areas.
13. Even small states had produced statewide coverage estimates
using previous methods. However, it is difficult for small states to
justify the use of $165,000 in infrastructure funding for a random
digit dialing immunization survey such as the NIS. Under a proposal
CDC has floated with states, surveys in small states would be
subsidized by "contributions" of a percentage of federal grant funds
from larger states. However, in view of immunization needs, 20 state
health officers surveyed by ASTHO could not justify devoting 6.5-10
percent of their infrastructure funds to survey support. Twenty-four
states told ASTHO they were willing to contribute 10 percent of their
1995 infrastructure grant toward the survey in the event that federal
funding was discontinued, but their prospective contributions would
have totaled $4.6 million--much less than the survey's reported
annual cost.
14. As we have noted, the precision of current estimates raises
questions about whether the survey does, in fact, provide an improved
basis for the distribution of incentive funds. Moreover, the amount
expended on the survey is substantial in comparison to the amount of
such funds available for distribution.
15. Most state immunization program managers indicated that the NIS
results were not useful in targeting their activities. Although a
low result may provide some states with a general incentive to do
better, it provides no guidance as to how to accomplish any
improvement.
16. The collection of such data will enhance the information derived
from the tremendous number of phone contacts with ineligible
households made in conducting the NIS. However, the collection of
immunization data may continue to drive the number of calls required
(and hence the cost of the survey) because households containing
two-year-olds would likely continue to be the rarest population
sampled. In any case, the utility of the survey for collecting other
data does not bear upon its usefulness for collecting information on
immunization.
17. CDC agrees with the cost we reported for the NIS. We did not
verify CDC's claims regarding its use of the funds that were not
applied to the survey. While CDC anticipates that future costs will
be lower, it has not requested modification of its fiscal 1997 budget
request to reflect these lower costs.
18. The poor quality of immunization data gathered from household
respondents had been documented before the NIS was planned. Thus,
although the provider surveys may have reduced the inaccuracies
contained in these household data, the survey might have been more
efficiently designed had the limitations of household data been
acknowledged in survey planning. Earlier recognition of this problem
would have supported more serious exploration of other survey
methods.
19. It should be noted that CDC's comments compare the survey
estimates to a standard different from the target established in the
contract and survey plan. Survey plans are ordinarily drawn by
determining the sample size necessary to achieve an acceptably
precise result if the value of the measured variable is near 50
percent, the point at which the largest sample will be required to
achieve a given level of precision (for example, plus or minus 5
percent with 95-percent confidence). This is exactly the sampling
target specified in CDC's contract with the survey organization.
Insofar as the immunization levels measured by the survey are well
above 50 percent, had the targets established in the contract been
met, the estimates would show precision better than plus or minus 5
percent.
Further, CDC's statement that, "Seventy-one of the 78 areas met or
exceeded the requirement that the margin of error be within five
percent of the value of the estimate itself," does not conform to the
first four quarters of survey results published by CDC (see MMWR,
Feb. 23, 1996, pp. 148-49). These indicate that, for 4:3:1
coverage, only 23 of the 78 estimates met or exceeded the criterion
that the margin of error be within 5 percent of the value of the
estimate itself. For 4:3:1:3 coverage, the number meeting or
exceeding this criterion was only 16 of the 78.
Whether the survey estimates met this or any other criterion is less
important than the fact that their precision, if not improved, is
generally only sufficient to detect, reliably, changes of a size
larger than has typically been observed on an annual basis.
While the addition of provider data has helped correct some
substantial errors incorporated in household responses, it has not
reduced the margins of error for survey estimates.
20. We do not find that the survey documents high levels of
variability in results across IAPs. Although CDC correctly states
that Alabama's result was statistically different from the result for
21 other IAP areas (11 states and 10 cities), it cannot be
statistically distinguished from the results in 56 others. CDC is
correct that, in most cases, differences of at least 10 points can be
statistically distinguished, as we show in figure 1 for 4:3:1
coverage, but there is only a 24-point range in the state estimates
for full coverage, so the majority of the state estimates--31--are
not far enough apart for their difference from the national estimate
to be confidently attributed to anything more than sampling error.
The range of estimates for coverage with particular vaccines is
generally narrower.
21. The NIS can detect reasonably small changes in national coverage
between consecutive four-quarter annualized estimates, though the
first two successive annualized estimates for 4:3:1 coverage were not
different. However, even at the national level, for most of the
antigens and series, the smallest reliably detectable change (at
conventional levels of significance) is slightly larger than 1
percent.
At conventional levels of significance, it is impossible to judge
differences as small as 5 percent to be statistically significant
when most estimates have 95 percent margins of error of 5 percent or
greater. Our report quotes a statement in a document issued by the
survey contractor noting that the survey can detect only very large
changes (for example, a 20-percent increase from 50 percent) between
successive quarterly annualized estimates in the various areas
surveyed. The margins originally planned would have been no larger
than plus or minus 5 percent. However, survey documentation NCHS
provided to us notes that "Confidence intervals for the vaccination
coverage estimates are somewhat wider than originally planned because
provider information is not available for all children in the
sample."\2 In addition, for data collected in quarters 2 through 4 of
1994, the number of completed child-level interviews was less than 90
percent of the sample size called for in the design specifications
for roughly a third of the IAPs. This too, would have the effect of
increasing the margins of error for survey estimates.
22. While the NIS applies the same methodology across states, the
range of state results is not as broad as expected and the
performance of many states cannot be differentiated. In any case, in
making such comparisons with the NIS, it is important to take into
account the wide variations in survey coverage and response rates
across states and urban areas.
23. We noted that the retrospective survey approach has both
advantages and disadvantages, including the timeliness of data.
Retrospective surveys do not produce results as quickly as the NIS;
however, even the NIS issues results about a year after data
collection, and thus it appears equally ill-suited to provide an
early warning.
24. As we have noted in appendix III, the NIS in some areas excludes
a similar proportion of children living in households without
telephone service.
25. This is generally true, although the costs of a household survey
can be comparable in some urban areas, as suggested by recent
experience in Norfolk and Seattle.
26. There may be some economies of scale in centralizing the surveys
under a single contract, but these must be weighed against the costs
of limiting potential bidders to firms equipped to handle a task of
this large scale. Conducting separate surveys would have the
advantage of permitting the questions to be tailored to provide
additional data about state and local initiatives.
27. It is true that the full cost of a random digit dialing survey
such as the NIS would be more difficult for smaller states to bear.
ASTHO officials reported that many smaller states were unwilling to
continue participation in the survey if it meant funding the full
cost of their own random digit dialing survey through their
infrastructure funding. However, it should be noted that all states
have recent experience conducting other types of statewide
immunization surveys.
28. Minimal staff hours are generally involved in retrospective
surveys. While this is not true of household surveys, states may
also contract for such services if they continue to be required.
29. As noted in our report, the Congress may wish to weigh the cost
of the NIS against its benefits in order to ensure the most efficient
use of immunization resources.
--------------------
\1 See Seattle-King County Department of Public Health and The
University of Washington School of Public Health and Community
Medicine (February 1996). "Household Immunization Survey of Central
and Southeast Seattle April-August 1995: Final Report."
\2 "National Immunization Survey: EZGuide to Estimation" (undated),
p. 7.
MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V
PROGRAM EVALUATION AND METHODOLOGY
DIVISION
Betty Ward-Zukerman, Project Manager
Richard C. Weston, Senior Social Science Analyst
Brian Keenan, Assistant Director for Survey Methodology
Venkareddy Chennareddy, Referencer
OFFICE OF GENERAL COUNSEL
George Bogart, Senior Attorney
*** End of document. ***