    ONDCP Media Campaign: Contractor's National Evaluation Did Not
    Find that the Youth Anti-Drug Media Campaign Was Effective in
    Reducing Youth Drug Use (25-AUG-06, GAO-06-818).

    Between 1998 and 2004, Congress appropriated over $1.2 billion to
    the Office of National Drug Control Policy (ONDCP) for the
    National Youth Anti-Drug Media Campaign. The campaign aimed to
    prevent the initiation of or curtail the use of drugs among the
    nation's youth. In 2005, Westat, Inc., completed a multiyear
    national evaluation of the campaign. GAO has been mandated to
    review various aspects of the campaign, including Westat's
    evaluation which is the subject of this report. Applying
    generally accepted social science research standards, GAO
    assessed (1) how Westat provided credible support for its
    findings and Westat's findings about (2) attitudes, beliefs, and
    behaviors of youth and parents toward drug use and (3) youth
    self-reported drug use.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-06-818
        ACCNO:   A59564
      TITLE:     ONDCP Media Campaign: Contractor's National Evaluation
    Did Not Find that the Youth Anti-Drug Media Campaign Was
    Effective in Reducing Youth Drug Use
         DATE:   08/25/2006
      SUBJECT:   Advertising
    	     Controlled substances
    	     Drug abuse
    	     Drugs
    	     Evaluation methods
    	     Program evaluation
    	     Youth
    	     Drug abuse prevention
    	     Data collection
    	     Public relations
    	     Surveys
    	     Teenagers
    	     ONDCP National Youth Anti-Drug Media
    	     Campaign

    	     National Survey of Parents and Youth

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-06-818

         * Results in Brief
         * Background
              * The National Youth Anti-Drug Media Campaign
              * Planning and the Underlying Logic of the Campaign
              * Campaign Activities during Phase III
              * Campaign Themes and Messages
              * Design of the Evaluation, Interim Evaluation Reports, and Re
              * Assessment of the Campaign by the Office of Management and B
              * Recent Research on the Effects of the Campaign in Local Sett
         * Westat's Evaluation Design, Use of Generally Accepted and Ap
              * Although Elements of the Campaign Limited Choices of Evaluat
              * Sample Coverage Issues Did Not Invalidate Westat's Assessmen
              * Sample Attrition across NSPY Interview Rounds Was Sufficient
              * The NSPY Data Could Be Used to Detect Reasonably Small Effec
              * Westat's Analytic Methods Aimed to Isolate Causal Effects of
         * The Phase III Evaluation Provided Mixed Evidence of the Camp
              * Youth and Parents' Recall of Campaign Advertisements Increas
              * Westat Found That the Campaign Generally Had No Effect on th
              * The Evaluation Reported Favorable Effects of the Campaign on
              * No Evidence of Favorable Effects of the Campaign on Youth Ou
         * The Phase III Evaluation Found No Significant Effects of Exp
              * Westat Tracked Trends in Marijuana Use from Several Sources
              * Westat Reported That Trends in Marijuana Offers Declined ove
              * On the Basis of Its Analysis of the Association between Expo
         * Conclusions
         * Matter for Congressional Consideration
         * Agency Comments and Our Evaluation
              * Westat Evaluation's Role in Judging the Impact of the Advert
              * ONDCP Made Campaign Changes as a Result of Westat Interim Fi
              * Other Youth Drug Use Findings
              * Steps Taken to Remedy Potential Problems
              * ONDCP Cites Major Changes in Campaign
              * ONDCP Offers an Alternative Explanation for Counterintuitive
              * ONDCP Takes Issue with the Timing of Our Review
              * Points Concerning Our Matter for Congressional Consideration
              * ONDCP Posits Consequences of Further Budget Cuts
              * Coverage in the NSPY
              * NSPY and CPS Comparisons of Distributions on Analyzed Variab
              * Undercoverage in the NSPY and Other Widely Known and Used Lo
              * Sample Attrition across NSPY Interview Rounds
              * Comparisons of Respondents and Nonrespondents across NSPY Su
              * Differences in Sampling Methodologies between NSPY and MTF
              * The Capacity of the NSPY to Detect Reasonably Small Effects
              * Westat Methods to Measure Outcomes
         * GAO Contacts
         * Acknowledgments
         * GAO's Mission
         * Obtaining Copies of GAO Reports and Testimony
              * Order by Mail or Phone
         * To Report Fraud, Waste, and Abuse in Federal Programs
         * Congressional Relations
         * Public Affairs

    Report to theSubcommittee on Transportation, Treasury, the Judiciary,
    Housing and Urban Development, and Related Agencies, Committee on
    Appropriations, U.S. Senate

    United States Government Accountability Office

    GAO

    August 2006

    ONDCP MEDIA CAMPAIGN

    Contractor's National Evaluation Did Not Find That the Youth Anti-Drug
    Media Campaign Was Effective in Reducing Youth Drug Use

    Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth
    Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug
    Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media
    Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign
    Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth
    Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug
    Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media
    Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign
    Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth
    Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug
    Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media
    Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign
    Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth
    Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug
    Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media
    Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign
    Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth
    Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug
    Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media
    Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign
    Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth
    Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug
    Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media
    Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign
    Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth
    Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug
    Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media
    Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign
    Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth
    Anti-Drug Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug
    Media Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media
    Campaign Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign
    Youth Anti-Drug Media Campaign Youth Anti-Drug Media Campaign

    GAO-06-818

    Contents

    Letter 1

    Results in Brief 5
    Background 8
    Westat's Evaluation Design, Use of Generally Accepted and Appropriate
    Sampling and Analytic Techniques, and Reliable Methods for Measuring
    Campaign Exposure Produced Credible Evidence to Support Its Findings 22
    The Phase III Evaluation Provided Mixed Evidence of the Campaign's
    Effectiveness on Intermediate Outcomes, but It Found No Effect of the
    Campaign on Parental Monitoring of Youth 32
    The Phase III Evaluation Found No Significant Effects of Exposure to the
    Campaign on Youth Drug Use Outcomes Other than Limited Unfavorable Effects
    on Marijuana Initiation 37
    Conclusions 42
    Matter for Congressional Consideration 44
    Agency Comments and Our Evaluation 44
    Appendix I Westat's Methods for Addressing Evaluation Implementation
    Issues 52
    Appendix II Comments from the Office of National Drug Control Policy 65
    Appendix III GAO Contacts and Staff Acknowledgments 72

    Tables

    Table 1: Appropriations for the National Youth Anti-Drug Media Campaign,
    Fiscal Years 1998 through 2006 10
    Table 2: NSPY Survey Rounds and Response Rates, Sampled and Surveyed Youth
    56

    Figures

    Figure 1: Data Collection Rounds and Waves of the NSPY 15

    Abbreviations:

    CPS Current Population Survey GRP gross rating points MTF Monitoring the
    Future NIDA National Institute on Drug Abuse NIP National Immunization
    Program NIS National Immunization Survey of Children NLSY National
    Longitudinal Survey of Youth NSDUH National Survey on Drug Use and Health
    NRC National Research Council NSPY National Survey of Parents and Youth
    OMB Office of Management and Budget ONDCP Office of National Drug Control
    Policy PART Performance Assessment Rating Tool PATS Partnership for a Drug
    Free America's Attitude Tracking Survey PDFA Partnership for a Drug Free
    America PME Performance Measures of Effectiveness YRBSS Youth Risk
    Behavior Surveillance System

    reproduce this material separately.

    This is a work of the U.S. government and is not subject to copyright
    protection in the United States. It may be reproduced and distributed in
    its entirety without further permission from GAO. However, because this
    work may contain copyrighted images or other material, permission from the
    copyright holder may be necessary if you wish to This is a work of the
    U.S. government and is not subject to copyright protection in the United
    States. It may be reproduced and distributed in its entirety without
    further permission from GAO. However, because this work may contain
    copyrighted images or other material, permission from the copyright holder
    may be necessary if you wish to

    United States Government Accountability Office

    Washington, DC 20548

    August 25, 2006

    The Honorable Christopher Bond Chairman The Honorable Patty Murray Ranking
    Member Subcommittee on Transportation, Treasury, the Judiciary, Housing
    and Urban Development, and Related Agencies Committee on Appropriations
    United States Senate

    Congressionally mandated under the Treasury and General Government
    Appropriations Act of 1998,1 the National Youth Anti-Drug Media Campaign
    had the primary goals of preventing the initiation of drug
    use-particularly the use of entry-level drugs marijuana and
    inhalants-among the nation's youth and stopping youth that have begun
    using drugs from continuing their use. Administered through the Office of
    National Drug Control Policy (ONDCP) and implemented in three phases, the
    campaign featured as its centerpiece a paid advertising effort in which
    campaign funds were used to purchase media time and space for
    advertisements that delivered anti-drug messages to the campaign's target
    audience-youth aged 9 to 18 and their parents-through strategic placement
    of anti-drug advertisements on television and radio and in print media. In
    addition to the advertising, the campaign included community outreach,
    work with the entertainment industry to encourage the accurate depiction
    of the consequences of drug use, outreach to faith-based organizations,
    and work with youth organizations. The campaign's first two phases, which
    ran from January 1998 through the summer of 1999, were pilot phases that
    focused primarily on informing the planning and development for phase III
    and included a 12-city pilot (phase I) and nationwide advertising (phase
    II). Phases I and II aimed to increase public awareness of anti-drug
    messages. Phase III of the campaign, which began in mid-1999, continued
    the nationwide advertising campaign begun during phase II and integrated
    the advertising with outreach efforts. From fiscal year 1998 through
    fiscal year 2006, Congress appropriated over $1.4 billion to support the
    campaign. For fiscal year 2007, the President's budget requested $120
    million for the campaign, an increase over the fiscal year 2006
    appropriation, to purchase additional media time and space to increase the
    reach and frequency of the campaign's messages, which would restore
    appropriations to their fiscal year 2005 level.

    1Pub. L. No. 105-61, 111 Stat. 1272.

    Congress first authorized funding for the campaign in fiscal year 1998
    with the expectation that demonstrable changes in youth drug behaviors
    would be apparent within 3 years, and Congress required ONDCP to assess
    whether the campaign's efforts have been effective in changing the drug
    use behaviors of America's youth. ONDCP also indicated that it anticipated
    that it would take 2 to 3 years for the campaign to affect drug use
    behavior, although ONDCP also indicated that it was with the
    implementation of phase III of the campaign, beginning in mid-1999, that
    ONDCP expected to see improvements in anti-drug attitudes that would lead
    to decreases in youth drug use within 3 years. We previously reported that
    ONDCP's evaluations of the first two phases of the campaign produced
    inconclusive results because of various evaluation implementation problems
    and limitations of the analyses used to support findings about effects
    during these pilot phases.2 In particular, we noted that the impact
    evaluations of phases I and II did not adequately gauge the overall level
    of anti-drug awareness generated by the campaign-the principal outcome
    measure for these two phases-and we identified site selection problems,
    unknown parent response rates, low school response rates, and data
    analysis issues contributing to the inconclusive results.

    To implement the phase III evaluation, ONDCP entered into an interagency
    agreement with the National Institute on Drug Abuse (NIDA), which in turn
    awarded contracts to Westat, Inc., through June 2005 for $42.7 million to
    conduct the evaluation. Westat subcontracted with the Annenberg School for
    Communication at the University of Pennsylvania, and staff from Westat and
    Annenberg were coprincipal investigators for the study.3 Westat's phase
    III evaluation covered the period from September 1999 through June 2004
    and studied the impact of the effectiveness of the nationwide campaign in
    reaching its target audience; affecting youth beliefs, attitudes,
    intentions, and behaviors with regard to drug use; and affecting their
    parents' beliefs and attitudes toward drug use and affecting parents'
    behaviors associated with interacting with their children and monitoring
    their activities.

    2See: GAO, Anti-Drug Media Campaign: ONDCP Met Most Mandates, but
    Evaluations of Impact Are Inconclusive, GAO/GGD/HEHS-00-153 (Washington,
    D.C.: July 31, 2000). We also reported on ONDCP's use of consultants in
    the campaign in GAO, Anti-Drug Media Campaign: An Array of Services Was
    Provided, but Most Funds Were Committed to Buying Media Time and Space,
    GAO-05-175 (Washington, D.C.: Mar. 31, 2005). We also described the phase
    III evaluation in GAO, Program Evaluation: Strategies for Assessing How
    Information Dissemination Contributes to Agency Goals, GAO-02-923
    (Washington, D.C.: Sept. 30, 2002).

    Westat evaluated the campaign using a longitudinal panel survey-the
    National Survey of Parents and Youth (NSPY)-that aimed to measure the
    campaign's effectiveness by assessing changes in various outcomes within
    individuals over time in relation to their exposure to campaign messages.
    Westat's evaluation assessed the effect of exposure to the campaign on
    youth drug use and on several key intermediate outcomes-such as youth and
    parent attitudes toward and beliefs about drug use and parental
    involvement with their children-that were believed to influence youth drug
    use. In conducting the evaluation, Westat submitted several interim
    reports that were used in part to inform decisions about the direction of
    the campaign. Westat submitted a draft of the final evaluation report to
    NIDA in February 2005.4

    Both Congress and ONDCP recognized the need for a separate evaluation of
    the campaign, because of limitations associated with existing national
    surveys of drug use. Congressional conferees acknowledged their intention
    to rely on the evaluation to gauge the impact of the campaign, and also
    indicated that if the campaign failed to show its effectiveness, they
    would be compelled to reevaluate the use of taxpayer money to support it.
    ONDCP acknowledged that existing national surveys of drug use would not be
    able to answer the critical question of whether changes in drug use
    behavior and attitudes were the result of the campaign. These surveys do
    not ask respondents about their exposure and reactions to the messages of
    the campaign that can then be linked to their drug-related attitudes and
    behavior. For example, in a 2001 report on youth drug use and the
    campaign, ONDCP officials noted that while national surveys of youth drug
    use showed flattening or declining youth marijuana use in 1999 and 2000
    and these trends suggested that the campaign may be having the desired
    impact, it was necessary to await the results of the campaign's
    independent evaluation before drawing any definitive conclusions regarding
    the campaign's contribution to changes in youth drug use.

    3Hereafter, we refer to the contractor as "Westat," and this implicitly
    includes Annenberg. In addition, a second subcontractor, the National
    Development and Research Institutes, Inc., provided expertise in
    developing drug use questions and assisted in preparing the first special
    topics report on trends in drug use.

    4Westat and Annenberg jointly submitted to NIDA all evaluation reports
    except for the final report, which was submitted by Westat only.

    In a committee report for the fiscal year 2004 appropriations cycle, the
    Senate Appropriations Committee directed us to review how consultants were
    used in support of the media campaign. 5 This is the second of two reports
    responding to this mandate.6 The first report provided information
    concerning ONDCP's use of consultants in the campaign. This second report
    addresses three questions related to Westat's evaluation of phase III of
    the media campaign: (1) How did Westat ensure that it could report
    credible results in its evaluation of the campaign? (2) What did the
    evaluation find about the effect of exposure to the campaign on key
    intermediate outcomes that were intended to lower youth drug use? (3) What
    did the evaluation find about the effect of exposure to the campaign on
    youth drug use?

    In addressing our objectives, a team of GAO social scientists reviewed and
    assessed materials related to Westat's phase III evaluation, applying
    generally accepted social science research standards, including such
    elements as when and how the sample data were collected, adjustments made
    to the sample to address nonresponse, how program effects were isolated
    (i.e., the use of statistical controls), and the appropriateness of
    outcome measures. The materials reviewed included interim and final
    evaluation reports, documentation and analyses provided by Westat to us in
    response to several sets of questions that we submitted about the details
    of its methodology, documentation pertaining to meetings of scientific
    panels that provided guidance on the evaluation, and documentation
    prepared by ONDCP about the design and implementation of the campaign. We
    conducted our work in accordance with generally accepted government
    auditing standards from October 2005 through June 2006.

    5Senate Report No. 108-146, at 143 (2003).

    6See GAO-05-175 for our review of ONDCP's use of consultants in the
    campaign.

                                    Results in Brief

    By collecting longitudinal data-i.e., multiple observations on the same
    persons over time-using generally accepted and appropriate sampling and
    analytic techniques, and establishing reliable methods for measuring
    campaign exposure, Westat was able to produce credible evidence to support
    its findings about the relationship between exposure to campaign
    advertisements and both drug use and intermediate outcomes. In
    implementing the study, Westat encountered problems that are common to
    large-scale longitudinal studies, and it addressed those using methods
    that are generally recognized as appropriate approaches for the study
    implementation challenges Westat faced. Challenges that Westat encountered
    were (1) lack of baseline data, which precluded Westat from comparing
    postprogram outcomes to preprogram conditions, and the redirection of the
    campaign; (2) sampling concerns, particularly ensuring the coverage of
    eligible households with youth in the targeted age range and ensuring that
    attrition over successive survey cycles did not result in insufficient
    sample size to detect campaign effects or in systematic bias within the
    sample; (3) establishing measures that would allow for both the sufficient
    detection of and the reliable measurement of exposure to the campaign on
    NSPY survey respondents; and (4) disentangling causal effects of exposure
    and drawing meaningful comparisons in the absence of ability to employ an
    experimental design where NSPY respondents would have been randomly
    assigned to various levels of exposure- the generally preferred approach
    for assessing program effects, where possible. Our examination of Westat's
    evaluation report and related documentation leads us to conclude that it
    addressed each of these challenges sufficiently to allow it to report
    credible findings about the effect of campaign exposure on drug use and
    intermediate variables believed to be precursors to drug use.
    Specifically, (1) several factors suggest that the lack of baseline data
    was not fatal to the evaluation's findings, and Westat was able to
    generate statistically significant findings related to the redirected
    campaign; (2) Westat found no evidence of bias in the NSPY estimates
    despite sample coverage losses, and it also maintained high follow-up
    response rates of sampled individuals to provide for robust longitudinal
    analysis; (3) the NSPY sample could be used to detect changes in outcomes
    that were on the order of magnitude of changes that ONDCP expected for the
    campaign, and Westat demonstrated that its measures of exposure were valid
    and could reliably predict outcomes, whether results of the associations
    between exposure and outcomes were favorable or unfavorable to the
    campaign; and (4) using sophisticated statistical methods, Westat matched
    respondents on their underlying propensity to be exposed to campaign
    advertisements and, by comparing differences in outcomes among groups with
    different levels of exposure resulting from its matching methods, isolated
    the effects of the campaign from other variables. (See appendix I for
    further details.)

    For intermediate outcome measures thought to influence the ultimate target
    of the campaign, youth drug use-for example, recall and identification of
    campaign messages, youth anti-drug attitudes, and parents' beliefs and
    behaviors-Westat found favorable effects for some measures and subgroups,
    as well as unfavorable effects and no significant effects for others. In
    general, both youth and parents' recall of specific campaign messages
    increased over the life of the campaign. In addition, NSPY trend data
    showed some increasing trends in anti-drug attitudes and beliefs as well
    as the proportion of youth who reported never intending to try marijuana.
    However, cross-sectional and longitudinal analysis provided no evidence
    that these trends resulted from campaign exposure. Westat's analysis also
    indicated that among current, non-drug-using youth, exposure to the
    campaign had unfavorable effects on their anti-drug norms and perceptions
    of other youths' use of marijuana-that is, greater exposure to the
    campaign was associated with weaker anti-drug norms and increases in the
    perceptions that others use marijuana. Data for parents in the NSPY on
    five intermediate measures show some favorable effects of campaign
    exposure on parents' behaviors and beliefs. However, for a major aim of
    the campaign, affecting parental behaviors regarding monitoring their
    children's whereabouts, activities, and friends, Westat found no evidence
    of a significant effect. Moreover, where the data showed favorable
    relationships between campaign exposure and parental beliefs and
    behaviors, Westat did not find that these effects on parents ultimately
    lead to corresponding changes in their children's beliefs and behaviors.

    Westat's evaluation found no significant favorable effects of campaign
    exposure on marijuana initiation among non-drug-using youth or cessation
    and declining use among prior marijuana users. Westat's NSPY data did show
    some declining trends in self-reported lifetime and past-month use of
    marijuana by youth over the period from 2002 to 2004 and declining trends
    in youth reports of offers to use marijuana. Declining drug use trends in
    the NSPY were consistent with trends in other national surveys of drug use
    over these years. However, Westat cautioned that because trends do not
    account for the relationship between campaign exposure and changes in
    self-reported drug use, trends alone should not be taken as definitive
    evidence that the campaign was responsible for the declines. ONDCP has
    also acknowledged the limitation of drug use trends for the purpose of
    demonstrating a causal link between campaign exposures and declines in
    drug use trends. Westat's analysis of the relationship between exposure to
    campaign advertisements and youth self-reported drug use in the NSPY data
    for the entire period covered by its evaluation-assessments that used
    statistical methods to adjust for individual differences and control for
    other factors that could explain changes in self-reported drug use- showed
    no significant effects of exposure to the campaign on initiation of
    marijuana by prior nonusing youth. Westat's analysis found significant
    unfavorable effects-that is, a relationship between campaign exposure and
    higher rates of initiation-during one round of NSPY data and for the whole
    period of the campaign among certain subgroups of the sample (e.g., 12
    1/2- to 13-year-olds and girls). Westat found no effects of campaign
    exposure on rates of quitting or use by prior users of marijuana.

    In light of the fact that the phase III evaluation of the media campaign
    yielded no evidence of a positive outcome in relation to teen drug use and
    congressional conferees' indications of their intentions to rely on the
    Westat study, Congress should consider limiting appropriations for the
    National Youth Anti-Drug Media Campaign beginning in the fiscal 2007
    budget year until ONDCP is able to provide credible evidence of the
    effectiveness of exposure to the campaign on youth drug use outcomes or
    provide other credible options for a media campaign approach. In this
    regard we believe that an independent evaluation of the new campaign
    should be considered as a means to help inform both ONDCP and
    Congressional decision making.

    We provided a draft of this report to the Director of ONDCP for comment.
    In response, ONDCP provided written comments (reproduced in appendix II),
    which stated that ONDCP was puzzled that we did not make recommendations
    to it about how to improve the campaign. However, the main purpose of our
    report was to assess Westat's evaluation rather than to comment on how to
    improve the media campaign. In so doing, we focused on Westat's methods.
    Our role was to inform Congress about the reliability of Westat's
    evaluation so that Congress could decide the extent to which it will
    continue to fund the campaign.

    ONDCP expressed a number of concerns about our assessment of Westat's
    evaluation and its implications concerning the effectiveness of the
    campaign. Most importantly, it stated that the Westat study is ill suited
    to assess impact and the study's findings are of limited relevance. Our
    extensive review of the Westat study does not support ONDCP's conclusion.
    Westat successfully addressed implementation challenges and used
    sophisticated analytic techniques to develop its findings. Another major
    issue ONDCP presents in its comments deals with the fact that the campaign
    has made major changes since the Westat data collection, rendering the
    study's findings irrelevant. Neither we nor ONDCP has factual data upon
    which to base an assessment of the effectiveness of the current campaign.
    However, other major efforts to substantially change the campaign during
    the time frame of the Westat data collection did not yield positive
    results. ONDCP raised a number of other issues that are generally related
    to the issues discussed above. These are addressed in the Agency Comments
    and Our Evaluation section of this report.

                                       Background

    The National Youth Anti-Drug Media Campaign

    As part of the Treasury and General Government Appropriation Act of 1998,7
    the Drug Free Media Campaign Act of 1998 required, among other things, the
    Office of National Drug Control Policy to conduct a national media
    campaign for the purpose of reducing and preventing drug abuse among young
    people in the United States.8 The National Youth Anti-Drug Media Campaign
    may be the most visible federal effort devoted to preventing drug use
    among the nation's youth. It aims to educate and enable America's youth to
    reject illegal drugs; to prevent youth from initiating use of drugs,
    especially marijuana and inhalants; and to convince occasional users of
    these and other drugs to stop using drugs. Administered by ONDCP, and
    implemented in three phases, the campaign has as its centerpiece a paid
    advertising effort in which campaign funds were used to purchase media
    time and space for advertisements that delivered anti-drug messages to the
    campaign's target audiences-youth aged 9 to 18 and their parents and adult
    caregivers-through strategic placement of anti-drug advertisements on
    television and radio and in print media.

    The campaign's first two phases were pilot phases that had as their
    objectives developing advertising concepts, creating limited
    advertisements, testing public awareness of the advertisements in 12
    metropolitan areas, and eventually extending the pilot program nationwide.
    Phase III of the campaign, which began in mid-1999, continued the
    nationwide advertising campaign begun during phase II and integrated the
    advertising with outreach efforts. In addition to the advertising, the
    fully integrated phase III campaign included community outreach, work with
    the entertainment and media industries to encourage the accurate depiction
    of the consequences of drug use, outreach to faith-based organizations,
    and work with youth organizations.

    7Pub. L. No. 105-61, 111 Stat. 1272.

    8Drug Free Media Campaign Act of 1998, 21 U.S.C. S: 1801 et. seq.

    During phase III, ONDCP had overall responsibility for developing and
    implementing the campaign, and to do so, it enlisted the support of
    nonprofit organizations, trade associations, private businesses, and
    federal agencies. Appropriated media campaign funds were to be used to
    cover the costs of actually making the advertisements as well as the costs
    for planning of purchase of media time and space. The campaign also
    included public outreach and specialized communications efforts. The
    purpose of public outreach and communications was to extend the reach and
    influence of the campaign through nonadvertising forms of marketing
    communications. Examples of these nonadvertising forms of communication
    included submitting articles related to key campaign messages such as
    effective parenting or the effects of marijuana on teen health to
    newspapers and magazines; building partnerships and alliances, for
    example, coordinating positive activities for teens with local schools and
    community groups; creating Web sites and exploring alternative media
    approaches; and entertainment industry outreach.

    According to the campaign's communications strategy, youth aged 9 to 18
    were segmented into three school and age risk-level categories: late
    elementary school adolescents, aged 9 to 11; middle school children, aged
    11 to 13; high school youth, ages 14 to 18. The campaign originally
    targeted youth aged 9 to 18 with a focus on middle school age adolescents
    (roughly 11-to 13-year-olds); its secondary focus was on high school-aged
    youth (approximately 14 to 18 years of age). In 2001, the campaign shifted
    its creative focus to 11- to 14-year-olds in order to more effectively
    reach youth at the time they are most at risk for trying drugs. In 2002,
    the campaign altered its target age group to focus primarily on 14- to
    16-year-olds. For all age groups, the communications strategy identified
    the primary focus of the campaign as at-risk nonusers and occasional users
    of drugs. For all groups, it was designed to give consideration to
    differences arising from gender, race, ethnicity, and regional and
    population density factors.

    From fiscal year 1998 through fiscal year 2004, Congress appropriated
    $1.225 billion to support the campaign (table 1).

    Table 1: Appropriations for the National Youth Anti-Drug Media Campaign,
    Fiscal Years 1998 through 2006

                    Fiscal year Final appropriations (in millions of dollars)
                           1998                                          $195
                           1999                                          $185
                           2000                                          $185
                           2001                                          $185
                           2002                                          $180
                           2003                                          $150
                           2004                                          $145
                           2005                                          $120
                           2006                                          $100
    Total, 1998 through 2006                                        $1,445
    Total, 1998 through 2005                                        $1,345
    Total, 1998 through 2004                                        $1,225

    Sources: Appropriations Acts from various years.

    Note: Appropriated amounts are prerescission amounts. For example,
    rescissions were 0.38 percent in fiscal year 2000, 0.22 percent in fiscal
    year 2001, 0.65 percent in fiscal year 2003, and 0.59 percent in fiscal
    year 2004.

    For fiscal year 2007, the President's budget requested $120 million for
    campaign activities. The 2007 request represents an increase of $21
    million above the fiscal year 2006 budget authority. The additional
    resources were requested to help to purchase additional media time and
    space to increase the reach and frequency of the campaign's messages.

    Planning and the Underlying Logic of the Campaign

    According to ONDCP, its planning for the campaign's communications
    strategy included reviews of published studies on the etiology and
    prevention of adolescent drug use, drug prevention campaigns, other public
    health campaigns, and general consumer marketing campaigns targeting youth
    and their parents. ONDCP also supplemented its research evidence with an
    extensive expert consultation process that included input from over 200
    experts in academia, civic and community organizations, government
    agencies, and the private sector. A campaign design expert panel that
    included experts in the fields of drug use and prevention, public health
    communication, advertising, market research, consumer marketing, and
    public policy met over a 4-day period during the fall of 1997 and played a
    key role in integrating diverse sources of information and guiding the
    development of the communications strategy for the campaign.

    The planning process resulted in a statement of ONDCP's communications
    strategy for the campaign, which described the premises of the campaign.
    Among these were the following: First, that the media can influence people
    in a variety of ways, such as informing and alerting them to important
    developments and shaping subsequent actions; satisfying leisure time
    needs, thereby influencing individuals' views and beliefs about the world;
    and stimulating interest in commercial goods and services, thereby
    influencing where and how people shop. Second, that media messages have
    more potential to reinforce rather than to alter existing attitudes and
    beliefs. Third, to the extent that youth attitudes, beliefs, and
    intentions toward drug use vary with their age, the potential of a media
    campaign to influence drug use may be directly related to the age of the
    youth. Fourth, the campaign had to be sustained over time and to have a
    significant media presence, and its central messages have to be repeated
    often and in a variety of ways. Citing research showing that attitudinal
    and behavioral change took time to occur, ONDCP reported that it expected
    to observe "improvements in anti-drug attitudes that would lead to
    decreases in youth drug use within three years" of the implementation of
    phase III of the campaign. Fifth, as parents and adult caregivers play a
    vital role in youth drug use behaviors, and by also targeting parents, the
    campaign would aim to affect the nature of their interaction with their
    children, thereby strengthening their children's capacity to resist using
    illicit drugs.

    The campaign focused on primary prevention-that is, preventing those who
    did not use drugs from starting to use drugs. According to ONDCP, a media
    campaign that focused on primary prevention targets the underlying causes
    of drug use and therefore has the greatest potential to reduce the scope
    of the problem over the long term. Further, a primary prevention campaign
    also has greater potential to affirm and reinforce anti-drug attitudes of
    nonusers than to persuade experienced users to change their behaviors, and
    a primary prevention campaign would also, over time, lessen the need for
    drug treatment services. With a focus on young, non-drug-using
    adolescents, an expectation underlying the campaign's potential success
    was that as these young, non-drug-using adolescents aged, the campaign's
    messages would intervene, retard the development of more pro-drug
    attitudes, and enable adolescents to continue to maintain their
    preexisting anti-drug attitudes. By maintaining these attitudes, or
    preventing the development of pro-drug sentiments, the campaign would
    affect drug use rates by lowering the rate at which youth initiated drug
    use, particularly the use of marijuana or inhalants.

    The campaign was designed to have a significant and sustained media
    presence. During planning, ONDCP acknowledged that the campaign would have
    to be sustained for a period of time sufficient to bring about a
    measurable change in the beliefs and behaviors of youth in the target
    audience. On the basis of the experiences of successful social marketing
    campaigns, ONDCP reported that it expected that changes in awareness or
    recall of the campaign would be detectable within a few months of the
    start of the campaign, that changes in perceptions and attitudes would be
    detectable within 1 to 2 years of the start of the campaign, and that
    changes in behavior would be detectable within 2 to 3 years.

    Campaign Activities during Phase III

    From mid-1999, the start of phase III, through June 2004, the end of the
    phase III evaluation, campaign activities included extensive media
    dissemination of campaign messages to a national audience of youth and
    parents; an interactive media component, which involved using
    content-based Web sites and Internet advertising; use of experienced
    individuals and organizations with expertise in marketing to teens,
    advertising and communications, behavior change, and drug prevention to
    inform the campaign strategy and implementation; use of multicultural
    initiatives that focused on sufficiently exposing campaign messages to
    African Americans, Asian Americans, Pacific Islanders, Hispanic Americans,
    American Indians, and Alaskan Natives; and the implementation of the
    integrated social marketing and public health communications campaign
    through the creation of partnerships with civic, professional, and
    community groups and outreach to media, entertainment, and sports
    industries. Through the partner organizations, the campaign attempted to
    strengthen local anti-drug efforts, and through outreach, it encouraged
    the news media to run articles that conveyed campaign messages. Youth and
    parent exposure to campaign messages could come from the direct, paid and
    donated advertising or from content delivered by news media and
    entertainment industries through the outreach efforts. Additional
    opportunities for exposure to anti-drug messages could be enhanced through
    personal involvement with organizations that became partners as a result
    of campaign outreach or by interaction with the campaign's Web site.
    Further, youth exposure to anti-drug messages could also occur through
    interactions with friends, peers, parents, or other adults that occurred
    directly from either campaign ads or outreach efforts.

    Campaign Themes and Messages

    Campaign messages for both youth and their parents and caregivers were to
    focus on common transitions-such as the transition from elementary to
    secondary school-and common situations-such as the amount of time spent in
    settings without adult supervision-that were believed to heighten
    adolescents' vulnerability to drug use initiation. In addition, messages
    were to focus on altering mediating variables-such as beliefs and
    intentions-that were known to have a significant impact on adolescent drug
    use. Finally, campaign messages were designed to create a "brand identity"
    in the minds of target audience members and through brand identity
    position campaign messages as credible and important. Throughout phase
    III, themes such as parents as "The Anti-Drug" and the "My Anti-Drug"
    theme for youth were designed to promote identification and positive
    associations with the campaign's messages.

    While they evolved throughout the campaign, the central strategic messages
    or themes for youth focused on resistance skills and self-efficacy to
    refuse drugs, normative education and positive messages, negative
    consequences of drug use, and early intervention. Resistance skills and
    self-efficacy advertisements were designed to enhance the personal and
    social skills of youth that promote lifestyle choices and to help build
    youth's confidence that they could resist drugs. Normative education
    themes attempted to instill the beliefs that most young people do not use
    drugs or convey messages that "cool people don't use drugs," while
    positive message themes reinforced the idea of positive uses of time as
    alternatives to illicit drugs. Negative consequences themes aimed to
    enhance youth perceptions that drug use is likely to lead to a variety of
    negatively valued consequences, such as loss of parental approval, reduced
    performance in school, and negative social, aspirational, and health
    effects. Negative consequences themes were the primary focus of the
    Marijuana Initiative, which was introduced during 2002. An early
    intervention theme sought to motivate youth to intervene with friends who
    they perceived as having problems with drugs or alcohol and tried to
    convince youth of their ability to take action and to give them the tools
    and skills they needed to intervene.

    For parents, the campaign's themes included messages that every child,
    including their own, was at risk of doing drugs; that they can learn
    parenting skills to help them help their children avoid drugs; that they
    need to be aware of the harmful effects of drugs including marijuana and
    inhalants; and, as part of the Early Intervention Initiative, that it was
    important that they intervene at the earliest possible opportunity in
    their child's life if their child was using drugs or alcohol.

    Design of the Evaluation, Interim Evaluation Reports, and Redirection of the
    Campaign

    ONDCP recognized the need for a separate evaluation of the campaign and
    for ongoing reporting of evaluation results. The need for a separate
    evaluation stemmed in part from the limitations of existing national
    surveys that monitor drug use, such as Monitoring the Future, which
    provides data on drug use by high school students, the National Household
    Survey on Drug Abuse,9 and the Youth Risk Behavior Survey, which addresses
    health risk behaviors including drug use. These recurring surveys provide
    very little information with which to evaluate the impact of the campaign,
    because they were not designed to evaluate it. As ONDCP has written, these
    surveys contain no questions about target audience exposure and response
    to the campaign, and as a result, any changes in attitudes, beliefs, and
    behaviors toward drug use could not be associated directly with the
    campaign. By comparison, ONDCP acknowledged that it was using the Westat
    evaluation to assess the extent to which changes in anti-drug attitudes
    and beliefs or drug-using behavior can be attributed to the campaign, as
    opposed to other socioeconomic factors. In addition, ONDCP indicated that
    for the campaign, data from Westat's evaluation would enable ONDCP to
    assess whether the campaign is working.

    The primary tool of the Westat evaluation was the National Survey of
    Parents and Youth. The NSPY is a longitudinal panel study of children and
    their parents' exposure and response to the campaign. The NSPY was
    designed to collect initial and follow-up data from nationally
    representative samples of youth aged 9 to 18 and from the parents of these
    youth. The sample was designed to represent youth living in homes in the
    United States and their parents. Data collection began in November 1999
    and was conducted over four rounds-each of which was about 1 year apart
    from the next round-in nine waves of interviews. An interview wave refers
    to the fielding of a survey round to a specific subsample in the NSPY. An
    interview round refers to the completion of interviews with the entire
    sample. Data for each of the nine waves were collected using a laptop
    computer and a combination of computer-assisted interview technologies. To
    collect sensitive data, audio computer-assisted self-interview technology
    was used, allowing respondents to self-administer the questionnaire in
    total privacy. The final wave of data collection was completed in June
    2004 (fig. 1). Eligible youth and parents were to be interviewed four
    times.

    9The survey is now known as the National Survey on Drug Use and Health.

    Figure 1: Data Collection Rounds and Waves of the NSPY

                                               Cohorts
                                             (groups) of
                                              youth and
                                               parents
    Phase of                  Data        interviewed
    data        Survey Survey collection  in specific
    collection  round  wave   period      survey waves
    Recruitment Round  Wave 1 11/99-5/00    Group A,
    phase       1                         recruitment
                                              interview
                          Wave 2 17/00-12/00               Group B,
                                                          recruitment
                                                           interview
                          Wave 3 11/01-6/01                            Group C,
                                                                      recruitment
                                                                       interview
    Follow-up   Round  Wave 4 17/01-12/01   Group A,
    phase       2                            first
                                              follow-up
                                              interview
                          Wave 5 11/02-6/02                Group B,    Group C,
                                                             first       first
                                                           follow-up   follow-up
                                                           interview   interview
                   Round  Wave 6 17/02-12/02   Group A,
                   3                            second
                                              follow-up
                                              interview
                          Wave 7 11/03-6/03                Group B,    Group C,
                                                            second      second
                                                           follow-up   follow-up
                                                           interview   interview
                   Round  Wave 8 17/03 -       Group A,
                   4             12/03          third
                                              follow-up
                                              interview
                          Wave 9 11/04-6/04                Group B,    Group C,
                                                             third       third
                                                           follow-up   follow-up
                                                           interview   interview

    Source: Adopted from Westat, 2005, Vol. 2: Appendices.

    The evaluation aimed to assess whether exposure to the campaign affected
    the self-reported knowledge, attitudes, beliefs, and drug use of youth.
    Because the campaign reached out to all youth nationwide, the evaluators
    could not assess its effects using experimental methods, in which some
    subjects are randomly assigned to the intervention and others are randomly
    assigned to control groups that were not exposed to the intervention.
    Westat's evaluation was designed to take into account the variation in
    self-reported exposure to the campaign messages and to assess how this
    variation in exposure was correlated with outcomes that the campaign
    intended to affect. To attribute changes in drug use attitudes and
    behaviors to the campaign, the evaluation was designed to assess exposure
    to the campaign and to compare differences in outcomes for groups of
    persons that were exposed to varying levels of the campaign's messages,
    and to use statistical controls to account for individual-level
    differences among survey respondents.

    Westat's evaluation assessed youth self-reported drug use and intermediate
    outcomes-such as youth and parent attitudes and beliefs toward drug use
    and parental involvement with their children-that were believed to
    influence youth drug use. The evaluation of phase III addressed issues
    related to (1) whether the campaign was reaching its target populations,
    (2) whether the desired outcomes moved in favorable or unfavorable
    directions, (3) whether the campaign was influencing changes in the
    desired outcomes, and (4) what could be learned from the overall
    evaluation to support ongoing decision making for the campaign. These
    issues led to the five major objectives for the evaluation:

               o  to measure changes in drug-related knowledge, attitudes,
               beliefs, and behavior in youth and their parents;
               o  to assess the relationship between changes in drug-related
               knowledge, attitudes, beliefs, and behavior and self-reported
               measures of media exposure, including the salience of the
               measures;
               o  to assess the association between parents' drug-related
               knowledge, attitudes, beliefs, and behavior and those of their
               children;
               o  to assess changes in the association between parents'
               drug-related knowledge, attitudes, beliefs, and behavior and those
               of their children that may be related to the campaign; and
               o  to compare groups of people with high exposure to other groups
               with low exposure.

    Westat submitted semiannual and special topic reports to NIDA, as the
    findings from these interim evaluation reports were to be used to support
    ongoing decision making for the campaign. Westat submitted the first
    semiannual report in November 2000. By December 2003, Westat had submitted
    six additional reports, four of which were labeled as semiannual reports,
    and the other two included a special report on historical trends in drug
    use and a 2003 report of findings.10 Westat submitted its first draft of
    its final report to NIDA in February 2005.

    In addition to Westat's evaluation of the relationship between exposure
    and outcomes, Westat also prepared a report on the environmental context
    of the campaign.11 In May 2002, Westat reported findings from this
    qualitative study of views of representatives from major national
    organizations and state prevention coordinators about the messages
    conveyed by the campaign and the role of the campaign as an organizing
    partner in helping to bolster local substance abuse prevention efforts.
    According to Westat, representatives felt that the campaign's messages
    reinforced their own messages that encouraged youth to find healthy
    alternatives to drug use and to raise public awareness of the issue of
    illicit drugs among youth. Westat also reported that representatives were
    less enthusiastic about the role of the campaign as an organizational
    partner in helping with local substance abuse prevention efforts.

    10All of these reports were submitted jointly by Westat and Annenberg.

    In November 2002, Westat submitted its fifth semiannual report to NIDA. In
    it, Westat reported that it found little evidence that the campaign had
    direct, favorable effects on youth self-reported drug use between 2000 and
    2002. Specifically, Westat reported:

    "There is little evidence of direct favorable Campaign effects on youth.
    There is no statistically significant decline in marijuana use to date,
    and some evidence for an increase in use from 2000 to 2001. Nor are there
    improvements in beliefs and attitudes about marijuana use between 2000 and
    the first half of 2002. Contrarily, there are some unfavorable trends in
    youth anti-marijuana beliefs. Also there is no tendency for those
    reporting more exposure to Campaign messages to hold more desirable
    beliefs."12

    Westat further reported that there were unfavorable delayed effects of
    campaign exposure on subsequent intentions to use marijuana and on other
    beliefs. By delayed effects, Westat referred to the relationship between
    exposure to the campaign measured in one survey round having an effect on
    intentions or beliefs outcomes at a subsequent survey round. For parents,
    Westat reported that the evidence was consistent with favorable campaign
    effects, as it found that there were favorable changes for three of five
    parents' belief and behavior outcome measures. However, Westat also
    reported that it found no evidence for favorable indirect effects on youth
    behavior as the result of their parents' exposure to the campaign.

    11Westat, "Environmental Context of the National Youth Anti-Drug Media
    Campaign: Findings from In-Depth Discussions with Representatives of
    National Organizations and State Prevention Coordinators." Report
    delivered to National Institute on Drug Abuse, National Institutes of
    Health, Rockville, Maryland, May 2002.

    12Hornik, Robert, et al. Evaluation of the National Youth Anti-Drug Media
    Campaign: Fifth Semi-Annual Report of Findings, (Rockville, Maryland:
    Westat, November 2002), p. xi.

    Congressional appropriators expressed concerns about the findings of
    Westat's fifth semiannual report. In the conference report for fiscal year
    2003 omnibus appropriations, the conferees reported that they were "deeply
    disturbed by the lack of evidence that the National Youth Anti-Drug Media
    Campaign has had any appreciable impact on youth drug use."13 The
    conferees further acknowledged that while the evaluation conducted under
    NIDA's auspices showed "slight and sporadic impact on the attitudes of
    parents, it has had no significant impact on youth behavior." The
    conferees further acknowledged that while other surveys of youth drug
    use-such as Monitoring the Future, a survey of high school youth-showed
    recent declines in drug use, "the NIDA study was undertaken to measure the
    specific impact of the Media Campaign, not simply to gauge general
    trends," and the conferees stated that they "intend to rely on the
    scientifically rigorous NIDA study to gauge the ultimate impact of the
    campaign" and to reevaluate the use of taxpayer money to support the
    campaign if the campaign continued to fail to demonstrate its
    effectiveness.

    In 2002, the strategy for the campaign was redirected. In the spring, the
    target age group of the campaign became 14- to 16-year-olds-youth who have
    higher rates of marijuana initiation than younger youth-from its original
    targeting of 11- to 13-year-olds. The shift to teens in the 14- to
    16-year-old range aimed to allow the campaign to more effectively reach
    youth during the time at which they are most at risk for trying drugs.
    ONDCP also required more rigorous copy test procedures of all television
    advertisements before they were aired, and ONDCP increased its oversight
    in guiding the development and production of advertisements. In October
    2002, ONDCP launched a new initiative called the Marijuana Initiative.
    This initiative contained more focused advertising to address youth
    marijuana use. In a hearing before the House Committee on Government
    Reform, ONDCP announced that it would reverse the ratio of campaign
    advertising expenditures directed to adults and youth, respectively.
    Previously, about 60 percent of expenditures were directed to adults and
    40 percent toward youth. Finally, during February 2004, it expanded the
    campaign's communications goals to include the Early Intervention
    Initiative. This intervention was targeted toward both parents and teen
    friends, and ONDCP intended to use parental and peer pressure to stop drug
    and alcohol use among teens.

    13Conference Report No. 108-10, at 1345 (2003).

    Assessment of the Campaign by the Office of Management and Budget and ONDCP's
    Current Approach

    To strengthen the linkages between resources and performance envisioned in
    the Government Performance and Results Act of 1993 (GPRA),14 the Office of
    Management and Budget (OMB) developed the Program Assessment Rating Tool
    (PART) to bring performance information into the executive budget
    formulation process. PART is designed to determine the strengths and
    weaknesses of federal programs by drawing upon available program
    performance and evaluation data so that the federal government can achieve
    better results. The PART therefore looks at factors that affect and
    reflect program performance, including program purpose and design;
    performance measurement, evaluations, and strategic planning; program
    management; and program results. Because the PART includes a consistent
    series of analytical questions, it allows programs to show improvements
    over time and allows comparisons between similar programs.

    OMB's PART rating of the campaign addressed issues related to its purpose
    and design, strategic planning, program management, and program results
    and accountability. OMB indicated that the purpose was clear-giving ONDCP
    a 100 percent score on this factor-and it rated the campaign's planning
    and management with scores of 67 percent and 70 percent, respectively. In
    its assessment of ONDCP's strategic planning, OMB noted that in response
    to its 2002 PART review, ONDCP revised the campaign's logic model and
    significantly changed its long-term and annual performance measures.

    However, OMB's assessment rating for the campaign was "results not
    demonstrated." OMB indicated that its assessment of the campaign's
    progress toward both the long-term goals and annual performance goals will
    be reviewed against the results of the NIDA-managed evaluation. OMB noted
    that while there is no federal program closely comparable to the campaign,
    evaluations of other health behavior change efforts found short-term
    effects after exposure to media. While acknowledging that a final
    assessment of the effects of the campaign awaited the final report from
    the NIDA-managed evaluation, OMB also indicated that "outcome data from
    the evaluation suggest little or no direct positive effect on youth
    behavior and attitudes attributable to the campaign to date. Perhaps some
    positive effect on parental attitudes/behavior but that has not yet
    translated into an effect on youth."

    14Pub. L. No. 103-62, 107 Stat. 285 (1993).

    ONDCP has credited the campaign, along with a variety of collective
    prevention efforts, with contributing to "significant success in reducing
    teen drug use, as evidenced by the 19 percent decline from 2001 to 2005."
    It has introduced a new youth brand approach to connect youth with
    aspiration themes. ONDCP also has indicated that while it awaits our
    formal assessment of the evaluation, that it will use existing national
    surveys to evaluate the campaign and suspend its request for proposals for
    a new evaluation contract. Specifically, ONDCP indicated that it would use
    the MTF survey to track improvements in perception of the risk of drug
    use-a predictor of lower drug use by youth-and it would use a special
    analysis of the PATS survey-the Partnership for a Drug-Free America's
    Attitude Tracking Survey-data on anti-drug messages. According to the 2005
    data from MTF, there were no significant 1-year declines in marijuana use
    for youth in any grade levels, and while gradual declines in the upper
    grades continued, declines halted for youth in the 8th grade.
    Additionally, for 8th graders, perceived risk of marijuana use held
    steady, while for youth in 10th and 12th grade, there was an increase in
    perceived risk of marijuana use.

    Recent Research on the Effects of the Campaign in Local Settings

    Two recently released studies have reported that exposure to the campaign
    was associated with changes in past-month marijuana use under certain
    conditions for certain groups of students exposed to the campaign. In one
    of the studies,15 45 South Dakota high schools and their middle-school
    feeder(s) were randomly assigned to three groups: (1) a basic prevention
    curriculum, (2) a group given this curriculum with booster lessons, and
    (3) a control group. All schools were exposed to the campaign during the
    fall of 1999 and spring of 2000. This permitted the researchers to test
    for a synergistic effect between exposure to the campaign's anti-drug
    messages and participation in the school-based drug prevention curriculum.
    The sample of about 4,100 youth were asked how often they had seen
    anti-drug advertisements in recent months in five media outlets that were
    used by the campaign, and the researchers measured exposure to the
    campaign that indicated whether or not the adolescents reported seeing ads
    at least one to three times per week in any of the five media outlets.
    Consistent with Westat's fifth interim report, the evaluation of the South
    Dakota drug prevention curriculum found no direct effects of exposure to
    the campaign on its sample of adolescents' use of illegal drugs. However,
    the evaluation also found that marijuana use in the past month was
    significantly less likely among adolescents who received both the
    curriculum with booster lessons and weekly exposure to the campaign's
    messages. In other words, neither the enhanced curriculum nor the campaign
    alone had a substantial effect on marijuana use in the absence of the
    other. In addition, this evaluation's measure of exposure was based on
    weekly exposure, suggesting that the synergistic effect of the campaign
    observed in these South Dakota schools was based on the delivery of
    repeated messages.

    15Longshore, Douglas, et al., "National Youth Anti-Drug Media Campaign and
    School-Based Drug Prevention: Evidence for a Synergistic Effect in ALERT
    Plus," Addictive Behaviors, Vol. 31, (2006) pp. 496-508.

    The second study used monthly random samples of 100 youth from the
    enrollment lists of 4th to 8th graders in the public schools in the spring
    of 1999 in two moderate-sized communities-Fayette County (Lexington),
    Kentucky, and Knox County (Knoxville), Tennessee-over 48 months from April
    1, 1999, through March 31, 2003.16 The study period included
    advertisements aired under the campaign's Marijuana Initiative. Students
    in the samples aged over time and were 13 to 17 years of age at the
    beginning of the Marijuana Initiative. Youth in the samples were measured
    on marijuana use during the past 30 days, as well as on their attitudes
    toward marijuana. Exposure to television and radio advertisements was
    measured by self-reported past-month exposure. The study found that among
    high-sensation-seeking youth-that is, youth who desire novel, complex, and
    intense sensations and experiences and who are willing to take social
    risks to obtain them-exposure to the first 6 months of the campaign's
    Marijuana Initiative led to reductions in marijuana use. The study's
    authors reasoned that the effects that they found for the Marijuana
    Initiative were consistent with an approach termed SENTAR (for
    sensation-seeking targeting), in which high-sensation-seeking youth are
    targeted with high sensation value messages to prevent risky behaviors.

    16Palmgreen, Philip, et al., "Effects of ONDCP's Marijuana Initiative
    Campaign on High Sensation-Seeking Youth." Paper presented to the American
    Public Health Association, Philadelphia, Pennsylvania, December 2005.

     Westat's Evaluation Design, Use of Generally Accepted and Appropriate Sampling
     and Analytic Techniques, and Reliable Methods for Measuring Campaign Exposure
                   Produced Credible Evidence to Support Its Findings

    Westat was able to produce credible evidence to support its findings about
    the relationship between exposure to campaign advertisements and both drug
    use and intermediate outcomes by employing a longitudinal panel
    design-i.e., collecting multiple observations on the same persons over
    time-using generally accepted and appropriate sampling and analytic
    techniques and establishing reliable and sufficiently powerful measures of
    campaign exposure. Westat encountered various challenges and threats to
    validity that are commonly associated with large-scale longitudinal
    studies, including lack of an opportunity to use experimental methods,
    lack of baseline data, and changes in campaign focus that were not timed
    with data collection; issues with ensuring adequate sample coverage and
    controlling for sample attrition over time; establishing measures that
    were sufficient to detect and reliably measure campaign effects; and
    disentangling causal effects without being able to employ an experimental
    design where subjects would have been randomly assigned to different
    levels of exposure. Our review of Westat's evaluation report and
    associated documentation leads us to conclude that the design and
    methodology used in its evaluation responded appropriately to these
    challenges, resulting in credible findings.

    Although Elements of the Campaign Limited Choices of Evaluation Designs and
    Affected Data Collection, Westat's Design Was Rigorous and Provided a Means to
    Test for Campaign Effects

    The nationwide scope of the campaign precluded Westat from using
    experimental methods or obtaining baseline data, and the timing of the
    introduction of some new campaign initiatives limited some of the data
    available to evaluate them. However, Westat's longitudinal panel survey
    design provided a framework for developing strong evidence of
    within-respondent changes in outcomes over time as a result of exposure to
    the campaign. The consensus of a scientific panel convened by NIDA in
    August 2002 to review the evaluation was that Westat's use of a national
    probability sample to study change arising from the campaign was
    preferable as the "gold standard" to a study based on other alternatives,
    such as in-depth community-based studies of the mechanisms of change and
    campaign effects. Additionally, the theoretical underpinnings of
    behavioral change through advertising, along with statistically
    significant outcomes in some but not all groups, suggest that the absence
    of baseline data and introduction of new campaign initiatives did not
    invalidate the evaluation's findings. Finally, despite the introduction of
    new campaign initiatives that were not timed with data collection cycles,
    Westat was able to assess change in the NSPY data and generate
    statistically significant findings using these data.

    Westat's longitudinal panel design was based on the premise that effects
    of exposure to the campaign on outcomes could be measured and detected
    within individuals over time, after controlling for various other factors
    that could have influenced outcomes. The design called for measuring the
    same respondents up to four times to assess how the natural variation in
    exposure to the campaign correlated with campaign outcomes. Westat's
    approach-an exposure (or dose)-response model-is based upon a premise that
    respondents' recall of advertisements (exposure or dose) is related to
    outcomes (response). In two recent studies of the effects of the campaign
    on specific groups of youth in local areas, an exposure-response approach
    has been shown to be an effective method for detecting effects of the
    campaign in reducing youth drug use in local settings. One of the studies
    reported a synergistic effect of exposure to the campaign and a
    classroom-based drug prevention curriculum among 9th grade students in 45
    South Dakota high schools. The other study reported reductions in drug use
    during the period of the redirected campaign among high-sensation-seeking
    youth in schools in Knoxville, Tennessee, and Lexington, Kentucky. To
    assess the possibility of preexisting differences between groups of
    exposed youth and parents that might explain both the variation in
    exposure to the campaign and variation in outcomes, Westat included in the
    NSPY structured interview many questions on personal and family history,
    and it used the responses to control statistically for differences in
    attributes of respondents in order to attempt to isolate the relationship
    between exposure to the campaign and outcomes.

    The absence of baseline data-that is, precampaign data on outcomes-was
    beyond Westat's control, as phase III of the campaign began before the
    first wave of data collection for the phase III evaluation began. The lag
    between the start of phase III of the campaign in mid-1999 and the
    completion of the evaluation's first round of data collection-around
    mid-2001-leaves open the possibility that there were effects of the
    campaign that occurred very early on in the campaign, prior to when Westat
    began data collection. Several factors suggest that the absence of
    pre-phase III baseline data was not fatal to the evaluation's findings.
    First, if there were effects of the campaign that could not be detected
    because of the absence pre-phase III baseline data, those effects must
    have occurred very rapidly and then endured throughout the remainder of
    the campaign, from 1999 through 2004. However, rapid changes in youth drug
    use were not observed in MTF data; rather, the overall trend in MTF past
    year drug use was flat between 1998 and 1999. Second, rapidly occurring
    effects were not expected by ONDCP in designing the campaign. As we
    reported in 2000, and as ONDCP wrote in 2001, ONDCP believed that it would
    take 2 to 3 years for changes in drug use to be evident as a result of the
    campaign.17

    Another campaign design factor that affected Westat's evaluation was the
    implementation of new campaign initiatives, such as the Marijuana
    Initiative, which were implemented at times that officials at ONDCP
    considered to be important, and therefore they may not have coincided with
    planned data collection for the evaluation, nor should they necessarily
    have done so. For example, the Marijuana Initiative was implemented in
    October 2002, and the NSPY data available to evaluate outcomes during it
    were limited to three complete survey waves. For its longitudinal analysis
    of change during the Marijuana Initiative, Westat was limited to data from
    two survey waves. Despite these limitations, the evaluation produced data
    that enabled Westat to detect effects during the period of the Marijuana
    Initiative.

    Sample Coverage Issues Did Not Invalidate Westat's Assessment of the
    Effectiveness of Exposure to the Campaign on Intermediate and Drug Use Outcomes

    During the enrollment phase of the NSPY, Westat experienced sample
    coverage problems, in that it enrolled-or rostered-a smaller percentage of
    households with youth in the targeted age range than would be expected
    based on comparable Current Population Survey (CPS) estimates-the data
    that Westat used to develop its expectations about the percentage of
    households having youth in the targeted age ranges. Coverage refers to the
    extent to which a sample is representative of the intended population on
    specified characteristics, and it is important because the omission of
    segments of the intended population from a sample-or undercoverage-can
    lead to biased results, in that omitted segments may differ in some
    important respect from those segments included. Westat estimated the
    extent of undercoverage in the NSPY to be about 30 percent as compared to
    the CPS estimates, and according to Westat and NIDA, the undercoverage
    arose during the stage of sampling in which Westat was developing rosters
    of households that were believed to contain youth in the target age range.
    At this stage, the survey rostering process required entry into the
    household, which may have led respondents in potentially eligible
    households to refuse to participate.

    Our review of Westat's documentation leads us to conclude that there was
    no evidence of biased results due to undercoverage and that the sample was
    sufficiently reliable both for the purposes of estimating changes over
    time in individual outcomes and for assessing the effectiveness of
    exposure to the campaign on outcomes. Westat's comparisons of the
    estimated population characteristics of the NSPY-such as race and
    ethnicity of head of household and race and ethnicity of youth in
    households-with the estimated population characteristics from the CPS show
    that they are generally similar. That is, the distributions of
    characteristics of eligible households with youth included in the NSPY
    were broadly consistent with a variety of corresponding distributions from
    the 1999 CPS. These comparisons suggest that the NSPY estimated population
    by race and ethnicity was similar to that of CPS. Westat also used
    multivariate modeling techniques to develop weighting adjustments, and it
    developed weights to adjust its sample for nonresponse that were
    reasonably effective in reducing nonresponse bias.

    17 GAO/GGD/HEHS-00-153 , (Washington, D.C.: July 31, 2000), p. 68.

    An additional test for bias in a sample is to compare estimates derived
    from it with estimates on the same variable derived from another sample.
    If the NSPY results were biased, then one would expect that estimates
    derived from it would differ from estimates derived from unbiased samples.
    For example, if eligible households refused to participate in the NSPY
    because they contained teens with drug issues and as a result avoided
    participation at a higher rate than did households containing teens
    without drug issues, then these higher refusal rates by households
    containing teens with drug issues would lead to NSPY estimates of the
    percentage of youth reporting that they used drugs that were lower than
    those obtained from other, comparable national surveys. According to data
    provided by NIDA officials and our review of Westat's final report,
    estimated self-reported drug use rates from the NSPY are comparable to
    estimates derived from other major surveys of drug use, such as the
    National Survey on Drug Use and Health. For example, in the NSPY, rates of
    past-month marijuana use among 12- to 18-year-olds were 7.2 percent in
    2000, 8 percent in 2001, 8.9 percent in 2002, and 7.9 percent in 2003.
    These rates were similar to those reported for 12- to 17-year-olds in the
    National Survey on Drug Use and Health (NSDUH) of 7.2 percent in 2000, 8
    percent in 2001, 8.2 percent in 2002, and 7.9 percent in 2003. If youth
    with known drug use problems consistently opted out of both the NSPY and
    the NSDUH-a hypothesis that is not testable with the available data-then
    the estimates from both the NSPY and the NSDUH of the true prevalence of
    youth drug use would be biased underestimates.

    Sample Attrition across NSPY Interview Rounds Was Sufficiently Low to Allow for
    Reliable Assessments of the Effect of Campaign Exposure on Outcomes

    As the NSPY was a longitudinal survey-in which eligible sample respondents
    were re-interviewed up to three times after their enrollment
    interviews-attrition was a concern with which Westat had to contend. If
    comparatively large numbers of sample respondents were not retained across
    successive rounds of the survey, the capacity of the NSPY to provide data
    to assess changes in outcomes in response to exposure over time would be
    greatly diminished. Further, if attrition was specific to certain groups,
    then the NSPY estimates would also be biased.

    For the purpose of estimating within-respondent changes in outcomes in
    response to changes in exposure across sample periods-the main use of the
    NSPY data-Westat achieved follow-up longitudinal response rates of between
    82 percent and 94 percent for waves 4 through 9, the follow-up waves to
    the first three enrollment waves. The longitudinal response rate consists
    of two elements: (1) the percentage of prior survey respondents that are
    tracked and for whom eligibility is determined and (2) the percentage of
    those eligible that actually complete an interview. Across the three
    follow-up survey rounds, Westat tracked and determined the eligibility to
    participate in a follow-up survey of between 92 percent and 96 percent of
    the youth and parents who completed a survey in the prior round. Of these,
    Westat obtained consent and completed extended interviews with between 94
    percent and 96 percent of youth and parents for whom eligibility for a
    follow-up survey had been determined.

    In our view, Westat's follow-up response rates resulted in a sample that
    was sufficient to provide reliable findings about the effects of exposure
    on outcomes. In addition, Westat's nonresponse adjustment methodology
    compensated for effects of differential response rates related to the
    percentage of persons in certain age groups, of certain races and
    ethnicities, of those that owned homes versus rented, those that were U.S.
    citizens versus noncitizens, and those with incomes below the poverty
    level.

    The NSPY Data Could Be Used to Detect Reasonably Small Effects, and Westat's
    Measurement of Exposure and Outcomes Were Valid and Could Detect Effects, if
    They Occurred

    The NSPY sample could be used to detect changes in outcomes that were on
    the order of magnitude of changes expected by ONDCP for the campaign, and
    its measures of exposure were valid and reliably predicted outcomes. In
    early meetings on the design of the evaluation of the media campaign,
    ONDCP officials reported that it had a specific Performance Measures of
    Effectiveness system and that the campaign was embodied within the first
    goal of the National Drug Strategy, which was to "educate and enable
    America's youth to reject illegal drugs as well as the use of alcohol and
    tobacco." Under this goal, ONDCP's proposed targets for reducing the
    prevalence of past-month use of illicit drugs and alcohol among youth from
    a 1996 base year-by 2002, reduce this prevalence by 20 percent, and by
    2007, reduce it by 50 percent. ONDCP officials identified other specific
    targets, again from the base year 1996-by 2002, increase to 80 percent the
    proportion of youth who perceive that regular use of illicit drugs,
    alcohol, and tobacco is harmful; and by 2002, increase to 95 percent the
    proportion of youth who disapprove of illicit drug, alcohol, and tobacco
    use. To achieve a goal of 80 percent of 12th grade youth who perceive that
    regular use of marijuana is harmful would require increasing the 1996
    baseline percentage of youth perceiving marijuana as harmful from 60
    percent, as measured by MTF, or by about 3.3 percentage points per year
    from 1996 to 2002. Westat's sample could be used to detect this amount of
    annual change in youth attitudes.

    In order to detect changes in outcomes due to exposure to the campaign, it
    also was necessary that Westat accurately measure and characterize
    exposure to the campaign. Westat provided evidence for the validity of its
    measures of self-reported exposure, and the evidence suggests that the
    measure of exposure was both valid and reliable. To measure exposure to
    the campaign for both youth and parents, NSPY interviewers asked
    respondents about their recall of anti-drug advertisements (general
    exposure) and their recognition of specific current or very recent
    television and radio advertisements (specific exposure).18 To facilitate
    measures of recall, respondents viewed television and radio advertisements
    on laptop computers. Youth and parents were only shown or listened only to
    advertisements targeted to their respective groups. In addition, both
    youth and parents were asked some general questions about their recall of
    advertisements seen or heard in various media, including television,
    radio, newspapers, magazines, movie theaters, billboards, and the
    Internet.

    Westat's assessments of the validity of its measure of exposure to
    campaign advertisements confirm that the NSPY data were able to measure
    exposure. First, Westat examined respondents' recall of campaign
    advertisements using "ringer" television advertisements-advertisements
    that never had appeared. According to Westat's analysis of ringer
    advertisements, youth were more likely to recognize an advertisement as a
    campaign advertisement when presented with an actual campaign
    advertisement than a bogus one. For example, a far lower percentage of
    respondents (11 percent) claimed to have seen a ringer, or bogus,
    advertisement than the percentage who claimed to have seen the broadcast
    advertisements (45 percent), particularly the advertisements that were
    delivered with high frequency. The result held for youth and for parents.

    18Each respondent was presented ads that had been broadcast nationally in
    the 2 calendar months prior to the interview.

    Second, comparing data on advertisement time purchases with self-reported
    exposure to these advertisements in the NSPY, Westat found a high
    correlation between advertising and exposure. Specifically, on the basis
    of analysis of individual advertisements' gross rating points (GRP)- a
    measure of the underlying reach and frequency of each advertisement-and
    self-reported exposures by respondents, Westat found a high correlation
    between GRPs purchased by the campaign and self-reported exposure to
    advertisements among youth. The correlation for parents was somewhat
    smaller, but was also significant. Third, Westat also compared
    self-reported exposure with recall of the correct brand phrase and found a
    strong association between self-reported exposure and correct recognition
    of the brand phrase. This is further evidence for the validity of its
    measures of self-reported exposure.

    Westat measured a variety of outcomes for youth and parents and took steps
    to ensure that the measures were consistent with existing research. The
    youth questionnaires included numerous questions that were designed to
    measure exposure to the campaign advertisements and other anti-drug
    messages. The youth question areas included exposure propensity to media;
    current and past use of tobacco, alcohol, marijuana, inhalants, and
    Ecstasy; past discussions with and communication of anti-drug messages
    from parents and friends; expectations of others about respondent's drug
    use; knowledge and beliefs about the positive and negative consequences of
    drug use; exposure to campaign messages; family and peer factors; personal
    factors; and demographic information. Westat used separate questionnaires
    for youth of different ages; one questionnaire was used for children (aged
    9 to 11) and another one was used for teens (aged 12 to 18).

    Westat's Analytic Methods Aimed to Isolate Causal Effects of the Campaign and
    Did So Using Sophisticated Techniques That Enhanced the Strength of Its Findings

    In it analysis, Westat used three types of evidence to draw inferences
    about the effects of the campaign: (1) trend data-data that describe
    increases or decreases in drug use and other outcomes over time; (2)
    cross-section analysis-measures of association between exposure to
    campaign messages and individual drug use beliefs, intentions, and
    behaviors, at the time data were collected; and (3) longitudinal
    analysis-measures of association, for youth and parents who were observed
    at two points in time, between exposure to campaign messages at the
    earlier time on outcomes at the later time.19 Westat indicated that trends
    over time, by themselves, could not be used to provide definitive support
    for campaign effects. Rather, the trends needed to be supported by
    measures of association. Westat also indicated that measures of
    association, whether cross-sectional or longitudinal, needed to control
    for variables that could influence outcomes independently of the campaign
    or otherwise confound the association between exposure and outcomes.
    Cross-section association between exposure and outcomes measured at the
    same time would provide stronger evidence of campaign effects than would
    trend data alone, provided that controls for other variables were
    introduced into the associational analyses. However, even if cross-section
    associations between exposure and outcomes hold after controlling for the
    effects of other variables, as Westat pointed out, there may remain an
    alternative explanation for cross-section associations: For example, an
    outcome-like perceptions of others' use of drugs-may be the cause of
    exposure rather than an effect of it. Westat's longitudinal analysis
    attempts to address the ambiguities that exist with cross-sectional
    associations. With longitudinal data, if, after controlling for other
    confounding variables, exposure measured at an earlier time is associated
    with an outcome at a later time, the inference made is that the causal
    direction is from exposure to outcome, since an effect cannot precede a
    cause in time.

    As the campaign was implemented nationally and it was therefore not
    possible to assign youth and their parents randomly to treatment and
    control groups, a major threat to the validity of the conclusions from the
    evaluation is that the observed correlations between exposure to the
    campaign and self-reported attitudes and behaviors could reflect
    preexisting differences among individuals in their underlying
    susceptibility to campaign messages. The evaluation's associations between
    exposure to the campaign and self-reported initiation of marijuana use
    took into account statistically the individual differences in attributes
    among youth who were exposed to various levels of campaign messages, and
    they adjusted for the influence of other variables that could determine
    marijuana initiation-called confounder variables. As such, Westat's
    evaluation of the associations between campaign exposure and marijuana
    initiation have accounted for individual differences among youth and can
    be viewed as comparisons of outcomes for statistically similar
    individuals. Further, the statistical test Westat used in assessing the
    relationship between exposure and initiation did not rely upon assumptions
    of linearity between levels of exposure and initiation. Instead, it tests
    for an ordered relationship between exposure and an outcome such as
    marijuana use initiation.

    19Westat also called its longitudinal analysis a "delayed effects"
    analysis.

    Westat used statistical methods to address the possibility that
    preexisting differences between individuals could have caused both
    reported levels of exposure and respondent outcomes, and its use of these
    methods contributed to the validity of its findings about the effects of
    the campaign on outcomes. If, independently of the campaign, individuals
    differed in their underlying tendencies to accept and recall campaign
    messages, and if the individuals who were more likely to recall
    advertisements also were those who were more likely to respond to
    advertisements, then, absent efforts to address this confounding factor,
    the findings about the evaluation would be questionable. This type of bias
    is often called a selection effect. If selection effects occurred in the
    campaign, then both exposure and reported changes in attitudes and
    behaviors could reflect underlying beliefs that were not affected by the
    campaign, despite the presence of statistical correlations between
    self-reported exposure and changes in attitudes and behaviors.

    To control for selection effects and the many factors that could have
    influenced both exposure and outcomes independently of, or in conjunction
    with, the campaign, Westat used propensity scoring methods. These methods
    limit the influence of preexisting differences among exposed groups by
    controlling for a wide range of possible confounding variables. Propensity
    score methods are used to create comparison groups that are similar on
    measured and potentially confounding variables but that differ on their
    levels of treatment. In the evaluation of the campaign, the comparison
    groups were similar on confounding variables but differed on their level
    of exposure to campaign messages. Propensity score methods replace a set
    of confounding variables with a single function of these variables, which
    is called the propensity score. In Westat's analysis, an individual's
    propensity score is considered to represent an individual's probability of
    being assigned to a particular level of exposure to the campaign,
    conditional upon the individual's values of the confounding variables. By
    including relevant, potentially confounding variables and matching
    individuals on their propensity scores, Westat was able to minimize bias
    due to selection effects. The comparison groups that Westat created by
    using propensity score methods can be considered as statistical analogues
    to randomly assigning individuals to different levels of exposure. After
    creating these groups, Westat then analyzed outcomes among the groups
    having different propensities to be exposed to campaign messages.

    Our assessment of Westat's methods leads us to conclude that Westat took
    reasonable steps to develop valid propensity models, and as a result of
    its models, its analysis identified the effects of the campaign, net of
    other factors included in its propensity score models. First, rather than
    simply compare individuals who were exposed to campaign messages with
    those who were not exposed, Westat estimated and compared groups of
    individuals with different levels of exposure, where the number of
    exposure groups was measured alternatively as a three- or four-level
    variable-e.g., low, medium, or high exposure.20 Second, for the results of
    propensity methods to be valid, it is important that the propensity
    scoring models include all relevant variables that could otherwise explain
    differences in both exposure and outcomes, as evaluators can adjust only
    for confounding variables that are observed and measured. If an important
    variable is omitted from the propensity model, the results of analyses may
    be affected. Westat's models included many relevant and potentially
    confounding variables. For example, in the youth models, the propensity
    score models included measures of demographic attributes, educational
    attainment and educational aspiration, family and parent background,
    parental consumption of television and other media, income and employment,
    reading habits, Internet usage, location of residence in an urban area,
    among other variables. Third, for propensity models to remove the effects
    of confounding variables from the association between exposure and
    response, it is necessary that the population means of the confounder
    variables not vary across exposure levels. If a confounder is successfully
    balanced, then it will have the same theoretical effect across all
    exposure levels. After estimating models, Westat also assessed and
    demonstrated the balance of variables in its propensity models.

    20Propensity score methods have been demonstrated to be robust against
    bias associated with the specification of incorrect functional forms-e.g.,
    linear rather than quadratic- of variables.

    The Phase III Evaluation Provided Mixed Evidence of the Campaign's Effectiveness
      on Intermediate Outcomes, but It Found No Effect of the Campaign on Parental
                                  Monitoring of Youth

    Westat reported mixed evidence about the effectiveness of the campaign on
    intermediate outcome measures-such as recall and identification of
    campaign messages, youth anti-drug attitudes, and parents' beliefs and
    behaviors-that were thought to be causal factors influencing youth drug
    use, the ultimate target of the campaign. Most parents and youth recalled
    exposure to campaign anti-drug messages, and for both groups, recall
    increased during the September 1999 to June 2004 period covered by the
    phase III evaluation. For current, non-drug-using youth-whose resistance
    to initiating marijuana use the campaign intended to affect-although NSPY
    data showed some favorable trends in anti-drug attitudes and beliefs and
    in the proportion of youth who said that they would definitely not try
    marijuana, there was no evidence that exposure to the campaign influenced
    these trends. Conversely, among current, non-drug-using youth, evidence
    suggested that exposure to the campaign had unfavorable effects on their
    anti-drug norms and perceptions of other youths' use of marijuana-that is,
    greater exposure to the campaign was associated with weaker anti-drug
    norms and increases in the perceptions that others use marijuana. On three
    of five parent belief and behavior outcome measures-including talking with
    children about drugs, doing fun activities with children, and beliefs
    about talking with children-the evidence pointed to a favorable campaign
    effect on parents. However, while there was mixed evidence on the effect
    of the campaign on parents' beliefs and attitudes about monitoring
    children's behaviors, there was no evidence to support a claim that the
    campaign actually affected parents' monitoring behaviors-an area of the
    campaign's focus for parents-and there was little evidence for favorable
    indirect effects on youth behavior or beliefs as the result of parental
    exposure to the campaign.

    Youth and Parents' Recall of Campaign Advertisements Increased over Time, Their
    Impressions of the Advertisements Were Favorable, and They Could Identify the
    Campaign Brand

    According to Westat, the campaign purchased enough advertising time over
    the 58-month period from September 1999 to June 2004 to achieve an average
    exposure of 2.5 youth-targeted ads per week for youth and an average of
    2.2 parent-targeted advertisements per week for parents. Westat's
    estimates include campaign advertisements intended for either all youth or
    all parents, but they do not include exposure of youth to parent
    advertisements or parents to youth advertisements, nor do they account for
    separate advertising targeted to specific race- or ethnicity-defined
    audiences.

    Using exposure indexes, Westat measured trends in general and specific
    exposure to campaign advertisements. The general exposure index was based
    on questions that asked about exposure to anti-drug messages in recent
    months through a variety of channels, including movies, television, radio,
    and billboards, and was not limited to campaign advertisements.21 The
    specific exposure index was based on recall of specific advertisements
    broadcast during the 60 days prior to the respondent's interview, and was
    limited to advertisements that targeted the respondent. For example, for
    youth, only youth advertisements were sampled to measure specific
    exposure. Youth aged 12 1/2 to 18 and their parents reported increasing
    levels of recall of specific but not general exposure to campaign
    advertisements over time. For both parents and youth, there was a sharp
    increase over time in the recall of specific exposure of television ads
    across the campaign. Westat speculated that the increase in specific
    recall may have arisen from better-placed, more memorable, or longer-aired
    advertisements rather than only to an overall increase in television
    advertisements. However, recall of all general anti-drug advertising was
    fairly stable over time, as there was no overall detectable change in
    reported general exposure over the course of the campaign.

    Beginning in 2001, when the evaluation started to measure brand phrase
    recall, and continuing through 2004, the evidence indicates that youth, in
    particular, exhibited increases in brand phrase recall. Advertising
    campaigns may use a brand phrase to provide a recognizable element, and to
    the extent that the brand is recognized and positively regarded, its
    familiarity may lead to a positive response to a new advertisement or
    increase the perception that each advertisement is part of a larger
    campaign. The campaign included both a parent and a youth brand. Brand
    messages may have involved a series of phrases or the portrayal of an
    activity or lifestyle as positive (e.g., participating in team sports) to
    set up the brand phrase of "The Anti-Drug." Westat reported that the
    evidence from the NSPY shows that the greater the exposure to media
    campaign advertising, the more likely respondents were to recall the brand
    phrase. In addition, the more that respondents recalled specific ads, the
    more likely they were to recognize the brand phrase, although over time
    even those with less exposure had learned the brand phrase.

    Overall, youth reported favorable impressions of the subset of campaign
    television advertisements that they were asked to evaluate, and their
    favorable impressions increased over time. Responses to the
    advertisements-whether they were attention getting, convincing, or said
    something important to the respondent-were positive among both youth and
    their parents. Parents' evaluations of the advertisements were generally
    more positive than those of youth, and parents' positive views also
    increased over time.

    21According to Westat, the reference period for the general exposure
    index, is "in recent months," and this wording was chosen to maintain
    equivalence to the wording used in the Monitoring the Future surveys in
    its questions about anti-drug advertising.

    In addition to distributing messages directly in media advertisements, the
    campaign aimed to reach its target audiences indirectly through other
    institutions and routes, such as community groups, in-school and
    out-of-school anti-drug education, and discussions among youth and
    parents, and youth and friends, concerning drug use and the drug
    advertisements. The NSPY data indicated that the campaign's messages were
    not accompanied by similar increases in exposure to messages from other
    sources. Both youth and parents reported receiving anti-drug messages from
    other sources, but they did not consistently report increases in exposure
    to messages from these sources. For example, from the 2000 to 2004
    samples, the percentages of youth reporting receiving in-school drug
    education messages and attending out-of-school drug education both
    declined.

    Westat Found That the Campaign Generally Had No Effect on the Attitudes of Youth
    Not Using Marijuana toward Its Use but That Exposure to the Campaign Was
    Associated with Unfavorable Effects on Youth Perceptions of Others' Use of
    Marijuana

    Westat generally found no significant effects of campaign exposure on the
    cognitive outcomes of adolescent nonusers of marijuana- i.e., development
    of anti-drug attitudes and beliefs. For current nonusers, the evaluation
    reported on four cognitive measures and a fifth measure of their
    perceptions of others' use of marijuana. Three of the four
    measures-attitudes and beliefs about the consequences of marijuana use;
    perceived social norms or pressures from parents, friends, and peers about
    infrequent or regular marijuana use; and perceived self-efficacy to avoid
    using marijuana, or their confidence to turn down use of marijuana under
    various circumstances-were premised to affect the fourth-youth intentions
    to use marijuana at all during the next year. The fifth outcome,
    perceptions of other youths' use of marijuana,22 was included to examine
    whether exposure to the campaign was leading to increased perception among
    youth that others use marijuana, and whether this perception, in turn,
    affected their own behaviors.

    Westat reported that the evidence from the analysis of trend data from
    2000 to 2004 for two of the youth cognitive measures-attitudes and beliefs
    about the consequences of marijuana use and intentions to use
    marijuana-showed significant increases in youth believing that marijuana
    use had negative consequences and significant increases in the percentage
    of youth that reported that they had no intention to use marijuana.
    However, evidence from both cross-section and longitudinal associations
    between exposure and these two cognitive outcomes did not substantiate
    that the favorable trends arose from exposure to the campaign.
    Specifically, the cross-sectional associations between both general and
    specific exposure to the campaign and intentions not to use marijuana show
    no significant favorable effects of exposure on this outcome. None of the
    cross-section associations between either general or specific exposure and
    intention to use marijuana are significant, and none of the longitudinal
    associations between specific exposure and intentions are significant. Two
    of the longitudinal associations between general exposure and intentions
    are significant, but the direction of the effect is unfavorable, in that
    greater exposure led to declines in intentions not to use marijuana. The
    evidence from the associational analyses between exposure and attitudes
    and beliefs about the consequences of marijuana use generally did not show
    an effect of the campaign. While there was one significant cross-section
    association between general exposure and attitudes and beliefs about
    consequences during the final two waves of survey data, there were no
    significant cross-section associations between specific exposure and
    attitudes and beliefs about consequences, nor were there any significant
    longitudinal associations with either general or specific exposure.

    22This was measured as the "Percent perceiving few other kids regularly
    use marijuana."

    The associational analysis also produced some evidence of unfavorable
    effects of exposure on social norms-i.e., social pressures from parents,
    peers, and other important persons about marijuana use. Westat's
    cross-section associations showed no significant effects of exposure on
    social norms, but its longitudinal associations showed that across all
    survey rounds, there was a significant relationship between specific
    exposure and weaker social norms. Westat's analysis of associations
    between exposure and perceptions of others' use of marijuana also produced
    significant results. Cross-section associations between specific exposure
    and perceptions of others' use were significant, as were longitudinal
    associations of this relationship. In other words, among youth who
    reportedly did not use marijuana at the time of their interview, there was
    a significant effect of specific exposure on the perception that others
    used marijuana, and the direction of the effect was unfavorable-that is,
    those reporting higher exposure to anti-drug ads were more likely to
    believe that their peers used marijuana regularly. A significant and
    unfavorable relationship between specific exposure and perceptions of
    others' use of marijuana was obtained for the data covering the entire
    period of the evaluation as well as for the period of the redirected
    campaign, from 2002 to 2004.

    The Evaluation Reported Favorable Effects of the Campaign on Three Parent
    Outcomes but Not on Parental Monitoring

    A theme of the campaign was to encourage parents to engage with their
    children to protect them against the risk of drug use, and parent skills
    were a focus of parent advertising almost since the start of the campaign.
    The campaign encouraged parents to monitor their children's behavior by
    knowing where they were and with whom, and to make sure that they had
    adult supervision. It also encouraged parents to talk with their children
    about drugs and to a lesser degree to engage in fun activities with their
    children. The evaluation observed five outcomes for parents, and for four
    of the five found significant and favorable effects of exposure to the
    campaign. For three outcomes-parent-child conversations about drugs
    (talking behavior), parents' beliefs and attitudes about talking with
    their children about drugs (talking beliefs), and parents' engagement with
    their children in in-home and out-of-home activities (fun activities)-both
    cross-section and longitudinal associations between exposure and outcomes
    were generally significant and favorable to the campaign. For parents'
    beliefs and attitudes toward monitoring their children's behaviors, Westat
    reported favorable trend and cross-sectional associations but no
    significant overall longitudinal effects of either general or specific
    exposure on this outcome. For the fifth outcome, parent monitoring
    behaviors-that is, parents' knowing or having a pretty good idea about
    what their child was doing or planned to do-the evidence did not support a
    finding of an effect of the campaign. There were no significant favorable
    trends in parents' reports of monitoring behaviors, and there were no
    significant cross-section or longitudinal associations of either general
    or specific exposure on monitoring behaviors.

    No Evidence of Favorable Effects of the Campaign on Youth Outcomes through
    Campaign Effects on Parental Outcomes

    Despite evidence of some favorable parental outcomes for the campaign,
    Westat found no significant evidence for the overall evaluation that these
    favorable parent outcomes affected youth attitudes and behaviors toward
    drug use. Specifically, for the entire period covered by the evaluation,
    Westat found no evidence of overall, indirect campaign effects on parents
    leading to changes in marijuana use, intentions to use marijuana, social
    norms, self-efficacy, or cognitions among youth who were not marijuana
    users. Westat found that there were some significant indirect effects of
    parental specific exposure on some youth outcomes for some subgroups. For
    example, parental specific exposure was favorably associated with
    intentions to use marijuana for 14- to 18-year-olds and for boys, and it
    was also associated favorably with attitudes and beliefs about the
    consequences of marijuana use for Hispanics. Westat also found significant
    but unfavorable indirect effects of parents' general exposure on subgroups
    of youth in other youth outcomes. For example, parental general exposure
    was unfavorably associated with youth social norms for 14- to 16-year-olds
    and for girls.

        The Phase III Evaluation Found No Significant Effects of Exposure to the
     Campaign on Youth Drug Use Outcomes Other than Limited Unfavorable Effects on
                                  Marijuana Initiation

    Westat reported that the NSPY data showed some declines in self-reported
    lifetime and past-month use of marijuana by youth over the period from
    2002 to 2004, and these trends in NSPY were consistent with trends in
    other national surveys of drug use over these years. Westat also reported
    that the NSPY data showed declining trends in youth reports of offers to
    use marijuana. However, Westat cautioned that because trends do not
    account for the relationship between campaign exposure and changes in
    self-reported drug use, drug use trends alone should not be taken as
    definitive evidence that the campaign was responsible for the declines. On
    the basis of the analysis of the relationship between exposure to campaign
    advertisements and youth self-reported drug use in the NSPY
    data-assessments that used statistical methods to adjust for individual
    differences and control for other factors that could explain changes in
    self-reported drug use-for the entire period covered by its evaluation,
    Westat found no significant23 effects of exposure to the campaign on
    initiation of marijuana by prior nonusing youth. The only significant
    effect indicated in Westat's analysis of the relationship between campaign
    exposure and self-reported drug use was an unfavorable effect of exposure
    on marijuana initiation-that is a relationship between campaign exposure
    and higher rates of initiation-for one round of NSPY data and similar
    unfavorable effects of campaign exposure on marijuana initiation among
    certain subgroups of the sample (e.g., 12 1/2- to 13-year-olds and girls).
    Westat found no effects of campaign exposure on rates of quitting or use
    by prior users of marijuana.

    23In discussing Westat's findings, any references to significance refers
    to statistical significance.

    Westat Tracked Trends in Marijuana Use from Several Sources and Reported That
    the Trend Data by Themselves Were Insufficient to Demonstrate Effects of
    Exposure to the Campaign

    Westat tracked trends in self-reported use of marijuana by youth and
    trends in youth reports of offers to use marijuana for the period from
    2000 to the first half of 2004 to determine if there were significant
    declines. Westat also assessed these trend data for changes occurring
    since 2002, or during the period of the redirected campaign. Westat's
    trend analysis was designed to provide supportive but not definitive
    evidence for campaign effects.

    In its trend analysis, Westat compared trends in self-reported drug
    use-lifetime, past year, and past month-in the NSPY with trend data on
    self-reported drug use from three other nationally representative surveys
    of drug use-Monitoring the Future, the Youth Risk Behavior Surveillance
    System (YRBSS), and the National Survey on Drug Use and Health.24 Both MTF
    and YRBSS are school-based surveys, and NSDUH is a household survey that
    provides estimates of drug use by the civilian, noninstitutionalized
    population of the United States aged 12 years and older. Methodological
    differences between the school-based surveys-MTF and YRBSS-and the
    household surveys-NSPY and NSDUH-have been shown to account for the some
    of the differences in estimates of marijuana use.

    According to Westat's analysis, the surveys of self-reported marijuana use
    show some similarities and differences in trends depending upon the
    measure, age group, or subperiod covered within the longer 2000 to 2004
    period. For example, the MTF data generally show declines in lifetime,
    past-year, and past-month self-reported drug use for 8th, 10th, and 12th
    graders over the years from 2000 to 2004, although only some of the
    year-to-year differences in the MTF self-reported drug use data were
    statistically significant. Nonetheless, for the subperiod from 2002 to
    2004, MTF data show statistically significant declines in past-year and
    past-month use for 8th graders and past-year use for 10th graders, and the
    NSPY data also show statistically significant declines in past-month use
    from 2002 to 2004 for youth aged 12 1/2 to 18 years old and for 14- to
    18-year-olds. On the other hand, the MTF data suggest a decline in
    past-year and past-month use by 10th graders from 2000 to 2002, but the
    NSPY data suggest an increase in past-month marijuana use during this
    period.25 Further, the data from NSDUH for 2000 and 2001 also show
    statistically significant increases in lifetime, past-year, and past-month
    marijuana use among youth aged 12 to 17, statistically significant
    increases in lifetime and past-year marijuana use for youth aged 16 to 17,
    and a statistically significant increase in past year use for youth aged
    14 to 15. The pattern of increase in NSDUH data from 2000 to 2001 is
    consistent with the 2000 to 2002 increases in past-month use in NSPY, but
    they differ from the MTF trends over this period.

    24This survey was formerly known as the National Household Survey on Drug
    Abuse.

    25Westat points out, however that the MTF decline in use among 10th
    graders between 2001 and 2002 was within the statistical confidence limits
    of NSPY.

    All four surveys generally show declines in marijuana use beginning in
    2002, but not all of the declines are statistically significant. Both MTF
    and NSPY show some statistically significant declines since 2002, and
    while NSDUH and YRBSS show declines, the declines were not statistically
    significant. These declines starting in 2002 coincide with the redirected
    campaign and the introduction of the Marijuana Initiative.

    Despite the concurrence of the trend data from all sources for the 2002 to
    2004 period, Westat concluded that the existence of declining trends in
    self-reported drug use by themselves do not provide definitive evidence
    that the campaign caused the declines because factors other than the
    campaign also could affect behavior. For example, changes in high-school
    completion rates among youth could affect drug use behaviors, as high
    school dropouts may have more involvement with drugs than youth who stay
    in school. Additionally, declines in self-reported drug use that began
    before the initiation of phase III of the campaign could not have been
    caused by the campaign. The declines reported in MTF began prior to the
    start of phase III of the campaign; therefore, factors other than the
    campaign had to have been responsible for the start of the decline
    occurring in these data. Further, ONDCP also has acknowledged the
    limitations of trends in the national surveys for determining whether
    changes in drug use were the result of the campaign. ONDCP's Office of
    Programs, Budget, Research and Evaluation wrote about the MTF, YRBSS, and
    NSDUH:26

    "They provide policy makers with broad indicators of the success of
    policy...However, they will not be able to answer the critical question of
    whether these changes were the result of the Media Campaign. These surveys
    do not ask respondents about their exposure and reactions to the messages
    of the Media Campaign that can then be linked to their drug-related
    attitudes and behavior."27

    26At the time that ONDCP prepared this document, NSDUH was still known as
    the National Household Survey on Drug Abuse, or NHSDA.

    Westat Reported That Trends in Marijuana Offers Declined over Time, but Factors
    Other than the Campaign Contributed to Changes in Offers

    Westat assessed trends in youth reports of receiving offers of
    marijuana-whether anyone had ever offered youth marijuana and the
    frequency of offers within the past 30 days. Marijuana offers are closely
    related to marijuana use, and the campaign aired messages that encouraged
    resistance to offers of marijuana. Over the 2000 to 2004 period, Westat
    found significant increases in the percentage of youth reporting that they
    had never received offers, and it also found significant decreases in the
    percentage of youth reporting that they had received offers in the prior
    month. Westat also found significant changes in offers over 2002 to 2004,
    during the period of the redirected campaign, and these changes were
    generally consistent with the trends for the overall 2000 to 2004 period.
    Further, on the basis of longitudinal analysis of the relationship between
    offers in one period and marijuana use in the subsequent period among
    youth who were nonusers in an initial survey round-an analysis that
    assesses whether offers precede use or are simply a correlate of it-Westat
    found that youth who reported having received a marijuana offer at one
    period were much more likely-between three and seven times more likely,
    depending upon age group-to have initiated marijuana use at a following
    period than nonusing youth who reported never having received such an
    offer. However, as Westat reported, while the findings on offers are
    favorable, they cannot be ascribed to the campaign because they may be
    caused by other factors, as the analysis of the relationship between
    offers and use did not take into account other factors that could affect
    use.

    27Office of Programs, Budget, Research and Evaluation, Office of National
    Drug Control Policy, "Youth Drug Use and the National Youth Anti-Drug
    Media Campaign," February, 2001 (Washington, D.C.: Executive Office of the
    President), p. 16.

    On the Basis of Its Analysis of the Association between Exposure and Drug Use
    Outcomes, Westat Found No Evidence That Exposure to the Campaign Affected
    Initiation or Cessation of Marijuana Use

    From its longitudinal analysis of associations between exposure and
    initiation of marijuana use, Westat found no evidence that increased
    exposure to the campaign reduced youth's initiation of marijuana use.
    Westat's longitudinal analysis assessed the effects of exposure at one
    survey wave on marijuana initiation at a subsequent survey wave,
    controlling for potential confounding variables that could affect the
    exposure initiation relationship. Westat assessed the effects of two types
    of exposure on initiation of marijuana use-general exposure and specific
    exposure. General exposure represents the sum of recalled exposure to
    anti-marijuana advertising in four types of sources of
    advertisements-television and radio, movies and videos, print media
    including newspapers and magazines, and outdoor media. Specific exposure
    represents the sum of recalled exposure to youth-targeted individual
    campaign television advertisements that had been aired in the 60 days
    prior to an interview.

    Westat found no significant effects of the level of general exposure on
    marijuana use initiation, either over the entire period of the campaign or
    between subperiods as defined by survey rounds.28 Westat also found no
    overall effects of levels of specific exposure on marijuana initiation
    during the entire period of the campaign, but it found one significant
    association between specific exposure and marijuana use initiation that
    occurred in the data from wave 7 and its wave 9 follow-up, or during the
    period of the Marijuana Initiative. Wave 7 was the first complete survey
    wave covering exposure to the Marijuana Initiative. The significant
    association from this analysis was that higher levels of specific exposure
    were associated with higher levels of initiation of marijuana use among
    previously nonusing youth.

    Westat also examined the longitudinal relationships between exposure and
    initiation for nine subgroups of youth (two sexes, three race/ethnicity
    groups, two risk groups, and two nonoverlapping age groups). For several
    subgroups, it found significant associations between specific exposure and
    marijuana initiation. These associations were in a direction that was
    unfavorable to the campaign, in that greater specific exposure was
    associated with higher levels of initiation. The subgroups for which these
    unfavorable associations were most pronounced included 12 1/2- to
    13-year-olds, girls, African Americans, and lower risk youth.

    28Westat's assessed the exposure-initiation relationship using data from
    survey rounds 1 and 2, survey rounds 2 and 3, and within survey round 4;
    it assessed the exposure-initiation relationship between waves 6 and 8 and
    waves 7 and 9.

    On cessation and reduction of marijuana use, Westat assessed two outcomes
    among current marijuana users: the rate at which they quit using marijuana
    and their frequency of use. The frequency of use measure allowed for
    campaign effects to be observed if users did not quit but reduced their
    use of marijuana. Westat estimated that the quit rate-the percentage of
    prior-year users reporting that they no longer used marijuana-among
    prior-year users of marijuana was 24.8 percent. However, it found no
    statistically significant association between general exposure and
    quitting or between specific exposure and quitting. It also found that
    among adolescent marijuana users, the frequency of use-increase, decrease,
    or no change-was not affected by exposure to the campaign.

                                      Conclusions

    A well-designed and executed multiyear study of the impact of the ONDCP
    anti-drug media campaign on teen initiation of drug use, or cessation of
    drug use, shows disappointing results for the campaign. The study provides
    no evidence that the campaign had a positive effect in relation to teen
    drug use, and shows some indications of a negative impact. Some
    intermediate outcomes, such as parents talking with children about drugs,
    and doing fun activities with their children, showed positive results in
    that the media campaign encouraged parents to adopt these behaviors.
    However, other intermediate outcomes, such as parents' monitoring of their
    children's behavior, were not shown to be affected by the campaign.
    Moreover, the evaluation did not provide evidence that intermediate
    outcomes that showed positive results translated into greater resistance
    to drugs among the teenage target population.

    Unfavorable preliminary findings from the evaluation were reported by
    Westat in 2002. Beginning in 2002, ONDCP took a number of steps that were
    intended to strengthen the power of the campaign to achieve positive
    results. These steps included more rigorous ad copy testing and a
    concentration on anti-marijuana messages. However, the post-2002 results
    yielded no evidence of positive impacts and some evidence of negative and
    unintended consequences in relation to marijuana use. Specifically,
    exposure to advertisements during the redirected campaign was associated
    with higher rates of marijuana use initiation among youth who were prior
    nonusers of marijuana.

    Most parents and youth recalled exposure to the campaign messages and,
    further, they recognized the campaign brand. Thus, the failure of the
    campaign to show positive results cannot be attributed to a lack of
    recognition of the messages themselves. This raises concerns about the
    ability of messages such as these to be able to influence teen drug
    attitudes and behaviors. It raises questions concerning the understanding
    of the factors that are most salient to teens' decision making about drugs
    and how they can be used to foster anti-drug decisions.

    Westat's evaluation is centered on this particular configuration of a
    media campaign as it was presented from 1999 to 2004, and its results
    pertained to the campaign nationwide. It cannot be construed to mean that
    a media campaign that is configured differently from this one cannot work.
    Nor should its results be construed to mean that in some locations, for
    some groups of youth, the campaign did not have an effect on drug use.
    However, substantial effort and expertise were brought to the task of
    designing the advertisements from the outset, and the 2002 redirection of
    the campaign placed even greater emphasis on copy testing and enhanced
    ONDCP oversight. This casts some doubt on the notion that a better media
    campaign can lead to positive results.

    It is also important to note that two recent smaller studies in three
    locations have provided evidence of a limited effect of the campaign for
    some youth, and it is quite possible that additional analyses of the NSPY
    data using different methods or measures may find other effects of the
    campaign, at least for some adolescents, than have been produced by
    Westat's evaluation team. The data from the evaluation have only recently
    been made available to academic and other researchers, and while the
    analyses undertaken by Westat are, as we have noted elsewhere, appropriate
    and thorough, they are not exhaustive.

    It is heartening that surveys intended to measure teen drug use, such as
    Monitoring the Future, are showing declines in marijuana use in recent
    years. Indeed, NPSY also shows some evidence of a decline in drug use
    among teens. However, Monitoring the Future and other surveys of teens
    concerning drug use are not linked to exposure to the media campaign, and
    NPSY shows no relationship between anti-drug media campaign exposure and
    favorable drug outcomes for teens. This seems to indicate that other
    unidentified factors, other than the anti-drug media campaign, are
    affecting drug use decisions among teens.

    Although ONDCP has pointed to declines in teen drug use and credited the
    campaign along with other prevention efforts as contributing to
    significant success in reducing teen drug use, trend data derived from the
    Monitoring the Future survey that show declines in teen marijuana use from
    2001 to 2005 do not explicitly take into account exposure to the campaign,
    and therefore, by themselves, cannot be used as evidence of effectiveness.
    ONDCP has indicated in the past, and we concur, that because these surveys
    cannot link their results with the media campaign, they do not measure
    campaign effectiveness. The evaluation of the media campaign reinforces
    the lack of linkage between the media campaign and teen drug use behavior.

    It is important to note that virtually all social science research is
    imperfect. Attempting to systematically observe and document human
    behavior in real-world settings is a daunting task given the extremely
    wide variation in both humans and settings. We believe that the evaluation
    of the ONDCP media campaign is credible in that it was well designed given
    the circumstance of the campaign, and appropriately executed.

                         Matter for Congressional Consideration

    In light of the fact that the phase III evaluation of the media campaign
    yielded no evidence of a positive outcome in relation to teen drug use and
    congressional conferees' indications of their intentions to rely on the
    Westat study, Congress should consider limiting appropriations for the
    National Youth Anti-Drug Media Campaign beginning in the fiscal 2007
    budget year until ONDCP is able to provide credible evidence of the
    effectiveness of exposure to the campaign on youth drug use outcomes or
    provide other credible options for a media campaign approach. In this
    regard we believe that an independent evaluation of the new campaign
    should be considered as a means to help inform both ONDCP and
    Congressional decision making.

                           Agency Comments and Our Evaluation

    We provided a draft of this report to the Director of the Office of
    National Drug Control Policy for comment on July 31, 2006. ONDCP provided
    us with written technical comments on the report, which we incorporated
    where appropriate. In addition, ONDCP provided written comments about our
    report in which it raised a question about our matter for congressional
    consideration and outlined a number of concerns that it had with our
    report on Westat's findings. These written comments are reproduced in
    appendix II. In our evaluation of ONDCP's written comments, we address
    each of the other concerns in the order ONDCP presented them.

    Westat Evaluation's Role in Judging the Impact of the Advertising Campaign

    ONDCP comments that Westat's evaluation is ill suited to judge the impact
    of an advertising campaign in part because Westat attempted to establish a
    causal relationship between exposure and outcomes, and this, ONDCP
    indicates, is something that major marketers rarely attempt because of its
    difficulty. ONDCP writes, "we take issue with the fundamental method
    pursued by Westat and GAO, and therefore, believe that the study's
    findings are deeply flawed." We find this response surprising for a number
    of reasons. First, ONDCP is on record as stating that the evaluation
    conducted by Westat would be the means to assess the impact of the
    campaign. Indeed, in February, 2001, in the ONDCP publication entitled
    Youth Drug Use and the National Youth Anti-Drug Media Campaign, ONDCP
    states:

    "ONDCP, on the other hand, is measuring the impact of the Media Campaign
    with a thorough, rigorous, and independent evaluation. The nationally
    representative evaluation is being conducted for ONDCP by the National
    Institute on Drug Abuse (NIDA)....The evaluation is a 4-year longitudinal
    study of parents' and their children's exposure and response to the Media
    Campaign....ONDCP will be able to assess the extent to which changes in
    anti-drug attitudes and beliefs or drug using behavior can be attributed
    to the Media Campaign."29

    ONDCP officials had opportunities during the evaluation to raise concerns
    about Westat's design and its efforts to establish a link between exposure
    to the campaign and outcomes, but we are not aware of their having done
    so. However, we are aware of ONDCP's participation in a NIDA-sponsored
    expert panel review of Westat's evaluation that was held in August 2002.
    Our review of the minutes of that meeting reveals that while an ONDCP
    official raised concerns about issues such as assessing the nonadvertising
    components of the campaign and the number of interim reports, ONDCP
    officials did not at that time raise concerns that the evaluation was
    fundamentally flawed. The consensus of the expert panel was that Westat's
    evaluation was "pretty impressive" given the challenges presented by the
    absence of baseline data and of an experimental design. Panel members also
    asserted that Westat's use of propensity score models to isolate the
    effects of the campaign was termed both "sensible" and "state-of-the-art."

    ONDCP further states that major advertisers evaluate the success of their
    campaigns by rigorously testing advertisements prior to airing and by
    developing correlations between messages and consumer attitudes and
    behavior. While we do not dispute whether this is a commonly used approach
    among major advertisers, we believe that in assessing the expenditure of
    public funds researchers should attempt, where feasible, to establish
    causal relationships or use research designs that attempt to isolate the
    effects of federally funded interventions. While we acknowledge that
    establishing causal relationships is difficult, we maintain that Westat
    used sophisticated and appropriate statistical methods that aimed to
    isolate the effects of recalled exposure to the campaign on youth drug
    use. Further, adopting a methodology that relies upon correlations between
    advertising messages and an outcome, such as reductions in youth drug use,
    without attempting to take into account many of the other factors that
    could affect drug use allows for too many post hoc explanations of
    findings. Westat's analysis included socioeconomic factors, parent
    characteristics, television viewing habits, risk of using drugs, and
    sensation-seeking tendencies to be able to determine whether exposure was
    related to drug use net of the influences of these factors. We conclude,
    on the basis of our assessment of Westat's methods, that exposure to
    campaign messages generally did not influence youth drug use net of these
    other influences.

    29Office of Programs, Budget, Research and Evaluation, Office of National
    Drug Control Policy, "Youth Drug Use and the National Youth Anti-Drug
    Media Campaign," February, 2001 (Washington, D.C.: Executive Office of the
    President), p. 3.

    ONDCP notes that correlational findings have been used to assess
    anti-tobacco advertising campaign results. We have not reviewed the
    anti-tobacco campaign and cannot comment on its relationship to youth
    smoking prevalence. We notice, however, that in ONDCP's comments on
    "Consequences of Further Budget Cuts," it appears to contradict its
    statements about establishing causal relationships to determine the effect
    of advertising campaigns. ONDCP writes, "Previous studies have established
    a relationship between exposure to anti-tobacco messages and smoking rates
    among teens." ONDCP goes on to draw an analogy between anti-smoking
    messages and anti-drug messages to write, "We should expect similar
    results for illicit drug use if anti-drug messages decline." These
    statements emphasize very directly the same kind of causal relationships
    that ONDCP cites as not appropriate in its opening comments.

    We also note that ONDCP indicates in its comments that it has made
    multiple refinements to the media campaign on the basis of earlier
    findings from the Westat study. This seems to be inconsistent with a
    position of major concerns with the fundamental soundness of the study.

    Finally, the three research papers that ONDCP cites on page 2 of its
    comments on "Conflicting Evidence from Other Research" all use
    exposure-response methodologies that are analogous to Westat's and all
    attempt to isolate the causal effects of exposure either to ONDCP's
    campaign or to other media campaigns. Thus, it would seem that ONDCP's
    comment that efforts to isolate causal effects of media campaigns are
    fundamentally flawed would also apply to these three studies.

    ONDCP Made Campaign Changes as a Result of Westat Interim Findings

    ONDCP indicates that it has sought to improve the performance of the media
    campaign by using the results from the Westat study and other data. We are
    aware that ONDCP redirected the campaign in 2002 in response to Westat's
    interim findings that indicated some negative impacts of the campaign on
    youth marijuana use. However, the 2002 to 2004 Westat study results also
    did not show positive outcomes. Westat's study is the only national
    evaluation of the campaign. Although Monitoring the Future provides
    context regarding general drug trends among youth, as ONDCP has stated:

    "These surveys [MTF, the National Household Survey on Drug Use, and the
    Youth Risk Behavior Survey] will permit the determination of whether drug
    use behavior and related attitudes and beliefs changed after the launching
    of Phase III of the Media Campaign in mid-1999. However, they will not be
    able to answer the critical question of whether these changes were the
    result of the Media Campaign. These surveys do not ask respondents about
    their exposure and reactions to the messages of the Media Campaign that
    can then be linked to their drug-related attitudes and behavior."30

    More recently in late 2005, ONDCP launched a newly designed campaign. The
    impact of this campaign is not known and should be independently
    evaluated.

    Other Youth Drug Use Findings

    ONDCP believes we did not provide adequate discussion of studies that
    report findings contrary to those of Westat. Our report mentions two of
    the three studies that ONDCP identifies-the Longshore and Palmgreen
    studies. Our report does not mention the third study, Slater, because it
    focused on a different anti-drug media campaign approach and not on the
    ONDCP media campaign. Overall, these studies' findings are not necessarily
    "contrary" to Westat's findings. Rather, they assess small slices of the
    youth population or particular circumstances (such as other programs that
    could reinforce an anti-drug message) and find some positive results. The
    Westat national findings do not preclude the findings of positive results
    for some subpopulations of youth. The Palmgreen study, for example found a
    positive effect for the media campaign on high-sensation-seeking youth,
    but did not find an effect on non-high-sensation-seeking youth in the two
    moderate size communities in which the study was conducted. The
    distribution of these youth in the nationwide population could be
    consistent with both studies being correct. Our objective was to assess
    the Westat study as a national evaluation of the impact of the national
    campaign. In the Slater study, after being trained in the use of campaign
    media materials, leaders in each of eight communities that received a
    media campaign were allowed to develop their own media strategies and were
    able to use whatever materials they chose or developed on their own. This
    approach emphasized the flexibility to adopt different media strategies
    deemed appropriate by individual communities and not the use of a single
    national strategy.

    30Office of Programs, Budget, Research and Evaluation, Office of National
    Drug Control Policy, "Youth Drug Use and the National Youth Anti-Drug
    Media Campaign," February, 2001 (Washington, D.C.: Executive Office of the
    President), p. 16.

    ONDCP expressed concern that we had not discussed Westat's hypothesis
    concerning why the campaign might have contributed to youth
    experimentation with marijuana. We are unable to draw a conclusion about
    this hypothesis based on Westat's report, nor do we have additional
    information upon which to base an assessment. ONDCP also faults our report
    for not discussing other potential competing explanations for the
    substantial downturn in teen drug use and increase in anti-drug attitudes.
    Although this is beyond the objectives of this report, we note that
    multiple other indicators of youth responsibility also seem to be trending
    in a positive direction at the same time that MTF reports declines in
    youth drug use. For example, from 1991 through 1999, the teen pregnancy
    rate declined by 27 percent and from 1991 through 2002, the teen birth
    rate fell 30 percent. Similarly, the number of juvenile homicides declined
    by 44 percent from 1993 to 2002, and the juvenile violent crime arrest
    rate fell by more than 40 percent from 1994 to 2003. All of these
    trends-including declines in drug use-could be related to broader
    environmental, familial, or other influences. The coincidence of these
    trends with drug use trends indicates that factors other than the campaign
    could be responsible for the decline in drug use and points to the
    necessity of trying to isolate the effects of the campaign, rather than
    relying upon simple correlations.

    Steps Taken to Remedy Potential Problems

    ONDCP states that it has taken extensive "due diligence" steps that are
    briefly acknowledged in our report, but that our report "fails to
    acknowledge the thoroughness of our actions to identify, assess, and
    attenuate any possible negative consequences of the campaign once Westat
    reported the possibility of such an effect." Apart from those actions
    described in Westat's evaluation reports, a full discussion of the steps
    that ONDCP took in response to Westat's interim evaluation reports that
    highlighted the possibility of unintended negative consequences of
    exposure to the campaign on youth initiation of marijuana was not salient
    to our assessing whether Westat took appropriate steps to address the
    evaluation implementation challenges that it faced. However, Westat's
    findings for the period from 2002 to 2004 showed that the campaign also
    was not effective after ONDCP took these steps.

    ONDCP Cites Major Changes in Campaign

    ONDCP states that the campaign is substantially different from what it was
    when the last data were collected by Westat more than 2 years ago. We are
    not in a position to comment on ONDCP's new campaign ("Above the
    Influence"), launched in November 2005, as these current efforts are
    beyond the scope of our report and outside the time frame of the Westat
    data collection. At this time, neither we nor ONDCP have empirical
    information with which to assess this revised campaign. However, Westat's
    evaluation showed that neither the campaign as initially implemented nor
    the redirected campaign implemented after 2002 was effective. Hence,
    although a new and improved campaign may be effective, Westat's findings
    raise concerns about whether any campaign can affect youth drug use,
    especially since the lack of effect does not seem to be related to
    recognition of campaign ads, but rather to subsequent impact on attitudes
    and behaviors. Finally, ONDCP cites the receipt of awards from both the
    advertising and communications industry for its newest campaign. While
    laudable, these awards are not evidence that the new campaign will change
    youth drug attitudes and behavior. Only an independent evaluation can
    assess the current campaign's effectiveness.

    ONDCP Offers an Alternative Explanation for Counterintuitive Results

    ONDCP stated that there is growing research evidence showing that asking
    people a question about their future behavior influences the subsequent
    performance of the behavior in question. ONDCP then indicates that the use
    of a panel design for the Westat study with repeated interviews of youth
    concerning drug attitudes and behaviors might, itself, have resulted in
    increased perceptions that drug use is widely pervasive among youth. If,
    during the course of the Westat study, ONDCP and NIDA, who acted as
    monitor for the study, felt that the study itself-that repeated interviews
    of youth by Westat concerning the campaign and drug attitudes and
    behavior-was resulting in a negative effect, it would have been
    appropriate for them to discontinue the study to avoid potential harm to
    subjects. Although ONDCP raised this issue in its comments to us, neither
    ONDCP nor NIDA mentioned this issue in any of our previous meetings
    specific to this engagement.

    ONDCP Takes Issue with the Timing of Our Review

    ONDCP said that the "long delay" in receiving our assessment of the Westat
    report has prevented it from making progress on the next round of
    evaluation. We note that Westat's draft final report was not made
    available to us until spring 2005 (not 2 years ago as seems to be
    indicated in ONDCP's comments). The volume of reports from the 4 1/2-year
    study, and the complexity of the review required a great deal of time from
    our most skilled social scientists and statisticians. Time was required to
    ensure that our review of the Westat study was both comprehensive and
    correct.

    Points Concerning Our Matter for Congressional Consideration

    ONDCP said that our matter for congressional consideration-that Congress
    consider limiting appropriations until ONDCP is able to provide credible
    evidence of the effectiveness of exposure to the campaign on youth drug
    use outcomes-offers insufficient detail concerning how to demonstrate
    satisfactory evidence of progress and that it was puzzled by our lack of
    recommendations to ONDCP for improving the campaign. Our mandate was to
    assess Westat's evaluation and to draw conclusions about the reliability
    of its findings so that Congress could make decisions about funding for
    the campaign, and developing suggestions for improvements to the media
    campaign itself was beyond our scope. In so doing, we focused on Westat's
    methods and efforts to address challenges in implementing the evaluation.
    Our matter for congressional consideration was intended to allow ONDCP to
    explore a number of approaches to providing credible evidence of campaign
    effectiveness to Congress. Our report clearly indicates that one approach
    is the one applied in the Westat evaluation, which is the focus of this
    report, but we do not want to rule out other approaches. At the same time,
    we acknowledge that providing such evidence is not easy.

    ONDCP Posits Consequences of Further Budget Cuts

    ONDCP states that further budget cuts to the campaign could have
    far-reaching and unfavorable consequences in youth drug use. Given that
    the Westat findings show that the campaign was not having a positive
    impact, we found no evidence that a reduction in campaign advertisements
    would have a negative impact. ONDCP cites the 2005 MTF as an indicator of
    media campaign effectiveness by indicating that the reduction in anti-drug
    messages has resulted in a flattening of 8th graders' perception of risk.
    Again, as ONDCP has indicated, the relationship cannot be assessed with
    MTF because it does not ask respondents about their exposure and reactions
    to the messages of the media campaign that can then be linked to their
    drug-related attitudes and behaviors.

    Failure to continue the media campaign's efforts, according to ONDCP, is
    "raising a white flag to those who favor drug legalization, with the
    expectation that youth drug use soon would begin to rise, reversing years
    of hard-earned positive news." In our view, on the other hand,
    continuation of programs that have been demonstrated not to work diverts
    scarce resources from programs that may be more effective.

    We are sending copies of this report to other interested congressional
    committees and the Director of the Office of National Drug Control Policy.
    We will make copies of the report available to others upon request. In
    addition, the report will be available at no charge on GAO's Web site at
    http://www.gao.gov.

    If you or your staff have any questions about this report, please contact
    either Nancy Kingsbury at 202-512-2700 or by e-mail at KingsburyN@gao.gov
    or Laurie Ekstrand at 202-512-8777 or by e-mail at EkstrandL@gao.gov .
    Contact points from our Offices of Congressional Relations and Public
    Affairs may be found on the last page of this report. Key contributors are
    listed in appendix III.

    Sincerely,

    Nancy Kingsbury, Managing Director Applied Research and Methodology

    Laurie E. Ekstrand, Director Homeland Security and Justice

    Appendix I: Westat's Methods for Addressing Evaluation
    Implementation Issues

    This appendix provides additional details about how Westat's addressed
    evaluation implementation issues related to the coverage of the National
    Survey of Parents and Youth (NSPY), sample attrition, and its analytic
    methods.

    Coverage in the NSPY

    The NSPY was a nationwide household survey of youth aged 9 to 18 and their
    parents. Westat used a dual-frame sampling frame-or list of the members of
    the population from which the sample was ultimately selected. One
    frame-the area frame-consisted of housing units that had been built by
    late 1991; the second frame-the building permit frame-consisted of
    building permits issued between January 1990 and December 1998 for new
    housing.1 Combined, these frames constituted an estimated 98 percent of
    dwelling units nationwide that existed by the end of 1998.

    A household had to meet two criteria in order to be eligible to be
    included in the NSPY sample: It had to (1) contain children within a
    specified age group and (2) be a housing unit that was built before April
    1, 1990, was a mobile home, or was selected from a roster of building
    permits for new housing units issued between January 1990 and December
    1998. To identify households that met these conditions, Westat drew a
    sample of dwelling units and from this sample it screened households to
    determine their eligibility for inclusion in the NSPY, that is, whether a
    household contained children in a specified age group, where the specified
    age groups were children aged 9 through 13, 12 and 13, or 9 through 18.

    According to estimates provided by Westat, after completing enrollment in
    the NSPY-which occurred during waves 1 through 3-the NSPY sample covered
    more than an estimated 95 percent of occupied dwelling units (households)
    nationwide. From its sample of occupied dwelling units, Westat developed
    rosters of households that were believed to contain youth in the target
    age range. At this second stage of sample enumeration, Westat experienced
    a drop-off in the coverage of households that were believed to be eligible
    for inclusion in the sample. The number of eligible households enumerated
    in the NSPY was 30 percent smaller than the number expected from the 1999
    Current Population Survey (CPS) data.

    1Housing units built after 1998 had no chance of selection in either
    sampling frame. Also, a housing unit had no chance of selection if it had
    been built during the 1990s in a jurisdiction where no permit was
    required. Finally, modular housing built during the 1990s was
    inadvertently omitted from the permit sample. Any biases resulting from
    excluding housing units built after 1999 are likely to be small, as they
    constituted a small fraction of all housing units in the NSPY sampling
    frame, and they were accounted for by Westat's poststratification
    adjustments. For example, housing units built after April 1999 accounted
    for an estimated 1.0 percent of all housing units in existence in the time
    period covered by the wave 1 sample.

    According to Westat, coverage losses in the NSPY could have occurred for
    several reasons: (1) because an interviewer may have decided to classify a
    household as an ineligible household rather than as a nonresponding
    household, (2) because the household respondent took cues from the
    screening questions to avoid selection into the sample by giving an
    incorrect answer, or (3) because the doorstep enumeration process was
    considered to be intrusive. Westat reported that it could not conclusively
    rule out the first explanation for coverage losses. However, it undertook
    sample validation procedures that examined whether ineligible households
    in the recruitment waves were misclassified, and it found none. Neither
    Westat nor the National Institute on Drug Abuse reported that
    undercoverage was primarily due to respondents avoiding selection into the
    sample by taking cues from the screening questions and giving incorrect
    answers as a way to avoid selection into the sample. Overall, Westat
    reported that the main reason for undercoverage was the rostering
    component of the survey, which required actual entry into the home, and
    led to "a great many respondents" asking the interviewer to come back at a
    later date, only to repeat the request when the interviewer reappeared.
    Westat inferred that this represented passive refusal to participate.
    Therefore, according to Westat, most of the coverage losses occurred
    during the doorstep screening process in which simple, focused screening
    questions about the composition of the household were used to identify
    households from which to sample eligible youth.

    NSPY and CPS Comparisons of Distributions on Analyzed Variables

    In response to questions from us, Westat provided data that indicated that
    the coverage losses in the NSPY did not result in differences in the
    estimated distributions of population characteristics from the NSPY as
    compared with those estimated from the CPS data. In other words, the
    distributions of characteristics of eligible households with youth
    included in the NSPY were broadly consistent with a variety of
    corresponding distributions from the 1999 Current Population Survey.

    The comparisons of NSPY-estimated populations to CPS-estimated populations
    were based on weighted NPSY estimates, where the weights adjusted for
    nonresponse at the doorstep and household enumeration (roster) stages, and
    the weights also reflected the differential probabilities of retaining a
    household for the NSPY depending on the screener group to which it was
    applied. These weights were calculated prior to Westat's
    poststratification calibration techniques, which brought the estimated
    NSPY population totals into line with the estimated CPS population totals.
    Hence, if upon using the weights based only on the probability of
    selection and nonresponse adjustments, the population characteristics in
    the NSPY differed widely from those derived from the CPS, this would
    constitute evidence of potential bias in the NSPY sample due to
    undercoverage.

    Westat compared NSPY and CPS distributions for each of the three
    enrollment waves of the NSPY (waves 1 through 3) on several variables,
    including the race/ethnicity of the householder and the presence of males
    28 years of age or older, the distribution of eligible households by the
    age of the youth in the household, the age and gender distributions of
    youth, and the age distributions of youth by race and ethnicity. Each of
    these comparisons involved discrete subgroups within the focused
    subpopulation of the NSPY. The largest differences between the NSPY and
    CPS estimates arose in the comparison of the distributions by
    race/ethnicity of household and the presence of a male 28 years of age or
    older in the household. Some of these differences could also arise from
    sampling variance, as both the NSPY and CPS estimates are based on samples
    that are subject to sampling errors. Although Westat did not provide
    sampling errors with the estimates that it provided to us, some of the
    differences in distributions could be apparent, as opposed to real,
    differences, in statistical terms.

    Undercoverage in the NSPY and Other Widely Known and Used Longitudinal Surveys

    Coverage issues are not an uncommon problem with surveys that focus on
    relatively small subpopulations within a larger population, such as
    occurred with the NSPY's focus on youth aged 9 to 18. The NSPY's target
    population of households with youth aged 9 to 18 focused on a
    subpopulation that, according to 1999 CPS data, constituted about 25
    percent of the roughly 104 million households in the United States.

    The estimated extent of undercoverage of eligible youth in the NSPY was
    comparable to the extent of undercoverage in other well-known and widely
    used longitudinal surveys. Both the National Longitudinal Survey of Youth
    (NLSY)-sponsored by the Bureau of Labor Statistics-and the National
    Immunization Survey of Children (NIS)-sponsored by the National
    Immunization Program (NIP) and conducted jointly by the NIP and the
    National Center for Health Statistics of the Centers for Disease Control
    and Prevention-focus on specific subpopulations, and both experienced
    undercoverage that was comparable to that of the NSPY. The 1979 NLSY is a
    nationally representative sample of men and women born in the years 1957
    to 1964 who were ages 14 to 22 when first interviewed in 1979. It had a
    coverage rate of 68 percent. The 1979 NLSY has been widely used and cited
    to examine a wide variety of policy issues. As documented in the National
    Longitudinal Surveys' annotated bibliography, about 3,100 journal
    articles, working papers, monographs, and other research documents have
    been catalogued as having used the 1979 NLSY data. The target population
    for the NIS is children between the ages of 19 and 35 months living in the
    United States at the time of the interview, and it has been conducted
    annually since 1994. The survey involves the selection of a quarterly
    probability sample of telephone numbers, and the coverage has been about
    20 percent lower than estimated by two other benchmark surveys. Survey
    data are used primarily to monitor immunization coverage in the preschool
    population in the nation and to provide national, state, and selected
    urban area estimates of vaccination coverage rates for these children.

    Sample Attrition across NSPY Interview Rounds

    In the NSPY, respondents initially recruited into the sample were to be
    tracked for three additional survey rounds that covered about a 3-year
    period following the recruitment round. By the final survey round of the
    NSPY, the cumulative response rate-the percentage of youth or parents in
    eligible households that completed all four interviews-reached between 50
    percent and 55 percent. These cumulative response rates after four survey
    rounds were determined largely by the response rates during the enrollment
    waves, as postenrollment, Westat was able to track, contact, determine
    eligibility for reinterview, and complete interviews for between 82
    percent and 94 percent of previously interviewed respondents between two
    successive interview waves. The response rates achieved for the first
    three survey waves-the enrollment waves-were generally similar.
    Specifically, about 74 percent to 75 percent of the dwelling units
    determined to be eligible for the survey in waves 1 through 3 completed
    the household enumeration (or rostering of youth). After obtaining consent
    to conduct interviews from parents and youth, interviewers completed
    extended interviews-that is, completed the full NSPY questionnaire-with
    about 91 percent of the sampled youth in each of waves 1 through 3. Among
    sampled parents, about 88 percent gave consent and completed extended
    interviews in the enrollment waves. (See table 2.)

    Table 2: NSPY Survey Rounds and Response Rates, Sampled and Surveyed Youth

    Rounds and stages of sampling                             Survey waves
    Round 1: enrollment waves                             Wave 1 Wave 2 Wave 3
          o  Percentage of sampled dwelling units for which   95.1%  95.7%  95.5%
          eligibility was determined
          o  Percentage of eligible dwelling units            74.4%  74.6%  75.3%
          completing household roster
          o  Percentage of youth completing interview         90.3%  91.9%  91.2%
          o  Cumulative (overall) response rate, enrollment   63.8%  65.5%  65.5%
          waves
    Round 2: first follow-up                                  Wave 4    Wave 5
          o  Percentage of dwelling units (from prior wave)       94.2%     92.9%
          refielded for follow-up
          o  Percentage of refielded dwelling units for           86.8%     93.8%
          which eligibility was determined
          o  Percentage of youth completing interview             93.5%     93.6%
          o  Cumulative (overall) response rate                   54.1%     58.4%
          o  Follow-up (conditional) longitudinal response        82.2%     88.8%
          rate
    Round 3: second follow-up                                 Wave 6    Wave 7
          o  Percentage of dwelling units (from prior wave)       85.1%     89.8%
          refielded for follow-up
          o  Percentage of refielded dwelling units for           93.1%     92.8%
          which eligibility was determined
          o  Percentage of youth completing interview             94.7%     93.8%
          o  Cumulative (overall) response rate                   53.1%     56.0%
          o  Follow-up (conditional) longitudinal response        93.4%     91.6%
          rate
    Round 4: third follow-up                                  Wave 8    Wave 9
          o  Percentage of dwelling units (from prior wave)       78.7%     83.5%
          refielded for follow-up
          o  Percentage of refielded dwelling units for           95.9%     94.8%
          which eligibility was determined
          o  Percentage of youth completing interview             94.0%     94.3%
          o  Cumulative (overall) response rate                   50.2%     53.4%
          o  Follow-up (conditional) longitudinal response        92.4%     93.4%
          rate

    Source: Westat, June 2005.

    Across the three follow-up rounds of the NSPY, Westat achieved between an
    82 percent and a 94 percent longitudinal response rate. Follow-up required
    that respondents be tracked over time and across places, as persons
    enrolled in the sample could move, and their eligibility for a follow-up
    interview had to be determined. For example, youth who turned 19 years of
    age between survey rounds would no longer be eligible for reinterview, as
    they were beyond the target age of the campaign. Efforts to track
    individuals prior to the second survey round included verifying address
    change information with the U.S. Postal Service and obtaining location
    information from a national database company. Westat obtained updated
    location information from these sources, and telephone interviewers placed
    calls to these households to verify the identity of respondents. According
    to Westat, a high proportion of the households that moved were contacted
    and respondents verified their new addresses. During the third and fourth
    survey rounds, Westat used procedures to track and verify addresses that
    were similar to those used to track respondents from the first to second
    survey rounds, although Westat modified these procedures as necessary. The
    key eligibility requirement for youth for a follow-up interview was the
    youth had to be 18 years of age or younger at the time of the interview.

    For the first follow-up round-waves 4 and 5-Westat located individuals and
    determined eligibility for 92 percent of the youth and 92 percent of the
    parents who completed an initial interview during the first round of the
    survey-that is, in waves 1, 2, and 3, and of these youth who were still
    eligible, 94 percent completed an interview. Among parents from the first
    round who were tracked and determined to be eligible in the second round,
    92 percent completed a second round interview. In the third and fourth
    survey rounds of the NSPY, between 96 percent and 97 percent of the youth
    and parents who had completed prior round surveys were tracked and
    determined to be eligible, and of these, the youth response rates were 96
    percent and the parent rates were 95 percent.

    Comparisons of Respondents and Nonrespondents across NSPY Survey Waves

    Even with the relatively high follow-up response rates that Westat
    achieved, it is possible that respondents could differ from nonrespondents
    in follow-up rounds, and if so, the NSPY estimates of the effects of
    exposure on outcomes would be biased. Westat provided data that compared
    nonrespondents to the respondents across the three enrollment waves,
    indicating that with some differences, nonrespondents were generally
    similar to respondents with respect to characteristics that might affect
    survey outcomes. Nonrespondents were compared to respondents on gender,
    age at interview, whether both parents were in the household, the number
    of youth in the household, the type of household dwelling, and the type of
    area in which the household was located. For example, apart from the three
    differences below, nonrespondents and respondents were similar in
    characteristics across survey waves: In the three enrollment waves,
    nonrespondents were proportionately older youth than respondents; in waves
    2 and 3, there were proportionately more youth living in cities among
    nonrespondents than respondents; and in wave 1, there were proportionately
    more youth in the building permit sample among nonrespondents than
    respondents.

    Differences in Sampling Methodologies between NSPY and MTF

    Westat compared estimates of drug-use prevalence from the NSPY data with
    those obtained from other national surveys such as Monitoring the Future
    (MTF). While the NSPY estimates of marijuana use prevalence differ over
    some periods covered by the NSPY from those derived from the MTF survey of
    youth in school, differences between the two surveys' sampling frames and
    methodologies mean that direct comparisons between the two surveys must be
    made with caution and must take the methodological differences into
    account. Specifically, MTF showed a decline in marijuana use for some
    teenage groups during the 2000 to 2002 period, while the NSPY showed the
    increases reported above. However, the difference in drug use rates
    reported from the two surveys could plausibly arise from differences in
    the sampling frames. The MTF sampling frame covers only youth who are in
    school and not those who drop out of school, who are truant on the survey
    day, or who are 17- and 18-year-olds who have graduated from high school.
    To the extent that high school dropouts and truants have more involvement
    with drugs than those who stay in school, the MTF estimates of drug use
    may underrepresent drug use among all youth of high school age. By
    comparison, the NSPY household survey includes youth who are not enrolled
    in school in its sampling frame. To the extent that dropping out of high
    school is correlated with drug use, and given that dropouts are excluded
    from the MTF sampling frame, differences in drug use between MTF and NSPY
    could reflect the fact that youth enrolled in high school reported drug
    use at different rates from all youth in the general population covered by
    the NSPY, which would include dropouts who may be at higher risk of using
    drugs.

    The Capacity of the NSPY to Detect Reasonably Small Effects

    One challenge in designing surveys to evaluate changes in outcomes as the
    result of an intervention lies in selecting a sample with sufficient power
    to detect differences between groups-including the same individuals at two
    points in time-or significant associations among variables, such as
    between levels of exposure to the campaign and outcomes. Sample size is a
    major factor determining a study's power to detect differences, and while
    larger sample sizes will generally allow researchers to detect smaller
    differences over time, as the size and power of a sample to detect changes
    increases, so too generally does its cost.

    In consultation with the National Institute on Drug Abuse (NIDA), Westat
    chose to compute power for analyses of annual change in a prevalence
    statistic-that is, change in the percentage of a population that reported
    an outcome. For purposes of its power analysis, Westat chose to assume
    different baseline prevalences for parents and for youth of all ages and
    to assume that the study should be able to detect reliably declines of
    specified sizes. For example, for youth of all ages, Westat assumed a
    baseline prevalence of 10 percent and determined the power of its sample
    for detecting a minimum downswing in an outcome-such as past-month drug
    use-of 2.3 percentage points over a year.2 The power of the sample to
    detect this difference was well within conventional power criteria.3

    As reported above, the sizes of differences that Westat's sample could
    detect were consistent with the Office of National Drug Control Policy's
    (ONDCP) goals for the campaign. In early meetings on the design of the
    evaluation of the media campaign, ONDCP officials reported that ONDCP had
    a specific Performance Measures of Effectiveness (PME) system and that the
    campaign was embodied within the first goal of the National Drug Strategy,
    which was to "educate and enable America's youth to reject illegal drugs
    as well as the use of alcohol and tobacco." Under this goal, ONDCP's PME
    proposed targets for reducing the prevalence of past-month use of illicit
    drugs and alcohol among youth from a 1996 base year: by 2002, reduce this
    prevalence by 20 percent, and by 2007, reduce it by 50 percent. ONDCP
    officials further identified specific targets for the media campaign,
    again with respect to a base year of 1996: by 2002, increase to 80 the
    percentage of youth who perceive that regular use of illicit drugs,
    alcohol, and tobacco is harmful; and by 2002, increase to 95 the
    percentage of youth who disapprove of illicit drug, alcohol, and tobacco
    use. To achieve a goal of 80 percent of youth who perceive that regular
    use of marijuana is harmful would require increasing the 1996 baseline
    percentage of youth perceiving marijuana as harmful from 60 percent, as
    measured by MTF, or by about 3.3 percentage points per year from 1996 to
    2002. Westat's sample had sufficient power to detect this amount of annual
    change in youth attitudes.

    The power of the NSPY to detect changes in outcomes due to exposure to the
    campaign also presumes that it was possible to accurately measure and
    characterize exposure to the campaign by the reported number of
    advertisements recalled by respondents. While the general question of how
    exposure to advertisements affected respondents was beyond the scope of
    the evaluation, if by exposure is meant a recognition-based task-or
    encoded exposure-then the NSPY measures of exposure can be viewed as
    valid. According to communications researchers, often what is of interest
    to campaign planners and evaluators is whether the presentation of
    campaign content generates at least a memory trace in individuals. At this
    point, a potential audience member can be said to have engaged the
    campaign's presentation in a meaningful sense, and this is what is meant
    by encoded exposure. To measure exposure to the campaign for both youth
    and parents, NSPY interviewers asked respondents about their recall of
    specific current or very recent television and radio advertisements.4

    2The power to detect differences for upswings in prevalence would depend
    upon the baseline level. However, the power to detect an upswing from a
    baseline of 90 percent of youth would be exactly the same as that for
    detecting a downswing from a 10 percent baseline.

    3Specifically, the minimum detectable difference for wave-to-wave changes
    was at least 80 percent using a one-sided hypothesis test at the 0.05
    level.

    There was variation in recall of advertisements by both youth and parent
    respondents, and this type of variation is needed in order to examine
    associations between levels of exposure and outcomes. For example, for the
    entire campaign, youth reported a median of 12 exposures per month, and
    76.7 percent reported 4 or more exposures per month. Comparatively few
    youth-about 6 percent-reported less than 1 exposure per month. Youth
    recall of specific exposure also varied, as 41.2 percent of youth reported
    12 or more television exposures per month throughout the campaign while
    reporting a median of 4.4 exposures to television advertisements.
    Additionally, Westat's measures of exposure and outcomes have demonstrated
    sensitivity to detect favorable campaign effects among parents.

    Westat's test for associations between exposure and outcomes-the gamma
    coefficient-was an ordinal test statistic for whether two variables (e.g.,
    exposure and marijuana use initiation) have a montonic, but not
    necessarily a linear, relationship. Therefore, were there nonlinear
    relationships, its test would have allowed for them. Finally, nonrandom
    measurement error in the measure of exposure is unlikely to have biased
    estimates of campaign effects, as if the nonrandom measurement error were
    constant, it would not affect measures of association, and if it was not
    constant, it would be addressed by Westat's statistical methods.

    4Each respondent was presented ads that had been broadcast nationally in
    the 2 calendar months prior to the interview.

    Westat Methods to Measure Outcomes

    Westat measured a variety of outcomes for youth and parents and took steps
    to ensure that the measures were consistent with existing research. The
    youth questionnaires included numerous questions that were designed to
    measure exposure to the campaign advertisements and other anti-drug
    messages. The youth question domains included exposure propensity to
    media; current and past use of tobacco, alcohol, marijuana, inhalants, and
    Ecstasy; past discussions with and communication of anti-drug messages
    from parents and friends; expectations of others about respondent's drug
    use; knowledge and beliefs about the positive and negative consequences of
    drug use; exposure to campaign messages; family and peer factors; personal
    factors; and demographic information. Westat used two separate
    questionnaires for youth of different ages; one questionnaire was used for
    children (aged 9 to 11) and another one was used for teens (aged 12 to
    18).

    The NSPY parent questionnaire also included numerous questions that were
    intended to measure parents' exposure to the campaign's messages and other
    anti-drug messages. The question domains for parents included media
    consumption; past discussions with child about drug attitudes and
    avoidance strategies; past child monitoring behaviors; self-efficacy of
    discussing drugs with child and monitoring of child's actions; belief that
    the child is at risk of drug use; belief that drug use has bad
    consequences; exposure to the campaign's advertising, including brand
    recognition; parent's own current and past use of tobacco, alcohol, and
    drugs; and demographic information.

    Westat followed generally accepted procedures in developing the survey
    instruments for the NSPY by using information from a prototype prepared by
    NIDA and using information from other surveys that addressed youth drug
    use and prevention. Prior to the phase III evaluation, and in preparation
    for the NSPY, NIDA convened an expert panel to assist in the development
    of the youth and parent questionnaires. The panel, which consisted of
    experts in adolescent drug use prevention and parenting behaviors, drafted
    NSPY survey questionnaires for children, teens, and parents, and NIDA
    shared these prototypes with Westat at the beginning of Westat's
    evaluation contract. In developing the final questionnaire for the NSPY,
    Westat created a questionnaire development team consisting of evaluation
    experts. In developing the final NSPY questionnaires, the Westat team
    reviewed NIDA's prototype and other surveys.

    Westat measured youth drug use by self-reported data on use. We have
    previously cautioned about limitations associated with self-reported data
    on youth drug use.5 Additionally, the National Research Council (NRC) of
    the National Academy of Sciences also has pointed out limitations
    associated with self-reported drug use in national surveys such as the
    National Survey of Drug Use and Health (NSDUH) and MTF.6 As NRC has
    pointed out, while self-reported data on drug use may have limitations for
    estimating the actual levels of use at a particular point in time, they
    may not suffer from these same limitations when they are used to assess
    changes in use over time, unless there is reason to believe that attitudes
    about drug use change in ways that affect respondents' willingness to
    honestly report drug use, or stigma.

    Specifically, if there is a stigma associated with self-reporting drug
    use, that stigma may affect the levels of use reported, as some have
    argued that the propensity of respondents to give valid responses may be
    affected by social pressures. In particular, the incentive to give false
    negative reports may increase over time if drug use becomes increasingly
    perceived as harmful or socially unacceptable. Using data from NSDUH and
    MTF, NRC showed an inverse relationship between the percentages of
    respondents who either disapproved of illegal drug consumption or
    perceived it to be harmful. Thus, as stigma increased, self-reported drug
    use decreased. As NRC cautioned, one could interpret this relationship as
    indicating that changes in stigma are associated with changes in invalid
    reporting, or as stigma increases, false negative reports increase, rather
    than necessarily indicating that as stigma increases, drug use decreases.

    The NRC analysis leads to two inferences: First, if social stigma remains
    constant over time, changes in the propensity to give valid responses
    would be unaffected and estimates of change in self-reported drug use
    would not be biased by social stigma. For the evaluation results, this
    would imply that its measures of changes in self-reported drug use would
    provide valid measures of changes in use, so long as factors other than
    stigma did not affect the propensity to self-report use. Second, if the
    social stigma associated with reporting drug use is inversely related to
    disapproval of illicit drug use or increased perceptions that it is
    harmful, then the estimates of self-reported drug use are likely to
    decrease as a result of the stigma. According to results from the
    evaluation, trends in youth attitudes and beliefs about illicit drugs
    changed significantly over the entire campaign in a direction that was
    favorable to the campaign. Specifically, the trends in youth attitudes and
    beliefs about illicit drug use meant that youth were more likely to
    believe, as the campaign went on, that use of illicit drugs was likely to
    have negative consequences. Alternatively, the social stigma associated
    with drug use increased over time. If the relationship between stigma and
    reporting that NRC found held and applied to the data in the evaluation of
    the campaign, this would imply that the increased stigma associated with
    drug use would lead to decreases in self-reports of drug use over time.

    5GAO: Drug Use Measurement: Strengths, Limitations, and Recommendations
    for Improvement, GAO/PEMD-93-18 (Washington, D.C.: June 25, 1993).

    6National Research Council, Informing America's Policy on Illegal Drugs:
    What We Don't Know Keeps Hurting Us. National Academy Press, Washington,
    D.C.: 2001.

    Westat�s Analytic Methods

    To control for the many factors that could have influenced both exposure
    and outcomes independently of, or in conjunction with, the campaign,
    Westat used propensity scoring methods to match individuals based on
    numerous measured attributes and to create groups of individuals who
    differed on their underlying propensity to be exposed to different levels
    of campaign advertisements. A propensity score is a weighted sum of the
    individual effects of variables in a model that predicts the likelihood of
    exposure to campaign messages. Westat's propensity scoring methods
    resulted in the creation of groups of individuals who were statistically
    similar on exposure propensities. These groups can be considered as
    statistical analogues to randomly assigning individuals to different
    levels of exposure. After creating these groups, Westat then analyzed
    outcomes between the groups having different propensities to be exposed to
    campaign messages.

    Westat used ordinal logit models to estimate the chances of being exposed,
    where exposure was measured alternatively as a three- or four-level
    variable-e.g., low, medium, or high exposure.7 Westat used a myriad of
    variables to predict exposure levels in both the youth and parent models.
    For example, in the youth models, the propensity score models included
    measures of demographic attributes, educational attainment and educational
    aspiration, family and parent background, parent consumption of television
    and other media, income and employment, reading habits, Internet usage,
    location of residence in urban areas, among other variables. After
    estimating models, Westat also assessed the balance of variables in its
    propensity models. For propensity models to remove the effects of
    confounding variables from the association between exposure and response,
    it is necessary that the population means of the confounder variables not
    vary across exposure levels. If a confounder is successfully balanced,
    then it will have the same theoretical effect across all exposure levels.

    7Propensity score methods have been demonstrated to be robust against bias
    associated with the specification of incorrect functional forms-e.g.,
    linear rather than quadratic- of variables.

    The net result of the propensity scoring models is to provide each
    individual with a score that reflects the individual's propensity to
    recall advertisements based upon a weighted sum of all of the variables in
    the model. Therefore, while two individuals may differ on the likelihood
    that a particular variable affects their chances of being exposed to
    messages or on their levels of a certain variable-such as age or
    education-they could be similar in their overall propensity to be exposed
    to campaign messages if the differential effects of any individual
    variables sum to the same total propensity.

    In order for the results of propensity methods to be valid, it is
    important that the propensity scoring models include all relevant
    variables that could otherwise explain differences in both exposure and
    outcomes. Propensity score models can adjust only for confounding
    variables that are observed and measured. In other words, they are built
    upon the assumption that all relevant variables are measured and
    controlled for. If an important variable is omitted from the propensity
    model, the results of analyses may be affected. Westat made reasonable
    attempts to identify and control for a variety of confounding variables,
    include them in its models, and reduce bias.

    Appendix II: Comments from the Office of National Drug Control Policy

    Appendix III: GAO Contacts and Staff Acknowledgments

    GAO Contacts

    Nancy Kingsbury, 202-512-2700
    Laurie E. Ekstrand, 202-512-8777

    Acknowledgments

    In addition to the contacts named above, contributors to this report
    included David P. Alexander, Billy Commons, James Fields, Kathryn Godfrey,
    Mary Catherine Hult, Jean McSween, Karen V. O'Conor, Mark Ramage, William
    J. Sabol, Barry J. Seltser, and Douglas Sloane.

    (440408)

    GAO's Mission

    The Government Accountability Office, the audit, evaluation and
    investigative arm of Congress, exists to support Congress in meeting its
    constitutional responsibilities and to help improve the performance and
    accountability of the federal government for the American people. GAO
    examines the use of public funds; evaluates federal programs and policies;
    and provides analyses, recommendations, and other assistance to help
    Congress make informed oversight, policy, and funding decisions. GAO's
    commitment to good government is reflected in its core values of
    accountability, integrity, and reliability.

    Obtaining Copies of GAO Reports and Testimony

    The fastest and easiest way to obtain copies of GAO documents at no cost
    is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts newly
    released reports, testimony, and correspondence on its Web site. To have
    GAO e-mail you a list of newly posted products every afternoon, go to
    www.gao.gov and select "Subscribe to Updates."

    Order by Mail or Phone

    The first copy of each printed report is free. Additional copies are $2
    each. A check or money order should be made out to the Superintendent of
    Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
    copies mailed to a single address are discounted 25 percent. Orders should
    be sent to:

    U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
    D.C. 20548

    To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
    512-6061

    To Report Fraud, Waste, and Abuse in Federal Programs

    Contact:

    Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
    Automated answering system: (800) 424-5454 or (202) 512-7470

    Congressional Relations

    Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
    Government Accountability Office, 441 G Street NW, Room 7125 Washington,
    D.C. 20548

    Public Affairs

    Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800 U.S.
    Government Accountability Office, 441 G Street NW, Room 7149 Washington,
    D.C. 20548

    www.gao.gov/cgi-bin/getrpt? GAO-06-818 .

    To view the full product, including the scope and methodology, click on the link above.

    For more information, contact Laurie Ekstrand, (202) 512-8777,
    ekstrandl@gao.gov.

    Highlights of GAO-06-818 , a report to the Chairman and Ranking Member,
    Subcommittee on Transportation, Treasury, the Judiciary, Housing and Urban
    Development, and Related Agencies, Committee on Appropriations, U.S.
    Senate

    August 2006

    ONDCP MEDIA CAMPAIGN

    Contractor's National Evaluation Did Not Find That the Youth Anti-Drug
    Media Campaign Was Effective in Reducing Youth Drug Use

    Between 1998 and 2004, Congress appropriated over $1.2 billion to the
    Office of National Drug Control Policy (ONDCP) for the National Youth
    Anti-Drug Media Campaign. The campaign aimed to prevent the initiation of
    or curtail the use of drugs among the nation's youth. In 2005, Westat,
    Inc., completed a multiyear national evaluation of the campaign.

    GAO has been mandated to review various aspects of the campaign, including
    Westat's evaluation which is the subject of this report.

    Applying generally accepted social science research standards, GAO
    assessed (1) how Westat provided credible support for its findings and
    Westat's findings about (2) attitudes, beliefs, and behaviors of youth and
    parents toward drug use and (3) youth self-reported drug use.

    What GAO Recommends

    Given that Westat's evaluation stated the campaign did not reduce youth
    drug use nationally, Congress should consider limiting appropriations for
    the campaign, beginning in the 2007 fiscal year budget until ONDCP
    provides credible evidence of a media campaign approach that effectively
    prevents and curtails youth drug use. ONDCP's written comments on our
    report generally disagreed with the findings. Specifically, ONDCP does not
    believe the Westat findings reflect the campaign's effectiveness. We
    believe the Westat study is sound.

    GAO's review of Westat's evaluation reports and associated documentation
    leads to the conclusion that the evaluation provides credible evidence
    that the campaign was not effective in reducing youth drug use, either
    during the entire period of the campaign or during the period from 2002 to
    2004 when the campaign was redirected and focused on marijuana use. By
    collecting longitudinal data-i.e., multiple observations on the same
    persons over time-using generally accepted and appropriate sampling and
    analytic techniques, and establishing reliable methods for measuring
    campaign exposure, Westat was able to produce credible evidence to support
    its findings about the relationship between exposure to campaign
    advertisements and both drug use and intermediate outcomes. In particular,
    Westat was able to demonstrate that its sample was not biased despite
    sample coverage losses, maintained high follow-up response rates of
    sampled individuals to provide for robust longitudinal analysis,
    established measures of exposure that could detect changes in outcomes on
    the order of magnitude that ONDCP expected for the campaign and that could
    reliably measure outcomes, and used sophisticated statistical methods to
    isolate causal effects of the campaign.

    Westat's findings on the effects of exposure on intermediate
    outcomes-theorized precursors of drug use-were mixed. Specifically,
    although sampled youth and parents' recall of campaign advertisements
    increased over time, they had good impressions of the advertisements, and
    they could identify the specific campaign messages, exposure to the
    advertisements generally did not lead youth to disapprove of using drugs
    and may have promoted perceptions among exposed youth that others' drug
    use was normal. Parents' exposure to the campaign led to changes in
    beliefs about talking about drug use with their children and the extent to
    which they had these conversations with their children. However, exposure
    did not appear to lead to increased monitoring of youth. Moreover, the
    evaluation was unable to demonstrate that changes in parental attitudes
    led to changes in youth attitudes or behaviors toward drug use.

    Westat's evaluation indicates that exposure to the campaign did not
    prevent initiation of marijuana use and had no effect on curtailing
    current users' marijuana use, despite youth recall of and favorable
    assessments of advertisements. Although general trend data derived from
    the Monitoring the Future survey and the Westat study show declines in the
    percentage of youth reportedly using marijuana from 2002 to 2004, the
    trend data do not explicitly take into account exposure to the campaign,
    and therefore, by themselves, cannot be used as evidence of effectiveness.
    In Westat's evaluation of relationships between exposure and marijuana
    initiation the only significant finding was of small unfavorable effects
    of the campaign exposure on marijuana initiation during some periods of
    data collection and in some subgroups.
    *** End of document. ***