Bureau of Justice Statistics: Quality Guidelines Generally	 
Followed for Police-Public Contact Surveys, but Opportunities	 
Exist to Help Assure Agency Independence (30-MAR-07, GAO-07-340).
                                                                 
The Bureau of Justice Statistics (BJS), a statistical agency of  
the Department of Justice's Office of Justice Programs, produces 
a recurring national Police-Public Contact Survey documenting	 
contacts between the police and the public, including instances  
involving the use or threat of force by police. BJS issues public
reports and sometimes press releases from survey results. For	 
reports and a press release issued from the 1999 and 2002 surveys
(the most recent available), GAO reviewed (1) the extent to which
BJS followed quality guidelines to ensure the accuracy and	 
integrity of its survey-related products, and (2) factors that	 
affected whether and how BJS followed available guidelines. GAO  
reviewed applicable federal data quality guidelines, policy and  
procedure documents, and interviewed current and former officials
familiar with BJS.						 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-340 					        
    ACCNO:   A67566						        
  TITLE:     Bureau of Justice Statistics: Quality Guidelines	      
Generally Followed for Police-Public Contact Surveys, but	 
Opportunities Exist to Help Assure Agency Independence		 
     DATE:   03/30/2007 
  SUBJECT:   Data integrity					 
	     Government information dissemination		 
	     Internal controls					 
	     Quality assurance					 
	     Statistical data					 
	     Statistical methods				 
	     Surveys						 
	     DOJ Police Public Contact Survey			 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-340

   

     * [1]Results in Brief
     * [2]Background

          * [3]Overview of BJS Mission and Organization
          * [4]The BJS Police-Public Contact Survey
          * [5]Quality Guidelines for Statistical Products

     * [6]BJS Followed Nearly All Available Quality Guidelines to Help

          * [7]BJS Fully Followed All Applicable Quality Guidelines for the
          * [8]BJS Fully Followed 7 of 10 Applicable Data Quality Guideline

     * [9]Differing Views on Applicability of Guidelines Raised Questi

          * [10]BJS Interpreted Guidelines as Not Applying to Press Releases
          * [11]The Roles Played by Certain Outside Noncareer Appointees in

     * [12]Agency Comments and Our Evaluation

          * [13]Review, Approval, and Dissemination Process for Reports Issu
          * [14]Review, Approval, and Dissemination Process for Press Releas
          * [15]The Scope of the Police-Public Contact Survey Has Expanded w
          * [16]The Police-Public Contact Survey Methodology Has Remained Co

     * [17]GAO Contacts
     * [18]Acknowledgments
     * [19]GAO's Mission
     * [20]Obtaining Copies of GAO Reports and Testimony

          * [21]Order by Mail or Phone

     * [22]To Report Fraud, Waste, and Abuse in Federal Programs
     * [23]Congressional Relations
     * [24]Public Affairs

Report to Congressional Requesters

United States Government Accountability Office

GAO

March 2007

BUREAU OF JUSTICE STATISTICS

Quality Guidelines Generally Followed for Police-Public Contact Surveys,
but Opportunities Exist to Help Assure Agency Independence

GAO-07-340

Contents

Letter 1

Results in Brief 3
Background 6
BJS Followed Nearly All Available Quality Guidelines to Help Ensure
Accuracy and Integrity of Products Issued from Police-Public Contact
Surveys 11
Differing Views on Applicability of Guidelines Raised Questions about
Clarity, while Roles of Certain Noncareer Appointees during Product
Issuance Process Affected BJS's Independence 14
Agency Comments and Our Evaluation 23
Appendix I Objectives, Scope, and Methodology 28
Appendix II Applicable Quality Guidelines for Statistical Products Issued
by the Bureau of Justice Statistics and Other Agencies 32
Appendix III Press Release Issued in 2001 Based on BJS's 1999
Police-Public Contact Survey 38
Appendix IV BJS Processes for the Review, Approval, and Dissemination of
Police-Public Contact Survey Products 40
Appendix V Changes in the Scope and Methodology of the Police-Public
Contact Surveys 43
Appendix VI Comments from the Department of Justice 47
Appendix VII GAO Contact and Acknowledgments 52

Tables

Table 1: Public Products Issued Based on 1999 and 2002 Police-Public
Contact Surveys 8
Table 2: Guidelines That BJS Fully Followed for the Reports Issued from
Its 1999 and 2002 Police-Public Contact Surveys 11
Table 3: Noncareer Appointees' Roles in the Review, Approval, and
Dissemination of BJS Press Releases 20
Table 4: Extent to Which BJS Followed Available Quality Guidelines for
1999 and 2002 Police-Public Survey Products 33
Table 5: Police-Public Contact Survey (PPCS) Methodologies for the Four
Surveys: 1996, 1999, 2002, and 2005 45

Figures

Figure 1: Office of Justice Programs Organizational Structure 7
Figure 2: Review, Approval, and Dissemination Process for BJS Survey
Reports 40
Figure 3: Review, Approval, and Dissemination Process for BJS Survey Press
Releases 41

Abbreviations

BJS Bureau of Justice Statistics
CATI computer-assisted telephone interview
DOJ Department of Justice
FBI Federal Bureau of Investigation
NRC National Research Council
OCOM Office of Communications
OJP Office of Justice Programs
OMB Office of Management and Budget
PAPI paper-and-pencil in-person face-to-face interview

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office
Washington, DC 20548

March 30, 2007 March 30, 2007

Congressional Requesters:

The Bureau of Justice Statistics (BJS), a statistical agency within the
Department of Justice's Office of Justice Programs, collects, analyzes,
publishes, and disseminates information on crime, criminal offenders,
victims of crime, and the operation of justice systems at all levels of
government. A critical aspect of the agency's mission is to produce
impartial, timely, and accurate statistics. An ongoing BJS data collection
activity is the Police-Public Contact Survey--one of many statistical
surveys the agency administers. This recurring, nationally representative
survey of the public is designed to document contacts between the police
and the public and the outcomes of those contacts, including instances
involving the use of force or the threat of force by the police. BJS
issues reports to the public based on the results of this survey, and
these may be accompanied by a press release. To date, BJS has conducted
the survey four times--a pilot survey was conducted in 1996, and more
extensive surveys were done in 1999, 2002, and 2005.^11The Bureau of
Justice Statistics (BJS), a statistical agency within the Department of
Justice's Office of Justice Programs, collects, analyzes, publishes, and
disseminates information on crime, criminal offenders, victims of crime,
and the operation of justice systems at all levels of government. A
critical aspect of the agency's mission is to produce impartial, timely,
and accurate statistics. An ongoing BJS data collection activity is the
Police-Public Contact Survey--one of many statistical surveys the agency
administers. This recurring, nationally representative survey of the
public is designed to document contacts between the police and the public
and the outcomes of those contacts, including instances involving the use
of force or the threat of force by the police. BJS issues reports to the
public based on the results of this survey, and these may be accompanied
by a press release. To date, BJS has conducted the survey four times--a
pilot survey was conducted in 1996, and more extensive surveys were done
in 1999, 2002, and 2005.

The Police-Public Contact Survey involves conducting interviews in person
and by telephone with tens of thousands of individuals to obtain
information about their encounters with police officers. The most
frequently reported reason for such contacts has involved traffic stops.
In this context, interviewers have gathered self-reported data from
motorists on such factors as the reason for the traffic stop and what
occurred after the traffic stop--for example, whether the individual was
ticketed, handcuffed, or searched, and whether the police used or
threatened to use force. The Police-Public Contact Survey involves
conducting interviews in person and by telephone with tens of thousands of
individuals to obtain information about their encounters with police
officers. The most frequently reported reason for such contacts has
involved traffic stops. In this context, interviewers have gathered
self-reported data from motorists on such factors as the reason for the
traffic stop and what occurred after the traffic stop--for example,
whether the individual was ticketed, handcuffed, or searched, and whether
the police used or threatened to use force.

In the spring of 2005, a disagreement arose between the then-director of
BJS and an official outside of BJS but within the Department of Justice
about the statistical content of a draft press release concerning the
results of the 2002 Police-Public Contact Survey (which at the time was
the most recent survey to have been completed), and no press release was
ultimately issued from this survey. In the spring of 2005, a disagreement
arose between the then-director of BJS and an official outside of BJS but
within the Department of Justice about the statistical content of a draft
press release concerning the results of the 2002 Police-Public Contact
Survey (which at the time was the most recent survey to have been
completed), and no press release was ultimately issued from this survey.

^1 At the time of our review, BJS had not issued any products from the
2005 survey.

As a result of this incident, some members of Congress, the research
community, and the media raised questions about the methods and procedures
BJS follows in reviewing and approving reports and other products based on
the agency's statistical information, and the extent of BJS's independence
as a statistical agency.

In light of the questions raised about BJS's ability to meet its
self-stated mission as a statistical agency committed to maintaining
public trust and confidence in its statistics, we reviewed what processes
are in place at BJS to help the agency ensure the integrity and
independence of its statistical studies. This report specifically
addresses the following two questions for the 1999 and 2002 Police-Public
Contact Surveys, the two surveys for which products had been issued as of
February 2007: (1) To what extent did BJS follow guidelines to help ensure
the accuracy and integrity of the review, approval, and dissemination of
reports and press releases based on its surveys? (2) What key factors
affected whether and how BJS followed available guidelines?

To assess the extent to which BJS followed guidelines to help ensure the
accuracy and integrity of its product issuance processes for the 1999 and
2002 Police-Public Contact Surveys, we reviewed quality guidelines
pertaining to the review, approval, and dissemination of written products
issued by the National Research Council (a private, nonprofit institution
that is a component of the congressionally chartered National Academies),
the Department of Justice, the Office of Justice Programs, and BJS itself.
The guidelines that we reviewed covered the period between February 2001,
when the first product based on the 1999 survey was issued, and June 2006,
when the most recent product based on the 2002 survey was issued. We
initially identified a total of 63 guidelines from the four organizations.
Because the organizations had a number of similar guidelines, we
eliminated the overlap by reducing the list to 24 nonduplicative
guidelines. To determine the extent to which BJS followed each guideline,
we developed a data collection instrument for recording information on
whether BJS fully, partially, or did not at all follow the guideline. For
BJS's 1999 and 2002 Police-Public Contact Survey reports and the one press
release that was issued, we asked BJS to assess the extent to which it
followed each of 15 guidelines, and provide us with documentary evidence
supporting its assessment. We completed the instrument for the remaining 9
guidelines for which we already had sufficient information from BJS and
Office of Justice Programs documents and interviews, and asked BJS to
confirm or not confirm our assessments. Two GAO analysts reviewed BJS's
responses and all available supporting documentary and testimonial
evidence, and made a final determination on the extent to which we
believed BJS followed each guideline. We provided our findings to BJS for
review and comment. We also reviewed guidelines issued by the Office of
Management and Budget (OMB), but did not specifically assess BJS's
practice with respect to following OMB's guidelines, because OMB issued
governmentwide policy and procedural guidance to federal agencies that
called for agencies to develop their own agency-specific guidelines. To
determine key factors that affected whether and how BJS followed
applicable guidelines, we reviewed BJS procedural documents and
interviewed both current and former officials involved in the review,
approval, and dissemination of BJS products, including past and present
BJS directors and assistant attorneys general within the Office of Justice
Programs. In addition, we reviewed pertinent statutory provisions relating
to the roles and responsibilities of officials with respect to BJS. It was
beyond the scope of this review to address any personnel issues that may
have arisen in connection with the disagreement over the content of the
draft press release. Appendix I provides additional details on our scope
and methodology. We conducted our work between April 2006 and February
2007 in accordance with generally accepted government auditing standards.

In addition, in response to your interest regarding changes that have
occurred in the scope and methodology of the Police-Public Contact Surveys
between 1996 and 2005, and the reasons for any changes, appendix V
provides detailed information on this issue.

Results in Brief

For the four issued reports and the one issued press release based on the
1999 and 2002 Police-Public Contact Surveys, BJS followed nearly all
quality guidelines related to product issuance developed by various
organizations. These reports and press release were issued between 2001
and 2006, and are the most recent products available from the surveys.
Specifically, for the four reports issued from these surveys, we found
that BJS fully followed all product issuance guidelines of the National
Research Council, the Department of Justice, the Office of Justice
Programs, and BJS itself, pertaining to how agencies should review
statistical information, obtain the approval of key decision makers, and
publicly disseminate the information. Because BJS followed these
guidelines, we believe it took proper steps to help ensure the accuracy
and integrity of the processes associated with issuing public reports
based on Police-Public Contact Surveys. BJS concurred with our analysis.
With respect to the single press release issued based on the Police-Public
Contact Survey, we found that BJS was in a position to fully follow 7 of
10 quality guidelines available for the one press release issued from the
1999 survey.^2 BJS did not, in our view, fully follow 3 other guidelines
pertaining to agency independence for reasons discussed below.

Two key factors affected whether and how BJS followed quality guidelines
during the review, approval, and dissemination of products issued from the
1999 and 2002 Police-Public Contact Surveys. First, BJS believed certain
quality guidelines, including those issued by the National Research
Council, applied to its survey-related reports, but not to press releases.
This was because BJS considered its reports to be statistical products
covered by the guidelines, and press releases not to be statistical
products, and therefore not covered by the guidelines. BJS cited a lack of
specificity in the National Research Council's guidelines as a basis for
this conclusion. While we believe that BJS's conclusion was not
unreasonable, our analysis of the press release issued led us to
conclude--based on its content as opposed to its label as a press release,
and regardless of the procedures used to develop and issue it--that it was
primarily a statistical product. In our view, it was therefore appropriate
to apply the council's guidelines to it. Given that the National Research
Council's printed guidelines did not specifically state that they covered
press releases based on the statistical reports of statistical agencies,
we do not fault BJS for concluding that the 2001 press release based on
the 1999 Police-Public Contact Survey was exempt from the guidelines.
Indeed, we noted in our May 2006 report on data quality that 2 of 14
statistical agencies we surveyed stated that there was ambiguity as to
whether a statistical press release was a statistical product. BJS was
among the 14 statistical agencies surveyed, but it was not 1of the 2
agencies reporting ambiguity in whether a statistical press release was a
statistical product. The second factor that affected whether and how BJS
followed guidelines in its product issuance process concerned the role
that certain noncareer appointees outside BJS can play in the product
issuance process. In particular, certain noncareer appointees outside of
BJS, including the Assistant Attorney General in the Office of Justice
Programs, have the ability under Department of Justice policy to become
involved in the review, approval, and dissemination of press releases; and
press releases are issued jointly by BJS, the Department of Justice, and
the Office of Justice Programs. Owing to these circumstances, BJS was not
in a position to follow the National Research Council guidelines
recommending that statistical agencies issue statistical products without
prior clearance; and control the scope, content, and timing of their
products--issues relating to BJS's independence as a statistical agency.

^2 Of the consolidated list of 24 review, approval, and dissemination
guidelines issued by the National Research Council, Department of Justice,
Office of Justice Programs, and Bureau of Justice Statistics, only 10 were
available in 2001, when the press release based on the 1999 survey was
issued.

To address conflicting interpretations of certain quality guidelines, we
recommended in a May 2006 report on the quality of federal data that OMB
issue a directive that, among other things, would provide clear
definitions about what guidelines cover.^3 As of January 2007, according
to an OMB analyst assigned to the effort, OMB was still working on this
directive. Because our May 2006 recommendation to OMB has not yet been
implemented, we are not making any new recommendations in this report. We
continue to believe, however, that implementing our recommendation could
help safeguard the integrity of federal statistical data. We also believe
it would reduce the likelihood that the type of disagreement discussed in
this report would recur, and help assure the independence of BJS as a
statistical agency.

We provided a draft of this report to the Department of Justice and
received written comments from the Justice Department's Office of Justice
Programs' Assistant Attorney General. The Assistant Attorney General
affirmed several of our findings and agreed that a need exists for clear
definitions about what constitutes a statistical product and that the
federal statistical community would benefit from clarity in this area. The
agency disagreed, however, with our characterization of the 2001
Police-Public Contact Survey press release as a statistical product, and
therefore did not believe that the National Research Council's or BJS's
own quality guidelines were applicable to it. We maintain that we made a
sound decision in applying these guidelines to the Police-Public Contact
Survey press release because we felt that the content of the press
release, which was almost entirely statistical in nature, was a more
important determinant of whether or not it was a statistical product than
the label attached to it. The Assistant Attorney General's comments appear
in appendix VI and our detailed response to these comments is contained in
the report.

^3 GAO, Data Quality: Expanded Use of Key Dissemination Practices Would
Further Safeguard the Integrity of Statistical Data, [25]GAO-06-607
(Washington, D.C.: May 31, 2006).

Background

Overview of BJS Mission and Organization

BJS was established by the Justice Systems Improvement Act of 1979.^4 In
1995, OMB identified BJS as one of 10 principal statistical agencies
within the federal government. As defined by OMB,^5 the statistical
activities of statistical agencies include the planning of statistical
surveys and studies; and the collection, processing, or tabulation of
statistical data for publication, dissemination, research, analysis, or
program management and evaluation. BJS publishes annual data on criminal
victimization, populations under correctional supervision, and federal
criminal offenders and case processing. It provides periodic data series
on the administration of law enforcement agencies and correctional
facilities, prosecutorial practices and policies, state court case
processing, felony convictions, the characteristics of correctional
populations, criminal justice expenditure and employment, civil case
processing in state courts, and special studies on other criminal justice
topics.

BJS is organizationally located within the Department of Justice's Office
of Justice Programs (see fig. 1). The highest-ranking executives of BJS
(BJS Director) and the department's Office of Justice Programs (Assistant
Attorney General) are both noncareer officials appointed by the President
and confirmed by the Senate. Within BJS, only the Director is a noncareer
appointee.

^4 Pub. L. No. 96-157, 93 Stat. 1167 (1979).

^5 Office of Management and Budget, Statistical Programs of the United
States Government: Fiscal Year 2006, pp. 3-4 (Washington, D.C.: 2005).

Figure 1: Office of Justice Programs Organizational Structure

The BJS Police-Public Contact Survey

BJS initiated the Police-Public Contact Survey pursuant to a mandate in
the Violent Crime Control and Law Enforcement Act of 1994,^6 which
required the Attorney General to collect information on the use of
excessive force by law enforcement officers. The data were to be used only
for research or statistical purposes and were not to contain any
information that could reveal the identity of the victim or any law
enforcement officer. BJS fielded its first pilot survey in 1996 with the
goal of better understanding the types and frequency of contacts between
the police and the public, and the conditions under which force may be
threatened or used. The pilot survey consisted of 6,421 respondents. The
three subsequent surveys (in 1999, 2002, and 2005) consisted of 80,543,
76,910, and 63,943 respondents, respectively. Multiple reports and press
releases may be issued in connection with any of the surveys. The years in
which reports and a single press release associated with the 1999 and 2002
surveys were issued are shown in table 1.

^6 Pub. L. No. 103-322, 108 Stat. 1796, 2071 (1994) (codified at 42 U.S.C.
14142).

Table 1: Public Products Issued Based on 1999 and 2002 Police-Public
Contact Surveys

Sources: GAO, BJS.

aContacts between Police and the Public, Findings from the 1999 National
Survey, February 2001.

bCharacteristics of Drivers Stopped by Police, 1999, March 2002.

cContacts between Police and the Public, Findings from the 2002 National
Survey, April 2005.

dCharacteristics of Drivers Stopped by Police, 2002, June 2006.

eForce or Threatened Force Used in Less than 1 Percent of All
Police-Public Interactions, March 2001.

Quality Guidelines for Statistical Products

Over the last several years, various types of guidance have been developed
to help federal agencies such as BJS ensure the integrity of statistical
information. In 1992, in response to requests from Congress and others as
to what constitutes an effective statistical agency, the National Research
Council began issuing best-practice guidelines.^7 According to the
Committee on National Statistics,^8 which authored the guidelines, the
guidelines have been widely cited and used by Congress and federal
agencies, and have shaped legislation and executive actions to establish
and evaluate statistical agencies. These recommended guidelines, which BJS
and other statistical agencies may choose to voluntarily follow, cover the
review, approval, and dissemination processes of products issued by
federal statistical agencies. In its guideline document, Principles and
Practices for Federal Statistical Agencies, the National Research Council
indicated, among other things, that statistical agencies should provide
high-quality data, take a strong position of independence, be perceived to
be free of political interference and policy advocacy, and strive for wide
dissemination of their results.^9 In particular, according to the National
Research Council, the quality guidelines are to cover the

^7 For ease of presentation, in this report we use the term "guidelines"
to refer to the principles and practices described in the National
Research Council document, as well as to guidelines issued by the
Department of Justice, Office of Justice Programs, and BJS.

^8 The National Research Council's Committee on National Statistics was
established in 1972 at the recommendation of the President's Commission on
Federal Statistics in order to improve the statistical methods and
information on which public policy decisions are based.

           o review process and include verification of sources and results,
           disclosure of limitations, and accuracy of results;
           o approval process and include who has authority over the content
           and timing of the release of a product, and separation of policy
           from statistical information; and
           o dissemination process and include the usability of information
           and its accessibility to a wide range of people.

In February 2002 and September 2006, pursuant to the Information Quality
Act of 2001, OMB issued policy and procedural guidance to federal
agencies, including statistical agencies such as BJS, directing them to
develop their own quality guidelines to help maximize the quality,
objectivity, utility, and integrity of the information they
disseminate.^10 OMB stated that it was essential that federal statistics
be collected, processed, and published in a manner that guarantees and
inspires confidence in their reliability.^11 Specifically, OMB directed
federal agencies to "adopt a basic standard of quality ... as a
performance goal," and "take appropriate steps to incorporate information
quality criteria into agency dissemination practices."

In response to OMB's February 2002 guidance, the Department of Justice,
Office of Justice Programs, and BJS issued their own guidelines later that
year. BJS issued a second edition of its guidelines in 2005. In
formulating its guidelines, BJS stated that it sought to provide the
public with additional information regarding its methods for ensuring the
quality, utility, objectivity, and integrity of the statistics it publicly
disseminates. As a component of the Department of Justice, BJS is governed
by its own data quality guidelines, as well as the information quality
guidelines promulgated by the Office of Justice Programs, Department of
Justice, and Office of Management and Budget. The Department of Justice's
Information Quality Guidelines are intended to (1) provide the
department's components with a foundation for developing their own, more
detailed procedures, (2) provide guidance to component staff, and (3)
inform the public of the agency's policies and procedures.

^9 Margaret E. Martin, Miron L. Straf, and Constance F. Citro (eds.),
Principles and Practices for a Federal Statistical Agency, (3rd ed.)
(Washington, D.C.: The National Academies Press, 2005).

^10 Section 515 of the Treasury and General Government Appropriations Act
for Fiscal Year 2001 (Pub. L. No. 106-554, 114 Stat. 2763, 2763A-125
(2000)), known as both the Information Quality Act and the Data Quality
Act, required OMB to issue governmentwide quality guidelines including the
requirement that such agencies issue their own quality guidelines.

^11 See Notice of revisions to OMB's Standards and Guidelines for
Statistical Surveys, 71 Fed. Reg. 55522 (Sept. 22, 2006).

The Office of Justice Programs' information quality guidelines require its
components--including BJS--to (1) assess the usefulness of the information
to be disseminated to the public by continuously monitoring information
needs, developing new information sources, or revising existing methods,
models, and information products where appropriate; (2) ensure
disseminated information is accurate, clear, complete, reproducible, and
presented in an unbiased manner by using reliable data sources, sound
analytical techniques, and documenting methods and data sources; and (3)
protect information from unauthorized access, corruption, or revision. As
in the case of the Department of Justice's guidelines, the Office of
Justice Programs provides guidance to its components in developing their
own, more specific quality guidelines.

BJS Followed Nearly All Available Quality Guidelines to Help Ensure Accuracy and
Integrity of Products Issued from Police-Public Contact Surveys

BJS Fully Followed All Applicable Quality Guidelines for the Four Reports Issued
from the 1999 and 2002 Police-Public Contact Surveys

For all four reports issued from the two Police-Public Contact Surveys, we
found that BJS fully followed all of the review, approval, and
dissemination guidelines available at the time of issuance. We considered
a guideline to have been fully followed if our independent analysts
determined that all aspects of the guideline were followed. (Our
methodology for how we determined the extent to which BJS followed the
guidelines is explained in app. I.) The extent to which BJS followed
applicable, available guidelines when it issued its Police-Public Contact
Survey reports is shown in table 2.

Table 2: Guidelines That BJS Fully Followed for the Reports Issued from
Its 1999 and 2002 Police-Public Contact Surveys

Source: GAO analysis

aThe number of guidelines varied across the time periods covered. One
National Research Council guideline was dropped when the council issued
the second edition of its guidelines. Had this guideline not been dropped,
13 guidelines would have been applicable to BJS's 2002 survey report,
instead of 12, and 24 guidelines would have been applicable to the 2005
and 2006 survey reports, instead of 23. Table 4 in appendix II reflects
which guidelines were applicable to each of the survey products.

For the first report issued from the 1999 survey, we found that BJS
voluntarily followed the National Research Council's 10 applicable
existing guidelines; for the second report, we found that BJS voluntarily
followed those 10, as well as 2 additional guidelines issued since the
first report, for a total of 12. For each of the two reports based on the
2002 survey, we found that BJS followed all 23 available data quality
guidelines that had by then been issued by the National Research Council,
the Department of Justice, the Office of Justice Programs, and BJS
itself.^12

The data quality guidelines that BJS followed describe how agencies should
review statistical information, obtain the approval of key decision
makers, and publicly disseminate the information. While not all of the
guideline-issuing organizations addressed the review, approval, and
dissemination process, in total across the four organizations--the
National Research Council, Department of Justice, Office of Justice
Programs, and Bureau of Justice Statistics--all three areas were
addressed. Some examples of the guidelines that BJS fully followed in its
report issuance process are listed below. (For a complete list of all
available data quality guidelines, see appendix II.)

           o Components of the Department of Justice and Office of Justice
           Programs will review all information dissemination products for
           their quality (including objectivity, utility, and integrity)
           before they are disseminated.
           o All BJS reports and other statistical products must be subject
           to an objective and appropriate verification process conducted by
           qualified BJS staff other than the author of the report.
           o The statistical agency has recognition by policy officials
           outside the statistical agency of its authority to release
           statistical information without prior clearance.
           o The statistical agency has authority for professional decisions
           over the scope, content, and frequency of data compiled, analyzed,
           or published.
           o The objectivity of BJS statistics must be vigilantly protected
           at all times by BJS staff.

On the basis of our analysis, BJS successfully followed all applicable
quality guidelines for these survey-based statistical reports, which both
BJS and we consider to be statistical products covered by the guidelines.
Thus, we believe the agency took proper steps to help ensure the accuracy
and integrity of the review, approval, and dissemination processes
associated with issuing public reports based on the two surveys we
reviewed. BJS concurred with our analysis. All of the reports were posted
to the BJS Web site, where the information is to be accessible to the
general public.^13

12 After OMB issued policy and procedural guidance to federal agencies in
2002, the Department of Justice, Office of Justice Programs, and BJS
developed their own guidelines.

BJS Fully Followed 7 of 10 Applicable Data Quality Guidelines for the Single
Press Release Issued from Its Police-Public Contact Surveys

For the single press release that was issued--that is, the 2001 press
release based on BJS's 1999 Police-Public Contact Survey--we determined
that BJS fully followed 7 of the 10 applicable National Research Council
guidelines available at the time.^14 The 7 federal data quality guidelines
that BJS fully followed are listed below.

           (1) A statistical agency should develop an understanding of the
           validity and accuracy of its data and convey the resulting
           measures of quality to users in ways that are comprehensible to
           nonexperts.
           (2) A statistical agency should use modern statistical theory and
           sound statistical practice in all technical work.
           (3) A statistical agency has maintenance of a clear distinction
           between statistical information and policy interpretations of such
           information by the President, the secretary of the department, or
           others in the executive branch.
           (4) A statistical agency should follow good practice, in reports
           and other data releases, in documenting concepts, definitions,
           data collection methodologies, and measures of uncertainty, and in
           discussing possible sources of error.
           (5) Effective dissemination programs include policies for the
           preservation of data that guide what data to retain and how they
           are to be archived for future secondary analysis.
           (6) An agency should have an established publications policy that
           describes, for a data collection program, the types of reports and
           other data released to be made available, the audiences served,
           and the frequency of release.
           (7) Dissemination of data and information (basic series, analytic
           reports, press releases, public use tapes) should be timely and
           public. Avenues of dissemination should be chosen to reach as
           broad a public as reasonably possible.

^13 Information posted on the BJS Web site includes hypertext links to
information that is created and maintained by other public and private
organizations in the United States and by other nations.

^14 Guidelines developed by BJS, the Office of Justice Programs, and the
Department of Justice were not available at the time the press release,
based on the 1999 survey, was released.

There were 3 other applicable quality guidelines that we determined BJS
was not in a position to follow in connection with this issued press
release, and a press release based on the 2002 Police-Public Contact
Survey findings was not issued. It is important to note that, for reasons
discussed later in this report, BJS officials did not believe these
guidelines were applicable to its press releases in the first place.

Differing Views on Applicability of Guidelines Raised Questions about Clarity,
while Roles of Certain Noncareer Appointees during Product Issuance Process
Affected BJS's Independence

Two key factors affected whether and how BJS followed quality guidelines
during the review, approval, and dissemination of products issued from the
1999 and 2002 Police-Public Contact Surveys. First, while BJS believed, as
noted earlier, that its survey reports were statistical products covered
by the quality guidelines, it did not believe that the survey-related
press release was a statistical product covered by the quality guidelines.
BJS cited a lack of specificity in the National Research Council's
guidelines as a basis for this conclusion. We believe, however, that while
BJS's interpretation of the guidelines was not unreasonable, there was
nonetheless sufficient evidence for a different interpretation; namely,
that this press release was a statistical product, that the available
guidelines did apply, and that BJS was not in a position to meet 3 of 10
guidelines for the single press release issued from the 1999 survey, owing
to a second factor. This second factor was the role that certain noncareer
appointees outside BJS have the ability to play, pursuant to Department of
Justice policy, in the product issuance process. In certain instances, the
roles of these non-BJS officials meant that BJS was not in a position to
fully follow all guidelines related to agency independence, and this holds
the potential for future actual or perceived political interference in
BJS's product issuance process for statistical products.

BJS Interpreted Guidelines as Not Applying to Press Releases, Raising Questions
about the Applicability and Clarity of Certain Guidelines

In both written documentation and oral comments, BJS officials stated that
they believed they were in full conformance with the National Research
Council's guidelines and disagreed with our determination that the agency
was not in a position to follow 3 of 10 guidelines for the 2001
Police-Public Contact Survey press release that was issued from the 1999
survey. The guidelines that we determined BJS was not in the position to
fully follow all pertain to the agency's independence and, in particular,
to its control over the issuance of press releases. These guidelines were:

           (1) The statistical agency has recognition by policy officials of
           its authority to release statistical information without prior
           clearance.
           (2) The statistical agency has authority for professional
           decisions over the scope, content, and frequency of data compiled,
           analyzed, or published.
           (3) The release of information should not be subject to actual or
           perceived political interference. In particular, the timing of the
           public release of data should be the responsibility of the
           statistical agency.

BJS officials asserted that, based on their interpretation of the National
Research Council's guidelines, BJS press releases did not qualify as
statistical products and, therefore, press releases did not fall within
the purview of the council's guidelines. They also asserted that neither
BJS's own quality guidelines, nor those issued by the Department of
Justice and the Office of Justice Programs, apply to BJS press releases.
Both BJS and Office of Justice Programs officials stated that the
applicability of the council's guidelines to BJS press releases was, at a
minimum, open to question because the council did not state that press
releases are data disseminations. In other words, according to BJS and the
Office of Justice Programs, press releases are not publications of data,
but rather they are simply announcements that a data publication is
forthcoming. In its communications with us, BJS stated that many of the
guidelines do not apply to press releases but apply only to statistical
products.

Based on its content rather than its label as a press release, and
notwithstanding that the policies and procedures for developing and
issuing products labeled by the Office of Justice Programs as press
releases differed from policies and procedures for products it labeled as
statistical products, we believe there is sufficient evidence for us to
conclude that the press release issued from the 1999 Police-Public Contact
Survey qualified as a statistical product to which the National Research
Council's quality guidelines appropriately apply. Our analysis of this
press release indicated that it was a data-based statistical product, more
than simply an announcement that a data publication was forthcoming. In
its entirety, the press release consisted of 20 sentences and one table
describing the survey's statistical findings; 3 sentences on the survey's
methodology; and 5 sentences on who prepared the report and how to obtain
copies. We found that this press release was a compilation of statistical
data that contained no interpretations, conclusions, or policy
statements.^15 (See Appendix III for a reproduction of the press release.)
In accordance with the council's guidelines, the release maintained "a
clear distinction between statistical information and policy
interpretations of such information."

To understand whether the National Research Council was purposeful in not
stating that its guidelines were applicable to statistical agency press
releases, we contacted the council to seek clarification. Officials from
the council's Committee on National Statistics, which authored the data
quality guidelines, stated that although the Principles and Practices
document did not specifically state that the guidelines covered the
content, scope, and timing of press releases issued by statistical
agencies, it was not the committee's intent to exclude press releases from
the guidelines. They stated that, in their view, press releases issued by
BJS are statistical products to which it is appropriate to apply the
guidelines.

BJS and we agree that the National Research Council's guidelines apply, in
general, to statistical products. In asserting that the press release that
BJS jointly issued with the Department of Justice and Office of Justice
Programs was not a statistical product, BJS correctly noted that the
National Research Council did not explicitly state that the guidelines
covered press releases. However, given the strong statistical content of
the Police-Public Contact Survey press release, we did not believe that
such an explication was necessary. Nonetheless, we acknowledge that it is
not unreasonable for BJS to reach a different conclusion given the lack of
specificity that existed in the council's printed guidelines. Because
BJS's own data quality guidelines, issued in 2002, state that they "govern
all justice statistics that BJS produces and disseminates for the general
public, including all statistics that are featured in BJS publications, on
the website, and in BJS press releases," we considered the BJS guidelines
to be applicable to press releases, as well. BJS, however, did not hold
this view.

^15 Our work focused exclusively on the Police-Public Contact Survey, and
we did not conduct a systematic review of other press releases that have
been issued based on BJS's work, to make an independent determination of
whether they constituted statistical products.

It is important to note that we are not finding fault with BJS for the
conclusions it drew with respect to the applicability of the quality
guidelines to its press release issuance process because the National
Research Council's guidelines were not explicit on this matter. Indeed, we
noted in our May 2006 report on data quality that 2 of 14 statistical
agencies we surveyed stated that there was ambiguity as to whether a
statistical press release was a statistical product, and if so, whether
statistical agencies could issue them without first getting releases
cleared at the departmental level. BJS was among the 14 statistical
agencies surveyed, but it was not one of the two agencies reporting
ambiguity in whether a statistical press release was a statistical
product. Overall, we believe that BJS made a good faith effort to follow
the guidelines it deemed to be applicable to the Police-Public Contact
Survey products.

Deciding which guidelines a statistical agency like BJS should follow is
further complicated by the fact that BJS's parent organizations--the
Department of Justice and Office of Justice Programs--have explicitly
stated that their own guidelines do not apply to press releases. However,
these organizations' guidelines are intended to be broadly applicable to
both statistical and nonstatistical agencies. For example, the Department
of Justice comprises 38 separate component organizations that produce a
variety of types of information, both statistical and nonstatistical in
nature. The Office of Justice Programs is composed of 6 bureaus and
program offices, and these, too, produce both statistical and
nonstatistical information. Because we believe that press releases issued
by the department and the Office of Justice Programs may in some, but not
all, instances be statistical products, we do not hold the view that
statistical guidelines should be universally applicable to all press
releases issued by the Department of Justice and Office of Justice
Programs. However, because different interpretations can arise, we believe
that clarification regarding which guidelines should be applied under
which circumstances--and, specifically, to press releases--would be
helpful to statistical agencies that are in situations similar to BJS's.

To address potential discrepancies such as these, in a May 2006 report on
the quality of federal data,^16 we recommended that to help improve
governmentwide data dissemination practices that would further safeguard
the integrity of federal statistical data, OMB should consider how best to
address the gaps we identified between agencies' data dissemination
practices and the National Research Council's guidelines. We noted in that
report that OMB, in concert with federal statistical agencies, was
developing a governmentwide directive on the release and dissemination of
statistical products that, according to OMB officials, parallels the
council's and other generally accepted dissemination practices. We pointed
out that it will be important for OMB's directive to consider, for
example, whether the directive should cover principal statistical agencies
only, the statistical functions of all agencies, or only statistical
products. OMB officials indicated that the guidance is intended to help
ensure that statistical products are policy-neutral, timely, and accurate.
We recommended that, among other things, OMB include in this directive (1)
clear definitions of what is and is not covered, (2) the extent to which
agencies should document their data dissemination guidance and how often
the guidance should be reviewed, (3) the amount of flexibility agencies
have in implementing OMB's guidance, and (4) procedures for monitoring
agencies' adherence to its directive. To the extent that statistical
agencies appropriately follow these practices, the directive could promote
more consistent adherence to practices that facilitate broader
dissemination of statistical data and enhance its credibility. Although
OMB did not provide comments on the recommendations in our 2006 report, an
OMB official told us that as of January 2007, OMB was still working on
this directive.^17 We believe it remains important for OMB to complete its
directive on the release and dissemination of statistical products in
order to help safeguard the integrity of federal statistical data, reduce
the likelihood that the type of disagreement discussed in this report
would recur, and help assure both the actual and perceived independence of
BJS.

^16 See [26]GAO-06-607 .

^17 OMB's February 2002 guidelines for federal agencies exempted press
releases from the guidelines. These guidelines, however, were designed to
apply to a wide variety of government information dissemination
activities, and be generic enough to fit all types of media. As we noted
in our May 2006 report ( [27]GAO-06-607 ), the directive on which OMB is
currently working pertains directly to the release and dissemination of
statistical products.

The Roles Played by Certain Outside Noncareer Appointees in the Press Release
Process Affected BJS Carrying Out Its Role as an Independent Statistical Agency

The second key factor that affected whether and how BJS followed
guidelines concerned the involvement of noncareer appointees outside of
BJS in the press release issuance process, and had implications for BJS's
independence as a statistical agency. Specifically, we determined that BJS
was not in a position to fully follow the 3 National Research Council
guidelines listed in the previous section for the 2001 press release based
on the 1999 survey (the only applicable, available data quality guidelines
in place in 2001) because certain noncareer appointees outside of BJS and
within the Department of Justice, are vested--pursuant to the Department
of Justice's and Office of Justice Programs' policies defining the roles
and responsibilities of their noncareer appointees--with the ability to
participate in the review, approval, and dissemination of press
releases.^18 In certain cases, the roles and responsibilities of these
noncareer appointees precluded BJS from being in the position to fully
follow certain guidelines. The Assistant Attorney General within the
Department of Justice's Office of Justice Programs has general statutory
responsibilities with respect to coordinating the activities of that
office and its various components, such as BJS.^19 These statutory
provisions do not specifically address the Office of Justice Programs'
role with respect to the review, approval, and dissemination of press
releases. However, under departmental policy, noncareer appointees within
the Department of Justice and outside of BJS have the ability to
participate in the press release issuance process.

Table 3 shows the type of involvement that the Assistant Attorney General
in the Office of Justice Programs and other noncareer appointees generally
have had in the press release review, approval, and dissemination process.
Appendix IV describes in more detail the responsibilities of these various
officials associated with review, approval, and dissemination procedures
for both BJS reports and press releases.

^18 We did not review the extent to which BJS followed guidelines for the
2005 draft press release (based on the 2002 survey) that was the source of
disagreement between the then-BJS Director and the then-Acting Assistant
Attorney General, because the press release was not issued.

^19 U.S. Code, Title 42, Chapter 46, Subchapter I (42 U.S.C. Sections 3711
through 3715a).

Table 3: Noncareer Appointees' Roles in the Review, Approval, and
Dissemination of BJS Press Releases

Source: BJS and Office of Justice Programs.

aTo the BJS web site, after approval by the BJS Director.

bTo Congress, the media, and executive department press offices, after
verification by the BJS Director.

With respect to the first of the three guidelines, which calls for a
statistical agency to have authority to release information without prior
clearance, it is our view that BJS was not in a position to follow this
independence-related guideline at all because it did not have the ability
to do so. This is because press releases are subject to review and
approval by not only the BJS Director, but also by other Department of
Justice noncareer appointees.^20 Outside of BJS, the noncareer appointees
participating in the clearance process are located in the Department of
Justice's Office of Justice Programs (these include the Office's Chief of
Staff, Deputy Assistant Attorney General, and Assistant Attorney General)
and Office of Public Affairs.

The current Assistant Attorney General and two former Assistant Attorneys
General in the Office of Justice Programs told us that there is no
written, formal policy or guidance that bounds their input and
decision-making roles and responsibilities with respect to BJS press
releases. BJS and OJP officials indicated that the Office of Justice
Programs' Assistant Attorney General has ultimate responsibility for the
review and approval of BJS press releases. Press releases are issued
jointly on letterhead listing BJS and the Department of Justice. The
current BJS Director confirmed that publication and dissemination
functions for press releases are considered to be within the Assistant
Attorney General's oversight authority. Because the National Research
Council stated that an aspect of independence includes "recognition by
policy officials outside the statistical agency of its authority to
release statistical information without prior clearance," we concluded
that BJS was not in the position to follow this guideline because, as we
have stated, we believe the Police-Public Contact Survey press release was
a statistical product that BJS could not issue independently.

^20 In contrast to the process followed for press releases, the only
noncareer appointee involved in the review, approval, and dissemination of
reports from Police-Public Contact Surveys is the BJS Director. Appendix
IV provides details on the roles of the Director and other employees in
this process.

In practice, the ways in which Assistant Attorneys General of the Office
of Justice Programs have exercised their authority have varied. For
instance, one former Office of Justice Programs' Assistant Attorney
General told us that she placed "self-imposed" limits on her decisions to
modify the content of a BJS press release based on her awareness of
congressional support for, and her own belief in, the independence of
statistical agencies. The current Office of Justice Programs' Assistant
Attorney General told us that she reviews only press releases that contain
quotes from the Attorney General. She said that since she assumed her
position in 2005, there have been no BJS press releases that have quoted
the Attorney General, and she has relied on her Deputy Assistant Attorney
General, the BJS Director, and others to ensure the accuracy and clarity
of press releases. Nevertheless, the BJS Director must obtain the approval
of the Office of Justice Programs' Assistant Attorney General and other
Justice noncareer appointees to issue a press release.

With respect to the second guideline, pertaining to the agency's decisions
over the scope, content, and frequency of data compiled, analyzed, or
published, we found that BJS was not in a position to fully follow this
independence-related guideline. Specifically, we found that BJS could
exercise professional decisions about the frequency of data analyzed and
published (within available budgets), but did not always have complete
control over the scope and content of survey press releases to be issued.
As noted above, this was due to the fact that press releases are joint
products of BJS, the Office of Justice Programs, and the Department of
Justice, and noncareer appointees outside of BJS can become involved in
the press release process. BJS's situation with respect to this second
guideline came to the fore during the drafting of a press release in 2005
based on the 2002 Police-Public Contact Survey. The press release that BJS
sought to publish would have included the following statistical findings
from the accompanying Police-Public Contact Survey report: (1) there was
no statistically significant difference between the rates that white and
minority drivers reported being stopped by police, and (2) once stopped, a
larger percentage of black and Hispanic minority drivers reported police
using or threatening to use force against them than did whites. The
then-BJS Director and the then-Acting Assistant Attorney General had a
difference of opinion regarding the presentation of the second statistical
finding, which was included in the Police-Public Contact Survey report.
Despite reported efforts on the part of both parties to negotiate
alternative language with respect to the content of the press release,
they could not resolve their differences and the BJS Director decided that
a press release would not be issued. The current BJS Director told us that
it is "inconceivable" that the Assistant Attorney General would issue a
press release without the BJS Director's prior approval.

According to current BJS officials (both career and noncareer) and the
Office of Justice Programs' Office of Communications staff, during the
period 1996-2006, this was the only instance in which a BJS press release
was prepared but not issued because the Office of Justice Programs and BJS
could not agree on the contents. In all other instances during this
period, according to these officials, when the parties disagreed on the
content of a press release, they were able to resolve their differences.

With respect to the third guideline, pertaining to actual or perceived
political interference and the timing of a release, we similarly believe
BJS was in not in a position to fully follow this independence-related
guideline for the 2001 press release, which, as discussed earlier, we
believe to be a statistical product. Although we found no evidence of
political interference with the timing of the 2001 survey press release
issued from the 1999 survey, we found that BJS does not have complete
control over the timing of press releases, as recommended by the National
Research Council. Since both noncareer appointees and career officials in
the Office of Justice Programs and the Department of Justice have a role
in reviewing and approving BJS press releases, they can affect the date
that a press release is issued. According to BJS, career and noncareer
appointees outside of BJS can delay the issuance of a press release for
reasons having nothing to do with political interference, such as a
determination that the press release is not sufficiently newsworthy at the
time that it was designated to be issued.

On balance, we believe that the noncareer appointees who played
decision-making roles in the Police-Public Contact Survey press release
process that we reviewed acted within the scope of the roles and
responsibilities accorded them under Department of Justice policies, and
that BJS made a reasonable effort to adhere to all applicable data quality
guidelines. The fact that certain noncareer officials have the ability to
make decisions that affect BJS's ability to fully meet federal data
quality guidelines suggests, however, that the potential exists for BJS's
review, approval, and dissemination process for statistical products to be
subject to political interference. Thus, certain actions by noncareer
appointees--though made on the basis of professional judgment--could put
them at odds with the very guidelines designed to ensure the statistical
independence and integrity of agencies such as BJS.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Justice for review
and comment. On March 13, we received written comments on the draft report
from the Office of Justice Programs' Assistant Attorney General, and the
comments are reproduced in full in appendix VI.

In her letter, the Assistant Attorney General affirmed several of our
findings and agreed that a need exists for clear definitions about what
quality guidelines cover. She noted that competing interpretations exist
about what constitutes a statistical product and that the federal
statistical community would benefit from clarity in this area.

However, the Assistant Attorney General disagreed with our
characterization of the 2001 Police-Public Contact Survey press release as
a statistical product and, therefore, with our conclusion that the
National Research Council's quality guidelines applied to this press
release. The Assistant Attorney General stated that "a press release ...
is a public relations announcement issued to encourage media coverage. The
mere presence of statistics in a press release does not transform a press
release into a statistical product." We do not believe and have not stated
that the mere presence of statistics in a press release in and of itself
transforms it into a statistical product  any more than we believe or have
stated that labeling a document lacking in statistics but called a
statistical product necessarily transforms it into one. The Assistant
Attorney General also stated that we "mischaracterized" BJS's data quality
guidelines as applying to press releases because the guidelines apply only
to the statistics contained in BJS press releases, and because BJS
conforms with OMB, the Department of Justice, and the Office of Justice
Programs in considering press releases to be outside the scope of the
guidelines. For the following reasons, we maintain that we made a sound
decision in applying BJS's guidelines to the Police-Public Contact Survey
press release: (1) BJS's guidelines state that they "govern all justice
statistics that BJS produces and disseminates for the general public,
including all statistics that are featured in BJS publications, on the
website, and in BJS press releases;" and (2) the Police-Public Contact
Survey press release was made up almost entirely of survey statistics,
indicating to us that it was a statistical product. In determining that
the Police-Public Contact Survey was a statistical product, we felt that
the content of the press release was a more important determinant than the
label attached to it, or the fact that the processes and staff involved in
developing the press release were different from those in BJS reports.

The Assistant Attorney General also noted that the National Research
Council's written guidelines did not explicitly cover press releases.
Because we agree, we contacted the National Research Council and consulted
with officials of the Council's Committee on National Statistics (the
authoring committee of the Principles and Practices). The officials
concurred with our view that BJS press releases referring to statistical
products (as opposed to press releases about the announcement of a new
agency head, for example) are statistical products to which it is
appropriate to apply the guidelines. Although the Principles and Practices
document did not specifically state that the guidelines covered the
content, scope, and timing of press releases issued by statistical
agencies, according to these officials it was not the Committee's intent
to exclude such press releases from the guidelines.

The Assistant Attorney General also felt that the draft report overstated
the potential threats to BJS's independence because we used the term
"statistical products" to refer to press releases. She was concerned with
our observation that the potential exists for BJS's review, approval, and
dissemination process for statistical products to be subject to political
interference since noncareer officials can affect BJS's ability to meet
federal data quality guidelines. We stand by this conclusion. Department
of Justice policy permits noncareer appointees within the Department but
outside of BJS to participate in the press release process. At the same
time however, we are unaware of anything that prevents future
modifications to that policy to similarly allow noncareer appointees to
participate in BJS's report issuance. Thus, we believe that we have
correctly assessed the risk of potential or actual threats to BJS's
independence.

Finally, the Assistant Attorney General stated that even if the council's
written guidelines explicitly applied to press releases, the BJS director
would not adhere to them and no current law can make him do so. We
recognize that they are voluntary and not legally required and never have
said otherwise.

As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from
its issue date. At that time, we will send copies of the report to the
Attorney General, the Director of the Office of Management and Budget, and
other interested parties. In addition, the report will be available at no
charge on GAO's home page at http://www.gao.gov.

Please contact Brian Lepore at (202) 512-4523 or [email protected] if you or
your staff have any questions concerning this report. Contact points for
our Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are listed
in appendix VII.

Brian Lepore, Acting Director Homeland Security and Justice Issues

List of Congressional Addressees

The Honorable John Conyers
Chairman
Committee on the Judiciary
House of Representatives

The Honorable William Delahunt
The Honorable Sheila Jackson Lee
The Honorable Zoe Lofgren
The Honorable Martin Meehan
The Honorable Jerrold Nadler
The Honorable Linda Sanchez
The Honorable Debbie Wasserman Schultz
The Honorable Robert C. "Bobby" Scott
The Honorable Mel Watt
The Honorable Robert Wexler
House of Representatives

Appendix I: Objectives, Scope, and Methodology

This report addresses the following two objectives for the 1999 and 2002
Police-Public Contact Surveys, the two surveys for which products had been
issued as of February 2007: (1) To what extent did the Bureau of Justice
Statistics (BJS) follow available guidelines to help ensure the accuracy
and integrity of the review, approval, and dissemination of reports and
press releases based on its surveys? (2) What key factors affected whether
and how BJS followed available guidelines? In addition, we provide
information on scope and methodology changes in the Police-Public Contact
Surveys over time (see app. V).

To address the first objective, regarding the extent to which BJS followed
guidelines, we obtained quality guideline documents from BJS, the
Department of Justice's Office of Justice Programs, the Department of
Justice, and the National Research Council. The guidelines that we
obtained from these organizations covered the period between February
2001, when the first product based on the 1999 Police-Public Contact
Survey was issued, and June 2006, when the most recent product based on
the 2002 survey was issued. We included these federal organizations in our
review because BJS is a component of the Office of Justice Programs, which
in turn is a component of the Department of Justice, and BJS considers
itself to be "governed by" the information quality guidelines of these
organizations. We included the National Research Council in our review
because it is a widely recognized organization that issued guidelines that
were intended to be statements of best practice and provide information on
what constitutes an effective statistical organization. We also reviewed
guidance and directives issued by the Office of Management and Budget
(OMB) because OMB is charged with issuing governmentwide policy and
procedural guidance to federal agencies, which are then encouraged to
issue their own implementation guidelines.

We took several steps to determine the extent to which BJS followed the
specific quality guidelines that it, the Office of Justice Programs, the
Department of Justice, and the National Research Council had issued. From
the documents provided by these four organizations, a GAO analyst
initially identified a total of 63 guidelines that pertained to product
review, approval, and dissemination processes. For verification purposes,
a GAO methodologist also reviewed the guideline documents. The GAO
methodologist agreed with the auditor that all 63 guidelines were
appropriate for inclusion in our review. Because many of the guidelines
issued by the four organizations were similar and overlapping, the GAO
auditor reduced the list to 24 nonduplicative guidelines. The GAO
methodologist again reviewed the work of the auditor, and in all cases
agreed with the auditor that similar guidelines were being appropriately
grouped.

We then developed a data collection instrument to determine whether BJS
was following guidelines for the 1999 and 2002 Police-Public Contact
Surveys, on which information could be recorded as to whether BJS fully
followed, partially followed, or did not at all follow each of the
guidelines. We defined "fully" as all aspects of the guideline being
followed; "partially" as some, but not all, aspects of the guideline being
followed; and "not at all" as no aspects of the guideline being followed.
We asked BJS to complete a separate data collection instrument with
respect to each of its 1999 and 2002 Police-Public Contact Survey reports
and one press release, and to support each response by providing
documentary evidence. To decrease the burden on BJS, GAO analysts
completed the data collection instrument for 9 of the 24 guidelines, for
which we already had sufficient information (for example, documents
describing agency processes and procedures, and interviews regarding the
roles and responsibilities of noncareer appointees).^1 We provided our
assessments regarding these guidelines to BJS and asked officials to
either confirm or not confirm them. Two GAO analysts reviewed BJS's
responses and all available supporting documentary and testimonial
evidence, and determined whether BJS fully, partially, or did not at all
meet each guideline. We provided our findings to BJS for review and
comment.

BJS's 1996 and 2005 Police-Public Contact Surveys were outside the scope
of our work. We excluded the 1996 survey because that was a relatively
small-scale pilot study; and we excluded the 2005 survey, the most recent
Police-Public Contact Survey conducted, because no reports or press
releases have yet been issued from this work.

To address the second objective, regarding key factors that affected
whether and how BJS followed guidelines, we reviewed processes and
procedures that described the review, approval, and dissemination
processes for BJS-generated reports and press releases, with particular
interest in identifying the roles of noncareer appointees involved in each
of these processes. We also reviewed pertinent statutory provisions
relating to the roles and responsibilities of officials with respect to
BJS. We conducted in-person interviews with, or obtained written responses
to our questions from, noncareer appointees in BJS, the Office of Justice
Programs, and the Department of Justice's Office of Public Affairs.
Specifically, we conducted in-person interviews with the current BJS
Director and the BJS Director who was involved in the disagreement with
the Acting Assistant Attorney General, as well as with the current
Assistant Attorney General and Deputy Assistant Attorney General in the
Office of Justice Programs. We obtained detailed written responses to our
questions from a former BJS Director, the Acting Assistant Attorney
General who was involved in the disagreement with the BJS Director, and
two former Assistant Attorneys General from the Office of Justice
Programs. We conducted a telephone interview with the current Deputy
Director of the Department of Justice's Office of Public Affairs. Among
other things, we asked these noncareer appointees to provide us with
information about BJS's process for reviewing, approving, and
disseminating reports and press releases; the roles and responsibilities
of noncareer appointees in that process; changes, if any, that had
occurred in the roles played by noncareer appointees; procedures used to
help ensure that BJS reports and press releases were accurate, reliable,
and unbiased; and any factors that may have affected BJS's independence in
the product issuance process. Finally, we reviewed the guidelines of BJS,
the Department of Justice's Office of Justice Programs, the Department of
Justice, and the National Research Council to determine that they
reflected the product issuance processes and to consolidate them in order
to eliminate duplication.

^1 The number of guidelines that were applicable to the review, approval,
and dissemination of Police-Public Contact Surveys products varied across
the time periods covered. In general, the number of guidelines increased
between February 2001, when the first survey report was issued, and June
2006, when the latest survey report was issued. However, one National
Research Council guideline, pertaining to the dissemination of data and
information to as broad a public as reasonably possible, was dropped when
the council issued the second edition of its guidelines between the time
that BJS issued its first report (in February 2001) and second report (in
March 2002) from the 1999 survey.

To determine what changes, if any, have occurred in the scope and
methodology of the Police-Public Contact Surveys between 1996 and 2006,
which we present in appendix V, we initially developed a matrix of key
scope and methodology dimensions, based on a review of the standard social
science literature. We then conducted interviews and reviewed documents
with respect to these dimensions, for all four Police-Public Contact
Surveys--the 1996 pilot survey and the surveys of 1999, 2002, and 2005. We
interviewed the current BJS Director and a former BJS Director, and
available report authors and the key statistician participating in
administrations of the survey, to ascertain their views concerning the
intended scope of the four surveys, the methodologies used, scope and
methodology changes that were made, and reasons for any changes. We also
obtained written responses to our questions from these officials. We
conducted a detailed documentary review of the scoping and methodology
sections of the issued Police-Public Contact Survey reports and press
releases, and extracted information about changes in the data collection
instruments used (for example, the numbers and types of questions asked
about searches and the use of force). In addition, we reviewed documents
prepared by the American Statistical Association and the U.S. Bureau of
the Census, which conducted field tests to ensure that Police-Public
Contact Survey questions were appropriately devised. In cases where we
noted that changes had been made between surveys, we reviewed Census
Bureau documentation and interviewed staff and officials at BJS.

It was beyond the scope of this review to address any personnel issues
that may have arisen in connection with the disagreement over the content
of the 2005 draft press release based on the 2002 Police-Public Contact
Survey. We conducted our work between April 2006 and January 2007 in
accordance with generally accepted government auditing standards.

Appendix II: Applicable Quality Guidelines for Statistical Products Issued
by the Bureau of Justice Statistics and Other Agencies

BJS followed numerous recommended data quality guidelines designed to help
ensure the accuracy and integrity of the Police-Public Contact Survey
products that it issued in 2001, 2002, 2005, and 2006 based on its 1999
and 2002 surveys. The product issuance guidelines were used to aid BJS's
efforts to review, approve, and disseminate these statistical products to
the public and others.

The guidelines were issued at various points in time by the following
organizations:

           o the National Research Council,
           o the Bureau of Justice Statistics,
           o the Department of Justice, and
           o the Office of Justice Programs.

In addition to reviewing the guidelines of these four organizations, we
also reviewed guidelines and directives issued by the Office of Management
and Budget (OMB). However, we did not specifically assess BJS's practice
with respect to following OMB's guidelines because OMB issued
governmentwide policy and procedural guidance to federal agencies that
called for agencies to develop their own implementing guidelines.

Table 4 shows the guidelines that were available at the time BJS's 1999
and 2002 Police-Public Contact Survey products were issued, and which
guidelines BJS was in a position to follow.

Table 4: Extent to Which BJS Followed Available Quality Guidelines for
1999 and 2002 Police-Public Survey Products

Source: GAO analysis of DOJ, OJP, and BJS data.

Legend: Guideline was available and BJS followed it fully. BJS was not in
the position to follow this guideline fully. BJS was not in the position
to follow this guideline at all. Guideline was unavailable during the
product issuance stage.

NRC National Research Council DOJ Department of Justice OJP Office of
Justice Programs

aForce or Threatened Force Used in Less Than 1 Percent of All
Police-Public Interactions (March 11, 2001).

bContacts between Police and the Public, Findings from the 1999 National
Survey (February 2001).

cCharacteristics of Drivers Stopped by Police, 1999 (March 2002).

dContacts between Police and the Public, Findings from the 2002 National
Survey (April 2005).

eCharacteristics of Drivers Stopped by Police, 2002 (June 2006).

Appendix III: Press Release Issued in 2001 Based on BJS's 1999
Police-Public Contact Survey

Appendix IV: BJS Processes for the Review, Approval, and Dissemination
of Police-Public Contact Survey Products

Review, Approval, and Dissemination Process for Reports Issued from
Police-Public Contact Surveys

Since the inception of the Police-Public Contact Survey in 1996, the BJS
Director has been the single noncareer appointee who has had a
decision-making role in BJS's review, approval, and dissemination
processes for reports. The BJS Director is a noncareer presidential
appointee subject to Senate confirmation. Figure 2 provides an overview of
the process followed by BJS in the review, approval, and dissemination of
Police-Public Contact Survey reports.

Figure 2: Review, Approval, and Dissemination Process for BJS Survey
Reports

As indicated by the figure, the BJS report author and supervisor prepare
the draft report for review and approval. The BJS Director reviews the
draft, requests any changes, approves the final draft, and transmits a
memorandum of notification through the Office of Justice Program's
Assistant Attorney General up the chain of command to the Attorney
General. The memorandum contains an abstract of the report, selected
survey findings, and a projected release date for the report. BJS sets the
release date for 30 days from the date that the Assistant Attorney General
signs the memorandum of notification. The report is posted to the Web site
at that time, or sooner if the date and time are specified in the
notification memo.

Review, Approval, and Dissemination Process for Press Releases Issued from
Police-Public Contact Surveys

In contrast to the process followed for survey reports, several noncareer
appointees are involved in the organizational review, approval, and
dissemination process, as shown in figure 3.

Figure 3: Review, Approval, and Dissemination Process for BJS Survey Press
Releases

As indicated in the figure, in addition to the Director of BJS, there are
three noncareer appointees within the Office of Justice Programs who
participate in the review and approval process--the Chief of Staff, the
Deputy Assistant Attorney General, and the Assistant Attorney General, and
at least one noncareer appointee within Department of Justice
headquarters: the Director of the Office of Public Affairs.

The BJS report author and supervisor jointly work with staff from the
Office of Justice Program's Office of Communications to prepare the press
release. The BJS director reviews the draft press release, requests any
changes, approves the final draft, and transmits the press release up the
chain of command to the Office of Justice Programs' Assistant Attorney
General for review and approval. Following approval by the Assistant
Attorney General, the Department of Justice's Office of Public Affairs
reviews the press release for clarity, and the BJS Director then verifies
that the information in the press release is accurate. The Department of
Justice's Office of Public Affairs is then responsible for disseminating
the press release to Congress, the media, and executive department press
offices, while BJS is responsible for disseminating the press release
through its Web site.

Appendix V: Changes in the Scope and Methodology of the Police-Public
Contact Surveys

BJS has conducted four Police-Public Contact Surveys as supplements to the
National Crime Victimization Survey. The first Police-Public Contact
Survey was conducted as a pilot in 1996. Three subsequent, more extensive
surveys were conducted at 3-year intervals: 1999, 2002, and 2005. Although
we do not discuss the 2005 survey in this report because no reports or
press releases have yet been issued from this survey, we present
information on the 2005 survey in this appendix because information is
available on this survey's scope and methodology. The scope of the
Police-Public Contact Surveys has consistently expanded over time, while
the methodology has remained generally consistent.

The Scope of the Police-Public Contact Survey Has Expanded with Each Additional
Survey

The pilot survey was designed to test whether the survey could be
effectively used as a supplement to the National Crime Victimization
Survey to collect data on (a) the types of contacts the public have with
the police, and (b) police use of force. To conduct this test, BJS
employed a representative sample of 6,421 U.S. residents. The pilot survey
yielded useful information on the various types of contacts the public had
with the police, and whether force was used by the police. However, the
sample size of the pilot survey was not sufficiently large for BJS to draw
inferences about the extent to which the population at large would report
that they experienced "excessive" use of force by police.

For its Police-Public Contact Survey in 1999, BJS increased the sample
size to a representative sample of 80,543 U.S. residents. The scope of the
survey was further enhanced by adding questions about traffic stops (the
most common form of public-police contact, as determined in the 1996 pilot
survey), and including a question on whether the police used excessive
force during any contact with the public. BJS officials told us that they
added the traffic stop questions, at least in part, to "address the
growing public concern about racial profiling in connection with traffic
stops."

In its 2002 survey, BJS expanded and refined its survey questions further.
Specifically, according to BJS officials, they added questions that would
help BJS estimate the extent to which U.S. residents nationwide would say
that (1) they were stopped by the police while driving, (2) they or their
vehicle were searched by the police without their permission during a
traffic stop, and (3) they were arrested as a result of the search. In
addition, BJS officials said that they added questions to estimate
differences, if any, among racial groups in their rates of traffic stops
at various times of the day, and whether police used force in situations
where persons were engaged in such behaviors as arguing with, cursing, or
disobeying police.

In 2005, according to BJS officials, the scope of the Police-Public
Contact Survey was further extended in several ways, including the
following: (a) respondents were asked whether they had been arrested for
driving under the influence of alcohol during the year (in order to make
comparisons with Federal Bureau of Investigation (FBI) arrest rates, so
that potential undercounting rates could be determined); (b) respondents
were permitted to group themselves into any combination of racial
categories (rather than choosing a single category) to better refine
respondent demographic status; (c) respondents were asked whether police
used force during any of their police contacts during the year, as opposed
to the more limiting question in 2002, which was directed only toward the
most recent contact with police; and (d) respondents were provided
open-ended response fields on the survey instrument to indicate any ways
they believed that the police had acted inappropriately toward them.

The Police-Public Contact Survey Methodology Has Remained Consistent Following
the 1996 Survey

The 1999, 2002, and 2005 Police-Public Contact Surveys have consistently
maintained a similar methodology. The methodological dimensions of the
surveys that have remained consistent are geographic coverage, target
population, sampling design, data collection method, sample size/response
rate, survey administration, and sample characteristics (as indicated in
table 5). To illustrate, all three surveys involved selecting nationally
representative, stratified, multi-cluster samples from the population of
U.S. residents 16 years of age or older.^1 The data were collected either
through face-to-face or computer-assisted telephone interviews. The
surveys were administered during the last 6 months of the year, and the
demographic characteristics of the samples were similar across time
periods.

The 1996 pilot survey differed in several ways from the subsequent three
surveys. Specifically, the pilot survey included residents younger than
16, included far fewer people than the subsequent surveys, and limited the
sampling to individuals who had participated in the last round of the
National Crime Victimization Survey. In addition, the percentage of
face-to-face interviews was lower, and the survey administration period
was shorter, and during a different time of the year, than in the other
three surveys.

^1 This included persons living in group quarters, but excluded
institutionalized persons and some members of the armed services.

Table 5: Police-Public Contact Survey (PPCS) Methodologies for the Four
Surveys: 1996, 1999, 2002, and 2005

Source: BJS data.

Note: Percentages may not sum to 100 percent because of rounding.

aPilot survey.

Appendix VI: Comments from the Department of Justice

Appendix VII: GAO Contact and Acknowledgments

GAO Contacts

Brian J. Lepore (202) 512-4523

Acknowledgments

In addition to the above, Evi. L. Rezmovic, Assistant Director; Ronald S.
Fecso, Chief Statistician; Jared A. Hermalin; Karen A. Jarzynka; Amanda K.
Miller; Amy L. Bernstein; Geoffrey R. Hamilton; Robert Alarapon; and Tracy
J. Harris made key contributions to this report.

(440500)

GAO's Mission

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

To Report Fraud, Waste, and Abuse in Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or (202)
512-7470

Congressional Relations

Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548

Public Affairs

Paul Anderson, Managing Director, [email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

www.gao.gov/cgi-bin/getrpt?GAO-07-340 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Brian Lepore at (202) 512-4523 or
[email protected].

Highlights of [35]GAO-07-340 , a report to congressional requesters

March 2007

BUREAU OF JUSTICE STATISTICS

Quality Guidelines Generally Followed for Police-Public Contact Surveys,
but Opportunities Exist to Help Assure Agency Independence

The Bureau of Justice Statistics (BJS), a statistical agency of the
Department of Justice's Office of Justice Programs, produces a recurring
national Police-Public Contact Survey documenting contacts between the
police and the public, including instances involving the use or threat of
force by police. BJS issues public reports and sometimes press releases
from survey results.

For reports and a press release issued from the 1999 and 2002 surveys (the
most recent available), GAO reviewed (1) the extent to which BJS followed
quality guidelines to ensure the accuracy and integrity of its
survey-related products, and (2) factors that affected whether and how BJS
followed available guidelines.

GAO reviewed applicable federal data quality guidelines, policy and
procedure documents, and interviewed current and former officials familiar
with BJS.

[36]What GAO Recommends

In a May 2006 report, GAO recommended that the Office of Management and
Budget (OMB) issue a directive to further safeguard the integrity of
federal statistical data and improve guideline clarity. Such a directive
could help address conflicting interpretations about the applicability of
quality guidelines. Since OMB is still working on this directive, GAO
makes no new recommendations in this report.

BJS followed nearly all quality guidelines for its 1999 and 2002
Police-Public Contact Surveys. Specifically, for the four public reports
issued from these surveys, BJS fully followed all data quality guidelines
available for reviewing statistical information, obtaining the approval of
key decision makers, and publicly disseminating information. These
guidelines were issued by the National Research Council, Department of
Justice, Justice's Office of Justice Programs, and BJS itself. GAO
believes that because BJS followed these guidelines, proper steps were
taken to help ensure the accuracy and integrity of the reports. BJS
followed 7 of the 10 quality guidelines available for the one press
release issued from its 1999 survey, but was not in a position to fully
follow 3 other guidelines for reasons discussed below.

Two key factors affected whether and how BJS followed quality guidelines.
The first concerned different interpretations about certain guideline
applicability. BJS considered its survey-related reports--but not its
press releases--to be statistical products covered by the National
Research Council's guidelines. BJS cited a lack of specificity in these
guidelines, which did not specifically state that they were applicable to
statistical agency press releases, as a basis for concluding that the
survey press releases need not conform to guidelines for statistical
products. We believe BJS's position was not unreasonable, and did not find
fault with the agency. However, we determined nonetheless that the single
press release issued from the 1999 survey was a statistical product, and
therefore believe the council's guidelines appropriately applied. Second,
certain noncareer appointees outside BJS may, in accordance with Justice
Department policy, make decisions about the review, approval, and
dissemination of press releases, and BJS press releases are jointly issued
with the Justice Department, with input from its Office of Justice
Programs. Both conditions can potentially affect BJS's independence. Owing
to these conditions, BJS was not, in our view, in a position to meet 3
council quality guidelines related to statistical agency independence,
including that it be able to issue statistical products without prior
clearance, and control the scope and content of its products.

Justice affirmed several of GAO's findings but disagreed with certain GAO
conclusions about the applicability of guidelines to a press release.
Justice's detailed comments and GAO's response are contained in the
report.

Guidelines BJS Followed for Products Issued from 1999 and 2002
Police-Public Contact Surveys

Source: GAO analysis of agency data.

aTotal guidelines were 24 across the different points in time; 23 was
maximum at any given time.

bFrom 1999 survey information.

References

Visible links
  25. http://www.gao.gov/cgi-bin/getrpt?GAO-06-607
  26. http://www.gao.gov/cgi-bin/getrpt?GAO-06-607
  27. http://www.gao.gov/cgi-bin/getrpt?GAO-06-607
  35. http://www.gao.gov/cgi-bin/getrpt?GAO-07-340
*** End of document. ***