Technical Assessment of Zhao and Thurman's 2001 Evaluation of the
Effects of COPS Grants on Crime (12-JUN-03, GAO-03-867R).	 
                                                                 
Community Oriented Policing Services (COPS) is a federal public  
safety program whose goals are to add officer positions to the	 
streets of communities nationwide and to promote community	 
policing. Since the program's inception in 1994, local law	 
enforcement agencies have received billions of dollars in grants 
to hire additional officers, acquire technology and civilian	 
personnel, and implement innovative crime-prevention programs. To
receive COPS grants, agencies are expected to implement or	 
enhance community policing strategies illustrating community	 
partnerships, problem solving, and organizational commitment.	 
Given the large expenditures of funds, it is important for policy
makers, among others, to have sound information on the		 
effectiveness of the COPS program in reducing crime. Congress	 
asked us to review one evaluation of the effectiveness of the	 
COPS program--by Zhao and Thurman--and to render an assessment of
its quality. In this report, we provide information on the extent
to which this particular study's conclusions are supported by the
data the researchers used and the analyses they conducted. GAO	 
statisticians and methodology specialists reviewed the study	 
using standard and widely accepted statistical and social science
research principles.						 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-03-867R					        
    ACCNO:   A07197						        
  TITLE:     Technical Assessment of Zhao and Thurman's 2001	      
Evaluation of the Effects of COPS Grants on Crime		 
     DATE:   06/12/2003 
  SUBJECT:   Crime prevention					 
	     Federal grants					 
	     Law enforcement agencies				 
	     Police						 
	     Program evaluation 				 
	     Community Oriented Policing Services		 
	     Program						 
                                                                 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-03-867R

Page 1 GAO- 03- 867R COPS Evaluation

United States General Accounting Office Washington, DC 20548

June 13, 2003 The Honorable F. James Sensenbrenner Chairman Committee on
the Judiciary House of Representatives

Subject: Technical Assessment of Zhao and Thurman's 2001 Evaluation of the
Effects of COPS Grants on Crime

Dear Mr. Chairman: Community Oriented Policing Services (COPS) is a
federal public safety program whose goals are to add officer positions to
the streets of communities nationwide and to promote community policing.
Since the program*s inception in 1994, local law enforcement agencies have
received billions of dollars in grants to hire additional officers,
acquire technology and civilian personnel, and implement innovative
crimeprevention programs. To receive COPS grants, agencies are expected to
implement or enhance community policing strategies illustrating community
partnerships, problem solving, and organizational commitment. Given the
large expenditures of funds, it is important for policy makers, among
others, to have sound information on the effectiveness of the COPS program
in reducing crime. You asked us to review one evaluation of the
effectiveness of the COPS program* by Zhao and Thurman 1 *and to render an
assessment of its quality. In this report, we provide information on the
extent to which this particular study*s conclusions are supported by the
data the researchers used and the analyses they conducted. GAO
statisticians and methodology specialists reviewed the study using
standard and widely accepted statistical and social science research
principles.

Our assessment of Zhao and Thurman*s work cannot be construed to be an
assessment of the COPS program itself. Since we have not reviewed the
quality of any other COPS evaluation or conducted an independent
evaluation of the program, we have no basis to judge whether or not the
program has been effective in achieving its stated goals. It is also
important to note that these types of aggregate level analyses that are
intended to assess program effectiveness are extremely difficult to
execute successfully, in part, because direct measures of important
variables are not always available.

1 Zhao, J. and Thurman, Q. A National Evaluation of the Effect of COPS
Grants on Crime from 1994 to 1999 (Dec. 2001).

GAO- 03- 867R COPS Evaluation Page 2 We conducted our review of Zhao and
Thurman*s study during a 3- week period in

May 2003. In addition to reviewing Zhao and Thurman*s December 2001
report, we reviewed a November 2002 journal article by Zhao, Scheider, and
Thurman based on the same study, 2 reviewed a May 2003 draft of an updated
COPS study by the same authors, and discussed data and statistical issues
with these researchers in a telephone call on May 27. In this report, we
focus the majority of our comments on Zhao et al. *s earlier COPS study
(reported in December 2001 and November 2002). We discuss differences
between the earlier study and the May 2003 follow- up study in a section
at the end of this report. For ease of presentation, we refer to their
original work as the *2001 study.*

Background

The Public Safety Partnership and Community Policing Act of 1994 3
authorized $8.8 billion in grants to be awarded to law enforcement
agencies for fiscal years 1995 to 2000. Focused on crime- prevention, the
act required, among other things, that half the grants go to law
enforcement agencies serving populations of 150,000 or less. The act also
required that grantees not supplant state and local funding, but rather
use the federal funds for additional law enforcement beyond what would
have been available without a grant. The Attorney General created the
Office of Community Oriented Policing Services to administer the grant
programs and advance community policing across the country.

The COPS office is tasked with promoting community policing through a
variety of types of grants, including:

Hiring grants, which are used to fund the hiring of additional police
officers. Through its Universal Hiring Program, the COPS program provides
funding directly to local, state, and tribal jurisdictions. The funding
provides up to 75 percent of the salaries and benefits for new officers
for 3 years up to a maximum of $75,000 per officer. According to the COPS
Office, 71,192 officers were funded and 63,592 officers were hired through
hiring grants as of July 26, 2002. The COPS Office estimated that hiring
grant awards totaled about $5.6 billion as of June 3, 2003.

Making Officer Redeployment Effective (MORE) grants, which are used to
fund up to 75 percent of the total cost of acquiring new technologies and
equipment and the hiring of civilians for 1 year. These are intended to
allow police to spend more time patrolling the streets instead of on
administrative and support tasks. According to the COPS Office, 24,436
full- time equivalent staff were redeployed through MORE grants as of July
26, 2002. The COPS Office estimated that MORE grant awards totaled about
$1.3 billion as of June 3, 2003.

Innovative grants, which are used to promote innovative approaches to
solving crime in specific areas such as domestic violence and drug abuse.
The COPS

2 Zhao, J., Scheider, C. and Thurman, Q. Funding Community Policing to
Reduce Crime: Have COPS Grants Made a Difference? Journal of Criminology &
Public Policy, Nov. 2002 (vol. 2, no. 1). 3 P. L. 103- 322.

GAO- 03- 867R COPS Evaluation Page 3 Office estimated that innovative
grant awards totaled $820 million as of June 3,

2003.

Results in Brief

Our review of the 2001 study on the effects of COPS grants on crime rates
indicated that the results of their study should be viewed as
inconclusive. We believe that the study*s limitations in data and methods
are significant and preclude meaningful interpretation of the results. We
cannot agree with Zhao et al. that their 2001 study shows that some COPS
grants (hiring and innovative) significantly reduced crime because, among
other things, important variables were omitted from their analyses, the
analytic models were misspecified, and the sample of cities included in
the study was limited. Further, we have concerns about the use of outdated
census data for control variables. Aside from concerns about data and
methods, we question whether the statistically significant crime
reductions that Zhao et al. found are significant in a practical sense.

While we cannot agree with the Zhao et al. *s conclusions, we also cannot
say that COPS grants are ineffective in reducing crime. A program*s
effects and researchers* ability to design studies that will accurately
measure those effects are two different things. Other studies, which we
have not reviewed, may have taken a more rigorous approach to assessing
the effects of COPS grants on crime. We believe that a more rigorous study
would incorporate, among other things, more reliable, valid, and complete
measures; a more complete and generalizable sample of cities; and
wellspecified analytic models.

In written comments on a draft of this report, the Department of Justice*s
COPS Office and Zhao and Thurman generally disagreed with our findings.
The comments reflected the view that our standards for critiquing Zhao et
al. *s work were too stringent, that we were incorrect in concluding that
their statistical models were misspecified, and that the statistical
controls incorporated into their analytic models were sufficient to
account for the types of missing data we identified as limitations of the
study. In our response, we address why we continue to believe that these
limitations render the findings of this particular study inconclusive.

Summary of Analysis and Results of the 2001 Study

The 2001 study presented a statistical analysis of the effects of three
types of grants* hiring, MORE, and innovation* on the reported rates of
violent and property crimes over a 5- year period across 6,100 U. S.
cities that received COPS grants. The analysis, which looked separately at
cities with populations greater than 10,000 and those with populations
less than 10,000, sought to determine how the reported crime rates varied
as a function of the amount of COPS funds received.

The variables used in the 2001 study are presented in table 1, along with
the averages and standard deviations for these variables across all cities
included in the analysis.

GAO- 03- 867R COPS Evaluation Page 4

Table 1: Averages Across All Cities from 1994 to 1999 Average Standard
deviation Dependent variables (1995- 1999) Violent crime rate (per 100,000
population) 769.63 674.50 Property crime rate (per 100,000 population)
5,016.39 2,820.74 Independent variables (1994* 1998) Hiring grants (per
resident) $2.38 3.72 Innovative grants (per resident) $0.42 2.45 MORE
grants (per resident) $0.65 1.45

Demographic control variables % unemployment (1994- 1998) 4.97 2.17 %
minority (1990 census) 30.40 23.32 % single parent households (1990
census) 10.59 4.09 % young people ages 15- 24 (1990 census) 15.43 4.59 %
home owners (1990 census) 56.92 14.62 % people in same household for 5 or
more years (1990 census) 50.66 10.03 Note: Zhao et al. used weighted
averages to estimate the means of COPS grants and control variables.
Source: Zhao et al., December 2001 and November 2002.

Zhao et al. found that hiring grants significantly reduced reported
violent and property crimes in larger cities, but significantly increased
those rates in smaller cities. They speculated that the addition of police
officers in smaller cities could produce an increase in reported crime
because, among other things, the increased interaction between police and
the community can help residents feel more comfortable and willing to
report crimes. Innovative grants also significantly reduced the reported
violent and property crime rates in larger cities, but had no significant
effect in smaller cities. MORE grants had no discernable effect in larger
cities, or on reported violent crimes in smaller cities, but they
significantly increased the rates of reported property crimes in the
smaller cities. Zhao et al. concluded that innovative programs, which are
targeted at specific crime problems or jurisdictions, had the strongest
effect on reducing reported crime rates. They also observed that *crime
reduction in the United States is not a unitary phenomenon* in light of
the different effects found in large versus smaller cities.

Our Review Indicated Several Problems with the 2001 Study

Our review revealed several problems with the 2001 study that cast doubt
on the validity of the conclusions about the effectiveness of COPS grants.
The problems we identified pertain to Zhao et al. *s interpretation of
their findings, omission of important variables from the analysis,
misspecifications in the analytic models used, and sample selection
issues. We also had some concerns about the outdated nature of census data
used as control variables in the 2001 study.

The Meaning of the Study*s Findings Can Be Interpreted Differently The
finding that COPS grants exerted different effects on crime patterns in
large versus small cities led the researchers to observe that crime
reduction is not a unitary phenomenon. While this may be the case, one can
also conclude that the study*s findings are equivocal, inconsistent, and
inconclusive.

Further, while the crime- reducing effects that Zhao et al. found for
hiring and innovative grants may have been statistically significant, they
could also be

GAO- 03- 867R COPS Evaluation Page 5 characterized as quite small in a
practical sense. 4 Table 2 demonstrates this point by

presenting a summary of Zhao et al. *s estimates.

Table 2: Estimates of the Effects of Three Types of COPS Grants, from Zhao
et al. (2001) City type

Crime type

Hiring Grants

Innovative grants

MORE grants

>10,000 Violent -5.26* -12.93* -0.11 Property -21.63* -45.53* -1.52 1,000
- 10,000 Violent 0.83* 1.06 2.48

Property 8.97* 11.98 31.20* All cities 1000+ Violent -1.86 -12.26* 0.28

Property -10.44 -43.85* -0.28 Note: An asterisk (*) denotes that the
estimated effect was statistically significant. Source: GAO summary of
Zhao et al. 2001 data.

The coefficients in table 2 indicate how much each grant dollar spent per
person in each city affected the rates of reported violent and property
crimes; in other words, how much of a change in the reported violent and
property crime rates we might expect if funding were increased by one
dollar per resident. As shown in table 1, the average annual COPS
innovative grant across all cities amounted to $0.42 per person, and the
average rates of reported violent and property crimes, respectively, were
about 770 and 5,016 per 100,000. These coefficients imply that if COPS
funding in larger cities for innovative grants were doubled (from $0.42 to
$0.84 per person), we would expect the violent crime rate to go down by
0.7 of 1 percent (from 770 to 765 per 100,000). 5 We would expect the
reported property crime rate to go down by 0.4 of 1 percent (from 5,016 to
4,997 per 100,000). 6 As small as the effects are, there are reasons to
question whether they accurately represent the expected returns on such an
investment, and these reasons are listed below in general order of
importance.

Important Variables Were Omitted from the Analysis While dummy variables
were used in the 2001 study to control for unmeasured differences across
counties, the only city- level variables in the analysis that were
measured and explicitly controlled in the models of estimated COPS grant
effects were (1) the 1994 crime rate and (2) the six demographic variables
shown in table 1. Most conspicuously absent from these models is a measure
of expenditures on police that were not derived from COPS grants. The
researchers told us they did not include

4 Statistical significance means that the observed effect does not result
from chance alone. The number of observations in a sample can be an
important determinant of statistical significance, with larger sample
sizes frequently being associated with statistically significant findings.
Zhao et al. *s 2001 study consisted of 36,605 observations, making it
possible that statistically significant effects could have been found even
when they were small on a practical level. 5 This is calculated as
follows: From table 2, we see that in cities larger than 10, 000, each
dollar of innovative grant funding was associated with a decrease of 12.93
violent crimes. $0.42 is 42 percent of 1 dollar, and 42 percent of 12. 93
crimes equals 5.4. This represents the decrease in the expected crime rate
as innovative grant funding increased by $0. 42 per person. If the violent
crime rate were 770 per

100,000 population, doubling the $0.42 innovative grant expenditure per
person would reduce the violent crime rate by 5, or to about 765 per
100,000 population. 6 The mean offered in Zhao and Thurman is a weighted
average for all cities and only approximates the

mean for large cities. Because of that, and the severe skew in the
distribution of average grant amounts across cities (note the standard
deviations in table 1), this may not be a very accurate way to estimate
the effect size. The skew in the distribution of grant amounts also
suggests that it might have been preferable to transform (using
logarithms) those amounts prior to the analyses.

GAO- 03- 867R COPS Evaluation Page 6 non- COPS funded police expenditures
because such data are not available. Because

the COPS program supports only a portion of police agency budgets,
however, we believe the absence of any control for state and local
expenditure to be a serious weakness.

Police departments that received COPS grants may have also received grants
from other programs (such as Byrne grants). T Th he es se e a am mo ou un
nt ts s c co ou ul ld d b be e c co or rr re el la at te ed d w wi it th h
C CO OP PS S f fu un nd di in ng g a am mo ou un nt ts s. . F Fo or r e ex
xa am mp pl le e, , i if f a a d de ep pa ar rt tm me en nt t i is s p pr
ro of fi ic ci ie en nt t i in n g ge et tt ti in ng g C CO OP PS S f fu
un nd di in ng g, , i it t m ma ay y b be e p pr ro of fi ic ci ie en nt t
i in n g ge et tt ti in ng g o ot th he er r f fu un nd di in ng g, , a as
s w we el ll l. . W Wi it th ho ou ut t s se ep pa ar ra at ti in ng g C
CO OP PS S f fu un nd di in ng g f fr ro om m o ot th he er r t ty yp pe
es s o of f f fu un nd di in ng g t th ha at t p po ol li ic ce e a ag ge
en nc ci ie es s r re ec ce ei iv ve e, , w we e c ca an nn no ot t b be e
s su ur re e h ho ow w m mu uc ch h o of f a an n e ef ff fe ec ct t C CO
OP PS S g gr ra an nt ts s b by y t th he em ms se el lv ve es s h ha av
ve e o on n c cr ri im me e r re ed du uc ct ti io on n. . The study also
lacked any measure of city size beyond the dichotomy (i. e., population

smaller or larger than 10,000) used to split the sample of cities prior to
model estimation. Other omitted measures include such socioeconomic
variables as per capita income and percent male. County dummy variables
controlled for some of the problems associated with omitted variables, but
they would not control effectively for variables that differed across
cities within counties, or variables that changed within counties over
time. For example, if state and local expenditures on police varied across
cities in a given county, using dummy variables to represent counties
would not take these differences or changes into account in estimating the
independent effect of COPS grants.

Misspecifications in the Analytic Models The models employed in Zhao et
al. *s analyses are two- factor fixed effects models that employ 2,674
dummy variables representing the counties and 5 dummy variables
representing the years included in the analysis. These dummy variables
controlled for unmeasured variability across counties and over time, and
they supplemented the controls for prior rate of crime and the 6
demographic variables described above. These models and the estimation
procedures they involve are fairly sophisticated, but since the data on
crime rates and COPS funds were measured at the city level, we believe
that unmeasured variability would have been more effectively controlled
had dummy variables been used to distinguish cities, instead of the
counties in which the cities were located. 7 With dummy variables
representing counties, any unmeasured and systematic variability across
cities within the same county remained uncontrolled and a potential source
of bias in the parameters representing the effects of the COPS grants
estimated in the models. Sample Selection Limited Z Zh ha ao o e et t a al
l. .* *s s a an na al ly ys si is s i is s f fo oc cu us se ed d o on nl
ly y o on n COPS program grants used to fund local city

police departments. Their report indicates that other law enforcement
agencies, such as state and county police agencies; sheriffs* offices;
campus police; and special purpose law enforcement agencies such as court,
forest, and park police, among others, were excluded from their study.
Since these other agencies accounted for

7 A footnote in the 2001 study indicates that the researchers conducted
initial analyses using city dummy variables. However, they ultimately
decided to use county dummy variables, and all the report findings are
derived from statistical models that included county rather than city
dummy variables.

GAO- 03- 867R COPS Evaluation Page 7 4,891 (or 40 percent) of the 12,070
law enforcement agencies receiving COPS grant

awards from 1994 to 1998, Zhao et al. *s study omitted a large portion of
COPS grant recipients. Further, there is likely to be considerable overlap
across jurisdictions receiving COPS grants (cities within counties, campus
police within city jurisdictions). 8 According to Zhao et al., the sample
of cities included in their study represented a

subset of 6,100 of the 7,179 cities whose local city police departments
received COPS grants at some point during the period from 1994 to 1998. 9
The researchers deleted 535 cities with populations less than 1,000, and
544 cities that lacked Uniform Crime Reports (UCR) data. 10,11 Four states
(Delaware, Illinois, Kansas, and Montana) contributed only 8 cities
between them owing to missing UCR data. These omissions may have affected
the study*s results. Of greater concern, however, is the omission of the
potentially large number of cities that received no COPS funding at all.
We believe that cities with no COPS funding should have been included in
the analyses in order avoid sample selection problems and ensure that the
results were generalizable across all cities. 12 Concerns about Measures
of Demographic Variables. While the rates of violent crimes and property
crimes were measured and allowed to

vary in each of the 5 years from 1994 to 1998, in the 2001 study at least
5 of the 6 demographic variables were derived from the 1990 census and
fixed at their 1990 levels. We believe the 1990 figures would be a poor
basis for estimates because in many cities, the demographic
characteristics of residents in 1990 would be expected to be quite
different from those in the mid- to late- 1990s; and in all cities, these
timeinvariant estimates would fail to account for the significant
demographic changes

8 For example, if the city of College Park, MD, received a COPS grant and
the University of Maryland campus police (located in College Park)
received a separate COPS grant, their joint impact on the city*s crime
rates would not be included in this analysis. 9 In the analysis, the crime
rates from 1995 to 1999 were intentionally lagged a year to allow these
agencies to receive and deploy these funds.

10 UCR is a nationwide database of police statistics consisting of crime
data voluntarily reported to the Federal Bureau of Investigation by nearly
17, 000 city, county, and state law enforcement agencies. UCR data form
the basis for a Crime Index, which is used to gauge fluctuations in the
nation*s overall volume and rate of crime. The offenses included in the
"violent crime" category are murder and nonnegligent manslaughter,
forcible rape, robbery, and aggravated assault. The offenses included in
the "property crime" category are burglary, larceny- theft, and motor
vehicle theft, and arson. 11 In personal discussions with the researchers,
we learned that their 2001 published study contained an error related to
missing data. Specifically, the researchers had intended to eliminate
cities from

their analysis if crime data were missing for even a single month of the
year. However, the dataset they obtained did not uniformly distinguish
between missing data and "zero" reported crimes. In those cases, the
analysis would have produced an underestimate of the 12- month crime rate.
After publishing their results, the researchers corrected these data
errors and reanalyzed the dataset. They told us that the revised results
did not differ substantially from those published. Time limitations
prevented us from assessing the revised results. 12 In a November 2002
publication in the Journal of Criminology and Public Policy, Zhao et al.
explained that their analyses omitted cities without COPS grants because
of concern that including

these cities would produce a downward bias in their estimation of COPS
program effects. They said this is because crime was decreasing across the
board between 1994 and 1998 in cities with and without COPS grants. We
disagree with their rationale. Since Zhao and Thurman controlled for the
baseline rate of crime by including the 1994 rate in their model as a
control variable, cities with COPS

grants would presumably have a higher rate of decrease than cities without
COPS grants. We continue to believe that Zhao and Thurman*s estimates of
COPS program effects were biased as a result of omitting cities that did
not receive COPS grants.

GAO- 03- 867R COPS Evaluation Page 8 that may have occurred over time. It
was not entirely clear to us how the

unemployment data derived from the U. S. Department of Labor Statistics
for the years 1994 to 1998 were used in these models. However, it too
represented a potentially poor measure of unemployment in many cities.
This is because data are not available for cities with populations less
than 25,000, and county- level rates were used for those cities instead.

Comments on Zhao et al. *s Draft Updated COPS Study

Our previous comments pertain to the unpublished 2001 study by Zhao and
Thurman and the 2002 publication by Zhao, Scheider, and Thurman which
resulted from the study and which was virtually identical to the
unpublished study in terms of the primary results that were reported. That
study, as we noted previously, relied on data from 6,100 cities for which
COPS grant data for the years 1994 to 1998, and UCR crime data for the
years 1994 to 1999, could be obtained. After reviewing that work, we
received a draft updated report from those authors that re- estimated the
effects of COPS grants on crime rates using data from an additional year
(e. g., COPS grant data

for 1994- 1999 and UCR data for 1994- 2000) 13 and models that
incorporated updated 2000 census data and allowed the demographic
characteristics to vary over time. While these newer estimates, like those
in the 2001 and 2002 reports, were derived from models that used county
dummy variables, we also received from the researchers additional
information that showed how results compared when they used dummy
variables representing cities in place of the county dummy variables.

These updated results are shown in table 3, along with the results from
the researchers* prior study. The researchers have asserted, both in the
draft updated report and in their conversations with us, that these
updated results are largely consistent with the previously published
results, and in a general sense we agree with this. That is, with or
without the newer data, regardless whether demographic factors are allowed
to vary, and regardless whether county or city or dummy variables are
used, both studies found (1) no evidence that COPS grants have diminished
the crime rates in cities with populations less than 10,000, and (2) some
evidence that they have done so in larger cities. Apart from this general
observation, however, the results of the two studies are inconsistent in
that the size and significance of some of the estimated effects of COPS
grants differed under alternative specifications. For example, when
updated data and the time varying covariates were used, the estimated
effects of innovative grants on violent and property crimes in large
cities declined in size to less than half of the prior estimates, while
the effects of MORE grants increased more than 10- fold, and became
statistically significant in the case of property crimes. 14 13 One
difference in the crime rates analyzed in the two studies was that arson
was included as a property crime in the newer study, but not in the 2001
study.

14 The authors provided us with additional information from their follow-
up study on the analytic results obtained when they used dummy variables
to represent cities instead of counties. They found that in large cities,
the estimated effects of hiring grants on violent crimes doubled, the
estimated effects of MORE grants doubled and became statistically
significant, and the effect of innovative grants became statistically not
significant. The effect of MORE grants on property crimes remained
significant in large cities when city dummy variables were used, but
diminished to half the size that was estimated by a model that used county
dummy variables.

GAO- 03- 867R COPS Evaluation Page 9 Since these newer results have not
been finalized, it is premature for us to make a

final determination of their validity and usefulness. The researchers are
to be commended for the considerable effort they made to determine how
reliable and robust the estimated effects of the different COPS grants
were over time, and under alternative specifications. Nonetheless, the
newer study that we reviewed had some of the same limitations as the 2001
study. Specifically, the newer study (1) omitted important variables,
including measures of expenditures on police apart from COPS grants, (2)
omitted a large number of cities that did not receive COPS grants, and (3)
did not control for the effect of city size on crime in a more refined
fashion than dichotomizing city populations. Our review of the results of
the newer analyses has not fundamentally altered our view that the
estimated effects of COPS grants on reported violent and property crimes
were small in a practical sense. Again, it is important to note that this
does not imply that COPS grants do not have positive effects in reducing
crime; only that it is hard to reach firm conclusions about their effects
from the particular studies we reviewed. Our technical assessment of Zhao
et al. *s work is not a commentary on the effectiveness of the COPS
program.

Table 3: Zhao et al. *s Estimates of the Effects of Three Types of COPS
Grants with Dummy Variables Representing Counties in (a) 2001 Study Using
1994- 1999 Data and (b) Draft Updated Study Using 1995- 2000 Data

City type

Crime type

COPS study Hiring grants

Innovative grants

MORE grants

>10,000 Violent 2001 study -5.26* -12.93* -0.11 Draft updated study -5.49*
-5.31* -2.00

Property 2001 study -21.63* -45.53* -1.52 Draft updated study -25.22*
-20.65* -21.47*

1,000 - 10,000 Violent 2001 study 0.83* 1.06 2.48 Draft updated study
1.47* 0.60 2.92

Property 2001 study 8.97* 11.98 31.20* Draft updated study 7.91* 1.30
30.51* Note: An asterisk (*) denotes that the estimated effect was
statistically significant. Source: GAO summary of Zhao et al. *s 2001 and
updated studies.

Agency Comments and Our Evaluation

The Acting Deputy Director of the COPS Office and Professors Zhao and
Thurman provided us with written comments on a draft of this report. Their
comments contained a number of points that disagreed with the limitations
we identified in our assessment. The comments reflected the view that we
(1) applied an overly stringent standard to the study*s design and failed
to consider the fact that this study was better and more comprehensive
than previous research on the subject; (2) were incorrect in concluding
that their statistical models were misspecified and did not control for
the effect of missing police expenditure data; (3) were ill- advised in
stating that including data on cities* access to grants other than COPS
grants would have improved the estimates of COPS grant effects; (4) were
ill- advised in stating that including data on such socioeconomic
variables as percentage of the population that is male would have improved
the estimates of COPS grant effects; (5) were incorrect in stating that
including data on COPS- funded jurisdictions within cities, such as

GAO- 03- 867R COPS Evaluation Page 10 university police, would have
improved the estimates of COPS grant effects; and (6)

were ill- advised in stating that including police departments in cities
that did not receive COPS funding would have improved the estimates of
COPS grant effects. We continue to disagree with the researchers on these
key points and discuss our reasons below.

First, with respect to the assertion that our standards were too high and
that we did not consider the advances made by this study, we would
reiterate that the purpose of our assessment was to determine the extent
to which the conclusions of this particular COPS study were supported by
the data used and analyses conducted. Because we were asked to review this
single study and did not have time to review any others, we cannot comment
on whether and how this study*s approach to evaluating the effectiveness
of the COPS program may have been an incremental improvement over other
similar efforts. We acknowledge in the introduction to this report that it
is extremely difficult to assess program effectiveness via aggregate level
analyses. We also believe that the researchers should be commended for
their efforts, which involved merging data on more than 6,000 towns and
cities over a multi- year period from four different sources and using
sophisticated methods to analyze those data under a variety of
specifications. But, in our estimation, the problems that we identified
with the research make the results more suggestive than conclusive.

Second, with respect to the assertion that the statistical models were
both correctly specified and sufficiently controlled for the effect of
missing data on police expenditures, we do not believe this was the case.
Zhao et al. believe that we are unjustifiably critical of their having
used county rather than city dummy variables in estimating the effects of
COPS grants on crime rates. They point out that they ran both their 2001
and 2003 analyses using both city and county dummy variables, and the
results of the two types of analyses did not differ substantially. 15
While the models incorporating county or city dummy variables do, as the
authors assert, explain a sizable portion (between 64 percent and 86
percent) of the variation in reported crime rates across cities over time,
this is not surprising and is largely attributable to the very large
number of dummy variables included in their models. The proportion of
variance explained, however, does not necessarily imply that the estimates
of the effects of COPS grants were unbiased. The authors, in our opinion,
are mistaken in their claim that the use of dummy variables controls for
the effects of all unmeasured differences between cities and over time.
That is, the county dummy variables do not control for unmeasured
differences between cities within counties, and even the combination of
city and year dummy variables do not control for differences within cities
over time, unless the changes in all cities are similar. Crime rates in
cities did not show similar changes over time, 16 however, and there are
many

15 The researchers noted that they recently collected original police
expenditure data from 55 of the largest police departments and found that
including these data in the statistical models showed that they had
virtually no effect on their estimates of the effects of COPS funding. We
appreciate the

difficulty of obtaining police expenditure data for large cities and
endorse efforts to marshal supporting evidence from a sample of those
cities. However, we have not seen the results of these analyses and have
no basis to judge how representative these 55 cities are of large cities
in general, or whether the estimated effects of COPS grants from the 55
cities are generalizable to larger cities generally. 16 Bureau of Justice
Statistics data on 62 local police departments serving cities with a
population of

250,000 or more revealed a high degree of change in violent and property
crime rates within the same city over time. For example, New York*s
reported violent crime rate dropped by 57 percent between

GAO- 03- 867R COPS Evaluation Page 11 factors that can change in cities
from one year to the next in ways that might affect

crime rates. For example, fluctuations in local, state, and other
expenditures on police could produce changes in crime rates within cities
over time, and the failure to control for such factors can seriously bias
the estimates of the effects of COPS grants.

Third, with respect to the assertion that omitting data on cities* access
to grants other than COPS grants probably did not affect the results, we
are not convinced. We agree that data on grants that cities receive are
not readily available. However, we believe that information on at least
major grant programs could be obtained from the Office of Justice
Programs. To the extent that cities that receive COPS grants may be more
likely to receive other types of grants, omitting consideration of other
grants that are also targeted at reducing crime may lead to an
overestimation of the effects of COPS grants. By restricting their
attention to COPS grants awarded to city and local police, the researchers
investigated the effects of only a portion of all COPS grants. They
ignored the effects of other grants and of state and local expenditures,
generally, and therefore increased the potential for obtaining biased
estimates of COPS grant effects.

Fourth, with respect to the assertion that the study*s results were not
impaired by the omission of such socioeconomic variables as percentage of
the city population that is male, we disagree with the researchers that
this is not problematic. They assert that (1) the dummy variables in their
statistical models controlled for the effects of socioeconomic variables
other than those in their analyses, (2) a city*s male

population should not significantly affect the estimated effects of COPS
grants on crime, (3) the socioeconomic variables included in the analyses
were sufficient and grounded in widely accepted social disorganization
theory, and (4) problems of multicollinearity 17 could have arisen had
they included additional socioeconomic variables. As with police
expenditures, we maintain that data on factors affecting crime rates that
vary across cities and over time should be included in analyses, and may
not be sufficiently controlled by statistical models that use dummy
variables to control for unmeasured differences. While we do not know
whether and how COPS grant amounts to cities may be associated with the
socioeconomic characteristics of city residents, the literature indicating
a gender difference in crime is extensive. 18 To the extent that
socioeconomic characteristics affect crime rates, and to the extent that
cities that received COPS grants may have different socioeconomic
characteristics, we believe it would be wise to incorporate such variables
into models to lessen any potential bias in the estimates of the COPS
grants on crime. Since this study was intended to be an evaluation of the
effects of COPS grants on crime and

1990 and 2000, while Nashville*s rate increased by 29 percent during that
same time period. Similarly, New York*s reported property crime rate
dropped by 60 percent, while Nashville*s rate increased by 22 percent.
(Police Departments in Large Cities, 1990- 2000. Department of Justice,
Bureau of Justice Statistics, May 2002). 17 Multicollinearity means that
the independent variables are highly correlated. If this occurs, it is
impossible to distinguish between them in estimating their effects on the
dependent variable. 18 For example, a Bureau of Justice Statistics study
indicated that men comprised 93 of the state prison

population in 2001; 93 percent of the federal prison population in 1997;
and 90 percent of the local jail population in 1996 (http:// www. ojp.
usdoj. gov/ bjs/ crimoff. htm# findings). Another study reported that in
1960, 1975, and 1990, men were arrested at much higher rates than women
for all crime categories

except prostitution (Steffenmeier, D. and Allen, E., *Gender and Crime:
Toward a Gendered Theory of Female Offending,* Annual Review of Sociology,
1996, vol. 22, pp. 459- 87).

GAO- 03- 867R COPS Evaluation Page 12 not as a test of social
disorganization theory, we do not believe that limiting the

socioeconomic control variables to those dictated by this particular
theory was warranted. Finally, with 36,000 observations in their study, we
do not believe that multicollinearity would have been a problem had
additional socioeconomic variables been included in the analyses.

Fifth, with respect to the assertion that including data on COPS- funded
agencies within cities would not have improved the estimates of COPS grant
effects, we continue to believe that this cannot be known. Zhao and
Thurman state that there is no meaningful way to include such agencies*
for example, park and university police* in their statistical models
because the jurisdictions overlap. They note that it was neither necessary
nor possible to estimate the effects of such agencies on crime rates
because they report crime incident data to the Federal Bureau of
Investigation separately and because census data for them are not readily
available. We maintain that by restricting their attention to crimes
reported to local and city police departments, the researchers are
investigating the effect of only a portion of all COPS grants and are
looking at only a subset of all crimes reported. Again, we do not know
whether these restrictions result in an overestimate or underestimate of
the effect of COPS grants on crimes, but they can potentially bias their
estimates. We acknowledge that data may not be readily available for such
an analysis, but that does not mean they cannot be collected or that they
are unimportant.

Sixth, with respect to the assertion that including data on police
departments in nonCOPS funded cities would not have improved the estimates
of COPS grant effects, we continue to disagree. Zhao and Thurman note that
because small cities are more likely than large cities to not receive COPS
funding, including nonfunded agencies in their analysis could bias the
findings towards showing an effect of COPS grants. It is our view that
missing cases, except when they are missing at random, should be regarded
as problematic. The 6,100 agencies that Zhao et al. analyzed represented
about 85 percent of the COPS- funded city and local police departments, 51
percent of the total number of COPS- funded agencies, and 36 percent of
the agencies that participate in the UCR system. Some of these exclusions
may have been unavoidable, but their cumulative impact is likely to be
non- negligible.

We do not know how or to what extent the findings that Zhao et al.
obtained would change if the limitations that we identified in our
assessment were successfully resolved. We do know, however, that while
Zhao et al. may have performed the most sophisticated and advanced
research on the topic, drawing inferences or making policy decisions about
COPS grant effects from this work are unwarranted at this time. Indeed,
Zhao and Thurman are themselves continuing this work, an indication that
they also believe refinements are needed.

The comments from the COPS office and the researchers are reproduced in
the enclosure to this report. The COPS Office also provided us with
technical comments, which we incorporated in the report as appropriate.

-- -- -- -- -- -- As agreed with your office, unless you publicly announce
the contents of this report earlier, we plan no further distribution of it
until 30 days from the date of this report.

GAO- 03- 867R COPS Evaluation Page 13 We will then send copies of the
report to the Attorney General and will make copies

available to others upon request. In addition, the report will be
available at no charge on GAO*s web site at http:// www. gao. gov.

If you have any questions about this report, please contact me at (202)
512- 8777. The key contributors to this report were David Alexander, Carl
Barden, Evi Rezmovic, and Douglas Sloane.

Sincerely yours, Laurie E. Ekstrand Director, Homeland Security

and Justice Issues Nancy Kingsbury Managing Director, Applied Research

and Methods Issues Enclosures - 2

Enclosure I Page 14 GAO- 03- 867R COPS Evaluation Enclosure I

Enclosure II Page 17 GAO- 03- 867R COPS Evaluation Enclosure II

Enclosure II Page 23 GAO- 03- 867R COPS Evaluation (440213)
*** End of document. ***