Violence Against Women Office: Problems with Grant Monitoring and
Concerns about Evaluation Studies (16-APR-02, GAO-02-641T).	 
                                                                 
The Violence Against Women Office (VAWO) was created in to lead  
the national effort to end violence against women, including	 
domestic violence, sexual assault, and stalking. Since its	 
inception, VAWO has grown both in the number of discretionary	 
grants awarded and dollars awarded for those grants.		 
Unfortunately, monitoring activities and impact evaluation data  
provide little basis to assess program results. Both VAWO and the
Office of Justice Programs made a commitment to improve, citing  
reorganization plans that anticipate management information	 
system as the foundation for improved grants management. However,
reorganization and management information tools and are only as  
good as the management that wields them. Commitment to		 
improvement and oversight are needed to ensure progress.	 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-02-641T					        
    ACCNO:   A03079						        
  TITLE:     Violence Against Women Office: Problems with Grant       
Monitoring and Concerns about Evaluation Studies		 
     DATE:   04/16/2002 
  SUBJECT:   Discretionary grants				 
	     Grant monitoring					 
	     Program evaluation 				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-02-641T
     
Testimony Before the Subcommittee on Crime and Drugs, Committee on the
Judiciary, U. S. Senate

United States General Accounting Office

GAO For Release on Delivery Expected at 10: 15 a. m., EDT, Tuesday, April
16, 2002 VIOLENCE AGAINST

WOMEN OFFICE Problems with Grant Monitoring and Concerns about Evaluation
Studies

Statement of Laurie E. Ekstrand, Director, Justice Issues

GAO- 02- 641T

Page 1 GAO- 02- 641T

Mr. Chairman and Members of the Subcommittee: Thank you for inviting me to
testify about our recent work concerning the Violence Against Women Office
(VAWO). This work has specifically focused on monitoring activities and
impact evaluations related to VAWO?s discretionary grant programs and is
part of a body of recent work concerning monitoring and evaluation of grants
by a number of Office of Justice Program?s (OJP) bureaus and offices.
Monitoring and evaluation are the activities that identify whether programs
are operating as intended, whether they are reaching those that should be
served, and ultimately whether they make a difference. In other words, these
are major elements of assessing results. Our recent work has shown a need
for improvement in VAWO grant monitoring and in the evaluations that are
intended to assess the impacts of VAWO programs.

VAWO was created in 1995 to carry out certain programs created under the
Violence Against Women Act of 1994. 1 The Victims of Trafficking and
Violence Prevention Act of 2000 reauthorized most of the existing VAWO
programs and added new programs. 2 VAWO?s mission is to lead the national
effort to end violence against women, including domestic violence, sexual
assault, and stalking. VAWO programs seek to improve criminal justice system
responses to these crimes by providing support for law enforcement,
prosecution, courts, and victim advocacy programs across the country. In
addition, programs are to enhance direct services for victims, including
victim advocacy, emergency shelter, and legal services. VAWO also addresses
violence against women issues internationally, including working to prevent
trafficking in persons. VAWO is one of seven program offices and five
bureaus in OJP. 3

VAWO?s discretionary grant programs have grown substantially since its
inception in 1995. Data provided by OJP showed that, between fiscal years

1 Title IV of the Violent Crime Control and Law Enforcement Act of 1994 (P.
L. 103- 322). 2 P. L. 106- 386. 3 OJP?s five bureaus are Bureau of Justice
Assistance, Bureau of Justice Statistics, National Institute of Justice,
Office of Juvenile Justice and Delinquency Prevention, and Office for
Victims of Crime. OJP?s seven program offices are American Indian and Alaska
Native Affairs Desk, Violence Against Women Office, Executive Office for
Weed and Seed, Corrections Program Office, Drug Courts Program Office,
Office for Domestic Preparedness, and Office of Police Corps and Law
Enforcement Education. Appendix I shows OJP?s current organizational
structure. Background

Page 2 GAO- 02- 641T

1995 and 2000, the yearly number of VAWO discretionary grant awards
increased about 362 percent- from 92 in fiscal year 1996, the first full
year of funding, to 425 in fiscal year 2000. In addition the yearly dollar
amount of VAWO discretionary grant awards increased about 940 percent- from
just over $12 million in fiscal year 1996, the first full year of funding,
to about $125 million in fiscal year 2000. Appendix II shows the number of
yearly VAWO discretionary grant awards for fiscal year 1995 through fiscal
year 2000. Appendix III shows the dollar amount of VAWO discretionary grant
awards, adjusted to constant fiscal year 2000 dollars, over the same period.

The monitoring of grant activities is a key management tool to help ensure
that funds awarded to grantees are being properly spent. In November 2001,
in response to a request by Senators Grassley and Sessions, we reported that
grant files for discretionary grants awarded by VAWO often lacked the
documentation necessary to ensure that the required monitoring activities
occurred. 4 Our review of grant files for a representative sample of VAWO
discretionary grants active in all of fiscal years 1999 and/ or 2000 showed
that:

 VAWO grant files did not always contain requisite grant monitoring plans.
When monitoring plans were in the files, grant managers did not consistently
document their monitoring activities, such as site visits, according to the
plans they developed.

 A substantial number of VAWO grant files did not contain progress and
financial reports sufficient to cover the entire grant period, contrary to
OJP guidelines. Furthermore, VAWO grantee progress and financial reports
were often submitted late by grantees. These reports are an important tool
to help managers and grant monitors determine if grantees are meeting
program objectives and financial commitments.

 VAWO grant files did not always contain the required closeout documents-
key documents by which OJP ensures that, among other things, the final
accounting of federal funds have been received.

We also found that, because documentation about monitoring activities was
not readily available, VAWO was not positioned to systematically determine
staff compliance with monitoring requirements and assess

4 U. S. General Accounting Office, Justice Discretionary Grants: Byrne
Program and Violence Against Women Office Grant Monitoring Should Be Better
Documented,

GAO- 02- 025 (Washington, D. C.: Nov. 28, 2001). Problems with VAWO

Discretionary Grant Monitoring

Page 3 GAO- 02- 641T

overall performance. Although VAWO officials said that they met with grant
managers weekly to discuss any grant problems or monitoring issues, VAWO did
not (1) have an overall system to track monitoring activities, other than
site visits and (2) appear to be routinely using OJPwide data on late
progress reports and financial reports. Furthermore, the lack of systematic
data associated with program monitoring activities and the documentation
problems we observed raised questions about whether VAWO was positioned to
measure its performance consistent with the Government Performance and
Results Act (GPRA) of 1993. Specifically, we pointed out that, in DOJ?s
Fiscal Year 2000 Performance Report and Fiscal Year 2002 Performance Plan,
DOJ failed to recognize the serious limitations associated with inconsistent
documentation and the lack of systematic monitoring data in measuring
whether VAWO was achieving its goals for formula and discretionary grants-
especially since the Report and Plan stated that VAWO would rely on grant
monitoring data to measure its performance.

We concluded that neither OJP, VAWO, nor GAO can determine the level of
monitoring performed by grant managers as required by OJP and the
comptroller general?s internal control standards, which call for
documentation of all transactions and other significant events to ensure
that management directives are being carried out. 5 We recommended that VAWO
review why documentation problems occurred and take steps to resolve these
problems.

VAWO and OJP officials have acknowledged that they need to take steps to
resolve some of the problems associated with grant monitoring, but it is too
early to tell if these steps will be effective. For example, in response to
our report, the assistant attorney general said that VAWO had begun to
develop both an internal monitoring manual that would include procedures for
developing monitoring plans using a risk- based assessment tool. They also
said they have developed a management information system that will
eventually track the submission of progress and financial reports.
Furthermore, while we were developing our report, VAWO officials said that
they were not satisfied with the performance measures they used to gauge
their performance under GPRA because they did not believe they are
meaningful for measuring program outcomes. They said that they are working
with other OJP officials and an outside contractor to

5 U. S. General Accounting Office, Internal Control: Standards for Internal
Control in the Federal Government, GAO/ AIMD- 00- 21. 1. 3. 1 (Washington,
D. C.: Nov. 1999). Too Early to Gauge Efforts

to Resolve Grant Monitoring Problems

Page 4 GAO- 02- 641T

develop new measures and hope to have them available for the fiscal year
2003 performance plan.

It is also important to note that VAWO?s efforts to address grant monitoring
problems need to be viewed in the context of OJP efforts in this area. Our
recent related reports discussed grant monitoring problems at other OJP
organizations, such as the Bureau of Justice Assistance (BJA) and the Office
of Juvenile Justice and Delinquency Programs (OJJDP) 6 , and pointed out
that, over the last few years, we and others, including OJP, have identified
various grant monitoring problems among OJP bureaus and offices. We
discussed how OJP had begun to work with bureaus and offices to resolve some
of the problems it and others have identified, including OJP efforts to
develop an automated grants management system as a way to standardize and
streamline the grant process.

Our report concluded that OJP efforts to automate the grant management
process, particularly in regard to grant monitoring, holds some promise if
OJP takes steps to ensure that all monitoring activities are consistently
recorded and maintained in a timely manner. We also said that current and
future efforts will be futile unless OJP and its bureaus and offices, such
as VAWO, periodically test grant manager compliance with OJP requirements
and take corrective action when needed to enforce those requirements. We
recommended that OJP (1) study and recommend ways to establish an approach
to systematically test or review grant files to ensure consistent
documentation across OJP and (2) explore ways to electronically compile and
maintain documentation of monitoring activities to facilitate more
consistent documentation, more accessible management oversight, and sound
performance measurement.

In January 2002, in response to our report, the assistant attorney general
said that OJP agreed that it needs to develop more consistent documentation
of monitoring activities. She said that among other things, OJP has created
a chief information officer position charged with planning and implementing
an agencywide grant management system. According to the assistant attorney
general, the new system is envisioned to produce reports in response to
informational requests, provide information pertaining to grantees and all
resources provided by OJP, and maintain

6 U. S. General Accounting Office, Juvenile Justice: OJJDP Reporting
Requirements for Discretionary and Formula Grantees and Concerns About
Evaluation Studies, GAO- 0223 (Washington, D. C.: Oct. 30, 2001).

Page 5 GAO- 02- 641T

information from the opening to the closing of a grant award. Although the
assistant attorney general said that OJP will consider the comptroller
general?s internal control standards in taking these steps, it is unclear
whether the new system will include the full range and scope of monitoring
activities carried out by grant managers in VAWO and other OJP
organizations.

We have also recently issued a report on work undertaken for Senators
Grassley and Sessions that addressed the methodological rigor of impact
evaluations of three VAWO discretionary grant programs. 7 During fiscal
years 1995 through 2001, the National Institute of Justice (NIJ) awarded
about $4 million for five VAWO discretionary grant program evaluations that
were intended to measure the impact of the VAWO programs. 8 Our indepth
review of the three program evaluations that had progressed beyond the
formative stage showed that all three had methodological problems that
raised concerns about whether the evaluations will produce definitive
results.

More specifically, our report stated that, although program evaluation is an
inherently difficult task, in all three VAWO evaluations, the effort was
particularly arduous because of variations across grantee sites in how the
programs are implemented. Our concerns about these efforts included problems
with both evaluation design and implementation. In particular, VAWO sites
participating in the impact evaluations had not been shown to be
representative of their programs, thereby limiting the evaluators? ability
to generalize results. Further, the lack of nonprogram participant
comparison groups hindered evaluators? ability to minimize the effects of
factors that are external to the program and isolate the impact of the
program alone. While in some situations, other means (other than comparison
groups) can be effective in isolating the impact of a program from other
factors, in these evaluations, effective alternative methods were not used.
In addition, data collection and analytical problems (e. g.,

7 U. S. General Accounting Office, Justice Impact Evaluations: One Byrne
Evaluation Was Rigorous; All Reviewed Violence Against Women Office
Evaluations Were Problematic,

GAO- 02- 309 (Washington, D. C.: Mar. 7, 2002). 8 Impact evaluations are
designed to assess the net effect of a program by comparing program outcomes
with an estimate of what would have happened in the absence of the program.
Concerns About

Evaluation Studies of VAWO Discretionary Grant Programs

Page 6 GAO- 02- 641T

related to statistical tests, assessment of change) compromised the
evaluators? ability to draw appropriate conclusions from the results.

We concluded that, despite great interest in assessing the results of OJP?s
discretionary grant programs, it can be extremely difficult to design and
execute evaluations that will provide definitive information. We further
concluded that, given that NIJ spends millions of dollars to evaluate OJP
grant programs, including those within VAWO, more up- front attention to the
methodological rigor of these evaluations will increase the likelihood that
they will produce meaningful results for policymakers. We recommended that
the attorney general direct the NIJ director to assess the two VAWO impact
evaluations still in the formative stage to address any design methodology
and implementation problems and, on the basis of that assessment, initiate
any needed interventions to help ensure that the evaluations produce
definitive results. We further recommended that the director of NIJ be
instructed to assess its evaluation process to develop approaches to ensure
that future impact evaluation studies are designed and implemented to
produce definitive results. The assistant attorney general commented that
she agreed with the substance of our recommendations and has begun or plans
to take steps to address them. It is still too early to tell whether these
actions will be effective in preventing or resolving the problems we
identified, but they appear to be steps in the right direction.

In summary, since its inception, VAWO has grown substantially both in terms
of the number of discretionary grants awarded and dollars awarded for those
discretionary grants- increasing the importance of ensuring that its
grantees are achieving intended results. Unfortunately, the lack of good
data from monitoring activities and impact evaluations leaves us with very
little basis to assess program results. Both VAWO and OJP have indicated a
commitment to making improvements, citing reorganization plans and the
anticipated management information system as the foundation for improved
grants management, including improvements in monitoring and evaluation. But,
reorganization and management information systems are only tools and are
only as good as the management that wields them. Commitment to improvement
and oversight are needed to ensure progress.

Mr. Chairman, this concludes my prepared statement. I would be pleased to
answer any questions that you or other members of the subcommittee may have.

Page 7 GAO- 02- 641T

For further information regarding this testimony, please contact Laurie E.
Ekstrand or John F. Mortin at (202) 512- 8777. Individuals making key
contributions to this testimony included Wendy C. Simkalo, Jared A.
Hermalin, and Chan My J. Battcher.

Page 8 GAO- 02- 641T

Note: The organization chart is current as of March 2002. Source: Prepared
by GAO based on OJP documentation.

Appendix I: OJP Organization Chart

Corrections Program

Office Office of Budget, & Management

Services Bureau of

Justice Assistance

Executive Office for

Weed & Seed Violence

Against Women

Office Office for

Domestic Preparedness Drug Courts

Program Office

Office of General Counsel Office for

Civil Rights

Office of the Comptroller

Bureau of Justice Statistics

National Institute of

Justice Office of Juvenile

Justice & Delinquency Prevention

Office for Victims of

Crime Office of Police

Corps & Law Enforcement

Education Office of Administration

Office of Congressional

& Public Affairs Office of the

Assistant Attorney

General Equal

Employment Opportunity Office American

Indian & Alaska Native Affairs

Desk

Page 9 GAO- 02- 641T

Source: OJP Office of the Comptroller.

Appendix II: Number of Yearly VAWO Discretionary Awards, Fiscal Years 1995-
2000

Page 10 GAO- 02- 641T

Note: The award amounts for each fiscal year are adjusted to constant fiscal
year 2000 dollars. Source: OJP Office of the Comptroller.

Appendix III: Dollar Amount of VAWO Discretionary Awards, Fiscal Years 1995-
2000

(440120)
*** End of document. ***