[Final Audit Report on Improvements Needed In Developing and Reporting On GPRA Goals and Measures, Bureau of Land Management]
[From the U.S. Government Printing Office, www.gpo.gov]

Report No. 2002-I-0047

Title: Final Audit Report on Improvements Needed In Developing
       and Reporting On GPRA Goals and Measures, Bureau of Land
       Management

  
Date:  September 25, 2002

**********DISCLAIMER********** 
This file contains an ASCII representation of an OIG report. No attempt has been made to display graphic images or illustrations. Some tables may be included, but may not resemble those in the printed version. A printed copy of this report may be obtained by referring to the PDF file or by calling the Office of Inspector General, Division of Acquisition and Management Operations at (202) 219-3841. 
******************************

Memorandum

To:		Assistant Secretary, Land and Minerals Management

From:		Robert Romanyshyn		
		Regional Audit Manager, Eastern Region

Subject:  Final Audit Report on Improvements Needed In Developing and Reporting On GPRA Goals and Measures, Bureau of Land Management (No. 2002-I-0047)

The attached report presents the results of our review of the Bureau of Land Management's (BLM) reporting of its fiscal year 2000 Government Performance and Results Act (GPRA) program activity 1.04 related to reducing threats to public health, safety, and property.  This program activity reports on BLM's efforts to protect public lands and facilities from unauthorized uses, hazardous materials, illegal dumping, theft, wildfire, and other unsafe conditions.

In the July 31, 2002 response to our draft report, the Director of BLM concurred with the report's six recommendations.  Based on this response and the milestone dates subsequently provided, we consider the recommendations to be resolved.  Accordingly, no further response from BLM is needed, and we are referring the recommendations to the Assistant Secretary for Policy, Management and Budget for tracking of implementation (see Appendix 4, Status of Audit Recommendations).

The legislation, as amended, creating the Office of Inspector General, (5 U.S.C. App. 3) requires semiannually reporting to Congress on all audit reports issued, actions taken to implement audit recommendations, and recommendations that have not been implemented.

We appreciate BLM's cooperative spirit in working together throughout the audit to communicate issues and obtain a consensus so that recommended actions address mutually agreed upon needed improvements in GPRA reporting.  If you have any questions regarding this report, please call me at (703) 487-8011.

Attachment

cc:	Director, Bureau of Land Management

	

Executive Summary

Bureau of Land Management
Improvements Needed In Developing and Reporting On
GPRA Goals and Measures 


The 1993 enactment of the Government Performance and Results Act (GPRA) requires each federal department and agency to prepare annual performance reports comparing planned accomplishments for mission-related activities with the results actually achieved.  The Bureau of Land Management (BLM) developed GPRA program activity 1.04 on health, safety, and property to address the increased risks and responsibilities resulting from population growth in the West; evaluate safety concerns; identify hazards and risks; and implement procedures to protect public lands as well as employees and visitors.

Our objectives were to determine whether BLM (1) developed appropriate performance goals and measures for program activity 1.04 - Reduce Threats to Public Health, Safety, and Property and (2) accurately reported performance data and related information for goals and measures under this activity.

BLM should revise several of its performance goals and measures to improve performance information provided to BLM managers and to decision makers in Congress and the Office of Management and Budget (OMB).  BLM should also consider adding at least two comprehensive safety and property goals and/or measures related to visitor and employee safety and deferred maintenance.  In addition, BLM staff and management should be held accountable for ensuring that the data used to report BLM performance are accurate.  Finally, there appears to be a lack of understanding throughout BLM of the increasing importance of performance management and results.

We made six recommendations based on our identifying the following opportunities for BLM to improve its GPRA reporting.
   
* BLM could build on its experience with collecting and reporting data for its GPRA goals and performance measures by updating the goals and measures to ensure that they meet the GPRA requirements of focusing on BLM's mission and performance.  BLM could also provide guidance and training for staff at all levels on how GPRA reporting requirements are related to the day-to-day operations of field personnel.

* BLM could establish a data verification and validation process to ensure the accuracy of the GPRA data reported at all levels.  Any such effort should include certification to the accuracy of the data included in the GPRA report.  BLM would also be more successful in reporting accurate data and related information if it developed procedures to accumulate information supporting the GPRA data reported.


The BLM concurred with the report's six recommendations.  Based on BLM's response and the milestone dates subsequently provided, we consider the recommendations resolved but awaiting implementation.


Table of Contents

Page

Introduction 		1
	Background 		1
	Objectives and Scope 		2

Results of Audit 		4
Improvements Needed In Developing and Reporting On
GPRA Goals and Measures 		4

Performance Goals and Measures Should be Revised 		4
Basis for Determining Appropriate Measures 		4
Measure Did Not Support Overall Goal 		5
Managers Lacked Control Over Achieving Results 		6
Measures Were Not Outcome Oriented 		7
Measures Did Not Address Key Safety Issues 		7
BLM Staff Lacked Experience with GPRA 		9

Improvements Needed for Accurate Reporting 		9
	Facilities and Roads Data Were Not Accurate 
		or Current		10
	CASHE Data Accuracy was Questionable 		12
	Abandoned Mines Reporting was Inaccurate 		13
	Hazardous Materials Reporting was Inaccurate 		14
	Law Enforcement Accomplishment was Overstated 		14
	Data Were Not Systematically Verified  		15
	Improving the Usefulness of the GPRA Report to 
Decision Makers 		17

Recommendations 		17

Agency Response and Office of 
	Inspector General Reply 		18

Appendices

1.	Scope and Methodology 		20
2.	Prior Audit Coverage 		22
3.	Agency Response 		23
4.	Status of Audit Recommendations 		27


Abbreviations

AMLIS  Abandoned Mine Lands Inventory System
BLM  Bureau of Land Management
CASHE  Compliance Assessment - Safety, Health, and the Environment
DOI  Department of the Interior
FIMMS  Facility Inventory Maintenance  Management System
GAO  General Accounting Office
GPRA  Government Performance and Results Act
MIS  Management Information System
OIG  Office of Inspector General
OMB  Office of Management and Budget

Introduction

GPRA requires federal departments to prepare annual performance plans setting out specific, measurable goals and to submit annual performance reports comparing actual performance with these goals.  The first annual performance report, which covered fiscal year 1999, was due March 31, 2000.  GPRA was a result of Congressional concern that "Federal managers are seriously disadvantaged in their efforts to improve program efficiency and effectiveness, because of insufficient articulation of program goals and inadequate information on program performance."  The purpose of GPRA was to "help Federal managers improve service delivery, by requiring that they plan for meeting program objectives and by providing them with information about program results and service quality."  BLM's performance plan and report for fiscal year 2000 included eight GPRA program activities.

BLM GPRA program activity 1.04 addresses the reduction of threats to public health, safety, and property.  Within this activity, BLM established five annual goals to assess its performance in protecting public lands and facilities from unauthorized uses, hazardous material, illegal dumping, theft, wildfire, and other unsafe conditions.  BLM also established twelve performance measures to evaluate the success of specific activities in meeting its five goals, as follows:

Table 1

Annual Goal
Performance Measure
In FY 2000, the percentage of administrative facilities maintained in fair or good condition will increase to 87%; bridges will increase to 94%, and dams will increase to 59%.  A baseline percentage for facilities in good safety, health, and environmental condition will be established.
Percent of administrative facilities in fair or good condition.

Percent of dams in fair or good condition.

Percent of bridges in fair or good condition.

Percent of facilities (CASHE Program) in good safety, health, and environmental condition.
In FY 2000, assess condition of 3% of BLM-maintained roads to identify access needs, maintenance requirements, and prospective road closures, and improve 60% of BLM roads to be in fair or good condition.
Percentage of roads assessed to identify access needs, maintenance requirements, and closures.

Percentage of roads in fair or good condition.
In FY 2000, identify and correct physical safety hazards at zero abandoned mines and cleanup 135 hazardous materials sites on public lands.
Number of abandoned mine physical safety hazards identified and corrected.

Number of hazardous materials cleanup actions completed.
In FY 2000, maintain an increase of at least 1% (to 47%) in the number of reported violations resulting in enforcement actions compared to 1999.
Percentage of reported violations resulting in enforcement actions.
In FY 2000, implement 35% of Fire Management Plans, reduce the 10-year average of human caused fires to 27%, and upgrade 9 (cumulative) fire facilities to approved standards.
Percentage of FMPs implemented.

Percentage of human-caused fires.

Number of fire facilities upgraded.

Our objectives were to determine whether BLM (1) developed appropriate performance goals and measures1 for program activity 1.04 - reduce threats to public health, safety, and property and (2) accurately reported performance data and related information for the goals and measures under this activity.

We selected GPRA program activity 1.04 in consultation with BLM staff.  We did not review the goal with three measures regarding the fire program for this activity because BLM had substantially changed the goal and measures for fiscal year 2001.  Like other federal agencies, BLM is working to improve its GPRA reporting as it gains experience in performance planning, management, and reporting.  Recognizing weaknesses in its fiscal year 2000 fire program goal and measures, BLM revised them for fiscal year 2001, eliminating two measures, revising the wording of the third, and adding four new measures.

Our review of program activity 1.04 involved several BLM offices:  the BLM Protection and Response Group, the National Law Enforcement Office, and the Management Systems Group.  The Protection and Response Group, which handles environmental compliance and protection, cleanup of hazardous material sites, abandoned mine lands, facilities maintenance, and engineering, is responsible for three goals with eight of the performance measures.  The National Law Enforcement Office, which provides BLM-wide leadership and guidance for law enforcement, security, and investigative actions to protect resources under BLM jurisdiction, oversees one goal with one performance measure.  The Management Systems Group works on GPRA requirements for strategic planning, performance measurement, and reporting; prepares annual GPRA plans and reports; and performs validation and verification reviews of performance data.

The scope and methodology of our review, including the sites visited, are further detailed in Appendix 1.  Prior reviews related to our GPRA audit are summarized in Appendix 2.

Results of Audit

Some fiscal year 2000 performance goals and measures for the health, safety, and property program activity were not appropriate; that is, they did not support the overall goal, or they were not under the direct control of federal managers or outcome oriented.  In addition, BLM did not have goals or measures for key safety issues.  We attributed this condition to a lack of experience in implementing GPRA.  BLM employees are still developing expertise in meeting GPRA reporting requirements.  BLM also reported inaccurate or incomplete data for eight measures because of limited data verification at all levels of the organization.  Additionally, BLM did not have procedures in place to accumulate sufficient documentation to support the performance data used in GPRA reporting.  Once appropriate goals and measures are selected, accurate data and information will improve program accountability of managers and allow Congress and OMB to better evaluate BLM's performance and make decisions on appropriations.


BLM needs to improve the performance goals and measures for its program activity to reduce threats to public health, safety, and property.  Four of the nine performance measures reviewed needed revision, and two additional measures should be developed to appropriately measure and report BLM's progress in this activity.  The measures were not appropriate because, like many federal agencies, BLM managers lacked experience in GPRA reporting and in linking GPRA reporting requirements to day-to-day operations.  The measures do not provide sufficient information for decision-making by BLM managers, Congress, or OMB.

The criteria we used for determining the appropriateness of the performance measures were general compliance with the requirements of GPRA and OMB.  Specifically, the performance aspect or target for the measure should support the overall goal of reducing threats; managers should have control over meeting accomplishments; and accomplishment of the performance measure should be outcome related whenever possible.  We found that four of the nine performance measures reviewed did not meet these criteria (please see Table 2).

      Table 2
      
Goal Number
Performance Measure
Appropriateness
01.04.01
Percent of administrative facilities in fair or good condition.
Yes

Percent of dams in fair or good condition.
Yes

Percent of bridges in fair or good condition.
Yes

Percent of facilities (CASHE Program) in good safety, health, and environmental condition.
Measure definition did not support overall goal.
01.04.02
Percentage of roads assessed to identify access needs, maintenance requirements, and closures.
Yes

Percentage of roads in fair or good condition.
Yes
01.04.03
Number of abandoned mine physical safety hazards identified and corrected.
No management control and "not outcome oriented."2

Number of hazardous materials cleanup actions completed.
No management control over illegal dumping.
01.04.04
Percentage of reported violations resulting in enforcement actions.
Not outcome oriented.

The Compliance Assessment - Safety, Health, and the Environment (CASHE) program evaluates BLM's facilities and operations for compliance with federal, state, and local safety, health, and environmental regulations.  These evaluations result in a list of findings to be addressed to bring a facility up to standard.  While this goal was appropriate, the definition of the CASHE measure did not support the overall goal of reducing threats to health and safety.  As defined during fiscal year 2000, a facility could be reported in GPRA as being in good safety, health, and environmental condition if the number of uncorrected high priority findings were four or less.  A high priority finding might be the combined storage of incompatible materials (such as corrosive and oxidizing materials stored with flammable and combustible liquids) or an uncovered electric junction box with exposed wiring that posed an electrocution and fire hazard.  Given these examples, we believe that the number of high priority uncorrected findings should be zero before a facility is classified as being in "good" condition.  According to the CASHE program manager, the zero standard will be phased in over the next few years and become effective in fiscal year 2005.  We believe, however, that it would be more consistent with the overall goal of reducing threats to health and safety to have the same standard for safety in 2000 as in 2005 and to adjust the baseline and targets.

Other concerns regarding the CASHE program-the length of time between inspections and facilities self-reporting that findings are corrected-are discussed on page 12.
BLM program managers for both abandoned mines and hazardous materials had limited control over the accomplishments of performance measures for these activities.  The correction of physical safety hazards at abandoned mines had zero budgets in 2000 and 2001 and zero planned accomplishments in 2000.  As a result, safety hazards were corrected in fiscal year 2000 through funding from benefiting programs, such as Recreation Management, the Mining Law Administration Program,3 and the Watershed Cleanup Initiative.  We recognize that the Program was able to accomplish significant corrections in 2000 without its own budget.  However, this type of funding largely removed control over accomplishments from the safety hazards staff, making it difficult for them to prioritize safety, set realistic targets, and be held accountable for meeting goals.

Likewise, accomplishing the hazardous materials performance measure, which reports the number of hazardous material cleanups completed by BLM, depends primarily on factors outside of BLM's control-that is, the amount of public dumping on BLM land.  The number of illegal dumping incidents, which account for approximately 60 percent of the hazardous materials workload, can fluctuate substantially from one year to the next.  This prevents BLM from establishing realistic targets and budgets for which it can be held accountable.  Field specialists stated that the current measure did not reflect actual day-to-day operations because staff have little control over the total number of cleanups performed during the year.  In effect, BLM cannot cleanup until the public dumps.  For example, if a BLM state office does not meet its target of 30 cleanups because only 20 dumpings occurred in a year, the threat to health and safety has actually diminished, but not as a result of BLM efforts.  An example of an approach that would give BLM managers control over accomplishment would be developing a measure that addressed the timeliness rather than the number of hazardous cleanup actions.

Two of the performance measures reviewed-the physical safety hazards of abandoned mines and law enforcement actions-were not results oriented.  In fiscal year 2000, the abandoned mines measure combined the identification of sites and the subsequent correction of safety hazards.  We concluded that to better focus on outcomes, BLM should remove the identification portion of the performance measure because mere identification of a site does not reduce safety hazards.  As a result of our discussions with BLM staff, they removed identification from the measure for fiscal year 2002, thereby resolving our concern that the performance measure was not outcome oriented.

The law enforcement measure-to increase the percentage of reported violations resulting in enforcement actions-was also based on output rather than outcome.  Increasing the number of enforcement actions may not actually reduce threats to public safety.  A 1 percent increase in the enforcement rate can result either from additional enforcement actions on the same number of offenses or from a reduction in the number of violations.  BLM law enforcement staff believe that a performance measure to reduce crime depends on too many fluctuating variables, such as staffing and workload, to accurately assess what is happening in the field.  In their opinion, the current measure, which targets increased enforcement actions, better addresses performance.

We believe, however, that a measure to reduce the rate of violations would more clearly focus BLM efforts on the goal of reducing threats.  For example, the Department of Justice combines strategies for increased enforcement and crime prevention with GPRA performance goals and measures.  These strategies stress the reduction and prevention of crime as well as an improved prosecution rate for offenses.  The Department of the Interior's (DOI) Bureau of Indian Affairs also has a law enforcement performance measure that takes the seriousness of an offense into account by concentrating on the reduction of violent crimes.  We understand that BLM's law enforcement environment is unique in many ways, but believe that BLM should consider making the measure more results oriented.

While BLM managers have told us that current policy is to keep the number of GPRA measures as low as possible, we believe that BLM should consider adding at least two more measures to GPRA program activity 1.04.  Measures related to reducing significant accidents, injuries, and incidents involving BLM visitors and employees and reducing the backlog of health and safety related deferred maintenance would be mission related and results oriented and would specifically address material weaknesses identified in past BLM internal reviews.  These comprehensive measures could either supplement or replace current measures covering only small, individual programs.

DOI's National Park Service has safety measures to reduce accidents and injuries to both the public and employees, and we believe BLM should consider a measure to decrease the rate of accidents, injuries, and incidents of BLM visitors and employees. Such a measure would address the recent emphasis on safety by both the White House and the Secretary of the Interior and would be an important indicator of BLM's progress in accomplishing the overall goal of reducing threats.  BLM staff expressed concerns about their ability to control public behavior resulting in accidents or injuries.  Visitor safety, however, will continue to be an issue as the urbanization of Western states results in increasing numbers of visitors to BLM-managed land.  BLM has not incorporated a GPRA measure for employee safety proposed by its Safety Program manager into the annual plan to date.  Data for this measure could be reported in BLM's existing Safety Management Information System database.

GPRA measures also need to be developed for deferred maintenance, especially maintenance related to health and safety issues.  Currently, neither the facilities' measures nor the CASHE measure directly address deferred maintenance, which is a material weakness for both BLM and DOI.  Recent Office of Inspector General (OIG) and DOI reviews have identified a serious backlog on health and safety deferred maintenance for BLM.  (Please see Appendix 2.)  Although health and safety projects are prioritized in BLM's deferred maintenance 5-year plan, a backlog of over $20 million still exists according to BLM data.  We noted that DOI, National Park Service, U.S. Fish and Wildlife Service, and Bureau of Indian Affairs have performance measures related to addressing deferred maintenance issues to improve health and safety for both visitors and employees.  A performance measure to reduce BLM's deferred maintenance backlog would provide information on progress being made to address the backlog and have a direct impact on reducing threats to public health, safety, and property.

We believe that the lack of experience with GPRA requirements and/or experience with linking GPRA reporting requirements to day-to-day operations contributed to inappropriate GPRA measures.  The fiscal year 2000 GPRA Report was the second GPRA report prepared by BLM.  We recognize that DOI bureaus will require time to understand and fully implement reporting requirements.  Program staff throughout BLM were unfamiliar with GPRA and its requirements.  Although staff in the field had experience with workload measures, they were not always knowledgeable about GPRA or how workload measures related to GPRA.  BLM staff at all levels had questions and comments related to the relevance and usefulness of the performance measures for their programs.  Management Systems Group managers stated that national program managers were now participating more actively in goal development and reporting.

Three of the measures (law enforcement, abandoned mines with physical safety hazards, and CASHE) were reported on for the first time in fiscal year 2000.  As a result, BLM had no practical experience with data collection and reporting for these measures prior to the 2000 report.  In the case of hazardous material cleanups, field personnel have recognized, after 2 years of operational experience, that the measure should be revised to better reflect day-to-day operations.  With additional experience in implementing the GPRA requirements BLM should be able to identify weaknesses in GPRA measures and make appropriate changes through actual collection and reporting of data and associated comments from BLM field specialists.  BLM could also evaluate and improve the GPRA process by using its Continuous Improvement Process, which relies on the expertise of BLM's workforce to build efficiency, effectiveness, and accountability into BLM's work processes.

Data reported in the fiscal year 2000 GPRA Report were inaccurate or incomplete for six of the nine performance measures reviewed (please see Table 3).  In addition, all but one performance measure lacked sufficient disclosure or provided incorrect information on data limitations and data sources.  As a result, BLM managers, Congress, and OMB did not have accurate or complete information about BLM's efforts to reduce threats to public health, safety, and property.  We identified several causes that contributed to the inaccurate or incomplete reporting of data and related information.  Some causes, such as the lack of management oversight; inadequate guidance and training; and lack of procedures to support GPRA data, were program specific.  Seven of the nine measures lacked procedures to accumulate supporting documentation and verify that data were accurately reported.

Table 3
      
Goal Number
Performance Measure
Fiscal Year 2000
Reported Accurately


Goal
Reported
Data
Related Information4
01.04.01
Percent of administrative facilities in fair or good condition.
  87
  84
No
No

Percent of dams in fair or good condition.
  59
  61
N/A5
No

Percent of bridges in fair or good condition.
  94
  92
Yes
No

Percent of facilities (CASHE Program) in good safety, health, and environmental condition.
  60
  59
No
No
01.04.02
Percentage of roads assessed to identify access needs, maintenance requirements, and closures.
    3
    4
Yes
No

Percentage of roads in fair or good condition.
  60
  82
No
No
01.04.03
Number of abandoned mine physical safety hazards identified and corrected.
  --
650
No
No

Number of hazardous materials cleanup actions completed.
135
290
No
Yes
01.04.04
Percentage of reported violations resulting in enforcement actions.
  47
  53
No
No

The percentage of both administrative facilities and roads reported in fair or good condition in the fiscal year 2000 GPRA Report was calculated inaccurately and did not agree with data in the Facility Inventory Maintenance Management System (FIMMS).  This system is the data source for the measures related to administrative facilities and roads.  Administrative facilities in good or fair condition were reported as 84 percent, whereas FIMMS data supports an 87 percent accomplishment.  The error for roads was significantly larger.  The national program manager for facilities and roads confirmed our calculation using FIMMS data, 64 percent of roads were in good or fair condition rather than the reported 82 percent.  The program manager could not explain how the incorrect data were reported, particularly since the goal was 60 percent, and an 82 percent reported actual accomplishment should have raised questions.

The accuracy and completeness of GPRA data reported for facilities and roads depended on the accuracy of FIMMS.  We found, however, that FIMMS data were not current or complete because field specialists did not always consistently perform cyclical condition assessments of these assets or update FIMMS when they did perform assessments.  As a result, there was no assurance that the numbers calculated from data in FIMMS were accurate.

According to BLM procedures, the condition of administrative facilities should have been assessed in 1999.  We found, however, that condition assessments had not been updated since 1998 for 72 percent of facilities sampled at eight field offices (34 of 47 facilities) because either assessments were not done or information was not entered into FIMMS.  The situation for roads was similar.  Engineers at six of eight field offices visited stated that condition assessments had not been performed for roads under their jurisdiction.  In one state, the two field offices visited were responsible for maintaining over 16,000 road segments.  For 94 percent of these segments, however, no date for condition assessments had been entered in FIMMS, and for another 5 percent, the last condition assessment date listed was 1991.  We concluded that FIMMS data for administrative facilities and roads were not current and therefore not reliable.  BLM engineers apparently agreed with us because only one state used FIMMS data to select projects to be included in BLM's 5-Year Deferred Maintenance Plan.  Several BLM staff told us that instead of using FIMMS, they relied on their own records or "cuff notes" to track facility and road conditions and maintenance work performed.

Although program staff at all levels were aware that FIMMS data were incomplete and the system was difficult to use, BLM did not disclose these limitations in the GPRA report.  Instead, the data verification and validation section of the report stated, "program staff have used FIMMS for many years, so data entry errors or inconsistent definitions are not likely."  Management Systems Group managers, who accumulated GPRA information and prepared the Report, agreed they included this information without consulting program managers.  This section also incorrectly identified the BLM Management Information System (MIS) as another data source for facilities and roads.  We also noted that BLM reported fiscal year 2000 accomplishments for all the facilities and roads performance measures as a percentage only and did not include information on the total number of facilities or the number of facilities in good or fair condition.  Inclusion of this information would have helped BLM identify errors in reporting on administrative facilities and roads accomplishments.

BLM provided limited training in the use of FIMMS, which negatively affected the willingness and ability of field office staff to enter information into FIMMS.  BLM staff told us that FIMMS was also not being updated because BLM planned to transfer its maintenance data to a new system called MAXIMO, which is currently a pilot program and which may not be operational for at least 3 to 4 years.  In addition, BLM recognized that it would take significant resources to correct and validate data from FIMMS prior to transferring the data to MAXIMO.

The accuracy of the CASHE data was questionable for several reasons:  the data was based on a projection, the facilities self-reported corrected deficiencies, and some data may be as old as 7 years.  The fiscal year 2000 GPRA Report did not disclose that the 59 percent accomplishment rate was based on a projection.  BLM made the projection by surveying all facilities with CASHE inspections and asking the facilities to identify how many uncorrected findings still remained.  The facilities determined when a finding was corrected, with no independent verification until the next CASHE inspection.  In addition, facilities were not required to correct CASHE health and safety findings within a specified time or to provide evidence of correction when closing a finding.  BLM used the responses, which were received from just over half of the facilities surveyed, to estimate the number of all facilities in good safety, health, and environmental condition.  In the fiscal year 2000 GPRA Report, BLM stated that it had validated the estimated 1998 baseline with fiscal year 2000 data, but did not disclose that this was done using a projection.

Another factor impacting the accuracy of fiscal year 2000 CASHE data was the length of time between inspections.  Completion of the first set of CASHE inspections for all BLM facilities occurred over a 7-year period.  Five of the facilities (10 percent) reported in good condition for fiscal year 2000 were inspected more than 5 years ago.  The second round of CASHE inspections are underway and should be completed within 3 years.  Both the shorter inspection cycle and the planned new data system (discussed on page 17 of this report) should improve the accuracy of CASHE GPRA reporting.

Our review of the data reported by BLM offices in four states for corrective actions taken at abandoned mine sites with physical safety hazards found that one state under-reported and two states over-reported the number of sites corrected (please see Table 4).  We verified reported corrections at state and field offices by requesting lists of mines where physical safety hazards were corrected in fiscal year 2000.  We then compared the lists to summary data in BLM's MIS database, which is used to report GPRA data.  We compared the description of the activity accomplished to the performance measure definition and determined whether the activity met the definition.  Where we identified discrepancies, we followed up with the state or field specialist.  The discrepancies identified were usually the result of reporting in the wrong year or reporting under the wrong program element.  Some discrepancies could not be explained.

Table 4

State
Hazards Corrected

Reported
Verified
Over (Under)
California
  30
  21
  9
Colorado
158
185
(27)
Nevada
321
321
  0
Oregon
  10
    4
  6

We also identified potential accuracy problems related to what actions should be counted as GPRA accomplishments.  Field staff stated that interpretations differed as to whether individual features at a site, such as a shaft, could be corrected and reported as an accomplishment or whether the entire site had to be corrected before reporting an accomplishment.  Current guidance does not provide a precise definition of a reportable site with physical safety hazards.  As a result, one field office reported 11 abandoned mine corrections associated with one site.  Another state reported six abandoned mine corrections at one site.  We believe that program staff should also consider whether temporary mine closures adequately fulfill the long-term GPRA goal of reducing threats to health and safety.  For example, erecting warning signs or fences around hazardous areas may or may not be sufficient action depending on the risk factors of the site.

We also determined that the data verification and validation section of the fiscal year 2000 GPRA Report incorrectly identified the Abandoned Mine Lands Inventory System (AMLIS) as the data source for abandoned mines.  The actual data source for GPRA is the MIS.  We confirmed with program staff that AMLIS was not fully functional as a database for abandoned mines with physical safety hazards.

BLM offices in four states over-reported cleanup accomplishments at hazardous material sites (please see Table 5).  We verified reported corrections at state and field offices by requesting lists of hazardous material sites where cleanup activity took place in fiscal year 2000.  We then compared the lists to summary data in BLM's MIS database, which was used to report GPRA data.  We compared the description of the activity accomplished to the performance measure definition and determined whether the activity met the definition.  Where we identified discrepancies, we followed up with the state or field specialist.  The discrepancies we identified were usually the result of reporting in the wrong year or reporting under the wrong program element.  Some discrepancies could not be explained.

Table 5

State
Cleanup Actions

Reported
Verified
Over
California
60
17
43
Colorado
13
11
  2
Nevada
25
24
  1
Oregon
85
31
54

Interpretations differed as to whether to count a response with no cleanup action as an accomplishment.  While additional guidance for reporting hazardous material accomplishments was provided in March 2000, field personnel stated that interpretation problems related to defining a reportable accomplishment still existed.  Acknowledging these problems, BLM completed four workshops at different state offices to clarify reporting requirements for fiscal year 2001.  The most recent workshop was held in November 2001 in California.

Although GPRA data for law enforcement were readily available from BLM's LAWNET database, the accomplishment for law enforcement was overstated by an estimated 2 percent nationwide.  The wording of the law enforcement goal "percentage of reported violations resulting in enforcement actions" implies a one-to-one relationship between violations and enforcement actions.  A violation, however, may have more than one enforcement action, and 127 of fiscal year 2000 violations had two or three enforcement actions6.  In one example, a "theft from vehicle" violation resulted in two enforcement actions-a criminal complaint lodged with state or local officials and an arrest.  Another example was a "marijuana cultivation" violation, which resulted in the execution of a search warrant and an arrest.

We noted that the LAWNET system had some procedures for verifying and supporting data.  Law enforcement officers in the field directly entered information into the LAWNET system, which was approved (verified) by managers before the entries were "uploaded."  Although this approval constituted a data quality control, one manager told us that his officers approved their own entries with his permission.  Law enforcement officers also kept logbooks of their activities, which supported entries into LAWNET.

While the net effect of multiple enforcements of 2 percent for fiscal year 2000 was small, the planned annual improvement in the goal was only 1 percent.  We understand that tracking all enforcement actions for a violation is important for the management of the law enforcement program; however, counting more than one enforcement action for any one violation is not accurate for GPRA reporting.  We believe this problem could be resolved by deducting multiple enforcements entered in LAWNET from the enforcement totals prior to submitting the data for inclusion in GPRA.

Many of the problems with the accuracy of data and related information in the fiscal year 2000 GPRA Report occurred because BLM managers at all levels did not systematically verify data.  BLM could have identified and corrected the data errors we noted if managers at each level had ensured the accuracy of reported data.

The Management Systems Group performs annual reviews of selected measures at judgmentally selected offices, but has reviewed only a portion of BLM's GPRA activities.  In 1999, a Management Systems Group specialist visited offices in three states and reviewed 13 of 67 GPRA measures.  In 2000, the same specialist visited two states and reviewed 9 of 45 GPRA measures.  During our audit, we reviewed Management Systems Group reports and accompanied the specialist on the 2001 review in Idaho.  We found the site visit and reports identified problems with data collection for GPRA measures and suggested corrective actions.  The Management Systems Group could improve its data validation and verification process by establishing a more systematic method of selecting measures and field offices for review and by establishing follow-up procedures to ensure that corrective actions were taken.

Data verification work at the program level was limited.  Headquarters program managers and some managers at state and field offices said that they compared targets and accomplishments for reasonableness.  They did not make these comparisons systematically, however, and this procedure does not ensure that reported data are accurate.  Performance data should be verified from the bottom up by BLM staff most familiar with day-to-day program activities.  Such verification, however, is not being done.  Field office staff told us that they were often not sure how program activities they performed were incorporated into the GPRA report or how the quality of their GPRA reporting could impact their program in the long term.

BLM has not established a systematic procedure to accumulate sufficient information to support the data included in the GPRA report for most of the measures.  This is particularly a problem for three of the measures we reviewed:  abandoned mines with physical safety hazards, hazardous materials cleanup, and CASHE.  BLM used the MIS to report GPRA data for abandoned mine lands and hazardous materials, and we expected that, at a minimum, BLM managers could provide us with a list of accomplishments reported.  The MIS, however, contains only cumulative numbers without specific details of actual accomplishments.

Field office managers were also unable to identify and verify accomplishments reported to the state office for GPRA purposes, in part because field office specialists were not required to provide documentary evidence to support the number entered in the MIS.  Maintenance of records of accomplishments by field specialists is vital and need not be burdensome.  A simple spreadsheet or journal entry of accomplishments completed with dates, contract dollars, and other pertinent information would be sufficient.  Some field specialists kept this kind of information, but apparently did not forward it to managers with the MIS data.

Likewise, state program managers should be able to identify and verify accomplishments from their field offices before forwarding this information to Headquarters.  The four state managers for the hazardous materials and abandoned mines programs, however, could not immediately identify their state's accomplishments without first contacting field office specialists.  National program managers, who should maintain minimal records of accomplishments being reported through them by states and incorporated into the annual GPRA report, also could not provide us with a listing of the 650 abandoned mine sites with physical safety hazards corrected or the 290 hazardous material sites cleaned up.

Another verification issue is related to the BLM CASHE facilities.  These facilities self-report that they have taken action to correct findings, but were not required to provide evidence showing that they handled the problems properly.  The CASHE Program also did not have a tracking system to document corrective actions taken.  The lack of a tracking system is significant because the findings requiring corrective actions could be violations of federal laws and regulations.  The CASHE program manager is developing a new information system capable of generating reports that describe uncorrected high priority findings.  The system will be tested in fiscal year 2002 and should be fully operational in fiscal year 2004, thereby supporting the compilation and verification of data that can be used to support this GPRA goal.

Establishing appropriate goals that focus on mission results is key to the performance management process.  If BLM's annual GPRA reports are to be useful to decision makers in BLM, Congress, and OMB, BLM should revise its GPRA goals and measures to focus more on mission, performance, and results and improve the accuracy and completeness of its data.  Through its choice of GPRA goals, BLM should be identifying its highest program priorities to its own managers and employees and to Congress and OMB.

Once BLM has selected appropriate goals and measures, it should ensure accurate, sufficient, and relevant information on program results to allow managers to assess what their programs do to achieve BLM objectives, how much they cost, and how effectiveness could be improved.  Accurate and timely performance data would also provide BLM with better control over resources used and improve accountability for results by program managers.

Appropriate goals and measures and accurate information are also needed to achieve the major GPRA objective of helping Congress develop a clearer understanding of what is being achieved in relation to what is being spent.  Congress can then determine whether progress is being made and whether the level of performance is sufficient to justify federal resources and effort.


We recommend that the Director, BLM:

1. Revise or develop the GPRA goals and measures, as needed, using the experience of 3 years of collecting and reporting GPRA data to ensure that all goals and measures support the overall goal of reducing threats, are under direct control of BLM managers, and are outcome based to the extent feasible.

2. Improve guidance and training on GPRA performance reporting at all levels of BLM, with emphasis on how GPRA reporting requirements link with the day-to-day operations of field personnel.

3. Develop procedures to accumulate information supporting the GPRA data reported.

4. Ensure the accuracy of the GPRA data reported by establishing a data verification and validation process that includes accountability by field, state, and national managers responsible for these programs.

5. Require certification of the accuracy of reported GPRA data at each level of reporting including field offices, state offices, and national program offices.

6. Ensure the completeness and accuracy of information in the GPRA report related to data sources and limitations.

In the July 31, 2002 response to the draft report (Appendix 3), the BLM Director concurred with the report's six recommendations.  The Director stated that BLM is taking steps to improve the GPRA data reported.  Subsequent contact with BLM officials provided the milestone dates for implementing these steps.  BLM is planning the following actions to address our respective recommendations.

1. BLM's Protection and Response Group has established a team to develop or improve performance measures to be more outcome oriented and link more directly to reducing threats.  BLM's new strategic plan, containing the revised measures, will be completed by September 30, 2003.

2. BLM's Business and Fiscal Resources Directorate is developing a training program which will emphasize how performance and cost data link to daily work and operations.  Training modules will be implemented by September 30, 2003.  Additionally, an Instruction Memorandum clarifying the GPRA reporting requirements and emphasizing how they link with day-to-day operations of field personnel will be distributed by October 31, 2002.

3. Through the Annual Work Plan Directives, BLM will require State and Field Office program leads to maintain information that supports the GPRA data reported.  The Annual Work Plan will be distributed by February 1, 2003.

4. BLM will require State and Field Offices to complete a Data Validation and Verification Assessment Matrix survey by July 30, 2003.  The survey results will be used by the Bureau to select sites and programs for evaluation to improve accountability by field, State, and national program managers in fiscal year 2004.

5. Through the Annual Work Plan Directives, BLM will direct the Washington Office, State and Field Offices to certify that accomplishments were reported accurately and consistent with data validation and verification procedures.  Certification will occur as part of BLM's quarterly review process.  The Annual Work Plan will be distributed by February 1, 2003.

6. Beginning with the fiscal year 2000 Accountability Report, BLM will implement the Department's data validation and verification requirements.  BLM's performance measure data will be reviewed internally and surnamed by the appropriate managers for accuracy.  The fiscal year 2002 Accountability Report will be issued by December 31, 2002.

Accordingly, the Director's response and supplemental information is sufficient to consider the six recommendations resolved, but not implemented.  The six recommendations will be referred to the Assistant Secretary for Policy, Management and Budget for tracking of implementation (Appendix 4).

Appendix 1
Scope and Methodology

The scope of our audit included a review of BLM procedures for establishing performance goals and measures, reporting performance accomplishments, and verifying and validating performance data.  To accomplish our objectives, we reviewed BLM performance goals, measures, and accomplishments for fiscal year 2000 and performance plans for fiscal years 2000 and 2001.  We interviewed BLM staff in Headquarters and in four state and eight field offices (please see Table 5).  We also reviewed the Management Systems Group's annual validation reports for 1999 and 2000 and accompanied BLM staff on the 2001 GPRA verification and validation review in Idaho (Table 5).  Our fieldwork was performed from April through September 2001.  Please see the Appendix 2 for prior reports and publications related to our review.

In addition, we reviewed relevant DOI and BLM annual reports and assurance statements for fiscal years 1998 through 2000.  These documents identified safety management as a material weakness in BLM's management controls and BLM and Department-wide maintenance management as mission critical material weaknesses.

Our audit was conducted in accordance with "Government Auditing Standards," issued by the Comptroller General of the United States.  Accordingly, we included such tests of records and other auditing procedures that were considered necessary to accomplish our objectives.

Appendix 1
Scope and Methodology

Table 6

Sites Visited 
Office
Location
Headquarters Office
Washington, DC
State Offices
Sacramento, California
Denver, Colorado
Reno, Nevada
Portland, Oregon
District/Field Offices
Folsom, California
Hollister, California
Gunnison, Colorado
Royal Gorge, Colorado 
Carson City, Nevada
Elko, Nevada
Eugene, Oregon
Prineville, Oregon
Verification And Validation Review
National Interagency Fire Center
Boise, Idaho
State Office
Boise, Idaho
Field Office
Shoshone, Idaho


Appendix 2
Prior Audit Coverage

Increasing interest in performance management has resulted in increased attention to GPRA planning and reporting by our office and the General Accounting Office (GAO).  Summaries of OIG reports and GAO testimony are as follows:

* In 2000, we suggested improvements in BLM's Fiscal Year 1999 Performance Report and 2001 Annual Performance Plan.7  Our suggestions included providing additional information to fully explain the goals and measures and the significance of each, describing the total program area for which a measure had been established, establishing more precise and limited goals to measure program performance, and adding annual goals or measures for some programs.

* In July 2000, GAO testified8 on the challenges identified in the implementation of GPRA.  The challenges included articulating a results orientation, coordinating crosscutting issues, showing performance consequences of budget decisions and the contribution of daily operations to results, and building the capacity to gather and use performance information.  GAO also highlighted the implementation of GPRA and the steps needed to maximize the usefulness of GPRA for Congress and the executive branch.

* In 2001, we issued a management letter9 identifying concerns about the GPRA reporting process.  We recommended that when using a percentage format, BLM report all performance indicators used in the calculation and all relevant facts necessary for the reader to understand the data.

Appendix 3
Agency Response

Appendix 3
Agency Response

Appendix 3
Agency Response

Appendix 3
Agency Response

Appendix 4
Status of Audit Recommendations

Recommendation
Status
Action Required
1 through 6
Resolved; not implemented
No further response to the Office of Inspector General is necessary.  The recommendations will be referred to the Assistant Secretary for Policy, Management and Budget for tracking of implementation.


1 BLM's GPRA performance measures are repeated in its performance goals.  As a result, if a measure is revised or added an associated goal must be revised or added.  To avoid repetition, we will use the term measure, rather than goal and measure, throughout the report.
2 Resolved when "Identification" was removed from the goal in fiscal year 2002.
3 An instruction memorandum was issued in April 2002 clarifying that MLAP funds may no longer be used for the correction of physical safety hazards or any other Abandoned Mine Lands activity.
4 Information included in the narrative sections of the GPRA Report such as data limitations and data sources.
5 The eight field offices visited during our fieldwork had oversight responsibility for only 31 (3 percent) of BLM's 1,124 dams.  The sample was too small for us to determine the accuracy of the reported data.
6 Our analysis included data reported as of June 2001 for fiscal year 2000.  This included an additional 1,833 violations not incorporated in the October MIS used for fiscal year 2000 GPRA reporting.
7 Review of the Fiscal Year 1999 Performance Reports and Fiscal Year 2001 Performance Plans for the U.S. Department of the Interior, OIG Report No. 00-I-533, June 2000.
8 Managing for Results:  Continuing Challenges to Effective GPRA Implementation, Testimony No. GAO/T-GGD-00-178, July 2000.
9 Management Letter:  Management Issues Identified During Audit of the Bureau of Land Management's Financial Statements for Fiscal Year 2000, OIG Management Report No. 01-I-309, March 2001.