Information Technology: Improvements for Acquisition of Customs  
Trade Processing System Continue, but Further Efforts Needed to  
Avoid More Cost and Schedule Shortfalls (25-OCT-07, GAO-08-46).  
                                                                 
The Department of Homeland Security (DHS) established the	 
Automated Commercial Environment (ACE) program to replace and	 
supplement existing cargo processing technology. According to the
fiscal year 2007 DHS appropriations act, DHS is to develop and	 
submit an expenditure plan for ACE that satisfies certain	 
conditions, including being reviewed by GAO. GAO reviewed the	 
plan to (1) determine whether the expenditure plan satisfies the 
legislative conditions, (2) determine the status of 15 open GAO  
recommendations, and (3) provide observations about the 	 
expenditure plan and DHS's management of the program. To address 
the mandate, GAO assessed plans and related documentation against
federal guidelines and industry standards and interviewed the	 
appropriate DHS officials.					 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-08-46						        
    ACCNO:   A77608						        
  TITLE:     Information Technology: Improvements for Acquisition of  
Customs Trade Processing System Continue, but Further Efforts	 
Needed to Avoid More Cost and Schedule Shortfalls		 
     DATE:   10/25/2007 
  SUBJECT:   Appropriation limitations				 
	     Cost analysis					 
	     Customs administration				 
	     Homeland security					 
	     Information resources management			 
	     Investment planning				 
	     Performance measures				 
	     Procurement planning				 
	     Program management 				 
	     Regulatory agencies				 
	     Reporting requirements				 
	     Risk management					 
	     Schedule slippages 				 
	     Strategic planning 				 
	     Cost overruns					 
	     Human capital management				 
	     Information security				 
	     Information security management			 
	     Cost estimates					 
	     Program implementation				 
	     Customs Service Automated Commercial		 
	     Environment System 				 
                                                                 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-08-46

   

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. 

Report to Congressional Committees: 

October 2007: 

Information Technology: 

Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls: 

GAO-08-46: 

GAO Highlights: 

Highlights of GAO-08-46, a report to congressional committees. 

Why GAO Did This Study: 

The Department of Homeland Security (DHS) established the Automated Commercial Environment (ACE) program to replace and supplement existing cargo processing technology. According to the fiscal year 2007 DHS appropriations act, DHS is to develop and submit an expenditure plan for ACE that satisfies certain conditions, including being reviewed by GAO. 

GAO reviewed the plan to (1) determine whether the expenditure plan satisfies the legislative conditions, (2) determine the status of 15 open GAO recommendations, and (3) provide observations about the expenditure plan and DHSï¿½s management of the program. To address the mandate, GAO assessed plans and related documentation against federal guidelines and industry standards and interviewed the appropriate DHS officials. 

What GAO Found: 

The ACE expenditure plan satisfies manyï¿½but not allï¿½of the legislative conditions specified in the fiscal year 2007 DHS appropriations act. Specifically, the plan (with related program documentation and officialsï¿½ statements) complies with acquisition rules, requirements, guidelines, and management practices of the federal government; includes a DHS certification that an independent verification and validation agent is under contract; was reviewed and approved by DHS and the Office of Management and Budget (OMB); and was reviewed by GAO. In addition, it partially satisfies conditions for meeting the capital planning and investment control review requirements established by OMB in Circular A-11 (part 7), including information security, and for complying with the DHS enterprise architecture. 

DHS has implemented eight open GAO recommendations made during the past 4 years, including those related to performance measures and targets, independent verification and validation, cost estimation, and program reporting. Seven other recommendations made during this time are in the process of being implemented. With respect to these seven, DHS has taken steps to satisfy each, such as establishing an accountability framework, reducing overlap and concurrence among ACE releases, and completing a privacy impact assessment, and actions are under way or planned to more fully address them. 

GAO is making three new observations about the expenditure plan and the management of ACE. First, the program is taking needed steps to redefine requirements for several ACE releases because of limitations in the completeness of original requirements, but this redefinition is likely to introduce significant program schedule delays and cost increases. Second, the changes to ACE requirements have led to replacement of a key commercial product, but the new product carries the risk of negatively impacting user productivity. Third, the automated database used for managing ACE risks is incomplete and does not contain information needed to adequately inform program decisions. 

All told, DHS has continued to make progress on ACE, and the program is better positioned today for delivering promised capabilities and benefits than it has been in the past. Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, GAO sees major program schedule delays and cost overruns on the horizon. To improve ACE management and minimize exposure to risk, it is important for DHS to remain vigilant in its efforts to satisfy ACE legislative requirements, fully implement prior GAO recommendations, and keep Congress fully informed about the programï¿½s status, plans, and risk. 

What GAO Recommends: 

GAO is making recommendations to further strengthen ACE management and accountability by disclosing program information in quarterly reports to Congress related to unmet legislative conditions, open GAO recommendations, program changes, and risk management. DHS agreed with GAOï¿½s findings and recommendations and described actions that it has under way and planned to address them. Most of the described actions are consistent with GAOï¿½s recommendations. 

To view the full product, including the scope and methodology, click on [hyperlink, http://www.GAO-08-46]. For more information, contact Randolph C. Hite at (202) 512-3459 or [email protected]. 

[End of section] 

Contents: 

Letter: 

Compliance with Legislative Conditions: 

Status of GAO Recommendations: 

Observations on the Expenditure Plan and Management of ACE: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendixes: 

Appendix I: Briefing to Subcommittees on Homeland Security, House and Senate Committees on Appropriations: 

Appendix II: Comments from the Department of Homeland Security: 

Appendix III: Contact and Staff Acknowledgments: 

ACE: Automated Commercial Environment: 

CBP: U.S. Customs and Border Protection: 

CIO: chief information officer: 

COTS: commercial off-the-shelf: 

DHS: Department of Homeland Security: 

GBB: global business blueprint: 

ITS: internet transaction server: 

OMB: Office of Management and Budget: 

PIA: privacy impact assessment: 

October 25, 2007: 

The Honorable Robert C. Byrd: 
Chairman: 
The Honorable Thad Cochran: 
Ranking Minority Member: 
Subcommittee on Homeland Security: 
Committee on Appropriations: 
United States Senate: 

The Honorable David E. Price: 
Chairman: 
The Honorable Harold Rogers: 
Ranking Minority Member: 
Subcommittee on Homeland Security: 
Committee on Appropriations: 
House of Representatives: 

The U.S. Customs and Border Protection (CBP), as part of the Department of Homeland Security (DHS), submitted to Congress in February 2007 its fiscal year 2007 expenditure plan for the Automated Commercial Environment (ACE) program pursuant to the Department of Homeland Security Appropriations Act, 2007.[Footnote 1] 

Begun in 2001, ACE is to replace and supplement existing cargo processing technology. The system is being developed and deployed in a series of increments to about 300 ports of entry and is expected to be fully deployed by the end of 2011. The goals of ACE include (1) supporting border security by enhancing analysis and information sharing with other government agencies and providing CBP with the means to decide before a shipment reaches the border what should be targeted because it is a security threat and what should be expedited because it complies with U.S. laws and (2) streamlining time-consuming and labor- intensive tasks for CBP personnel and the trade community through a national trade account and a single Web-based interface. 

As required by the appropriations act, we reviewed ACE's fiscal year 2007 expenditure plan. Our objectives were to (1) determine whether the expenditure plan satisfies the legislative conditions, (2) determine the status of 15 open GAO recommendations for ACE,[Footnote 2] and (3) provide observations about the expenditure plan and DHS's management of the program. 

On July 26, 2007, we provided a briefing to the staffs of the Subcommittees on Homeland Security, Senate and House Committees on Appropriations on the results of our review. This report transmits those results. The full briefing, including our scope and methodology, is reprinted in appendix I. 

Compliance with Legislative Conditions: 

The ACE expenditure plan--including related program documentation and program officials' statements--satisfies or partially satisfies the six legislative conditions specified in the appropriations act. Specifically, the plan satisfies the conditions that it (1) comply with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government;[Footnote 3] (2) include a certification by the DHS Chief Information Officer that an independent verification and validation agent is currently under contract for the program; (3) be reviewed by the DHS Investment Review Board, the Secretary of DHS, and Office of Management and Budget (OMB); and (4) be reviewed by us. The plan and its supporting information partially satisfies the conditions that it (5) meet the capital planning and investment control review requirements established by OMB (including Circular A-11, part 7) and that it (6) comply with the DHS enterprise architecture. For example, although DHS determined that ACE was aligned with the DHS enterprise architecture, the agency's analysis did not address key aspects of an architectural alignment, such as alignment with the architecture's data reference model. 

Status of GAO Recommendations: 

DHS has implemented some, but not all, of the recommendations pertaining to ACE that we have made since 2003. These recommendations, along with their status, are summarized here. 

Eight recommendations have been implemented. For example, DHS has: 

* developed a range of ACE performance measures and targets needed to support an outcome-based, results-oriented accountability framework, including user satisfaction with ACE; 

* aligned ACE program goals, benefits, desired business outcomes, and performance measures; 

* addressed those legislative conditions associated with measuring ACE performance and results and employing effective independent verification and validation practices; and: 

* ensured that future expenditure plans are based on cost estimates that are reconciled with independent cost estimates. 

The remaining seven recommendations are in the process of being implemented, as described here. 

* Recommendation: Define and implement an ACE accountability framework that fulfills several conditions,[Footnote 4] to include ensuring the currency, relevance, and completeness of commitments made to Congress in expenditure plans and reporting in future expenditure plans the progress against commitments that were contained in prior expenditure plans. 

In progress. The program has established an accountability framework that is providing input for both the annual expenditure plan and quarterly congressional reports. However, as with prior expenditure plans, the fiscal year 2007 expenditure plan did not reflect the most current program information. For example, information on milestones, earned value management, and risks were about 4 months old when the plan was submitted to the appropriations committees in February 2007, and program commitments were no longer current. Further, while program officials continue to use the quarterly ACE status reports to provide the appropriations committees with more detailed information, these reports are generally submitted to Congress 3 to 4 months after the end of each quarter, thus limiting their currency and relevance as well. CBP and DHS officials told us that the delays were due to the DHS review and approval process and that they are exploring ways to accelerate the process. 

The 2007 expenditure plan did not adequately report on progress against previous plan commitments. For example, the plan did not (1) report progress against milestones in the fiscal year 2006 plan or explain why these milestones were not achieved, (2) report actual obligation or expenditure of funds relative to the planned uses of these funds in prior expenditure plans, or (3) address progress against the milestone dates for each stage of a release that was included in the prior year's plan. According to program officials, the quarterly congressional reports provide more current information on the program's progress against prior commitments. However, these reports have also not fully addressed the commitments made in prior expenditure plans. Program officials stated that they intend to start providing this information in the last quarterly report of each year. 

* Recommendation: Define measures and collect and use associated metrics for determining whether prior and future program management improvements are successful. 

Planned. The program office has made changes that are to improve overall program management and, according to its December 2006 quarterly report to Congress, the program office plans to measure the impact of future management improvements. Moreover, this report stated that the program anticipates more changes, including creation of a cargo requirement management board to decide the disposition of all change requests to production systems; establishment of a new invoice review policy; and colocation of personnel within a given business area. However, program officials told us that they have yet to define measures to determine the impact of such changes and thus are not yet positioned to determine their success. 

* Recommendation: Minimize the degree of overlap and concurrency across ongoing and future ACE releases and capture and mitigate the associated risks of any residual concurrence. 

In progress. Since May 2006, the program office has reduced overlap and concurrence of ACE releases and has taken actions to reduce potential contention for limited resources by, for instance, decoupling (i.e., reducing dependencies among) certain program components; dividing releases into smaller subreleases to provide more flexibility in scheduling; improving planning for development, integration, testing, training activities, and milestones to better schedule use of development and test environments; and centralizing management of shared software services. Further, the program office conducts regular integration meetings with the teams supporting each release to discuss concerns, decisions, and schedules associated with resource availability and is using a software tool to track and mitigate release-specific concurrency risks. However, this tool contains vague or incomplete data relative to mitigating these risks. These data limitations make it difficult to determine the status or the effectiveness of the efforts to reduce the risks associated with overlap and concurrence among releases. 

* Recommendation: Direct the appropriate departmental officials to fully address those legislative conditions associated with having an approved privacy impact assessment (PIA)[Footnote 5] and ensuring architectural alignment. 

In progress. One legislative condition states that the plan should meet OMB's capital planning and investment control review requirements, which include addressing security and privacy issues. The program office has developed a PIA for ACE release 4 (e-Manifest: Trucks), which is currently operational. DHS approved this PIA on July 14, 2006. This PIA addressed the major elements of DHS's guidance, and program officials stated that they would update the assessment for each ACE release. However, this PIA did not cover other recently completed screening releases. Program officials told us that these screening releases were considered to be part of the Automated Targeting System and were covered by the Automated Targeting System's PIA. However, this PIA does not specifically identify or address ACE screening releases. 

With respect to architectural alignment, DHS determined in May 2007 that all required ACE products and technologies were aligned with the DHS technical reference model and that ACE was thus aligned with the DHS enterprise architecture. However, we have yet to receive sufficient documentation describing the criteria and methodology used to make these determinations or verifiable analysis supporting the determinations. Moreover, the determinations were based on technical alignment and did not address other relevant aspects of program alignment to an enterprise architecture, such as data alignment. 

* Recommendation: Develop and implement key human capital management practices. 

In progress. In June 2006, the Office of Information Technology Strategic Human Capital Management Plan, which included an ACE-specific appendix as a road map for effective management of human capital resources, was approved. However, the plan does not address the basic tenets of effective human capital management, such as defining the positions needed (including core competencies) to perform core program functions, assessing and inventorying current workforce skills and abilities, assessing any gaps between needed and existing workforce levels and capabilities, and filling identified gaps. Officials acknowledged these limitations in the plans and stated that they are developing an implementation plan to address these shortfalls. 

* Recommendation: Include in the June 30, 2006, quarterly update report to the appropriations committees a strategy for managing ACE human capital needs and the ACE framework for managing performance and ensuring accountability. 

In progress. The June 30, 2006, quarterly report to the House and Senate Appropriations Committees included the ACE program's strategy for meeting its human capital needs and its accountability framework; however, the human capital strategy does not meet the basic tenets of strategic human capital management, as previously explained. 

* Recommendation: Accurately report to the appropriations committees on progress in implementing our prior recommendations. 

In progress. Quarterly reports to the House and Senate Appropriations Committees have contained information on the status of our open recommendations since November 2002, but recent reports remain outdated due to a 2-to 7-month time lapse between when the reports are produced and when they are provided to the appropriations committees. DHS and program officials stated that they are exploring ways to accelerate the review process and thereby improve the timeliness and accuracy of their reports to Congress. 

Observations on the Expenditure Plan and Management of ACE: 

We have three observations about ACE requirements, commercial product selection, and risk management. 

* Requirements: Redefinition of requirements for several ACE releases is now under way to address limitations in completeness of originally defined requirements, and this redefinition is likely to introduce program schedule delays and cost increases. 

In defining ACE requirements, the program office discovered that its original approach did not adequately engage all key stakeholders, such as software programmers and subject matter experts, for the legacy system ACE is intended to replace. To address this, key stakeholders are now collaborating and decomposing legacy code. However, this effort is expected to significantly delay some system releases and drops. Program officials have taken several actions that they say will minimize the impact of the delays, such as prioritizing shared functionality, dividing releases/drops into smaller increments, and changing release deployment strategies. However, these changes have yet to be approved, and the full extent of the cost and schedule implications is not yet known. Moreover, neither the fiscal year 2007 expenditure plan nor the ACE quarterly reports have disclosed the requirements redefinition and its impact, and neither has addressed any changes to release deployment strategies. 

* Commercial product selection: Significant changes to ACE requirements have led to reevaluation and replacement of a key commercial off-the- shelf (COTS) product previously selected and being prepared for use. 

The program office conducted several analyses to determine which COTS products would best meet ACE system requirements, including a general review of various packages in 2002 to select a provider and a more detailed analysis in 2004 to define and allocate ACE requirements and select a specific product--the SAP Enterprise Portal. In December 2006, however, the ACE Chief System Architect determined through additional analysis that all planned SAP functionality could be provided by Internet Transaction Server (ITS) technology and recommended that ITS be adopted. The program office subsequently stopped work on SAP Enterprise Portal design and configuration efforts and reported that ITS would be used for release 5/drop A2 instead of the SAP Enterprise Portal. This decision is expected to have some near-term schedule impacts because much of the completed work for A2 had been based on the planned use of SAP Enterprise Portal. Further, use of ITS raises the risk of inadequate user response time, which would, in turn, negatively impact user productivity and introduce a high probability of significant cost and schedule impacts. Program officials reported that actions are under way to mitigate the risk through performance modeling and test planning. However, neither the fiscal year 2007 expenditure plan nor the quarterly reports to Congress disclose this COTS product change, its impact on release schedules and cost estimates, or the risk to future system performance. 

* Risk management: All program risks are not being effectively managed. 

The program office has developed a process guide and implemented an automated tool (database) for managing ACE risks in accordance with relevant guidance and best practices. Although the database contains fields to provide a description, level (high, medium, or low), and mitigation strategy (including start and end dates, exit criteria, and implementation status) for each risk, the completeness and quality of this information varies. Because of such database limitations, we could not determine the status of and mitigation progress on 17 risks. Moreover, these database limitations were not reflected in the documentation used at key program events, indicating that the program does not have the risk-related information that it needs to inform its program decisions and to reduce the chances of potential problems becoming actual problems. Program officials stated that they are taking steps to improve risk management, including establishing a group to ensure the quality and completeness of the database, holding regular group meetings with contract staff and team leads to discuss risks and their impacts, and conducting risk management training. To date, however, program risks have not been communicated to oversight organizations through the 2007 expenditure plan or recent quarterly reports to the House and Senate Appropriations Committees. 

Conclusions: 

Over the past 7 years, CBP and DHS have worked to fulfill legislatively mandated annual expenditure plan requirements and to implement dozens of our recommendations related to these plans and management of the program. Among other things, these requirements and recommendations have promoted effective program management and accountability for performance and results. As a result of these years of effort, the ACE program is better positioned today for delivering promised capabilities and benefits than it has been in the past. 

Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management continue to remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, avoiding major program schedule delays and cost overruns remains a challenge as more of each appears to be on the horizon. To further improve ACE management and minimize its exposure to risk, it is important for CBP and DHS to remain vigilant in their efforts to satisfy ACE legislative requirements and to fully implement our prior recommendations. Moreover, it is important that they keep Congress fully informed on where the program stands and what changes are planned to address emerging cost overruns and schedule delays. 

Recommendations for Executive Action: 

To further strengthen ACE management and promote accountability for ACE performance and results, we are recommending that the Secretary of Homeland Security direct the CBP Commissioner to ensure that future quarterly reports to the House and Senate Appropriations Committees disclose: 

* the risks and associated mitigation strategies of not having fully satisfied the expenditure plan legislative conditions and not having completed implementation of all our prior recommendations; 

* the status and impacts on the program's estimated cost and schedule and lessons learned from ongoing efforts to redefine requirements and to implement a different COTS product than originally selected; and: 

* the program's plans and actions for improving ACE risk management and its current inventory of program risks, including their associated mitigation strategies and the status of the strategies' implementation. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report signed by the Director, Departmental GAO/OIG Liaison, and reprinted in appendix II, DHS agreed with our findings and stated that it is committed to addressing them. Further, the department agreed with our recommendations and described actions that it said are under way or planned to address them. While most of the department's stated actions are consistent with our recommendations, in one case they may not be sufficient. Specifically, the department stated that it would ensure that future expenditure plans meet all OMB capital planning and investment control review requirements by attaching the required OMB budget submission for ACE, referred to as an Exhibit 300, to all future plans. However, we reviewed the fiscal year 2007 ACE Exhibit 300 as part of our review of the fiscal year 2007 ACE expenditure plan and found that it did not meet the OMB requirements cited above. Therefore, unless DHS improves the quality of future ACE Exhibit 300s by addressing the weaknesses we have identified, this action alone may not fully address our recommendation. 

We are sending copies of this report to the Chairmen and Ranking Minority Members of the other Senate and House committees and subcommittees that have authorization and oversight responsibilities for homeland security. We are also sending copies to the DHS Secretary, the CBP Commissioner and, on their request, to other interested parties. In addition, the report will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. 

Should you or your offices have any questions on matters discussed in this report, please contact me at (202) 512-3459 or at h [Hyperlink, [email protected]] [email protected]. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other contacts and key contributors to this report are listed in appendix III. 

Signed by:  

Randolph C. Hite: 
Director, Information Technology Architecture: 
and Systems Issues: 

[End of section] 

Appendix I: Briefing to Subcommittees on Homeland Security, House and Senate Committees on Appropriations: 

Information Technology: Management Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls: 

Briefing to the Staffs of the Subcommittees on Homeland Security, Senate and House Committees on Appropriations:  

July 26, 2007: 

Briefing Overview: 

Introduction: 

Objectives: 

Results in Brief: 

Background: 

Results: 

* Legislative Conditions: 

* Status of Recommendations: 

* Observations: 

Conclusions: 

* Recommendations for Executive Action: 

Agency Comments: 

Attachment 1. Scope and Methodology: 

Attachment 2. Related GAO Products: 

Introduction: 

The U.S. Customs and Border Protection[Footnote 6](CBP) is developing a new import and export processing system, referred to as the Automated Commercial Environment (ACE), to replace and supplement existing cargo processing technology. Begun in 2001, this system is being developed and deployed in a series of increments to about 300 ports of entry and is expected to be fully deployed by the end of 2011. 

The goals of ACE include: 

* supporting border security by enhancing analysis and information sharing with other government agencies and providing CBP with the means to decide before a shipment reaches the border what should be targeted because it is a security threat and what should be expedited because it complies with U.S. laws; and: 

* streamlining time-consuming and labor-intensive tasks for CBP personnel and the trade community through a national trade account and a single Web-based interface. 

The Department of Homeland Security Appropriations Act, 2007,[Footnote 7] states that DHS may not obligate $216.8 million of the $316.8 million appropriated for ACE until the House and Senate Committees on Appropriations receive a plan for expenditure that: 

1. meets the capital planning and investment control review requirements established by the Office of Management and Budget (OMB), including Circular A-11, part 7;[Footnote 8]: 

2. complies with DHSï¿½s information systems enterprise architecture; 

3. complies with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government; 

4. includes a certification by the Chief Information Officer (CIO) of DHS that an independent verification and validation agent (IV&V) is currently under contract for the project;

5. is reviewed and approved by the DHS Investment Review Board (IRB),[Footnote 9] the Secretary of Homeland Security, and OMB; and: 

6. is reviewed by GAO.

On February 6, 2007, DHS submitted its fiscal year 2007 expenditure plan for $316.8 million to the House and Senate Appropriations Subcommittees on Homeland Security. In addition, CBP submits quarterly reports to the House and Senate Appropriations Committees to keep them apprised of ACE progress and issues.

Objectives: 

As agreed, our objectives were to: 

1. determine whether the ACE fiscal year 2007 expenditure plan satisfies the legislative conditions,

2. determine the status of 15 open GAO recommendations for ACE,[Footnote 10] and: 

3. provide observations about the expenditure plan and DHSï¿½s management of the ACE program. 

We conducted our work at CBP headquarters and contractor facilities in the Washington, D.C., metropolitan area from December 2006 through July 2007 in accordance with generally accepted government auditing standards. Details of our scope and methodology are provided in attachment 1. Related GAO products are in attachment 2.

Results in Brief: Objective 1: 

Legislative Conditions: 

Table: Summary of satisfaction of legislative conditions: 

Legislative condition: Meets the capital planning and investment control review requirements established by OMB, including OMB Circular A-11, part 7; 
Status [A] [B]: Partially satisfied. 

Legislative condition: Complies with the DHS enterprise architecture; 
Status [A] [B]: Partially satisfied. 

Legislative condition: Complies with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government; 
Status [A] [B]: Satisfied. 

Legislative condition: Includes a certification by the DHS CIO that an IV&V agent is currently under contract for the project; 
Status [A] [B]: Satisfied. 

Legislative condition: Is reviewed and approved by the DHS IRB, Secretary of DHS, and OMB; 
Status [A] [B]: Satisfied. 

Legislative condition: Is reviewed by GAO; 
Status [A] [B]: Satisfied. 

Source: GAO. 

[A] Partially satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying many, but not all, key aspects of the condition that we reviewed. 

[B] Satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying every aspect of the condition that we reviewed. 

[End of table] 

Results in Brief: Objective 2: 

Open Recommendations: 

Table: Implementation of prior GAO recommendations GAO Recommendations[A]: 

GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures; 
Status [B,C,D]: [Empty]. 

GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures: a. coverage of all program commitment areas, including key expected or estimated system (1) capabilities, use, and quality; (2) benefits and mission value; (3) costs; and (4) milestones and schedules; 
Status [B,C,D]: Complete. 

GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures: b. currency, relevance, and completeness of such commitments made to Congress in expenditure plans; 
Status [B,C,D]: In progress. 

GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures: c. reliable data relevant to measuring progress against commitments; 
Status [B,C,D]: Complete. 

GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures: d. reporting in future expenditure plans progress against commitments contained in prior expenditure plans; 
Status [B,C,D]: In progress. 

GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures: e. use of criteria for exiting key readiness milestones that adequately consider indicators of system maturity, such as severity of open defects, and document milestone decisions in a way that reflects the risks associated with proceeding with unresolved severe defects and provides for mitigating these risks.[E]; 
Status [B,C,D]: Complete. 

GAO Recommendation: 2. Develop the range of realistic performance measures and targets needed to support an outcome-based, results-oriented accountability framework, including user satisfaction; 
Status [B,C,D]: Complete. 

GAO Recommendation: 3. Explicitly align program goals, benefits, desired business outcomes, and performance measures; 
Status [B,C,D]: Complete. 

GAO Recommendation: 4. Define measures and collect and use associated metrics for determining whether prior and future program management improvements are successful; 
Status [B,C,D]: Planned. 

GAO Recommendation: 5. Fully address those legislative conditions associated with measuring performance and results and employing effective IV&V practices; 
Status [B,C,D]: Complete. 

GAO Recommendation: 6. Ensure that future expenditure plans are based on cost estimates that are reconciled with independent cost estimates; 
Status [B,C,D]: Complete. 

GAO Recommendation: 7. Develop and implement a rigorous and analytically verifiable cost estimating program that embodies the tenets of effective estimating; 
Status [B,C,D]: Complete. 

GAO Recommendation: 8. Use earned value management (EVM)[F]in developing all existing and future releases; 
Status [B,C,D]: Complete. 

GAO Recommendation: 9. Have future expenditure plans specifically address any proposals or plans for extending and using ACE infrastructure to support other homeland security applications; 
Status [B,C,D]: Complete. 

GAO Recommendation: 10. Minimize the degree of overlap and concurrence across ongoing and future releases, and capture and mitigate the associated risks of any residual concurrence; 
Status [B,C,D]: In progress. 

GAO Recommendation: 11. Fully address those legislative conditions associated with having an approved privacy impact assessment (PIA) and ensuring architectural alignment; 
Status [B,C,D]: In progress. 

GAO Recommendation: 12. Develop and implement missing human capital management practices; 
Status [B,C,D]: In Progress. 

GAO Recommendation: 13. Include in the June 30, 2006, quarterly update report to the appropriations committees a strategy for managing human capital needs and the framework for managing performance and ensuring accountability; 
Status [B,C,D]: In progress. 

GAO Recommendation: 14. Report to House and Senate Appropriations Committees on a quarterly basis on efforts to address open GAO recommendations; 
Status [B,C,D]: Complete. 

GAO Recommendation: 15. Accurately report to the appropriations committees on CBP's progress in implementing our prior recommendations; 
Status [B,C,D]: In Progress. 

Source: GAO. 

[A] With respect to the fiscal year 2007 expenditure plan.

[B] Complete means that actions have been taken to fully implement the recommendation. 

[C] In progress means that actions are under way to implement the recommendation. 

[D] Planned means actions are planned to implement the recommendation.

[E] This is a combination of two related recommendations.

[F] EVM is a management tool for measuring progress and is both an industry accepted practice and an OMB requirement. 

[End of table] 

Results in Brief: Objective 3: 

Observations: 

Summary of observations: 

* Limitations in the completeness of the original requirements for several ACE releases have resulted in changes to how requirements are being defined, as well as changes to the requirements themselves. These changes are likely to produce significant schedule delays and cost growth, and neither the expenditure plan nor the quarterly reports to Congress have disclosed these risks. 

* Requirements changes have led to reevaluation and replacement of a key commercial software product that was previously selected based on incomplete requirements. This change carries schedule and cost risks that neither the expenditure plan nor the quarterly reports have disclosed. 

* Management of ACE risks has not been effective, but improvements are planned to better anticipate and avoid problems that cause schedule delays and cost growth. ACE risks are not disclosed in either the expenditure plan or in the quarterly reports. 

Results in Brief: 

Recommendations and Agency Comments: 

To reduce ACE exposure to future schedule delays and cost growth and to promote greater accountability, we are making recommendations to DHS aimed at disclosing the nature of and progress in addressing the risks associated with not having fully satisfied expenditure plan legislative conditions, not having completed implementation of all prior GAO recommendations, and having to redefine ACE requirements and reselect a key commercial software product. 

In oral comments on a draft of this briefing, DHS and CBP officials agreed with our conclusions and recommendations, and provided clarifying information and technical comments that we incorporated in the briefing, as appropriate. 

Background: 

Program Overview: 

CBP is about 6 years into its trade processing modernization program, known as ACE. Among other things, ACE is to introduce reengineered business processes and next generation cargo processing technology, and it is to support CBPï¿½s mission of (1) protecting the American public against terrorism and (2) enforcing the laws of the United States while fostering our nationï¿½s economic security through lawful international trade and travel. 

ACE is also to support provisions of Title VI of the North American Free Trade Agreement, commonly known as the Customs Modernization Act. Subtitle B of the Act[Footnote 11] contains provisions that were intended to enable the government to modernize international trade processes and permit CBP to adopt an informed compliance approach with industry using automated systems. 

The goals of ACE are to: 

* enhance analysis and information sharing with other government agencies relative to new national security threats; 

* provide CBP personnel with the technology and information needed to decide, before a shipment reaches the border, what should be targeted because it is a security threat and what should be expedited because it complies with U.S. laws;

* enable the efficient collection, processing, and analysis of commercial import and export data via an integrated, fully automated information system; 

* reduce costs for the government and the trade community by streamlining time-consuming and labor-intensive tasks; 

* enable government and trade communities to process, view, and manage accounts nationally and obtain historical information on cargo, conveyances, and crew, based on screening and targeting rules; and: 

* enable government to comply with legislative mandates to improve efficiency/effectiveness and provide better customer service to U.S. citizens. 

Background: 

Program Organization: 

Several CBP components support the execution of the ACE program. 

* The Cargo Systems Program Office (referred to in this briefing as the program office) is responsible for the implementation of ACE. This office has primary responsibility for ACE program management; is responsible for the day-to-day management of ACE modernization activities; and develops the policies, standards, processes, and metrics by which the program is managed and measured. 

* The ACE program office is located within CBPï¿½s Office of Information Technology (OIT), which is headed by the Assistant Commissioner for Information and Technology.

* The ACE program office is supported by the Targeting and Analysis Systems Program Office (a merger of ACE screening and targeting resources and the Office of Information Technologyï¿½s Interprocess Solutions Branch), which is responsible for acquisition and development of ACE screening and targeting functionality.

* The program office is supported by CBPï¿½s Office of Finance, Office of Strategic Trade, Office of Regulations and Rulings, and more than 100 participating government agencies and offices. 

Background: 

Contractors: 

CBP awarded a 15-year, indefinite delivery/indefinite quantity, prime integration contract (5-year base with two 5-year options) to IBM Global Services in April 2001 for development and implementation of ACE. CBP exercised the first option in April of 2006. 

* IBM and its subcontractors-collectively called the ACE Support Team[Footnote 12]-provide ACE with program management support, enterprise engineering, systems planning and development, and systems business process reengineering; technology architectures; prototyping; systems and application design; software development, testing, and evaluation; deployment; and developmental operations support. 

* IBM is also under contract for operations and maintenance of deployed ACE releases, as well as for ongoing enhancements (new capabilities, referred to as program baseline enhancements) implemented in sub-releases that are scheduled in-between major ACE releases. 

Background: 

Acquisition Approach and Cost: 

CBP also relies on support contractors to provide a range of management support, such as program management, financial management, process improvement, quality assurance, requirements and configuration management, program communication, organizational learning, human capital planning, specialized cost analysis and life cycle cost estimating/modeling services, and procurement support.

In 2002, ACE was to be completed in four increments over 4 years at an estimated cost of between $1.5 billion and $1.6 billion. 

The ACE Program Plan (version 2.1, dated August 2006) states that ACE is expected to be fully deployed and operational by the end of 2011 at a cost of about $3.3 billion. However, program officials told us that cost and schedule estimates are being revised, but have not yet been approved and therefore were not provided for our review. In the fiscal year 2007 ACE expenditure plan, CBP reports that it has spent about $1.7 billion on the program. 

ACE is now being acquired and implemented through a series of 10 increments (referred to as releases), which are further divided into major system deliveries, known as ï¿½drops.ï¿½ (See the following six slides for a description of the increments and their status.)

Background: 

Description of ACE Increments: 

A description of the 10 ACE increments (releases and related drops) follows. 

Release 1 (ACE Foundation): Computer hardware and system software (infrastructure) to support subsequent system releases. 

Release 2 (Account Creation): Initial group of national account managers1 and 41 importers access to account information, such as trade activity. 

Release 3 (Periodic Payment): Additional account managers and importers, as well as brokers and carriers,2 with access to account information; provides initial financial transaction processing and revenue collection capability, allowing monthly payments of duties and fees. 

Release 4 (e-Manifest: Trucks): Electronic truck manifest3 processing and interfacing to legacy enforcement systems and databases. 

Background: 

Description of ACE Increments: 

Release 5 (Entry Summary, Accounts, and Revenue (ESAR)): SAP[Footnote 16] technologies to enhance and expend accounts management, financial management, and entry summary functionality. 

* Master Data and Enhanced Accounts (drop A1): SAP to deliver enhanced account creation and maintenance functionality and expand the types of accounts managed in ACE. 

* Entry Summary and Revenue (drop A2): Entry summary, interfaces with participating government agencies, calculation of duties and fees, reconciliation processing, and refunds. 

Release 6 (e-Manifest, All Modes, and Cargo Release): Electronic manifest capability for rail, air, and sea shipments; provides a multimodal manifest;[Footnote 17] enables full tracking of cargo, conveyances, individuals, and equipment; and enhances enforcement processes for rail, air, and sea. 

* e-Manifest: Rail and Sea Manifest (drop M1): Electronic manifest functionality for rail and sea shipments; rail, sea, and truck electronic manifests into the multimodal manifest. 

* e-Manifest: Air Manifest and Cargo Release (drop M2): Electronic manifest to air shipment and brings all modes of transportation into the multimodal manifest.

* e-Manifest: Exports and Mail Entry Writing System (drop M3): Tracking of cargo, conveyances, individuals, and equipment for truck, sea, rail, and air manifests. 

Release 7 (Exports and Cargo Control): Remaining accounts management, revenue, manifest, release, and export functionality. 

* ESAR: Drawback, Protest, and Importer Activity Summary Statement (IASS) (drop A3): Import activity summary statement,[Footnote 18] drawback functionality, and enhanced protest; online processing for trade account applications.

E-Manifest: Final Exports and Manifest (drop M4): Electronic manifest for mail, pipeline, and hand carry; electronic export processing. 

Screening S1 (Screening Foundation): Foundation for screening cargo and conveyances by centralizing criteria and results into a single standard database; allows user definition and maintenance of data sources and business rules for air, rail, sea, and truck modes of transportation. 

Screening S2 (Targeting Foundation): Platform and foundation for advanced targeting capabilities by enabling CBPï¿½s National Targeting Center to search multiple databases for relevant facts and actionable intelligence and infer relationships between entities and data elements; architecture for integrating new data sources (including integrating external data sources and providing a single sign on capability), implementing analytical tools, and deploying analytical capabilities.

Screening S3 (Advanced Targeting Capabilities): Screening for reconciliation, intermodal manifest, Food and Drug Administration data, and in-bond, warehouse, and foreign trade zone authorized movements; integrates additional data sources into targeting capability; and risk management capability.

Background: 

Status of ACE Increments: 

According to CBP, as of July 2007,: 

* Five increments are fully operational: Releases 1, 2, and 3, and Screenings S1 and S2.

* One increment (Release 4) has been deployed at 94 of the 99 truck land border ports.

* Three increments are at various stages of development and/or deployment: Release 5/Drops A1 and A2, Release 6/Drop M1, and Screening 

Background: 

Status of ACE Increments: 

Figure: ACE Schedule: 

This figure is a bar chart showing the ACE schedule. 

[See PDF for image] - graphic text: 

Source: GAO analysis based on CBP data. 

[End of table] 

Background: 

Status of ACE Use: 

CBP reports increased use of operational ACE increments. For example: 


* The number of external accounts[Footnote 19] grew from 41 in June 2003 to 736 in August 2005, and stood at about 4,100 in October 2006. As of November 2006, about 4,500 corporate entities had been approved to pay monthly duties and fees.

* Total revenue collections through ACE grew from $84,673 in June 2004 to $1.3 billion in July 2005, and as of October 2006 stood at about $8 billion. 

* The number of e-Manifests that were filed using ACE grew from 20,847 in December 2006 to 45,548 in January 2007. 

Background: 

Funding for ACE Expenditure Plans: 

Table: Funding for ACE Expenditure Plans: 

Release: 1: ACE Foundation:-through Operational Readiness; 
Budgeted amount: Prior years[A]: 114.8; 
Budgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: 114.8. 

Release: 2: Account Creation-through Operational Readiness; 
Budgeted amount: Prior years[A]: 114.8; 
Budgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: 114.8. 

Release: 3: Periodic Payment:-through Operational Readiness; 
Budgeted amount: Prior years[A]: 172.9; 
dgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: 183.2. 

Release: 4: e-Manifest Trucks-through Operational Readiness; 
Budgeted amount: Prior years[A]: 10.3; 
Budgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: 183.2. 

Release: 5(A1): ESAR: Master Data and Enhanced Amounts; 
Budgeted amount: Prior years[A]: 172.3; 
Budgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: 233.6. 

Release: 5(A2): ESAR: Entry Summary and Revenue; 
Budgeted amount: Prior years[A]: 23.3; 
Budgeted amount: Fiscal year: 38.0; 
Budgeted amount: Total: 233.6. 

Release: 6(M1): e-Manifest- Rail and Sea Manifest; 
Budgeted amount: Prior years[A]: 46.3; 
Budgeted amount: Fiscal year: 38.3; 
Budgeted amount: Total: 95.0. 

Release: 6(M2): e-Manifest- Air and Cargo Release; 
Budgeted amount: Prior years[A]: 10.4; 
Budgeted amount: Fiscal year: 38.3; 
Budgeted amount: Total: 95. 

Release: 6(M3): Exports and Mail Entry Writing System; 
Budgeted amount: Prior years[A]: [Empty]; 
Budgeted amount: Fiscal year: 38.3; 
Budgeted amount: Total: 95. 

Release: 7(A3): ESAR: Drawback, Protest and IASS[B]; 
Budgeted amount: Prior years[A]: [Empty]; 
Budgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: [Empty]. 

Release: 7(M4): e-Manifest Custodial Entities, Pipelins, and Batch Processes[C]; 
Budgeted amount: Prior years[A]: [Empty]; 
Budgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: [Empty]. 

Release: S1: Screening Foundation; 
Budgeted amount: Prior years[A]: 48.7; 
Budgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: 48.7. 

Release: S2: Targeting Foundation; 
Budgeted amount: Prior years[A]: 39.3; 
Budgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: 39.3. 

Release: S3: Advanced Targeting; 
Budgeted amount: Prior years[A]: 21.8; 
Budgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: 21.8. 

Activity: ACE Operations and Maintenance; 
Budgeted amount: Prior years[A]: 143.1; 
Budgeted amount: Fiscal year: 44.2; 
Budgeted amount: Total: 187.3. 

Activity: ACE Production Baselines; 
Budgeted amount: Prior year[A]: [Empty]; 
Budgeted amount: Fiscal year: 22.6; 
Budgeted amount: Total: 22.6. 

Activity: ACE Foundation Architecture and Engineering; 
Budgeted amount: Prior year[A]: 90.4; 
Budgeted amount: Fiscal year: 35.7; 
Budgeted amount: Total: 126.1. 

Activity: ACE Implementation Infrastructure and Support; 
Budgeted amount: Prior year[A]: 269.5; 
Budgeted amount: Fiscal year: 62.9; 
Budgeted amount: Total: 332.4. 

Activity: ACE Foundation Program Management; 
Budgeted amount: Prior year[A]: 137.5; 
Budgeted amount: Fiscal year: 18.9; 
Budgeted amount: Total: 156.4. 

Activity: ACE Communications, Training, Outcome, and Deployment; 
Budgeted amount: Prior year[A]: 31.7; 
Budgeted amount: Fiscal year: 24.4; 
Budgeted amount: Total: 56.1. 

Activity: Other Tasks; 
Budgeted amount: Prior year[A]: 20.6; 
Budgeted amount: Fiscal year: [Empty]; 
Budgeted amount: Total: 20.6. 

Activity: Cargo Systems Program Office; 
Budgeted amount: Prior year[A]: 201.2; 
Budgeted amount: Fiscal year: 25.3; 
Budgeted amount: Total: 226.5. 

Activity: International Trade Data System; 
Budgeted amount: Prior year[A]: 59.4; 
Budgeted amount: Fiscal year: 16; 
Budgeted amount: Total: 75.4. 

Activity: Management Reserve; 
Budgeted amount: Prior year[A]: 92; 
Budgeted amount: Fiscal year: 4.6; 
Budgeted amount: Total: 96.6. 

Activity: International Trade Data System Funding for Development[D]; 
Budgeted amount: Prior year[A]: [Empty]; 
Budgeted amount: Fiscal year: -14; 
Budgeted amount: Total: -14. 

Appropriated total; 
Budgeted amount: Prior year[A]: 1707; 
Budgeted amount: Fiscal year: 317; 
Budgeted amount: Total: 2024. 

Source: GAO analysid of CBP data. 

[A] Prior funding consists of stopgap funding, approved by the appropriations committees in March 2001, and the seven previous expenditure plans. 

[B,C] Funding for A3 and M4 has not yet been included in the expenditure plans. 

[D] This amount consists of prior year unobligated funds that ITDS provided to ACE to address requirements for participating government agencies. 

[End of table] 

Objective 1: Legislative Conditions: 

Condition 1: Partially Satisfied: 

The 6 legislative conditions have been either satisfied or partially satisfied. 

Table: Legislative condition: 

Legislative condition: 1. Meets the capital planning and investment control review requirements established by the Office of Management and Budget (OMB), including OMB Circular A-11, part 7; 
Status [A,B]: Partially satisfied. 

Legislative condition: 2. Complies with DHSï¿½s enterprise architecture; 
Status [A,B]: Partially satisfied. 

Legislative condition: 3. Complies with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government; 
Status [A,B]: Satisfied. 

Legislative condition: 4. Includes a certification by the DHS CIO that an IV&V agent is currently under contract for the project; 
Status [A,B]: Satisfied. 

Legislative condition: 5. Is reviewed and approved by the DHS IRB, Secretary of Homeland Security, and OMB; 
Status [A,B]: Satisfied. 

Legislative condition: 6. Is reviewed by GAO; 
Status [A,B]: Satisfied. 

Source: GAO. 

[A] Partially satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying many, but not all, key aspects of the condition that we reviewed.

[B] Satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying every aspect of the condition that we reviewed. 

[End of table] 

Condition 1: The plan, including related program documentation and program officialsï¿½ statements, partially satisfies the capital planning and investment control review requirements established by OMB, including Circular A-11, part 7.[Footnote 20] 

The table that follows provides an overview of the results of our analysis and selected examples in areas where A-11 requirements have or have not been fully satisfied. Given that the A-11 requirements are intended to minimize a programï¿½s exposure to risk, permit performance measurement and oversight, and promote accountability, any areas in which the program falls short of the requirements reduces the chances of delivering cost effective capabilities and measurable results on time and within budget. 

Objective 1: Legislative Conditions: 

Condition 1: Partially Satisfied:  

Table: Condition 1: Partially Satisfied:  

Table: Examples of A-11 Conditions Results of Our Analysis: 

Provide a brief description of the investment and its status in the capital planning and investment control review process, including major assumptions made about the investment; 
The expenditure plan and the ACE Exhibit 300 budget submission provide brief descriptions of the ACE investment and its releases. The Exhibit 300 also describes the status of ACE relative to DHSï¿½s capital planning and investment control process, and states that ACE is in the ï¿½controlï¿½ stage of the process;  
The Exhibit 300 and the ACE program plan also describe investment assumptions, such as (a) an annual budget of $305.5 million will be provided; (b) incremental deliveries within releases will mitigate risks and reduce uncertainty of program estimates; (c) ITDS funding will be timely, separate, and adequate; (d) stakeholder representation and resources will be adequate. 

Report performance goals and measures for existing investments and show how the Federal Enterprise Architecture (FEA) Performance Reference Model (PRM) applies to this investment; 
The expenditure plan reports the programï¿½s performance goals, benefits, objectives, performance measures, and their relationships for some, but not all, increments, including actual performance for fiscal years 2005 through 2006. The plan includes the ACE accountability framework, which is the programï¿½s overall tool for implementing CBPï¿½s PRM, which in turn is based on the FEA PRM. CBPï¿½s PRM includes, for example, performance measures for user satisfaction, efficiency, productivity, and data reliability, among other elements, and the ACE accountability framework incorporates a subset of these PRM measures, augmented by additional measures focusing on the programï¿½s status; 
ACEï¿½s Exhibit 300 also presents performance goals, targets, measures, and results for 2003 through 2012. However, these are not fully consistent with the expenditure plan. According to CBP, this is because the ACE performance measures were aligned with DHSï¿½s goals, objectives, strategies, and desired results as of July 2006, which is after the fiscal year 2007 Exhibit 300 was submitted. In addition, the performance measures continue to evolve as more understanding is gained in the usefulness and meaning of measures. (See the open recommendations section of this briefing for more information on performance measures.)

Provide a summary of the investmentï¿½s risk assessment, including how 19 OMB-identified risk elements are being addressed; 
The expenditure plan presents some, but not all, of the identified risks for the programï¿½s releases and activities. The Exhibit 300 also presents program risks, assessments, mitigation strategies, and status, and it organizes them by OMB risk categories. However, the risks in the expenditure plan and the Exhibit 300 are not fully consistent with each other or with the risks being tracked in the programï¿½s risk database. Further, the most recent risk assessment is limited to security risks for Release 4; 
The ACE program has a documented risk management process that includes identifying, classifying, reporting, and tracking risks. However, this process has not been fully implemented. For example, some risks that are identified in the risk database and the accountability framework have missing or outdated status information and mitigation strategies. As a result, the status of all risks is not clear. Program officials stated that risk management is not yet mature, but that they are taking steps to improve it. (See the open recommendation section of this briefing for more information on risk management.)

Provide a summary of the investmentï¿½s status in accomplishing baseline cost and schedule goals through the use of an earned value management (EVM) system or operational analysis, depending on the life-cycle stage; 
The expenditure plan reported some, but not all, EVM data for ACE releases and activities. For example, the data includes planned, estimated, and actual schedule milestones, but it does not include schedule variances; 
The EVM data in the expenditure plan was from the ACE accountability framework that program officials rely on to manage the program. (See the open recommendations section of this briefing for more information on the accountability framework.) According to program officials, they are using EVM to manage all ACE releases and screenings under contract and EVM tracking begins with an Integrated Baseline Review, which forms the basis for a realistic plan against which to objectively measure work to be completed during the period of performance; 
The Exhibit 300 also includes EVM data and describes contractor requirements for EVM, including verification of EVM compliance with industry standards. According to program officials, these contractual requirements are still operative; 
Nevertheless, the EVM program has still not been certified, as required by OMB. According to program officials, this certification is scheduled for the beginning of fiscal year 2008. As an interim measure, the prime contractor performed an EVM compliance and internal surveillance audit against relevant EVM criteria[Footnote 21] in March 2006 and found no issues. 

Provide a description of the investment's privacy and security issues. Summarize the agency's ability to manage security at the system or application level. Demonstrate compliance with the certification and accreditation processes as well as the mitigation of IT security weaknesses; 
Privacy: The expenditure plan does not discuss privacy issues. However, the Exhibit 300 states that ACE is subject to the privacy provisions of the E-Government Act of 2002. It further states that a privacy impact assessment was conducted in 2005 for Release 4 and that separate assessments will be conducted for each subsequent release. This assessment was approved by DHS on July 14, 2006, and our analysis shows that it complied with relevant DHS guidance. This assessment does not cover other recently completed screening releases S1 and S2. According to program officials, S1 and S2 are considered to be part of the Automated Targeting System (ATS), and are therefore covered by the ATS PIA. However, our analysis of the ATS PIA showed that while it addresses screening and targeting functions, it does not specifically identify or address releases S1 and S2. (See the open recommendations section of this briefing for further information on privacy issues); 
Security: CBP has taken a number of actions related to ACE security management. It conducted a security self-assessment for Release 4 in September 2006 using the National Institute of Standards and Technology (NIST) Security Self-Assessment Guide for Information Technology Systems and did not report any major security vulnerabilities. In addition, CBP officials stated that another security self-assessment was conducted in April 2007. However, we have yet to receive this self-assessment. CBP also reports that it has conducted two full contingency plan tests at its disaster recovery facility; 
Further, CBP accredited ACE Release 4 on November 22, 2004; Release S1on July 18, 2006; and Release S2 on October 18, 2006. The program office plans to conduct its next ACE certification and accreditation of ACE in September 2007 for Release 5/Drop A1. However, since November 2004, the deployed ACE system has been subject to considerable change. Specifically, about 7,100 trouble tickets, trouble reports, change requests and install requests were generatedï¿½some of which have resulted in system changes. Federal guidance recommends reaccreditation whenever significant changes to the system or its operational environment are likely to affect a systemï¿½s security posture.[Footnote 22] However, CBP has not established explicit criteria for when to reaccredit/recertify a system in the face of changes, or documented a systematic procedure for determining the security risk from the cumulative impact of changes made to the system. As a result, CBP has not recertified or reaccredited the system; 
In addition, the ACE security plan and security risk assessment have not been updated to reflect key information, such as the results of the April 2007 security self-assessment or a system access control risk identified in May 2006 and included in the programï¿½s risk database. 

Source: OMB and NIST criteria and GAO analysis of DHS documentation. 

[End of table] 

Objective 1: Legislative Conditions: 

Condition 2: Partially Satisfied: 

Condition 2: The plan, including related program documentation and program officialsï¿½ statements, partially satisfies the condition that it comply with the DHS information systems enterprise architecture (EA) as currently defined. 

According to federal guidelines and best practices, investment compliance with an EA is essential for ensuring that an organizationï¿½s investment in new and existing systems is defined, designed, and implemented in a way that promotes integration and interoperability and minimizes overlap and redundancy, thus optimizing enterprisewide efficiency and effectiveness. A compliance determination is not a one-time event that occurs when an investment begins, but is, rather, a series of determinations that occurs throughout an investmentï¿½s life cycle as changes to key aspects of both the EA and the investmentï¿½s architecture are made (e.g., data, business, services, and technology). 

The DHS Enterprise Architecture Board, supported by the Enterprise Architecture Center of Excellence, is responsible for ensuring that projects demonstrate adequate technical and strategic compliance with the departmentï¿½s EA. 

During 2006, the DHS Enterprise Architecture Board conducted three reviews of ACE architectural alignment with the DHS EA Technical Reference Model (TRM)[Footnote 23] as part of key milestone reviews for three release/screening increments. 

* Master Data and Enhanced Accounts (Drop A1), Critical Design Review, June 2006; 

* E-Manifest: Rail and Sea (Drop M1), Critical Design Review, September 2006, and: 

* Targeting Foundation (S2), Production Readiness Review, November 2006. 

On May 1, 2007, DHS EA officials reported that all required products and technologies were aligned to the TRM, and thus that ACE was in alignment with the DHS EA. In addition, the DHS CIO certified as part of the above milestone reviews that each increment was in alignment with the TRM. More specifically, 

* In December 2006, the Center of Excellence determined that ACE was conditionally compliant with the DHS EA, contingent on actions taken in regard to four conditions. Three of the four conditions were resolved by March 2007.

* On March 29, 2007, the EAB recommended approval of ACEï¿½s decision request for program alignment, with one remaining conditionï¿½that by June 30, 2007, the program submit technology insertions packages for the products identified as needing alignment with the DHS TRM. 

Notwithstanding these EA compliance determinations, DHS did not provide us with sufficient documentation of its determinations to allow us to understand the methodology and criteria, or to verify the analyses, that were used to arrive at them. Moreover, documentation that was provided showed that the determinations focused on ACE technical alignment, and did not address, for example, alignment to the DHS EA Data Reference Model. 

Until DHS demonstrates, through verifiable documentation and methodologically- based analysis, that ACE is aligned with all relevant aspects of the DHS EA, including corporate data structures and standards, the program will be at risk of being designed and implemented in a way that does not support optimized departmental operations, performance, and achievement of strategic goals and outcomes, including those related to information sharing. 

Objective 1: Legislative Conditions: 

Condition 3: Satisfied: 

Condition 3: The plan, including related program documentation and program officialsï¿½ statements, satisfies the condition that it comply with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government.[Footnote 24] 

Federal acquisition rules, requirements, guidelines, and management practices provide an acquisition management framework that is based on the use of rigorous and disciplined processes for planning, managing, and controlling the acquisition of IT resources.[Footnote 25] These acquisition management processes are embodied in published best practices models, such as the Capability Maturity Modelsï¿½ developed by Carnegie Mellon Universityï¿½s Software Engineering Institute (SEI). These models explicitly define, among other things, acquisition process management controls that are recognized hallmarks of successful organizations and that, if implemented effectively, can greatly increase the chances of acquiring software-intensive systems that provide promised capabilities on time and within budget. 

In our prior reviews of the ACE program,[Footnote 26] we reported that ACE had satisfied this condition based on SEIï¿½s November 2003 assessment of the program against the Software Acquisition Capability Maturity Model (SA- CMMï¿½).[Footnote 27] That assessment assigned the program an SEI level 2[Footnote 28] rating, indicating that CBP had instituted basic acquisition management processes and practices. Since receiving its 2003 rating, the program office has not conducted another acquisition capability assessment to ensure that it is continuing to employ these basic acquisition controls, and does not plan to do so.

Furthermore, program officials told us that although the prime contractor was contractually required to have an SEI CMMï¿½ Level 3[Footnote 29] capability, they are no longer required to do so because funding contractor program management-related activities is now a lower priority relative to other competing demands for ACE funding. 

According to CBP officials, several steps have been taken to mitigate any impact of no longer focusing on SEI CMMï¿½ compliance, such as: 

* requiring the prime contractor and the program officeï¿½s support contractors to follow plans, processes, and procedures for such key areas as EVM, configuration management, and problem reporting; 

* having the program officeï¿½s quality assurance staff, in collaboration with the prime contractorï¿½s quality assurance function, monitor adherence to key management controls through reviews of ACE processes and products, including 48 such reviews in fiscal year 2006; and: 

* having the program officeï¿½s contracting function continuously monitor adherence to contractual provisions; 

* leveraging OIT assistance, through the Process Asset Group, to conducts reviews of new and updated plans, policies, and procedures;[Footnote 30] and: 

* cataloguing job aids in a Process Asset Group library that specifies the required activities for program management processes and disseminating them to ACE staff. 

Objective 1: Legislative Conditions: 

Condition 4: Satisfied: 

Condition 4: The plan satisfies the condition that it include certification by the DHS CIO that an IV&V agent is currently under contract. 

On October 24, 2006, the DHS Deputy CIO certified in writing that an IV&V agent is under contract for ACE and that the agent met applicable requirements and standards. (See open recommendations section of this briefing for more information on IV&V.)

Objective 1: Legislative Conditions: 

Condition 5: Satisfied: 

Condition 5. The plan, including related program documentation and program officialsï¿½ statements, satisfies the requirement that it be reviewed and approved by the DHS IRB, the Secretary of Homeland Security, and OMB. 

* The DHS IRB reviewed the program and approved the expenditure plan on December 12, 2006.

* The DHS Under Secretary for Management approved the expenditure plan on behalf of the Secretary of Homeland Security on February 6, 2007. 

* OMB approved the expenditure plan on January 22, 2007. 

Objective 1: Legislative Conditions: 

Condition 6: Satisfied: 

Condition 6. The plan satisfies the requirement that it be reviewed by GAO. Our review was completed on July 26, 2007. 

Objective 2: Open Recommendations: 

Accountability Framework: 

Open Recommendation 1: Define and implement an ACE accountability framework that fulfills these conditions: 

a. Covers al program commitment areas, including key expected or estimated system (1) capabilities, use, and quality; (2) benefits and mission value; (3) costs; and (4) milestones and schedules. 

Status: Complete: 

Effective program management includes defining and measuring progress against program commitments and being held accountable for results. Such commitments generally cover expected or estimated (1) capabilities and their associated use and quality, (2) benefits and mission value, (3) costs, and (4) milestones and schedules.

Since 2003, we have reported that such commitments for ACE have not always been defined, although improvements have been made. Most recently, we reported[Footnote 31] that the program office had developed an initial version of an accountability framework for measuring several program commitments, such as capabilities and milestones, but that other commitments, such as benefits, had not been as well defined. 

During the last year, the program office has continued to improve its coverage of program commitments, as evidenced by the content of key program documentsï¿½ ACE accountability framework, ACE Program Plan (August 2006), the CBP/ACE performance reference model, periodic Program Management Review (PMR) reports, and the fiscal year 2007 expenditure plan. For example, 

* The accountability framework now captures and integrates data on ACE functional and performance capabilities, benefits, and mission value, estimated and actual costs, milestones and schedule, accomplishments, and earned value management status. Further, the framework provides visibility into each ACE release at several levels of detail, and it is being used by the ACE Executive Director and others as the means for monitoring program progress, issues, and decisions. 

* The PMR reports have included the accountability framework data. 

* The fiscal year 2007 ACE expenditure plan included an example of the accountability framework commitments from the September 2006 PMR report.

b. Ensures currency, relevance, and completeness of such commitments made to Congress in expenditure plans. 

Status: In progress: 

We have previously reported that commitments made in expenditure plans relative to program capabilities, benefits, costs, and schedules need to be current, relevant, and complete. To the extent that they are not, the currency and relevance of the plan and its utility to Congress as an accountability mechanism are limited.

ACE expenditure plans have not always included such information. To address this limitation, we reported last year[Footnote 32] that CBP was relying on quarterly reports to Congress to provide the appropriations committees with more current, relevant, and complete information about the program than could be provided in the expenditure plan. However, we also noted that the quarterly reports were generally submitted to Congress 3 to 4 months after the end of each quarter, thus limiting their currency and relevance. 

The fiscal year 2007 expenditure plan continues to include information about program commitments that is not current. Specifically, 

* The latest expenditure plan was provided to the appropriations committees on February 6, 2007. However, information from the ACE accountability framework that was included in the plan, such as milestones, EVM values, and risks, were as of October 2006, making it about 4 months old. For example, the expenditure plan listed the milestone for the Production Readiness Review (PRR) for a key release, Release 5/Drop A1, as March 1, 2007. However, this milestone had already slipped by more than 4 months--to July 12, 2007--by the December 2006 PMR (held on January 5, 2007).

According to program officials, they continue to use the quarterly ACE status reports to provide the appropriations committees with more current information.[Footnote 33] However, recent quarterly reports have not been current. For example, 

* The December 2006 quarterly report was not provided until February 26, 2007, and it contained the same outdated PRR milestone as the expenditure plan did for the release previously mentioned. 

* Other quarterly reports have been submitted to the appropriations committees as many as 7 months after the end of the quarter. 

CBP and DHS officials told us that the delays were due to the DHS review and approval process and that incorporating the most current program information would delay the reports even longer. According to the ACE and DHS officials, they are exploring ways to accelerate the review process. 

c. Ensures reliable data relevant to measuring progress against commitments. Status: Complete: 

The quality of the capabilities that a program is to deliver is a relevant program commitment. One measure of the quality of system capabilities is the trend in the number and severity of unresolved system defects or problems. Reliable data about these defects are needed so that system maturity can be understood and informed investment decisions can be made. 

We previously reported [Footnote 34] that ACE defect data were not always consistent because the two tools that the program used to track defects were not integrated and reconciled. As a result, the true status of ACE defects, and thus an important measure of system quality, was not known. 

Since then, the program office has integrated the two tools using a cross- referencing function and has instituted manual processes for reconciling the data in each tool. The December 2006 quarterly congressional report stated that these efforts have improved the ability to record, assess, and report on system quality and performance.

d. Ensures reporting in future expenditure plans progress against commitments contained in prior expenditure plans. 

Status: In progress: 

The value and utility of an expenditure plan for congressional 
oversight and agency accountability depends in large part on whether the plan reports on progress against commitments--system capabilities, benefits, costs, and schedules--made in prior plans. 

Historically, the ACE expenditure plans have not reported adequately on progress against previous plan commitments. Most recently, we reported[Footnote 35] that the fiscal year 2006 expenditure plan contained several such reporting gaps, such as whether funding amounts were actually obligated and expended as planned and whether releases met schedules. In particular, we reported that the plan did not address whether the design and development for Release 5 was actually accomplished as planned. 

The fiscal year 2007 expenditure plan still does not adequately describe the programï¿½s progress against the commitments that were made in the fiscal year 2006 plan. For example, 

* The plan does not report progress against milestones in the fiscal year 2006 plan or explain why these milestones were not achieved. 

* The plan does not report actual obligation or expenditure of funds relative to the planned uses of these funds in prior expenditure plans. 

*  The plan did not address progress against the milestone dates for each stage of a release that was included in the prior yearï¿½s plan. As a case in point, the fiscal year 2006 expenditure plan described specific functionality planned for Release 5/Drop A1 (Master Data and Enhanced Accounts), such as online registration for trade representatives. However, the fiscal year 2007 expenditure plan does not include information on whether these functions were delivered, stating only that Release 5 functionality would be deployed in two phases. 

According to program officials, the quarterly congressional reports provide more current information on the programï¿½s progress against prior commitments. However, we found that these reports have also not fully addressed the commitments made in prior expenditure plans. For example, recent quarterly reports describe progress against developmental milestones for each release, but do not report whether key functionality associated with each release was delivered. In particular, the report for the first quarter of fiscal year 2007 identified planned future capabilities and associated milestones for Release 5/Drop A1 but did not address the status of the key functions associated with this release. 

Program officials stated that they intend to start providing this information in the last quarterly report of each year. 

e. Ensures use of criteria for exiting key readiness milestones that adequately consider indicators of system maturity, such as severity of open defects, and document the milestone decisions in a way that reflects the associated risks and plans for mitigating them.[Footnote 36] 

Status: Complete: 

As noted earlier, one measure of the quality of system capabilities being delivered is the number and severity of unresolved system defects or problems. As such, information on such defects is an important consideration when program decisions, such as key milestone decisions, are made. 

We previously reported[Footnote 37] that several key milestones were passed that had severe open defects and that program officials were unable to provide documentation regarding how the risks associated with these defects were assessed. We also reported that these risks were not being tracked in the programï¿½s risk database. 

Since then, the system life cycle gate review process for Office of Information and Technology projects, including ACE, has been amended to include milestone readiness decisions based on an assessment and acceptance of risks (including the risks related to unresolved defects). Further, the ACE Risk and Issue Management Process guidance provides for the systematic identification, analysis, prioritization, planning, execution, evaluation, and documentation of program risks and issues and requires that any risks associated with going forward should be identified as part of each life cycle gate review. 

This process has been applied in the following two major gate reviews: 

* Release A1 (ESAR) Critical Design Review (May 2006) and: 

* Release M1 (E-Manifest: Rail and Sea Manifest) Critical Design Review (August 2006).

In the case of Release 5/Drop A1, program officials reported that no severe defects existed, but that all known risks associated with proceeding past the milestone had been entered into the programï¿½s risk database and documented in the certification package submitted to the DHS CIO. 

The quarterly reports to Congress similarly state that risks related to gate review decisions and the associated impacts are entered into a database to ensure visibility and mitigation and are included in the package submitted to the DHS CIO for review and certification to pass a given milestone. Going forward, officials stated that similar assessments will be part of the following planned gate reviews for Release 5/Drop A1: 

* Operational Readiness Review (planned for August 23, 2007) and: 

* Live operations (planned for August 25, 2007). 

Open recommendation 2: Develop the range of realistic ACE performance measures and targets needed to support an outcome-based, results-oriented accountability framework, including user satisfaction with ACE. 

Status: Complete: 

We have previously reported on both the absence of meaningful performance measures to understand ACE progress, quality, and results, and have raised concerns about the practicality and applicability of some measures that have been defined. Most recently, we reported[Footnote 38] that defined ACE performance targets are not always realistic and that goals, expected mission benefits, and performance measures are not fully defined and adequately aligned. 

In response, the program office has established an initial set of performance measures that are tied to program objectives and related performance goals, and it has established processes for the collecting, analyzing, and integrating performance data for each measure. Among other things, these performance measures address user satisfaction, efficiency, and productivity. Moreover, these measures have been made an integral part of the ACE accountability framework and they were approved by the CBP Commissioner and the DHS CIO on June 30 and July 6, 2006, respectively. 

Objective 2: Open Recommendations: 

Performance Measures: 

According to program officials, these initial measures will continue to be revised and augmented, and thus will evolve over time based on lessons learned, changes to existing release capabilities, and refinements of requirements for future releases. For example, the program office reported in March 2007 that new measures are still being developed, reviewed, and evaluated for Releases 2, 3, and 4 at the same time that performance data is being collected and reported for existing measures for these releases. According to program officials, the new measures will be combined with existing performance measures. 

In addition, CBPï¿½s December 2006 quarterly congressional report stated that performance measures for future releases, such as Release 5, are being scheduled for identification and development and that changes are being made to existing measures when they are determined to be inappropriate. For example, the program office learned that the performance measure "percentage of truck manifests being filed electronically" did not recognize that empty trucks are not required to file manifests, and thus it revised the metric used to divide the number of electronically filed truck manifests by the total number of required truck manifests instead of the total number of trucks in order to get a more meaningful reflection of ACE performance. 

To assist in managing these performance measures, the program has established a life cycle process for performance measures as well as a database tool to record and manage modifications.

Objective 2: Open Recommendations: 

Performance Measures Alignment: 

Open recommendation 3: Explicitly align ACE program goals, benefits, desired business outcomes, and performance measures.

Status: Complete: 

As just discussed, the program office has developed an initial set of ACE program measures that are tied to program goals, among other things, and they intend to continue to update and evolve these as the program moves forward. In addition, the program office has developed an ACE Performance Reference Model that explicitly aligns the CBP strategic goals, objectives, strategies, and desired results applicable to the ACE program with specific performance measures. For example,

Table: 

CBP Strategic Goal; 
Preventing terrorism at ports of entry: Prevent terrorists and terrorist weapons, including weapons of mass destruction and weapons of mass effect, from entering the U. S.. 

CBP Objective; 
Improve information and targeting. 

CBP Strategy
Use advanced passenger and cargo information (NTC, ATS-Air, ATS, Screening and Targeting-ACE) to pre-screen, target, and identify potential terrorists and terrorist shipments and any related activity.

Desired Result; 
Increased use of targeting. 

Performance Measure; 
Number of security-focused selections generated by system (= or above intensive threshold) by type. 

Further, this model links the measures to specific ACE releases. For example, the above performance measure applies to Screening Foundation (S1) and Targeting Foundation (S2). 

Objective 2: Open Recommendations: 

Management Improvement Measures: 

Open recommendation 4: Define measures and collect and use associated metrics for determining whether prior and future program management improvements are successful. 

Status: Planned: 

As we have previously reported,[Footnote 39] investments in program management improvements should include defined measures of progress and results. To date, the program office has implemented a number of such improvements; however, it has not had measures or metrics to determine the success of the improvements. Moreover, the December 2006 quarterly congressional report stated that the program anticipates more changes, including: 

* creation of a cargo requirement management board to decide the disposition of all change requests to production systems; 

* establishment of a new invoice review policy; and: 

* co-location of personnel within a given business area. 

According to its December 2006 quarterly reports to Congress, the program office plans to measure the impact of future management improvements. However, program officials told us that they have yet to define them and thus are not yet using such measures.

Objective 2: Open Recommendations: 

Legislative Conditions: 

Open recommendation 5: Fully address those legislative conditions associated with measuring ACE performance and results and employing effective IV&V practices.

Status: Complete: 

Among the legislative conditions that ACE expenditure plans have been required to meet are satisfaction of OMB guidance, including that associated with measuring program performance and results and use of effective IV&V practices. These conditions reflect good program management practices and, if implemented properly, can reduce program risks. 

Performance and Results: 

As previously discussed in this briefing, the program office has developed a range of ACE performance measures and taken steps to align them with program, CBP, and DHS strategic goals and outcomes. According to program officials, they plan to continue to evolve and refine the performance measures and include the measures in the ACE accountability framework. 

IV&V Practices: 

In January 2006, the DHS CIO certified that an IV&V agent was under contract for the ACE program, but noted several issues, including these: 

* mechanisms were needed to ensure that products were complete, of sufficient quality, and met the needs of the user and: 

* a more explicit technical approach, describing when and how certain activities should (or should not be) performed, was needed. 

Moreover, we subsequently reported[Footnote 40] that the scope of the contractorï¿½s activities did not extend to both IV&V of key system products and development processes. 

On October 24, 2006, the Deputy CIO again certified that an IV&V agent was under contract and stated that the previous issues had been addressed. Since then, the program office has also developed an IV&V Implementation Management Plan that addresses the concerns raised by the DHS CIO and us. In particular, the plan: 

* requires an IV&V program consistent with the industry standard; 

* provides a set of objectives, guidelines, and expectations for IV&V activities, including periodic independent reports on status, observations, recommendations, and activities; and: 

* addresses satisfaction of quality standards for ACE products and user needs. 

Program officials told us that IV&V has allowed early identification and correction of program process and product weakness. 

Objective 2: Open Recommendations: 

Reconciliation of Cost Estimates: 

Open recommendation 6: Ensure that future expenditure plans are based on cost estimates that are reconciled with independent cost estimates. 

Status: Complete: 

It is important that expenditure plans be based on reliable estimates of costs, to include reconciling differences between government and independent cost estimates. We recently reported[Footnote 41] that the cost estimate in the fiscal year 2006 expenditure plan was based on government and independent cost estimates that had been compared and found to be consistent. 

For the fiscal year 2007 expenditure plan, our analysis showed that the government and independent cost estimates differed by about 15 percent. According to program officials, they reconciled these differences in January 2007, and concluded that the results did not warrant changes to the expenditure plan because the government estimates used in the expenditure plan were more accurate. 

According to program officials, two primary factors account for the difference in estimates. Specifically,

* The estimates assumed different timelines for completing development of all releases. The independent estimate assumed development completion by fiscal year 2010, while the government estimate assumed fiscal year 2011. According to program office officials, this accounts for about five percent of the difference. 

* The government estimate included a number of items that the independent estimator did not. For example, the independent estimator did not include training and outreach items, such as the cost of conferences with trade associations, training materials, and associated travel. As a result, the government estimated training and outreach costs over the life of the program to be about $90 million, while the independent estimate put these costs at about $39 million. The independent estimate has since been amended to include the missing items, and the reconciliation process is continuing. 

Objective 2: Open Recommendations: 

Rigorous Cost Estimation: 

Open recommendation 7: Develop and implement a rigorous and analytically verifiable cost estimating program that embodies the tenets of effective estimating.[Footnote 42] 

Status: Complete: 

The reliability of cost estimates is largely a function of the quality of the estimating process used to derive them. We previously reported[Footnote 43] that the program did not have a well-defined cost estimating process, but that it has since made progress in strengthening its cost estimating program. Specifically, the program office has: 

* defined and documented processes for estimating program costs (including management reserve costs) and: 

* hired a contractor to develop costs estimates that were independent of the government estimates.

In September 2006, an ACE support contractor reported that both the government and independent cost estimation processes demonstrated significant conformance to effective estimating practices and concluded that the program is using a rigorous and verifiable cost estimating approach. 

Objective 2: Open Recommendations: 

Earned Value Management: 

Open recommendation 8: Use EVM in developing all existing and future releases. 

Status: Complete: 

EVM is a program management tool to measure progress by comparing, during a given period of time, the value of work accomplished with the amount of work expected to be accomplished. This comparison permits performance to be evaluated based on calculated variances from the planned (baselined) cost and schedule. EVM is both an industry accepted practice and an OMB requirement. 

We recently reported[Footnote 44] that the program office was not using EVM for all releases (e.g., Release 5) due to changes to release baselines and the lack of familiarity on the part of program staff with EVM practices. 

Since then, the program office has established the performance baselines needed to implement EVM for Release 5/Drops A1 and A2, Screening 3, and Release 6/Drop M1. 

For these releases, the program office reports that it has also applied EVM standards in establishing baselines and included EVM data in the accountability framework for management and decision-making purposes. Program officials also reported that they plan to implement EVM on future releases and task orders. 

Objective 2: Open Recommendations: 

ACE Infrastructure: 

Open recommendation 9: Have future ACE expenditure plans specifically address any proposals or plans, whether tentative or approved, for extending and using ACE infrastructure to support other homeland security applications, including any impact on ACE of such proposals and plans. 

Status: Complete: 

Together, the fiscal year 2007 expenditure plan and the quarterly reports to Congress address ACEï¿½s relationships with other trade processing and DHS applications. For example, the plan discusses efforts to work with participating government agencies in defining and deploying the International Trade Data System (ITDS). The stated goal is to deliver ACE/ITDS in an integrated and coordinated manner. In this regard, the plan discusses workshops to gather requirements from participating agencies for Release 6/Drop M2 (first held on July 19, 2006), a working group to address these agenciesï¿½ HAZMAT issues (established on July 17, 2006), and efforts to develop data element inputs for ACE from other government agencies. 

In addition, the quarterly reports to Congress cite coordination efforts with related homeland security programs. For example, the report for the first quarter of 2007 states that: 

* Container Security Initiative will be supported by ACE Release 6 and Screening 1-3 capabilities;

* Customs-Trade Partnership Against Terrorism is coordinating with ACE Release 5/Drop A1 capabilities to provide both CBP and trade representatives with the ability to view the status of CBP programs; and: 

* U.S.-Mexico Border Partnership Plan will coordinate with ACE to implement any cargo screening standards derived from partnership plan agreements. 

In addition, CBP reports that ACE and other CBP applications[Footnote 45] share infrastructure (desktops, clients, and local area networks) at the ports of entry. This infrastructure is purchased and maintained by CBPï¿½s OIT Program Integration Division. Each application, including ACE, is responsible for complying with OITï¿½s infrastructure standards. 

Objective 2: Open Recommendations: 

Overlap and Concurrence: 

Open recommendation 10: Minimize the degree of overlap and concurrency across ongoing and future ACE releases and capture and mitigate the associated risks of any residual concurrence. 

Status: In progress: 

Significant overlap and concurrency among major program activities, such as releases, introduce considerable cost and schedule risks as they can create contention for limited resources among the releases. We have continued to report on extensive overlap and concurrence in ACE releases and the cost overruns and schedule delays that have resulted. Most recently, we reported[Footnote 46] that the ACE schedule continued to provide for such overlap and concurrency and that the risks associated with doing so were not being effectively addressed. Since then, the program office has reduced this overlap and concurrence. For example, it has: 

* decoupled (i.e., reduced dependencies among) certain ACE program components by separating Screening 1-3 from Releases 4-6 and: 

* aligned the development and delivery of functionality for different releases with the availability of required hardware environments. 

Recent quarterly congressional reports stated that CBP has taken actions to reduce potential contention for limited resources. Examples include: 

* dividing releases into smaller subreleases, called drops, to provide more flexibility in scheduling;

* improving planning for development, integration, testing, and training activities and milestones to better schedule use of development and test environments; and: 

* centralizing management of shared software services to address, among other things, allocation of resources and responsiveness to workload peaks and use of consistent technical management approaches across releases. Moreover, the programï¿½s risk database identifies overlap and concurrency-related risks. However, the mitigation strategies for these risks contain vague or incomplete data. These data limitations make it difficult to determine the status or the effectiveness of the efforts to reduce the risks associated with overlap and concurrence among releases. (See the observations section of this briefing for additional information on program risk management.) 

The program office is also conducting regular integration meetings with the teams supporting each release to discuss concerns, decisions, and schedules associated with resource availability and is using a software tool to track and mitigate release- specific concurrency risks.
Notwithstanding these steps, the program continues to face challenges in managing dependencies among releases, which any associated concurrency in the programï¿½s development exacerbates. For example, the schedule slips expected with Release 5/Drop A2 will affect Release 6/Drop M1, as resources have been shifted from M1 to address the A2 delay. In addition, further delays in A1 and A2 will impact the implementation of M1. 

Objective 2: Open Recommendations: 

Legislative Conditions: 

Open recommendation 11: Direct the appropriate departmental officials to fully address those legislative conditions associated with having an approved privacy impact assessment (PIA)[Footnote 47] and ensuring architectural alignment. 

Status: In progress: 

The department approved the ACE PIA for Release 4 in July 2006, however, this assessment does not cover other completed releases. Further, DHS has determined that ACE is aligned with DHS architecture, but we have yet to receive documentation to adequately understand and verify the determination. 

PIA (In progress): 

In March 2006, DHS developed guidance on the development and content of 
a PIA. As we have previously reported,[Footnote 48] the purpose of a PIA is to ensure that there is no collection, storage, access, use, or dissemination of identifiable personal or business information that is not both needed and permitted. Development and use of a PIA are both a requirement of OMB and the E-Government Act of 2002.[Footnote 49] 

The program office has developed a PIA for Release 4 of the ACE e-Manifest: Trucks and the International Trade Data System, and DHS approved it on July 14, 2006. As noted earlier, our analysis shows that this PIA addresses the major elements of DHSï¿½s guidance. Program officials stated that they would update the assessment for each ACE release. This assessment does not cover other recently completed screening releases (i.e., S1 and S2). However, program officials told us that S1 and S2 are considered to be part of the Automated Targeting System (ATS), and are therefore covered by the ATS PIA. However, our analysis of the ATS PIA showed that while it addresses screening and targeting functions, it does not specifically identify or address releases S1 and S2. 

Architectural Alignment (in progress): 

As discussed in the legislative conditions section of this briefing, DHS has determined that ACE is aligned with the DHS EA. 

For example, 

* In December 2006, DHSï¿½s EA Center of Excellence determined that ACE was conditionally compliant with the DHS architecture, but that the program needed to take actions to address four conditions. 

* On March 2007, the DHS EA Board reported that ACE had satisfied three of the conditions and recommended approval of ACEï¿½s request for program alignment with one remaining conditionï¿½alignment with the DHS Technical Reference Model.

* In May 2007, DHS EA officials reported that all required products and technologies were aligned with the Technical Reference Model and that ACE was thus aligned with the DHS EA. 

However, as was also discussed earlier in this briefing, we have yet to receive sufficient documentation describing the criteria and methodology used to make these determinations or verifiable analysis supporting the determinations. Moreover, the determinations were based on technical alignment and did not address other relevant aspects of program alignment to an EA, such as data alignment. Thus, the program is at risk of being designed and implemented in a way that is not consistent with all relevant aspects of the DHS EA, such as data structures and standards, and thus does not support optimized DHS-wide operations, performance, and results. 

Objective 2: Open Recommendations: 

Human Capital Practices: 

Open recommendation 12: Develop and implement key human capital management practices. 

Status: In progress: 

As we have previously reported,[Footnote 50] effective strategic management of program human capital includes, among other things, defining the skill sets needed to perform program functions, assessing and inventorying the skill sets of the programï¿½s current workforce and assessing any associated skill gaps in meeting future needs, and developing strategies to fill any identified gaps. Moreover, it includes having a well-defined plan that provides for effective implementation of these processes. 

In June 2006, CBP executives approved the OIT Strategic Human Capital Management Plan. The plan is intended to be a road map for supporting CBPï¿½s workforce vision and ensuring effective management of human capital resources across OIT to address enterprise-wide priorities. An ACE-specific plan was included as an appendix to the CBP plan. This appendix was intended to provide better near- and long-term human capital management practices for the OIT offices involved in ACE development.

Neither the OIT nor the ACE-specific plan addresses the basic tenets of effective human capital management. For example, neither provides for: 

* defining the positions needed (including core competencies) to perform core program functions;

* assessing and inventorying current workforce skills and abilities; 

* assessing any gaps between needed and existing workforce levels and capabilities; and: 

* filling identified gaps via such means as hiring new staff, training existing staff, and augmenting staff with contractor support. 

OIT officials acknowledged these limitations in the plans and stated that they are developing an implementation plan to address these shortfalls. The officials stated that the implementation plan will include accountability, timeframes, and metrics to carry out the larger OIT Strategic Human Capital Management Plan. 

According to these officials, this implementation plan was to be completed in January 2007, following presentation to the OIT Deputy Director's Council and approval by the OIT Office Directors and Assistant Commissioner. However, as of July 2007, it had not yet been approved and thus was not available for our review. 

In the interim, OIT and program officials reported taking various steps to address ACE human capital needs. These include, for example, 

* Using various methods to fill staff vacancies. As of May 2007, the program reports that it has 62 full-time employees and one vacancy and that additional staff are working on ACE either full- or part-time. 

* Lobbying OPM for more flexibility in recruiting, to include higher salaries and direct hiring authority. 

ï¿½ Reorganizing OIT into six program offices aligned to major mission areas so that the number of government personnel responsible for ACE development activities can be augmented by IT functional program management expertise.

* Offering training to improve the skills of staff. 

Nevertheless, these steps have not been guided by a well-defined plan and thus represent activities that are not tied to strategic goals and outcomes and thus cannot be measured against them. Without a plan, it is unlikely that the program will be able to adequately ensure that it has the right people at the right time to deliver ACE successfully. 

Objective 2: Open Recommendations: 

Quarterly Reports: 

Open recommendation 13: Include in the June 30, 2006, quarterly update report to the appropriations committees a strategy for managing ACE human capital needs and the ACE framework for managing performance and ensuring accountability. 

Status: In progress: 

The June 30, 2006 quarterly report to the House and Senate Appropriations Committees included the programï¿½s strategy for meeting its human capital needs and its accountability framework. 

Human Capital Strategy (In Progress): 

The June 30, 2006, quarterly report to Congress included a description of the previously described ACE-specific appendix to the OIT Strategic Human Capital Management Plan, including the planï¿½s five goals and strategies. Moreover, the report states that this ACE-specific plan is aligned with OPMï¿½s Human Capital Assessment and Accountability Framework.[Footnote 51] As previously stated, however, this ACE-specific plan does not meet the asic tenets of strategic human capital management, as defined in this framework and other relevant guidance. 

Objective 2: Open Recommendations: 

Quarterly Reports: 

According to program officials, they are developing a new OIT human capital strategy and implementation plan, but they have yet to provide us with a date when it will be completed.

Accountability Framework (Complete): 

The June 30, 2006, quarterly report included a description of the ACE accountability framework. As previously discussed, this framework is the means by which the program measures performance relative to promised ACE capabilities, costs, schedules, earned value, risks, and mission values and benefits. The report also included an appendix illustrating the format and content of the accountability framework tool. At that time, CBP reported that it would continue to work with stakeholders to further enhance the format, readability, and utility of the framework as a program management and reporting tool. 

Since June 2006, the program office has refined and implemented the accountability framework. The framework covers all program commitment areas and key system aspects to support executive decision making and provides external program stakeholders with information on the program. 

Objective 2: Open Recommendations: 

Quarterly Report on Open Recommendations: 

Open recommendation 14: Report quarterly to the House and Senate Appropriations Committees on efforts to address open GAO recommendations. 

Status: Complete: 

CBP has submitted quarterly reports on ACE to both the House and the Senate Appropriations Subcommittees since November 2002, including reports for each quarter of fiscal year 2006 and the first and second quarters of fiscal year 2007. These reports have addressed CBPï¿½s efforts to address open GAO recommendations. 

Objective 2: Open Recommendations: 

Accurately Report on Open Recommendations: 

Open recommendation 15: Accurately report to the appropriations committees on CBP's progress in implementing our prior recommendations. 

Status: In progress: 

As previously stated, CBP has included information on progress in meeting GAOï¿½s recommendations in its quarterly reports[Footnote 52] to the appropriations committees since November 2002. However, some of this information is dated and thus inaccurate due to the time lapse between when the reports are produced and when they are provided to the appropriations committees. Recently, this time lapse has been between about two and seven months. According to program officials, they are exploring ways to accelerate the review process and thereby improve the timeliness and accuracy of their reports to Congress. DHS officials also told us that they have ongoing efforts to improve the review process; however, the process has not improved consistently or significantly. 

Objective 3: Observations: 

Requirements Limitations Likely to Cause Major Delays: 

Observation 1: Redefinition of requirements for several ACE releases is now under way to address limitations in completeness of originally defined requirements, and this redefinition is likely to introduce program schedule delays and cost increases.

A key aspect of successful system acquisition programs is having well-defined requirements. Among other things, requirements should be complete, and to accomplish this, best practices advocate engaging all key stakeholders in the requirements definition and management process. 

In defining ACE requirements, the program office discovered that its 
original requirements definition approach did not adequately engage all key stakeholders, and to its credit, has since taken steps to address this. Specifically, 

* In the spring of 2004, the program office and the ACE support team conducted more than 300 business process workshops with the ACE user community to help define ACE requirements for the future releases. This requirements definition approach was referred to as the Global Business Blueprint (GBB) effort.

Objective 3: Observations: 

Requirements Limitations Likely to Cause Major Delays: 

* In June and July of 2005, a key group of ACE stakeholders that were not originally involved in the GBB (application programmers familiar with the legacy system that ACE is to replace) raised questions about the completeness of the requirements. To address these questions, the program office examined the GBBï¿½s completeness by first reverse-engineering the legacy software for a small number of legacy system programs and then comparing the reverse-engineered requirements to the GBB-derived requirements. Based on this, which is referred to as legacy code decomposition, the program office found that the GBB-derived requirements were missing about 20 percent of needed ACE functionality. 

* In August 2005, the program office decided that it needed to decompose all of the legacy system code in order to completely capture the requirements for all ACE releases. Work began with decomposition of the legacy code related to ACE Release 5/Drop A2 (Entry Summary Accounts and Revenue). 

* In September 2006, the program office determined that the legacy code decomposition approach alone was not sufficient to gain a full understanding of the requirements given the size and complexity of A2. As a result, the requirements definition process was expanded to engage another key stakeholder group (business process subject matter experts). Under the expanded approach, referred to as legacy code decomposition and collaboration, legacy system software programmers and subject matter experts were to examine the code line by line in defining ACE requirements. 

* In November 2006, the legacy system code decomposition and collaboration effort for Release 5/Drop A2 fell behind schedule because of lack of personnel with legacy system expertise and experience. The schedule for A2 was tentatively revised, but at the time of our review, the schedule was still under review, had not yet been approved, and thus was not yet available for our review. 

The program office has identified the A2 legacy code decomposition and collaboration process as a high risk item that will significantly impact the A2 schedule. For example, the decomposition and collaboration process for part of the functionality on A2ï¿½s critical pathï¿½the Authorized Broker Interface Entry Summaryï¿½is not expected to be completed before early 2008, at the earliest. 

According to program officials, delays for other releases/drops, such as Release 5/Drop A1 and Release 6/Drop M1, will not be as significant. They also said that, while they have not yet estimated how long A2 will be delayed and what the associated cost implications are, they do not expect the cost increases to breach the current acquisition program baseline of $3.3 billion, which translates into a cost increase of less than $200 million. Moreover, they said that several actions have been taken to minimize the impact of the delays. For example, 

* A2 functionality necessary for M1 has been given priority in order to support M1 deployment as originally planned and: 

* A2 scope is being divided into increments to allow some functionality to be delivered sooner and to minimize the impact on other drops. 

However, these actions carry consequences, such as missed opportunities to combine field training and an increase in the number of legacy interfaces, thus increasing the potential for introducing software problems. 

In addition, program officials told us that they are considering changes to the A2 and M1 deployment strategies to address stakeholder concerns, and they said that these changes could also minimize the magnitude of the A2 and M1 delays. Specifically, they said that they had planned to deploy on a national basis, meaning that A2 and M1 functionality would be deployed to all ports at the same time and concurrently adopted by all users nationwide. However, deployment may change to a filer-by-filer basis, meaning that A2 and M1 functionality would be deployed to all ports at the same time, but not all filers would begin using it at the same time. 

Objective 3: Observations
Requirements Limitations Likely to Cause Delays According to program officials, they believe that the change in the deployment strategy would both address stakeholder concerns and minimize any A2 and M1 schedule delays caused by the redefinition of requirements. However, they added that these changes have yet to be approved, and the full extent of the cost and schedule implications are not yet known. Moreover, neither the fiscal year 2007 expenditure plan nor the ACE quarterly reports have disclosed the A2 and M1 requirements redefinition and its impact, and neither has addressed any changes to their deployment strategies. 89
Objective 3: Observations
Key COTS Product Being Replaced Observation 2: Significant changes to ACE requirements have, in turn, led to reevaluation and replacement of a key commercial off-the-shelf product (COTS) previously selected and being prepared for use. When acquiring commercial component-based systems, like ACE, best practices advocate basing decisions on whether to employ a given COTS product on thorough, rigorous, and continuous analysis of a number of factors, including how well competing products satisfy defined system requirements. To the program officeï¿½s credit, it reports having followed the CBP system life cycle methodology to determine which COTS product would best meet the requirements for Release 5/Drop A2. These analyses include: ï¿½ In 2002, the program office reviewed, in general terms, various COTS packages and determined that a solution using SAP (formerly Systems Application and Products), a COTS provider, combined with other commercial solutions and customized development, provided the best combination of capability, performance, and cost for ACE. 90
Objective 3: Observations
Key COTS Product Being Replaced ï¿½ In 2004, a more detailed analysis was conducted as part of the previously mentioned GBB process, which was intended to define and allocate ACE requirements to future ACE releases and provide the basis for, among other things, selecting a specific SAP product. At that time, the SAP Enterprise Portal product was selected for Release 5/Drop A2. ï¿½ In December 2006, the ACE Chief System Architect recommended that the Internet Transaction Server (ITS) technology already used by CBP should be adopted instead of the SAP Enterprise Portal, based on the determination that all currently planned SAP functionality could be presented using the ITS technology. This decision was based on improved understanding of the requirements and previous analyses of ITS. 91
Objective 3: Observations
Key COTS Product Being Replaced On the basis of these analyses and the Chief System Architectï¿½s recommendation, the program office subsequently stopped work on SAP Enterprise Portal design and configuration efforts and, in March 2007, the program reported that the SAP Internet Transaction Server would be used for Release 5/Drop A2 instead of the SAP Enterprise Portal. While this decision was expected to have some near-term schedule impact because much of the completed work for A2 had been based on the planned use of SAP Enterprise Portal, the program office reports that these impacts are offset by the cost advantages of other ACE releases already using the Internet Transaction Server technology. However, in March 2007, program officials identified a risk of inadequate response time for the Internet Transaction Serverï¿½thus negatively impacting user productivityï¿½and that there was high probability of significant cost and schedule impacts. Actions are underway to mitigate the risk through performance modeling and test planning.
Neither the fiscal year 2007 expenditure plan nor the quarterly reports to Congress disclose this COTS product change, its impact on release schedules and cost estimates, or risk to future system performance. 92
Objective 3: Observations
Risks Not Being Effectively Managed Observation 3: All program risks are not being effectively managed. Risk management is a continuous, forward-looking process that is intended to either prevent program cost, schedule, and performance problems from occurring or to minimize the impact if they occur. According to relevant guidance and best practices, effective risk management involves proactively identifying, assessing, and disclosing risks; defining and implementing cost-effective strategies for mitigating these risks; and measuring and disclosing progress in doing so. To its credit, the program office has developed a process guide and implemented an automated tool (database) for managing ACE risks in accordance with relevant guidance and best practices. Among other things, the database contains the description, level (high, medium, or low), and mitigation strategy (including start and end dates, exit criteria, and implementation status) for each risk. 93
Objective 3: Observations
Risks Not Being Effectively Managed 1 As of May 23, 2007 the risk database contained 46 risks. However, the completeness and quality of the information on each of the risks in the risk database1 vary. For example, many risks were missing information on the status of efforts to implement the mitigation strategy and 
ï¿½ many risks (18) were missing criteria for completing mitigation steps, clear descriptions of what the risk entailed, and start and end dates for planned mitigation activities.
Because of such database limitations, we could not determine the status of and mitigation progress on 17 risks. Moreover, these database limitations were reflected in the documentation used at key program events, such as PMRs. This means that the program does not have the risk-related information that it needs to inform its program decisions and to reduce the chances of potential problems becoming actual problems.
94
Objective 3: Observations
Risks Not Being Effectively Managed Program officials, including the official responsible for risk management, stated that risk management is immature and needs to be strengthened. The reasons they gave for these risk management weaknesses are due to ï¿½ all staff not being trained on how to use the tool (last training was provided in 2003, and the since then a number of people have joined the program); ï¿½ the tool is unique to the ACE program and thus no CBP guidance exists on its use); and
ï¿½ each ACE group addresses risk differently in its weekly meetings. To improve ACE risk management, program officials told us that they are: ï¿½ establishing a group to ensure the quality and completeness of the database; ï¿½ holding regular group meetings with contract staff and team leads to discuss risks and their impacts; and ï¿½ conducting risk management training. 95
Objective 3: Observations
Risks Not Being Effectively Managed If implemented effectively, such steps should result in more meaningful information about program risks that can be useful to DHS in managing the program and to Congress in overseeing it. To date, however, ACE program risks have not been communicated to oversight organizations through the 2007 expenditure plan or recent quarterly reports to the House and Senate Appropriations Committees. 96
Conclusions
Over the past 7 years, CBP and DHS have worked to fulfill legislatively mandated annual expenditure plan requirements and to implement dozens of our recommendations related to these plans and management of the program. Among other things, these requirements and recommendations have promoted effective program management and accountability for performance and results. As a result of these years of efforts, the ACE program is better positioned today for delivering promised capabilities and benefits than it has been in the past. Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management continue to remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, avoiding major program schedule delays and cost overruns remains a challenge as more of each appear to be on the horizon. 97
Conclusions
To further improve ACE management and minimize its exposure to risk, it is important for CBP and DHS to remain vigilant in their efforts to satisfy ACE legislative requirements and to fully implement our prior recommendations. Moreover, it is important that they keep the Congress fully informed on where the program stands and what changes are planned to address emerging cost overruns and schedule delays.
98
Recommendations for Executive Action To further strengthen ACE management and promote accountability for ACE performance and results, we are making the following recommendation to the Secretary of Homeland Security to direct the CBP Commissioner to ensure that future quarterly reports to the House and Senate Appropriations Committees disclose
(1)the risks and associated mitigation strategies of not having fully satisfied the expenditure plan legislative conditions and not having completed implementation of all our prior recommendations; (2)the status and impacts on the programï¿½s estimated cost and schedule and lessons learned from ongoing efforts to redefine requirements and to implement a different COTS product than originally selected; and (3)the programï¿½s plans and actions for improving ACE risk management and its current inventory of program risks, including their associated mitigation strategies and the status of the strategiesï¿½ implementation. 99
Agency Comments
In oral comments on a draft of this briefing, DHS and CBP officials agreed with our conclusions and recommendations, and provided clarifying information and technical comments that we incorporated in the briefing, as appropriate. 100
Attachment 1
Scope and Methodology
1 SEIï¿½s institutional and project-specific estimating guidelines are defined in Robert E. Park, Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, Pa.: Carnegie Mellon University Software Engineering Institute, 1995) and A Managerï¿½s Checklist for Validating Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburg, Pa: 1995), respectively. Scope and Methodology
To accomplish our objectives, we analyzed the ACE fiscal year 2007 expenditure plan and supporting documentation, and compared them to relevant federal requirements and guidance, applicable best practices, and our prior recommendations. We also interviewed DHS and CBP officials and ACE program contractors. In particular, we reviewed ï¿½ DHS and CBP investment management practices, using OMB A-11, part 7; ï¿½ DHS and CBP certification activities for ensuring ACE compliance with the DHS enterprise architecture; ï¿½ DHS and CBP acquisition management efforts, using SEIï¿½s SA-CMM; ï¿½ CBP cost estimating program and cost estimates, using SEIï¿½s institutional and project-specific estimating guidelines;1 101
Attachment 1
Scope and Methodology
ï¿½ independent verification and validation (IV&V) activities using the Institute of Electrical and Electronics Engineers standard for software verification and validation;2
ï¿½ CBP actions to coordinate ACE with related programs; ï¿½ CBPï¿½s reorganization documentation, including the organizational charts and roles and responsibilities matrix; ï¿½ ACEï¿½s accountability framework; and ï¿½ cost and schedule data and program commitments from program management documentation.
2Institute of Electrical and Electronics Engineers (IEEE) Computer Society, Standard for Software Verification and Validation 1012-1998 (June 8, 2005). 102
Attachment 1
Scope and Methodology
For DHS-, CBP-, and contractor-provided data that we did not substantiate, we have made appropriate attribution indicating the data's source. We conducted our work at CBP headquarters and contractor facilities in the Washington, D.C., metropolitan area from December 2006 through July 2007 in accordance with generally accepted government auditing standards. 103
Attachment 2
Related GAO Products
104
Attachment 2
Related GAO Products
105




Appendix II Comments from the Department of Homeland Security: 

[See PDF for image] - graphic text: 

[End of figure] - graphic text: 

[See PDF for image] - graphic text: 

[End of figure] - graphic text: 

[See PDF for image] - graphic text: 

[End of figure] - graphic text: 

[End of section] 

Appendix III Contact and Staff Acknowledgments: 

GAO Contact: 

Randolph C. Hite, (202) 512-3459: 

Staff Acknowledgments: 

In addition to the person named above, Daniel Castro, Dawn Day, Neil Doherty, Nancy Glover, Paula Moore, Jamelyn Payan, Nik Rapelje, and Karen Talley made key contributions to this report. 

[End of section] 

Footnotes: 

[1] Pub. L. No. 109-295 (Oct. 4, 2006). 

[2] Two related open recommendations have been combined. 

[3] We did not evaluate the compliance of the ACE program with respect to the Federal Acquisition Regulation or OMB directives, but we did evaluate its compliance with established best practices models that incorporate IT investment requirements for federal agencies. 

[4] This recommendation has multiple conditions, of which three are completed and two are in progress. A sixth condition of this recommendation--that the ACE accountability framework clearly and unambiguously delineate roles and responsibilities of the government and the prime contractor--was previously completed. See Information Technology: Customs Has Made Progress on Automated Commercial Environment System, but It Faces Long-Standing Management Challenges and New Risks, GAO-06-580 (Washington, D.C.: May 31, 2006). 

[5] 44 U.S.C. ï¿½ 3501 note. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: [email protected] 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations

Gloria Jarmon, Managing Director:
[email protected]:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, DC 20548:

Public Affairs: 

Chuck Young, Managing Director:
[email protected]:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:

*** End of document. ***