    DOD Systems Modernization: Planned Investment in the Naval
    Tactical Command Support System Needs to be Reassessed
    (05-DEC-05, GAO-06-215).

    Because it is important that the Department of Defense (DOD)
    adheres to disciplined information technology (IT) acquisition
    processes to successfully modernize its business systems, GAO was
    asked to determine whether the Naval Tactical Command Support
    System (NTCSS) is being managed according to important aspects of
    DOD's acquisition policies and guidance, as well as other
    relevant acquisition management best practices. NTCSS was started
    in 1995 to help Navy personnel effectively manage ship,
    submarine, and aircraft support activities. To date, about $1
    billion has been spent to partially deploy NTCSS to about
    one-half its intended ashore and afloat sites.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-06-215
        ACCNO:   A42697
      TITLE:     DOD Systems Modernization: Planned Investment in the
    Naval Tactical Command Support System Needs to be Reassessed
         DATE:   12/05/2005
      SUBJECT:   Best practices
    	     Best practices methodology
    	     Defense procurement
    	     Economic analysis
    	     Enterprise architecture
    	     Evaluation criteria
    	     Internal controls
    	     Procurement planning
    	     Procurement policy
    	     Program evaluation
    	     Naval Tactical Command Support System

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-06-215

         * Report to the Subcommittee on Readiness and Management Support,
           Committee on Armed Services, U.S. Senate
              * December 2005
         * DOD SYSTEMS MODERNIZATION
              * Planned Investment in the Naval Tactical Command Support System
                Needs to Be Reassessed
         * Contents
              * Results in Brief
              * Background
                   * NTCSS Genesis and Status Overview
                   * NTCSS Oversight and Management Roles and Responsibilities
                   * NTCSS Participation in DOD's Rapid Improvement Team Pilot
                   * Prior Review Identified Strengths and Weaknesses in DOD's
                     Acquisition Policies and Guidance
              * NTCSS Has Not Been Managed in Accordance with DOD and Other
                Relevant System Acquisition and Development Guidance
                   * The Navy Has Not Economically Justified Investment in NTCSS
                     on the Basis of Costs and Benefits
                        * The Latest NTCSS Cost Estimate Was Not Derived Reliably
                   * The Latest NTCSS Economic Analysis Did Not Meet Key Federal
                     Guidance
                        * The Latest NTCSS Economic Analysis Was Not
                          Independently Reviewed
                   * The Navy Has Yet to Measure Whether Actual Benefits Have
                     Accrued from Deployed NTCSS Capabilities
                   * The Navy Recently Decided to Prepare a Benefits Assessment
                   * The Navy Has Not Defined and Developed NTCSS within the
                     Context of an Enterprise Architecture
                   * Key Program Management and Oversight Activities Have Not
                     Been Effectively Performed
                        * The Navy is Not Adequately Measuring Progress Against
                          Planned Cost and Scheduled Work Commitments
                             * DOD Has Adopted Industry Standards for Earned
                               Value Management
                             * NTCSS Has Not Effectively Implemented EVM
                             * Two NTCSS Projects Illustrate How EVM Has Been
                               Poorly Implemented
                        * The Navy Has Not Adequately Reported NTCSS's Progress
                          and Problems
                             * Navy Reporting Requirements for NTCSS Have Changed
                               over the Last Several Years
                             * The Navy Has Not Satisfied All NTCSS Reporting
                               Requirements
                        * The Navy Has Not Properly Budgeted for NTCSS
                        * Navy Oversight of NTCSS Has Not Been Adequate
                             * The Milestone Decision Authority Has Not
                               Adequately Overseen the Program
                             * Other Navy Organizations Have Not Conducted
                               Program Oversight
                   * NTCSS Requirements and Test Management Weaknesses Have
                     Contributed to Deployment Delays and System Quality Problems
                        * The Navy Has Not Adequately Managed Requirements for
                          the NTCSS Application Currently Under Development
                             * Requirements for OOMA Release 4.10 Were Not Traced
                             * Requirements for OOMA Release 4.10 Were Not
                               Prioritized
                        * The Navy's Developmental Testing for OOMA Has Not Been
                          Effective, but Improvements Planned
                             * Navy Operational Testing Organization Reported
                               That Developmental Testing Has Failed to Identify
                               Problems
                             * Developmental Test Documentation Has Not Been
                               Adequate, but Improvements Planned
                        * Central Design Agency Reports Management Improvements
                          are Under Way
              * Conclusions
              * Recommendations for Executive Action
              * Agency Comments and Our Evaluation
         * Objective, Scope, and Methodology
         * Trouble Reports and Change Proposals Assessment
              * Trouble Reports
              * Change Proposals
         * Earned Value Management Assessment
         * Comments from the Department of Defense
         * GAO Contact and Staff Acknowledgments

    Report to the Subcommittee on Readiness and Management Support, Committee
    on Armed Services, U.S. Senate

    December 2005

    DOD SYSTEMS MODERNIZATION

    Planned Investment in the Naval Tactical Command Support System Needs to
    Be Reassessed

    Contents

    Tables

    Figures

    December 5, 2005Letter

    The Honorable John Ensign Chairman The Honorable Daniel K. Akaka Ranking
    Minority Member Subcommittee on Readiness and Management Support Committee
    on Armed Services United States Senate

    Because it is so important that the Department of Defense (DOD) adhere to
    disciplined information technology (IT) acquisition processes in order to
    successfully modernize its business systems, you requested that we
    determine whether the department is following its own revised policies and
    guidance for acquiring systems,1 which it issued in May 2003. As part of
    our response to your request, we agreed to review the Naval Tactical
    Command Support System (NTCSS) program. NTCSS was started in 1995 and is
    intended to help Navy personnel effectively manage ships, submarines, and
    aircraft support activities. The Navy expects to spend $348 million on
    NTCSS between fiscal years 2006 and 2009, for a total of approximately
    $1.45 billion since program inception.

    As agreed, our objective was to determine whether NTCSS is being managed
    according to important aspects of DOD's acquisition policies and guidance,
    as well as other relevant acquisition management best practices. We
    focused on the program's (1) economic justification; (2) architectural
    alignment; (3) project management, including progress measurement,
    progress reporting, funding disclosure, and oversight activities; and
    (4) system development, including requirements management and testing. For
    requirements management and testing, we focused on the NTCSS application
    that is currently being developed, known as the Optimized Organizational
    Maintenance Activity (OOMA).

    We conducted our review from September 2004 through November 2005 in
    accordance with generally accepted government auditing standards. For
    details on our objective, scope, and methodology, see appendix I.

    Results in Brief

    The Navy has not managed its NTCSS program in accordance with key aspects
    of the department's system acquisition policies and related guidance,
    including federal and recognized best practice guidance. Collectively,
    these policies and guidance are intended to reasonably ensure that
    investment in a given IT system represents the right solution to fill a
    mission need-and if it is, that acquisition and deployment of the system
    are handled in a manner that maximizes the chances of delivering defined
    system capabilities on time and within budget. In the case of NTCSS,
    neither of these outcomes is being realized. As a result, the Navy does
    not currently have a sufficient basis for determining whether NTCSS is the
    right systems solution for its aircraft, ship, and submarine tactical
    command support needs, and it has not pursued the proposed solution in the
    right way, meaning in a fashion that increases chances of delivering
    defined capabilities on time and within budget. Key areas in which the
    Navy did not follow relevant policies and guidance are described here.

    o The Navy has not economically justified its ongoing and planned
    investment in NTCSS on the basis of reliable estimates of future costs and
    benefits. The most recent economic justification's cost estimates were not
    reliably derived, and return on investment was not properly calculated. In
    addition, independent reviews of the economic justification to determine
    its reliability did not occur, and the Navy has not measured whether
    already deployed and operating components of the system are producing
    expected value.

    o The Navy has not invested in NTCSS within the context of a well-defined
    enterprise architecture, which is an institutional blueprint to guide and
    constrain program investment decisions in a way that promotes
    interoperability and reduces redundancy among related and dependent
    systems. As we recently reported,2 DOD's business enterprise architecture
    does not contain sufficient context (depth and scope of operational and
    technical requirements) to effectively guide and constrain business
    transformation and system modernization efforts. Further, the Navy does
    not yet have a defined architecture, although it plans to develop one.
    Investing in systems, in the absence of an enterprise architecture,
    requires explicit recognition and deliberate consideration of the inherent
    risks to ensure fully informed investment decision making.

    o The Navy has not effectively performed key measurement, reporting, and
    oversight activities. In particular, earned value management, which is a
    means for determining and disclosing actual performance against budget and
    schedule estimates, and revising estimates based on performance to date
    has not been implemented effectively. Also, complete and current reporting
    of NTCSS progress and problems in meeting cost, schedule, and performance
    goals has not occurred, leaving oversight entities without the information
    needed to mitigate risks, address problems, and take corrective action. In
    addition, NTCSS budgets have not reflected the proper category of
    appropriated funds associated with system development efforts. Further,
    oversight entities' roles and responsibilities have not been fully
    discharged.

    o The Navy has not adequately conducted requirements management and
    testing activities. For the NTCSS application that is currently under
    development, the Navy has not adequately managed requirements, as
    evidenced by the absence of requirements traceability to system design
    specifications and testing documents, and the lack of prioritization of
    the requirements. The lack of requirements traceability and other issues
    have in turn contributed to problems with developmental testing, including
    the failure of these tests to identify problems that subsequently
    prevented the system from passing operational testing twice over the last
    4 years. Based on the Navy's data, the recent trend in key indicators of
    system maturity, such as the number and nature of reported systems
    problems and change proposals, shows that problems with NTCSS persist and
    that these problems could involve costly and timely rework to address.3

    Reasons the Navy cited for not following policies and guidance included
    questioning their applicability to the NTCSS program, having insufficient
    time in which to apply them, and believing that plans to adopt them were
    not meant to be applied retroactively. In some cases, the Navy did not
    acknowledge that any deviations from policies and guidance had occurred,
    but in these cases, it has yet to provide us with documentation
    demonstrating that it did adhere to them. Collectively, this means that
    after investing 10 years and $1 billion on NTCSS, it is unclear whether
    the Navy's planned future investment in the program is warranted. Even if
    key uncertainties are addressed and it can be demonstrated that NTCSS is
    the right solution, then the manner in which NTCSS is being defined,
    developed, tested, measured, and overseen is also of concern. Accordingly,
    we are making recommendations to the Secretary of Defense aimed at
    developing the basis needed to determine whether continued investment in
    NTCSS is a prudent use of limited departmental resources. We are also
    making recommendations to strengthen management of the program,
    conditional upon a decision to proceed with further investment in the
    NTCSS program.

    The Office of the Assistant Secretary of Defense for Networks and
    Information Integration provided written comments on a draft of the
    report. In its comments, DOD concurred with two of the recommendations and
    partially concurred with the remaining five. DOD also stated that while
    some of our findings are valid, our overall findings significantly
    understated and misrepresented the program's level of discipline and
    conformance with applicable guidance and direction. We do not agree. Our
    report cites numerous instances, supported by analyses, where the Navy did
    not comply with either DOD acquisition policies and guidelines or industry
    best practices. DOD's comments are reprinted in their entirety in appendix
    IV of this report, along with our detailed responses to each.

    Background

    The Navy's primary mission is to organize, train, maintain, and equip
    combat-ready naval forces capable of winning the global war on terror and
    any other armed conflict, deterring aggression by would-be foes,
    preserving freedom of the seas, and promoting peace and security. To
    support this mission, the Navy performs a variety of interrelated and
    interdependent business functions such as logistics and financial
    management. The Navy requested, for fiscal year 2005, about $3.5 billion
    to operate, maintain, and modernize its business systems and related IT
    infrastructure that support these business functions. This request
    represents about 27 percent of the $13 billion that DOD requested for all
    of its business systems for fiscal year 2005. Of the 4,150 business
    systems that DOD reports in its current inventory, the Navy accounts for
    2,353, or about 57 percent, of the total.

    In 1995, we designated DOD's business systems modernization efforts as a
    high-risk program and continue to designate it as such today4 for several
    reasons, including the department's challenges in implementing effective
    IT investment management structures and processes, developing and
    implementing an enterprise architecture, and implementing effective IT
    system acquisition and development processes.

    NTCSS Genesis and Status Overview

    In the early 1990s, the Navy employed a variety of IT systems to support
    the management of information, personnel, materials, and funds required to
    maintain and operate ships, submarines, and aircraft. Three core
    systems-each managed by a separate program office-consisting of nine major
    applications, provided this support: (1) the Shipboard Non-Tactical
    Automated Data Processing Program (SNAP), managed by the Space and Naval
    Warfare Systems Command; (2) the Naval Aviation Logistics Command
    Management Information System (NALCOMIS), managed by the Naval Air Systems
    Command; and (3) the Maintenance Resource Management System (MRMS),
    managed by the Naval Sea Systems Command. See table 1 for a description of
    these three legacy systems and a list of their respective applications.

    Table 1: Legacy Systems and Applications

    Legacy system Description                          Application
    SNAP systems  Manages systems for maintenance,     SNAP I:
                     supply, and financial operations at
    SNAP I        the organizational and intermediate  o Shipboard Uniform
                     levels.a                             Automated Data
    SNAP II                                            Processing System
                     Manages medical and dental services,
                     pay and personnel administration,    o Organizational
                     food service, retail sales and       Maintenance Management
                     service, training programs,          System
                     technical data storage and
                     retrieval, support and test          o Administration Data
                     equipment, and other mission         Management I
                     support-related areas at the
                     organizational level.                SNAP II:

                     SNAP I was developed for the Navy's  o Supply and Financial
                     larger ships, marine aviation        Management
                     logistics squadrons,b training
                     sites, and selected activities       o Organizational
                     ashore.                              Maintenance Management
                                                          System II Maintenance
                     SNAP II provides the same            Data System
                     functionality as SNAP I, but it was
                     developed for use on smaller ships   o Administration Data
                     and submarines. SNAP II was also     Management II
                     modified to use microcomputers as
                     the computing platforms when it is
                     deployed on ships with constricted
                     physical space; this version is
                     known as MicroSNAP.
    NALCOMIS      Supports day-to-day aircraft         o NALCOMIS
                     maintenance and related material     Organizational
                     maintenance functionality both at    Maintenance Activity
                     sea and ashore.
                                                          o NALCOMIS Intermediate
                     Provides the initial maintenance     Maintenance Activity
                     response when a problem is
                     reported-including aircraft
                     component troubleshooting,
                     servicing, inspection, and removal
                     and replacement at the
                     organizational level.

                     Supports, at the intermediate
                     maintenance level, the repair of
                     components after defective parts
                     have been removed from an aircraft
                     and sent to a central location to be
                     refurbished.
    MRMS          Supports intermediate-level ship and o Maintenance Resource
                     submarine maintenance at ashore      Management System
                     facilities by providing management
                     information such as planning,
                     scheduling, workload forecasting,
                     work progression, production
                     control, productivity analysis, and
                     resource management.

    Source: Navy.

    aThe "organizational" level is the first stage of aircraft maintenance
    activity that is performed on individual planes and involves the upkeep
    and servicing of the aircraft at the location where it is deployed, such
    as a ship. Components or parts that cannot be repaired at the
    organizational level are removed from the plane and sent to a central
    location for repair. This second stage of maintenance is known as the
    "intermediate" level, and it normally occurs on land. If the defective
    part cannot be fixed at the intermediate level, it is then sent to a third
    stage of maintenance, known as the "depot" level, which is not in the
    scope of the NTCSS program.

    bMarine aviation logistics squadrons are groups of planes that are
    land-based but that can be deployed on an aircraft carrier for a specific
    mission. When the mission is completed, these planes return to their land
    base.

    In 1992, we recommended that the Navy merge the management of all
    shipboard nontactical programs under a single command that would have
    authority and control over funding and development.5 In 1993, the Navy
    developed a strategy to do so. In 1994, the Navy also identified a number
    of problems with the three legacy systems. Specifically, the Navy
    determined that (1) the individual systems did not consistently handle
    increasing workloads and provide the flexibility to meet changing
    operational demands; (2) the systems' software architectures were
    ineffective and inefficient; (3) the hardware was outdated, slow,
    expensive to maintain, and nonstandard; and (4) the systems could not
    support modernized applications.

    To address these concerns, the Navy initiated the NTCSS program in 1995 to
    enhance the combat readiness of ships, submarines, and aircraft. To
    accomplish this, NTCSS was to provide unit commanding officers and crews
    with information about, for example, maintenance activities, parts
    inventories, finances, technical manuals and drawings, and personnel.
    According to the Navy, it spent approximately $1.1 billion for NTCSS from
    its inception through fiscal year 2005 and expects to spend another $348
    million between fiscal years 2006 and 2009, for a total of approximately
    $1.45 billion.

    The Navy defined a three-stage acquisition process for NTCSS.

    Stage 1: Purpose was to replace hardware in order to establish a common
    infrastructure across all force-level ships, unit-level ships, aviation
    squadrons, Naval air stations, marine aviation logistics squadrons, and
    other maintenance activities-both at sea and ashore.6 During this stage,
    software and business processes were not to be changed. This phase was
    begun in 1994 under the legacy SNAP and NALCOMIS programs and, according
    to program officials, it is fundamentally complete-although technology
    refresh or replacement activities are still occurring.

    Stage 2: Purpose was to provide the functionality of the legacy systems
    software with more efficient, more easily maintained software and to
    eliminate functional overlap among the systems. This stage was to involve
    software technology modernization but no changes in software functionality
    or business processes. Existing legacy systems used flat files and
    hierarchical databases, which were to be converted to relational
    databases, and the existing application code was to be rewritten using
    modern software languages. A common hardware and systems software
    environment was also to be implemented, and functionality found in eight
    of the nine legacy applications was to be consolidated and rewritten as
    four new NTCSS applications. Development of these four applications began
    in 1995 and was reportedly completed in 2000. This stage was known as
    NTCSS Optimization. See table 2 for a description of the functionality of
    these new applications.

    Stage 3: Purpose was to improve NTCSS's functionality by implementing
    business process improvements. According to Navy officials, this stage is
    known as NTCSS Modernization and, to date, includes two efforts:
    (1) replace the last legacy application and (2) create a Web-enabled
    version of the three unit-level Optimized NTCSS applications that were
    developed under Stage 2. See table 3 for a description of the
    functionality of these business process improvements.

    Table 2: Optimized Applications Developed During Stage 2 of the NTCSS
    Program

    NTCSS Optimized     Description                       Status
    applications
    Relational Supply   Supports supply chain management, Operational, as of
                           inventory management, and         September 1998, on
                           financial management processes.   large force-level
                                                             ships, smaller
                           Provides Navy personnel with      unit-level ships,
                           access to the supply support      and at air stations
                           functions they perform most       and marine aviation
                           often-ordering, receiving, and    logistics
                           issuing necessary supplies and    squadrons.a
                           materials; maintaining financial
                           records; and reconciling supply,
                           inventory, and financial records
                           with the Navy's shore
                           infrastructure.
    Organizational      Assists shipboard personnel in    Operational, as of
    Maintenance         planning, scheduling, reporting,  September 1998,
    Management          and tracking maintenance and      primarily on large
    System-Next         related logistics support         force-level ships
    Generation          actions.                          and smaller
                                                             unit-level ships.
                           Maintains online lists of
                           maintenance actions to be
                           performed, parts required to
                           maintain shipboard equipment, and
                           parts carried onboard ship to
                           support maintenance actions.

                           Interfaces with Relational Supply
                           to requisition parts that are not
                           onboard.
    Relational          Automates the management of       Operational, as of
    Administration Data personnel awards and decorations, April 2000, on large
    Management          work assignments, and berthing    force-level ships,
                           assignments.                      smaller unit-level
                                                             ships, and at air
                                                             stations and marine
                                                             aviation logistics
                                                             squadrons.
    Optimized           Provides online                   Operational, as of
    Intermediate        intermediate-level aviation       April 2000, at
    Maintenance         maintenance, configuration, and   force-level ships
    Activities          logistics management support.     and at air stations
                                                             and marine aviation
                           Interfaces with other major       logistics squadrons.
                           integrated logistics support
                           systems within the Naval aviation
                           community.

    Source: Navy.

    aRelational Supply is also in use at additional sites that are not a part
    of the NTCSS program.

    Table 3: Modernized Applications Developed During Stage 3 of the NTCSS
    Program

    NTCSS modernized Description                        Status
    applications
    Optimized        Is to support day-to-day           Initiated in 1999,
    Organizational   maintenance management tools for   withdrawn from
    Maintenance      aviation squadrons and other       operational testingb
    Activity (OOMA)  organizational-level maintenance   in April 2001 when it
                        activities.                        became clear that it
                                                           would fail. Failed
                        Is to provide the foundation for   operational testing
                        achieving a completely automated   again in May 2004.
                        maintenance environment, such as a Scheduled for third
                        single point of data entry,        operational test in
                        automated and assisted pilot and   the third quarter of
                        maintenance debrief, online        fiscal year 2006.
                        diagnostics, structural life
                        prognostics,a interactive          Fielded at 77 sites as
                        electronic technical manuals, and  of June 2005.
                        forecasting and tracking of
                        maintenance schedules.
    eNTCSS           Was to provide a Web-enabled       Initiated in 2001.
                        version of NTCSS, and allow users  Cancelled in April
                        to access the three unit-level     2004.
                        Optimized applications from any
                        workstation on a ship's local area Fielded on one
                        network via a standard Web browser submarine and
                        and to execute work activities in  scheduled to be
                        a Web-server environment.          fielded on one more.

                                                           Is to be replaced with
                                                           the Optimized
                                                           applications, but a
                                                           date has yet to be
                                                           determined.

    Source: Navy.

    aAccording to the U.S. Marine Corps Logistics Directorate, structural life
    prognostics is defined as the ability to reliably predict the remaining
    useful life of mechanical or structural components, within an actionable
    time period, and within acceptable confidence limits.

    bAccording to the DOD Defense Acquisition Guidebook, the primary purpose
    of operational test and evaluation is for representative users to evaluate
    systems in a realistic environment in order to determine whether these
    systems are operationally effective and suitable for their intended use
    before production or deployment.

    As of April 2005, legacy applications were still in use at 51 percent of
    the Navy's 659 sites. These 659 sites either have legacy, Optimized, or
    modernized applications. Table 4 shows the distribution of the legacy,
    Optimized, and modernized applications.

    Table 4: Applications in Operation as of April 2005

    Applications                           Number of sites Percentage of total
    Legacy applications
    SNAP Ia, b                                          10
    SNAP IIa, b, c                                      68
    MicroSNAP                                           32
    NALCOMIS Organizational Maintenance                214
    Activityd
    NALCOMIS Intermediate Maintenance                   10
    Activityb
    Maintenance Resource Management                      2
    Systeme
    Subtotal                                           336                  51
    Optimized applicationsf
    Relational Supplyc
    Organizational Maintenance Management
    System - Next Generation
    Relational Administration Data
    Management
    Optimized Intermediate Maintenance
    Activities
    Subtotal                                           229                  35
    Modernized applications
    Optimized Organizational Maintenance                93
    Activity
    eNTCSS                                               1
    Subtotal                                            94                  14
    Total                                              659                 100

    Source: Navy.

    aSNAP I and SNAP II are each composed of three different legacy
    applications (see table 1).

    bThe Navy plans to decommission some of the ships that use these
    applications and upgrade the remaining ships to NTCSS Optimized
    applications.

    cThis application also is in use at additional sites that are not a part
    of the NTCSS program.

    dThe functionality included in this application is to be replaced in the
    future by Optimized Organizational Maintenance Activity.

    eThe Navy plans to incorporate this functionality into Organizational
    Maintenance Management System-Next Generation at a future date.

    fThese four applications are deployed as a single software package at all
    229 sites.

    According to Navy officials, about $1.1 billion was spent on NTCSS between
    1995 and 2005. This includes about $1 billion on NTCSS Optimized
    applications7 and $91 million on OOMA and eNTCSS. Table 5 shows NTCSS's
    budget totals from the time the program began in fiscal year 1995 through
    fiscal year 2005.

    Table 5: NTCSS Budget from FY 1995 through FY 2005

     Dollars
    in
    thousands
               FY 95  FY 96  FY 97   FY 98   FY 99   FY 00   FY 01   FY 02  FY 03   FY 04  FY 05     Total
    NTCSS     83,537 69,794 69,075 123,469 119,822  91,053  95,322  95,549 82,708 108,087 71,926 1,010,342
    Optimized
    OOMA                920    700     983   4,724  16,527  20,854  14,920  3,981   2,871 13,291    79,771
    eNTCSS                                                   5,000   5,309    985                  11, 294
    Total     83,537 70,714 69,775 124,452 124,546 107,580 121,176 115,778 87,674 110,958 85,217 1,101,407

    Source: Navy.

    NTCSS Oversight and Management Roles and Responsibilities

    A number of Navy and DOD organizations are involved in overseeing and
    managing the NTCSS program. Table 6 lists the organizations involved in
    NTCSS oversight and their respective roles and responsibilities.

    Table 6: NTCSS Oversight Roles and Responsibilities

    Oversight entity               Roles and responsibilities
    Deputy Assistant Secretary of  Currently serves as the milestone decision
    the Navy for Command, Control, authority. Assigned overall responsibility
    Communication, Computers and   for the NTCSS program; approves the program
    Intelligence, and Space        to proceed through its acquisition cycle on
                                      the basis of a review of key documents,
                                      such as an acquisition plan, an
                                      independently evaluated life cycle
                                      cost-and-benefits estimate, Acquisition
                                      Program Baseline documents, and Defense
                                      Acquisition Executive Summary reports.
    Program Executive Office for   Serves as the program executive office.
    Command, Control,              Assigned overall responsibility for NTCSS
    Communication, Computers and   program oversight; reviews the component
    Intelligence, and Space; Space cost analysis, acquisition strategy, and
    and Naval Warfare Systems      Acquisition Program Baseline prior to
    Command                        approval by the milestone decision
                                      authority.
    Department of Navy Chief       Reviews the acquisition program during the
    Information Officer            department's planning, programming,
                                      budgeting, and execution processes to
                                      ensure that the program's goals are
                                      achievable and executable; ensures
                                      conformance to appropriation law, financial
                                      management regulations, and Navy, DOD, and
                                      federal IT policies in several areas (e.g.,
                                      security, architecture, and investment
                                      management); works closely with the program
                                      office during milestone review assessments.
    Assistant Secretary of the     Ensures system compliance with
    Navy, Research Development and architectural standards and promotes
    Acquisition, Chief Engineer    interoperability of the Navy's systems.
    Office of the Secretary of     Verifies and validates the reliability of
    Defense, Office of the         cost and benefit estimates found in
    Director for Program Analysis  economic analyses and provides its results
    and Evaluation                 to the milestone decision authority.
    Naval Cost Analysis Division   Performs independent cost estimates,
                                      maintains cost analysis tools, and focuses
                                      on cost analysis policy and oversight.
    Executive Steering Committee   Establishes priorities for NTCSS
                                      development and implementation and for
    Members are representatives    defining long-term architectural goals;
    from:                          meets after regularly scheduled NTCSS
                                      meetings (e.g., Requirements Integrated
    Office of the Chief of Naval   Product Team meetings and Forum meetings).a
    Operations for Material
    Readiness and Logistics
    Operations (Chairman);

    Commander in Chief, U.S.
    Atlantic Fleet;

    Commander in Chief, U.S.
    Pacific Fleet;

    Commandant of the Marine
    Corps; and

    Program Executive Office for
    Command, Control,
    Communication, Computers and
    Intelligence, and Space.

    Source: Navy.

    aThe Requirements Integrated Product Team is chartered to collect and
    analyze users' requirements, input these requirements into the NTCSS
    requirements management process, and provide recommendations to the
    program office on these requirements. The Forum brings together
    stakeholders and acquisition and development personnel to (1) discuss
    issues and requirements related to current and future system readiness,
    (2) develop specific action items and recommendations that will result in
    improved program products and services to the Fleet, and (3) facilitate
    key decisions by senior program leadership at Executive Steering Committee
    meetings.

    There have been three milestone decision authorities for NTCSS since the
    program was begun. Initially, the milestone decision authority was in the
    Office of the Assistant Secretary of Defense for Networks and Information
    Integration/Chief Information Officer. In July 1999, this authority was
    delegated to the Assistant Secretary of the Navy for Research,
    Development, and Acquisition, who then delegated oversight authority to
    Deputy Assistant Secretary of the Navy for Command, Control,
    Communication, Computers and Intelligence, and Space in March 2000.

    Table 7 lists the organizations involved in NTCSS management and their
    respective roles and responsibilities.

    Table 7: NTCSS Management and Stakeholder Roles and Responsibilities

    Entity                     Roles and responsibilities
    Program Manager, Warfare;  Serves as the program office. Assigned
    Space and Naval Warfare    responsibility for day-to-day program
    Systems Command            management of NTCSS and, as such, is the single
                                  point of accountability for managing the
                                  program's objectives through development,
                                  production, and sustainment. Manages cost,
                                  schedule, and performance reporting. Prepares
                                  and updates the acquisition strategy, component
                                  cost analysis, and acquisition program
                                  baselines. Coordinates all testing activities
                                  in coordination with requirements.
    Space and Naval Warfare    Serves as the central design agency. Assigned
    Systems Command, Systems   responsibility for software development,
    Center Norfolk             including application design, development, and
                                  testing activities. Responsible for managing
                                  trouble reports and change proposals.a Manages
                                  Space and Naval Warfare Systems Command,
                                  Systems Center Norfolk Detachment San Diego,
                                  which installs the initial NTCSS systems on
                                  ships, submarines, and at land sites and
                                  performs subsequent on-site software
                                  maintenance.
    Space and Naval Warfare    Serves as the in-service engineering activity.
    Systems Command, Systems   Provides engineering support and installs and
    Center Charleston          integrates hardware.
    Office of the Chief of     Serves as the program and resource sponsor.
    Naval Operations for       Balances user requirements with available
    Material Readiness and     resources. Works with users to ensure that
    Logistics Operations       operational and functional requirements are
                                  prioritized correctly and are supported.
                                  Addresses various issues pertaining to Navy
                                  policy, requirements, resources, and schedules.
    Functional Managers        Represent the system users. Participate in the
                                  process of establishing functional requirements
    Includes representatives   for input into the change management and system
    from:                      design processes. Prepare test plans and test
                                  analysis reports to support functional
    Naval Sea Systems Command; certification of software.

    Naval Supply Systems
    Command;

    Naval Air Systems Command;
    and

    Commander in Chief, U.S.
    Atlantic Fleet.

    Source: Navy.

    aNavy officials provided data regarding trouble reports and change
    proposals for the Optimized and modernized NTCSS applications. For details
    see appendix II.

    NTCSS Participation in DOD's Rapid Improvement Team Pilot

    In 2001, the DOD Chief Information Officer and the Undersecretary of
    Defense for Acquisition, Technology, and Logistics chartered a pilot
    project aimed at saving time by significantly reducing the reporting and
    oversight requirements. The ultimate goal was to enable the acquisition
    process to deliver mission-effective IT systems within 18 months. Known as
    the Rapid Improvement Team (RIT) for IT Acquisition Management
    Transformation, the pilot was to cover a 2-year period from January 1,
    2002, through December 31, 2003. Nine programs from the military services
    participated in the pilot. NTCSS was selected to participate in the pilot
    by its milestone decision authority due to its longevity and because of
    its perceived low risk, stability, and compliance with IT management best
    practices. It was also believed that little system development remained to
    be done. NTCSS began participating in the RIT pilot in October 2002.

    The RIT pilot relieved the program office of the normal acquisition
    process activities, such as preplanned, formal milestone decision reviews
    or briefings, and it granted the program office the authority to pass key
    milestones once it determined that established requirements had been met.
    This streamlined approach was considered possible because all information
    related to these requirements was to be continually updated and available
    to oversight organizations and stakeholders via a RIT Web site. More
    specifically, the program office was to update the Web site monthly via a
    set of electronic forms with the kind of data that were traditionally
    found in DOD oversight documents. The program office was also to use the
    Web site to input key acquisition documents (e.g., acquisition plans,
    economic analyses, requirements documents and test plans) in an electronic
    library. In turn, the milestone decision authority and other oversight
    organizations were to review these data on at least a monthly basis and
    directly retrieve any acquisition documents to be reviewed from the
    library. No response from the milestone decision authority would indicate
    implicit approval of the program data. Although the formal RIT pilot ended
    in December 2003, program officials told us that they continued to operate
    using the RIT pilot's procedures and continued to update program
    information on the Web site through December 2004.

    According to a memorandum issued by the Office of the Assistant Secretary
    of Defense for Networks and Information Integration/Chief Information
    Officer and the Undersecretary of Defense for Acquisition, Technology, and
    Logistics, the principal output of the pilot would be a blueprint for IT
    acquisition that is transferable to other systems. A report summarizing
    the results of the entire RIT pilot program was published in

    April 2005.8 This report concluded that (1) by instituting risk-based
    governance, the milestone decision authority can be assigned to an
    organization subordinate to the Office of the Secretary of Defense without
    adding unacceptable risk to the investment process and (2) the success of
    risk-based governance and cycle time reduction is predicated on the
    adoption of net-centricity9 by the business community.

    Prior Review Identified Strengths and Weaknesses in DOD's Acquisition
    Policies and Guidance

    In July 2004, we reported10 that DOD's revised systems acquisition
    policies and guidance incorporated many best practices for acquiring
    business systems, such as (1) justifying system investments economically,
    on the basis of costs, benefits, and risks, and (2) continually measuring
    an acquisition's performance, cost, and schedule against approved
    baselines. However, the revised policies and guidance did not incorporate
    a number of other best practices, particularly those associated with
    acquiring commercial component-based business systems, and DOD did not
    have documented plans for incorporating these additional best practices
    into its policies. We also reported that the department's revised
    acquisition policies did not include sufficient controls to ensure that
    military services and defense agencies would appropriately follow these
    practices. We concluded that, until these additional best practices were
    incorporated into DOD's acquisition policies and guidance, there was
    increased risk that system acquisitions would not deliver planned
    capabilities and benefits on time and within budget and increased risk
    that an organization will not adopt and use best practices that were
    defined. Accordingly, we made 14 recommendations to the Secretary of
    Defense that were aimed at strengthening DOD's acquisition policy and
    guidance by including additional IT systems acquisition best practices and
    controls for ensuring that these best practices were followed. DOD agreed
    with most of our recommendations and has since issued additional system
    acquisition guidance.11

    NTCSS Has Not Been Managed in Accordance with DOD and Other Relevant
    System Acquisition and Development Guidance

    DOD system acquisition and development policies and guidance, along with
    other federal and best practices guidance, provide an effective framework
    within which to manage IT business system programs and investments, like
    NTCSS. Proper implementation of this framework can minimize program risks
    and better ensure that system investments deliver promised capabilities
    and benefits on time and within budget. The Navy has not managed NTCSS in
    accordance with many key aspects of these policies and guidance. For
    example, the Navy has not economically justified its investment in NTCSS
    on the basis of cost and benefits. It has not invested in NTCSS within the
    context of a well-defined enterprise architecture. Further, the Navy has
    not effectively performed key measurement, reporting, and oversight
    activities, and has not adequately conducted requirements management and
    testing activities. Reasons the Navy cited for not following policies and
    guidance included questioning their applicability to the NTCSS program,
    having insufficient time in which to apply them, and believing that plans
    to adopt them were not meant to be applied retroactively. In some cases,
    the Navy did not acknowledge that any deviations from policies and
    guidance had occurred but, in these cases, it has yet to provide us with
    documentation demonstrating that it did adhere to them. As a result, the
    Navy does not currently have a sufficient basis for determining whether
    NTCSS is the right system solution for its tactical command support needs,
    and it has not pursued the proposed solution in a way that increases the
    likelihood of delivering defined capabilities on time and within budget.

    The Navy Has Not Economically Justified Investment in NTCSS on the Basis
    of Costs and Benefits

    The decision to invest in any system should be based on reliable analyses
    of estimated system costs and expected benefits over the life of the
    program. DOD policy requires such analyses, and other relevant acquisition
    management practices provide guidance on how these analyses should be
    prepared. However, the current economic analysis for the NTCSS program
    does not meet this guidance. Additionally, the analysis was not
    independently reviewed in accordance with DOD guidance. Finally, contrary
    to DOD policy and relevant acquisition management practices, the Navy has
    not demonstrated that NTCSS Optimized applications are producing expected
    benefits. Without such reliable analyses, an organization cannot
    adequately know that a given system investment is justified.

    The Latest NTCSS Cost Estimate Was Not Derived Reliably

    According to DOD guidance,12 the cost estimates used to economically
    justify an investment should be reasonable, traceable, and based on
    realistic assumptions. Our research shows that a reliable cost estimate
    should meet nine specific criteria developed by Carnegie Mellon
    University's Software Engineering Institute (SEI),13 such as appropriately
    sizing the task being estimated and identifying and explaining estimate
    assumptions.

    In March 2004, the NTCSS program office prepared its fifth NTCSS economic
    analysis. This analysis examined the costs associated with three
    alternative NTCSS hardware, software, operating system and data base
    management configurations, and was to be used to inform decisions about
    system development and implementation. The analysis did include estimated
    costs for each alternative. However, it did not include measurable,
    quantifiable benefits for each alternative. Rather, it included only
    qualitative benefits. Further, the cost estimates used in this analysis
    did not meet six of the nine criteria associated with reliable cost
    estimates. For example, while the estimate's purpose was stated in
    writing, the system life cycle used was 6 years rather than the 10 years
    recommended. Also, documentation showing that the costs were based on data
    from the program's demonstrated accomplishments has yet to be provided to
    us, and the assumptions used to create the cost estimate were not
    identified and explained. See table 8 for the results of our analyses
    relative to each of the nine criteria.

    Table 8: Navy Satisfaction of Cost Estimating Criteria

            Criterion             Explanation       Criterion    GAO analysis
                                                      meta
    The objectives of    The objectives of the   Yes       The objective of
    the program are      program should be                 the program was
    stated in writing.   clearly and concisely             clearly stated.
                            stated for the cost
                            estimator to use.
    The life cycle to    The life cycle should   No        The life cycle was
    which the estimate   be clearly defined to             not clearly defined
    applies is clearly   ensure that the full              to ensure that the
    defined.             cost of the                       full cost of the
                            program-that is, all              program is
                            direct and indirect               included. The life
                            costs for planning,               cycle defined was 6
                            procurement, operations           years past full
                            and maintenance, and              operational
                            disposal-are captured.            capability, instead
                            For investments such as           of the full 10
                            NTCSS, the life cycle             years defined in
                            should cover 10 years             the DOD guidance.
                            past full operational
                            capability of the
                            system.b
    The task has been    An appropriate sizing   Yes       The method used in
    appropriately sized. metric should be used             the model lends
                            in the development of             itself to being
                            the estimate, such as             appropriately
                            the amount of software            sized.
                            to be developed and the
                            amount of software to
                            be revised.
    The estimated cost   Estimates should be     No        No documentation
    and schedule are     validated by relating             was provided to
    consistent with      them back to                      show the use of
    demonstrated         demonstrated and                  historical data to
    accomplishments on   documented performance            produce the
    other projects.      on completed projects.            estimate.
    A written summary of If a parametric         No        The model used
    parameter values and equation was used to              undocumented values
    their rationales     generate the estimate,            as the source of
    accompany the        the parameters that               the estimate for
    estimate.            feed the equation                 multiple elements.
                            should be provided,
                            along with an
                            explanation of why they
                            were chosen.
    Assumptions have     Accurate assumptions    No        Any assumptions
    been identified and  regarding issues such             used in the model
    explained.           as schedule, quantity,            were not
                            technology, development           identified.
                            processes,
                            manufacturing
                            techniques, software
                            language, etc., should
                            be understood and
                            documented.
    A structured         A work breakdown        Yes       A work breakdown
    process, such as a   structure or similar              structure was
    template or format,  structure that                    provided and
    has been used to     organizes, defines, and           included all the
    ensure that key      graphically displays              standards elements.
    factors have not     the individual work
    been overlooked.     units to be performed
                            should be used. The
                            structure should be
                            revised over time as
                            more information
                            becomes known about the
                            work to be performed.
    Uncertainties in     For all major cost      No        No risk analysis
    parameter values     drivers, an uncertainty           was documented in
    have been identified analysis should be                the estimate.
    and quantified.      performed to recognize
                            and reflect the risk
                            associated with the
                            cost estimate.
    If more than one     The primary methodology No        No secondary model
    cost model or        or cost model results             was discussed in
    estimating approach  should be compared with           the estimate
    has been used, any   any secondary                     documentation.
    differences in the   methodology (for
    results have been    example, cross-checks)
    analyzed and         to ensure consistency.
    explained.

    Sources: SEI criteria, DOD guidance, and GAO analysis of Navy data.

    a"Yes" means that the program provided documentation demonstrating
    satisfaction of the criterion. "Partially" means that the program provided
    documentation demonstrating satisfaction of part of the criterion. "No"
    means that the program has yet to provide documentation demonstrating
    satisfaction of the criterion.

    bDOD, DOD Automated Information System (AIS) Economic Analysis (EA) Guide,
    May 1, 1995.

    Program officials told us that they did not develop the 2004 cost estimate
    in accordance with all of the SEI cost estimating criteria because they
    had only a month to complete the economic analysis. By not following
    practices associated with reliable estimates, the Navy has decided on a
    course of action that is not based on one of the key ingredients to sound
    and prudent decision making- a reliable estimate of system life cycle
    costs. Among other things, this means that the investment decision made by
    the Navy has not been adequately justified and, that to the extent that
    program budgets were based on cost estimates, the likelihood of funding
    shortfalls and inadequate funding reserves is increased.

    The Latest NTCSS Economic Analysis Did Not Meet Key Federal Guidance

    According to Office of Management and Budget (OMB) guidance,14 economic
    analyses should meet certain criteria to be considered reasonable, such as
    comparing alternatives on the basis of net present value and conducting an
    uncertainty analysis of costs and benefits.

    The latest NTCSS economic analysis, prepared in March 2004, identified
    potential costs and benefits from three alternative NTCSS hardware,
    software, operating system, and data base management configurations.
    However, the analysis provided only monetized costs for each alternative.
    It did not provide monetized benefits. Further, the analysis did not meet
    five of eight OMB criteria. For example, the alternatives were not
    compared on the basis of their net present values, an appropriate interest
    rate was not used to discount the net present values, and the uncertainty
    associated with the cost estimates was not disclosed and used in the
    analysis. See table 9 for the results of our analyses relative to each of
    the eight criteria.

    Table 9: Navy Satisfaction of OMB Economic Analysis Criteria

           Criterion           Explanation       Criterion      GAO analysis
                                                   meta
    The economic       The economic analysis  Yes       The economic analysis
    analysis clearly   should clearly explain           explained why the
    explained why the  the reason why the               status quo alternative
    investment was     investment is needed,            was not viable.
    needed.            i.e., why the status
                          quo alternative is
                          unacceptable.
    At least two       At least two           Yes       Three alternatives to
    alternatives to    meaningful                       the status quo were
    the status quo     alternatives to the              considered.
    were considered.   status quo should be
                          examined to help
                          ensure that the
                          alternative chosen was
                          not preselected.
    The general        The general rationale  Yes       The rationale for each
    rationale for the  for the inclusion of             alternative was
    inclusion of each  each alternative                 discussed.
    alternative was    should be discussed to
    discussed.         enable reviewers of
                          the analysis to gain
                          an understanding of
                          the context for the
                          selection of one
                          alternative over the
                          others.
    The quality of the The quality of the     No        The cost estimates
    cost estimate for  cost estimate of each            were not complete and
    each alternative   alternative should be            did not meet a
    was reasonable.    complete and                     majority of the SEI
                          reasonable for a net             criteria.
                          present value to be
                          accurate. One measure
                          of a cost estimate's
                          reasonableness is its
                          satisfaction of
                          earlier cited SEI
                          criteria.
    The quality of the The quality of the     No        Monetized estimates of
    benefits to be     benefit estimate of              benefits were not
    realized from each each alternative                 provided, and no
    alternative was    should be complete and           explanation was given
    reasonable.        reasonable for a net             as to why these
                          present value to be              estimates were not
                          calculable and                   provided.
                          accurate.
    Alternatives were  The net present value  No        The economic analysis
    compared on the    should be calculated             stated that all costs
    basis of net       because it                       and benefits were
    present value.     consistently results             expressed in
                          in the selection of              undiscounted constant
                          the alternative with             fiscal year 2004
                          the greatest benefit             dollars; however,
                          net of cost.                     monetized benefits
                                                           were not reported in
                                                           the economic analysis.
                                                           As a result, the net
                                                           present value was not
                                                           calculated.
    The proper         OMB Circular A-94 is   No        Since all dollar
    discount rate used the general guidance             amounts are expressed
    for calculating    for conducting                   in undiscounted
    each alternative's cost-benefit analyses            constant fiscal year
    overall net        for federal government           2004 dollars, the
    present value      programs and provides            discount rate used in
    should be used.    specific guidance on             the economic analysis
                          the discount rates to            is, by default, zero.
                          be used in evaluating            The discount rates
                          those programs whose             provided by OMB
                          benefits and costs are           Circular No. A-94 are
                          distributed over time.           all positive (i.e.,
                                                           greater than zero).
    An uncertainty     Estimates of benefits  No        No uncertainty
    analysis of costs  and costs are                    analysis for the
    and benefits was   typically uncertain              overall reported costs
    included.          because of imprecision           was included.
                          in both underlying
                          data and modeling
                          assumptions. Because
                          such uncertainty is
                          basic to virtually any
                          cost-benefit analysis,
                          its effects should be
                          analyzed and reported.

    Sources: OMB guidance and GAO analysis of Navy data.

    a"Yes" means that the program provided documentation demonstrating
    satisfaction of the criterion. "Partially" means that the program provided
    documentation demonstrating satisfaction of part of the criterion. "No"
    means that the program has yet to provide documentation demonstrating
    satisfaction of the criterion.

    Program officials told us that they did not adhere to the OMB criteria
    because they had only a month to complete the economic analysis and,
    therefore, did not have the time necessary to comply with it. By not
    following established OMB guidance, the reliability of the latest NTCSS
    economic analysis is questionable. This further increases the risk that
    the Navy is following a course of action that will not produce the
    expected return on investment.

    The Latest NTCSS Economic Analysis Was Not Independently Reviewed

    DOD guidance15 states that economic analyses and cost estimates should be
    independently reviewed and assessed. In this regard, the Office of Program
    Analysis and Evaluation, located in the Office of the Secretary of
    Defense, is responsible for verifying and validating the reliability of
    economic analyses and providing the results to the milestone decision
    authority; the Naval Cost Analysis Division is responsible for preparing
    independent cost estimates.

    However, neither of these offices reviewed the most recent economic
    analysis for NTCSS. An official from the Office of Program Analysis and
    Evaluation told us that this office did not review the 2004 economic
    analysis because, once NTCSS entered the RIT Pilot, the program office no
    longer provided documentation needed to review the analysis. Officials
    from the Naval Cost Analysis Division also stated that they did not review
    the estimates in this economic analysis. According to officials from this
    office, they are only required to review cost estimates that are prepared
    for milestone reviews, and staffing limitations do not permit them to
    review all cost estimates.

    By not having the economic analysis reviewed by independent parties, the
    Navy has no independent verification that the estimates of life cycle
    costs and benefits are reasonable and traceable, that the cost estimates
    are built on realistic program and schedule assumptions, or that the
    return on investment calculation is valid. This casts further doubt on the
    reliability of the economic analysis the Navy has used to justify its
    ongoing investment in NTCSS.

    The Navy Has Yet to Measure Whether Actual Benefits Have Accrued from
    Deployed NTCSS Capabilities

    The Clinger-Cohen Act of 1996 and OMB guidance16 emphasize the need to
    develop information to ensure that IT projects are actually contributing
    to tangible, observable improvements in mission performance. DOD
    guidance17 also requires that analyses be conducted to validate estimated
    benefits and measure the extent to which desired outcomes have been
    achieved. To this end, agencies should define and collect metrics to
    determine whether expected benefits are being achieved and modify
    subsequent applications and investments to reflect the lessons learned.

    However, the Navy has yet to measure whether NTCSS Optimized applications
    are actually producing expected benefits commensurate with actual costs.
    For example, in 1999 the Navy projected that deploying the NTCSS Optimized
    applications would result in reduced costs associated with NTCSS
    maintenance, training, and other support activities. However, the Navy
    does not know the extent to which NTCSS Optimized applications are meeting
    these expectations-even though these applications have been deployed to
    229 user sites since 1998-because metrics to demonstrate that these
    expectations have been met have not been defined and collected.

    Program officials and officials representing the milestone decision
    authority stated that the Navy is not required to measure actual accrual
    of benefits because DOD guidance to do so was not yet in effect when the
    NTCSS Optimized applications were deployed, and there was no explicit
    requirement to apply this guidance retroactively. Program officials also
    stated that it will not be possible to measure actual return-on-investment
    for the already deployed NTCSS Optimized applications until the entire
    NTCSS system is deployed and operational. Similarly, an official with the
    milestone decision authority stated that actual NTCSS return-on-
    investment has not yet been measured.

    Because it is not measuring whether cost and benefit projections are being
    met, the Navy lacks important information that it will need to inform
    future economic analyses and investment decisions.

    The Navy Recently Decided to Prepare a Benefits Assessment

    In February 2005, officials from the Office of the Chief of Naval
    Operations for Material Readiness and Logistics Operations18 and
    representatives from key user organizations questioned whether NTCSS can
    cost effectively meet users' future needs. Initially this office tasked
    the program office to develop a new economic analysis to determine whether
    to continue investing in NTCSS or in some other system solution, such as
    the Navy enterprise resource planning (ERP) program.19 In November 2005,
    officials from the Office of the Chief of Naval Operations for Material
    Readiness and Logistics Operations stated that they were no longer
    planning to develop a new economic analysis but planning to conduct a
    benefits assessment to evaluate changing NTCSS to some solution to enable
    the system to perform future ashore activities. These officials
    acknowledged that this assessment will be less than the initially planned
    economic analysis in that it will exclude any analysis of costs and
    alternative solutions. However, they also acknowledged that DOD policy and
    guidance does not address benefits assessments as a recognized acquisition
    program document. They stated that this assessment will be prepared for
    inclusion in the 2006 budget submission.

    Without knowing the extent to which NTCSS Optimized applications are
    meeting cost and benefit expectations, the Navy is not in a position to
    make informed, and thus justified, decisions on whether and how to proceed
    with the program. Such a situation introduces a serious risk that the Navy
    will not be able to demonstrate whether NTCSS is cost-effective until it
    has already spent hundreds of millions of dollars more on the NTCSS
    Optimized applications and OOMA.

    The Navy Has Not Defined and Developed NTCSS within the Context of an
    Enterprise Architecture

    DOD policy and guidance,20 as well as federal and best practice
    guidance,21 recognize the importance of investing in IT business systems
    within the context of an enterprise architecture. Our research and
    experience in reviewing federal agencies shows that not doing so often
    results in systems that are duplicative, not well integrated,
    unnecessarily costly to interface and maintain, and do not optimally
    support mission outcomes.22 NTCSS has not been defined and developed in
    the context of a DOD or Navy enterprise architecture because a
    well-defined version of either has not existed to guide and constrain the
    program, and meaningful analysis showing how NTCSS aligns to evolving DOD
    and Navy architecture efforts was not produced. This means that the Navy
    does not have a sufficient basis for knowing if NTCSS, as defined,
    properly fits within the context of future DOD and Navy business
    operational and technological environments.

    More specifically, a well-defined enterprise architecture provides a clear
    and comprehensive picture of an entity, whether it is an organization
    (e.g., a federal department) or a functional or mission area that cuts
    across more than one organization (e.g., personnel management). This
    picture consists of snapshots of both the enterprise's current or "As Is"
    environment and its target or "To Be" environment, as well as a capital
    investment road map for transitioning from the current to the target
    environment. These snapshots consist of integrated "views," which are one
    or more architecture products that describe, for example, the enterprise's
    business processes and rules; information needs and flows among functions;
    supporting systems, services, and applications; and data and technical
    standards and structures. GAO has promoted the use of architectures to
    guide and constrain systems modernization, recognizing them as a crucial
    means to a challenging goal: agency operational structures that are
    optimally defined in both the business and technological environments.

    DOD has long operated without a well-defined enterprise architecture for
    its business environment. In 2001, we first reported that DOD did not have
    such an architecture and recommended that it develop one to guide and
    constrain IT business systems, like NTCSS.23 Over the next 4 years, we
    reported that DOD's architecture development efforts were not resulting in
    the kind of business enterprise architecture that could effectively guide
    and constrain business system investments,24 largely because the
    department did not have in place the architecture management structures
    and processes described in federal guidance. In particular, we most
    recently reported in July 200525 that despite spending about $318 million
    producing eight versions of its architecture, DOD's latest version still
    did not have, for example, a clearly defined purpose that could be linked
    to the department's goals and objectives and a description of the "As Is"
    environment and a transition plan. Further, we reported that the
    description of the "To Be" environment was still missing important content
    (depth and scope) relative to, for example, the actual systems to be
    developed or acquired to support future business operations and the
    physical infrastructure (e.g., hardware and software) that would be needed
    to support the business systems. Over the last several years, we have also
    reported that DOD's efforts for determining whether ongoing investments
    were aligned to its evolving architecture were not documented and
    independently verifiable.26 On September 28, 2005, DOD issued the next
    version of its business enterprise architecture,27 which we are required
    to review, along with other things such as the department's efforts to
    review certain investments' alignment with the architecture, pursuant to
    the Fiscal Year 2005 National Defense Authorization Act.28

    The Navy has also not had an enterprise architecture to guide and
    constrain its IT system investments. For example, in February 2002 and
    November 2003, we reported that while the Navy was developing an
    enterprise architecture, the architecture products were not complete and
    they were not, for example, under configuration management.29 Since that
    time, the Navy has yet to develop an enterprise architecture. In response
    to our request for the latest version of its architecture, the Assistant
    Secretary of the Navy, Research Development and Acquisition, Chief
    Engineer, provided us documentation that describes high-level principles
    or goals that the Navy wants to achieve, such as systems interoperability.
    However, most of the critical products that an enterprise architecture
    should include were not provided, such as (1) a data dictionary, which is
    a repository of standard data definitions for applications; (2) a logical
    database model that provides the data structures that support information
    flows and that provides the basis for developing the schemas for
    designing, building, and maintaining the existing physical databases; and
    (3) an analysis of the gaps between the baseline and target architecture
    for business processes, information/data, and services/application systems
    to define missing and needed capabilities. According to the Deputy
    Assistant Secretary of the Navy for Command, Control, Communication,
    Computers and Intelligence, and Space, the Navy does not have an
    enterprise architecture. However, these officials stated that the Navy
    recognizes the importance of developing and using one and is taking steps
    to do so. They did not have a time frame as to when this would be
    accomplished, however.

    In addition, NTCSS program officials told us that the system has been
    assessed against DOD's business enterprise architecture, and based on this
    assessment, the system is aligned. However, our analysis of the alignment
    documentation showed while NTCSS could be mapped to several enterprise
    architecture elements (e.g., strategic goals and organizational roles), it
    was not mapped to other important elements (e.g., technical standards and
    data model). Moreover, as previously discussed, the version of the
    enterprise architecture used to assess alignment lacked utility and did
    not provide a sufficient basis for making informed investment decisions.

    These officials stated that they have not yet assessed the system against
    the Navy's architecture because (1) the architecture has yet to be
    sufficiently developed and (2) compliance with this architecture may not
    be required.

    Without having a well-defined architecture to set the institutional
    context within which a given investment like NTCSS must fit and taking
    proactive and verifiable steps to understand the extent to which the
    system as it is defined fits within this context, misalignments can occur
    that can introduce redundancies and incompatibilities and that can produce
    inefficiencies and require costly and time consuming rework to fix. In the
    case of NTCSS, this could be a problem because of the Navy's ongoing
    investment in its ERP program.31 this program is 30 As we recently
    reported,intended to provide functionality in such areas as supply and
    workforce management for ashore activities, which is functionality similar
    to that of NTCSS for afloat activities. However, both programs have
    proceeded without a common, institutional frame of reference (i.e.,
    enterprise architecture) that can be used to effectively manage their
    relationships and dependencies. Our research and experience in reviewing
    federal agencies shows that managing such relationships on a program to
    program basis is untenable and has proven unsuccessful. This is why the
    inherent risks associated with investing in systems in the absence of a
    well-defined architecture need to be explicitly disclosed and deliberately
    evaluated in order to make a well-informed investment decision.

    Key Program Management and Oversight Activities Have Not Been Effectively
    Performed

    Key aspects of effective program management include reliable progress
    measurement and reporting, appropriate budgeting, and meaningful
    oversight. DOD policy requires such activities, and DOD and other industry
    best practices provide guidance on how these activities should be
    conducted. However, these activities have not been effectively performed
    on the NTCSS program. Specifically, the Navy has not adequately measured
    progress against planned cost and scheduled work commitments, fulfilled
    defined reporting requirements, properly budgeted for expenditures, and
    conducted meaningful program oversight. As a result, opportunities for
    proactive program intervention and actions to address risks and problems
    were missed, allowing the program to proceed largely unchecked.

    The Navy is Not Adequately Measuring Progress Against Planned Cost and
    Scheduled Work Commitments

    Measuring and reporting progress against cost and schedule commitments is
    a vital element of effective program management. DOD policy and guidance
    recognize this by requiring the use of earned value management, and
    describing how it is to be performed. The NTCSS program has elected to use
    earned value management; however, it is not doing so effectively. As a
    result, the program, as well as Navy and DOD oversight authorities, have
    not had access to the kind of reliable and timely information they need to
    make informed decisions.

    DOD Has Adopted Industry Standards for Earned Value Management

    According to DOD policy and guidance,32 program offices should obtain data
    from contractors and central design agencies on work progress, and these
    data should relate cost, schedule, and technical accomplishments.
    Moreover, the guidance states that these data should be valid, timely, and
    auditable. The tool that many DOD entities, including the NTCSS's program
    office and its central design agency, use to obtain and report these data
    is known as earned value management (EVM). Through EVM, program offices
    and others can determine a contractor's or central design agency's ability
    to perform work within cost and schedule estimates. It does so by
    examining variances between the actual cost and time to perform work tasks
    and the budgeted/estimated cost and time to perform the tasks.

    In 1996, DOD adopted industry guidance33 that identifies 32 criteria that
    a reliable EVM system should meet. The 32 criteria are organized into five
    categories: organization, planning and budgeting, accounting, analysis and
    management reports, and revisions and data maintenance (see app. III for
    the 32 criteria). As we previously reported,34 EVM offers many benefits
    when done properly. In particular, it is a means to measure performance
    and serves as an early warning system for deviations from plans. It
    therefore enables a program office to mitigate the risk of cost and
    schedule overruns.

    NTCSS Has Not Effectively Implemented EVM

    The EVM system that NTCSS has implemented to measure program performance
    does not provide the kind of reliable and timely data needed to
    effectively identify and mitigate risks. According to the NTCSS central
    design agency's self-assessment of its earned value management system, 17
    of the 32 industry best practice criteria are not being satisfied by the
    EVM system it has implemented. For example, the central design agency
    reported that the system cannot (1) establish and maintain a budget
    baseline against which program performance can be measured over time, (2)
    identify management reserves in case of contingencies, (3) record all
    indirect costs35 that will be allocated to the work, (4) summarize data
    elements and associated variances through the work breakdown structure to
    support management needs, and (5) develop revised estimates of cost at
    completion based on performance to date.

    Beyond this self-assessment, our review showed that 29 of the 32 criteria
    were not satisfied. For example, the system does not (1) provide for the
    integration of planning, scheduling, budgeting, work authorization, and
    cost accumulation management process; (2) identify physical products,
    milestones, technical performance goals, or other indicators used to
    measure progress; (3) reconcile current budgets to prior budgets in terms
    of changes to the authorized work and internal replanning; and (4) control
    retroactive changes to records. See appendix III for the Navy's complete
    self-assessment and our full analysis of the extent to which the 32
    criteria are satisfied.

    Officials with the program office and the central design agency stated
    that although they chose to use EVM, they are not required by DOD policy
    to do so and, therefore, do not have to comply with the 32 criteria. These
    officials stated that one reason they are not required to use it is
    because the program office and the central design agency are part of the
    same organization (the Space and Naval Warfare Systems Command) and thus a
    formal contract or written agreement between them does not exist. They
    also stated that although the program as a whole exceeds dollar thresholds
    for which EVM is required,36 they have chosen to break the program into
    smaller projects managed on a fiscal year basis, and none of these
    projects individually exceeds either the new or old DOD policy thresholds
    that would require the use of EVM.

    We do not agree that the absence of a contractual relationship or the
    decomposition of the program into small, fiscal year-based projects is a
    valid reason for not effectively implementing EVM. DOD and OMB guidance
    require that the Navy base programmatic decisions on reliable analyses of
    estimated system's costs and expected benefits over the life of the
    program. The program office chose to use EVM as a means to satisfy these
    requirements and to measure progress and identify potential problems
    early, so that they could be effectively addressed. To accomplish this,
    EVM must be performed correctly. By not implementing it correctly on
    NTCSS, the Navy is losing an opportunity to gain the kind of visibility
    into program progress needed to identify problems and risks early and
    better ensure program success. Moreover, by tracking individual projects
    on a yearly basis the program office cannot adequately understand the
    status of the NTCSS program as a whole, which hinders its ability to
    accurately forecast program costs at completion and provide realistic
    schedule projections. In short, without reliable, timely, and auditable
    EVM data, the program office cannot adequately manage technical, cost, and
    schedule risks and problems.

    Two NTCSS Projects Illustrate How EVM Has Been Poorly Implemented

    Two of the individual NTCSS projects for which EVM activities were
    reportedly being performed are (1) 2004 OOMA software development and (2)
    2004 NTCSS hardware installation and integration (for both OOMA and
    Optimized NTCSS). For the OOMA software project, EVM was performed by the
    central design agency and for the NTCSS hardware project it was performed
    by the Space and Naval Warfare Systems Command Systems Center, Charleston.
    On both projects, we found several examples of ineffective EVM
    implementation, including the following:

    o An integrated baseline review was not conducted for either of the
    projects. According to DOD guidance and best practices, an integrated
    baseline review should be conducted as needed throughout the life of a
    program to ensure that the baseline for tracking cost, technical, and
    schedule status reflects (1) all tasks in the statement of work, (2)
    adequate resources in terms of staff and materials to complete the tasks,
    and (3) integration of the tasks into a well-defined schedule. Further,
    program managers are to use cost performance reports that have been
    validated by an integrated baseline review. Without verifying the
    baseline, monthly cost performance reporting, which is to track against a
    set budget and schedule, does not have sufficient meaning or validity.

    o The estimate at completion for the 2004 OOMA software project, which is
    a forecast value expressed in dollars representing the final projected
    costs of the project when all work is completed, showed a negative cost
    for a 6-month period (November 2003 to April 2004). When EVM is properly
    implemented, this amount should include all work completed and always be a
    positive number. The negative estimate at completion for this project
    would mean that the central design agency had incurred a savings rather
    than spending money, even though during that time more than $1.7 million
    had been spent.

    o The schedule performance index for the OOMA software project, which is
    to reflect the critical relationship between the actual work performed
    versus the costs expended to accomplish the work, showed program
    performance during a time when the program office stated no work was being
    performed. Specifically, the reports showed the schedule performance
    fluctuating between $0.21 worth of work performed for every dollar spent
    to more than $3.75 worth of work performed for every dollar spent during a
    time that the program office claims all work was halted. Perfect
    performance would indicate schedule indices equal to 1.0 at best (i.e.,
    for every dollar spent there was 100 percent of the schedule achieved).

    o The estimate at completion for the OOMA hardware installation project
    showed that almost $1 million in installation costs had been removed from
    the total sunk costs, but no reason for doing so was provided in the cost
    performance report.

    o The cost and schedule indices for the OOMA hardware installation project
    showed improbably high program performance during a time when the
    installation schedules and installation budget had been drastically cut
    because OOMA software failed operational testing. Specifically, the
    reports between March 2004 and July 2004 showed the current cost
    performance fluctuating between $0.07 worth of work performed for every
    dollar spent to $8.48 worth of work performed for every dollar spent.

    Navy officials cited several reasons for these shortcomings. For the
    software project, program officials stated that prior to the operational
    testing of OOMA in 2003, the central design agency's implementation of EVM
    was primitive at best and that the resulting data were not usable. They
    also stated that after the project failed operational testing, they did
    not see the value in rebaselining the project and thus all EVM analysis
    was halted. They did, however, continue to invest in OOMA. For the
    hardware installation project, a Charleston Center official responsible
    for developing the installation reports stated that there were problems
    with collecting actual costs because the individuals responsible for doing
    the work were covered by other contracts, and there was no way to ensure
    that the costs were being reported consistently. Regarding the
    approximately $1 million in installation costs that were removed from the
    total sunk costs, this official stated that these costs were erroneously
    charged to this project and were thus removed because they were not part
    of the original plan.

    Ineffective implementation of EVM, as occurred on these two projects,
    precludes NTCSS program officials from having reliable and timely
    information about actual program status and does not provide these
    officials with a sound basis for making informed program decisions.

    The Navy Has Not Adequately Reported NTCSS's Progress and Problems

    One essential aspect of effective program management is complete and
    current reporting by the program office to oversight organizations
    responsible for making decisions regarding the program's future. DOD
    policy recognizes this, stating that the program office is accountable for
    providing credible schedule, performance, and cost reporting information
    to the milestone decision authority. Officials from the NTCSS milestone
    decision authority told us that they relied on the program office to fully
    disclose progress against, and deviations from, program cost, schedule,
    and performance goals. However, the program office has not reported
    consistently or reliably on the program's progress and, as a result, has
    not fully disclosed program status to Navy and DOD oversight authorities
    who are responsible for making proper investment decisions.

    Navy Reporting Requirements for NTCSS Have Changed over the Last Several
    Years

    Since program inception, NTCSS requirements for reporting cost, schedule,
    and performance information have changed. Prior to October 2002, the
    program office was required to comply with applicable DOD acquisition
    policies and guidance.37 This guidance generally required the program
    office to provide oversight organizations with the following three key
    reports:

    o The Acquisition Program Baseline, which describes the program's cost,
    schedule, and performance goals. This baseline document is to be developed
    when the program is initiated, and it is to be updated for each milestone
    review. Within 90 days of a program breach,38 unless the program is back
    within its baseline goals, a new Acquisition Program Baseline is to be
    prepared by the program office and approved by the milestone decision
    authority.

    o The Program Deviation Report, which is to be prepared when the program
    office identifies deviations from the approved Acquisition Program
    Baseline goals. More specifically, when the program office has reason to
    believe that a program breach will occur, it is to immediately notify the
    milestone decision authority. Within 30 days, the program office is to
    inform the milestone decision authority of the reason for the deviation
    and the actions it considers necessary to bring the program back within
    baseline goals.

    o The Defense Acquisition Executive Summary, which is prepared to inform
    the milestone decision authority on the program's progress against cost,
    schedule, and performance goals reflected in the Acquisition Program
    Baseline. Prepared quarterly, the summary is designed to provide an early
    warning to the DOD Chief Information Officer (CIO) and the milestone
    decision authority by identifying existing and potential program problems
    and describing mitigating actions that have been taken.

    Between October 2002 and December 2004, the reporting requirements for the
    program changed.39 As previously discussed, NTCSS was selected by its
    milestone decision authority to participate in the RIT pilot, which was
    aimed at saving time in the acquisition management process by reducing
    traditional DOD reporting and oversight requirements, while still adhering
    to DOD acquisition guidance. Under the RIT pilot, the program office was
    required to prepare the following two monthly electronic reports:

    o The Monthly Acquisition Program Review, which was to assess the current
    health of the program on a monthly basis in such areas as cost and
    schedule performance, testing, funding, and contracting. This report was
    broken into eight parts. According to the program office, the main part
    for NTCSS was the Program Manager Assessment.

    o The Smart Chart, which was to address risks for different projects
    within the program, including a description of the risk, actions taken to
    address the risk, and recommendations for further actions. The Smart Chart
    was also to contain any updates to the Acquisition Program Baseline.

    In short, the RIT reporting was to provide the same information reported
    via the traditional acquisition baseline and the summary report, but it
    was to be more frequent (monthly versus quarterly) and use a different
    format (electronic versus paper). In addition, under the RIT pilot,
    certain acquisition documents, such as acquisition plans, economic
    analyses, requirements documents, and test plans, were to be posted to the
    RIT Web site's electronic library rather than sent in hard copy to the
    program's stakeholders.

    In December 2004, the program office and the milestone decision authority
    agreed to discontinue use of the RIT pilot procedures. In January 2005,
    the reporting requirements reverted to the acquisition policies and
    procedures as prescribed in the updated DOD 5000 series. Currently, the
    program office is required to prepare the summary report quarterly and the
    acquisition baseline as needed. Also, in January 2005, the Navy required
    the program office to begin making entries into the Dashboard. The
    Dashboard, like the summary report, is prepared by the program office on a
    quarterly basis for the milestone decision authority and is to provide an
    assessment of the program in such areas as cost, schedule, and performance
    characteristics.

    The Navy Has Not Satisfied All NTCSS Reporting Requirements

    The program office did not comply with the reporting requirements that
    were in effect during the 27 months of the RIT pilot. Specifically:

    o The Smart Chart was not updated for 19 of the 27 months. Specifically,
    the data were updated eight times between October 2002 and November 2003;
    the data were not updated after November 2003.

    o The Program Manager Assessment was not updated for 11 of the 27 months.
    In addition, the updates were not always made in a timely manner. For the
    16 months that were updated, 7 were done after the month had ended, and
    most of these updates were a month late.

    o Of the 15 essential acquisition documents that the program office
    committed to entering in the RIT electronic library, 10 were not entered.
    For example, the most recent economic analysis and the test and evaluation
    master plan for OOMA were not entered.

    o The Program Deviation Report and Acquisition Program Baseline were not
    prepared in a timely manner. Specifically, in April 2004, the acquisition
    of eNTCSS was cancelled and, in May 2004, OOMA did not pass operational
    testing-two events that caused the related cost and schedule thresholds in
    the Acquisition Program Baseline to be breached. While program officials
    had notified the milestone decision authority of these events via (1)
    e-mail, (2) entries into the Program Manager Assessment on the RIT Web
    site, and (3) briefings, the program office did not prepare a Program
    Deviation Report until about 15 months later. Moreover, this deviation
    report addressed only the OOMA failure, not the cancellation of eNTCSS and
    reprogramming of unexpended eNTCSS funding. In addition, program officials
    have yet to provide us with a new Acquisition Program Baseline to reflect
    the program breach or documentation showing that this revised baseline has
    been approved by the milestone decision authority.

    For the DOD and Navy reporting requirements in effect since January 2005,
    the Navy has satisfied some, but not all, of the reporting requirements.
    For example, the program office has prepared the Dashboard reports
    quarterly as required. However, it has not prepared the Defense
    Acquisition Executive Summary quarterly as required; the first report was
    not prepared until June 2005-6 months after the requirement resumed and
    the report was due.

    Program officials provided various reasons why the required program
    reporting has not occurred. In the case of the Smart Charts and the
    Program Manager Assessment reports, a contractor supporting the Assistant
    Program Manager stated that the data may have been entered into the Web
    site but not properly saved. Regarding the posting of documents into the
    electronic library, an official from the milestone decision authority
    stated that there was no documentation from the Office of the Assistant
    Secretary of Defense for Networks and Information Integration/Chief
    Information Officer that directed which, if any, acquisition documents
    were to be entered into the RIT Web site. Similarly, a contractor
    supporting the Assistant Program Manager stated that the folders in the
    electronic library were established by the Army and thus the Navy was not
    required to use them. However, our review of documentation provided by the
    program office shows that it clearly states which specific documents
    should be included in the electronic library. Regarding the delay in
    preparation of the Program Deviation Report and subsequent Acquisition
    Program Baseline revision, a contractor supporting the Assistant Program
    Manager stated that a new baseline should have been prepared sooner, but
    that this reporting was delayed due to the uncertainty of which reporting
    methods to use after the end of the formal RIT pilot.

    Officials representing the milestone decision authority stated that they
    relied on program office reporting on program status and progress, and
    that they expected the program office to inform them if the program
    exceeded its cost, schedule, and performance thresholds. Without adequate
    reporting, oversight officials were not positioned to effectively execute
    their roles and responsibilities.

    The Navy Has Not Properly Budgeted for NTCSS

    In September 1999, the Navy Comptroller issued guidance directing program
    offices to review their budgets and identify efforts that were being
    improperly funded and to take the steps necessary to realign these funds
    to "Research, Development, Test and Evaluation" as quickly as possible.
    Further, DOD Financial Management Regulation40 requires that IT
    development, test, and evaluation requirements generally be funded in the
    "Research, Development, Test and Evaluation" appropriations. More
    specifically it states that, "The Research, Development, Test and
    Evaluation funds should be used to develop major upgrades increasing the
    performance envelope of existing systems, purchase test articles, and
    conduct developmental testing and/or initial operational test and
    evaluation prior to system acceptance." Similarly, Navy financial
    management policy41 states that, "All costs associated with software
    development/modification efforts that provide a new capability or expand
    the capability of the current software program (i.e., expand the
    performance envelope) are funded in the Research, Development, Test and
    Evaluation appropriation."42

    However, this has not occurred. Since 1997, the program office has not
    identified "Research, Development, Test and Evaluation" funds in five of
    its seven Acquisition Program Baseline documents, three of which were
    prepared after the guidance was issued by the Comptroller of the Navy.
    Instead, the Navy funded these activities primarily out of the "Operations
    and Maintenance," "Other Procurement," and "Ship Construction"
    appropriations. (See table 10.)

    Table 10: Threshold Amounts in NTCSS Acquisition Program Baselines

    Dollars in
        thousands
    Acquisition Date       Operations       Other         Ship       Research,
    program     prepared          and procurement construction    development,
    baseline              maintenance                                 test and
                                                                       evaluation
    Revision 0  March         182,986     199,636       11,683               0
                   1997
    Revision 1  March         257,542     303,565       23,836           3,026
                   1998
    Revision 2  December      223,370     285,550       18,220               0
                   1998
    Revision 3  January       276,100     382,000       27,300               0
                   2001
    Revision 4  January       276,100     382,000       27,300               0
                   2003
    Revision 5  July 2003     276,100     382,000       27,300               0
    Revision 6  January       376,400     346,600       25,700          29,800
                   2004

    Source: Navy.

    Program officials agreed that they have funded NTCSS development
    activities, such as those associated with OOMA, out of the "Operation and
    Maintenance" appropriation rather than the "Research, Development, Test
    and Evaluation" appropriation. A contractor supporting the Assistant
    Program Manager stated that, although they were aware of the Comptroller
    of the Navy's budget guidance, the program office chose not to comply
    because program officials believed in 1999 that the OOMA application,
    which had been under development for 3 years, would pass developmental
    testing and operational testing by 2001. As a result, program officials
    determined that the effort required to reprogram funding from the
    "Operation and Maintenance" appropriation into the "Research, Development,
    Test and Evaluation" appropriation was not warranted. Further, the
    official stated that although OOMA did not pass operational testing in
    2001, the program office did not fund OOMA with "Research, Development,
    Test and Evaluation" funds until 2004 because it continued to consider
    OOMA as being close to becoming operational.

    The lack of proper budgeting for "Research, Development, Test and
    Evaluation" funding has given oversight authorities the misleading
    impression that NTCSS development activities were completed and that the
    system was fully operational. Specifically, officials from the Office of
    the Assistant Secretary of Defense for Networks and Information
    Integration/Chief Information Officer, which was the original NTCSS
    milestone decision authority, stated that since most of the "Research,
    Development, Test and Evaluation" funding appeared to have been spent,
    they concluded that the development portion of NTCSS was essentially
    complete. As a result, these officials stated that they had considered
    taking NTCSS off of the list of programs subject to oversight reviews.
    However, after 9 years and over $79 million in expenditures, the OOMA
    application still has not passed operational testing and thus is still in
    development.

    Navy Oversight of NTCSS Has Not Been Adequate

    DOD and Navy policies task a number of organizations with oversight of IT
    system acquisition and development programs. For example, DOD policy
    states that a milestone decision authority has overall program
    responsibility. In addition, the Navy Chief Information Officer is
    responsible for reviewing programs at certain points in the acquisition
    cycle. Finally, the NTCSS Executive Steering Committee is responsible for
    monitoring the near-term development and evolution of the NTCSS program.
    However, effective oversight by these entities has not occurred. As a
    result, opportunities to address long-standing program weaknesses have
    been missed, and the program has been allowed to proceed virtually
    unchecked.

    The Milestone Decision Authority Has Not Adequately Overseen the Program

    DOD acquisition policy43 states that a milestone decision authority is the
    designated individual with overall responsibility for a program and is to
    ensure accountability and maximize credibility in program cost, schedule,
    and performance reporting. In this role, the milestone decision authority
    is responsible for reviewing the program throughout its acquisition life
    cycle, including: (1) whenever the program reaches a milestone decision
    point; (2) whenever cost, schedule, or performance goals are baselined or
    must be changed; and (3) periodically through review of management
    information such as that found in the Defense Acquisition Executive
    Summary reports.

    However, the Navy milestone decision authority44 has not conducted such
    reviews. Specifically:

    o The NTCSS program has not reached a milestone decision point in over 5
    years. The last such milestone was in April 2000 when the final two NTCSS
    Optimized applications became operational. The next scheduled milestone
    was to be in 2001, but because OOMA operational testing was stopped and
    has yet to be successfully completed, a milestone decision point has yet
    to occur. As a result, there has not been a triggering event that would
    cause the milestone decision authority to formally review the program or
    any of its projects. We discussed the state of NTCSS in March 2005 with
    the milestone decision authority's representatives. In July 2005, the
    authority was briefed by the program office. According to program
    officials, this was the first formal program review to occur since
    termination of the RIT pilot in December 2003. These officials also stated
    that quarterly acquisition team meetings have since resumed-with the first
    meeting having occurred in September 2005 and the next scheduled for
    December 2005-to prepare for the next milestone review of OOMA.

    o The program office notified the milestone decision authority in April
    and June 2004 that OOMA failed operational testing and that eNTCSS was
    cancelled via e-mail, entries into the Program Manager Assessment on the
    RIT Web site, and briefings. According to officials with the milestone
    decision authority, they followed up with the program office and provided
    guidance; however, these events did not trigger a formal program review.

    o The milestone decision authority did not contact the program office to
    inquire as to the reason why monthly reports were not being prepared as
    agreed to after the formal RIT pilot had ended. For example, Smart Charts
    were not prepared after November 2003. However, according to milestone
    decision authority officials, they did not seek an explanation from the
    program office as to why. Milestone decision authority officials told us
    that they were relying on the Dashboard report in order to stay informed
    on the program's progress. However, they did not require the program
    office to begin preparing the Dashboard report until January 2005.

    According to DOD and Navy officials, including officials from the Office
    of the Assistant Secretary of Defense for Networks and Information
    Integration/Chief Information Officer, the Navy milestone decision
    authority, and the program office, NTCSS participation in the RIT pilot
    resulted in disruption of normal oversight activities, which have yet to
    be fully restored. They added that compounding this is the fact that the
    Navy's milestone decision authority's staffing has been reduced in recent
    years. According to these officials, approximately 2 years ago the number
    of full time staff in the Office of the Deputy Assistant Secretary of the
    Navy for Command, Control, Communication, Computers and Intelligence, and
    Space was reduced from 16 to 6 people, and these 6 are responsible for
    reviewing approximately 60 acquisition programs. The officials stated
    that, given the large number of programs and limited staffing, they are
    unable to fully perform oversight activities so they have increasingly
    relied on the program executive office's assistance to perform detailed
    oversight of this program. Without adequate oversight by the milestone
    decision authority, the NTCSS program has been allowed to proceed despite
    the program weaknesses discussed in this report.

    Other Navy Organizations Have Not Conducted Program Oversight

    While the milestone decision authority is the main program oversight
    entity, two other Navy organizations have oversight responsibilities.
    However, these entities also have not performed effective oversight of the
    program. Specifically,

    o Department of Navy CIO is responsible for reviewing programs at certain
    points in the acquisition cycle to ensure, among other things, that
    program goals are achievable and executable and that the program is
    providing value (i.e., producing a positive return-on-investment). Navy
    CIO officials stated that they have overseen NTCSS primarily by reviewing
    the Capital Investment Reports45 prepared by the program office. They
    stated that they have not performed any proactive activities to verify and
    validate the program's status and progress. Instead, they rely on
    information in the Capital Investment Reports, such as economic
    justification; budget information by appropriation type; and cost,
    schedule, progress, and status. However, as was discussed previously, the
    program office does not have or has not reported reliable information on
    these topics.

    o The NTCSS Executive Steering Committee is responsible for establishing
    priorities for NTCSS development and implementation and determining the
    strategic direction of the program. Among other things, it is to meet
    immediately following each major NTCSS program meeting. However, it has
    not met since December 2002, even though the program office convened both
    a Requirements Integrated Product Team meeting and a Forum meeting in
    February 2005. Further, during this period, major setbacks occurred on the
    program, including the failure of OOMA to pass operational testing and the
    cancellation of eNTCSS, which were issues that affected the direction of
    the program and its priorities and thus were consistent with the
    committee's charter. Program officials agreed that the Executive Steering
    Committee has not formally convened during this time frame. However,
    program officials stated that members of the committee informally met to
    discuss and provide advice regarding OOMA concerns, and Navy officials
    higher than the Executive Steering Committee made the decision to cancel
    eNTCSS. Therefore, these officials stated there was no need to formally
    convene an Executive Steering Committee meeting. Program officials stated
    that the Executive Steering Committee will be meeting in January 2006.

    NTCSS Requirements and Test Management Weaknesses Have Contributed to
    Deployment Delays and System Quality Problems

    As we have previously reported,46 the effectiveness of the processes used
    to develop a system is a reliable predictor of the quality of the system
    products produced. Two key system development processes are requirements
    development and management and test management. For the NTCSS application
    currently under development, we found weaknesses with both of these
    process areas. While improvements are planned, until they are implemented
    effectively, the risk of continued NTCSS cost, schedule, and performance
    shortfalls persists.

    The Navy Has Not Adequately Managed Requirements for the NTCSS Application
    Currently Under Development

    Well-defined requirements can be viewed as a cornerstone of effective
    system development and implementation. Accordingly, DOD guidance and
    industry best practices recognize effective requirements development and
    management as an essential system development and acquisition management
    process. For the NTCSS application that is currently under
    development-OOMA-the Navy has not adequately managed its 732 requirements,
    as evidenced by a lack of requirements traceability and prioritization.
    NTCSS program officials told us that NTCSS requirements development
    practices have historically been poor, but that improvements are under
    way. Without effective requirements management, it is likely that the
    Navy's challenges to date in developing NTCSS applications that meet user
    needs on time and on schedule will continue.

    Requirements for OOMA Release 4.10 Were Not Traced

    DOD guidance and industry best practices also recognize the importance of
    requirements traceability.47 The purpose of requirements traceability is
    to ensure that the finished product is compliant with the requirements. To
    do this, the system documentation should be consistent and thus complete,
    allowing for requirements traceability. Requirements traceability involves
    both the alignment and consistency backward to system documentation and
    forward to system design and test documentation.

    OOMA release 4.10 requirements were not traced to an Operational
    Requirements Document. According to DOD guidance,48 an Operational
    Requirements Document translates nonsystem-specific statements of a needed
    operational capability into a set of validated and prioritized user
    requirements. However, the Navy did not develop an Operational
    Requirements Document for NTCSS. As a result, the Navy did not take a
    basic validation step to ensure that the requirements to which it designed
    and built the application were complete and correct. In addition, release
    4.10 requirements were not always traceable to associated system
    specifications. Specifically, we were unable to trace 215 requirements
    found in the system segment specification to the requirements listed in
    the requirements checklist. Requirements should also be traced to test
    cases, but the program office has yet to provide us with the developmental
    test cases used to test the OOMA release 4.10 so that we could verify this
    traceability.

    Program officials acknowledged that release 4.10 requirements were not
    traceable but that improvements are planned for the next OOMA release. We
    found that 97 percent of the OOMA release 5.0 requirements found in the
    system segment specification were traceable to the requirements listed in
    the requirements checklist. However, these documents have yet to be
    approved. Requirements should also be traced to test cases, but the
    program office has yet to provide us with the developmental test cases
    used to test the OOMA release 5.0 so that we could verify this
    traceability. Without this traceability, the Navy has not had a sufficient
    basis for knowing that the scope of its development efforts, including
    testing, provides adequate assurance that applications will perform as
    intended.

    Requirements for OOMA Release 4.10 Were Not Prioritized

    According to published best practices guidance,49 any project with
    resource limitations should establish the relative priorities of the
    requested features or requirements. Prioritization helps the project
    office resolve conflicts, make trade-off decisions among competing
    requirements, and helps to ensure that the delivered system will be
    operationally suitable.

    However, OOMA's approximately 732 requirements have never been
    prioritized, and a program official told us that they are all considered
    to be equally important. This means, for example, that a requirement that
    dictates basic application functionality (e.g., if text can be entered on
    a particular screen) is as important as a requirement addressing safety
    issues that, if not met, could result in the loss of an aircraft or even a
    life.

    This lack of requirements prioritization contributed to release 4.10
    passing developmental testing but failing operational testing. (See later
    section of this report for a detailed discussion of OOMA testing.) A
    developmental testing threshold that the Navy set for release 4.10 was
    that each requirement was to be tested, and 95 percent of the requirements
    had to pass in order for the application to proceed to operational
    testing. For developmental testing of the OOMA release 4.10, 97 percent of
    the requirements passed. Of the 3 percent of the requirements that failed
    this test, some of these deficiencies seriously impacted squadron level
    operations. Further, for operational testing of OOMA release 4.10, 96
    percent of the requirements passed. However, the remaining 4 percent
    contained significant defects. Specifically, the release provided
    inconsistent and inaccurate flight and usage hours, as well as incorrect
    aircraft usage records. According to the Navy's independent operational
    test organization, these deficiencies impacted aircraft and component
    time-based inspection cycles and thus were the basis for the system
    failing operational testing. The Navy has yet to provide evidence that the
    requirements have been prioritized for the OOMA release 5.0.

    The Navy's Developmental Testing for OOMA Has Not Been Effective, but
    Improvements Planned

    Both DOD policy and relevant guidance recognize that effective testing is
    an essential component of system development or acquisition programs.
    Generally, testing can be viewed as consisting of two major phases-a
    developmental phase in which tests are performed to ensure that defined
    system requirements and specifications are met and an operational phase
    that includes tests to determine if the system meets user needs and is
    suitable in an operational environment. The OOMA application has failed
    operational testing twice over the last 4 years reportedly because of
    deficiencies in developmental testing. Program officials attributed
    developmental testing deficiencies to poor software development practices,
    such as the earlier discussed requirements development problems. These
    testing deficiencies can also be attributed to incomplete testing
    documentation. Without effective developmental testing, there is an
    increased risk that application problems will be detected later in the
    system life cycle when they are more expensive and difficult to fix.

    Navy Operational Testing Organization Reported That Developmental Testing
    Has Failed to Identify Problems

    According to DOD guidance and recognized best practices,50 the purpose of
    developmental testing is to provide objective evidence that the product
    (e.g., software module, application, system) satisfies defined
    requirements and performs as intended. Successful completion of
    developmental testing provides the basis for proceeding into operational
    testing to determine whether the integrated product (e.g., application,
    system, system of systems) performs as intended in an operational or
    real-world setting.

    OOMA operational testing results over the last 4 years show that the
    program office's developmental testing efforts have not been effective in
    identifying critical product problems. In particular, the application has
    failed operational testing twice during this time frame and, according to
    an official in the office of the Director of Navy Test and Evaluation and
    Technology Requirements, the failures occurred in operational testing
    because they were not identified during developmental testing. More
    specifically,

    o In March 2001, the program office certified that OOMA release 3.25 had
    passed developmental testing and was ready for operational testing.
    However, 1 month into a scheduled 3-month operational test, the decision
    was made to cease further testing because of significant problems with
    system reliability, data transfer between the application and the
    database, and user training on the application. As a result, the program
    office decertified this release, and the Navy's independent test
    organization recommended discontinuing OOMA deployment.

    o Using results from the failed operational test, the central design
    agency developed release 4.0. In February and March 2002, developmental
    testing of this release was conducted. Test results showed that the
    application was not ready for operational testing because it did not
    satisfy key functional requirements. Subsequently, the central design
    agency incorporated software fixes in release 4.10. In August and
    September 2002, developmental testing was conducted on this release and,
    while a number of deficiencies were verified as fixed, additional
    corrections were needed. From January to June 2003, developmental testing
    was again conducted on OOMA release 4.10.

    o From August 2002 to April 2003, the Naval Audit Service51 reviewed OOMA
    and reported several problems that would affect the application's
    readiness for operational testing. For example, it reported that controls
    to prevent unauthorized access were not in place, Privacy Act information
    was not adequately protected, and backup and recovery procedures were not
    in place. It also reported that the program had not adopted and
    implemented a risk-based system life cycle management approach. According
    to the report, these weaknesses could compromise safety, affect planning,
    and distort readiness reporting if OOMA was implemented throughout the
    Navy.

    o In June 2003, the program office certified OOMA release 4.10 as having
    passed developmental testing and being ready for operational testing. The
    Navy's independent operational test organization subsequently conducted
    testing from August to December 2003 and, in May 2004,52 this organization
    concluded that OOMA was not operationally effective or suitable and thus
    it again failed operational testing. In particular, the operational
    testing results showed that the application was incorrectly calculating
    flight and component usage hours-defects, which according to an official
    in the office of the Director of Navy Test and Evaluation and Technology
    Requirements, could have resulted in the loss of aircraft or life. The
    Assistant Program Manager also told us that release 4.10 did not address
    all of the deficiencies reported by the Naval Audit Service.

    For about a year, the central design agency has been developing and
    testing OOMA release 5.0 to fix the problems found in the prior version.
    The program office expects that this release will be certified as ready
    for operational testing sometime between April and June 2006. In
    preparation for operational testing, the Navy's independent operational
    test organization has been observing OOMA 5.0 developmental testing. A
    memo from this organization states that this release is an improvement
    over the previous releases.

    According to Navy officials, including the NTCSS Assistant Program Manager
    and the official responsible for OOMA developmental testing, previous
    application development practices were poor, which led to testing
    problems. Specifically, they cited poor requirements definitions, poor
    documentation, and concurrent development of application releases as
    examples. Further, Navy officials stated that the central design agency
    has not had a developmental testing lab to facilitate effective testing of
    application components and their integration. To address the poor
    development practices, program officials told us that they are in the
    process of implementing a new system life cycle management process that
    they said incorporates industry best practices, including those related to
    testing. However, the program office has yet to provide us with
    information defining how the practices in this plan will be implemented.
    To address the need for a developmental testing lab, the Naval Air Systems
    Command organization representing NTCSS users recently created a lab to
    strengthen the program's developmental testing capability. According to
    officials associated with the lab, they are finding defects that the
    central design agency should have found.

    It is important that the NTCSS program improve its developmental testing.
    Without effective developmental testing, there is an increased risk that
    system application problems will be detected late in the system life
    cycle, such as during operational testing. Generally, problems discovered
    late in the cycle are more expensive and difficult to fix than those
    discovered early.

    Developmental Test Documentation Has Not Been Adequate, but Improvements
    Planned

    To be effective, testing should be approached in a rigorous and
    disciplined fashion. One aspect of such testing is developing and using
    various testing

    documentation. DOD policy, guidance, and related best practices53 state
    that such documentation includes a test and evaluation master plan for the
    program, as well as documentation that is system product (e.g., module,
    application, system) and test type (e.g., integration, stress, regression,
    developmental) specific. This documentation includes approved test plans,
    test procedures and cases, and test results. According to DOD and other
    guidance, test plans should include, among other things, objectives,
    responsibilities, resources (tools, people, and facilities), schedules,
    and performance and exit criteria; test procedures should include detailed
    test scenarios, test events, steps, inputs, and expected outputs that are
    traced back to requirements. Test results include the test scenarios that
    passed and failed, assessments of deviations from test plans, and the
    extent to which requirements have been met.

    The NTCSS test and evaluation master plan identified, among other things,
    three phases of developmental testing for OOMA release 4.10. However, key
    test documentation for each of these phases was not produced.
    Specifically,

    o For the first phase, a test report was produced that contained detailed
    information on test results, but the program office has yet to provide us
    with a test plan or test procedures.

    o For the second phase, a test report was produced but it only contained
    the number of defects found (organized by severity) and did not include
    any other information on test results. Moreover, the program office has
    yet to provide us with a test plan or test procedures.

    o For the third phase, both a test plan and test report were produced, and
    the plan included the test purpose and objectives, schedule,
    responsibilities, and people resources, while the test report described
    test issues and contained detailed test results. However, the program
    office has yet to provide us with test procedures.

    According to Navy officials, including the Assistant Program Manager and
    officials responsible for developmental testing, the previously mentioned
    poor application development practices contributed to the absence of
    testing documentation. To address these poor practices, the program has
    developed a system life cycle plan that they said incorporates industry
    best practices, including those associated with testing documentation.
    However, the program has yet to provide us with plans defining how these
    practices will be implemented. Moreover, while the plan contains a
    recommended list of testing documents (e.g., test plan, test procedures,
    and test results report), our review of OOMA release 5.0 developmental
    testing documentation shows that not all the documentation is being
    prepared. Specifically, available documentation included an updated test
    and evaluation master plan and two test reports. Documentation not yet
    provided to us included test procedures, which would include documentation
    tracing test cases to requirements.

    The lack of a full set of developmental test documentation is a problem.
    Without such documentation, the adequacy and reliability of developmental
    testing cannot be substantiated, and thus the quality of the associated
    system products is in doubt.

    Central Design Agency Reports Management Improvements are Under Way

    In an effort to improve its performance on NTCSS and other programs,
    central design agency officials told us that they chose to undergo an SEI
    Capability Maturity Model Software Capability Appraisal in July and August
    2005. Carnegie Mellon University's SEI, recognized for its expertise in
    software and system processes, has developed the Capability Maturity
    Model(TM) for Software (SW-CMM)54 to provide guidance on how to gain
    control of their processes for developing and maintaining software and how
    to evolve toward a culture of software engineering and management
    excellence.

    In brief, SW-CMM calls for assessing different process areas-clusters of
    related activities such as project planning, requirements management, and
    quality assurance-by determining whether key practices are implemented and
    whether overarching goals are satisfied. Successful implementation of
    these practices and satisfaction of these goals result in the achievement
    of successive maturity levels. SW-CMM maturity levels range from 1 to 5,
    with level 1 meaning that the process is either characterized as ad hoc
    and occasionally even chaotic with few processes defined and success
    depending on individual effort; level 2 meaning that the process is
    repeatable; level 3 meaning that the process is defined; level 4 meaning
    that the process is managed; and level 5 meaning that the process is
    optimized.

    According to the central design agency they achieved a maturity rating of
    level 3 against the SW-CMM based on 13 process areas, including
    requirements management, software project planning, software project
    tracking and oversight, subcontract management, software quality
    assurance, software configuration management, organizational process
    focus, organizational process definition, training program, integrated
    software management, software product engineering, intergroup
    coordination, and peer reviews. Further, we were told that NTCSS was one
    of the programs included in the review. However, we have yet to receive
    the appraisal report to determine the extent to which the appraisal
    addressed the weaknesses discussed in this report. Nevertheless, our
    research has shown that properly performing such appraisals can be a
    useful starting point for making software and system related development
    improvements.

    Conclusions

    It is unclear whether the Navy's planned investment in NTCSS is warranted.
    Of particular concern is the absence of reliable analysis showing that
    further investment will produce future mission benefits commensurate with
    estimated costs, as well as the void in information concerning whether the
    deployed and operational components of NTCSS are actually producing
    expected value. Compounding this uncertainty is the inherent risk of
    defining and developing NTCSS outside the context of either a well-defined
    DOD or Navy enterprise architecture. Without this information, the Navy
    cannot determine whether NTCSS as defined, and as being developed, is the
    right solution to meet its strategic business and technological needs.

    Even if these uncertainties were to be addressed, and the Navy had the
    data needed to demonstrate that NTCSS plans are the right course of
    action, then the manner in which NTCSS is being defined, developed,
    tested, measured, and overseen would still be of concern. While any one of
    the concerns that we found is troubling, their combination subjects the
    program to an unacceptably high risk of failure. These effects are being
    realized on NTCSS, as evidenced by the cancellation of one system
    component and the repeated failure of another key component to pass
    testing.

    It is extremely important that Navy and DOD authorities responsible and
    accountable for ensuring prudent use of limited resources reassess whether
    allowing NTCSS to continue as planned is warranted. It is also important
    that the decision on how to proceed be based on reliable data about
    program cost, benefits, risk, and status.

    Recommendations for Executive Action

    We recommend that the Secretary of Defense direct the Secretary of the
    Navy to determine if continued investment in NTCSS, as planned, represents
    a prudent use of the department's limited resources. To accomplish this,
    the Secretary of the Navy should direct the program office to take the
    following three actions:

    o collaborate with the Office of the Assistant Secretary of Defense for
    Networks and Information Integration/Chief Information Officer, the Office
    of Program Analysis and Evaluation, and the Naval Cost Analysis Division
    to prepare a reliable economic analysis that encompasses all viable
    alternatives, including the Navy's recent enterprise resource planning
    program;

    o ensure that development of this economic analysis (1) complies with cost
    estimating best practices, including recognition of costs to resolve open
    trouble reports and change proposals, and relevant OMB cost benefit
    guidance and (2) incorporates available data on whether deployed NTCSS
    capabilities are actually producing benefits; and

    o collaborate with the Undersecretary of Defense for Acquisition,
    Technology, and Logistics and the Under Secretary of Defense (Comptroller)
    to ensure that NTCSS is adequately aligned with evolving DOD and Navy
    enterprise architectures.

    In addition, we recommend that the Secretary of Defense direct the
    Secretary of the Navy to present the results of these analyses to the
    Deputy Secretary of Defense, or his designee, and seek a departmental
    decision on how best to proceed with the program. Until this is done, we
    recommend that the Secretary of Defense direct the Secretary of the Navy
    to halt further deployment of NTCSS and to limit future investment in
    already deployed applications to essential operation and maintenance
    activities and only developmental activities deemed essential to national
    security needs.

    If-based on reliable data-a decision is made to continue the NTCSS
    program, we recommend that the Secretary of Defense direct the Secretary
    of the Navy to ensure that the following two actions are taken:

    o the NTCSS program implements effective program management activities,
    including earned value management, requirements development and
    management, and test management; and

    o key stakeholders, such as the central design agency and the
    developmental testing organization, have the people, processes, and tools
    to effectively execute their respective roles and responsibilities.

    Finally, we recommend that Secretary of Defense reestablish the Office of
    the Assistant Secretary of Defense for Networks and Information
    Integration/Chief Information Officer as the milestone decision authority
    and direct the Secretary of the Navy to take steps to ensure that Navy
    oversight entities fulfill their roles and responsibilities on NTCSS,
    including ensuring that reliable program reporting occurs and is acted
    upon.

    Agency Comments and Our Evaluation

    In its written comments on our draft report, signed by the Deputy to the
    Assistant Secretary of Defense for Networks and Information Integration
    (Command, Control, Communications, Intelligence, Surveillance, and
    Reconnaissance and Information Technology Acquisition) and reprinted in
    appendix IV along with our detailed responses, DOD stated that some of our
    findings are valid. For example, it acknowledged that NTCSS was defined
    and implemented without a complete and formal enterprise architecture.
    However, it also commented that our overall findings significantly
    understated and misrepresented the program's level of discipline and
    conformance with applicable guidance and direction. The department added
    that NTCSS "has proven to be the right solution to meet the Navy's
    strategic business and technological needs," and that sound program
    management practices are in place and improving.

    Neither DOD's comment about our overall findings nor its claims about
    NTCSS being the right solution and being effectively managed are
    adequately supported, as evidenced by the numerous factual instances that
    we site in the report where the Navy did not comply with either DOD
    acquisition policies and guidelines or industry best practices.
    Specifically, the report shows that the program's latest economic analysis
    did not provide the Navy a reliable basis upon which to make investment
    decisions. For example, the analysis did not include measurable,
    quantifiable benefits for each alternative, and the cost estimates did not
    meet six of the nine criteria associated with reliable cost estimates. The
    analysis also was not independently reviewed in accordance with DOD
    guidance and the Navy had yet to demonstrate that already deployed NTCSS
    Optimized applications are producing expected benefits. We appropriately
    concluded that the Navy does not know whether the program as defined is
    the right solution to meet the Navy's strategic business and technological
    needs.

    With respect to our recommendations, DOD fully concurred with two of the
    recommendations and partially concurred with the remaining five
    recommendations. The five areas of disagreement, DOD's basis for its
    disagreement, and our response to DOD's position follow.

    First, DOD stated that it does not see merit in conducting a formal
    economic analysis for the NTCSS program that would address all viable
    alternatives because, at this late stage, NTCSS is a "very mature
    program," and the final application (OOMA) is about to be fielded.
    Further, DOD said it saw no merit in seeking Office of Program Analysis
    and Evaluation (PA&E) review of the economic analysis. Instead, it said
    that it will "coordinate" with PA&E in analyzing the relationship of NTCSS
    with other programs that may provide similar functionality and "brief the
    results" to selected stakeholders.

    We do not agree that NTCSS is a "very mature program." In particular, the
    Navy still plans to spend in fiscal years 2006 through 2009 an additional
    $348 million, which is approximately one-third of what has been spent on
    the program to date. Further, there is no evidence to support the claim
    that the OOMA application is about to be fielded. OOMA has failed
    operational testing twice and is not yet fully developed or tested despite
    the Navy's initial plan to field it in 2001. In addition, the Navy's
    stated intention to develop an economic analysis for OOMA only and then,
    separately, prepare an "analysis to determine the relationship" of NTCSS
    and other alternative programs is not consistent with guidance and best
    practice, which advocate basing such analyses on the full scope of the
    planned investment. In addition, the proposal to limit key stakeholders'
    involvement in developing the economic justification to "coordinating" and
    "briefing would be inappropriate." These stakeholders have specific
    expertise and roles relative to economically justifying system investments
    that should be exploited. Until it conducts a complete and disciplined
    analysis of the entire NTCSS program (reviewed and approved by PA&E and
    the Naval Cost Analysis Division) and provides this analysis to all key
    stakeholders, the Navy's investment decisions will continue to be made
    without complete and reliable data.

    Second, the department stated that further deployment of NTCSS should not
    be limited at this time. Nevertheless, it stated that it will use the
    results of the analysis referred to above that depicts NTCSS's
    relationship with other programs to provide appropriate direction to the
    program. We do not agree that development should not be limited and would
    note that the department's own comment acknowledges the need to decide on
    an appropriate direction for the program. In our view, prudent use of
    taxpayer resources warrant both a reliable economic analysis that can be
    used to inform any decision on this direction and fiscal restraint to
    investing until an informed decision can be made.

    Third, DOD said that the Navy does not need to be directed to ensure that
    effective program management activities are implemented because it is
    continuously improving program management activities. Further, DOD stated
    that, although it is not required to implement an earned value management
    system because the individual projects do not meet the dollar threshold
    and there are no formal contract deliverables, it is nevertheless adhering
    to the 32 earned value management criteria set forth in applicable
    standards. The department added that it intends to have the Navy Inspector
    General conduct a separate study to further ensure that the program is
    using the best program management activities.

    We do not agree with these comments. In particular, neither during our
    review nor in its comments did the Navy provide evidence that it has
    implemented effective program management activities or has improvements
    under way. As we state in our report, neither the decomposition of the
    program into small, fiscal year-based projects nor the absence of a
    contractual relationship is a valid reason for not effectively
    implementing earned value management. Further, the Navy's earned value
    management self-assessment showed that it had not adhered to 17 of the 32
    earned value management standards. Without reliable, timely, and auditable
    earned value management data, the program office cannot adequately manage
    technical, cost, and schedule risks and problems.

    Fourth, the department stated that key stakeholders of the NTCSS program
    have the necessary people, processes, and tools to effectively execute
    their respective roles and responsibilities, noting in particular that the
    central design agency has demonstrated its competency and capability and
    was certified as SW-CMM maturity level 3. Nevertheless, the department
    agreed to address this recommendation.

    We support the Navy's stated commitment to address this recommendation. In
    addition, we would note that DOD's comment that stakeholders have the
    resources they need is not consistent with statements from stakeholders
    during our review who said that there were manpower and resource
    shortfalls that affected the oversight and execution of program
    activities. Further, despite the Navy's statement that the central design
    agency achieved SW-CMM maturity level 3, no documentation supporting this
    statement, such as appraisal reports, were provided. Furthermore, Navy
    officials told us that the central design agency did not have a
    development testing lab and was therefore unable to effectively execute
    testing activities.

    Fifth, DOD stated that it is "premature" to reestablish the DOD Chief
    Information Officer as the milestone decision authority as NTCSS
    development is over 95 percent complete. Instead, it stated that existing
    oversight entities would ensure that effective program management and
    reporting was occurring.

    We do not agree that elevating the milestone decision authority at this
    time is premature based on the statement that the program is 95 percent
    complete. For programs that have not been developed using industry best
    practices and technical and management discipline, which is the case for
    NTCSS, such claims of being essentially complete have historically proven
    inaccurate because they are not grounded in reliable performance data.
    Moreover, the Navy still plans to spend $348 million on NTCSS over the
    next three fiscal years. Finally, as stated in our report, the current
    milestone decision authority has allowed the program to operate unchecked
    although a major application has repeatedly failed operational testing,
    and another application was cancelled.

    We are sending copies of this report to interested congressional
    committees; the Director, Office of Management and Budget; the Secretary
    of Defense; the Deputy Secretary of Defense; the Under Secretary of
    Defense for Acquisition, Technology and Logistics; the Under Secretary of
    Defense (Comptroller); the Assistant Secretary of Defense (Networks and
    Information Integration)/Chief Information Officer; the Deputy Assistant
    Secretary of the Navy for Command, Control, Communication, Computers and
    Intelligence, and Space; the Program Executive Office for Command,
    Control, Communication, Computers and Intelligence, and Space within the
    Space and Naval Warfare Systems Command; the Department of the Navy Chief
    Information Officer; and the Office of the Chief of Naval Operations for
    Material Readiness and Logistics Operations. This report will also be
    available at no charge on our Web site at http://www.gao.gov.

    If you or your staff have any questions on matters discussed in this
    report, please contact me at (202) 512-3439 or hiter@gao.gov . Contact
    points for our Offices of Congressional Relations and Public Affairs may
    be found on the last page of this report. GAO staff who made major
    contributions to this report are listed in appendix V.

    Randolph C. Hite Director, Information Technology Architecture     and
    Systems Issues

    Objective, Scope, and MethodologyAppendix I

    Our objective was to determine whether the Naval Tactical Command Support
    System (NTCSS) is being managed according to important aspects of the
    Department of Defense's (DOD) acquisition policies and guidance, as well
    as other relevant acquisition management best practices. To accomplish our
    objective, we focused on the program's (1) economic justification; (2)
    architectural alignment; (3) program management, namely progress
    measurement and reporting, funding disclosure, and oversight; and (4) key
    system development activities, namely requirements development and
    management, test management, and system maturity indicators. For
    requirements and test management, we focused on the one NTCSS application
    that is currently being acquired, known as the Optimized Organizational
    Maintenance Activity (OOMA).

    To determine whether the Navy has economically justified its investment in
    NTCSS, we reviewed the latest economic analysis to determine the basis for
    the cost and benefit estimates and net present value calculations. This
    included evaluating the analysis against DOD and Office of Management and
    Budget (OMB) guidance, as well as relevant best practices.1 It also
    included interviewing program officials, including the Assistant Program
    Manager; the office of the Deputy Assistant Secretary of the Navy for
    Command, Control, Communication, Computers and Intelligence, and Space;
    the Office of Program Analysis and Evaluation; and the Naval Cost Analysis
    Division as to their respective roles, responsibilities, and actual
    efforts in developing and/or reviewing the economic analysis. In addition,
    we also interviewed the Assistant Program Manager and the office of the
    Deputy Assistant Secretary of the Navy for Command, Control,
    Communication, Computers and Intelligence, and Space about the purpose and
    use of the analysis for managing the Navy's investment in the NTCSS
    program including the extent to which measures and metrics showed that
    projected benefits in the economic analysis were actually being realized.

    To determine whether the Navy has aligned NTCSS to either the DOD business
    enterprise architecture2 or a Navy architecture, we relied on our prior
    reports addressing DOD and Navy architecture development and
    implementation efforts, a memo and analysis results on NTCSS's compliance
    with the business enterprise architecture, and documents on the Navy's
    architecture efforts. We also interviewed Navy officials from the program
    office; the office of the Deputy Assistant Secretary of the Navy for
    Command, Control, Communication, Computers and Intelligence, and Space;
    the office of the Navy Research, Development, and Acquisition Chief
    Engineer; and the Office of the Assistant Secretary of Defense for
    Networks and Information Integration/Chief Information Officer about DOD
    and Navy architecture efforts and NTCSS's alignment to them.

    To determine whether the Navy was effectively measuring, reporting, and
    overseeing the program, we did the following:

    o We first asked the central design agency to self-assess their
    satisfaction of 32 best practice criteria regarding their earned value
    management system. Using the results of their self-assessment to target
    our analysis, we then assessed those aspects of the earned value
    management system the self-assessment reported as meeting the criteria, by
    comparing the documentation with relevant DOD guidance and best
    practices.3 We selected these two projects as case studies to determine
    the degree to which earned value management was being implemented. The two
    projects selected were (1) 2004 OOMA software project and (2) 2004 NTCSS
    hardware installation and integration (for both OOMA and Optimized NTCSS).
    We selected these two because they were the projects for which Navy
    provided us the most earned value management related documentation. To
    understand the Navy's reasons why they were not performing certain
    elements of earned value management, we interviewed officials including
    the Assistant Program Manager, and officials at the central design agency
    in Norfolk and the in service engineering agency in Charleston.

    o To assess reporting capabilities, we reviewed program documentation such
    as Acquisition Program Baselines, program deviation reports, and Defense
    Acquisition Executive Summary reports. We also reviewed information and
    documentation on the Rapid Improvement Team pilot Web site including a
    report that assesses the current health of the program on a monthly basis
    and a report that address risks for different projects within the program.

    o To assess compliance with budget policies and guidance, we compared
    NTCSS budget documentation with DOD and Navy financial management policies
    and guidance.

    o To assess oversight of the program, we interviewed the program manager,
    milestone decision authority, functional sponsor, Navy Chief Information
    Officer, and a representative of the program's executive steering
    committee.

    o To determine whether the Navy was effectively managing key system
    development activities, namely requirements management, testing, and
    system maturity indicators, we did the following:

    o To assess requirements development and management capabilities, we
    reviewed program documentation such as the official list of requirements
    and system specifications, and evaluated them against relevant best
    practices4 for several characteristics including traceability and
    prioritization. We attempted to trace requirements to both higher level
    documents and lower level specifications. We also attended the NTCSS Forum
    where requirements were gathered and discussed. We interviewed Navy
    officials such as the Assistant Program Manager, Commanding Officer and
    Executive Director of the central design agency, and the OOMA Functional
    Manager to discuss their roles and responsibilities for developing and
    managing requirements.

    o To assess test management, we reviewed program documentation such as the
    test and evaluation master plan, test plans, test reports, and guidance.
    We then compared these documents with DOD guidance and best practices and
    focused on the effectiveness of developmental testing and the adequacy of
    developmental testing documentation.5 Our review

    o also included an audit report prepared by the Naval Audit Service6 and a
    test report prepared by Navy's independent operational test organization.7
    We interviewed Navy officials such as the Assistant Program Manager,
    Commanding Officer and Executive Director of the central design agency,
    OOMA Functional Manager, and an official in the office of the Director of
    Navy Test and Evaluation and Technology Requirements to discuss their
    roles and responsibilities for test management.

    We did not independently validate information on the program's cost and
    budget or the number of trouble reports and change proposals.

    We conducted our work at DOD headquarters in Arlington, Virginia; at Space
    and Naval Warfare Center, San Diego, California; Space and Naval Warfare
    Systems Center, Norfolk, Virginia; and Naval Air Systems Command in
    Patuxent River, Maryland. We performed our work from September 2004
    through November 2005 in accordance with generally accepted government
    auditing standards.

    Trouble Reports and Change Proposals AssessmentAppendix II

    One indicator of system quality, and thus the effectiveness of the
    development activities used to produce system products, is the volume and
    significance of system problems and change proposals. For the Naval
    Tactical Command Support System (NTCSS), trouble reports are prepared to
    document system defects, and change proposals are prepared to introduce
    additional system functionality. Priority levels are assigned to trouble
    reports and change proposals, with 1 being the most critical and 5 being
    the least critical. Table 11 defines the 5 priority levels.

    Table 11: NTCSS Trouble Report and Change Proposal Priorities

    Priority level Definition
    Priority 1     Prevents the accomplishment of an operational or
                      mission-essential capability; and jeopardizes safety or
                      security.
    Priority 2     Adversely affects the accomplishment of an operational or
                      mission-essential capability, and no work-around solution
                      is available.
    Priority 3     Adversely affects the accomplishment of an operational or
                      mission-essential capability, but a work-around solution is
                      available.
    Priority 4     Results in user/operator inconvenience or annoyance but
                      does not affect a required operational or mission-essential
                      capability.
    Priority 5     Any other effect.

    Source: Navy.

    Available data on the number and significance of open trouble reports and
    change proposals over the last 2 years do not demonstrate that NTCSS
    overall is a high-quality system that is delivering promised or expected
    capabilities. Specifically, the data shows that hundreds of open (yet to
    be resolved) trouble reports and change proposals have continued to affect
    the system.

    Trouble Reports

    The total number of NTCSS priority 1, 2, and 3 trouble reports have stayed
    about the same over the last 2 years-totaling about 700. Of this total,
    NTCSS priority 1 and 2 trouble reports have decreased by 117, with
    priority 1 trouble reports being virtually eliminated. While this is
    movement in a positive direction, about 300 priority 2 trouble reports
    still remain open and these by definition are adversely affecting
    accomplishment of an operational or mission-essential capability. (See
    figs. 1 and 2.)

    Figure 1: Total Number of Open NTCSS and OOMA Priority 1, 2, and 3 Trouble
    Reports

    Figure 2: Open NTCSS Priority 1 and 2 Trouble Reports

    Further, open priority 3 trouble reports have increased during this time
    to about 250 and, given that priority 3s require work-arounds, they
    decrease system capability and performance. Neither the number of priority
    2 trouble reports, which continue to be in the hundreds, nor the upward
    trend in priority 3 trouble reports are indicative of a maturing,
    high-quality system. (See fig. 3.)

    Figure 3: Open NTCSS Priority 3 Trouble Reports

    With respect to the OOMA application in particular, the trend in the
    volume of significant trouble reports shows that this application is
    particularly problematic. Specifically, while priority 1 OOMA open trouble
    reports have been virtually eliminated, the number of open priority 2 OOMA
    trouble reports has risen significantly from 12 to 90 in about the last 2
    years. (See fig. 4.)

    Figure 4: Open OOMA Priority 1 and 2 Trouble Reports

    Moreover, the number of open OOMA priority 3 trouble reports has not
    significantly declined over the last 2 years, with these remaining at
    roughly 160. (See fig. 5.)

    Figure 5: Open OOMA Priority 3 Trouble Reports

    Change Proposals

    The picture for NTCSS change proposals is similar to that for trouble
    reports. Specifically, the total number of open NTCSS priority 1, 2, and 3
    change proposals has increased over the last 2 years-going from about 325
    to 425. Of this total, NTCSS priority 2 change proposals have increased by
    72, with 247 priority 2 proposals still being open. (See figs. 6 and 7.)

    Figure 6: Total Number of Open NTCSS and OOMA Priority 1, 2, and 3 Change
    Proposals

    Figure 7: Open NTCSS Priority 1 and 2 Change Proposals

    Further, NTCSS priority 3 change proposals have increased during this time
    to about 81, and given that priority 3 change proposals require current
    work-arounds, this is not a positive trend. (See fig. 8.)

    Figure 8: Open NTCSS Priority 3 Change Proposals

    With respect to OOMA specifically, the number of open priority 2 change
    proposals has risen slightly from 7 to 12. (See fig. 9.) Similarly, the
    number of open priority 3 change proposals has also increased somewhat
    from 78 to 97. (See fig. 10.) While the number of priority 2 change
    proposals is not large, the trend in these, as well as the trend in the
    more significant number of priority 3 change proposals, is not consistent
    with those of a maturing system.

    Figure 9: Open OOMA Priority 1 and 2 Change Proposals

    Figure 10: Open OOMA Priority 3 Change Proposals

    Earned Value Management AssessmentAppendix III

    Earned value management (EVM) guidance was developed by the American
    National Standards Institute/Electronic Industries Alliance.1 This
    guidance identifies 32 criteria that reliable EVM systems should meet. The
    32 criteria are organized into the following five categories:

    o Organization: Activities that define the scope of the effort and assign
    responsibilities for the work;

    o Planning and budgeting: Activities for planning, scheduling, budgeting,
    and authorizing the work;

    o Accounting: Activities to accumulate the costs of work and material
    needed to complete the work;

    o Analysis: Activities to compare budgeted, performed, and actual costs;
    analyze variances; and develop estimates of final costs; and

    o Revisions and data maintenance: Activities to incorporate internal and
    external changes to the scheduled, budgeted, and authorized work.

    NTCSS central design agency (CDA) officials provided a self-assessment of
    their compliance with each of the criteria, reporting that they met 15 of
    the 32 criteria (see table 12). Using the results of their self-assessment
    to target our analysis, we then assessed those aspects of the EVM system
    the self-assessment reported as meeting the criteria, by comparing the
    documentation with relevant Department of Defense (DOD) guidance and best
    practices.2 Our assessment indicates that the NTCSS program satisfied two,
    and partially satisfied one, of the 32 criteria (see table 12).3

    Table 12: Navy Satisfaction of EVM Criteria

          Criteriaa         Definitions        Self-        GAO       GAO analysis
                                             assessment  assessment
      Organization
      Define the        The work breakdown   Yes         Yes         The EVM reports
      authorized work   structure is a                               for the OOMA
      elements for the  direct                                       software
      program. A work   representation of                            development
      breakdown         the work scope in                            project and the
      structure,        the project,                                 NTCSS hardware
      tailored for      documenting the                              installation
      effective         hierarchy and                                project had a
      internal          description of tasks                         work breakdown
      management        to be performed and                          structure.
      control, is       the relationship to
      commonly used in  the product
      this process.     deliverables. The
                        work breakdown
                        structure breaks
                        down all authorized
                        work scope into
                        appropriate elements
                        for planning,
                        budgeting,
                        scheduling, cost
                        accounting, work
                        authorization,
                        measuring progress,
                        and management
                        control. It also
                        ensures the
                        statement of work is
                        entirely captured
                        and allows for
                        integration of
                        technical, schedule,
                        and cost
                        information.
      Identify the      The organizational   Yes         No          CDA officials
      program           structure identifies                         have yet to
      organizational    the organization                             provide
      breakdown         responsible for each                         documentation
      structure,        segment of work,                             to demonstrate
      including the     including                                    satisfaction of
      major             subcontracted and                            this criterion.
      subcontractors    intra-organizational                         Such
      responsible for   effort. In order to                          documentation
      accomplishing the meet this guideline,                         includes an
      authorized work,  objective evidence                           organizational
      and define the    requires a work                              breakdown
      organizational    breakdown structure                          structure with
      elements in which intersection with an                         detail
      work will be      organizational                               regarding
      planned and       breakdown structure.                         subcontractors.
      controlled.
      Provide for the   The integration of   Yes         No          The CDA has yet
      integration of    planning,                                    to provide
      the company's     scheduling,                                  documentation
      planning,         budgeting, work                              to demonstrate
      scheduling,       authorization, and                           satisfaction of
      budgeting, work   cost accumulation                            this criterion.
      authorization,    management processes                         Such
      and cost          provides the                                 documentation
      accumulation      capability for                               includes copies
      processes with    establishing the                             of master,
      each other and,   performance                                  intermediate,
      as appropriate,   measurement                                  and detailed
      the program work  baseline,                                    schedules;
      breakdown         identifying work                             operational
      structure and the progress, and                                schedules;
      program           collecting of actual                         control account
      organizational    costs for management                         plans;
      structure.        analysis and                                 performance
                        corrective actions.                          reports by work
                                                                     breakdown
                                                                     structure and
                                                                     organizational
                                                                     breakdown
                                                                     structure;
                                                                     responsibility
                                                                     assignment
                                                                     matrix;
                                                                     statement of
                                                                     work; work
                                                                     authorization
                                                                     documents; and
                                                                     work breakdown
                                                                     structure and
                                                                     organizational
                                                                     breakdown
                                                                     structure
                                                                     documentation.
      Identify the      Visibility into      No          No          We did not
      company           direct and indirect                          analyze this
      organization or   costs is essential                           criterion
      function          for successful                               because it was
      responsible for   management of a                              self-assessed
      controlling       project. Therefore,                          by the CDA as
      overhead          project managers                             not being met.
      (indirect costs). should clearly
                        identify managers
                        who are responsible
                        for controlling
                        indirect costs,
                        including overhead,
                        burden, general and
                        administrative
                        costs, and who has
                        authority to approve
                        expenditure of
                        resources. They
                        should also document
                        the process for
                        management and
                        control of indirect
                        costs.
      Provide for       The integration of   No          No          We did not
      integration of    the work breakdown                           analyze this
      the program work  structure and                                criterion
      breakdown         organizational                               because it was
      structure and the breakdown structure                          self-assessed
      program           establishes where                            by the CDA as
      organizational    the performance                              not being met.
      structure in a    measurement
      manner that       necessary for
      permits cost and  project management
      schedule          is performed. This
      performance       intersection results
      measurement by    in designation of a
      elements of       focal point for
      either or both    management control
      structures, as    (the control account
      needed.           manager). It is also
                        the initiation point
                        for work
                        authorization,
                        performance
                        management, and
                        performance
                        measurement. The
                        control account
                        manager identifies
                        the plan for work
                        task accomplishment,
                        including defining
                        the effort required,
                        cost elements
                        (labor, material,
                        etc.), and the
                        resources required
                        to do the job.
      Planning and
      budgeting
      Schedule the      The scheduling of    Yes         Yes         Detailed
      authorized work   authorized work                              schedule
      in a manner that  facilitates                                  documents for
      describes the     effective planning,                          both projects
      sequence of work  reporting, and                               describe the
      and identifies    forecasting, which                           sequence and
      significant task  is critical to the                           interdependence
      interdependencies success of all                               of work
      required to meet  projects. An                                 relative to
      the program       integrated network                           project
      requirements.     scheduling system                            requirements.
                        has distinct tasks
                        that can be
                        summarized by work
                        breakdown structure
                        and organizational
                        breakdown structure
                        identifiers to track
                        progress and measure
                        performance.
      Identify physical Objective indicators Yes         No          The metrics in
      products,         enable measurement                           the NTCSS
      milestones,       of work                                      hardware
      technical         accomplished,                                installation
      performance       thereby allowing                             project reports
      goals, or other   accurate comparison                          contained
      indicators that   to planned work.                             unexpectedly
      will be used to   Meaningful                                   and
      measure progress. performance metrics                          unrealistically
                        enable better                                large
                        management insight                           improvements in
                        and decision making,                         performance
                        allowing maximum                             that were not
                        time for management                          explained. In
                        action to keep the                           addition, the
                        project on plan.                             program office
                                                                     told us that
                                                                     the measurement
                                                                     data for the
                                                                     OOMA software
                                                                     project is
                                                                     distorted due
                                                                     to numerous
                                                                     baseline
                                                                     changes and
                                                                     requirements
                                                                     changes.
                                                                     Satisfying this
                                                                     criterion
                                                                     requires valid
                                                                     data.
      Establish and     The assignment of    No          No          We did not
      maintain a        budgets to scheduled                         analyze this
      time-phased       segments of work                             criterion
      budget baseline,  produces a plan                              because it was
      at the control    against which actual                         self-assessed
      account level,    performance can be                           by the CDA as
      against which     compared. This is                            not being met.
      program           called the
      performance can   performance
      be measured.      measurement
      Budget for        baseline. The
      far-term efforts  establishment,
      may be held in    maintenance, and use
      higher-level      of the performance
      accounts until an measurement baseline
      appropriate time  are indispensable to
      for allocation at effective program
      the control       management.
      account level.
      Initial budgets
      established for
      performance
      measurement will
      be based on
      either internal
      management goals
      or the external
      customer
      negotiated target
      cost, including
      estimates for
      authorized but
      undefinitized
      work. On
      government
      contracts, if an
      over-target
      baseline is used
      for performance
      measurement
      reporting
      purposes, prior
      notification must
      be provided to
      the customer.
      Establish budgets An essential part of No          No          We did not
      for authorized    project planning and                         analyze this
      work with         establishing a                               criterion
      identification of performance                                  because it was
      significant cost  measurement baseline                         self-assessed
      elements (e.g.,   is the establishment                         by the CDA as
      labor and         of budgets for all                           not being met.
      material) as      work authorized.
      needed for        Identification of
      internal          the budget cost
      management and    elements documents
      for control of    the required
      subcontractors.   resources and
                        integrates the work
                        scope with the
                        performing
                        organization.
      To the extent it  The effort contained Yes         No          The CDA has yet
      is practical to   within a control                             to provide
      identify the      account is                                   documentation
      authorized work   distributed into                             to demonstrate
      in discrete work  either work packages                         satisfaction of
      packages,         or planning                                  this criterion.
      establish budgets packages. Work                               Such
      for this work in  packages are single                          documentation
      terms of dollars, tasks, assigned to a                         includes
      hours, or other   performing                                   control account
      measurable units. organization for                             plans divided
      Where the entire  completion, and                              into work and
      control account   should be natural                            planning
      is not subdivided subdivisions of                              packages, or
      into work         control account                              control account
      packages,         effort resulting in                          schedules and
      identify the      a definable end                              time-phased
      far-term effort   product or event.                            budgets.
      in larger         Budgets established
      planning packages at the work package
      for budget and    level provide the
      scheduling        detail for effective
      purposes.         execution of the
                        baseline plan. This
                        approach provides
                        meaningful product
                        or
                        management-oriented
                        events for
                        performance
                        measurement.
      Provide that the  The integrity of the No          No          We did not
      sum of all work   performance                                  analyze this
      package budgets,  measurement baseline                         criterion
      plus planning     is maintained when                           because it was
      package budgets   the budget of the                            self-assessed
      within a control  control account                              by the CDA as
      account, equals   equals the sum of                            not being met.
      the control       its work and
      account budget.   planning package
                        budgets. This
                        prevents duplicate
                        recording of
                        budgets.
      Identify and      Meaningful events    No          No          We did not
      control the level are critical for                             analyze this
      of effort         performance                                  criterion
      activity by       measurement.                                 because it was
      time-phased       Measurement of level                         self-assessed
      budgets           of effort activity                           by the CDA as
      established for   provides no                                  not being met.
      this purpose.     visibility into
      Only that effort  actual performance.
      that is           Level of effort
      unmeasurable or   activity is defined
      for which         as having no
      measurement is    measurable output or
      impractical may   product at the work
      be classified as  package level and,
      level of effort.  therefore, must be
                        limited to avoid
                        distorting project
                        performance data.
      Establish         Indirect costs are   No          No          We did not
      overhead budgets  for common                                   analyze this
      for each          activities that                              criterion
      significant       cannot be                                    because it was
      organizational    specifically                                 self-assessed
      component of the  identified with a                            by the CDA as
      company for       particular project                           not being met.
      expenses that     or activity and
      will become       should typically be
      indirect costs.   budgeted and
      Reflect in the    controlled
      program budgets,  separately at the
      at the            functional or
      appropriate       organization manager
      level, the        level. It is
      amounts in        important to have an
      overhead accounts indirect budgeting
      that are planned  and forecasting
      to be allocated   process because
      to the program as indirect costs
      indirect costs.   account for a major
                        portion of the cost
                        of any project. As
                        such, the budgetary
                        control and
                        management of this
                        category cannot be
                        overlooked or
                        minimized.
      Identify          Project managers     No          No          We did not
      management        need to realize the                          analyze this
      reserves and      performance                                  criterion
      undistributed     measurement baseline                         because it was
      budget.           planning process                             self-assessed
                        contains risk and                            by the CDA as
                        identify a                                   not being met.
                        management reserve
                        contingency for
                        unplanned activity
                        within the project
                        scope.
      Provide that the  A project baseline   No          No          We did not
      program target    that reflects the                            analyze this
      cost goal is      common agreement                             criterion
      reconciled with   between the two                              because it was
      the sum of all    parties provides a                           self-assessed
      internal program  common reference                             by the CDA as
      budgets and       point for progress                           not being met.
      management        assessment. It
      reserves.         provides recognition
                        of contractual
                        requirements and
                        precludes
                        unauthorized changes
                        to the performance
                        measurement
                        baseline.
      Accounting
      considerations
      Record direct     A project            No          No          We did not
      costs in a manner cost-charging                                analyze this
      consistent with   structure                                    criterion
      the budgets in a  established in the                           because it was
      formal system     accounting system                            self-assessed
      controlled by the ensures that actual                          by the CDA as
      general books of  direct costs are                             not being met.
      account.          accumulated and
                        reported in a manner
                        consistent with the
                        way the work is
                        planned and
                        budgeted.
      When a work       Actual costs need to No          No          We did not
      breakdown         be available at all                          analyze this
      structure is      levels of the work                           criterion
      used, summarize   breakdown structure                          because it was
      direct costs from to support project                           self-assessed
      control accounts  management with                              by the CDA as
      into the work     performance                                  not being met.
      breakdown         measurement data.
      structure without Cost collection
      allocation of a   accounts mapped to
      single control    the work breakdown
      account to two or structure ensure
      more work         performance
      breakdown         measurement data
      structure         integrity.
      elements.
      Summarize direct  To ensure            No          No          We did not
      costs from the    performance                                  analyze this
      control accounts  measurement data                             criterion
      into the          integrity, actual                            because it was
      contractor's      costs need to be                             self-assessed
      organizational    available at all                             by the CDA as
      elements without  levels of the                                not being met.
      allocation of a   organizational
      single control    breakdown structure.
      account to two or
      more
      organizational
      elements.
      Record all        All indirect costs   No          No          We did not
      indirect costs    should be recorded                           analyze this
      that will be      in the accounting                            criterion
      allocated to the  system. Allocating                           because it was
      project.          indirect costs to                            self-assessed
                        the appropriate                              by the CDA as
                        direct costs assures                         not being met.
                        that all projects
                        benefiting from
                        indirect costs
                        receive their fair
                        share.
      Identify unit     A manufacturing      Yes         No          The CDA has not
      costs, equivalent accounting system                            yet provided
      unit costs, or    capable of isolating                         documentation
      lot costs when    unit and lot costs                           to demonstrate
      needed.           in a production                              satisfaction of
                        environment allows                           this criterion.
                        the flexibility to                           Such
                        plan, measure                                documentation
                        performance, and                             includes a
                        forecast in a more                           manufacturing
                        efficient way when                           resource
                        there are multiple                           planning
                        projects in the same                         project cost
                        production line.                             collection
                                                                     structure or an
                                                                     enterprise
                                                                     resource
                                                                     planning system
                                                                     that supports
                                                                     the
                                                                     identification
                                                                     of unit costs,
                                                                     equivalent unit
                                                                     costs, or lot
                                                                     costs when
                                                                     needed
                                                                     including
                                                                     differentiation
                                                                     of work in
                                                                     process.
      For EVM, the      Material items       No          No          We did not
      material          consumed in the                              analyze this
      accounting system production of                                criterion
      will provide (1)  project deliverables                         because it was
      accurate cost     are accounted for                            self-assessed
      accumulation and  and progress is                              by the CDA as
      assignment of     measured at the                              not being met.
      costs to control  point most closely
      accounts in a     aligned to the
      manner consistent actual consumption.
      with the budgets  Material accounting
      using recognized, systems should
      acceptable,       adhere to these
      costing           three
      techniques; (2)   characteristics: (1)
      cost performance  the material
      measurement at    accounting system
      the point in time provides full
      most suitable for accountability and
      the category of   effective
      material          measurement of all
      involved, but no  material purchased;
      earlier than the  (2) material costs
      time of progress  should be accurately
      payments or       charged to control
      actual receipt of accounts using
      material; and (3) recognized,
      full              acceptable costing
      accountability of techniques; and (3)
      all material      when necessary, the
      purchased for the use of estimated
      program,          actual costs to
      including the     ensure accurate
      residual          performance
      inventory.        measurement should
                        be used.
      Analysis and
      management
      reports
      At least on a     Visibility into      Yes         No          In order to
      monthly basis,    project performance                          produce
      generate the      helps the project                            reliable and
      following         manager focus                                accurate
      information at    resources on those                           variance
      the control       areas in need of                             reports, many
      account and other attention. Accurate                          of the
      levels as         and reliable EVM                             aforementioned
      necessary for     data supports                                criteria that
      management        management control                           our analysis
      control using     needs by allowing                            showed that the
      actual cost data  the project manager                          CDA did not
      from, or          to identify root                             perform must be
      reconcilable      causes for variances                         satisfied.
      with, the         and establish                                Therefore, this
      accounting        actions to minimize                          criterion is
      system: (1)       impact at the                                not being
      comparison of the control account                              satisfied.
      amount of planned level.
      budget and the
      amount of budget
      earned for work
      accomplished
      (this comparison
      provides the
      schedule
      variance) and (2)
      comparison of the
      amount of the
      budget earned and
      the actual
      (applied where
      appropriate)
      direct costs for
      the same work
      (this comparison
      provides the cost
      variance).
      Identify, at      The analysis of      Yes         No          The metrics in
      least monthly,    deviations from plan                         the NTCSS
      the significant   for both schedule                            hardware
      differences       and cost at least                            installation
      between both      monthly provides                             project reports
      planned and       management at all                            contained
      actual schedule   levels the ability                           unexpectedly
      performance and   to rapidly and                               and
      planned and       effectively                                  unrealistically
      actual cost       implement corrective                         large
      performance and   actions with an                              improvements in
      provide the       understanding of the                         performance
      reasons for the   project risk and                             that were not
      variances in the  causes of risk.                              explained. In
      detail needed by                                               addition, the
      program                                                        program office
      management.                                                    told us that
                                                                     the measurement
                                                                     data for the
                                                                     OOMA software
                                                                     project is
                                                                     distorted due
                                                                     to numerous
                                                                     baseline
                                                                     changes and
                                                                     requirements
                                                                     changes.
                                                                     Satisfying this
                                                                     criterion
                                                                     requires valid
                                                                     data.
      Identify budgeted Ongoing indirect     No          No          We did not
      and applied (or   cost analysis                                analyze this
      actual) indirect  provides visibility                          criterion
      costs at the      into potential                               because it was
      level and         indirect cost                                self-assessed
      frequency needed  overruns and the                             by the CDA as
      by management for opportunity to                               not being met.
      effective         develop and
      control, along    implement management
      with the reasons  action plans to meet
      for any           project objectives.
      significant
      variances.
      Summarize the     Variances provide an No          No          We did not
      data elements and understanding of the                         analyze this
      associated        conditions, allowing                         criterion
      variances through the project manager                          because it was
      the program       to properly allocate                         self-assessed
      organization      available resources                          by the CDA as
      and/or work       to mitigate project                          not being met.
      breakdown         risk. They also
      structure to      identify significant
      support           problem areas from
      management needs  all levels of the
      and any customer  organization and
      reporting         project scope of
      specified in the  work, derived from
      project.          the same data
                        sources. Thus,
                        variances provide
                        valuable management
                        information.
      Implement         Earned value data    Yes         No          The metrics in
      managerial        must be utilized by                          the NTCSS
      actions taken as  all levels of                                hardware
      the result of     management for                               installation
      earned value      effective project                            project reports
      information.      execution. Because                           contained
                        of this, the data                            unexpectedly
                        produced by the EVM                          and
                        system must be                               unrealistically
                        available to                                 large
                        managers on a timely                         improvements in
                        basis and must be of                         performance
                        sufficient quality                           that were not
                        to ensure that                               explained. In
                        effective management                         addition, the
                        decisions can be                             program office
                        made as a result of                          told us that
                        its analysis.                                the measurement
                                                                     data for the
                                                                     OOMA software
                                                                     project is
                                                                     distorted due
                                                                     to numerous
                                                                     baseline
                                                                     changes and
                                                                     requirements
                                                                     changes.
                                                                     Satisfying this
                                                                     criterion
                                                                     requires valid
                                                                     data.
      Develop revised   Estimates at         No          No          We did not
      estimates of cost completion based on                          analyze this
      at completion     predictive                                   criterion
      based on          performance measures                         because it was
      performance to    increase the                                 self-assessed
      date, commitment  probability that the                         by the CDA as
      values for        project can be                               not being met.
      material, and     executed within the
      estimates of      reported estimates
      future            at completion. When
      conditions.       estimates at
      Compare this      completions are
      information with  analyzed at least
      the performance   monthly and updated
      measurement       as required, the
      baseline to       robustness of the
      identify          financial reporting
      variances at      requirements is
      completion        enhanced, thereby
      important to      reducing the
      company           potential for
      management and    surprises. Monthly
      any applicable    estimates at
      customer          completion reviews
      reporting         are essential for
      requirements,     management decisions
      including         including the
      statements of     planning of project
      funding           future funding
      requirements.     requirements.
      Revisions and
      data maintenance
      Incorporate       The incorporation of Yes         No          The CDA has yet
      authorized        authorized changes                           to provide
      changes in a      in a timely manner                           documentation
      timely manner,    maintains the                                to demonstrate
      recording the     integrity of the                             satisfaction of
      effects of such   performance                                  this criterion.
      changes in        measurement baseline                         Such
      budgets and       and thus its                                 documentation
      schedules. In the effectiveness as a                           includes change
      directed effort   baseline against                             control logs
      prior to          which to manage and                          and work
      negotiation of a  control performance.                         authorization
      change, base such                                              documents.
      revisions on the
      amount estimated
      and budgeted to
      the program
      organizations.
      Reconcile current Budget changes       Yes         No          The CDA has yet
      budgets to prior  should be controlled                         to provide
      budgets in terms  and understood in                            documentation
      of changes to the terms of scope,                              to demonstrate
      authorized work   resources, and                               satisfaction of
      and internal      schedule, and that                           this criterion.
      replanning in the budgets should                               Such
      detail needed by  reflect current                              documentation
      management for    levels of authorized                         includes change
      effective         work. Furthermore,                           documents or
      control.          budget revisions                             change control
                        should be traceable                          logs.
                        to authorized
                        contractual targets
                        and control account
                        budgets.
      Control           Retroactive changes  Yes         No          The CDA has yet
      retroactive       to the baseline may                          to provide
      changes to        mask variance trends                         documentation
      records           and prevent use of                           to demonstrate
      pertaining to     the performance data                         satisfaction of
      work performed    to project estimates                         this criterion.
      that would change of cost and schedule                         Such
      previously        at completion.                               documentation
      reported amounts  Retroactive budget                           includes change
      for actual costs, adjustments may                              control logs or
      earned value, or  delay visibility of                          approved
      budgets.          overall project                              retroactive
      Adjustments       variance from plan,                          change
      should be made    thus reducing the                            controls.
      only for          alternatives
      correction of     available to
      errors, routine   managers for project
      accounting        redirection or
      adjustments,      termination.
      effects of
      customer or
      management
      directed changes,
      or to improve the
      baseline
      integrity and
      accuracy of
      performance
      measurement data.
      Prevent revisions Changes made outside Yes         No          The CDA has yet
      to the program    the authorized                               to provide
      budget except for baseline control                             documentation
      authorized        processes compromise                         to demonstrate
      changes.          the integrity of                             satisfaction of
                        performance trend                            this criterion.
                        data and delay                               Such
                        visibility into                              documentation
                        overall project                              includes change
                        variance from plan.                          control logs,
                                                                     control
                                                                     accounts, and
                                                                     work package
                                                                     plans.
      Document changes  By ensuring that            Yes     Partial  We were
      to the            budget and schedule                          provided
      performance       revisions are                                documentation
      measurement       documented and                               showing eight
      baseline.         traceable, the                               baseline
                        integrity of the                             changes for the
                        performance                                  NTCSS hardware
                        measurement baseline                         installation
                        is maintained and                            project.
                        can be verified. The                         However, the
                        performance                                  program office
                        measurement baseline                         told us that
                        should reflect the                           the EVM data
                        most current plan                            for the OOMA
                        for accomplishing                            software
                        the effort.                                  project is
                        Authorized changes                           distorted due
                        should be quickly                            to numerous
                        recorded in the                              baseline
                        system and                                   changes and
                        incorporated into                            requirements
                        all relevant                                 changes.
                        planning. Planning
                        and authorization
                        documents must also
                        be updated
                        accordingly prior to
                        commencement of new
                        work.
      Number satisfied                       15          2
      Number partially                       0           1
      satisfied
      Number not                             17          29
      satisfied
      Total                                  32          32

    Sources: Navy CDA self-assessment and GAO analysis of Navy provided data.

    aBased on the National Defense Industrial Association Program Management
    Systems Committee Intent Guide (January 2005).

    Comments from the Department of DefenseAppendix IV

    The following are GAO's comments on the Department of Defense's letter
    dated November 23, 2005.

    1.See the Agency Comments and Our Evaluation section of this report.

    2.We disagree. Our report contains numerous instances where the Navy did
    not comply with either DOD acquisition policies and guidelines or industry
    best practices, in the areas of (1) economic justification; (2)
    architectural alignment; (3) project management, including progress
    measurement and reporting, funding disclosure, and oversight activities;
    and (4) system development, including requirements management and testing.
    Moreover, the Navy has not provided any evidence to demonstrate that our
    report is incorrect with respect to the level of program discipline and
    conformance with applicable policy and guidance in the areas that we
    reviewed.

    3.We disagree. Knowing that NTCSS is the right solution to meet the Navy's
    strategic business and technological needs would require that a frame of
    reference articulating these needs be available as a point of comparison.
    Such a frame of reference is an enterprise architecture. However, the Navy
    stated the system was defined and implemented without a complete and
    formal enterprise architecture. Our experience with federal agencies has
    shown that investing in an information technology solution without
    defining the solution in the context of an architecture often results in
    systems that are duplicative, not well integrated, and unnecessarily
    costly to maintain and interface. In addition, in February 2005, key
    program stakeholders and representatives from user organizations
    questioned whether NTCSS as defined was the right solution to cost
    effectively meet users' needs. At that time, program officials stated
    their intent to develop a new economic analysis to gather the information
    needed to determine whether to continue investing in NTCSS. In November
    2005, program officials told us that they no longer planned to develop
    this economic analysis. Without a well-defined architecture and a reliable
    economic analysis, the Navy cannot be sure that NTCSS is the right
    solution.

    4.See comment 2.

    5.We acknowledge DOD's comment but would note that it is contrary to
    statements made to us during the audit. For example, officials with the
    milestone decision authority stated that, due to staffing reductions, the
    office was unable to fully perform oversight activities and has had to
    delegate completion of these activities. Also, Naval Cost Analysis
    Division officials stated that they only review cost estimates that are
    prepared for milestone reviews because staffing limitations do not permit
    them to review all cost estimates. Further, Navy officials stated that the
    central design agency was unable to effectively execute testing activities
    because it did not have a development testing lab.

    6.We disagree with this approach because its scope is narrower than our
    recommendation. Specifically, we recommended that the Navy develop a
    reliable economic analysis of the NTCSS program that includes all viable
    alternatives, including the Navy's Enterprise Resource Planning program.
    DOD acquisition policy and guidance provide detailed instructions on how
    economic analyses should be performed to obtain information that is
    critical for decisions regarding investments of scarce resources. Without
    such information, Navy risks that its continued investment in the system
    may not be justified.

    7.We disagree. With respect to the statement that NTCSS is a "very mature
    program," NTCSS has been under development for about 10 years at a cost of
    about $1.1 billion, and the Navy plans to spend an additional $348 million
    between fiscal years 2006 and 2009. Further, as appendix II of our report
    shows, there are hundreds of open trouble reports and change proposals
    that need to be addressed before the system can deliver promised or
    expected capabilities. In addition, should the OOMA application pass
    operational testing and be fielded, there are over 200 sites where the
    necessary hardware must be installed and related training must occur.
    These two efforts will require a significant investment of time and
    resources, and it is therefore critical that the Navy ensure that NTCSS is
    the proper system before investing additional funds. With respect to the
    statement that "the final application is about to fielded," there is no
    evidence to support this. Since its originally planned fielding date of
    2001, OOMA has failed operational testing twice, and the application is
    still under development. Therefore, it is premature to assert that the
    application will soon pass developmental and follow-on operational
    testing.

    8.See comment 6. Further, we disagree with the proposal to limit key
    stakeholders' involvement in developing the economic justification to
    "coordinating" and "briefing." These stakeholders have specific expertise
    and roles relative to economically justifying system investments that
    should be exploited. Until it conducts a complete and disciplined analysis
    of the entire NTCSS program (reviewed and approved by the Office of
    Program Analysis and Evaluation and the Naval Cost Analysis Division) and
    provides this analysis to all key stakeholders, the Navy's investment
    decisions will continue to be made without complete and reliable data.

    9.We disagree. As discussed in our report, the 2004 economic analysis did
    not adhere to five of eight criteria elements contained in the Office of
    Management and Budget Circulars A-94 and A-11.

    10.We disagree. The 2004 economic analysis that the Navy provided us
    focused on three fielding alternatives for the NTCSS program, not just the
    OOMA application. The Navy did not provide a 2004 economic analysis for
    just OOMA as the final NTCSS application.

    11.We disagree. As stated in our report, officials from the Office of
    Program Analysis and Evaluation and the Naval Cost Analysis Division told
    us that they did not review the 2004 NTCSS economic analysis.

    12.See comment 10.

    13.We agree that the Navy ERP program did not exist when the original
    NTCSS analysis of alternatives was conducted. However, the Navy ERP
    program was initiated in 1998 and therefore did exist when the Navy
    conducted subsequent analysis of alternatives.

    14.See comment 9.

    15.We do not question whether these annual reviews occurred and what
    resulted from them. However, the point in our report is that NTCSS has not
    been defined and developed in the context of a DOD or Navy enterprise
    architecture because a well-defined version of either has not existed to
    guide and constrain the program. As a result, meaningful analysis showing
    how NTCSS aligns to evolving DOD and Navy architecture efforts could not
    be produced. This means that the Navy does not have a sufficient basis for
    knowing if NTCSS, as defined, properly fits within the context of future
    DOD and Navy business operational and technological environments.

    16.We disagree. Our recommendation to limit further deployment of NTCSS is
    a way of ensuring that the Navy takes a "strategic pause" while it takes
    the time to ensure that decisions regarding future investment are made
    using reliable information, which our report shows has not historically
    been the case. As long as the Navy is not appropriately limiting work on
    NTCSS, it is continuing to invest resources without having justified doing
    so.

    17.See comment 6.

    18.See comment 2.

    19.We disagree. As we state in our report, neither the decomposition of
    the program into small, fiscal year-based projects nor the absence of a
    contractual relationship is a valid reason for not effectively
    implementing earned value management. Without reliable, timely, and
    auditable earned value management data, the program office cannot
    adequately manage technical, cost, and schedule risks and problems.

    20.We disagree. The Navy's own self-assessment of compliance with the 32
    criteria, detailed in appendix III of our report, showed that 17 of these
    criteria were not being satisfied. Further, our assessment showed that the
    Navy did not satisfy 29 of the 32 criteria, and program officials did not
    provide any evidence to refute the results of our assessment.

    21.The Navy did not provide us with a copy of the CDA Software Measurement
    Plan.

    22.See comment 5. Further, the Navy's position that "key stakeholders of
    the NTCSS program do, in fact, have the people, processes and tools to
    effectively execute their respective roles and responsibilities," is not
    consistent with its comment that this area will be part of a planned
    review.

    23.We disagree. Although the Navy states that the program is 95 percent
    complete, it still plans to spend $348 million over the next three fiscal
    years, which is approximately 32 percent of what has been spent on the
    program to date. In addition, because the Navy lacks disciplined
    acquisition management practices, as discussed in our report, including
    earned value management, we question how it is able to reliably determine
    what percentage of the work has been completed and the percentage that
    remains to be done. As stated in our report, the current milestone
    decision authority has allowed the program to proceed while a major
    application repeatedly failed operational testing, and another application
    was cancelled. In addition, the Navy stated its intent to revisit the need
    to change milestone decision authority.

    GAO Contact and Staff AcknowledgmentsAppendix V

    Randolph C. Hite (202) 512-3439 or hiter@gao.gov

    In addition to the contact named above, Cynthia Jackson, Assistant
    Director; Harold J. Brumm; Calvin L. H. Chang; Jennifer K. Echard; Joanne
    Fiorino; Neelaxi Lakhmani; Freda Paintsil; Jamelyn Payan; Karen Richey;
    Dr. Karl Seifert; Andrea Smith; and Dr. Rona B. Stillman made key
    contributions to this report.

    (310287)

    transparent illustrator graphic

    www.gao.gov/cgi-bin/getrpt? GAO-06-215 .

    To view the full product, including the scope

    and methodology, click on the link above.

    For more information, contact Randolph C. Hite at (202) 512-3439 or
    hiter@gao.gov.

    Highlights of GAO-06-215 , a report to the Subcommittee on Readiness and
    Management Support, Committee on Armed Services, U.S. Senate

    December 2005

    DOD SYSTEMS MODERNIZATION

    Planned Investment in the Naval Tactical Command Support System Needs to
    Be Reassessed

    Because it is important that the Department of Defense (DOD) adheres to
    disciplined information technology (IT) acquisition processes to
    successfully modernize its business systems, GAO was asked to determine
    whether the Naval Tactical Command Support System (NTCSS) is being managed
    according to important aspects of DOD's acquisition policies and guidance,
    as well as other relevant acquisition management best practices. NTCSS was
    started in 1995 to help Navy personnel effectively manage ship, submarine,
    and aircraft support activities. To date, about $1 billion has been spent
    to partially deploy NTCSS to about one-half its intended ashore and afloat
    sites.

    What GAO Recommends

    GAO is making recommendations to the Secretary of Defense to develop the
    analytical basis to determine if continued investment in NTCSS represents
    prudent use of limited resources. GAO is also making recommendations to
    strengthen management of the program, conditional upon a decision to
    proceed with further investment in the program. DOD either fully or
    partially concurred with the recommendations. It also stated that while
    some of GAO's findings are valid, the overall findings understated and
    misrepresented the program's level of discipline and conformance with
    applicable guidance and direction.

    The Department of the Navy has not managed its NTCSS program in accordance
    with key aspects of the department's policies and related guidance,
    including federal and recognized best practice guidance. Collectively,
    these policies and guidance are intended to reasonably ensure that
    investment in a given IT system represents the right solution to fill a
    mission need and, if it is, that acquisition and deployment of the system
    are handled in a manner that maximizes the chances of delivering defined
    system capabilities on time and within budget. In the case of NTCSS,
    neither of these outcomes is being realized. Specifically,

               The Navy has not economically justified its ongoing and planned
               investment in NTCSS. Specifically, it (1) has not reliably
               estimated future costs and benefits and (2) has not ensured that
               independent reviews of its economic justification were performed
               to determine its reliability.

               The Navy has not invested in NTCSS within the context of a
               well-defined DOD or Navy enterprise architecture, which is
               necessary to guide and constrain NTCSS in a way that promotes
               interoperability and reduces redundancy with related and dependent
               systems.

               The Navy has not effectively performed key measurement, reporting,
               budgeting, and oversight activities. In particular, earned value
               management, which is a means for determining and disclosing actual
               performance against budget and schedule estimates, has not been
               implemented effectively, and oversight entities have not had the
               visibility into the program needed to affect its direction.

               The Navy has not adequately conducted requirements management and
               testing activities. For example, requirements were neither
               prioritized nor traced to related documentation to ensure that the
               system delivers capabilities that meet user needs. This
               contributed to failures in developmental testing that have
               prevented the latest component of NTCSS from passing operational
               testing twice over the last 4 years.

    Reasons the Navy cited for not following policies and guidance ranged from
    their not being applicable to the NTCSS program, to lack of time available
    to apply them, to plans for strengthening system practices not being
    applied retroactively. Nevertheless, the Navy has begun taking steps and
    is considering other steps intended to address some of the above problems.
    Until program management improves, NTCSS will remain a risky program.
    *** End of document. ***