United States Government Publishing Office publisher pbl distributor dst United States Government Accountability Office National Security and International Affairs Division author aut Government Organization text government publication eng GAOREPORTS Legislative Agency Publications gao legislative 2010-08-12 U.S. Government Printing Office 2000-07-31 monographic deposited born digital 71 p. GA 1.13:NSIAD-00-199 https://www.govinfo.gov/app/details/GAOREPORTS-NSIAD-00-199 P0b002ee1803779a6 f:ns00199 DGPO 2010-08-12 2011-03-24 GAOREPORTS-NSIAD-00-199 machine generated eng fdlp REPORT GAOREPORTS-NSIAD-00-199 NSIAD-00-199 Cost effectiveness analysis Defense cost control Weapons systems Comparative analysis Operational testing Weapons research and development Private sector practices Defense procurement Procurement planning DarkStar Unmanned Aerial Vehicle SDI Theater High Altitude Area Defense System Standoff Land Attack Missile F-22 Raptor Aircraft Chapter Report NSIAD Pursuant to a congressional request, GAO provided information on the role of testing and evaluation in product development, focusing on: (1) how the conduct of testing and evaluation affects commercial and Department of Defense (DOD) program outcomes; (2) how best commercial testing and evaluation practices compare with the DOD's; and (3) what factors account for the differences in these practices.<p/>GAO noted that: (1) for the leading commercial firms GAO visited, the proof of testing and evaluation lies in whether a product experiences what one firm called "late cycle churn," or the scramble to fix a significant problem discovered late in development; (2) late cycle churn has been a fairly common occurrence on DOD weapon systems; (3) often, tests of a full system identify problems that could have been found earlier; (4) leading commercial firms GAO visited use testing and other techniques to expose problems earlier than the DOD programs GAO reviewed; (5) the firms focus on validating that their products have reached increasing levels of product maturity at given points in time; (6) the firms' products have three maturity levels in common--components work individually, components work together as a system in a controlled setting, and components work together as a system in a realistic setting; (7) the key to minimizing late surprises is to reach the first two levels early, limiting the burden on the third level; (8) by concentrating on validating knowledge rather than the specific technique used commercial firms avoid skipping key events and holding hollow tests that do not add knowledge; (9) on the weapon programs, system level testing carried a greater share of the burden; (10) earlier tests were delayed, skipped, or not conducted in a way that advanced knowledge; (11) the differences in testing practices reflect the different demands commercial firms and DOD impose on program managers; (12) leading commercial firms insist that a product satisfy the customer and make a profit; (13) success is threatened if managers are unduly optimistic or if unknowns about a product are not resolved early, when costs are low and more options are available; (14) the role of testing under these circumstances is constructive, for it helps eliminate unknowns; (15) product managers view testers and realistic test plans as contributing to a product's success; (16) success for a weapon system program is different--it centers on attempting to provide a superior capability within perceived time and funding limits; (17) success is influenced by the competition for funding and the quest for top performance--delivering the product late and over cost does not necessarily threaten success; (18) testing plays a less constructive role in DOD because a failure in a key test can jeopardize program support; (19) specifically, test results often become directly linked to funding and other key decisions for programs; and (20) such a role creates a more adversarial relationship between testers and program managers. https://www.govinfo.gov/content/pkg/GAOREPORTS-NSIAD-00-199/html/GAOREPORTS-NSIAD-00-199.htm https://www.govinfo.gov/content/pkg/GAOREPORTS-NSIAD-00-199/pdf/GAOREPORTS-NSIAD-00-199.pdf GAO/NSIAD-00-199 https://www.govinfo.gov/app/details/GAOREPORTS-NSIAD-00-199 Chapter Report GAO/NSIAD-00-199; Best Practices: A More Constructive Test Approach Is Key; Cost effectiveness analysis Defense cost control Weapons systems Comparative analysis Operational testing Weapons research and development Private sector practices Defense procurement Procurement planning DarkStar Unmanned Aerial Vehicle SDI Theater High Altitude Area Defense System Standoff Land Attack Missile F-22 Raptor Aircraft Public Law 105-261