    Information Quality Act: National Agricultural Statistics Service
    Implements First Steps, but Documentation of Census of
    Agriculture Could Be Improved (23-SEP-05, GAO-05-644).

    The Information Quality Act (IQA) required the Office of
    Management and Budget to issue guidelines for ensuring the
    quality, objectivity, utility, and integrity of information
    disseminated by federal agencies. As part of our long-term
    examination of the quality of federal information, under the
    Comptroller General's authority, we reviewed how the act was
    implemented by the National Agricultural Statistics Service
    (NASS), and assessed the transparency of the documentation
    supporting its Census of Agriculture. NASS is part of the U.S.
    Department of Agriculture (USDA).
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-05-644
        ACCNO:   A37937
      TITLE:     Information Quality Act: National Agricultural Statistics
    Service Implements First Steps, but Documentation of Census of
    Agriculture Could Be Improved
         DATE:   09/23/2005
      SUBJECT:   Census
    	     Data collection
    	     Data integrity
    	     Federal law
    	     Government information
    	     Government information dissemination
    	     Quality assurance
    	     Reporting requirements
    	     Transparency

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-05-644

                     United States Government Accountability Office

    GAO

                           Report to Congressional Committees

    September 2005

    INFORMATION QUALITY ACT

          National Agricultural Statistics Service Implements First Steps, but
                Documentation of Census of Agriculture Could Be Improved

                                           a

    GAO-05-644

    [IMG]

    September 2005

    INFORMATION QUALITY ACT

    National Agricultural Statistics Service Implements First Steps, but
    Documentation of Census of Agriculture Could Be Improved

                                     What GAO Found

    NASS fulfilled its various procedural responsibilities and reporting
    requirements under the Office of Management and Budget's (OMB) guidelines
    for implementing the act. For example, NASS drafted its own implementation
    guidance, and developed a mechanism allowing affected parties to request
    the correction of information they believe is of poor quality. As a result
    of our review, NASS has also taken steps to better document the criteria
    it uses to evaluate data users' input on the content of the Census of
    Agriculture.

    The Census of Agriculture Provides a Detailed Picture of U.S. Farms and
    Ranches

    Building on these efforts, better documentation could improve the
    transparency of census data products. For example, the nine key products
    from the 2002 Census we examined lacked, among other things, discussions
    of any data limitations. This is contrary to NASS's own guidelines for
    ensuring transparency, which stress the importance of describing the
    methods, data sources, and other items to help users understand how the
    information was designed and produced.

    Although NASS complied with OMB's requirement to establish a mechanism
    under IQA to address requests to correct information, NASS has not
    documented its approach for handling correction requests not filed under
    IQA (NASS handles these correction requests using an existing, informal
    method). Agency officials told us that data users have been satisfied with
    the way NASS had responded to these requests. However, because NASS does
    not document its informal procedures for handling correction requests and
    lacks a recordkeeping system to log and track them, NASS could not provide
    us with specific data on the number of such requests it has handled, the
    nature of those requests, and whether and how they were addressed.

                     United States Government Accountability Office

    Contents

        Letter                                                                  1
                                      Results in Brief                          3
                                         Background                             5
                             Objectives, Scope, and Methodology                 7
                  NASS Met the Procedural and Reporting Requirements of OMB's
                                       IQA Guidelines                           9
                  Better Documentation Could Improve the Transparency of Data
                             Products and Correction Procedures                10
                                         Conclusions                           18
                            Recommendations for Executive Action               19
                             Agency Comments and Our Evaluation                19

    Appendixes

    Appendix I: Comments from the Department of Agriculture 21

    Appendix II:	How Census of Agriculture Reports Address Various
    Documentation Elements 23

    Appendix III: GAO Contact and Staff Acknowledgments 25

    Tables Table 1: NASS Addressed OMB's Agencywide Guidelines for

    Implementing IQA 9

    Table 2: Census Reports Need More Robust Documentation 11

    Table 3: NASS Is Using More Extensive Outreach to Develop the

    2007 Census Compared to 2002 16

    This is a work of the U.S. government and is not subject to copyright
    protection in the United States. It may be reproduced and distributed in
    its entirety without further permission from GAO. However, because this
    work may contain copyrighted images or other material, permission from the
    copyright holder may be necessary if you wish to reproduce this material
    separately.

    A

    United States Government Accountability Office Washington, D.C. 20548

    September 23, 2005

    The Honorable Saxby Chambliss
    Chairman
    The Honorable Tom Harkin
    Ranking Democratic Member
    Committee on Agriculture, Nutrition and Forestry
    United States Senate

    The Honorable Collin C. Peterson
    Ranking Democratic Member
    Committee on Agriculture
    House of Representatives

    The information disseminated by federal agencies is a critical strategic
    asset. For example, data collected for statistical purposes provide
    indicators of the economic and social well-being of the nation, while
    health,
    safety, environmental, and other scientific data help inform agencies'
    rule
    making activities. Given the widespread use and impact of federal
    information, it is important for it to meet basic quality standards.

    Section 515 of the Treasury and General Government Appropriations Act
    for Fiscal Year 2001-legislation that has come to be known as the
    Information Quality Act (IQA)1-required the Office of Management and
    Budget (OMB) to issue governmentwide guidelines that provide policy and
    procedural guidance to federal agencies for ensuring and maximizing the
    quality, objectivity, utility, and integrity of information disseminated
    by
    federal agencies.

    OMB's guidelines, issued in final form in February 2002, directed agencies
    covered by the act to issue their own quality guidelines, and noted that,
    where appropriate, agencies should support their data with transparent
    documentation. OMB's guidelines also required agencies to, among other
    actions, report annually to the Director of OMB on the number and nature
    of complaints received regarding compliance with the guidelines, and
    establish an administrative mechanism whereby affected parties can
    request the correction of information they deem to be of poor quality.

    1Consolidated Appropriations - Fiscal Year 2001, Pub. L. No. 106-544, S:
    515, 114 Stat. 2763A153 to 2763A-154 (2000) (44 U.S.C. S: 3516 note).

    As part of our long-term examination of the collection, dissemination, and
    quality of federal information, we are reviewing the governmentwide
    implementation of the IQA. As an initial step in our research on the
    quality of statistical data, under the Comptroller General's authority, we
    conducted a case study of the National Agricultural Statistics Service
    (NASS), and its Census of Agriculture. NASS is a statistical agency within
    the U.S. Department of Agriculture (USDA). We selected the census because
    it is one of the largest government surveys with a universe of 2.3 million
    respondents and an estimated paperwork burden in excess of 1.3 million
    burden hours.2 The last census took place in 2002 and the next census is
    scheduled for 2007.

    Specifically, our objectives were to (1) review how NASS met OMB's
    guidelines covering the IQA, and (2) examine the transparency of the
    documentation behind the Census of Agriculture's processes and products,
    both for the recently completed work on the 2002 Census and the efforts
    underway for the 2007 Census. To achieve both objectives, we reviewed
    OMB's and NASS's information quality guidelines and other relevant
    documents. We also interviewed senior agency officials and other personnel
    responsible for implementing the census including the NASS Administrator,
    Associate Administrator, and Deputy Administrator for Programs and
    Products.

    To evaluate the transparency of census products, we reviewed nine census
    products--eight reports and the Frequently Asked Questions (FAQ) section
    on NASS's 2002 Census Web site--to determine the extent to which NASS
    followed its own documentation guidelines. To obtain an external
    perspective of how NASS processes and products address the IQA guidelines,
    we interviewed six data users from different types of agricultural and
    research organizations. We selected these six because they use census data
    on a regular basis and have attended NASS's outreach meetings. Additional
    information on our approach is provided in the Objectives, Scope, and
    Methodology section below.

    We performed our work in Washington, D.C., from August 2004 through August
    2005 in accordance with generally accepted government auditing standards.

    2Under the Paperwork Reduction Act, agencies must estimate the burdens
    their data collections impose on the public.

    Results in Brief	NASS fulfilled its various procedural responsibilities
    and reporting requirements under OMB's guidelines. For example, NASS (1)
    drafted its own implementation guidelines and posted them on its Web site,
    (2) developed an administrative mechanism allowing affected persons to
    seek the correction of information, (3) designated an official responsible
    for ensuring NASS's compliance with OMB's requirements, and (4) reported
    to OMB the number of complaints received regarding data quality they
    received under the IQA.3

    With respect to the transparency of the documentation underlying the
    Census of Agriculture's data products and processes, as a result of our
    review, NASS has taken steps to better document the criteria it uses to
    evaluate data users' suggestions on the questionnaire content. The
    development of the 2002 and 2007 Censuses was led by the Census Content
    Team, which consisted of experienced NASS statisticians. The 2002 Team
    assessed users' input using a documented set of criteria, which considered
    such factors as whether the questionnaire items were mandated by Congress
    or whether they would provide data on current agricultural issues.
    However, because of staff turnover and reassignments, the 2007 Team was
    unaware of the 2002 criteria, and initially relied on professional
    judgment rather than documented factors to evaluate input on the 2007
    Census. According to NASS, our review raised the 2007 Team's awareness of
    the earlier criteria, and it has since developed similar documentation
    that it will use in the future. This approach is more consistent with
    NASS's own IQA guidelines concerning transparency, and could help create a
    closer link between the questions included in the census and evolving
    agricultural policy requirements, and thus a more cost-effective data
    collection program. Documenting the content selection criteria will also
    guard against the loss of institutional memory to the extent there is
    further turnover in Content Team membership.

    Building on these efforts, better documentation could improve the
    transparency of census data products. Although NASS's guidelines for
    ensuring transparency stress the importance of describing the methods,
    data sources, assumptions, and other items in order to help users
    understand how information was designed and produced, the eight census
    reports as well as the FAQ section we examined lacked a discussion of such
    important documentation practices as the limitations of the data; the

    3NASS officials reported that they have not received any complaints under
    the IQA.

    impact of imputations, by item; and whether any of the collected data have
    been suppressed for data quality reasons.

    The transparency of NASS's procedures for handling data correction
    requests filed outside of the IQA could also be improved. NASS complied
    with OMB's requirement to establish a mechanism to process requests for
    correction of disseminated information under the IQA. As part of this
    process, an individual must state that their request is being submitted
    under the IQA. To date, no individual has done so. NASS handles all other
    correction requests using its existing informal, undocumented procedures.
    Agency officials told us that it has resolved the informal requests it has
    handled so far to the data users' satisfaction. Nevertheless, because NASS
    does not document its informal procedures for handling correction requests
    and lacks a recordkeeping system to log and track them, NASS could not
    provide us with specific data on the number of requests it has handled,
    the nature of those requests, and whether and how they were addressed.

    To help enhance the transparency of the Census of Agriculture's processes
    and products, we recommend that the Secretary of Agriculture direct NASS
    to (1) ensure its products fully address its own requirements for
    transparent data documentation or at least contain links to such
    information and (2) document and post on its Web site its procedures for
    handling data correction requests not filed under the IQA and track the
    disposition of those requests.

    The NASS Administrator provided written comments on a draft of this report
    (see app. I). NASS said the information was insightful, and noted it will
    be used to strengthen the transparency of its methods and procedures. In
    particular, NASS agreed with our findings and, consistent with one of our
    two recommendations, said it will take steps to better document its
    specialized reports. NASS also said it plans to make a list of "common
    issues" raised by data users available on its Web site, which is in line
    with our second recommendation to improve the transparency of its
    procedures for handling data correction requests not filed under the IQA.
    NASS's commitment to continually improve its products is commendable, and
    its efforts to improve the transparency of its processes and products
    would be further enhanced if, consistent with our recommendations, it (1)
    ensures that all of its census products fully address NASS's own
    guidelines for data documentation and (2) posts on its Web site its
    procedures for handling correction requests not filed under the IQA.

    Background	The IQA directed OMB to issue guidelines to federal agencies
    covered by the Paperwork Reduction Act designed to ensure the "quality,
    objectivity, utility, and integrity" of information disseminated to the
    public.4 The IQA also directed OMB to include in its guidelines
    requirements for agencies to (1) develop their own information quality
    guidelines, (2) establish administrative mechanisms for affected persons
    to seek correction of information that does not comply with OMB's
    guidelines, and (3) annually report to OMB the number and nature of
    complaints they receive regarding the accuracy of the information they
    disseminate.5

    Prior to the IQA, there were several governmentwide actions aimed at
    improving agency data. For example, Statistical Policy Directive No. 2,
    first issued in 1952, required statistical agencies to inform users of
    conceptual or other limitations of the data, including how the data
    compare with similar statistics. In 1996, the Federal Committee on
    Statistical Methodology-an OMB-sponsored interagency committee dedicated
    to improving the quality of federal statistics-established a subcommittee
    to review the measurement and reporting of data quality in federal data
    collection programs. The results of the subcommittee's work were published
    in a 2001 report that addressed such issues as what information on sources
    of error federal data collection programs should provide, and how they
    should provide it.6 For all federal government information collections,
    the 1995 amendments to the Paperwork Reduction Act called on federal
    agencies to manage information resources with the goal of improving "the
    integrity, quality, and utility of information to all users within and
    outside the agency."7

    4Agencies covered by the IQA include all agencies subject to the Paperwork
    Reduction Act (PRA) - cabinet departments, independent regulatory agencies
    (e.g., Federal Communications Commission), and other independent agencies
    (e.g., the Environmental Protection Agency). 44 U.S.C. S: 3502(1).

    5Discussion of the IQA often centers on its impact on agencies' regulatory
    activities. Supporters of IQA, many of whom represent businesses and other
    regulated entities, maintain that IQA could enhance the quality of agency
    science and improve the rule-making process. Critics of IQA, including
    some environmental and public interest groups, view the law as a device to
    curtail health, safety, and other regulations.

    6OMB, Measuring and Reporting Sources of Error in Surveys, Statistical
    Policy Working Paper 31, July 2001.

    744 U.S.C. S: 3506(b)(1)(C).

    OMB's IQA guidelines were issued in final form in February 2002.8 They
    required agencies subject to the IQA to take such steps as

    o 	issue information quality guidelines designed to ensure the quality,
    objectivity, utility, and integrity of information disseminated to the
    public;

    o 	establish administrative mechanisms for affected persons to seek
    correction of information they believe is not in compliance with the
    guidelines;

    o 	report annually to the Director of OMB on the number and nature of
    complaints received regarding compliance with the guidelines and how the
    agencies handled those complaints; and

    o 	designate an official responsible for ensuring compliance with OMB's
    guidelines.

    The OMB guidelines defined quality as an encompassing term comprising

    o  utility, which is the usefulness of the information to its intended
    users;

    o 	integrity, which refers to the security of information and its
    protection from unauthorized access or revision; and

    o 	objectivity, which addresses both presentation (i.e., whether the
    information is being presented in an accurate, clear, complete, and
    unbiased manner) and substance (i.e., whether the information is accurate,
    reliable, and unbiased).

    In addition, OMB addresses transparency within the definition of
    objectivity and utility. As recognized in OMB's guidelines, agencies that
    disseminate influential scientific, financial, or statistical information
    must demonstrate a high degree of transparency about data and methods.
    These measures are in place to facilitate the information's
    reproducibility by an outside party or reanalysis of an agency's results.

    The National Research Council of the National Academies considers
    transparency a key principle for federal statistical agencies, and stated
    in a

    867 Fed. Reg. 8452 (Feb. 22, 2002).

    recent report that transparency, which it defines as "an openness about
    the sources and limitations of the data," is particularly important for
    instilling credibility and trust among data users and providers. 9

    As an agency within USDA, NASS is required to comply with the IQA. One
    statistical program administered by NASS is the quinquennial Census of
    Agriculture. According to NASS, the census provides a detailed picture of
    U.S. farms and ranches every 5 years and is the only source of uniform,
    comprehensive agricultural data at the county level. The results are
    published in 18 reports divided among three categories: Geographic Area
    Series, Census Quick Stats, and Specialty Products and Special Studies.
    Users of this information include federal agencies (for program and
    statistical purposes), farm organizations, businesses, universities, state
    departments of agriculture, elected representatives, legislative bodies at
    all levels of government, and academia. The next Census of Agriculture is
    scheduled for 2007.

      Objectives, Scope, and Methodology

    Our objectives were to (1) review how NASS met OMB's guidelines covering
    the IQA and (2) examine the transparency of the documentation behind the
    Census of Agriculture's processes and products, including the recently
    completed work on the 2002 Census, and the efforts currently underway for
    the 2007 Census.

    To achieve both of these objectives, we reviewed OMB's and NASS's
    information quality guidelines, Census of Agriculture reports,10
    submissions to OMB, and other relevant documents. We also interviewed NASS
    officials about how NASS conducted the 2002 Census and how it is planning
    for the 2007 Census. The officials included the NASS Administrator,
    Associate Administrator, and Deputy Administrator for Programs and
    Products.

    In addition, to evaluate the transparency of Census of Agriculture
    products, we reviewed eight census reports and the Frequently Asked
    Questions area of the 2002 Census Web site, to determine the extent to
    which NASS

    9Margaret E. Martin, Miron L. Straf, and Constance F. Citro eds.,
    Principles and Practices for a Federal Statistical Agency, (3rd ed.)
    (Washington, D.C.: National Academies Press, 2005), pp. 8, 29.

    10Reports can be obtained from USDA's Web site; see www.usda.gov/nass.

    followed its own procedures for ensuring the transparency of its
    information products. NASS's IQA guidelines define transparency as, "a
    clear description of the methods, data sources, assumptions, outcomes, and
    related information that allows a data user to understand how an
    information product was designed and produced."

    NASS's guidelines state that its survey activities include such activities
    as sample design, questionnaire design, pre-testing, analysis of sampling,
    and imputation of missing data. However, the guidelines were not clear as
    to the specific activities to be documented. Consequently, we reviewed the
    practices employed by such statistical agencies as the National Academies
    of Sciences, International Monetary Fund, and U.S. Census Bureau, and
    developed a set of 20 practices associated with transparent documentation
    that encompassed the items NASS laid out in its own guidelines. The
    practices include such actions as defining data items, discussing sample
    design, and describing how the content of the survey differs from past
    iterations (see app. II).

    We looked for the presence or absence of these practices in 9 out of the
    18 census reports and related forms of data that NASS disseminates, and
    verified the results with a second, independent analysis. In instances
    where a report did not include a particular documentation practice, we
    reviewed whether the report instead informed data users where to obtain
    this information. We chose these 9 reports because they all stem from the
    original census data collection, represent different product categories,
    and were available on the census Web site as of February 1, 2005.

    To obtain an external perspective of how NASS processes and products
    address the IQA guidelines, we interviewed six data users from different
    types of agricultural and research organizations. We selected these data
    users from lists of registrants for USDA and NASS outreach meetings within
    the past 5 years. We selected these six data users because they use
    information from the census on a regular basis. Moreover, these data users
    attended the most recent NASS outreach meeting, which specifically
    addressed the 2002 and 2007 Censuses. Some data users had also provided
    NASS with feedback on the content of the agricultural census. Their views
    cannot be projected to the larger population of census data users.

    We requested comments on a draft of this report from the Secretary of
    Agriculture. On September 8, 2005, we received the NASS Administrator's
    written comments and have reprinted them in appendix I. They are

    addressed in the Agency Comments and Our Evaluation section of this
    report.

    NASS Met the NASS fulfilled the various procedural responsibilities and
    reporting

    requirements under OMB's guidelines. For example, NASS released its
    ownProcedural and IQA guidelines for public comment on March 27, 2002.
    NASS officials Reporting stated they received no substantive comments on
    them and OMB approved Requirements of OMB's the guidelines with only
    minimal changes. The officials also noted that no

    revisions have been made since then. Table 1 shows in greater detail
    howIQA Guidelines NASS addressed OMB's guidelines.

    Table 1: NASS Addressed OMB's Agencywide Guidelines for Implementing IQA

    OMB directed agencies to: NASS's response

    o  Prepare a draft report explaining how  o  NASS posted its draft report
    and guidelines to their guidelines will ensure and its Web site from March
    27, 2002, until maximize the quality of information. September 30, 2002.

    o  Publish a notice in the Federal  o  Notice of availability was
    published in the June Register announcing the availability of 4, 2002,
    Federal Register a and the report itself this report on the agency's Web
    site was available on NASS's Web site. for public comment.

    o  Post their final report and guidelines  o  Final guidelines are on the
    NASS Web site. to their Web sites.

    o  Develop administrative mechanisms  o  NASS outlines its correction
    procedures in allowing affected persons to correct detail on its Web site.
    disseminated information that does not comply with the OMB guidelines.

    o  Submit a report by January 1 of each  o  USDA submitted a report to OMB
    both years year on the number and nature of since the guidelines have been
    in effect; NASS complaints received in the prior fiscal reported no
    complaints. year.

    o  Designate an official to be responsible  o  USDA has designated its
    Chief Information for the agency's compliance with Officer (CIO) as this
    official. The CIO in turn OMB's guidelines. delegates compliance questions
    to lower-level offices, including the Standards Officer within the office
    of the Associate Administrator at NASS.

    Source: GAO analysis of OMB and NASS documents.

    a67 Fed. Reg. 38,467.

      Better Documentation Could Improve the Transparency of Data Products and
      Correction Procedures

    NASS's IQA guidelines define transparency as, "a clear description of the
    methods, data sources, assumptions, outcomes, and related information that
    allows a data user to understand how an information product was designed
    and produced." NASS's guidelines also note that "NASS will make the
    methods used to produce information as transparent as possible" and that
    its "internal guidelines call for clear documentation of data and methods
    used in producing estimates and forecasts. . . ."

    To assess the extent to which NASS processes help ensure the transparency
    of the information it publishes, we examined key publications from the
    2002 Census of Agriculture. Census reports vary in terms of scope and
    intended audience (see table 2). On the one hand, the United States
    Summary and State Data report contains over 100 data tables, an
    introduction, and four appendices. On the other hand, County Profile
    reports summarize each county's agricultural situation on two pages.

    Overall, we assessed eight census reports within three product categories,
    as well as the Frequently Asked Questions (FAQ) section of the 2002 Census
    Web site, to determine the extent to which NASS followed its own
    guidelines for ensuring the transparency of its products. As shown in
    table 2, the transparency of the data documentation in the reports we
    reviewed varied between the Geographic Area Series reports-which are the
    most comprehensive of NASS's products and addressed 15 of the 20 data
    documentation practices-and the Specialty Products and Special Studies
    which, depending on the specific product, addressed no more than 1 of the
    practices.

                 Table 2: Census Reports Need More Robust Documentation

                                                                    Portion of 20
                                                                    documentation
                                                                        practices
        Product category    Product title    General description        addressed

         Geographic Area   United States Summary and  Contains over 100 national
                                                        and state data tables.
             Series               State Data
                             State and County Data   Contains over 100 state and
                                                         county data tables.
                           2002 Census of              A database for users to
    Census Quick Stats: Agriculture                download and generate data
    Ag Statistics Data  Downloadable Application    tables at the national,
                                                      state, and county levels.
              Base

    Specialty Products and State and County Profiles Two-page reports
    containing summary data about a Special Studies state or county.

    Quick Facts from the 2002 This report presents national data in 16 charts
    or Census of Agriculture graphs.

    Ranking of Market Value of This report contains state tables that rank the

    Agricultural Products Soldagricultural products sold by market value. The
    report also includes table definitions.

    Congressional District ProfilesEach profile is a two-page report that
    contains summary data about one congressional district.

    Ranking of Congressional Contains tables for 46 data items, such as number
    of

    Districtsfarms, and ranks the congressional districts for each of these
    data items.

    Additional Information Frequently Asked QuestionsThis section of the
    census Web site contains questions and answers grouped into four
    categories.

    Source: GAO analysis of NASS reports.

    All eight reports and the FAQ Web site lacked a discussion of four
    documentation practices, including the following:

    1.	Questionnaire testing. NASS produced a separate, internal report that
    discusses questionnaire testing in detail; however, publicly available
    census publications do not address this topic.

    2.	Limitations of the data. NASS does not discuss data limitations in the
    census reports we reviewed.

    3.	Impact of imputations, by item. When a statistical agency receives a
    report form with missing values, it normally estimates or "imputes" those
    values based on comparable data sources such as a similar farm operation.
    Although NASS uses a complex editing and imputation process to estimate
    missing values, and describes this process in the

    United States Summary and State Data report appendix, it does not quantify
    the impact of imputations by item in reports.

    4.	Whether any of the collected data have been suppressed for data quality
    reasons. Without information on whether any of the data had been
    suppressed because the quality was lacking, data users must assume that
    reports include all data items collected in the census had met agency
    publication standards.

    Although NASS appropriately recognizes the variation in data user needs by
    publishing several types of specialized reports, none of the reports we
    reviewed direct data users where to find either a complete set of
    documentation or additional documentation. For example, given the short
    length and summary format of the County Profile reports, it is not
    surprising that they lack documentation. However, in order for users to
    assess the quality of the data contained in the reports, it is important
    for NASS to at least provide links on its Web site or to other
    publications where users can access definitions, response rates, and other
    relevant information.

        NASS Should Document Its Procedures for Handling Correction Requests Not
        Filed under the IQA

    NASS has two methods for handling data correction requests, depending on
    how they are submitted: a formal approach prescribed by OMB for correction
    requests filed under IQA, and an informal approach that NASS uses to
    address correction requests that are not filed under IQA. NASS's informal
    correction procedures lack transparency because they are not documented
    and individual cases are not tracked. As a result, we could not determine
    the nature of these correction requests or whether or how they were
    addressed.

    Consistent with OMB's guidelines, NASS detailed its procedures to request
    corrections under IQA on its Web site, and posted appropriate Federal
    Register notices. For example, NASS's Web site explains that to seek a
    correction under IQA, petitioners must, among other steps: (1) state that
    their request for correction is being submitted under IQA, (2) clearly
    identify the information they believe to be in error, and (3) describe
    which aspects of NASS's IQA guidelines were not followed or were
    insufficient.11

    11See www.usda.gov/nassinfo/infocorrection.htm.

    According to the instructions posted on its Web site, NASS's IQA
    procedures are triggered only when petitioners explicitly state they are
    submitting a correction request under IQA. To date, none have done so.
    NASS addresses all other correction requests using informal, undocumented
    procedures that were in place before IQA was enacted. NASS officials
    explained that such requests are forwarded to the agency official
    responsible for preparing the report containing the information in
    question. That official, in turn, determines if the request can be
    resolved by clarifying the data, or whether a correction is needed. If a
    data item needs to be corrected, NASS has a set of procedures for
    documenting errors and issuing errata reports that are detailed in its
    Policy and Standards Memorandum No. 38. The memorandum describes the
    circumstances under which errata reports will be printed, and provides a
    mechanism for NASS staff to describe the nature of the error, its cause,
    and the action taken to resolve it.

    According to the Administrator, Associate Administrator, and other senior
    NASS officials we interviewed, the requests it has handled from the 2002
    Census have so far been resolved to the petitioners' satisfaction, and
    none resulted in any corrections to the data from the 2002 Census.
    However, because NASS does not document its informal procedures for
    handling inquiries and data correction requests, and lacks a recordkeeping
    system to log and track them, NASS could not provide us with firm
    information on the number of inquiries it has handled, the nature of those
    inquiries, and whether and how they were addressed.

    This is not to say that all complaints should follow the same procedures
    required by the IQA mechanism. For efficiency's sake, it is important for
    agencies to respond to complaints in accordance with the magnitude of the
    problem. However, to provide a more complete picture of the questions NASS
    receives about its data and how those questions were handled, it will be
    important for NASS to better document its approach for handling correction
    requests not filed under IQA, and track their disposition.

    NASS Has Taken Steps to The 2002 Census of Agriculture was the first in
    which NASS developed the Better Document Its Criteria questionnaire (the
    1997 Census of Agriculture was moved from the Census for Assessing Input
    on Bureau to NASS after the content had been determined). In doing so,
    NASS

    went to great lengths to obtain input from data users on what questions to

    Census Content	ask, and evaluated their suggestions using a documented set
    of criteria. In preparing for the 2007 Census, NASS sought feedback on the
    questionnaire content from a broader spectrum of data users, in part
    because NASS

    solicited suggestions via the Internet. However, unlike the 2002 cycle,
    the criteria NASS used to assess the feedback were not initially
    documented, which is contrary to NASS's IQA guidelines. However, as a
    result of our review, NASS has developed documented criteria similar to
    that used during the previous census.

    Under the Paperwork Reduction Act, agencies must obtain OMB's approval
    prior to collecting information from the public. As part of this process,
    agencies must certify to OMB that, among other things, the effort is
    necessary for the proper performance of agency functions, avoids
    unnecessary duplication, and reduces burden on small entities. Agencies
    must also provide an estimate of the burden the information collection
    would place on respondents.12

    For the 2002 Census, NASS submitted its request for approval-a form called
    "OMB 83-I"-in August 2001, and OMB approved it in October 2001. NASS
    estimated that the census would require a cumulative total of more than
    1.3 million hours for respondents to complete and would cost them, in
    terms of their time, in excess of $21 million.

    OMB's approval process also requires agencies to solicit input from
    external sources. NASS obtained input on the 2002 Agricultural Census
    content through a Federal Register notice, meetings with data users, and
    by contacting federal and state agencies that use census statistics to
    discuss data needs.

    Likewise, NASS is obtaining input on the content of the 2007 Census
    through a variety of channels. According to an agency official, the
    process began around June 2004, when NASS began releasing publications
    from the 2002 Census. NASS sent an evaluation form to its state offices
    requesting feedback on the census, including their suggestions for
    changing the content. NASS also asked the state offices to identify users
    from whom it could obtain additional feedback.

    NASS solicited further input by reaching out to data users within USDA and
    other federal agencies, querying organizations included in a list of
    "typical" data users maintained by NASS's Marketing and Information
    Services Office, and holding periodic regional meetings with data users.
    NASS also

    1244 U.S.C. S: 3506(c).

    has a "hot button" on its Web site where visitors are asked what items, if
    any, should be added or deleted from the census.13

    In all, NASS obtained input on the 2007 Census through 10 distinct
    conduits. Moreover, compared to the process used to develop the content of
    the 2002 Census, its 2007 efforts were open to a wider spectrum of
    customers, and involved more direct contact with data users during the
    planning phase. Indeed, as shown in table 3, NASS's outreach via the
    Internet, regional meetings, and queries to data users was over and above
    the steps it took when developing the 2002 Census. This openness was
    reflected in the comments of the six data users we interviewed. Five of
    the six users said NASS's approach to eliciting input was adequate, while
    three of the six had requested new content items for the 2007 Census to
    better meet the needs of their organizations.

    The content evaluation process began in December 2004, and NASS is
    currently testing the questionnaire content. Following any refinements,
    mail-out of the actual census is scheduled for December 2007.

    13See http://www.nass.usda.gov/census/census02/feedback.htm.

    Table 3: NASS Is Using More Extensive Outreach to Develop the 2007 Census
    Compared to 2002

                       Method of outreach 2002 Census 2007 Census

    Posted Federal Register notices X X

    Solicited input from state agricultural statistical offices X X

    Solicited input from state governors X

    Solicited input from Advisory Committee on Agricultural Statistics X X

    Solicited input from land grant universities X X

    Solicited input from federal data users X X

    Held federal data user working group meetings X X

    Held USDA-wide national data user outreach meeting X X

    Solicited input from a list of "typical" census users maintained by NASS's
    Marketing and Information X Services Office

    Solicited input via Web site feedback form X

    Held NASS-specific, regional data user meeting X

    Source: GAO analysis of NASS data.

    For both the 2002 and 2007 Census cycles, the solicitation, review, and
    ultimate determination of the questionnaire content was led by the Census
    Content Team, a group consisting of experienced NASS statisticians
    representing different segments of the agency such as livestock, crops,
    and marketing. The 2002 Content Team used specific, documented criteria to
    inform its decisions. Specifically, suggestions were assessed according to
    the following factors, which were also made available to data users:

    o 	items directly mandated by Congress or items that had strong
    congressional support;

    o 	items proposed by other federal agencies where legislation called for
    that agency to provide data for Congress;

    o  items needed for evaluation of existing federal programs;

    o 	items which, if omitted, would result in additional respondent burden
    and cost for a new survey for other agencies or users;

    o  items required for classification of farms by historical groupings;

    o  items needed for improving coverage in the census; and

    o  items that would provide data on current agricultural issues.

    However, the criteria the 2007 Team used to assess input on the
    questionnaire content were not initially documented. According to agency
    officials we interviewed, NASS largely relied on professional judgment to
    evaluate the feedback it received, considering such factors as the need to
    keep the data comparable to past censuses and not increase the length of
    the questionnaire.

    Although a certain amount of professional judgment will invariably be used
    in making determinations on questionnaire content, the absence of
    documented assessment criteria is inconsistent with NASS's guidelines.
    Indeed, these guidelines note that transparent documentation "allows a
    data user to understand how an information product was designed and
    produced." Moreover, without documented criteria, it is not clear whether
    members of the Content Team are considering the same set of factors, or
    even if they are weighing those factors in the same manner.

    According to NASS, the shift in approach stemmed from staff turnover and
    reassignments of members of the 2002 Team and, as a result, the 2007 Team
    was not aware of the criteria used in 2002. Our review made the 2007 Team
    aware of the earlier set of criteria, and the Team has since developed
    similar documentation. NASS noted that all future content teams will use
    and update these criteria when developing the content of subsequent
    censuses.

    It will be important for NASS to continue with this approach because it is
    more consistent with its own IQA guidelines, and will also help NASS to do
    the following:

    Ensure the utility and relevance of information. A key principle for
    federal statistical agencies is to provide information relevant to issues
    of public policy.14 However, the nation's information needs are constantly
    evolving, and it is important for statistical agencies to adapt
    accordingly. This is particularly true with agriculture, where a variety
    of factors such as changing technology and agricultural trends can affect
    what information should be collected. Rigorous content selection criteria
    could help NASS

    14Margaret E. Martin, Miron L. Straf, and Constance F. Citro eds.,
    Principles and Practices for a Federal Statistical Agency, (3rd ed.)
    (Washington, D.C.: National Academies Press, 2005), p. 4.

    methodically evaluate the needs of different users, establish priorities,
    and keep the census synchronized with changing public policy requirements.

    Maximize cost-effectiveness and reduce public burden. As with all federal
    surveys, there are financial and nonfinancial costs to conducting the
    Census of Agriculture. These costs include the direct expenditures related
    to planning, implementing, and analyzing the census, as well as
    disseminating the information. There is also a cost to respondents in
    terms of the time they take to complete the questionnaire. Additionally,
    there are opportunity costs in that for every question that is included in
    the census, another question might need to be excluded so as not to
    increase the length of the census. Rigorous, consistently applied criteria
    can help promote cost-effectiveness because they can ensure that only
    those questions that meet a particular, previously identified need are
    included in the census. Applying such criteria also help inform decisions
    on the appropriate role of the federal government in collecting the data,
    and whether a particular question might be more appropriately addressed by
    a different survey, government organization, or the by the private sector.

    Maintain credibility. Content selection criteria provide a basis for
    consistent decision making on what to include in the census and what gets
    left off. This is especially important for maintaining NASS's credibility
    given the input it receives from various sources. Without documented
    criteria, NASS's actions could be perceived as arbitrary or
    disproportionately swayed by one particular interest or another; thus,
    NASS's decisions would be more defensible.

    Further, documented criteria will guard against the loss of institutional
    memory to the extent there is further turnover in Content Team membership.

    Conclusions	NASS satisfied the procedural responsibilities and reporting
    requirements under OMB's IQA guidelines. Moreover, to the extent that NASS
    continues to use the documented criteria it developed to inform future
    decisions on the content of the Census of Agriculture, it could help
    establish a closer alignment between the questions included in the census
    and evolving agricultural policy requirements, resulting in a more
    cost-effective data collection program.

    Building on these efforts, the transparency of census data products could
    be improved with more robust documentation. NASS's procedures for

    addressing correction requests not filed under IQA could be more
    transparent as well. More than just a paperwork issue, greater
    transparency will help enhance NASS's accountability to public data users
    and increase the credibility of census information.

    Recommendations for 	To help enhance the transparency of the Census of
    Agriculture's processes and products, we recommend that the Secretary of
    Agriculture direct NASS

    Executive Action to take the following two steps:

    1.

    2.

    Ensure that census products fully address NASS's own guidelines for data
    documentation or at least contain links to such information. The list of
    20 documentation practices that we developed, while not necessarily
    exhaustive, represents sound actions used by other statistical agencies
    and could form a starting point for NASS.

    Document and post on NASS's Web site its procedures for handling data
    correction requests not filed under IQA, and track the disposition of
    those requests.

      Agency Comments and Our Evaluation

    The NASS Administrator provided written comments on a draft of this report
    on September 8, 2005, which are reprinted in appendix I. NASS noted that
    our "report and recommendations are insightful and will be used to further
    strengthen the transparency of NASS methods and procedures."

    In particular, NASS concurred with our finding that the methods and
    procedures in its specialized reports should be better documented and,
    consistent with our recommendation, stated that these products "will now
    provide links to this information." NASS's efforts, if fully implemented,
    should make it easier for data users to understand how these products were
    designed and produced, and NASS should be commended for its actions to
    continually improve its products and better meet the needs of its
    customers.

    While NASS's more comprehensive products were better documented, our
    analysis found that they could also benefit from more robust
    documentation. Thus, in keeping with our recommendation, it will be
    important for NASS to ensure that all of its census products-its larger
    reports and more focused studies--fully address NASS's own guidelines for
    data documentation.

    In commenting on our recommendation for NASS to document and post on its
    Web site its procedures for handling data correction requests not filed
    under IQA, NASS concurred with our view that this information would
    provide it with a better sense of the questions it receives about its
    data, but added that "a detailed recordkeeping system to log and track
    every inquiry" would not be the best use of its resources. Instead, NASS
    plans to "compile a listing of the more common issues" and make them
    available on its Web site in the form of frequently asked questions. NASS
    believes this approach would be useful for future planning, as well as
    provide answers to questions most likely to arise among other data users.

    As noted in our report, our recommendation stemmed from our finding that
    NASS could not provide us with information on the number of inquiries not
    filed under IQA, the characteristics of those inquiries, and how they were
    addressed. Although the details remain to be seen, NASS's proposed
    approach could provide this information and, consistent with the intended
    outcome our recommendation, address the need for greater transparency.
    NASS's efforts will be further strengthened if, consistent with our
    recommendation, it posts on its Web site its procedures for handling
    correction requests not filed under IQA.

    We will send copies of this report to other interested congressional
    parties,
    the Secretary of Agriculture, and the NASS Administrator. Copies will be
    made available to others on request. This report will also be available at
    no
    charge on GAO's Web site at http://www.gao.gov.

    If you or your staff have any questions about this report, please contact
    me
    at (202) 512-6806 or williamso@gao.gov. Contact points for our Offices of
    Congressional Relations and Public Affairs may be found on the last page
    of this report. GAO staff who made major contributions to this report are
    listed in appendix III.

    Orice M. Williams
    Director
    Financial Markets and

    Community Investments

    Appendix I

    Comments from the Department of Agriculture

    Appendix I
    Comments from the Department of
    Agriculture

    Appendix II

    How Census of Agriculture Reports Address Various Documentation Elements

                                     Product titles

    United                                         Ranking:
    States          State and   Quick                Market
    Summary       County        Stats: Ag          Value Ag
    and State     Reports       Statistics         Products
    Report        (Alabama)     Database               Sold

    State and County Profiles Congress. District Profiles Ranking of Congress.
    Districts Frequently Asked Questions Documentation practices

                                      Quick Facts

    Discussion of how NASS developed the content of the census.

                                         XXX X

             Discussion of             X           X           X                X
              2002 Census
                content
            consistency with
                 1997.

            Description of X         X       X                                  X
            how the content
                differs.
            Discussion of X          X       X
             why NASS made
            content changes.
          Definition of the X        X       X          X                       X
              population.
          Definition of data X       X       X               X          X
                 items.
            Discussion of X          X       X
             sample design.
            Discussion of X          X       X
             questionnaire
                design.

    Discussion of questionnaire testing.

    Copy of the         X       X
    questionnaire.
    Discussion of       X       X      X
    data collection
    procedures.
    Discussion of       X       X      X
    nonresponse
    followup.
    Discussion of       X       X      X
    data entry
    procedures.

                                      Appendix II
                       How Census of Agriculture Reports Address
                             Various Documentation Elements

    (Continued From Previous Page)

    Product titles

                  United                                      Ranking:
                  States  State and   Quick                    Market
                                               State                             Ranking
                  Summary  County   Stats: Ag  and            Value Ag Congress. of        Frequently
    Documentation and      Reports  Statistics  County  Quick Products District  Congress.   Asked
                  State
      practices   Report  (Alabama) Database   Profiles Facts   Sold   Profiles  Districts  Questions
    Discussion of    X        X         X
    data editing
     procedures.

    Quantify the impact of imputations, by item.

             Discussion of             X           X           X                X
            response rates.
             Discussion of             X           X           X
              nonsampling
                 error.

    Discussion of the limitations of the data.

    Discussion of whether any of the collected data have been suppressed for
    data quality reasons.

    Comparison of X
    census results to
    other survey
    results for
    consistency.

    Source: GAO analysis of NASS data.

    Note: See the Objectives, Scope, and Methodology section of this report
    for a complete explanation of our analysis.

    Appendix III

                         GAO Contact and Staff Acknowledgments

    GAO Contact Orice M. Williams, (202) 512-6806

    Acknowledgments	In addition to the contact named above, Robert Goldenkoff,
    Assistant Director; David Bobruff; Jennifer Cook; Richard Donaldson;
    Andrea Levine; Robert Parker; John Smale; and Michael Volpe made key
    contributions to this report.

    GAO's Mission	The Government Accountability Office, the audit, evaluation
    and investigative arm of Congress, exists to support Congress in meeting
    its constitutional responsibilities and to help improve the performance
    and accountability of the federal government for the American people. GAO
    examines the use of public funds; evaluates federal programs and policies;
    and provides analyses, recommendations, and other assistance to help
    Congress make informed oversight, policy, and funding decisions. GAO's
    commitment to good government is reflected in its core values of
    accountability, integrity, and reliability.

    Obtaining Copies of The fastest and easiest way to obtain copies of GAO
    documents at no cost

    is through GAO's Web site (www.gao.gov). Each weekday, GAO postsGAO
    Reports and newly released reports, testimony, and correspondence on its
    Web site. To Testimony have GAO e-mail you a list of newly posted products
    every afternoon, go to

    www.gao.gov and select "Subscribe to Updates."

    Order by Mail or Phone	The first copy of each printed report is free.
    Additional copies are $2 each. A check or money order should be made out
    to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
    Orders for 100 or more copies mailed to a single address are discounted 25
    percent. Orders should be sent to:

    U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
    D.C. 20548

    To order by Phone: 	Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
    512-6061

      To Report Fraud, Contact:
      Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm

    E-mail: fraudnet@gao.govFederal Programs Automated answering system: (800)
    424-5454 or (202) 512-7470

    Congressional 	Gloria Jarmon, Managing Director, JarmonG@gao.gov (202)
    512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125

    Relations Washington, D.C. 20548

    Public Affairs	Paul Anderson, Managing Director, AndersonP1@gao.gov (202)
    512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
    Washington, D.C. 20548
    *** End of document. ***