Information Quality Act: National Agricultural Statistics Service
Implements First Steps, but Documentation of Census of		 
Agriculture Could Be Improved (23-SEP-05, GAO-05-644).		 
                                                                 
The Information Quality Act (IQA) required the Office of	 
Management and Budget to issue guidelines for ensuring the	 
quality, objectivity, utility, and integrity of information	 
disseminated by federal agencies. As part of our long-term	 
examination of the quality of federal information, under the	 
Comptroller General's authority, we reviewed how the act was	 
implemented by the National Agricultural Statistics Service	 
(NASS), and assessed the transparency of the documentation	 
supporting its Census of Agriculture. NASS is part of the U.S.	 
Department of Agriculture (USDA).				 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-05-644 					        
    ACCNO:   A37937						        
  TITLE:     Information Quality Act: National Agricultural Statistics
Service Implements First Steps, but Documentation of Census of	 
Agriculture Could Be Improved					 
     DATE:   09/23/2005 
  SUBJECT:   Census						 
	     Data collection					 
	     Data integrity					 
	     Federal law					 
	     Government information				 
	     Government information dissemination		 
	     Quality assurance					 
	     Reporting requirements				 
	     Transparency					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-05-644

                 United States Government Accountability Office

GAO

                       Report to Congressional Committees

September 2005

INFORMATION QUALITY ACT

      National Agricultural Statistics Service Implements First Steps, but
            Documentation of Census of Agriculture Could Be Improved

                                       a

GAO-05-644

[IMG]

September 2005

INFORMATION QUALITY ACT

National Agricultural Statistics Service Implements First Steps, but
Documentation of Census of Agriculture Could Be Improved

                                 What GAO Found

NASS fulfilled its various procedural responsibilities and reporting
requirements under the Office of Management and Budget's (OMB) guidelines
for implementing the act. For example, NASS drafted its own implementation
guidance, and developed a mechanism allowing affected parties to request
the correction of information they believe is of poor quality. As a result
of our review, NASS has also taken steps to better document the criteria
it uses to evaluate data users' input on the content of the Census of
Agriculture.

The Census of Agriculture Provides a Detailed Picture of U.S. Farms and
Ranches

Building on these efforts, better documentation could improve the
transparency of census data products. For example, the nine key products
from the 2002 Census we examined lacked, among other things, discussions
of any data limitations. This is contrary to NASS's own guidelines for
ensuring transparency, which stress the importance of describing the
methods, data sources, and other items to help users understand how the
information was designed and produced.

Although NASS complied with OMB's requirement to establish a mechanism
under IQA to address requests to correct information, NASS has not
documented its approach for handling correction requests not filed under
IQA (NASS handles these correction requests using an existing, informal
method). Agency officials told us that data users have been satisfied with
the way NASS had responded to these requests. However, because NASS does
not document its informal procedures for handling correction requests and
lacks a recordkeeping system to log and track them, NASS could not provide
us with specific data on the number of such requests it has handled, the
nature of those requests, and whether and how they were addressed.

                 United States Government Accountability Office

Contents

    Letter                                                                  1 
                                  Results in Brief                          3 
                                     Background                             5 
                         Objectives, Scope, and Methodology                 7 
              NASS Met the Procedural and Reporting Requirements of OMB's 
                                   IQA Guidelines                           9 
              Better Documentation Could Improve the Transparency of Data 
                         Products and Correction Procedures                10 
                                     Conclusions                           18 
                        Recommendations for Executive Action               19 
                         Agency Comments and Our Evaluation                19 

Appendixes

Appendix I: Comments from the Department of Agriculture 21

Appendix II:	How Census of Agriculture Reports Address Various
Documentation Elements 23

Appendix III: GAO Contact and Staff Acknowledgments 25

Tables Table 1: NASS Addressed OMB's Agencywide Guidelines for

Implementing IQA 9

Table 2: Census Reports Need More Robust Documentation 11

Table 3: NASS Is Using More Extensive Outreach to Develop the

2007 Census Compared to 2002 16

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

A

United States Government Accountability Office Washington, D.C. 20548

September 23, 2005

The Honorable Saxby Chambliss
Chairman
The Honorable Tom Harkin
Ranking Democratic Member
Committee on Agriculture, Nutrition and Forestry
United States Senate

The Honorable Collin C. Peterson
Ranking Democratic Member
Committee on Agriculture
House of Representatives

The information disseminated by federal agencies is a critical strategic
asset. For example, data collected for statistical purposes provide
indicators of the economic and social well-being of the nation, while
health,
safety, environmental, and other scientific data help inform agencies'
rule
making activities. Given the widespread use and impact of federal
information, it is important for it to meet basic quality standards.

Section 515 of the Treasury and General Government Appropriations Act
for Fiscal Year 2001-legislation that has come to be known as the
Information Quality Act (IQA)1-required the Office of Management and
Budget (OMB) to issue governmentwide guidelines that provide policy and
procedural guidance to federal agencies for ensuring and maximizing the
quality, objectivity, utility, and integrity of information disseminated
by
federal agencies.

OMB's guidelines, issued in final form in February 2002, directed agencies
covered by the act to issue their own quality guidelines, and noted that,
where appropriate, agencies should support their data with transparent
documentation. OMB's guidelines also required agencies to, among other
actions, report annually to the Director of OMB on the number and nature
of complaints received regarding compliance with the guidelines, and
establish an administrative mechanism whereby affected parties can
request the correction of information they deem to be of poor quality.

1Consolidated Appropriations - Fiscal Year 2001, Pub. L. No. 106-544, S:
515, 114 Stat. 2763A153 to 2763A-154 (2000) (44 U.S.C. S: 3516 note).

As part of our long-term examination of the collection, dissemination, and
quality of federal information, we are reviewing the governmentwide
implementation of the IQA. As an initial step in our research on the
quality of statistical data, under the Comptroller General's authority, we
conducted a case study of the National Agricultural Statistics Service
(NASS), and its Census of Agriculture. NASS is a statistical agency within
the U.S. Department of Agriculture (USDA). We selected the census because
it is one of the largest government surveys with a universe of 2.3 million
respondents and an estimated paperwork burden in excess of 1.3 million
burden hours.2 The last census took place in 2002 and the next census is
scheduled for 2007.

Specifically, our objectives were to (1) review how NASS met OMB's
guidelines covering the IQA, and (2) examine the transparency of the
documentation behind the Census of Agriculture's processes and products,
both for the recently completed work on the 2002 Census and the efforts
underway for the 2007 Census. To achieve both objectives, we reviewed
OMB's and NASS's information quality guidelines and other relevant
documents. We also interviewed senior agency officials and other personnel
responsible for implementing the census including the NASS Administrator,
Associate Administrator, and Deputy Administrator for Programs and
Products.

To evaluate the transparency of census products, we reviewed nine census
products--eight reports and the Frequently Asked Questions (FAQ) section
on NASS's 2002 Census Web site--to determine the extent to which NASS
followed its own documentation guidelines. To obtain an external
perspective of how NASS processes and products address the IQA guidelines,
we interviewed six data users from different types of agricultural and
research organizations. We selected these six because they use census data
on a regular basis and have attended NASS's outreach meetings. Additional
information on our approach is provided in the Objectives, Scope, and
Methodology section below.

We performed our work in Washington, D.C., from August 2004 through August
2005 in accordance with generally accepted government auditing standards.

2Under the Paperwork Reduction Act, agencies must estimate the burdens
their data collections impose on the public.

Results in Brief	NASS fulfilled its various procedural responsibilities
and reporting requirements under OMB's guidelines. For example, NASS (1)
drafted its own implementation guidelines and posted them on its Web site,
(2) developed an administrative mechanism allowing affected persons to
seek the correction of information, (3) designated an official responsible
for ensuring NASS's compliance with OMB's requirements, and (4) reported
to OMB the number of complaints received regarding data quality they
received under the IQA.3

With respect to the transparency of the documentation underlying the
Census of Agriculture's data products and processes, as a result of our
review, NASS has taken steps to better document the criteria it uses to
evaluate data users' suggestions on the questionnaire content. The
development of the 2002 and 2007 Censuses was led by the Census Content
Team, which consisted of experienced NASS statisticians. The 2002 Team
assessed users' input using a documented set of criteria, which considered
such factors as whether the questionnaire items were mandated by Congress
or whether they would provide data on current agricultural issues.
However, because of staff turnover and reassignments, the 2007 Team was
unaware of the 2002 criteria, and initially relied on professional
judgment rather than documented factors to evaluate input on the 2007
Census. According to NASS, our review raised the 2007 Team's awareness of
the earlier criteria, and it has since developed similar documentation
that it will use in the future. This approach is more consistent with
NASS's own IQA guidelines concerning transparency, and could help create a
closer link between the questions included in the census and evolving
agricultural policy requirements, and thus a more cost-effective data
collection program. Documenting the content selection criteria will also
guard against the loss of institutional memory to the extent there is
further turnover in Content Team membership.

Building on these efforts, better documentation could improve the
transparency of census data products. Although NASS's guidelines for
ensuring transparency stress the importance of describing the methods,
data sources, assumptions, and other items in order to help users
understand how information was designed and produced, the eight census
reports as well as the FAQ section we examined lacked a discussion of such
important documentation practices as the limitations of the data; the

3NASS officials reported that they have not received any complaints under
the IQA.

impact of imputations, by item; and whether any of the collected data have
been suppressed for data quality reasons.

The transparency of NASS's procedures for handling data correction
requests filed outside of the IQA could also be improved. NASS complied
with OMB's requirement to establish a mechanism to process requests for
correction of disseminated information under the IQA. As part of this
process, an individual must state that their request is being submitted
under the IQA. To date, no individual has done so. NASS handles all other
correction requests using its existing informal, undocumented procedures.
Agency officials told us that it has resolved the informal requests it has
handled so far to the data users' satisfaction. Nevertheless, because NASS
does not document its informal procedures for handling correction requests
and lacks a recordkeeping system to log and track them, NASS could not
provide us with specific data on the number of requests it has handled,
the nature of those requests, and whether and how they were addressed.

To help enhance the transparency of the Census of Agriculture's processes
and products, we recommend that the Secretary of Agriculture direct NASS
to (1) ensure its products fully address its own requirements for
transparent data documentation or at least contain links to such
information and (2) document and post on its Web site its procedures for
handling data correction requests not filed under the IQA and track the
disposition of those requests.

The NASS Administrator provided written comments on a draft of this report
(see app. I). NASS said the information was insightful, and noted it will
be used to strengthen the transparency of its methods and procedures. In
particular, NASS agreed with our findings and, consistent with one of our
two recommendations, said it will take steps to better document its
specialized reports. NASS also said it plans to make a list of "common
issues" raised by data users available on its Web site, which is in line
with our second recommendation to improve the transparency of its
procedures for handling data correction requests not filed under the IQA.
NASS's commitment to continually improve its products is commendable, and
its efforts to improve the transparency of its processes and products
would be further enhanced if, consistent with our recommendations, it (1)
ensures that all of its census products fully address NASS's own
guidelines for data documentation and (2) posts on its Web site its
procedures for handling correction requests not filed under the IQA.

Background	The IQA directed OMB to issue guidelines to federal agencies
covered by the Paperwork Reduction Act designed to ensure the "quality,
objectivity, utility, and integrity" of information disseminated to the
public.4 The IQA also directed OMB to include in its guidelines
requirements for agencies to (1) develop their own information quality
guidelines, (2) establish administrative mechanisms for affected persons
to seek correction of information that does not comply with OMB's
guidelines, and (3) annually report to OMB the number and nature of
complaints they receive regarding the accuracy of the information they
disseminate.5

Prior to the IQA, there were several governmentwide actions aimed at
improving agency data. For example, Statistical Policy Directive No. 2,
first issued in 1952, required statistical agencies to inform users of
conceptual or other limitations of the data, including how the data
compare with similar statistics. In 1996, the Federal Committee on
Statistical Methodology-an OMB-sponsored interagency committee dedicated
to improving the quality of federal statistics-established a subcommittee
to review the measurement and reporting of data quality in federal data
collection programs. The results of the subcommittee's work were published
in a 2001 report that addressed such issues as what information on sources
of error federal data collection programs should provide, and how they
should provide it.6 For all federal government information collections,
the 1995 amendments to the Paperwork Reduction Act called on federal
agencies to manage information resources with the goal of improving "the
integrity, quality, and utility of information to all users within and
outside the agency."7

4Agencies covered by the IQA include all agencies subject to the Paperwork
Reduction Act (PRA) - cabinet departments, independent regulatory agencies
(e.g., Federal Communications Commission), and other independent agencies
(e.g., the Environmental Protection Agency). 44 U.S.C. S: 3502(1).

5Discussion of the IQA often centers on its impact on agencies' regulatory
activities. Supporters of IQA, many of whom represent businesses and other
regulated entities, maintain that IQA could enhance the quality of agency
science and improve the rule-making process. Critics of IQA, including
some environmental and public interest groups, view the law as a device to
curtail health, safety, and other regulations.

6OMB, Measuring and Reporting Sources of Error in Surveys, Statistical
Policy Working Paper 31, July 2001.

744 U.S.C. S: 3506(b)(1)(C).

OMB's IQA guidelines were issued in final form in February 2002.8 They
required agencies subject to the IQA to take such steps as

o 	issue information quality guidelines designed to ensure the quality,
objectivity, utility, and integrity of information disseminated to the
public;

o 	establish administrative mechanisms for affected persons to seek
correction of information they believe is not in compliance with the
guidelines;

o 	report annually to the Director of OMB on the number and nature of
complaints received regarding compliance with the guidelines and how the
agencies handled those complaints; and

o 	designate an official responsible for ensuring compliance with OMB's
guidelines.

The OMB guidelines defined quality as an encompassing term comprising

o  utility, which is the usefulness of the information to its intended
users;

o 	integrity, which refers to the security of information and its
protection from unauthorized access or revision; and

o 	objectivity, which addresses both presentation (i.e., whether the
information is being presented in an accurate, clear, complete, and
unbiased manner) and substance (i.e., whether the information is accurate,
reliable, and unbiased).

In addition, OMB addresses transparency within the definition of
objectivity and utility. As recognized in OMB's guidelines, agencies that
disseminate influential scientific, financial, or statistical information
must demonstrate a high degree of transparency about data and methods.
These measures are in place to facilitate the information's
reproducibility by an outside party or reanalysis of an agency's results.

The National Research Council of the National Academies considers
transparency a key principle for federal statistical agencies, and stated
in a

867 Fed. Reg. 8452 (Feb. 22, 2002).

recent report that transparency, which it defines as "an openness about
the sources and limitations of the data," is particularly important for
instilling credibility and trust among data users and providers. 9

As an agency within USDA, NASS is required to comply with the IQA. One
statistical program administered by NASS is the quinquennial Census of
Agriculture. According to NASS, the census provides a detailed picture of
U.S. farms and ranches every 5 years and is the only source of uniform,
comprehensive agricultural data at the county level. The results are
published in 18 reports divided among three categories: Geographic Area
Series, Census Quick Stats, and Specialty Products and Special Studies.
Users of this information include federal agencies (for program and
statistical purposes), farm organizations, businesses, universities, state
departments of agriculture, elected representatives, legislative bodies at
all levels of government, and academia. The next Census of Agriculture is
scheduled for 2007.

  Objectives, Scope, and Methodology

Our objectives were to (1) review how NASS met OMB's guidelines covering
the IQA and (2) examine the transparency of the documentation behind the
Census of Agriculture's processes and products, including the recently
completed work on the 2002 Census, and the efforts currently underway for
the 2007 Census.

To achieve both of these objectives, we reviewed OMB's and NASS's
information quality guidelines, Census of Agriculture reports,10
submissions to OMB, and other relevant documents. We also interviewed NASS
officials about how NASS conducted the 2002 Census and how it is planning
for the 2007 Census. The officials included the NASS Administrator,
Associate Administrator, and Deputy Administrator for Programs and
Products.

In addition, to evaluate the transparency of Census of Agriculture
products, we reviewed eight census reports and the Frequently Asked
Questions area of the 2002 Census Web site, to determine the extent to
which NASS

9Margaret E. Martin, Miron L. Straf, and Constance F. Citro eds.,
Principles and Practices for a Federal Statistical Agency, (3rd ed.)
(Washington, D.C.: National Academies Press, 2005), pp. 8, 29.

10Reports can be obtained from USDA's Web site; see www.usda.gov/nass.

followed its own procedures for ensuring the transparency of its
information products. NASS's IQA guidelines define transparency as, "a
clear description of the methods, data sources, assumptions, outcomes, and
related information that allows a data user to understand how an
information product was designed and produced."

NASS's guidelines state that its survey activities include such activities
as sample design, questionnaire design, pre-testing, analysis of sampling,
and imputation of missing data. However, the guidelines were not clear as
to the specific activities to be documented. Consequently, we reviewed the
practices employed by such statistical agencies as the National Academies
of Sciences, International Monetary Fund, and U.S. Census Bureau, and
developed a set of 20 practices associated with transparent documentation
that encompassed the items NASS laid out in its own guidelines. The
practices include such actions as defining data items, discussing sample
design, and describing how the content of the survey differs from past
iterations (see app. II).

We looked for the presence or absence of these practices in 9 out of the
18 census reports and related forms of data that NASS disseminates, and
verified the results with a second, independent analysis. In instances
where a report did not include a particular documentation practice, we
reviewed whether the report instead informed data users where to obtain
this information. We chose these 9 reports because they all stem from the
original census data collection, represent different product categories,
and were available on the census Web site as of February 1, 2005.

To obtain an external perspective of how NASS processes and products
address the IQA guidelines, we interviewed six data users from different
types of agricultural and research organizations. We selected these data
users from lists of registrants for USDA and NASS outreach meetings within
the past 5 years. We selected these six data users because they use
information from the census on a regular basis. Moreover, these data users
attended the most recent NASS outreach meeting, which specifically
addressed the 2002 and 2007 Censuses. Some data users had also provided
NASS with feedback on the content of the agricultural census. Their views
cannot be projected to the larger population of census data users.

We requested comments on a draft of this report from the Secretary of
Agriculture. On September 8, 2005, we received the NASS Administrator's
written comments and have reprinted them in appendix I. They are

addressed in the Agency Comments and Our Evaluation section of this
report.

NASS Met the NASS fulfilled the various procedural responsibilities and
reporting

requirements under OMB's guidelines. For example, NASS released its
ownProcedural and IQA guidelines for public comment on March 27, 2002.
NASS officials Reporting stated they received no substantive comments on
them and OMB approved Requirements of OMB's the guidelines with only
minimal changes. The officials also noted that no

revisions have been made since then. Table 1 shows in greater detail
howIQA Guidelines NASS addressed OMB's guidelines.

Table 1: NASS Addressed OMB's Agencywide Guidelines for Implementing IQA

OMB directed agencies to: NASS's response

o  Prepare a draft report explaining how  o  NASS posted its draft report
and guidelines to their guidelines will ensure and its Web site from March
27, 2002, until maximize the quality of information. September 30, 2002.

o  Publish a notice in the Federal  o  Notice of availability was
published in the June Register announcing the availability of 4, 2002,
Federal Register a and the report itself this report on the agency's Web
site was available on NASS's Web site. for public comment.

o  Post their final report and guidelines  o  Final guidelines are on the
NASS Web site. to their Web sites.

o  Develop administrative mechanisms  o  NASS outlines its correction
procedures in allowing affected persons to correct detail on its Web site.
disseminated information that does not comply with the OMB guidelines.

o  Submit a report by January 1 of each  o  USDA submitted a report to OMB
both years year on the number and nature of since the guidelines have been
in effect; NASS complaints received in the prior fiscal reported no
complaints. year.

o  Designate an official to be responsible  o  USDA has designated its
Chief Information for the agency's compliance with Officer (CIO) as this
official. The CIO in turn OMB's guidelines. delegates compliance questions
to lower-level offices, including the Standards Officer within the office
of the Associate Administrator at NASS.

Source: GAO analysis of OMB and NASS documents.

a67 Fed. Reg. 38,467.

  Better Documentation Could Improve the Transparency of Data Products and
  Correction Procedures

NASS's IQA guidelines define transparency as, "a clear description of the
methods, data sources, assumptions, outcomes, and related information that
allows a data user to understand how an information product was designed
and produced." NASS's guidelines also note that "NASS will make the
methods used to produce information as transparent as possible" and that
its "internal guidelines call for clear documentation of data and methods
used in producing estimates and forecasts. . . ."

To assess the extent to which NASS processes help ensure the transparency
of the information it publishes, we examined key publications from the
2002 Census of Agriculture. Census reports vary in terms of scope and
intended audience (see table 2). On the one hand, the United States
Summary and State Data report contains over 100 data tables, an
introduction, and four appendices. On the other hand, County Profile
reports summarize each county's agricultural situation on two pages.

Overall, we assessed eight census reports within three product categories,
as well as the Frequently Asked Questions (FAQ) section of the 2002 Census
Web site, to determine the extent to which NASS followed its own
guidelines for ensuring the transparency of its products. As shown in
table 2, the transparency of the data documentation in the reports we
reviewed varied between the Geographic Area Series reports-which are the
most comprehensive of NASS's products and addressed 15 of the 20 data
documentation practices-and the Specialty Products and Special Studies
which, depending on the specific product, addressed no more than 1 of the
practices.

             Table 2: Census Reports Need More Robust Documentation

                                                                Portion of 20 
                                                                documentation 
                                                                    practices 
    Product category    Product title    General description        addressed 

     Geographic Area   United States Summary and  Contains over 100 national  
                                                    and state data tables.    
         Series               State Data         
                         State and County Data   Contains over 100 state and  
                                                     county data tables.      
                       2002 Census of              A database for users to    
Census Quick Stats: Agriculture                download and generate data  
Ag Statistics Data  Downloadable Application    tables at the national,    
                                                  state, and county levels.   
          Base                                   

Specialty Products and State and County Profiles Two-page reports
containing summary data about a Special Studies state or county.

Quick Facts from the 2002 This report presents national data in 16 charts
or Census of Agriculture graphs.

Ranking of Market Value of This report contains state tables that rank the

Agricultural Products Soldagricultural products sold by market value. The
report also includes table definitions.

Congressional District ProfilesEach profile is a two-page report that
contains summary data about one congressional district.

Ranking of Congressional Contains tables for 46 data items, such as number
of

Districtsfarms, and ranks the congressional districts for each of these
data items.

Additional Information Frequently Asked QuestionsThis section of the
census Web site contains questions and answers grouped into four
categories.

Source: GAO analysis of NASS reports.

All eight reports and the FAQ Web site lacked a discussion of four
documentation practices, including the following:

1.	Questionnaire testing. NASS produced a separate, internal report that
discusses questionnaire testing in detail; however, publicly available
census publications do not address this topic.

2.	Limitations of the data. NASS does not discuss data limitations in the
census reports we reviewed.

3.	Impact of imputations, by item. When a statistical agency receives a
report form with missing values, it normally estimates or "imputes" those
values based on comparable data sources such as a similar farm operation.
Although NASS uses a complex editing and imputation process to estimate
missing values, and describes this process in the

United States Summary and State Data report appendix, it does not quantify
the impact of imputations by item in reports.

4.	Whether any of the collected data have been suppressed for data quality
reasons. Without information on whether any of the data had been
suppressed because the quality was lacking, data users must assume that
reports include all data items collected in the census had met agency
publication standards.

Although NASS appropriately recognizes the variation in data user needs by
publishing several types of specialized reports, none of the reports we
reviewed direct data users where to find either a complete set of
documentation or additional documentation. For example, given the short
length and summary format of the County Profile reports, it is not
surprising that they lack documentation. However, in order for users to
assess the quality of the data contained in the reports, it is important
for NASS to at least provide links on its Web site or to other
publications where users can access definitions, response rates, and other
relevant information.

    NASS Should Document Its Procedures for Handling Correction Requests Not
    Filed under the IQA

NASS has two methods for handling data correction requests, depending on
how they are submitted: a formal approach prescribed by OMB for correction
requests filed under IQA, and an informal approach that NASS uses to
address correction requests that are not filed under IQA. NASS's informal
correction procedures lack transparency because they are not documented
and individual cases are not tracked. As a result, we could not determine
the nature of these correction requests or whether or how they were
addressed.

Consistent with OMB's guidelines, NASS detailed its procedures to request
corrections under IQA on its Web site, and posted appropriate Federal
Register notices. For example, NASS's Web site explains that to seek a
correction under IQA, petitioners must, among other steps: (1) state that
their request for correction is being submitted under IQA, (2) clearly
identify the information they believe to be in error, and (3) describe
which aspects of NASS's IQA guidelines were not followed or were
insufficient.11

11See www.usda.gov/nassinfo/infocorrection.htm.

According to the instructions posted on its Web site, NASS's IQA
procedures are triggered only when petitioners explicitly state they are
submitting a correction request under IQA. To date, none have done so.
NASS addresses all other correction requests using informal, undocumented
procedures that were in place before IQA was enacted. NASS officials
explained that such requests are forwarded to the agency official
responsible for preparing the report containing the information in
question. That official, in turn, determines if the request can be
resolved by clarifying the data, or whether a correction is needed. If a
data item needs to be corrected, NASS has a set of procedures for
documenting errors and issuing errata reports that are detailed in its
Policy and Standards Memorandum No. 38. The memorandum describes the
circumstances under which errata reports will be printed, and provides a
mechanism for NASS staff to describe the nature of the error, its cause,
and the action taken to resolve it.

According to the Administrator, Associate Administrator, and other senior
NASS officials we interviewed, the requests it has handled from the 2002
Census have so far been resolved to the petitioners' satisfaction, and
none resulted in any corrections to the data from the 2002 Census.
However, because NASS does not document its informal procedures for
handling inquiries and data correction requests, and lacks a recordkeeping
system to log and track them, NASS could not provide us with firm
information on the number of inquiries it has handled, the nature of those
inquiries, and whether and how they were addressed.

This is not to say that all complaints should follow the same procedures
required by the IQA mechanism. For efficiency's sake, it is important for
agencies to respond to complaints in accordance with the magnitude of the
problem. However, to provide a more complete picture of the questions NASS
receives about its data and how those questions were handled, it will be
important for NASS to better document its approach for handling correction
requests not filed under IQA, and track their disposition.

NASS Has Taken Steps to The 2002 Census of Agriculture was the first in
which NASS developed the Better Document Its Criteria questionnaire (the
1997 Census of Agriculture was moved from the Census for Assessing Input
on Bureau to NASS after the content had been determined). In doing so,
NASS

went to great lengths to obtain input from data users on what questions to

Census Content	ask, and evaluated their suggestions using a documented set
of criteria. In preparing for the 2007 Census, NASS sought feedback on the
questionnaire content from a broader spectrum of data users, in part
because NASS

solicited suggestions via the Internet. However, unlike the 2002 cycle,
the criteria NASS used to assess the feedback were not initially
documented, which is contrary to NASS's IQA guidelines. However, as a
result of our review, NASS has developed documented criteria similar to
that used during the previous census.

Under the Paperwork Reduction Act, agencies must obtain OMB's approval
prior to collecting information from the public. As part of this process,
agencies must certify to OMB that, among other things, the effort is
necessary for the proper performance of agency functions, avoids
unnecessary duplication, and reduces burden on small entities. Agencies
must also provide an estimate of the burden the information collection
would place on respondents.12

For the 2002 Census, NASS submitted its request for approval-a form called
"OMB 83-I"-in August 2001, and OMB approved it in October 2001. NASS
estimated that the census would require a cumulative total of more than
1.3 million hours for respondents to complete and would cost them, in
terms of their time, in excess of $21 million.

OMB's approval process also requires agencies to solicit input from
external sources. NASS obtained input on the 2002 Agricultural Census
content through a Federal Register notice, meetings with data users, and
by contacting federal and state agencies that use census statistics to
discuss data needs.

Likewise, NASS is obtaining input on the content of the 2007 Census
through a variety of channels. According to an agency official, the
process began around June 2004, when NASS began releasing publications
from the 2002 Census. NASS sent an evaluation form to its state offices
requesting feedback on the census, including their suggestions for
changing the content. NASS also asked the state offices to identify users
from whom it could obtain additional feedback.

NASS solicited further input by reaching out to data users within USDA and
other federal agencies, querying organizations included in a list of
"typical" data users maintained by NASS's Marketing and Information
Services Office, and holding periodic regional meetings with data users.
NASS also

1244 U.S.C. S: 3506(c).

has a "hot button" on its Web site where visitors are asked what items, if
any, should be added or deleted from the census.13

In all, NASS obtained input on the 2007 Census through 10 distinct
conduits. Moreover, compared to the process used to develop the content of
the 2002 Census, its 2007 efforts were open to a wider spectrum of
customers, and involved more direct contact with data users during the
planning phase. Indeed, as shown in table 3, NASS's outreach via the
Internet, regional meetings, and queries to data users was over and above
the steps it took when developing the 2002 Census. This openness was
reflected in the comments of the six data users we interviewed. Five of
the six users said NASS's approach to eliciting input was adequate, while
three of the six had requested new content items for the 2007 Census to
better meet the needs of their organizations.

The content evaluation process began in December 2004, and NASS is
currently testing the questionnaire content. Following any refinements,
mail-out of the actual census is scheduled for December 2007.

13See http://www.nass.usda.gov/census/census02/feedback.htm.

Table 3: NASS Is Using More Extensive Outreach to Develop the 2007 Census
Compared to 2002

                   Method of outreach 2002 Census 2007 Census

Posted Federal Register notices X X

Solicited input from state agricultural statistical offices X X

Solicited input from state governors X

Solicited input from Advisory Committee on Agricultural Statistics X X

Solicited input from land grant universities X X

Solicited input from federal data users X X

Held federal data user working group meetings X X

Held USDA-wide national data user outreach meeting X X

Solicited input from a list of "typical" census users maintained by NASS's
Marketing and Information X Services Office

Solicited input via Web site feedback form X

Held NASS-specific, regional data user meeting X

Source: GAO analysis of NASS data.

For both the 2002 and 2007 Census cycles, the solicitation, review, and
ultimate determination of the questionnaire content was led by the Census
Content Team, a group consisting of experienced NASS statisticians
representing different segments of the agency such as livestock, crops,
and marketing. The 2002 Content Team used specific, documented criteria to
inform its decisions. Specifically, suggestions were assessed according to
the following factors, which were also made available to data users:

o 	items directly mandated by Congress or items that had strong
congressional support;

o 	items proposed by other federal agencies where legislation called for
that agency to provide data for Congress;

o  items needed for evaluation of existing federal programs;

o 	items which, if omitted, would result in additional respondent burden
and cost for a new survey for other agencies or users;

o  items required for classification of farms by historical groupings;

o  items needed for improving coverage in the census; and

o  items that would provide data on current agricultural issues.

However, the criteria the 2007 Team used to assess input on the
questionnaire content were not initially documented. According to agency
officials we interviewed, NASS largely relied on professional judgment to
evaluate the feedback it received, considering such factors as the need to
keep the data comparable to past censuses and not increase the length of
the questionnaire.

Although a certain amount of professional judgment will invariably be used
in making determinations on questionnaire content, the absence of
documented assessment criteria is inconsistent with NASS's guidelines.
Indeed, these guidelines note that transparent documentation "allows a
data user to understand how an information product was designed and
produced." Moreover, without documented criteria, it is not clear whether
members of the Content Team are considering the same set of factors, or
even if they are weighing those factors in the same manner.

According to NASS, the shift in approach stemmed from staff turnover and
reassignments of members of the 2002 Team and, as a result, the 2007 Team
was not aware of the criteria used in 2002. Our review made the 2007 Team
aware of the earlier set of criteria, and the Team has since developed
similar documentation. NASS noted that all future content teams will use
and update these criteria when developing the content of subsequent
censuses.

It will be important for NASS to continue with this approach because it is
more consistent with its own IQA guidelines, and will also help NASS to do
the following:

Ensure the utility and relevance of information. A key principle for
federal statistical agencies is to provide information relevant to issues
of public policy.14 However, the nation's information needs are constantly
evolving, and it is important for statistical agencies to adapt
accordingly. This is particularly true with agriculture, where a variety
of factors such as changing technology and agricultural trends can affect
what information should be collected. Rigorous content selection criteria
could help NASS

14Margaret E. Martin, Miron L. Straf, and Constance F. Citro eds.,
Principles and Practices for a Federal Statistical Agency, (3rd ed.)
(Washington, D.C.: National Academies Press, 2005), p. 4.

methodically evaluate the needs of different users, establish priorities,
and keep the census synchronized with changing public policy requirements.

Maximize cost-effectiveness and reduce public burden. As with all federal
surveys, there are financial and nonfinancial costs to conducting the
Census of Agriculture. These costs include the direct expenditures related
to planning, implementing, and analyzing the census, as well as
disseminating the information. There is also a cost to respondents in
terms of the time they take to complete the questionnaire. Additionally,
there are opportunity costs in that for every question that is included in
the census, another question might need to be excluded so as not to
increase the length of the census. Rigorous, consistently applied criteria
can help promote cost-effectiveness because they can ensure that only
those questions that meet a particular, previously identified need are
included in the census. Applying such criteria also help inform decisions
on the appropriate role of the federal government in collecting the data,
and whether a particular question might be more appropriately addressed by
a different survey, government organization, or the by the private sector.

Maintain credibility. Content selection criteria provide a basis for
consistent decision making on what to include in the census and what gets
left off. This is especially important for maintaining NASS's credibility
given the input it receives from various sources. Without documented
criteria, NASS's actions could be perceived as arbitrary or
disproportionately swayed by one particular interest or another; thus,
NASS's decisions would be more defensible.

Further, documented criteria will guard against the loss of institutional
memory to the extent there is further turnover in Content Team membership.

Conclusions	NASS satisfied the procedural responsibilities and reporting
requirements under OMB's IQA guidelines. Moreover, to the extent that NASS
continues to use the documented criteria it developed to inform future
decisions on the content of the Census of Agriculture, it could help
establish a closer alignment between the questions included in the census
and evolving agricultural policy requirements, resulting in a more
cost-effective data collection program.

Building on these efforts, the transparency of census data products could
be improved with more robust documentation. NASS's procedures for

addressing correction requests not filed under IQA could be more
transparent as well. More than just a paperwork issue, greater
transparency will help enhance NASS's accountability to public data users
and increase the credibility of census information.

Recommendations for 	To help enhance the transparency of the Census of
Agriculture's processes and products, we recommend that the Secretary of
Agriculture direct NASS

Executive Action to take the following two steps:

1.

2.

Ensure that census products fully address NASS's own guidelines for data
documentation or at least contain links to such information. The list of
20 documentation practices that we developed, while not necessarily
exhaustive, represents sound actions used by other statistical agencies
and could form a starting point for NASS.

Document and post on NASS's Web site its procedures for handling data
correction requests not filed under IQA, and track the disposition of
those requests.

  Agency Comments and Our Evaluation

The NASS Administrator provided written comments on a draft of this report
on September 8, 2005, which are reprinted in appendix I. NASS noted that
our "report and recommendations are insightful and will be used to further
strengthen the transparency of NASS methods and procedures."

In particular, NASS concurred with our finding that the methods and
procedures in its specialized reports should be better documented and,
consistent with our recommendation, stated that these products "will now
provide links to this information." NASS's efforts, if fully implemented,
should make it easier for data users to understand how these products were
designed and produced, and NASS should be commended for its actions to
continually improve its products and better meet the needs of its
customers.

While NASS's more comprehensive products were better documented, our
analysis found that they could also benefit from more robust
documentation. Thus, in keeping with our recommendation, it will be
important for NASS to ensure that all of its census products-its larger
reports and more focused studies--fully address NASS's own guidelines for
data documentation.

In commenting on our recommendation for NASS to document and post on its
Web site its procedures for handling data correction requests not filed
under IQA, NASS concurred with our view that this information would
provide it with a better sense of the questions it receives about its
data, but added that "a detailed recordkeeping system to log and track
every inquiry" would not be the best use of its resources. Instead, NASS
plans to "compile a listing of the more common issues" and make them
available on its Web site in the form of frequently asked questions. NASS
believes this approach would be useful for future planning, as well as
provide answers to questions most likely to arise among other data users.

As noted in our report, our recommendation stemmed from our finding that
NASS could not provide us with information on the number of inquiries not
filed under IQA, the characteristics of those inquiries, and how they were
addressed. Although the details remain to be seen, NASS's proposed
approach could provide this information and, consistent with the intended
outcome our recommendation, address the need for greater transparency.
NASS's efforts will be further strengthened if, consistent with our
recommendation, it posts on its Web site its procedures for handling
correction requests not filed under IQA.

We will send copies of this report to other interested congressional
parties,
the Secretary of Agriculture, and the NASS Administrator. Copies will be
made available to others on request. This report will also be available at
no
charge on GAO's Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact
me
at (202) 512-6806 or [email protected]. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last page
of this report. GAO staff who made major contributions to this report are
listed in appendix III.

Orice M. Williams
Director
Financial Markets and

Community Investments

Appendix I

Comments from the Department of Agriculture

Appendix I
Comments from the Department of
Agriculture

Appendix II

How Census of Agriculture Reports Address Various Documentation Elements

                                 Product titles

United                                         Ranking: 
States          State and   Quick                Market 
Summary       County        Stats: Ag          Value Ag 
and State     Reports       Statistics         Products 
Report        (Alabama)     Database               Sold 

State and County Profiles Congress. District Profiles Ranking of Congress.
Districts Frequently Asked Questions Documentation practices

                                  Quick Facts

Discussion of how NASS developed the content of the census.

                                     XXX X

         Discussion of             X           X           X                X 
          2002 Census                                             
            content                                               
        consistency with                                          
             1997.                                                

        Description of X         X       X                                  X 
        how the content                                               
            differs.                                                  
        Discussion of X          X       X                            
         why NASS made                                                
        content changes.                                              
      Definition of the X        X       X          X                       X 
          population.                                                 
      Definition of data X       X       X               X          X 
             items.                                                   
        Discussion of X          X       X                            
         sample design.                                               
        Discussion of X          X       X                            
         questionnaire                                                
            design.                                                   

Discussion of questionnaire testing.

Copy of the         X       X   
questionnaire.                  
Discussion of       X       X      X 
data collection                 
procedures.                     
Discussion of       X       X      X 
nonresponse                     
followup.                       
Discussion of       X       X      X 
data entry                      
procedures.                     

                                  Appendix II
                   How Census of Agriculture Reports Address
                         Various Documentation Elements

(Continued From Previous Page)

Product titles

              United                                      Ranking:                     
              States  State and   Quick                    Market                      
                                           State                             Ranking              
              Summary  County   Stats: Ag  and            Value Ag Congress. of        Frequently
Documentation and      Reports  Statistics  County  Quick Products District  Congress.   Asked    
              State                                                                    
  practices   Report  (Alabama) Database   Profiles Facts   Sold   Profiles  Districts  Questions 
Discussion of    X        X         X                                                  
data editing                                                                           
 procedures.                                                                           

Quantify the impact of imputations, by item.

         Discussion of             X           X           X                X 
        response rates.                                           
         Discussion of             X           X           X      
          nonsampling                                             
             error.                                               

Discussion of the limitations of the data.

Discussion of whether any of the collected data have been suppressed for
data quality reasons.

Comparison of X
census results to
other survey
results for
consistency.

Source: GAO analysis of NASS data.

Note: See the Objectives, Scope, and Methodology section of this report
for a complete explanation of our analysis.

Appendix III

                     GAO Contact and Staff Acknowledgments

GAO Contact Orice M. Williams, (202) 512-6806

Acknowledgments	In addition to the contact named above, Robert Goldenkoff,
Assistant Director; David Bobruff; Jennifer Cook; Richard Donaldson;
Andrea Levine; Robert Parker; John Smale; and Michael Volpe made key
contributions to this report.

GAO's Mission	The Government Accountability Office, the audit, evaluation
and investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost

is through GAO's Web site (www.gao.gov). Each weekday, GAO postsGAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To Testimony have GAO e-mail you a list of newly posted products
every afternoon, go to

www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone	The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: 	Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

  To Report Fraud, Contact:
  Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470

Congressional 	Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125

Relations Washington, D.C. 20548

Public Affairs	Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***