Student Testing: Issues Related to Voluntary National Mathematics and
Reading Tests (Letter Report, 06/18/98, GAO/HEHS-98-163).

Pursuant to a congressional request, GAO reviewed the Department of
Education's plan to develop voluntary national tests for fourth grade
reading and eighth grade mathematics, focusing on: (1) the relationship
between the National Assessment Governing Board and Education; (2) costs
of developing the national voluntary tests and procedures for hiring
contractors; and (3) possible explanations for the differences in scores
on state achievement tests and National Assessment of Educational
Progress (NAEP) tests.

GAO noted that: (1) since 1992, the Governing Board and Education have
operated under a Memorandum of Understanding based on NAEP's authorizing
legislation and designed to ensure the Governing Board's independence
from its parent agency in fulfilling its responsibilities for NAEP; (2)
in accordance with this memorandum, the Governing Board establishes
program policies and standards, oversees contracts to develop test
specifications, makes its own personnel decisions, obligates funds, and
awards contracts; (3) because they share responsibilities for the
program, however, Education and the Governing Board collaborate
extensively; (4) although the Governing Board helps in developing
Education's budget request for NAEP-related work, Education has final
authority over the request forwarded to the Office of Management and
Budget; (5) when Congress gave the Governing Board exclusive authority
for the voluntary national test development contract in November 1997,
it also altered the relationship between the Governing Board and
Education regarding this contract; (6) because it now had exclusive
authority for this development contract, the Governing Board used no
Education assistance in reviewing the contract as it had typically done
for NAEP-related contracts; (7) as a result of its review, the Governing
Board substantially changed the test development contract; (8) another
modification specified that the Governing Board, not Education, would
make all policy decisions as specified in the original contract; (9) nor
did the Governing Board help in developing Education's budget request
for voluntary national test development funds; (10) on reviewing the
Governing Board's test development contract, GAO estimated that the cost
to the federal government for developing one complete set of these tests
would be $15 million; (11) the cost of implementing the fourth grade
reading and eighth grade mathematics testing program has been estimated
at up to $96 million if all fourth and eighth grade students in the
public and private schools participated; (12) most explanations of the
differences in scores on the NAEP and state achievement tests fell into
three main categories: (a) differences in how the tests define
proficiency; (b) differences in the kinds of tests administered; and (c)
differences in students taking the test and when they take it; and (13)
NAEP, for example, tests students with disabilities or limited-English
proficiency; some states may not.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  HEHS-98-163
     TITLE:  Student Testing: Issues Related to Voluntary National 
             Mathematics and Reading Tests
      DATE:  06/18/98
   SUBJECT:  Educational testing
             Mathematics
             Secondary school students
             Elementary school students
             Academic achievement
             Education program evaluation
             Educational standards
             Educational research
             Interagency relations
             Literacy
IDENTIFIER:  Dept. of Education National Assessment of Educational 
             Progress
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Honorable
William F.  Goodling, Chairman, Committee on Education and the
Workforce, House of Representatives, and The Honorable John Ashcroft,
U.  S.  Senate

June 1998

STUDENT TESTING - ISSUES RELATED
TO VOLUNTARY NATIONAL MATHEMATICS
AND READING TESTS

GAO/HEHS-98-163

Student Testing

(104917)


Abbreviations
=============================================================== ABBREV

  AIR - American Institutes for Research
  FAR - Federal Acquisition Regulation
  FIE - Fund for the Improvement of Education
  NAE - National Academy of Education
  NAEP - National Assessment of Educational Progress
  NAS - National Academy of Sciences
  NCES - National Center for Education Statistics
  NCEST - National Council on Education Standards and Testing
  OERI - Office of Educational Research and Improvement
  OMB - Office of Management and Budget
  RFP - request for proposal

Letter
=============================================================== LETTER


B-278968

June 18, 1998

The Honorable William F.  Goodling
Chairman, Committee on Education and the Workforce
House of Representatives

The Honorable John Ashcroft
United States Senate

In February 1997, the Clinton administration announced a plan to
develop voluntary national tests for fourth grade reading and eighth
grade mathematics.  The tests were proposed as a way to provide
parents and teachers with information about their students'
performance relative to widely agreed-upon standards of what students
should know and be able to do.  Concerns have been raised, however,
about the need for such tests and the potential for inappropriate
federal influence on school curriculum, which is established by the
states and localities. 

Originally, the Department of Education had complete responsibility
for the testing initiative.  In November 1997, however, the
Department's Appropriations Act transferred responsibility for
developing the tests from Education to the National Assessment
Governing Board (referred to as the "Governing Board"),\1

a board originally created as part of the Education Department in
1988 to set policy for a program called the National Assessment of
Educational Progress (NAEP).  In addition, the Congress directed the
Department, including the Governing Board, to use no fiscal year 1998
funds to, among other things, pilot test, field test, or administer
these tests. 

The proposed voluntary national tests are designed to be based on
NAEP.  As part of your ongoing review of the proposed testing
program, you asked us to provide you with information about three
main issues: 

  -- the relationship between the Governing Board and the Department
     of Education,

  -- costs of developing the national voluntary tests and procedures
     for hiring contractors, and

  -- possible explanations for the differences in scores on state
     achievement tests and NAEP tests. 

To address the first two issues and other specific questions (see
app.  I), we interviewed officials at both the Governing Board and
the Department of Education, reviewed contract files, and analyzed
contracts and other relevant documents.  To determine reasons for
differences between state and NAEP test results, we interviewed
testing experts with the Governing Board, the Department, and the
National Center for Research on Evaluation Standards on Student
Testing and reviewed selected literature on testing standards.  We
also observed a Governing Board meeting and explored whether outside
advisers to the voluntary national test specification process had
written textbooks or educational materials for elementary and
secondary education classrooms.  We conducted our work between
January and May 1998 in accordance with generally accepted government
auditing standards. 


--------------------
\1 Departments of Labor, Health, Human Services, and Education and
Related Agencies Appropriations Act, 1998.  P.L.  105-78, sec.  307
(a), 111 Stat.  1467, 1505 (1997). 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :1

Since 1992, the Governing Board and the Department of Education have
operated under a Memorandum of Understanding based on NAEP's
authorizing legislation and designed to ensure the Governing Board's
independence from its parent agency in fulfilling its
responsibilities for NAEP, a national assessment program.  In
accordance with this memorandum, the Governing Board establishes
program policies and standards, oversees contracts to develop test
specifications, makes its own personnel decisions, obligates funds,
and awards contracts.  Because they share responsibilities for the
program, however, the Department and the Governing Board collaborate
extensively.  Although the Governing Board helps in developing the
Department's budget request for NAEP-
related work, the Department has final authority over the request
forwarded to the Office of Management and Budget (OMB). 

When the Congress gave the Governing Board exclusive authority for
the voluntary national test development contract in November 1997, it
also altered the relationship between the Governing Board and the
Department regarding this contract.  Because it now had exclusive
authority for this development contract, the Governing Board used no
Department assistance in reviewing the contract as it had typically
done for NAEP-
related contracts.  As a result of its review, the Governing Board
substantially changed the test development contract.  For example,
under the Department's contract, the planned administration date of
the first test was March 1999; under the Governing Board's contract,
the planned administration date is March 2001.  Another modification
specified that the Governing Board, not the Department, would make
all policy decisions as specified in the original contract.  Nor did
the Governing Board help in developing the Department's budget
request for voluntary national test development funds. 

On reviewing the Governing Board's test development contract, we
estimated that the cost to the federal government for developing one
complete set of tests would be $15 million.  The cost of implementing
the fourth grade reading and eighth grade mathematics testing program
has been estimated at up to $96 million if all fourth and eighth
grade students in public and private schools participated.  Who would
pay for implementing the test has not been determined.  For fiscal
years 1997 and 1998, contracts awarded for voluntary national tests
totaled about $17 million.  Most of this amount--about $13
million--was for the 1997 contract for developing these tests;
however, only about $1.7 million of this contract award had been paid
to the contractor as of April 1998.  In keeping with the
congressional mandate to restrict the use of fiscal year 1998 funds,
the only contract awarded with these funds was for a new
congressionally mandated study by the National Academy of Sciences
(NAS), which included examining the feasibility of comparing
commercial and state tests with NAEP tests.  In addition to those
expenditures for which cost figures were available, the Department
also spent an undetermined amount on distributing information about
the national tests in fiscal years 1997 and 1998 by using web sites,
press releases, articles in trade magazines, Department newsletters,
and public presentations.  Department officials said these
dissemination activities are not directly related to developing the
test and were part of the Department's ongoing function to educate
the public about its program initiatives. 

We heard various explanations for the differences in scores on the
NAEP and state achievement tests; far fewer students achieve
proficiency on recent NAEP tests than on state tests.  Most
explanations of the differences fell into three main categories:  (1)
differences in how the tests define proficiency and determine the
threshold test score for proficiency (for example, one test may
define proficiency as getting 66 percent of all items correct;
another might define proficiency as scoring a minimum of 66 percent
on each test section); (2) differences in the kinds of tests
administered (for example, multiple choice versus problem solving);
and (3) differences in students taking the test and when they take
it.  NAEP, for example, tests students with disabilities or
limited-English proficiency; some state tests may not. 


   BACKGROUND
------------------------------------------------------------ Letter :2

Created by the Congress in 1988, the Governing Board provides
independent, bipartisan policy guidance for NAEP, a congressionally
authorized program in place since 1969.  The Secretary of Education
appoints the 25 voting members of the Governing Board to represent a
wide range of legislatively prescribed categories, including two
state governors from different political parties, a school board
member, a school district superintendent, classroom teachers, a
business or industry representative, and curriculum specialists.  The
Department's Assistant Secretary for Educational Research and
Improvement serves as a nonvoting, ex officio member. 

The Governing Board has joint responsibilities with the Department of
Education for NAEP.  The Governing Board sets policy for NAEP, and
the Department's National Center for Education Statistics (NCES)
oversees the development and implementation of the assessment
according to the Governing Board's guidance.  The NCES commissioner
is responsible for NAEP operations and technical quality control.  In
November 1997, the Congress gave the Governing Board exclusive
authority--separate from the Department--for the voluntary national
test development contract. 


      THE GOVERNING BOARD AND THE
      DEPARTMENT SHARE
      RESPONSIBILITY FOR NAEP
---------------------------------------------------------- Letter :2.1

NAEP primarily tracks the educational achievements of students in
grades 4, 8, and 12 in selected subject areas such as reading,
writing, mathematics, and U.S.  history.\2 The Governing Board and
the Department share responsibility for developing and implementing
each test. 

For example, the Governing Board selects the subjects for each
assessment cycle\3 and oversees the creation of the conceptual
frameworks for the tests and the specifications for developing the
tests.  The frameworks identify subject-specific content areas and
specify such test attributes as the cognitive skills to be tested and
the type and number of test items in a particular content area.  The
frameworks are developed through a national consensus process that
involves educators, academicians, policymakers, the general public,
and others.  NCES in turn hires contractors who use these frameworks
to develop the actual assessments (which includes writing and pilot
testing the test items and field testing the test booklet) and
administer them.  The Governing Board has final approval over each
NAEP test item.  Participation of students, schools, districts, and
states in the assessment is voluntary.\4


--------------------
\2 Thus far, the Governing Board has overseen the creation of
assessment frameworks for reading, writing, mathematics, civics,
arts, science, geography, and U.S.  history.  More frameworks for
additional subject areas are planned. 

\3 Since 1990, various state and national NAEP assessments have been
administered every 2 years, according to a schedule determined by the
Governing Board. 

\4 In 1994, 39 states participated in the fourth grade reading
state-level assessment and met the minimum school participation
guidelines for public schools.  In 1996, 40 states participated in
the eighth grade state-level mathematics assessment and met the
minimum guidelines. 


      NAEP GIVES STATE AND
      NATIONAL--NOT
      INDIVIDUAL--RESULTS
---------------------------------------------------------- Letter :2.2

NAEP does not report on the performance of individual students or
individual schools.  It tests samples of students and reports the
overall performance of student populations at the state and national
levels.  As required in its authorizing legislation, the Governing
Board has developed student performance levels for each age and grade
in each subject area tested.  Starting with the 1990 mathematics
assessment, the Governing Board has reported results using
performance standards that define what students in a particular grade
level should know and be able to do in each subject area tested. 
Student performance on the assessment is categorized as basic,
proficient, or advanced.\5


--------------------
\5 "Basic" denotes partial mastery of prerequisite knowledge and
skills for proficient work, "proficient" denotes solid academic
performance that demonstrates competency in challenging subject
matter, and "advanced" denotes superior performance beyond proficient
grade-level mastery. 


      NAEP STANDARDS ARE HIGH AND
      UNDER SCRUTINY
---------------------------------------------------------- Letter :2.3

Experts generally consider NAEP assessments to be challenging, with
high standards for achieving proficiency.  In the 1994 NAEP reading
test--the most recent one to measure reading achievement--only 36
percent of the nation's 12th graders were considered to be at or
above the proficient level.  Although the comparability of NAEP
assessments with state assessments is being studied, state
achievement tests usually result in a much larger percentage of
students achieving proficiency. 

Given the national significance of NAEP tests, the Governing Board's
process for developing the performance standards has been widely
scrutinized and evaluated.  Reviews of the process have been done by
NCES, research firms under contract either to the Governing Board or
the Department, and other independent groups.  For example, we
previously studied the Governing Board's 1990 standard-setting
process,\6 and the National Academy of Education (NAE) studied mainly
the 1992 process.\7 Both of these reviews disagreed with several
aspects of the Governing Board's approach, which used a well-known
standard-setting process with some novel modifications.  The
Governing Board considered these views along with several other
diverse points of view\8 and changed its standard-
setting process but kept its basic approach.  The Governing Board
plans to use the same approach for setting performance standards for
the voluntary national tests as for the NAEP assessments.\9 (App. 
III describes the Governing Board's response to NAE's and our
recommendations in more detail.)


--------------------
\6 Educational Achievement Standards:  NAGB's Approach Yields
Misleading Interpretations (GAO/PEMD-93-12, June 23, 1993). 

\7 Lorrie Shepard, Robert Glaser, Robert Linn, and George Bohrnstedt,
Setting Performance Standards for Student Achievement, A Report of
the National Academy of Education Panel on the Evaluation of the NAEP
Trial State Assessment:  An Evaluation of the 1992 Achievement
Levels, National Academy of Education (Stanford, Calif.:  Stanford
University, 1993).  NAE is composed of 125 scholars and education
leaders who promote scholarly inquiry and discussion about the ends
and means of education in the United States and abroad.  NAE has been
in existence for about 33 years. 

\8 See Proceedings of the Joint Conference on Standard Setting for
Large-Scale Assessments of the National Assessment Governing Board
(NAGB) and the National Center for Education Statistics (NCES),
Volume II, and Executive Summary, Volume 1, prepared by Aspen Systems
under contract to the Governing Board and NCES (Washington, D.C.: 
Oct.  1995). 

\9 The Governing Board further clarified how this will be done: 
"[T]he contract and specifications approved by the Governing Board
require that the proposed tests be designed to show whether
individual students can meet NAEP standards for basic, proficient, or
advanced performance.  This will be done to link various forms of the
tests.  No new standard-setting is anticipated.  However, the Board
may make some modifications or amplifications in the written
descriptions of achievement levels to improve the clarity and
usefulness of the performance standard." See app.  IV. 


   DIFFERENCES BETWEEN NAEP AND
   PROPOSED TESTS
------------------------------------------------------------ Letter :3

The plan announced by President Clinton to develop voluntary national
tests for fourth grade reading and eighth grade mathematics would be
based on NAEP test frameworks.  This means, among other things, that
broad consensus would be obtained on the tests' subject matter and
approach.  The proposed tests would also differ from NAEP in several
key respects, however, including providing individual results (see
table 1 for a comparison of differences between NAEP and the proposed
national tests).  In addition, like NAEP, the proposed tests would
have voluntary participation.  The administration has announced that
it would administer the first tests in March 1999. 



                                Table 1
                
                  Key Differences Between NAEP and the
                  Voluntary National Testing Proposal

                                                Voluntary national
Characteristic          NAEP                    testing proposal\a
----------------------  ----------------------  ----------------------
What grades are tested  Primarily students in   Students in grades 4
                        grades 4, 8, and 12     and 8

What subject areas are  Selected areas          Reading (grade 4) and
tested                  determined by the       mathematics (grade 8)
                        Governing Board

Which students are      Random sample of        All students in
tested                  students                participating
                                                communities; but
                                                participation of
                                                individual students by
                                                law is voluntary

Extent of testing       Each student is tested  Each student is tested
                        on only some test       on all items, with up
                        items, with 45 to 60    to 90 minutes of
                        minutes of testing per  testing per student
                        student

Nature of               Voluntary on the part   Voluntary on the part
participation           of every student,       of every student,
                        school, district, and   school, district, and
                        state                   state

How results are         Individual scores not   Individual scores
reported                available; results are  available and results
                        aggregated into a       can be aggregated for
                        state or national       schools, districts,
                        score                   and states

Availability of test    Some test items         Test released to the
                        released to the         public (as a result, a
                        public, but most are    new test must be used
                        not (as a result,       for each test cycle)
                        parts of the test are
                        used in subsequent
                        years)
----------------------------------------------------------------------
\a In general, these elements were initially proposed by the
Department of Education but have not all been completely approved by
the Governing Board since it acquired the test development contract,
a Board official said.  Although the Board has approved the test
grades and subject matter and the extent of testing, it has made no
decision about the type of communities participating in the test, the
extent to which tests will be released to the public, or the level to
which test scores will be aggregated.  Such decisions are pending the
Board's review and any further congressional action, according to the
Board official. 


      CONCERNS ABOUT PROPOSED
      TESTS
---------------------------------------------------------- Letter :3.1

The voluntary national test proposal has raised many concerns.  Among
the concerns are that the tests (1) were not needed given the number
of tests already being administered, (2) were being developed too
quickly without allowing for enough congressional debate and
consideration, (3) might result in inappropriate federal influence on
state and local curricula, (4) would discriminate against students
with disabilities and limited-
English proficiency, and (5) might be used inappropriately to base
federal aid received by a state on testing performance. 

Partially in response to these concerns, the Congress took several
steps to change the testing proposal and the timetable.  It passed
the Department of Education's Appropriations Act for fiscal year
1998, which transferred authority for the contract to develop the
national tests from the Education Department to the Governing Board
(P.L.  105-78, sec.  307 (a)).  The Department had already awarded
the test development contract to the American Institutes for Research
(AIR) in August 1997.\10 The Appropriations Act required the
Governing Board to review this contract and change it or negotiate a
new one as necessary.  The act also prohibits the Department from
spending fiscal year 1998 funds to field test, pilot test,
administer, distribute, or implement any national tests (sec.  305
(a)).  The act does not specifically prohibit using fiscal year 1998
funds, however, to develop test specifications or write test items. 
The act does prohibit the federal government from requiring states or
districts to participate in any aspect of the test; it also prohibits
the federal government from requiring any student to take the test
(sec.  310 (a)).  Finally, the act mandates NAS to conduct some
studies on the comparability of NAEP assessments with state tests and
commercial tests (sec.  306 (a)). 


--------------------
\10 AIR is a 50-year-old not-for-profit organization that specializes
in conducting educational research, including large-scale
assessments, policy analysis, and program evaluation. 


   GOVERNING BOARD'S RELATIONSHIP
   WITH THE DEPARTMENT DEPENDS ON
   WHETHER THE FOCUS IS ON NAEP OR
   VOLUNTARY NATIONAL TESTS
------------------------------------------------------------ Letter :4

The relationship between the Governing Board and the Department
depends on whether the focus is on NAEP or the proposed voluntary
national tests.  To clarify this relationship regarding NAEP matters,
the Department and the Governing Board signed a 1992 Memorandum of
Understanding that while providing for collaboration, seeks to ensure
the Governing Board's independence from the Department for its NAEP
functions, powers, and duties.  The memorandum, for example, calls
for the Secretary to delegate to the Governing Board the authorities
to hire personnel and enter into contracts.  Funds for the Governing
Board's operating budget and NAEP-related contracts come from the
Department's Office of Educational Research and Improvement's (OERI)
appropriations for NAEP tests.  The Governing Board helps in
developing the Department's budget request for these funds. 

Passage of the Department's Appropriations Act in November 1997
changed the relationship between the Governing Board and the
Department by transferring exclusive authority for the voluntary
national test development contract from the Department to the
Governing Board.  Since that time, both Governing Board and
Department officials have indicated that congruent with the act, the
Department has had almost no involvement with the Governing Board's
handling of this contract.  Thus, unlike the two bodies' typical
collaboration on NAEP matters, almost no collaboration takes place on
voluntary national test matters, officials report.  The Department
funds the Governing Board's voluntary national test development
contract costs with moneys drawn from another OERI program, the Fund
for the Improvement of Education (FIE).  The Governing Board has not
helped in developing the Department's budget request for FIE funds. 


      MEMORANDUM DETERMINES
      RELATIONSHIP REGARDING NAEP
      MATTERS
---------------------------------------------------------- Letter :4.1

The 1992 Memorandum of Understanding is based on the 1988 legislation
authorizing NAEP.\11 The memorandum calls for the Department to
delegate to the Governing Board the authority to, among other
activities, appoint excepted service personnel; obligate funds;
obtain the services of experts and consultants; and make, enter into,
and perform contracts.  The memorandum also calls for the Department
to provide funding and administrative support services to the
Governing Board. 


--------------------
\11 A copy of the Memorandum of Understanding appears as app.  II. 


         PERSONNEL MATTERS
-------------------------------------------------------- Letter :4.1.1

In addition to its 25 members, the Governing Board has a staff of 10: 
6 professional staff with technical expertise, assisted by 2 support
and 2 clerical staff.  Recently, the Governing Board hired three
part-time, temporary consultants to help with its new
responsibilities for the voluntary national tests, a Governing Board
official reported. 

In accordance with its authority, the Governing Board makes all staff
hiring decisions, according to Board officials.  In addition, Board
officials said they determine positions' specifications,
classifications, and salaries.  For administrative purposes, however,
Governing Board staff are considered Department of Education
employees, and, in some cases, the Governing Board uses the
Department's administrative services to advertise and conduct initial
screenings for these staff positions, according to Board officials. 
Both Governing Board and Department officials told us that no
Department employee or appointee has ever been loaned or detailed to
work for the Governing Board.  Department staff are available,
however, when the Governing Board requests assistance. 

Regarding the Board, the Secretary of Education continues--although
it is not required--to select the Board members from a list of
candidates nominated by the Board, rather than select members without
the Board's help.  The original 1988 NAEP legislation required the
Secretary to choose from the Governing Board's nominations; the 1994
NAEP legislation removed this requirement, although the Secretary
opts to use the Board's nominations. 


         CONTRACTING AND RELATED
         MATTERS
-------------------------------------------------------- Letter :4.1.2

The Governing Board has full authority to hire contractors for
NAEP-related work.  The Department does not approve these contracts
in any way, according to Governing Board and Department officials. 
The Governing Board has three types of contracts for NAEP work--two
are for technical contracts that develop test frameworks or set
performance standards, and the third is for operational contracts
that arrange the logistics of meetings.  In developing these
contracts, the Governing Board involves some Department personnel,
but these staff have no approval authority for the contracts,
Governing Board officials said.  For example, the Board has asked
NCES to participate with outside experts in a peer review of the
technical contracts. 

Coordination between the Board and the Department is important in
developing NAEP contracts, according to officials of both bodies. 
For example, to ensure appropriate time lines for the Governing Board
contractor to develop achievement levels for the civics test
administered in 1998, the Board had to determine when the
Department's contract work for developing the civics test items would
be completed.  In addition, department officials said that in the
past, the Department has provided the Board with training and
assistance on using computer software related to contract
documentation.  The Governing Board has not requested such
assistance, however, in the past 2 years. 


         COLLABORATION ON NAEP
         MATTERS
-------------------------------------------------------- Letter :4.1.3

The Memorandum of Understanding has provisions for collaboration on
NAEP matters between the Governing Board and the Department. 
Officials of both bodies said that collaboration on NAEP matters was
important and perhaps essential for the Department to implement NAEP
according to the Board's policy and procedures.  This collaboration
does not affect the Board's independence from the Department,
according to Board officials.  The Governing Board and Department
officials described the following examples of their collaboration on
NAEP matters: 

  -- By participating as a nonvoting Board member, the Department's
     ex officio member helps to ensure that the Department
     understands the Governing Board's policy guidance on NAEP
     frameworks and achievement levels.  Similarly, a Governing Board
     staff member attends NCES meetings on test development to convey
     the Board's policies. 

  -- Regarding formulating NAEP policy for test frameworks or
     achievement levels, the Governing Board seeks technical advice
     from a broad range of experts, some of whom include experts with
     NCES and its contractors. 

  -- The memorandum calls for the Department and the Board to seek
     mutually satisfactory resolutions when NCES disagrees with the
     Governing Board's policy guidance.  In a few cases, the
     Secretary of Education has been the final arbiter of conflicts,
     according to Governing Board and Department officials, according
     to Department officials. 


         FUNDING FOR OPERATING
         BUDGET AND NAEP CONTRACTS
-------------------------------------------------------- Letter :4.1.4

The funding for the Governing Board's operating budget and
NAEP-related contracts comes from OERI's appropriations for NAEP
assessments.  From these appropriations, the Governing Board was
allocated about $2.9 million and $3.5 million in fiscal years 1997
and 1998, respectively.  The Board helped develop the Department's
budget request for the Board's part of NAEP's appropriations. 

Using Department guidance, the Governing Board develops its annual
budget request on the basis of the estimated cost of its staff
salaries, Board member expenses, and NAEP contracts and submits it to
the Department, Governing Board officials said.  Department budget
officials said this guidance consists of the typical instructions
required by OMB or congressional directives.  The Department reviews
the Governing Board request, makes adjustments, and includes the
request in its total agency request to OMB.  The amount of the
Department's request may differ from the amount the Governing Board
submitted, and the Board is generally not involved in the
Department's negotiations with OMB during OMB's review.  For example,
the Governing Board submitted a request of $4.8 million for fiscal
year 1999, and the Department reduced this request to $3.4 million,
according to officials of both bodies.  After reevaluating the
Governing Board's funding needs, however, the Department later
petitioned OMB to increase the Board's budget to $4 million, which
OMB did. 


      EXCEPT FOR FUNDING, THE
      DEPARTMENT HAS HAD LITTLE
      INVOLVEMENT IN THE GOVERNING
      BOARD'S VOLUNTARY NATIONAL
      TEST WORK
---------------------------------------------------------- Letter :4.2

Governing Board and Department officials view their relationship
regarding the voluntary national test work differently from their
collaboration on NAEP, reflecting the Appropriations Act provisions. 
After the Board received exclusive authority for the test development
contract, officials of both bodies said they discussed the mechanics
of smoothing the transfer of authority, the legal availability of
funds to pay contractors for pilot and field tests, and a legal
matter involving students with disabilities and limited-English
proficiency.  Other than these instances, officials have not
collaborated on voluntary national test work, they said. 


         DEPARTMENT NOT INVOLVED
         IN CONTRACT REVIEW
-------------------------------------------------------- Letter :4.2.1

Upon assuming exclusive authority for the voluntary national test
development contract, the Governing Board was required to review the
contract the Department had already awarded and modify it or
negotiate a new contract as necessary.  To review the contract, the
Governing Board formed a special committee of its Board members that
did not include any Department officials.  On the basis of the
special committee's recommendations, the Governing Board approved
some major changes to the contract, including the timing of pilot
testing, field testing, and test administration.  For example, under
the Department's contract, the first test was to be developed in time
to be administered in March 1999; under the Governing Board's
contract, the test is to be developed in time to be administered in
March 2001.  In addition, the Board's contract changed responsibility
for contract development policy decisions from the Department to the
Governing Board, in keeping with the Appropriations Act. 

The act also directed the Governing Board to determine, among other
things, whether the test development process and test items consider
the needs of disadvantaged, limited-English proficient, and disabled
students.  Because the Board has no legal counsel of its own, the
Board sought advice from the Department and other sources about the
access of such students to voluntary national tests, according to a
Board official. 


         FUNDING FOR THE GOVERNING
         BOARD'S VOLUNTARY
         NATIONAL TEST DEVELOPMENT
         CONTRACT
-------------------------------------------------------- Letter :4.2.2

The Department funds the Governing Board's voluntary national test
development contract costs with moneys from FIE, a discretionary
account that gives the Department the authority, under 20 U.S.C. 
8001(a), to support nationally significant programs and projects to
improve education.  As of April 1998, the Department has paid the
contractor a total of about $1.7 million, all with fiscal year 1997
FIE funds.  The Department has allotted about a third of these
payments ($540,000) to the Governing Board for payment to the
contractor.  The Governing Board has not been involved in developing
the Department's budget request for FIE funds, Department and
Governing Board officials said. 


   FISCAL YEARS 1997 AND 1998
   CONTRACT AWARDS AND
   EXPENDITURES AND FUTURE COST
   ESTIMATES
------------------------------------------------------------ Letter :5

Except for the test development contract specifically transferred to
the Governing Board, the Department still has responsibility for
other contracts related to the proposed voluntary national tests. 
These other contracts cover such activities as developing test
specifications based on NAEP frameworks,\12

comparing NAEP with other state and commercial tests, and evaluating
the voluntary national test items.  The Appropriations Act requires
these last two activities.\13 The Department awarded voluntary
national testing contracts totaling about $17 million for fiscal
years 1997 and 1998.  Most of this amount obligated the test
development contract that was subsequently transferred to the
Governing Board.  Contract awards represent obligated funds; payments
of such funds to contractors are termed "outlays," a Department
budget official said.  Outlays and other expenditures for voluntary
national tests in fiscal years 1997 and 1998 totaled almost $3
million as of April 1998.  (See table 1 for a summary of contract
awards and expenditures.) Under the Governing Board's revised
contract terms, we estimated the cost for developing one complete set
of voluntary national tests is $15 million, which the federal
government is expected to pay.  The Department determined that costs
to implement the fourth grade reading and the eighth grade
mathematics testing program in a year could total up to $96 million,
depending on the number of students tested.  Who would pay these
costs has not been determined. 



                          Table 2
          
          Education Department Contract Awards and
            Expenditures for Voluntary National
             Tests, Fiscal Years 1997 and 1998

                              Contract
Purpose and     Contract    outlay and  Contract
type of            award         other  or or
expenditure\   amount or  expenditures  recipien  Funding
a              expense\b            \c  t         source
------------  ----------  ------------  --------  --------
Fiscal year 1997
----------------------------------------------------------
Task order      $960,473      $882,655  MPR       FIE
contract for                            Associat
test                                    es
specificatio
ns
(awarded 5/
1/97)

Contract for  13,035,848     1,718,728  AIR       FIE
test
development
(awarded 8/
18/97)

Contract for   1,238,800       260,988  NAS       FIE
test
evaluation
(awarded 9/
30/97)

Task order        26,439        17,858  Professi  FIE
contract for                            onal and
meetings                                Scientif
with experts                            ic
and                                     Associat
transcripts                             es
(awarded 2/
14/97)

Expenses          28,720        28,720  Loews     OERI
related to                              L'Enfant  salaries
August 1997                             Plaza     and
conference                              Hotel,    expenses
with state                              American  funds
education                               Express
officials

Fiscal year   15,290,280     2,908,949  Not       Not
1997 totals                             applicab  applicab
                                        le        le


Fiscal year 1998
----------------------------------------------------------
Contract for   1,800,000        80,270  NAS       FIE
test
equivalency
scale,
forms, and
use studies
(awarded 2/
3/98)
----------------------------------------------------------
\a Information in this table is based on contract records and
expenditure summaries provided by the Department of Education and the
National Assessment Governing Board. 

\b With the exception of the test development contract, these
contracts and expenses are under the jurisdiction of the Department. 
The test development contract, originally signed by the Department in
Aug.  1997, was transferred to the Governing Board in Nov.  1997. 

\c These outlays were made as of Apr.  1998.  The only outlays made
by the Governing Board account for about $540,000 of the $1.7 million
outlaid for the test development contract.  The Department made all
other outlays. 


--------------------
\12 The Department's test specifications contract was completed
before Nov.  1997, when the Board received authority for the test
development contract. 

\13 The Department signed the test evaluation contract before the
passage of the Appropriations Act; however, Department officials said
that the contract fulfills the act's mandate (sec.  308) for NAS to
evaluate the test items developed for the voluntary national tests. 


      CONTRACT AWARDS IN FISCAL
      YEARS 1997 AND 1998
---------------------------------------------------------- Letter :5.1

In fiscal year 1997, the Department awarded contracts worth about $15
million for voluntary national test activities.  Preparing for these
contracts, which began just before the President's announcement of
the voluntary national test initiative, represents the Department's
first spending for the initiative, Department officials said.\14 The
Department based its authority for funding the activities of the
national testing program on the general authority under FIE.\15

The total $15 million awarded to the various contracts came from FIE
funds, which are administered by OERI.\16

Most ($13 million) of the fiscal year 1997 award amount was for the
test development contract awarded to AIR.  As of April 1998, only
about $1.7 million had been spent on this contract, however, because
of a partial stop-work order first initiated by the Department and
later extended by the Governing Board.\17

The Department has carried forward the obligated balance, about $11.3
million, of the fiscal year 1997 FIE funds to pay the contractor's
costs, now that the stop-work order has been lifted and the
contractor has resumed working, a Department budget officer reported. 

Of the three other fiscal year 1997 contracts, whose award amounts
totaled about $2 million, the contractors have received only about
$1.2 million.  In addition, the Department used $28,720 of OERI
salary and expense fund moneys to pay for invitational travel,
meeting space, use of video and audio equipment, and a working lunch
for nonfederal participants during the Secretary's conference on the
voluntary national tests with state education officials. 

For fiscal year 1998, the Department had planned to make another $15
million in FIE funds available for voluntary national test
activities.  As mandated by the Department's 1998 Appropriations Act,
the Department awarded $1.8 million of these funds to NAS to conduct
studies on test equivalency scales and other issues.  For this
contract, NAS will study (1) the feasibility of developing
equivalency scales for comparing test scores on different commercial
tests and state tests with each other and with NAEP scores and (2)
the steps needed to ensure that test results are not used in a
discriminatory way for student promotion, tracking, or graduation and
that test forms will accurately gauge student achievement in reading
and mathematics.  As of April 1998, the Department had spent about
$80,270 on this NAS contract. 

None of the remaining fiscal year 1998 FIE funds available for
voluntary national test activities has been obligated, a Department
budget officer reported.  As identified in its December 1997 spending
plan, the Department had planned to use $13 million from the fiscal
year 1998 FIE funds for the second year of the test development
contract.\18 Contract delays, a congressional prohibition on using
fiscal year 1998 funds for some test development activities, and
contract modifications, however, will probably reduce the amount of
fiscal year 1998 FIE funds needed for the second year of the
contract, a Department budget officer said.  The Department is
waiting for the Governing Board to communicate the amount needed for
the contract's second year and will allot the amount requested, the
budget officer explained.  The budget officer also said that no other
FIE or Department funds appropriated in fiscal year 1998 are being
spent on development activities for the proposed tests. 

In addition to those expenditures for which a cost figure was
available, the Department spent an undetermined amount when
distributing information about the national tests in fiscal years
1997 and 1998.  The Department has used various means to inform the
public about the tests, including web sites, press releases, articles
in trade magazines, Department newsletters, and presentations. 
Department officials said these activities are not directly related
to developing the test but were part of the Department's ongoing
function to educate the public about Department initiatives. 
Department officials had no estimate of the costs of these
activities, they said, because the costs were relatively small and an
integral part of the larger costs of salaries, supplies, and
services. 


--------------------
\14 Because the voluntary national tests use NAEP as a foundation,
funds spent on NAEP (for example, about $33 million appropriated in
fiscal year 1997 and about $35 million in fiscal year 1998) relate
somewhat to voluntary national tests, although they are technically
not part of the development costs. 

\15 Under this authority, the Department is further authorized to (1)
support the development of model strategies for assessing student
learning, (2) develop and evaluate strategies for integrating
instruction and assessment so that teachers and administrators can
focus on what students should know and be able to do at particular
grade levels, (3) support nationally significant programs and
projects to improve the quality of education, and (4) support
activities to raise standards and expectations for all students'
academic achievement. 

\16 FIE's budget request for fiscal year 1997 was $40 million; the
overall OERI appropriation, including FIE, was $293.1 million. 

\17 The Department issued a partial stop-work order in Sept.  1997 to
save the government from spending funds for work that may have to be
redone if the Congress transferred the contract to the Governing
Board.  The Governing Board extended the stop-work order until Feb. 
1998 to allow it time to review the contract. 

\18 The Department's appropriations for fiscal year 1998 specifically
excluded the use of fiscal year 1998 funds for some but not all of
the activities covered under the 1997 test development contract.  For
example, the contractor could use fiscal year 1998 funds to write
test items, but it could not conduct field or pilot testing.  In any
event, no fiscal year 1998 funds have been spent on the contract for
developing voluntary national tests, a Department budget official
reported. 


      TEST COST ESTIMATES FOR
      FUTURE YEARS
---------------------------------------------------------- Letter :5.2

The Department originally estimated that the total cost for the
original 5-year contract would be about $64 million.  During the
5-year term of the contract, the contractor would have developed four
complete test sets and part of a fifth.\19 On the basis of these
figures, the average cost of developing one complete test set is
about $14 million.\20 The Governing Board's revised test development
contract also covers a 5-year period.  To ensure that pilot testing
and then field testing take place in the same month of subsequent
school years, however, the Governing Board increased the time for
developing one complete test set from about 15 months to about 34
months.\21 During the 5-year period, the contractor would be required
to develop three test sets for a total award amount of about $46
million.  As a result, the estimated average cost of developing a
complete set of tests under the revised contract is about $15
million, which the federal government is expected to pay. 

To help estimate the implementation costs of administering the test
to students, a Department official asked some states what they spent
administering state or commercial assessments.  Per pupil costs
ranged from $10 to $12.  Because the methods used to administer and
score voluntary national tests may differ from the various state
methods, however, these figures may not be appropriate for comparison
purposes, according to Department officials.  Nonetheless, assuming
it cost $12 to test each of the estimated 8 million students eligible
to take the test, the Department estimated that implementing the
fourth grade reading and eighth grade mathematics testing program
would cost up to $96 million a year.\22 The administration had
originally proposed that the federal government would reimburse
states and localities for costs in the first year of full testing. 
The Governing Board now has to address this issue, however,
Department officials said.  Board officials believe the Board is
legally responsible only for developing the assessment--not
implementing it.  Who would pay for implementing these tests is
pending the Congress' decision, according to the Board. 


--------------------
\19 In the 5-year period covered by the original contract, the
contractor would have been required to complete all activities needed
to develop and finalize four sets of voluntary national tests for
mathematics and reading, one set for each year the test was to be
administered in years 1999 to 2002.  Each set of tests required about
15 months to complete and included writing new test items along with
the pilot and field testing of these items.  To meet the planned
schedule for administering the tests, the contractor would have to
develop some test sets concurrently. 

\20 A test set includes several different but equivalent tests for
each subject.  The set includes, for example, a copy used for the
actual test, a backup copy in case security has been breached, a copy
reserved for research purposes, and a sample copy released to the
public. 

\21 The pilot and field testing of students is scheduled to take
place a year apart in the same month of the school year to ensure
that tested students have about the same amount of prior knowledge. 
The contractor is expected to use the results of pilot testing to
modify the test items as needed before conducting field testing. 

\22 This estimate is based on the assumptions that (1) all fourth and
eighth graders in public and private schools will take the voluntary
national tests in 1999 and (2) such students will number 4 million in
each of these grades in 1999.  In 1995, the most recent year for
public school enrollment figures, about 3.4 million students were
enrolled in each of grades 4 and 8. 


   CONTRACTOR SELECTION
------------------------------------------------------------ Letter :6

The contractor selection process for the 3 fiscal year 1997 contracts
directly related to the national tests involved developing test
specifications,\23 developing the test, and evaluating the voluntary
test initiative.\24 The Department used a competitive procurement
process for all three contracts but was not able to obtain more than
one proposal for each.  The Department mainly used its own staff to
evaluate the proposals' technical and cost aspects.  For the test
development contract, the Department also formed a technical review
panel of federal and nonfederal experts. 

In addition, none of the individuals selected as advisers for
developing test specifications, including members of the National
Test Panel and the three advisory committees,\25 had produced
textbooks or materials used in primary and secondary school
classrooms, according to our review. 


--------------------
\23 The Governing Board later replaced the test specifications
developed for the voluntary national tests for this contract with a
set of specifications that are closely based on the assessment
framework the Board previously adopted.  (See app.  IV.)

\24 We did not review the selection process for the fiscal year 1998
contract because the Congress had specifically mandated the awardee,
and we did not review the task order contract that established
administrative arrangements for meetings with experts because it was
small and not directly related to the test. 

\25 The Reading Committee, composed of teachers, scholars,
teacher-educators, and reading specialists, developed a set of
recommended item and test specifications for fourth grade reading. 
The Mathematics Committee, composed of mathematics teachers,
supervisors, and educators and university mathematicians, developed
recommended item and test specifications for eighth grade
mathematics.  The Technical Advisory group, composed of measurement
and assessment experts, assisted the other panels to ensure that the
item and test specifications would result in fair, valid, and
reliable measures. 


      TEST SPECIFICATIONS CONTRACT
---------------------------------------------------------- Letter :6.1

In 1996, the Department established two competitive task order
contracts with two firms, AIR and MPR Associates.  Under the terms of
these contracts, each firm agreed to compete for task orders that
would provide technical and analytical support to OERI on an ad hoc
basis.  In 1997, the Department determined that the voluntary
national test specifications work could be done as a competitive task
order under the competitive task order contracts with AIR or MPR
Associates.  The Department had determined, however, that firms could
not compete for both the test specifications task order and the test
development contract because competing for the test specifications
task order would reveal information that would give a firm an unfair
advantage in competing for the test development contract--in
violation of the Federal Acquisition Regulation (FAR).\26 Because AIR
intended to compete for the test development contract, it did not
compete for the test specifications task order. 

The Department, therefore, entered into sole-source negotiations with
MPR as the only firm available to perform the test specifications
task order under the competitive task order contract, said Department
officials.  The Department awarded MPR the task order under MPR's
existing task order contract in May 1997. 

To help reach a national consensus on the appropriateness of the
specifications, the contract called for involving a wide range of
experts.  The contractor formed a National Test Panel to make the
final recommended item and test specifications.  This panel included
individuals from key constituencies and policy-making groups for
elementary and secondary education at the state and local levels.  To
facilitate the panel's work, the contractor also established three
committees. 

According to our review of databases for documents written by any of
the 56 panel and committee members, some members had written
technical papers, analyses of education issues, or resource books for
teachers.\27 No members, however, had written textbooks or
educational materials for elementary and secondary classroom
instruction in grades kindergarten through 12. 


--------------------
\26 The FAR requires contracting officers to avoid, neutralize, or
mitigate conflicts of interest before awarding a contract.  The
Department determined that an organizational conflict of interest
would result if it permitted the same firm to compete for both the
test specifications and test development contracts (FAR 9.504(2),
9.501). 

\27 The four databases we searched were the Educational Resources
Information Center of the U.S.  Department of Education; the Library
of Congress catalog; Books in Print, a comprehensive source of books
published in the United States, including bibliographic records from
over 44,000 publishers; and the IAC Business A.R.T.S., a scholarly
business database designed to help on-line searchers identify current
research, analysis, trends, and expert opinion in a variety of
disciplines. 


      TEST DEVELOPMENT CONTRACT
---------------------------------------------------------- Letter :6.2

The Department contracted the development of the voluntary national
tests by competitive negotiated acquisitions, according to Department
officials and a review of contract records.  Department officials
said, however, that the contracting process was more open and public
than usual.  For example, the Department held a series of public
meetings for market research purposes, to inform the public, and to
gather advice for the project.  The Department also publicized a
draft statement of work and the request for proposal (RFP) through
both the Commerce Business Daily and the Department's web site in
spring 1997.  The Department expected the increased publicity to
produce a number of proposals, Department officials said. 

Although about 90 offerors requested copies of the RFP and an
undetermined number of individuals downloaded it from the web site,
only one firm, AIR, submitted a proposal by the June 1997 closing
date.  AIR's proposal included seven subcontractors, who had
extensive collective experience in developing, field testing, and
publishing standardized tests.\28
The Department formed a technical review panel to determine if AIR's
proposal met the RFP's criteria and recommended whether it was
acceptable to perform the RFP's tasks.  Panel members included seven
Department staff representing a range of expertise in assessment,
legal issues, reading, mathematics, and other areas and two outside
experts, one representing a district-level perspective in math
assessments and the other experienced in state-level assessments.  A
Department individual who had been involved in the test initiative
served as the panel's technical adviser.  The panel determined that
the proposal could be made acceptable through negotiation.  The
Department's contract officer reviewed AIR's and the subcontractors'
cost information.  Before negotiating with AIR, the contract officer
requested advice from Department staff who knew about the testing
initiative's activities that occurred in public hearings and
high-level policy meetings.  Accordingly, the Department's Director
of the Voluntary National Test Initiative and an OERI budget officer
were briefed on maintaining procurement integrity and made
procurement officials.  These two individuals helped formulate the
Department's negotiation strategy.  After the Department negotiated
with AIR to clarify technical, business, and cost issues, AIR
provided its best and final offer, which the Department accepted in
August 1997. 

When the Governing Board received exclusive authority for the
contract, the Board had 90 days to review the contract and accept,
modify, or terminate it.  The Governing Board ultimately decided to
modify the contract and submitted a revised statement of work to AIR. 
The Governing Board and AIR signed a revised contract in February
1998. 


--------------------
\28 The subcontractors were Riverside Publishing; Harcourt Brace
Educational Measurement; Westat, Inc.; California Test Bureau;
National Computer Systems, Inc.; Educational Testing Service; and
Council for Basic Education. 


      TEST EVALUATION CONTRACT
---------------------------------------------------------- Letter :6.3

Although OERI originally requested that the test evaluation contract
be awarded on a sole-source basis to NAS, the Department's contract
officer recommended a search for other qualified sources. 
Accordingly, the Department issued a sources-sought announcement for
this work in the Commerce Business Daily in July 1997.  No firms
responded to the announcement, however.  As a result, the Department
began sole-source procurement with NAS.  After the Department's
contracting personnel conducted technical, business, and cost reviews
of NAS' proposal, the Department eventually awarded this contract to
NAS for about $1.2 million in September 1997. 


   SEVERAL EXPLANATIONS OFFERED
   FOR DIFFERENCES BETWEEN NAEP
   AND STATE TEST RESULTS
------------------------------------------------------------ Letter :7

Some controversy has developed over NAEP's framework for voluntary
national testing because of the differences in results between NAEP
and some state-level proficiency tests.  Many states conduct their
own tests to measure student proficiency using state-developed tests
or those from a private testing service adapted for states' use.  In
his September 1997 testimony, the Secretary of Education presented
data from a study that compared test results in 11 states (see fig. 
3).\29 In most cases, the percentage of students meeting the NAEP
proficiency standard is much lower than the percentage meeting the
proficiency standard for the state examination.  The study's author
suggested that one important reason for this difference is because
most state proficiency standards appear to be set at a lower level
than NAEP's.  Our discussions with several testing experts indicated
that different proficiency standards could be one explanation, but
other explanations are also possible. 

   Figure 1:  Smaller Percentage
   of Students Meeting the
   Proficiency Standard on NAEP
   Fourth Grade Reading Test
   Compared With Most State Tests,
   1994

   (See figure in printed
   edition.)

Note:  State results for New Hampshire, Wisconsin, Maryland, Georgia,
Delaware, South Carolina, and Louisiana are based on reading tests of
third rather than fourth graders. 

Source:  This figure is based on data from "Setting Education
Standards High Enough," by Mark D.  Musick, President, Southern
Regional Education Board, July 1996.  We confirmed that NAEP's scores
reported in the figure were the same as those reported by NCES for
the 1994 NAEP reading assessment. 


--------------------
\29 Mark D.  Musick conducted this study in his capacity as president
of the Southern Regional Education Board.  Mr.  Musick is also a
Governing Board member. 


      DEFINING PROFICIENCY VARIES
---------------------------------------------------------- Letter :7.1

In his testimony in September 1997, the Secretary of Education
highlighted the differences in proficiency scores between NAEP and
state tests used by the 11 states involved in the study.  The study's
author suggested that the differences may be partly attributed to
NAEP's assessments having a higher proficiency standard.  That is, a
student might have the same correct response rate on two comparable
tests, but the expectations of what amounts to a proficient score
would be higher for one test than for the other. 

Experts we spoke with agree that differences in tests' proficiency
definitions can produce differences in the number of students judged
to be proficient.  For example, one test expert described to us how
two states using about the same test had different expectations of
the amount and type of content the student had to master.  One state
defined proficiency as correctly answering 66 percent of the items in
each major content area of the test, while the other state required
an overall correct response rate of 66 percent.  Under the latter
approach, a student might score below 66 percent in some content
areas but still get a proficient score, while a student scoring the
same in the former state would not pass the test. 

Furthermore, experts explained that even if NAEP and states had
common definitions of proficiency, different approaches for setting
the proficiency threshold score at or above which a student is scored
proficient could lead to different scores being designated as
proficient.  For example, one approach, called "item-centered,"
establishes a proficiency score, called a "cut" score, for each
performance level based on an evaluation of the test items.  NAEP
uses an item-centered approach to determine its cut score.  Another
approach establishes cut scores on the basis of actual test
performances by students judged by other means to be at a certain
competency level. 


      TEST INSTRUMENTS VARY
---------------------------------------------------------- Letter :7.2

Another possible reason for the difference in test results, experts
we spoke with said, may be the actual tests.  NAEP's assessments may
differ from an individual state's test by the amount and type of
content covered, the format (such as the extent to which a test uses
multiple-choice or open-ended questions), and the type and extent of
cognitive skills tested (such as the extent to which the student is
asked to simply recall facts or apply logic in solving problems). 

A test's relationship to a particular curriculum being taught could
also affect scores, according to experts.  One expert suggested that
compared with NAEP's assessment, some state tests may more closely
relate to what was actually being taught.  In contrast, NAEP
assessments relate less to state school curricula, this expert said. 


      STUDENT PROFILES VARY
---------------------------------------------------------- Letter :7.3

Although not generally considered an important reason for explaining
the differences in NAEP and state test results, differences in
students taking the test and when they take it may affect the
results, according to experts.  For example, experts we spoke with
cited students who take the NAEP tests include those with
disabilities and those with limited-English proficiency; state tests
may not include such students.  In addition, a student taking a test
early in the school year may have less knowledge than one taking it
later in the year. 


   AGENCY COMMENTS
------------------------------------------------------------ Letter :8

We provided both the Governing Board and the Department of Education
the opportunity to comment on a draft of this report.  The Governing
Board said that the report was fair, balanced, and accurate.  It also
suggested several specific technical changes that we incorporated as
appropriate.  The full text of its comments appears in appendix IV. 
The Department of Education did not directly comment on the report
but said that the Governing Board has made good progress and should
continue to move forward in its efforts to develop national tests. 
The full text of the Department's comments appears in appendix V. 


---------------------------------------------------------- Letter :8.1

We are sending copies of this report to appropriate congressional
committees and members of the Congress, the Secretary of Education,
the National Assessment Governing Board, and other interested
parties.  Please call me at (202) 512-7014 or Eleanor L.  Johnson,
Assistant Director, at (202) 512-7209 if you or your staff have any
questions. 

Cornelia M.  Blanchette
Associate Director, Education and
 Employment Issues


SPECIFIC STUDY QUESTIONS
=========================================================== Appendix I

Our study addressed three main issues:  (1) the relationship between
the Governing Board and the Department of Education, (2) costs of
developing the national voluntary tests and procedures for hiring
contractors, and (3) possible explanations for the differences in
scores on state achievement tests and National Assessment of
Educational Progress (NAEP) tests.  Following is a list of specific
study questions grouped by main issue. 


   RELATIONSHIP BETWEEN THE
   GOVERNING BOARD AND THE
   DEPARTMENT OF EDUCATION
--------------------------------------------------------- Appendix I:1

1.  What is the relationship between the Department of Education and
the Governing Board?  Specifically,

1.1.  For each of the past 3 years, have any Department of Education
employees or appointees been loaned, detailed, or otherwise made
available to perform services for the Governing Board?  If so, please
list their names, titles, ranks, and months of service to the
Governing Board. 

1.2.  What, if any, role does the Department of Education play in
procurement for the Governing Board and NAEP? 

1.3.  Is the Department of Education involved in reviewing or
approving contracts or any other activities or policies of the
Governing Board?  If yes, explain the specific way in which the
Department is involved. 

1.4.  Is the Department of Education involved in the hiring of staff
of the Governing Board and, if so, in what ways? 

1.5.  What role and influence, if any, does the Department of
Education play or exercise in the distribution of annual
appropriations to the Governing Board (for the activities of the
Board and NAEP)?  What role and influence, if any, does the
Department of Education play or exercise in the establishment of the
budget of the Board?  Please provide any and all documentation of any
agreements or memoranda of understanding? 

2.  How has the Governing Board responded to GAO reports and National
Academy of Education analyses and recommendations? 


   COSTS OF DEVELOPING THE
   NATIONAL VOLUNTARY TESTS AND
   PROCEDURES FOR HIRING
   CONTRACTORS
--------------------------------------------------------- Appendix I:2

3.  How were contractors for national test development activities and
individuals providing advice on the process selected? 

4.  Is the Department of Education using any fiscal year 1998 funds
for activities related to developing the national tests? 

5.  How much money did the Department of Education spend on the
concept or specific development of the national tests in fiscal year
1997 and fiscal year 1998 (1) before the testing initiative was
announced and (2) after it was announced; and what are the details of
these expenditures, such as the sources of funds, dates, and purposes
of expenditures? 

6.  What is the Department of Education's estimate of the total cost
of developing the proposed costs and the annual implementation costs
that would be borne by (1) the federal government and (2) state and
local governments? 

7.  What textbooks or educational materials have members of the
national test panel, the technical advisory committee, or the math or
reading committee authored?  What, if any, federal funds were used to
develop or produce these materials? 


   POSSIBLE EXPLANATIONS AS TO WHY
   SCORES ON STATE ACHIEVEMENT
   TESTS DIFFERED FROM NAEP TEST
   RESULTS
--------------------------------------------------------- Appendix I:3

8.  What are the alternative explanations for differences between
student scores on state achievement tests and on the NAEP reading
tests exhibited by Secretary Riley during his September testimony? 




(See figure in printed edition.)Appendix II
MEMORANDUM OF UNDERSTANDING
BETWEEN THE DEPARTMENT OF
EDUCATION AND THE NATIONAL
ASSESSMENT GOVERNING BOARD
=========================================================== Appendix I



(See figure in printed edition.)



(See figure in printed edition.)


NATIONAL ASSESSMENT GOVERNING
BOARD'S RESPONSE TO OUTSIDE
RECOMMENDATIONS
========================================================= Appendix III

This appendix presents background information on the Governing Board
and standard setting as well as (1) an overview of our
recommendations and the Governing Board's responses to them, (2) an
overview of the National Academy of Education's (NAE) recommendations
and the Governing Board's response to them, and (3) some perspectives
on the Governing Board's approach to standard setting in view of
NAE's, our, and other experts' reviews. 


   BACKGROUND
------------------------------------------------------- Appendix III:1

The National Assessment of Educational Progress' (NAEP) 1988
authorization assigned the Governing Board to identify appropriate
achievement goals for each age and grade in each subject area tested
under the national assessment.  To carry out this responsibility, the
Governing Board set achievement levels for NAEP as performance
standards that define what students in grades 4, 8, and 12 should
know and be able to do in each subject area.  In pursuing this task,
the Governing Board identified scores (called cut points) on the NAEP
scale for basic, proficient, and advanced achievement levels for each
grade and used these levels in reporting most NAEP results in 1990,
1992, and 1994.  Using performance standards to report NAEP scores
(criterion-referenced) rather than just reporting the distribution of
test results (norm-referenced) was viewed by the Governing Board as a
way to provide more useful and meaningful information to policymakers
and the public. 

In developing achievement levels based on performance standards, the
Governing Board faced many policy, procedural, and technical issues. 
In addition, educational practice offered little guidance on the best
approach for setting standards for broad-based assessments such as
NAEP.  After consulting the National Center for Education Statistics
(NCES) and other testing experts, the Governing Board chose a
well-known standard-setting method, an item-centered approach, but
modified it in novel ways.\30

Given the national significance of NAEP tests, the Governing Board's
process for developing achievement levels was widely scrutinized and
evaluated.  NCES officials, research firms under contract either to
the Governing Board or to the Department of Education, and other
independent experts conducted reviews of the process.  Two such
reviews included our 1993 study focusing mainly on the Governing
Board's 1990 standard-setting process\31 and the NAE's 1993 study
mainly of the 1992 process.\32 NAE's study was done at NCES' request. 


--------------------
\30 In item-centered approaches, panelists judge how students who
have the capabilities needed to meet a given standard would perform
on each test item.  On the basis of the panelists' judgment, a cut
point is identified as the threshold score for meeting the standard. 

\31 Educational Achievement Standards:  NAGB's Approach Yields
Misleading Interpretations (GAO/PEMD-93-12, June 23, 1993). 

\32 Lorrie Shepard, Robert Glaser, Robert Linn, and George
Bohrnstedt, Setting Performance Standards for Student Achievement, A
Report of the National Academy of Education Panel on the Evaluation
of the NAEP Trial State Assessment:  An Evaluation of the 1992
Achievement Levels (Stanford, Calif.:  Stanford University, 1993). 


   OVERVIEW OF OUR RECOMMENDATIONS
   AND THE GOVERNING BOARD'S
   RESPONSES
------------------------------------------------------- Appendix III:2

In 1993, we reviewed the Governing Board's approach for establishing
the 1990 performance standards for NAEP's mathematics assessment.  We
analyzed the validity of issues raised by the Governing Board's
approach, examined alternative approaches to setting performance
standards, and reviewed the Governing Board's capacity to provide
sound guidance to the Department of Education on technical issues. 
The Governing Board's 1990 standard-setting approach was procedurally
flawed, resulting in interpretations of scores of which the validity
was doubtful, according to our review.  We recommended reopening the
question of setting and interpreting performance standards.  We also
recommended reviewing the structures and procedures governing the
assessment to ensure the technical soundness of the Governing Board's
policies as well as its responsiveness to constituent interests. 

We made a total of 10 recommendations to the Governing Board.  In
reviewing the status of these recommendations, we determined by
October 1995 that the Governing Board had implemented 5 of the 10
recommendations but had not fully responded to the remaining 5.  For
example, the Governing Board had reexamined its approach to standard
setting but chose to continue using the item-centered approach with
some modification.  The Governing Board also took some steps to
strengthen its capacity to give sound policy advice to the Department
by, for example, instituting a formal procedure to obtain NCES review
of its proposed policies.  The 10 recommendations and the Governing
Board's responses as determined by us are summarized in table III.1
at the end of this appendix. 


   OVERVIEW OF NAE'S
   RECOMMENDATIONS AND THE
   GOVERNING BOARD'S RESPONSES
------------------------------------------------------- Appendix III:3

On the basis of several studies it commissioned to review the Board's
process for setting performance standards, NAE found that the Board's
item-centered approach was fundamentally flawed for setting the 1992
performance standards and that the standards were set unreasonably
high.  NAE also concluded that the flawed performance standards would
not enhance the ability to interpret NAEP scores and could harm
NAEP's credibility.  NAE called for the Governing Board to stop using
an item-centered approach to setting performance standards and not
report the 1992 NAEP results using these standards.  NAE also
determined that the 1992 NAEP test items, particularly at the
advanced level, were not sufficiently congruent with emerging
national content standards.\33 Consequently, NAE reported that the
public may perceive that the expectations for developing students'
thinking and mastery of challenging subject matter were low.  Among
its recommendations, NAE urged the Governing Board to wait until the
availability of national content standards and then follow a more
coherent process for developing performance standards in conjunction
with content standards.  NAE also urged the Governing Board to
explore setting standards using several different approaches, none of
which were item centered. 

On the basis of its review of the NAE study and reviews by outside
experts, the Governing Board chose to continue using an item-centered
approach.  The Governing Board has researched and experimented,
however, with alternative approaches for setting the 1994 and 1998
standards, it has reported.  It has also reported that it has widened
its participant base in setting the performance standards to ensure a
more thorough national consensus in setting the standards. 


--------------------
\33 The movement to establish national content standards in the
various disciplines began, as NAE describes, with the 1989 Education
Summit when President Bush and the nation's governors identified six
broad education goals.  The creation of the goals led to initiatives
to establish national content standards, including the Congress'
creation of the National Council on Education Standards and Testing
(NCEST).  Several groups, such as the National Academy of
Sciences/National Research Council, National Council of Teachers of
English, and National Council of Teachers of Mathematics, have
already developed or are developing content standards in mathematics,
history, civics, geography, English, and other disciplines.  NCEST
distinguishes content standards from performance standards as
follows:  Content standards describe the knowledge, skills, and other
understandings that schools should teach for students to attain high
levels of competency in challenging subject matter; performance
standards define various levels of competency in the challenging
subject matter set out in the content standards. 


   PERSPECTIVES ON THE GOVERNING
   BOARD'S CONTINUED USE OF AN
   ITEM-CENTERED APPROACH TO
   STANDARD SETTING
------------------------------------------------------- Appendix III:4

Because the main issue underlying our and NAE's recommendations was
the Governing Board's item-centered approach to standard setting, we
consulted Governing Board officials and records to explain why the
Board continued to use this approach--with some modifications.  In
general, Board officials said that they had considered our and NAE's
recommendations, explored alternative standard-setting approaches,
sought advice from many different experts, and made some
modifications when their research warranted it but still concluded
that the item-centered approach was the best available.  The
following examples provide some perspective on the Board's position: 

  -- The Governing Board did not, as we recommended, withdraw its
     instructions to the Department for reporting the 1992 NAEP
     results because, after the 1990 effort and before beginning the
     1992 process, Governing Board officials said they consulted with
     representatives from NCES and several national research firms
     and education associations that unanimously advised them that
     its standard-setting approach was the best known procedure.\34

  -- As we reported in 1993, the Governing Board knew of some of the
     flaws in its 1990 procedures and acted to correct them for 1992. 
     For example, instead of a group of ad hoc advisers, the Board
     secured an experienced contractor to set the 1992 standards who
     made improvements that strengthened the item-centered approach. 
     These included careful attention to panelist selection, improved
     training, development of guiding definitions for each subject
     and grade before beginning the item judgment process, and review
     of the reliability of the judgment results.  We concluded,
     however, that these changes did not address the fundamental
     problem of finding a test score that the Board could interpret
     validly using its definitions and descriptions. 

  -- Regarding NAE's recommendation urging the Governing Board to
     altogether discontinue using its item-centered approach for
     setting standards, the Board contracted with two national
     testing experts to conduct an independent review of NAE's study. 
     After evaluating NAE's study and the background studies, both
     experts concluded that the evidence did not warrant NAE's
     conclusions.  State testing directors who reviewed NAE's study
     and heard the experts' rebuttal concluded that the Board should
     continue using its standard-setting approach, Board officials
     reported. 

  -- The Governing Board and NCES jointly sponsored an October 1994
     national conference of testing experts to discuss setting
     standards for large-scale assessments.  No consensus emerged
     from this conference regarding an alternative method to be used
     in place of the item-centered approach. 

  -- Before awarding the 1994 contract for developing performance
     standards, the Governing Board instructed all offerors to
     propose alternative designs and methodologies and to respond to
     issues raised by NAE and us, according to a Governing Board
     summary of its response.  The Board reported that the contractor
     searched the literature and sought advice from many national and
     international consultants but found no better alternatives to
     the item-centered approach.  The contractor also explored new
     approaches that called for judges to evaluate whole test
     booklets but reported that the judges chose to evaluate items
     individually. 

  -- The contractor is experimenting with other approaches, including
     some recommended by NAE, to setting performance standards for
     the 1998 assessment, the Governing Board reported. 



                                   Table III.1
                     
                         Our 1993 Recommendations and the
                      National Assessment Governing Board's
                            Actions Taken in Response

Our recommendation\a                     Governing Board action\b
---------------------------------------  ---------------------------------------
Because the current Governing Board's
approach to setting standards has
yielded unsupported interpretations of
NAEP scores, we recommend that

1. The Governing Board withdraw its      1. Action taken not fully responsive.
instructions to NCES to publish 1992     The Board did not follow this
NAEP results primarily in terms of       recommendation, but it did change its
achievement levels. The conventional     methods and reporting practices to
approach to scoring interpretation       guard against misleading
should be retained until an alternative  interpretations of the levels.
has been shown to be sound.

2. The Chairman of the Governing Board   2. Action completed. A conference to
and NCES Commissioner develop a joint    examine standard-setting methods was
plan and schedule for a review of the    held in 1994. Methods have been
Board's achievement levels approach,     amended.
taking into account evaluations under
way and providing for additional
activities as needed. The NCES
Commissioner should determine whether
the Governing Board's approach will
produce invalid interpretations of NAEP
scores and should not be pursued or
whether the approach is sufficiently
promising that a specific plan for
preparing for NCES prepublication
review should be designed and
implemented.

3. The Governing Board should withdraw   3. Action taken not fully responsive.
its policy of applying the 1990          The Board did not withdraw its policy,
achievement levels approach to future    but it did change its approach.
NAEP tests and join with NCES in
exploring alternatives for setting both
content-based and overall performance
standards regarding NAEP. This inquiry
should examine issues of purpose,
technical feasibility, cost, fairness,
credibility, and usefulness.

To strengthen the Governing Board's
capacity to give sound policy
direction, we recommend that the
Governing Board

4. Obtain NCES review of the technical   4. Action completed. The Board
strengths and weaknesses of proposed     instituted a formal procedure to obtain
policies that implement the Governing    NCES review.
Board's statutory responsibilities
before making final decisions on such    5. Action taken not fully responsive.
policies.                                The Board's response stated that
                                         policies already in place provide for
5. Analyze the probable effect of        adequate monitoring and proposed no
proposed policies on NAEP's ability to   additional action.
present achievement fairly and
accurately and to support both valid     6. Action completed. The 1992 and 1994
and reliable trend reporting.            procedures were pilot tested.

6. Pilot test and thoroughly evaluate
any new design or analysis procedure     7. Action taken not fully responsive.
before it is fully implemented and       The Board reported it had no plan to
results are reported.                    issue publications that would require
                                         the adoption of technical standards but
7. Adopt standards of technical quality  that if such plans arose, it would
(to be applied internally) for           apply standards and reviews.
publications issued under its own
authority and also secure competent
external technical review of such
publications before authorizing their
release.


8.The Chairman of the Governing Board    8. Action completed. The Board has
should review actions taken regarding    undertaken the review.
its statutory responsibility in the
past 2 years, identify those whose
technical consequences have not been
sufficiently examined, and secure
technical review as necessary to ensure
that these actions will generate no
unanticipated technical difficulties in
the future.

9. The Chairman of the Governing Board   9. Action taken not fully responsive.
should review proposed policy to ensure  The Board stated that its policy will
that the Board prescribes policy ends,   remain in force but proposes no
not technical details of                 specific actions.
implementation.

10. Regarding the Governing Board's      10. Action completed. The Board reports
membership, the Board should nominate    that new appointees have been
for the testing and measurement          appropriately qualified.
positions people who are trained in
designing and analyzing large-scale
educational tests. The Board should
also ensure that two or more of its
elected officials, educators, and
representatives of the general public
have significant technical knowledge
and experience.
--------------------------------------------------------------------------------
\a Our recommendations are from Educational Achievement Standards: 
NAGB's Approach Yields Misleading Interpretations (GAO/PEMD-93-12,
June 23, 1993). 

\b We identified the Governing Board's actions by reviewing the
status of our recommendations (made in report GAO/PEMD-93-12) as of
Oct.  31, 1995. 



(See figure in printed edition.)Appendix IV

--------------------
\34 The Governing Board reports that representatives from the
Educational Testing Service, the National Council of Teachers of
Mathematics, American College Testing, and the Mathematics Sciences
Education Board were included in this meeting. 


COMMENTS FROM THE NATIONAL
ASSESSMENT GOVERNING BOARD
========================================================= Appendix III



(See figure in printed edition.)




(See figure in printed edition.)Appendix V
COMMENTS FROM THE DEPARTMENT OF
EDUCATION
========================================================= Appendix III


GAO CONTACTS AND STAFF
ACKNOWLEDGMENTS
=========================================================== Appendix 0

GAO CONTACTS

Eleanor L.  Johnson, Assistant Director, (202) 512-7209
Barbara A.  Billinghurst, Evaluator-in-Charge (206) 287-4867

STAFF ACKNOWLEDGMENTS

In addition to those named above, the following individuals
contributed to the report:  Peter J.  Bylsma helped plan the job,
collect information, and prepare an early draft of the letter;
Stanley G.  Stenersen helped organize the report topics and write
report sections; Julie A.  Rachiele searched databases for textbook
authors; and Mary W.  Reich and Dayna K.  Shah provided legal advice. 


*** End of document. ***