[Federal Register Volume 64, Number 89 (Monday, May 10, 1999)]
[Notices]
[Pages 25028-25029]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 99-11660]


-----------------------------------------------------------------------

DEPARTMENT OF EDUCATION


National Assessment Governing Board; Public Forum

AGENCY: National Assessment Governing Board; Education.

ACTION: Notice of information collection activity.

-----------------------------------------------------------------------

SUMMARY: This notice announces that the National Assessment Governing 
Board (NAGB) will submit an Information Collection Request (ICR) to the 
Office of Management and Budget for approval.
    The ICR is: An Investigation of Alternative Methods for Scale 
Anchoring and Item Mapping in the National Assessment of Educational 
Progress.

DATES: Public comments must be submitted on or before June 9, 1999.

ADDRESSES: Written comments should be submitted by June 9, 1999. Mail 
to Patricia Hanick, NAEP ALS Project Manager, ACT, Inc., 2255 North 
Dubuque Road, P.O. Box 168, Iowa City, IA 52243-0168. Copies of the 
complete ICR and accompanying appendices may be obtained from the NAEP 
ALS Project Manager at the address above.
    Comments may also be submitted electronically by sending electronic 
mail (e-mail) to [email protected]. Electronic comments must be identified 
by the title of the ICR. No confidential business information should be 
submitted through e-mail.
    Information submitted as a comment concerning this document may be 
claimed confidential by marking any part or all of that information as 
confidential business information (CBI). Information so marked will not 
be disclosed except in accordance with the procedures set forth in 40 
CFR part 2. A copy of the comment that does not contain CBI must be 
submitted for inclusion in the public record. Information not marked 
confidential may be disclosed publicly by NAGB without prior notice.

FOR FURTHER INFORMATION CONTACT: Patirica Hanick, NAEP ALS Project 
Manager, ACT, Inc., 2255 North Dubuque Road, P.O. Box 168, Iowa City, 
IA 52243-0168, Telephone: (319) 337-1452 or (800) 525-6930, e-mail: 
[email protected].

SUPPLEMENTARY INFORMATION: Electronic copies of this ICR can be 
obtained from the contact person listed above.

I. Information Collection Request

    NAGB is seeking comments on the following Information Collection 
Request (ICR).
    Title: An Investigation of Alternative Methods for Scale Anchoring 
and Item Mapping in the National Assessment of Educational Progress.
    Affected Entities: Parties affected by this information collection 
are persons who served as panelists for pilot studies, achievement 
levels-setting (ALS) meetings, and validation research for the 1998 
Science NAEP.
    Abstract. The purpose of this information collection activity is to 
gather research information for NAGB to be used in evaluating the 
procedures for the selection of exemplar items to represent student 
performance on NAEP. Exemplar items are used as one of the primary 
means of communicating student performance on NAEP. The criteria for 
selecting exemplar items is critical to the ``message'' portrayed by 
the items when reporting the outcomes of NAEP to the public.
    Two statistical criteria currently guide the selection of exemplar 
items: One based on item difficulty, and the other on item 
discrimination. Part One of the study will examine that statistical 
criteria used to select exemplar items by comparing various scale 
anchoring and item mapping methodologies. The results of these 
systematic comparisons will be judged not only statistically, but also 
according to the degree to which they agree with informed judgments 
about item difficulty. Collecting responses from informed judges is 
Part Two of the study. Because these individuals have participated in 
the process of setting achievement levels for the 1998 Science NAEP, 
they are likely to be keenly interested in the research study.
    In Part One of the study, several technical aspects of the 
anchoring process will be investigated:
    (1) Stringency of difficulty criterion (response probability 
criterion);
    (2) Point versus interval-based estimates;
    (3) Empirical versus model-based estimates;
    (4) Type of discrimination criterion.
    Before conducting the data analysis of the anchoring process, cross 
validation analyses of random half-samples of the data will be done. 
Four methods of analyzing the student data are planned:
    (1) Empirical/interval estimation;
    (2) Empirical/point estimation;
    (3) Model-based/interval estimation;
    (4) Model-based/point estimation.
    Three factors will be considered as the criteria to determine which 
anchoring/mapping method is best:
    (1) Consistency of results across subsamples;
    (2) The degree to which the results are supported by informed 
judgment;

[[Page 25029]]

    (3) The number of exemplar items produced by the method.
    In Part Two of the study each prospective respondent will be sent a 
selection of items from the 1990 Science NAEP Physical Science item 
pool. All of the items have been ``released'' for public review. 
Respondents will be asked to rank order the items according to the 
perceived level of difficulty. The ranking task will involve four 
overlapping sets of 8-9 items per set. Each item will be displayed on a 
separate card. Item sets will include obtained and bogus clusters. 
Participants will be asked if the items appear to cluster together, and 
if so, what are the common tasks or content areas that form the basis 
of the cluster. They will be asked to determine if the clusters reflect 
Basic, Proficient, or Advanced performance, based on the NAGB policy 
definitions of achievement levels. Finally, participants will be asked 
the following question:

    If you were told that American students can do item N, what 
would you assume this meant? Specifically, what percent of students 
would need to be able to answer correctly in order for you to agree 
that students can do the item? (40%, 50%, 60%, 70%, 80%, (90%, 
100%).

The results of Part One will then be evaluated according to the degree 
to which they agree with the informed judgments about item difficulty.
    The response rate for the survey used in Part Two is expected to be 
80% or higher. Only persons who served as panelists for pilot studies, 
achievement levels-setting meetings, and validation research for the 
1998 Science NAEP will be invited to participate. These individuals 
have shown keen interest in NAEP by their participation in the ALS 
process. The mailing list for these individuals is being updated, and 
those who can be contacted will be asked in advance if they will agree 
to participate in the survey. Only persons who consent will receive the 
survey and accompanying materials. Follow-up procedures will include 
mailing reminder postcards, making telephone calls, and sending 
replacement materials.
    No third party notification or public disclosure burden is 
associated with this collection.
    Burden Statement: The estimated total respondent burden is 228 
hours, and the average burden per respondent is 1.5 hours. This is a 
one-time survey. Neither small business nor other small entities are 
included in the survey.

II. Request for Comments

    NAGB solicits comments to:

    (i) Evaluate whether the proposed collection of information is 
an appropriate method to determine the ``message'' portrayed by the 
items regarding student performance on NAEP reported to the public.
    (ii) Enhance the accuracy, quality, and utility of the 
information to be collected.
    (iii) Evaluate whether the design of this survey maximizes the 
response rate, i.e. the number of selected persons who will respond.

    Records are kept of all public comments and are available for 
public inspection at the U.S. Department of Education, National 
Assessment Governing Board, Suite 825, 800 North Capitol Street, NW, 
Washington, DC from 8:30 a.m. to 5 p.m.

    Dated: May 5, 1999.
Roy Truby,
Executive Director, National Assessment Governing Board.
[FR Doc. 99-11660 Filed 5-7-99; 8:45 am]
BILLING CODE 4000-01-M