[Federal Register Volume 63, Number 207 (Tuesday, October 27, 1998)] [Rules and Regulations] [Pages 57570-57575] From the Federal Register Online via the Government Publishing Office [www.gpo.gov] [FR Doc No: 98-28729] [[Page 57569]] _______________________________________________________________________ Part VII Department of Education _______________________________________________________________________ 34 CFR Part 702 Standards for Conduct and Evaluation of Activities Carried Out by the Office of Educational Research and Improvement (OERI)--Evaluation of the Performance of Recipients of Grants, Cooperative Agreements, and Contracts; Final Rule Federal Register / Vol. 63, No. 207 / Tuesday, October 27, 1998 / Rules and Regulations [[Page 57570]] DEPARTMENT OF EDUCATION 34 CFR Part 702 RIN 1850-AA54 Standards for Conduct and Evaluation of Activities Carried Out by the Office of Educational Research and Improvement (OERI)--Evaluation of the Performance of Recipients of Grants, Cooperative Agreements, and Contracts AGENCY: Office of Educational Research and Improvement, Department of Education. ACTION: Final regulations. ----------------------------------------------------------------------- SUMMARY: The Assistant Secretary establishes regulations pursuant to OERI's authorizing legislation, the Educational Research, Development, Dissemination, and Improvement Act of 1994. The major purpose of these standards is to ensure that the research, development, and dissemination activities carried out by the recipients of grants from and contracts and cooperative agreements with OERI meet the highest standards of professional excellence. EFFECTIVE DATE: These regulations take effect November 27, 1998. FOR FURTHER INFORMATION CONTACT: Sharon Bobbitt, U.S. Department of Education, 555 New Jersey Avenue, NW, Room 508C, Washington, D.C. Telephone: (202) 219-2126. Internet: (Sharon--B[email protected]). Individuals who use a telecommunications device for the deaf (TDD) may call the Federal Information Relay Service (FIRS) at 1-800-877-8339 between 8 a.m. and 8 p.m., Eastern time, Monday through Friday. Individuals with disabilities may obtain this document in an alternate format (e.g., Braille, large print, audiotape, or computer diskette) on request to the contact person listed in the preceding paragraph. SUPPLEMENTARY INFORMATION: Background On March 31, 1994, President Clinton signed Pub. L. 103-227, which includes Title IX, the Educational Research, Development, Dissemination, and Improvement Act of 1994 (the Act). The Act restructured OERI and provided it with a broad mandate to conduct an array of research, development, dissemination, and improvement activities aimed at strengthening the education of all students. Statutory Requirements The Act directed the Assistant Secretary to develop, in consultation with the National Educational Research Policy and Priorities Board (the Board), such standards as may be necessary to govern the conduct and evaluation of all research, development, and dissemination activities carried out by OERI to ensure that these activities meet the highest standards of professional excellence. The Board is responsible for reviewing and approving the standards. The legislation requires that the standards be developed in three phases. In the first phase, standards were created and promulgated to establish the peer review process and evaluation criteria to be used for the review of applications for grants and cooperative agreements and proposals for contracts. The final regulations setting out these standards were published on September 14, 1995 (60 FR 47808). In the second phase, standards were created and promulgated to establish the criteria to be used in reviewing potentially exemplary and promising educational programs. The final regulations setting out these standards were published on November 17, 1997 (62 FR 61427). In the third phase, which is the subject of these final regulations, the Act requires that OERI develop standards for evaluating and assessing the performance of all recipients of grants from and cooperative agreements and contracts with OERI. This evaluation must take place both during and at the conclusion of the performance of the grant, cooperative agreement, or contract, and must include the use of a system of peer review for the final assessment. In developing the standards, the Assistant Secretary was required to review the procedures utilized by the National Institutes of Health (NIH), the National Science Foundation (NSF), and other Federal departments or agencies engaged in research and development and to solicit recommendations from research organizations and members of the general public. OERI has reviewed the procedures used to evaluate the performance of recipients of grants, contracts, or cooperative agreements by several offices within NIH and NSF, the Office of Energy Research in the Department of Energy, the Food and Drug Administration, the National Institute of Standards and Technology, the National Aeronautics and Space Administration, and the University Research Initiative of the Department of Defense. Recommendations concerning these standards have been obtained from the American Educational Research Association, the Council for Educational Development and Research, and the Organization of Research Centers. Standards The standards have been developed by the Assistant Secretary in consultation with the Board. These standards cover all grants, cooperative agreements, and contracts administered by OERI, ranging from the smallest purchase orders and commissioned papers to the largest research projects and research centers. The standards:Require at least one interim assessment as well as a final assessment of the performance of recipients of grants, cooperative agreements, and contracts. Establish procedures for selecting peer review panels to conduct the assessments. Establish procedures and criteria that the peer review panels use in conducting the assessments. Establish specific additional criteria that peer review panels use in conducting the assessments for National Research and Development Centers, Regional Educational Laboratories, Field-Initiated Studies, and ERIC Clearinghouses. In an effort to fulfill the law's intention of ensuring high- quality research, development, and evaluation, OERI has developed standards in which interim and final assessments may be supplemented by a self-assessment by the recipient of a grant, cooperative, agreement, or contract. The Board and the Assistant Secretary believe that the collection and review of evidence on one's own performance is itself a useful tool for improvement. The Government Performance and Results Act requires the establishment of performance indicators for Department activities. Information collected pursuant to those indicators will be considered, as appropriate, in the evaluation of individual recipients. On February 24, 1998, the Assistant Secretary published a notice of proposed rulemaking (NPRM) for these standards in the Federal Register (63 FR 9393). These final regulations contain four major changes from the NPRM. These changes are fully explained in the ``Analysis of Comments and Changes'' elsewhere in this preamble. The major changes pertain to clarification of the purpose of the regulation, how OERI determines the number of interim assessments necessary, the role of Department of Education staff in the assessments, and the use of interim assessments as a source of information for the final assessment. Analysis of Comments and Changes In response to the Secretary's invitation in the NPRM, four parties [[Page 57571]] submitted comments on the proposed regulations. In addition to the public comment, comments from the Board's Subcommittee on Standards are addressed as required by the legislation. The full Board approved the final regulations at a meeting on September 18, 1998. An analysis of the comments and of the changes in the regulations since publication of the NPRM follows. Major issues are grouped according to subject with appropriate sections of the regulations referenced in parentheses. Technical and other minor changes--and suggested changes the Secretary is not legally authorized to make under the applicable statutory authority--are not addressed. Purpose (Sec. 702.1) Comments: Three commenters suggested that the purpose of the standards be clarified. One commenter suggested that the standards themselves cannot ensure the highest standards of professional excellence. Another commenter asked specifically whether the purpose for conducting assessments was to make decisions about future funding or to provide a system for monitoring and enhancing current and future projects. Discussion: The Secretary agrees that the purpose of the standards should be clarified to go beyond their stated statutory purpose, which is to ``ensure the highest standards of professional excellence,'' to include the objectives of continuously improving the quality of funded activities and of considering the results as one of the factors in determining continuation funding for multi-year awards. Changes: Section 702.1 has been modified to include a provision that the purpose of the standards is to provide feedback to help improve the quality of funded activities and to provide information for consideration as continuation funding decisions are made. Additional Activities that May be Evaluated (Sec. 702.3) Comment: One commenter thought that the statement that these standards could be applied to other activities funded by the Department was too broad and should be deleted. Discussion: The Secretary believes that this statement is necessarily broad to allow all Department programs to use these standards, when appropriate, to assess the performance of any of their funded activities without developing their own unique regulations. This statement is also consistent with the earlier standards which established the peer review process and evaluation criteria to be used for the review of applications for grants and cooperative agreements and proposals for contracts. Changes: None. Number of Interim Assessments (Sec. 702.4) Comments: Two commenters suggested changes to this provision. One commenter suggested since there may be more than one interim assessment, that it be clear in Sec. 702.4(d)(1). The OERI Board suggested that the requirements for a single interim assessment for total awards of $5,000,000 or less be modified to reflect total awards of $3,000,000 or less. Discussion: In response to the comments, the Secretary now believes that considerations such as difficulty in achieving project objectives rather than the dollar levels of awards should determine whether a particular project merits more than one interim assessment. Elimination of the dollar threshold clarifies the original intent of this section which is to require that all awards receive one interim assessment. More than one interim assessment will be performed only when a recipient is having difficulty achieving project objectives as determined by the initial interim assessment or through the monitoring efforts of Department of Education staff. The Assistant Secretary will make the determination of the number of interim assessments on a case- by-case basis. Changes: Section 702.4(b) has been modified to delete the dollar threshold and to reflect that all awards will receive at least one interim assessment. A new paragraph 702.4(c) has been added to clarify that the Assistant Secretary will require more than one interim assessment when a recipient has been identified, either in the initial interim review or through monitoring efforts of Department of Education staff, as having difficulty in achieving project objectives. Former paragraph 702.4(c) has been redesignated as Sec. 702.4(d). Section 702.4(d)(1) has been modified to define an interim assessment as ``any assessment'' conducted during a recipient's period of performance. Definitions (Sec. 702.5) Comment: One commenter suggested that the terms referred to in this section include the specific definitions and not references to the OERI statute and to the Education Department General Administrative Regulations. Discussion: The Secretary believes that providing the citations for specific terms rather than the definitions themselves keeps regulations short and concise while still cross referencing easily accessible resources for the definitions. Changes: None. Characteristics of Peer Reviewers (Sec. 702.10) Comment: One commenter suggested that paragraph 702.10(a) ``(4) knowledge of a broad range of education policies and practices;'' be deleted from the list of knowledge and expertise required of peer reviewers, because it is redundant with the other criteria and is very vague. Discussion: The Secretary believes that this criterion provides for a balance between specific program knowledge and a broader perspective of education policies and practices and is therefore not redundant with the other, more focused, characteristics required of peer reviewers. Changes: None. Role of Department Staff (Sec. 702.10) Comments: Two commenters expressed concern over the appropriate role of the OERI staff in the review process. One commenter urged the Department to use all outside reviewers. The other commenter acknowledged the knowledge and skills of the OERI staff but suggested that staff not serve as peer reviewers within the primary division of an agency in which they work and that each peer review panel be limited to one Department staff person. This commenter suggested that the staff focus on the important role of mentoring and designing competitions. Discussion: The Secretary agrees that the primary role of the OERI staff should be management of competitions including assessing the results of peer reviews and monitoring awards. The Secretary believes that the purpose of the peer review process should be to acquire the perspective of outside experts independent of OERI. The Secretary also believes that there may be exceptional circumstances where expertise resides in OERI or in the Department, or where outside reviewers are not required such as in the review of small purchase orders. The exceptions should be determined by the Assistant Secretary. Changes: Section 702.10(d) has been reworded to preclude OERI and other Department staff from serving as peer reviewers except in exceptional circumstances as determined by the Assistant Secretary. Conflict of Interest (Sec. 702.11) Comment: One commenter was concerned that while the conflict of interest requirements were ``legally correct'' they failed to address the problem occasioned by reviewers who [[Page 57572]] may have ideological or methodological view points that differ from those of the recipient to be evaluated, or who are affiliated with competing institutional organizations. Discussion: The commenter appears to be concerned that the proposed conflict of interest provision does not address the potential problem of bias on the part of a panel against a particular grantee on ideological or other grounds. The Secretary first believes that it is essential to retain the present language, which parallels the provision in the standards at 34 CFR 701.11(c), because it highlights the important issue of improper financial gain or the appearance of improper gain. However, the Secretary agrees that adding a requirement to the effect that panels selected by the Assistant Secretary reflect a broad range of perspectives could strengthen the regulation. Changes: A new paragraph ``(c)'' has been added to Sec. 702.13 requiring the Assistant Secretary, to the greatest extent feasible, to select peer reviewers for each evaluation who represent a broad range of perspectives. Sources of Information (Secs. 702.22 and 702.23) Comment: One commenter suggested that the use of Government Performance and Results Act (GPRA) information should be encouraged rather than required for both interim and final assessments. The commenter is concerned that information currently being collected under GPRA to evaluate the effectiveness of a program or a system-level activity will not provide information relevant to the assessment of individual awards under that program or system-level activity and therefore should not be required. Discussion: The Secretary agrees that information obtained by GPRA- related reports on the effectiveness of a program or system level activity, e.g., how effectively a program is meeting the overall objectives defined for it in its authorizing legislation, may not necessarily include information related to an individual award being reviewed under this regulation. However, the Secretary believes that information on the effectiveness of the particular program under which a recipient receives funding will help to provide a context for the review of an individual award and must be considered by the panel. Moreover, these regulations make it clear that the GPRA information is only one of a number of sources used in conducting the review. Changes: None. Comment: One commenter suggested that the findings and information from interim assessments would be an important source of information for the final assessments and should be included under Sec. 702.23(a). Discussion: The Secretary agrees that the results of interim assessments should be a source of information for final assessments. Change: Section 702.23(a) has been modified to add a new paragraph (Sec. 702.23(a)(5)) to require that the results of interim assessments be considered as a source of information for final assessments. Evaluation Criteria (Sec. 702.24) Comments: Two commenters suggested changes to this section. One commenter suggested that there be a single menu of criteria for the standards, because the proposed menu is too long. The second commenter suggested that since Field Initiated Studies are not likely to provide services, the word ``services'' be deleted from the criterion in Sec. 702.24(e)(4)(ii): ``* * * addresses issues of national significance through its products or services, or both.'' Discussion: The Secretary believes the current menu approach provides a comprehensive strategy for assessing the performance of all activities, ranging from the smallest purchase order to the largest research investments. The categories in the regulation reflect the specific authorities in the OERI statute. In addition, the menu provides for other criteria for future research investments that do not fit within the statutory authorities yet also must be assessed. A single menu would, of necessity, be too generic to apply to the wide range of activities covered by these standards. The Secretary agrees that assessing ``services'' is not appropriate for Field Initiated Studies projects. Change: Section 702.24(e)(4)(ii) has been modified to delete the word, ``services.'' Paperwork Reduction Act of 1995 Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number assigned to the collection of information in these final regulations is displayed at the end of the affected sections of the regulations. Assessment of Educational Impact In the NPRM the Secretary requested comments on whether the proposed regulations would require transmission of information that is being gathered by or is available from any other agency or authority of the United States. Based on the response to the NPRM and on its own review, the Department has determined that the regulations in this document do not require transmission of information that is being gathered by or is available from any other agency or authority of the United States. Electronic Access to This Document Anyone may view this document, as well as all other Department of Education documents published in the Federal Register, in text or portable document format (pdf) on the World Wide Web at either of the following sites: http://ocfo.ed.gov/fedreg.htm http://www.ed.gov/news.html To use the pdf you must have the Adobe Acrobat Reader Program with Search, which is available free at either of the previous sites. If you have questions about using the pdf, call the U.S. Government Printing Office at (202) 512-1530 or, toll free, at 1-888-293-6498. Anyone may also view these documents in text copy only on an electronic bulletin board of the Department. Telephone: (202) 219-1511 or, toll free, 1-800-222-4922. The documents are located under Option G--Files/Announcements, Bulletins and Press Releases. Note: The official version of this document is the document published in the Federal Register. List of Subjects in 34 CFR Part 702 Education, Educational research, Reporting and recordkeeping requirements. (Catalog of Federal Domestic Assistance Number does not apply.) Dated: October 22, 1998. C. Kent McGuire, Assistant Secretary for Educational Research and Improvement. The Secretary amends Chapter VII of Title 34 of the Code of Federal Regulations by adding a new Part 702 to read as follows: [[Page 57573]] PART 702--STANDARDS FOR CONDUCT AND EVALUATION OF ACTIVITIES CARRIED OUT BY THE OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT (OERI)--EVALUATION OF THE PERFORMANCE OF RECIPIENTS OF GRANTS, COOPERATIVE AGREEMENTS, AND CONTRACTS Subpart A--General Sec. 702.1 What is the purpose of these standards? 702.2 What activities must be evaluated by these standards? 702.3 What additional activities may be evaluated by these standards? 702.4 When is performance assessed under these standards? 702.5 What definitions apply? Subpart B--Selection of Peer Review Panels 702.10 What are the characteristics of peer reviewers? 702.11 What constitutes a conflict of interest for grants and cooperative agreements? 702.12 What constitutes a conflict of interest for contracts? 702.13 How are peer reviewers selected for panels? Subpart C--The Evaluation Process 702.21 How does a peer review panel evaluate the performance of a recipient? 702.22 What information does a peer review panel consider for an interim assessment? 702.23 What information does a peer review panel consider for a final assessment? 702.24 What evaluation criteria must be used for performance assessments? Authority: 20 U.S.C. 6011(i), unless otherwise noted. Subpart A--General Sec. 702.1 What is the purpose of these standards? (a) The standards in this part implement section 912(i) of the Educational Research, Development, Dissemination, and Improvement Act of 1994 (the Act). (b) These standards establish criteria and a peer review process to provide recipients of OERI grants, cooperative agreements and contract awards with assessments of their projects. (1) The purpose of the assessments is to provide feedback to recipients to improve the quality of funded activities and to provide information to OERI as it determines if a recipient of a multi-year award merits continuation funding. (2) The criteria and peer review process are intended to address the statutory requirement that the research, development, and dissemination activities carried out by the recipients of grants from and contracts and cooperative agreements with the Office of Educational Research and Improvement (OERI) meet the highest standards of professional excellence. (Authority: 20 U.S.C. 6011(i)(2)(F)) Sec. 702.2 What activities must be evaluated by these standards? These standards apply to activities carried out by OERI using funds appropriated under section 912(m) of the Act including activities carried out by the following entities or programs: (a) The National Education Research Institutes. (b) The Office of Reform Assistance and Dissemination. (c) The Educational Resources Information Center. (d) The Regional Educational Laboratories. (e) The Teacher Research Dissemination Demonstration Program. (f) The Goals 2000 Community Partnerships Program. (g) The National Educational Research Policy and Priorities Board. (Authority: 20 U.S.C. 6011(i)(1)) Sec. 702.3 What additional activities may be evaluated by these standards? The Secretary may apply these standards to other activities funded by the Department, as appropriate. (Authority: 20 U.S. C 6011 (i)(1)) Sec. 702.4 When is performance assessed under these standards? (a) The Secretary will assess the performance of recipients of OERI grants, contracts, and cooperative agreements subject to these standards during and at the conclusion of their period of performance. (b) The Department requires at least one interim assessment by a peer review panel for all awards. (c) The Assistant Secretary will approve and require more than one interim assessment when an award is identified, either by the initial interim review or by Department of Education staff monitoring the award, as having difficulty in achieving project objectives. (d) A final assessment by a peer review panel is required for all awards. (e) As used in this part-- (1) Interim assessment is any assessment conducted during a recipient's period of performance. (2) Final assessment is one conducted at the conclusion of a recipient's period of performance. (Authority: 20 U.S.C. 6011(i)(2)(F)) Sec. 702.5 What definitions apply? (a) Definitions in the Educational Research, Development, Dissemination, and Improvement Act of 1994. The following terms used in this part are defined in 20 U.S.C. 6011(l)(1): Development Dissemination Educational Research (b) Definitions in the Education Department General Administrative Regulations. The following terms used in this part are defined in 34 CFR 77.1: Application Award Department Grant Project Secretary (c) Definitions in the Federal Acquisition Regulation. The following term used in this part is defined in 48 CFR Chapter 1: Contract Proposal. (Authority: 20 U.S.C. 6011(i)(2)(F) Subpart B--Selection of Peer Review Panels Sec. 702.10 What are the characteristics of peer reviewers? (a) The Assistant Secretary selects each peer reviewer. Each peer reviewer must have the necessary knowledge and expertise in the area of the project being reviewed to evaluate the performance of a recipient. This experience may include-- (1) Expert knowledge of subject matter in the area of the activities to be reviewed; (2) Expert knowledge of theory or methods or both in the area of the activities to be reviewed; (3) Practical experience in the area of the activities or type of institution or both to be reviewed; (4) Knowledge of a broad range of education policies and practices; (5) Experience in managing complex organizations; or (6) Expertise and experience in evaluation theory and practice. (b) Each peer reviewer must be free of conflict of interest, as determined in accordance with Sec. 702.11 or Sec. 702.12. (c) The Assistant Secretary may solicit nominations for peer reviewers from professional associations, nationally recognized experts, and other sources. (d) OERI and other Department staff who possess the qualifications in paragraphs (a) and (b) of this section may serve as peer reviewers only in exceptional circumstances as determined by the Assistant Secretary. (Authority: 20 U.S.C. 6011(i)(2)(B)) Sec. 702.11 What constitutes a conflict of interest for grants and cooperative agreements? A peer reviewer assessing the performance of the recipient of a grant [[Page 57574]] from or cooperative agreement with OERI is considered an employee of the Department for the purposes of conflict of interest analysis. As an employee of the Department, the peer reviewer is subject to the provisions of 18 U.S.C. 208, 5 CFR 2635.502, and the Department's policies used to implement those provisions. (Authority: 20 U.S.C. 6011(i)(2)(B)) Sec. 702.12 What constitutes a conflict of interest for contracts? A peer reviewer assessing the performance of the recipient of a contract with OERI is considered an employee of the Department in accordance with the Federal Acquisition Regulation (FAR), 48 CFR 3.104- 4(h)(2). As an employee of the Department, the peer reviewer is subject to the provisions of the FAR, 48 CFR Part 3, Improper Business Practices and Personal Conflict of Interest. (Authority: 41 U.S.C. 423) Sec. 702.13 How are peer reviewers selected for panels? (a) The Assistant Secretary assigns peer reviewers to panels that conduct the performance assessments. (b) The Assistant Secretary may establish panels by category of recipient, such as a panel to review the performance of all Regional Educational Laboratories. Each recipient is evaluated individually by reviewers who have been assigned to this type of panel. (c) In establishing panels, the Assistant Secretary, to the greatest extent feasible, selects peer reviewers for each evaluation who represent a broad range of perspectives. (Authority: 20 U.S.C. 6011(i)(2)(B)) Subpart C--The Evaluation Process Sec. 702.21 How does a peer review panel evaluate the performance of a recipient? (a) In each evaluation, a peer review panel-- (1) Considers relevant information about the recipient's performance, as described in Secs. 702.22 and 702.23; and (2) Makes judgments about the recipient's performance, using the criteria in Sec. 702.24. (b) Each peer reviewer prepares a report based on the reviewer's assessment of the quality of the project according to the evaluation criteria. (c) After each peer reviewer has evaluated each project independently, the panel may be convened to discuss the strengths and weaknesses of the project. Each reviewer may then independently re- evaluate each project with appropriate changes made to the written report. (d) The report of the interim assessment must include any recommendations the peer reviewer may have for improving the recipient's performance. (e) The report of the final assessment must contain each peer reviewer's evaluative summary of the recipient's performance, from the beginning of the contract, grant, or cooperative agreement to its conclusion. (Authority: 20 U.S.C. 6011(i)(2)(F)) Sec. 702.22 What information does a peer review panel consider for an interim assessment? (a) Sources of information for the interim assessment must include-- (1) The original request for proposals or grant announcement and the contract proposal or grant application; (2) Documentation of any changes in the work described in the contract, grant, or cooperative agreement, including reasons for the changes; (3) Any progress reports delivered to the Department or made available to the public by the recipient; (4) Examples of products delivered to the Department or made available to the public by the recipient; (5) Any relevant reports written by OERI staff, including reports of site visits by OERI staff; (6) Any performance evaluations conducted under the FAR or the Education Department General Administrative Regulations (34 CFR Part 75). (7) Any relevant information provided by the recipient in response to Government Performance and Results Act (GPRA) (Pub. L. 103-62) requirements; and (8) Any reports from program evaluations commissioned by the Department. (b) Sources of information for the interim assessment may also include-- (1) A self-assessment, prepared by the recipient, addressing the criteria in Sec. 702.24; (2) One or more site visits by the peer review panel; (3) One or more oral or written presentations to the panel by the recipient describing its performance; or (4) Other information about the recipient's performance. (Approved by the Office of Management and Budget under control number 1850-0746) (Authority: 20 U.S.C. 6011(i)(2)(F)) Sec. 702.23 What information does a peer review panel consider for a final assessment? (a) Sources of information for the final assessment must include-- (1) The original request for proposals or application notice and the contract proposal or grant application, together with documentation of any changes in the work described in the proposal or application, including reasons for the changes; (2) If consistent with the recipient's contract, grant, or cooperative agreement with OERI, a written report or oral presentation or both by the recipient summarizing its activities and accomplishments; (3) Any relevant information provided by the recipient in response to Government Performance and Results Act (GPRA) (Pub. L. 103-62) requirements; (4) Any reports from program evaluations commissioned by the Department; and, (5) Any relevant information provided by the interim assessment. (b) The final assessment may also include other sources of information, such as one or more of those listed in Sec. 702.22. (Approved by the Office of Management and Budget under control number 1850-0746) (Authority: 20 U.S.C. 6011(i)(2)(F)) Sec. 702.24 What evaluation criteria must be used for performance assessments? (a) Peer reviewers (and those recipients who conduct self- evaluations) shall use the criteria in paragraph (b) of this section to assess performance and, in case of interim assessments, to identify areas in which the performance of recipients may need improvement. (b) The following evaluation criteria are to guide the assessment process undertaken by peer reviewers. The peer reviewers determine the extent to which recipients meet these criteria: (1) Implementation and management. (i) Peer reviewers shall consider the degree to which the recipient has fully executed its program of work. In doing so, peer reviewers shall consider evidence on the extent to which the recipient completes the work described in the approved application or contract, including any approved modifications, in the time period proposed and in an efficient manner. (ii) In examining the degree of implementation, peer reviewers may also consider evidence on the extent to which-- (A) The recipient implements and utilizes a quality assurance system for its products or services or both; and (B) The recipient conducts self-assessment or self-evaluation activities, including periodically seeking out independent critiques and evaluations of its work, and uses the results to improve performance. (2) Quality. (i) Peer reviewers shall consider the degree to which the [[Page 57575]] recipient's work approaches or attains professional excellence. In determining quality, peer reviewers shall consider evidence on the extent to which-- (A) The recipient utilizes processes, methods, and techniques appropriate to achieve the goals and objectives for the program of work in the approved application; and (B) The recipient applies appropriate processes, methods, and techniques in a manner consistent with the highest standards of the profession. (ii) In determining quality, peer reviewers may also consider the extent to which the recipient conducts a coherent, sustained program of work informed by relevant research. (3) Utility. (i) In determining the utility of the recipient's products or services or both, peer reviewers shall consider evidence on the extent to which the recipient's work (including information, materials, processes, techniques, or activities) is effectively used by and is useful to its customers in appropriate settings. (ii) In determining utility, peer reviewers may also consider the extent to which the recipient has received national recognition; e.g., articles in refereed journals and presentations at professional conferences. (4) Outcomes and impact. (i) Peer reviewers shall consider the results of the recipient's work. In examining outcomes and impact, peer reviewers shall consider evidence on the extent to which-- (A) The recipient meets the needs of its customers; and (B) The recipient's work contributes to the increased knowledge or understanding of educational problems, issues, or effective strategies. (ii) In examining outcomes and impact, peer reviewers may also consider the extent to which recipients address issues of national significance through its products or services or both. (c) For National Research and Development Centers, peer reviewers also shall consider evidence on the extent to which recipients meet the following criteria: (1) Quality. (i) The recipient uses a well-conceptualized framework and sound theoretical and methodological tools in conducting professionally rigorous studies; and (ii) The recipient conducts work of sufficient size, scope, and duration to produce sound guidance for improvement efforts and future research. (2) Utility. The recipient documents, reports, and disseminates its work in ways to facilitate the effective use of its work in appropriately targeted settings. (3) Outcomes and impact. (i) The recipient's work contributes to the development and advancement of theory in the field of study, including its priority area; and (ii) The recipient addresses issues of national significance through its products or services or both. (d) For the Regional Educational Laboratories, peer reviewers also shall consider evidence on the extent to which recipients meet the following criteria: (1) Quality. (i) The recipient utilizes a well-conceptualized framework and sound theoretical and methodological tools in conducting professionally rigorous studies; (ii) The recipient conducts work of sufficient size, scope, and duration to produce sound guidance for improvement efforts; and (iii) The recipient's products are well tested and based on sound research. (2) Utility. The recipient documents, reports, and disseminates its work in ways to facilitate its effective use in appropriately targeted settings, particularly in school improvement efforts of States and localities. (3) Outcomes and impact. (i) The recipient assists States and localities to implement comprehensive school improvement strategies through the provision of research-based information (including well- tested models and strategies), materials and assistance; and (ii) The recipient's work results in widespread access to information regarding research and best practices, particularly within its region. (e) For Field-Initiated Studies, peer reviewers also shall consider evidence on the extent to which recipients meet the following criteria: (1) Implementation and management. The recipient's work responds to the goals, objectives and mission of the National Institute from which it is funded. (2) Quality. The recipient utilizes a well-conceptualized framework and sound theoretical and methodological tools in conducting professionally rigorous studies. (3) Utility. The recipient documents, reports, and disseminates its work in ways to facilitate its effective use in appropriately targeted settings. (4) Outcomes and impact. (i) The recipient's work contributes to the development and advancement of theory and knowledge in the field of study; and (ii) The recipient addresses issues of national significance through its products. (f) For the ERIC Clearinghouses, peer reviewers also shall consider evidence on the extent to which recipients meet the following criteria: (1) Quality. The recipient applies an integrated approach to acquiring and disseminating significant and high-quality educational literature and materials to maintain and enhance the ERIC database. (2) Utility. The recipient contributes to the development of the ERIC database as a source of literature and materials that reflects trends and issues within its scope. (3) Outcomes and impact. (i) The recipient meets the informational and educational needs of its customers through dissemination and outreach approaches and the development of an array of print and non- print materials; and (ii) The recipient provides national leadership on the use of current computer, networking, and information technology. (Approved by the Office of Management and Budget under control number 1850-0746) (Authority: 20 U.S.C. 6011(i)(2)(F)) [FR Doc. 98-28729 Filed 10-26-98; 8:45 am] BILLING CODE 4000-01-U