[House Hearing, 113 Congress]
[From the U.S. Government Publishing Office]



 
                 SCIENTIFIC INTEGRITY AND TRANSPARENCY

=======================================================================

                                HEARING

                               BEFORE THE

                        SUBCOMMITTEE ON RESEARCH

              COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY
                        HOUSE OF REPRESENTATIVES

                    ONE HUNDRED THIRTEENTH CONGRESS

                             FIRST SESSION

                               __________

                         TUESDAY, MARCH 5, 2013

                               __________

                           Serial No. 113-10

                               __________

 Printed for the use of the Committee on Science, Space, and Technology


       Available via the World Wide Web: http://science.house.gov



                  U.S. GOVERNMENT PRINTING OFFICE
79-929                    WASHINGTON : 2013
-----------------------------------------------------------------------
For sale by the Superintendent of Documents, U.S. Government Printing Office, 
http://bookstore.gpo.gov. For more information, contact the GPO Customer Contact Center, U.S. Government Printing Office. Phone 202ï¿½09512ï¿½091800, or 866ï¿½09512ï¿½091800 (toll-free). E-mail, [email protected].  


              COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY

                   HON. LAMAR S. SMITH, Texas, Chair
DANA ROHRABACHER, California         EDDIE BERNICE JOHNSON, Texas
RALPH M. HALL, Texas                 ZOE LOFGREN, California
F. JAMES SENSENBRENNER, JR.,         DANIEL LIPINSKI, Illinois
    Wisconsin                        DONNA F. EDWARDS, Maryland
FRANK D. LUCAS, Oklahoma             FREDERICA S. WILSON, Florida
RANDY NEUGEBAUER, Texas              SUZANNE BONAMICI, Oregon
MICHAEL T. McCAUL, Texas             ERIC SWALWELL, California
PAUL C. BROUN, Georgia               DAN MAFFEI, New York
STEVEN M. PALAZZO, Mississippi       ALAN GRAYSON, Florida
MO BROOKS, Alabama                   JOSEPH KENNEDY III, Massachusetts
RANDY HULTGREN, Illinois             SCOTT PETERS, California
LARRY BUCSHON, Indiana               DEREK KILMER, Washington
STEVE STOCKMAN, Texas                AMI BERA, California
BILL POSEY, Florida                  ELIZABETH ESTY, Connecticut
CYNTHIA LUMMIS, Wyoming              MARC VEASEY, Texas
DAVID SCHWEIKERT, Arizona            JULIA BROWNLEY, California
THOMAS MASSIE, Kentucky              MARK TAKANO, California
KEVIN CRAMER, North Dakota           VACANCY
JIM BRIDENSTINE, Oklahoma
RANDY WEBER, Texas
CHRIS STEWART, Utah
VACANCY
                                 ------                                

                        Subcommittee on Research

                   HON. LARRY BUCSHON, Indiana, Chair
STEVEN M. PALAZZO, Mississippi       DANIEL LIPINSKI, Illinois
MO BROOKS, Alabama                   ZOE LOFGREN, California
STEVE STOCKMAN, Texas                AMI BERA, California
CYNTHIA LUMMIS, Wyoming              ELIZABETH ESTY, Connecticut
JIM BRIDENSTINE, Oklahoma            EDDIE BERNICE JOHNSON, Texas
LAMAR S. SMITH, Texas


                            C O N T E N T S

                         Tuesday, March 5, 2013

                                                                   Page
Witness List.....................................................     2

Hearing Charter..................................................     3

                           Opening Statements

Statement by Representative Larry Bucshon, Chairman, Subcommittee 
  on Research, Committee on Science, Space, and Technology, U.S. 
  House of Representatives.......................................     5
    Written Statement............................................     6

Statement by Representative Daniel Lipinski, Ranking Minority 
  Member, Subcommittee on Research, Committee on Science, Space, 
  and Technology, U.S. House of Representatives..................     7
    Written Statement............................................     8

                               Witnesses:

Dr. Bruce Alberts, Editor-in-Chief, Science Magazine and 
  Professor Emeritus of Biochemistry and Biophysics, University 
  of California - San Francisco
    Oral Statement...............................................     9
    Written Statement............................................    12

Dr. Victoria Stodden, Assistant Professor of Statistics, Columbia 
  University
    Oral Statement...............................................    20
    Written Statement............................................    22

Dr. Stanley Young, Assistant Director for Bioinformatics, 
  National Institutes of Statistical Sciences
    Oral Statement...............................................    48
    Written Statement............................................    51

Mr. Sayeed Choudhury, Associate Dean for Research Data Management 
  at Johns Hopkins University and Hodson Director of the Digital 
  Research and Curation Center
    Oral Statement...............................................    54
    Written Statement............................................    56

Discussion.......................................................    63

             Appendix I: Answers to Post-Hearing Questions

Dr. Bruce Alberts, Editor-in-Chief, Science Magazine and 
  Professor Emeritus of Biochemistry and Biophysics, University 
  of California - San Francisco..................................    74

Dr. Victoria Stodden, Assistant Professor of Statistics, Columbia 
  University.....................................................    80

Dr. Stanley Young, Assistant Director for Bioinformatics, 
  National Institutes of Statistical Sciences....................    86

Mr. Sayeed Choudhury, Associate Dean for Research Data Management 
  at Johns Hopkins University and Hodson Director of the Digital 
  Research and Curation Center...................................    92


                 SCIENTIFIC INTEGRITY AND TRANSPARENCY

                              ----------                              


                         TUESDAY, MARCH 5, 2013

                  House of Representatives,
                                   Subcommittee on Research
               Committee on Science, Space, and Technology,
                                                   Washington, D.C.

    The Subcommittee met, pursuant to call, at 10:01 a.m., in 
Room 2318 of the Rayburn House Office Building, Hon. Larry 
Bucshon [Chairman of the Subcommittee] presiding.
[GRAPHIC] [TIFF OMITTED] T9929.001

                     U.S. HOUSE OF REPRESENTATIVES

              COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY

                        SUBCOMMITTEE ON RESEARCH

                            hearing charter

                 Scientific Integrity and Transparency

                         tuesday, march 5, 2013
                        10:00 a.m. to 12:00 p.m.
                   2318 rayburn house office building

Purpose

    At 10 AM on Tuesday, March 5, 2013, the Subcommittee on Research 
will hold a hearing titled Scientific Integrity and Transparency. This 
hearing will provide Members an opportunity to understand the problem 
of access to underlying data from published research funded by the 
federal government, and why access to this underlying data is vital to 
scientific integrity and transparency for peer reviewed research. On 
March 29th, 2012 the Investigation and Oversight Subcommittee held a 
hearing entitled, ``Federally Funded Research: Examining Public Access 
and Scholarly Publication Interests.'' \1\ The focus of this past 
hearing was on open access to publications, whereas the focus of this 
hearing is on open access to data used in federal research.
---------------------------------------------------------------------------
    \1\ http://science.house.gov/hearing/subcommittee-investigations-
and-oversight-hearing-examining-public-access-and-scholarly

---------------------------------------------------------------------------
Witnesses

      Prof. Bruce Alberts, Professor of Biochemistry, 
University of California San Francisco

      Prof. Victoria Stodden, Assistant Professor of 
Statistics, Columbia University

      Dr. Stanley Young, Assistant Director for Bioinformatics, 
National Institute of Statistical Sciences

      Mr. Sayeed Choudhury, Associate Dean for Research Data 
Management at Johns Hopkins University and Hodson Director of the 
Digital Research and Curation Center

Overview

    The bedrock of the scientific process is the ability to replicate 
the experimental claims made by researchers. These claims include both 
the generation of data and the analysis of data by computer software 
and code. Scientists rarely reproduce the work of others since they 
neither have the time nor the resources to reliably replicate the work 
of their colleagues; instead, they often trust these claims and rely on 
the peer review process and their colleagues to share their data and 
analysis methods when needed. This exchange allows for scientists and 
companies to exploit the latest insights to develop new directions in 
their research, and allows them to maximize the impact of federal 
research investment. Thus, scientific progress cannot occur unless 
there is a strong culture of integrity and transparency.
    Unfortunately, the current system has demonstrated several flaws. 
The current incentive system rewards researchers who publish in 
journals, but preparation of data for others' use is not an important 
part of this reward structure. The process of peer review, which the 
scientific community views as its primary means to check scientific 
integrity in journal publications, oftentimes does not try to replicate 
the results of submitted papers. Fellow researchers conducting the peer 
review for publication rarely ask for the original data of the 
submitted paper they are reviewing, and focus instead on whether the 
claims made in the paper are plausible. They simply assume the 
underlying data is valid. In a recent study by Young and Karr, upwards 
of 90% of clinical trial claims for new medicines cannot be replicated. 
\2\ The inability to replicate published results is not unique to 
clinical trials and occurs across scientific disciplines. \3\
---------------------------------------------------------------------------
    \2\ http://science.house.gov/sites/republicans.science.house.gov/
files/documents/hearings/HHRG-112-SY20-WState-SYoung-20120203.pdf
    \3\ ``Again, and again, and again.'' p1225 Science Vol 334 2 
December 2011
---------------------------------------------------------------------------
    This hearing will attempt to understand the scope of the problem 
with scientific integrity, especially how thorough researchers deal 
with underlying data. This issue of scientific integrity should be 
differentiated from cases of scientists knowingly and intentionally 
committing scientific fraud, fabricating data, or plagiarism though 
these might be inter-related depending on individual circumstances. 
This hearing will focus primarily on how data is collected, shared, and 
analyzed by the scientific community and policies for what, how, and 
when federally funded research data should be shared, as well as the 
cost of making this data available to the scientific community and 
public. Current federal laws governing the sharing of data include the 
Data Access Act (DAA) of 1999 and the Information Quality Act (IQA) of 
2001. \4\ Introduced by Senator Richard Shelby, the DAA (sometimes 
known as ``the Shelby Amendment'' within the science community) 
requires that data from federally funded research be made available 
under the Freedom of Information Act procedures. The IQA requires the 
OMB to issue regulations for ensuring the quality and integrity of all 
information disseminated by federal agencies. However, the Government 
Accountability Office reported in September 2007 that federal agencies 
rarely monitor whether researchers make data available. \5\
---------------------------------------------------------------------------
    \4\ National Research Council, Ensuring the Integrity, 
Accessibility, and Stewardship of Research Data in the Digital Age 
(Washington, DC: National Academy Press), 2009.
    \5\ http://www.gao.gov/products/GAO-07-1172
---------------------------------------------------------------------------
    In response to these aforementioned issues, the Office of Science 
and Technology Policy (OSTP) released guidance to federal agencies on 
February 22nd about increasing access to the results of federally 
funded scientific research which includes a discussion about access to 
non-classified digital data. In this memo, OSTP outlines the following 
principles for federal funding agencies to follow when a issuing a data 
access plan \6\:
---------------------------------------------------------------------------
    \6\ http://www.whitehouse.gov/sites/default/files/microsites/ostp/
ostp-public-access-memo-2013
.pdf

      Maximize access to scientific data created with federal 
---------------------------------------------------------------------------
funds;

      Ensure that researchers develop data management plans, 
and allow inclusion for costs in proposals along with proper 
evaluations of these proposals;

      Include mechanisms to ensure compliance with data 
management plans and policies;

      Promote the deposit of data in publicly accessibly 
databases;

      Encourage cooperation with the private sector to improve 
data access and compatibility;

      Develop approaches for identifying/providing appropriate 
attribution to data sets;

      Support the training, education and workforce development 
related to data management; and

      Provide assessment of long-term needs for the 
preservation of scientific data.

    This hearing will address how such principles might best be 
implemented by federal research agencies and members of the scientific 
community conducting such research.
    Chairman Bucshon. The Subcommittee on Research will now 
come to order.
    Good morning. Welcome to today's hearing entitled 
``Scientific Integrity and Transparency.'' In front of you are 
packets containing the written testimonies, biographies and 
Truth-in-Testimony disclosures for today's witness panel. I 
recognize myself for five minutes for an opening statement.
    I want to welcome everyone to today's Research Subcommittee 
hearing on the issue of scientific integrity and transparency.
    An editorial in the March 29, 2012, edition of Nature 
magazine entitled: ``Must try harder: too many sloppy mistakes 
are creeping into scientific papers. Lab heads must look more 
rigorously at the data and at themselves.'' I found this 
editorial particularly interesting because of my background as 
a cardiothoracic surgeon and my professional interest in 
medicine. The editorial goes on to cite a recent study 
contained in this specific issue by Glenn Begley and Lee Ellis, 
which analyzes the low number of cancer research studies that 
have been converted into clinical success, and concludes that a 
major factor is the overall poor quality of published 
preclinical data. This is one of the many similar studies that 
I have read.
    The growing lack of scientific integrity and transparency 
has many causes but one thing is very clear: without open 
access to data, there can be neither integrity nor transparency 
from the conclusions reached by the scientific community. 
Furthermore, when there is no reliable access to data, the 
progress of science is impeded and leads to inefficiencies in 
the scientific discovery process. Important results cannot be 
verified, and confidence in scientific claims dwindles.
    The Federal Government is the main sponsor of basic 
scientific research, with over $140 billion spent in fiscal 
year 2013. The American scientific community has made enormous 
contributions in many scientific fields from federally 
sponsored research. I believe our Nation's scientists will 
continue to develop the breakthrough discoveries and 
innovations of tomorrow. However, scientists receiving federal 
funding need to be accountable and responsible stewards of 
taxpayers' resources. Hardworking Americans trust our 
scientists to be genuine and authentic in the way they conduct 
and share federally funded research.
    The focus of this hearing will be on scientific research 
data funded by the Federal Government. There are key issues 
that data-sharing policies should address including what is 
data, how it should be shared, when it should be shared, and 
what potential costs might result in making this data available 
to the research community. We want to maximize access to data 
while protecting personal privacy, avoid any negative impact on 
intellectual property rights and innovation, and preserve data 
without ridiculous cost or administrative burdens.
    In an attempt to begin addressing this issue, the Office of 
Science and Technology Policy released guidelines on February 
22nd of this year that recognized the problem of data access. 
These guidelines, intended for federal science agencies, are to 
be followed when determining a policy for public access to 
scientific data in digital formats. As part of this hearing, I 
look forward to hearing the witnesses' opinions on these 
federal guidelines.
    Our witnesses today offer input from a variety of 
scientific fields, as this problem is not exclusive to one 
scientific field, community or discipline. I would like to 
thank them for coming and taking the time to offer their 
expertise. I would also like to thank Ranking Member Lipinski 
and everyone else participating in today's hearing.
    [The prepared statement of Mr. Bucshon follows:]

              Prepared Statement of Chairman Larry Bucshon

    I want to welcome everyone to today's Research subcommittee hearing 
on the issue of scientific integrity and transparency.
    An editorial in the March 29, 2012 edition of Nature magazine was 
entitled: ``Must try harder: too many sloppy mistakes are creeping into 
scientific papers. Lab heads must look more rigorously at the data--and 
at themselves.'' I found this editorial particularly interesting 
because of my background as a cardiothoracic surgeon and my 
professional interest in medicine. The editorial goes on to cite a 
recent study (contained in this specific issue) by Glenn Begley and Lee 
Ellis which analyzes the low number of cancer-research studies that 
have been converted into clinical success, and concludes that ``a major 
factor is the overall poor quality of published pre-clinical data.'' 
This is one of many similar studies that I have read.
    The growing lack of scientific integrity and transparency has many 
causes but one thing is very clear: without open access to data, there 
can be neither integrity nor transparency from the conclusions reached 
by the scientific community. Furthermore, when there is no reliable 
access to data, the process of science is impeded and leads to 
inefficiencies in the scientific discovery process. Important results 
cannot be verified, and confidence in scientific claims dwindles.
    The federal government is the main sponsor of basic science 
research, with over $140 billion spent in fiscal year 2013. The 
American scientific community has made enormous contributions in many 
scientific fields from federally sponsored research. I believe our 
nation's scientists will continue to develop the breakthrough 
discoveries and innovations of tomorrow. However, scientists receiving 
federal funding need to be accountable and responsible stewards of tax-
payer resources. Hard-working Americans trust our scientists to be 
genuine and authentic in the way they conduct and share federally 
funded research.
    The focus of this hearing will be on scientific research data 
funded by the federal government. There are key issues that data-
sharing policies should address including: what is data, how it should 
be shared, when it should be shared, and what potential costs might 
result in making this data available to the research community. We want 
to maximize access to data while protecting personal privacy, avoid any 
negative impact on intellectual property rights and innovation, and 
preserve data without ridiculous cost or administrative burdens. In an 
attempt to begin addressing this issue, the Office of Science and 
Technology Policy released guidelines on February 22nd of this year 
that recognized the problem of data access. These guidelines, intended 
for federal science agencies, are to be followed when determining a 
policy for public access to scientific data in digital formats. As part 
of this hearing, I look forward to hearing the witness's opinions on 
these federal guidelines.
    Our witnesses today offer input from a variety of scientific 
fields, as this problem is not exclusive to one scientific field, 
community, or discipline. I'd like to thank them for coming and taking 
time to offer their expertise. I'd also like to thank Ranking Member 
Lipinski and everyone else participating in today's hearing.

    Chairman Bucshon. With that, I now recognize the Ranking 
Member, the gentleman from Illinois, Mr. Lipinski, for an 
opening statement.
    Mr. Lipinski. Thank you, Chairman Bucshon. I think this is 
our third hearing in three weeks, and we have another one next 
week that I will now label you the hardest-working Chairman in 
Washington, D.C. So it is good to be at work here and I want to 
thank all the witnesses for being here.
    The United States has for decades represented the world's 
gold standard for scientific integrity. But no one should 
mistake this observation as an argument for complacency. In the 
COMPETES Act of 2007, which we worked on in this Subcommittee, 
then-Subcommittee Chairman Brian Baird included a provision on 
Responsible Conduct of Research that required every institution 
receiving NSF grant funding to provide training on the ethical 
conduct of science to all students and postdocs covered under 
those grants. Today, all U.S. research universities have 
implemented research ethics training for their STEM students 
and trainees, which we all can agree is a good thing.
    The bigger challenge to the progress of science is not 
misconduct, but rather poor methodology and bad statistical 
analysis that take a long time to uncover. Or for that matter, 
discoveries in one field that have broad multidisciplinary 
relevance but take time to be known in other fields. To that 
end, the open sharing of scientific data is good for science 
and it is good for society. We must, of course, respect issues 
of privacy and intellectual property. But the more data are 
open, the faster we will validate new theories and overturn old 
ones, and the more efficiently we will transform new 
discoveries into innovations that will create jobs and make us 
healthier and more prosperous. The movement toward open data is 
not primarily about scientific integrity; it is mostly about 
speeding up the process of scientific discovery and innovation.
    However, there are some big challenges to the widespread 
implementation of open data. Someone must define what exactly 
data sharing is going to mean and how it is going to be done, 
beginning with a standard. The February 22nd OSTP memo, which 
the Chairman mentioned, on increasing access to the results of 
federally funded scientific research, which by the way was also 
a direct response to requirements in the COMPETES Act, takes on 
many of these issues in detail. But specifically, here are some 
questions that we have to consider, and some of these questions 
were questions raised by the Chairman. First, what does it 
entail and how much does it cost for researchers to develop a 
data management plan and to prepare their own data for sharing? 
Do they have adequate assistance from professional information 
managers? Are funding agencies sufficiently aware of the costs 
and skills required for good data management plans, and how 
should they evaluate and budget for data management proposals? 
What are the IT infrastructure needs for data sharing, 
including technical standards, and what, if any, scientific or 
technical barriers exist to developing that infrastructure? 
What are the most important factors to consider in the 
economics of digital data access and preservation? What should 
be the respective roles of science agencies, universities, and 
the private sector in supporting and preserving public 
databases? How can these groups work together to minimize costs 
and maximize benefit to the scientific community? And finally, 
are there any policy or legal barriers for sustainable digital 
access and preservation?
    In light of the majority's suggestion of a possible 
legislative outcome for this hearing, I hope that today's 
dialogue will include a thoughtful discussion of some of these 
practical issues of implementation. I know that all four expert 
witnesses before us have a lot to contribute to this discussion 
and I look forward to learning from them because this is 
certainly something that is important for us to pursue but we 
need to make sure that we are covering all our bases here and 
do this in the right manner.
    With that, I yield back.
    [The prepared statement of Mr. Lipinski follows:]

     Prepared Statement of Ranking Minority Member Daniel Lipinski

    Thank you Chairman Bucshon and thanks to all of the witnesses for 
being here.
    The U.S. has for decades represented the world's gold standard for 
scientific integrity. But no one should mistake this observation as an 
argument for complacency. In the COMPETES Act of 2007, which we worked 
on in this subcommittee, then Subcommittee Chairman Brian Baird 
included a provision on Responsible Conduct of Research that required 
every institution receiving NSF grant funding to provide training on 
the ethical conduct of science to all students and postdocs covered 
under those grants. Today, all U.S. research universities have 
implemented research ethics training for their STEM students and 
trainees.
    The bigger challenge to the progress of science is not misconduct, 
but rather poor methodology and bad statistical analysis that take a 
long time to uncover. Or for that matter, discoveries in one field that 
have broad multidisciplinary relevance but take time to be known in 
other fields. To that end, the open sharing of scientific data is good 
for science and it's good for society. We must, of course, respect 
issues of privacy and intellectual property. But the more data are 
open, the faster we will validate new theories and overturn old ones, 
and the more efficiently we will transform new discoveries into 
innovations that will create jobs and make us healthier and more 
prosperous. The movement toward open data is not primarily about 
scientific integrity, it's mostly about speeding up the process of 
scientific discovery and innovation.
    However, there are some big challenges to the widespread 
implementation of open data. Someone must define what exactly data 
sharing is going to mean and how it is going to be done, beginning with 
a standard. The February 22nd OSTP memo on increasing access to the 
results of federally funded scientific research, which by the way was 
also a direct response to a requirement in COMPETES, takes on many of 
these issues in detail.

Specifically, we must consider such questions as:

      What does it entail and how much does it cost for 
researchers to develop a data management plan and to prepare their own 
data for sharing? Do they have adequate assistance from professional 
information managers?

      Are funding agencies sufficiently aware of the costs and 
skills required for good data management plans, and how should they 
evaluate and budget for data management proposals?

      What are the IT infrastructure needs for data-sharing, 
including technical standards, and what, if any, scientific or 
technical barriers exist to developing that infrastructure?

      What are the most important factors to consider in the 
economics of digital data access and preservation?

      What should be the respective roles of science agencies, 
universities, and the private sector in supporting and preserving 
public databases? How can these groups work together to minimize costs 
and maximize benefit to the scientific community?

      And finally, are there any policy or legal barriers for 
sustainable digital access and preservation?

    In light of the Majority's suggestion of a possible legislative 
outcome for this hearing, I hope that today's dialogue will include a 
thoughtful discussion of some of these practical issues of 
implementation. I know that all four expert witnesses before us have a 
lot to contribute to this discussion and I look forward to learning 
from them.

    With that I yield back.

    Chairman Bucshon. Thank you, Mr. Lipinski.
    If there are Members who wish to submit additional opening 
statements, your statements will be added to the record at this 
point.
    At this time I would like to introduce our witnesses. Our 
first witness is Dr. Bruce Alberts, Editor-in-Chief of Science 
Magazine and Professor Emeritus of Biochemistry and Biophysics 
at the University of California-San Francisco. Welcome. Our 
next witness is Dr. Victoria Stodden, Assistant Professor of 
Statistics at Columbia University. Our third witness is Dr. 
Stanley Young, the Assistant Director of Bioinformatics at the 
National Institutes of Statistical Sciences. That was hard to 
say. Our fourth and final witness today is Mr. Sayeed 
Choudhury, Associate Dean for Research Data Management at Johns 
Hopkins University and Hodson Director of the digital Research 
and Curation Center.
    As our witnesses should know, spoken testimony is limited 
to five minutes each after which Members of the Committee will 
have five minutes each to ask questions.
    I now recognize Dr. Alberts to present his oral testimony.

                TESTIMONY OF DR. BRUCE ALBERTS,

             EDITOR-IN-CHIEF, SCIENCE MAGAZINE AND

       PROFESSOR EMERITUS OF BIOCHEMISTRY AND BIOPHYSICS,

            UNIVERSITY OF CALIFORNIA - SAN FRANCISCO

    Dr. Alberts. It is a pleasure to be here today. I would 
just like to start by emphasizing something that Science 
Magazine covers repeatedly, which is the fact that our strength 
in science and technology in the United States underlies both 
our economic success and our military dominance in the world. 
As you all know, many other nations are increasingly making 
investments in this area, and I find it distressing that 
although this Committee has long supported fundamental, long-
term scientific research, the investment in the United States 
has been stagnant for many years. The investment in this kind 
of research was 1.25 percent of GDP in 1985, has dropped to .87 
percent of GDP in 2013, a big drop, and of course, the current 
sequester will now make our situation even worse. I believe 
that this is dangerous for America's future, for my 
grandchildren's future.
    But this hearing, of course, is to focus on the quality and 
not the quantity of U.S. research. I would like to address 
first the data availability issue, which of course is crucial 
for science. Science builds by one scientist testing and 
building on and maybe refuting the data of other scientists, 
very much a community endeavor. And the privilege of publishing 
in a journal like ours demands data sharing. Otherwise science 
doesn't work.
    So our journal has been working on this. This is a special 
issue we published, 14 long articles about all these issues, 
February 2011, and we are publishing more and more about this. 
It is accompanied by a survey, a useful survey of scientists, 
how they use data and whether they have enough access. And we 
have stressed over and over again that our policy is ``that all 
data necessary to understand, assess and extend the conclusions 
of the manuscript must be available to any reader of science.'' 
In this issue, we announced a new policy. This includes 
computer codes involved in the creation or analysis of data, 
and I am pleased to say that we are getting good compliance 
with those policies.
    Of course, there are problems that remain. You will hear 
about them from the rest of the group here. But one I would 
like to emphasize is guaranteeing funding for the public 
databases, the critical ones, funding long term so that the 
community and journals like ours can rely on them. This is 
really a major issue. In my field, the protein database, for 
example, is absolutely crucial. It has got 100,000 different 
protein coordinates in it. You know, if funding lapses, then we 
lose all this, and these places play major roles in setting 
standards as well.
    And secondly, I would like to emphasize that we need tools 
for interacting with the largest data sets that are now 
increasingly provided as supplemental online information and 
journal publications like ours, so when we demand the data, we 
put the data not in the written paper but most of it in a big 
electronic supplement, and other journals are doing that as 
well, but we need ways to help people analyze that data who are 
not the original authors. And of course, every journal needs to 
stress clear and complete presentation of all the materials and 
methods that were used in the research.
    So the other issue is data reproducibility. Mr. Chairman, 
you quoted from that paper. My conclusion, and talking to 
people at Genentech who would agree with that paper that you 
cited from Bayer Health Care is that the scientific standards 
are lower in some fields of science and others that we need to 
work on setting higher standards.
    In addition, human cells are incredibly complex and it is 
easy to get a result that looks right when it is really wrong, 
and one can easily be fooled. Every scientist must be trained 
to be highly suspicious about his or her own results, and this 
again is a major issue. And finally, I believe we are 
overemphasizing research directly aimed at finding drugs at the 
expense of the high-quality discovery-driven basic research 
that is urgently needed to improve the search for disease 
treatments. We are just mostly stabbing in the dark.
    So my suggestions for improving this situation would demand 
a community effort from scientific journals like ours. We have 
new policies in the last three years that every senior author 
for each part of the results being published must confirm that 
he or she has personally reviewed the original data generated 
by that unit, specifying where exactly those results appear in 
the paper. It used to be that we wanted one author to take 
responsibility. That is totally unreasonable now. Half of our 
papers have authors in different countries. We would have to 
have a set of senior authors. We are developing checklists in 
various fields of science to help journals and scientists. 
There is a biosketch issue. People should not be listing huge 
lists of publications to impress other people who are giving 
them grant funds. They need to focus on their five or ten most 
important contributions, and quality is critical, not quantity. 
And funding agencies have a role to play here as well.
    I just want to emphasize my own role at universities. I am 
still teaching. I am going to be teaching a 2-week minicourse 
on ethics and research standards this May, so I am very much 
involved in these issues. Thank you.
    [The prepared statement of Dr. Alberts follows:]

    [GRAPHIC] [TIFF OMITTED] T9929.005
    
    [GRAPHIC] [TIFF OMITTED] T9929.006
    
    [GRAPHIC] [TIFF OMITTED] T9929.007
    
    [GRAPHIC] [TIFF OMITTED] T9929.008
    
    [GRAPHIC] [TIFF OMITTED] T9929.009
    
    [GRAPHIC] [TIFF OMITTED] T9929.010
    
    [GRAPHIC] [TIFF OMITTED] T9929.011
    
    
    
    Chairman Bucshon. Thank you.
    I now recognize Dr. Stodden for five minutes to present her 
testimony.

               TESTIMONY OF DR. VICTORIA STODDEN,

               ASSISTANT PROFESSOR OF STATISTICS,

                      COLUMBIA UNIVERSITY

    Dr. Stodden. Thank you for the privilege of addressing you, 
and thank you for your very lucid comments. I agree with just 
about everything both of you have said, and I also agree on how 
important this issue is. So I would like to spend my remaining 
time on two aspects. One is, I would like to scope the problem 
for you, and the second is, I would like to scope the action I 
think that is available to you here.
    So the first thing I want to say is that there is not a 
crisis of integrity in terms of scientists and scientists' 
behavior. What has happened in science is that like all sectors 
of the economy, and all across America, we are taking advantage 
of technological revolutions. What we are doing is using far 
more computers, far more data-oriented and data-driven 
research, far more high-powered investigation in all the 
research all across the sciences. This isn't just in the life 
sciences. This is in engineering, this is in English 
departments who are doing word counts in Shakespeare. This is 
something that is really pervasive in the scientific enterprise 
as a whole, and this is something that is having ramifications 
in the way that we disseminate and communicate science. It is 
not a question of personal integrity.
    So what this means is, to scope the issue, I think that we 
need to think about this issue in terms of reproducibility, so 
as Dr. Alberts outlined, open data itself is a very broad 
notion. I think this needs to be scoped to data and software 
required to reproduce published results, and what that means to 
a scientist is clear. There are details, of course, but that is 
something that a scientist can understand. This is something 
that institutions in the scientific enterprise can understand. 
And I reiterate that it is not just about data, it must include 
the codes and the software that take that data to the published 
results so that those results can be validated and verified.
    You mentioned in your opening remarks about statistical 
errors, about other issues. I would like to scope the problem 
to this computational issue, which I believe is reflected in 
the language around this, digital data, and the reason for that 
is clarity. I agree with you that as a statistician, there are 
lots of statistical errors that are in the literature that are 
being worked out. This is in part because doing computational 
work is new to many fields, and I believe the core issue is 
sharing data, sharing code and things like sort of biological 
materials or the mathematics and the statistics, those will 
work out as corollary issues. Right now the issue needs to be 
scoped on data and code that allow those results to be 
understood, validated and reproduced by other members in the 
community.
    So secondly, I would like to talk about the scope of action 
that I think is available and important for you to think about. 
The first thing is, as Dr. Alberts outlined, scientists are 
very interested in these issues of reproducibility. As we know, 
it is a cornerstone. We don't accept scientific findings until 
there is replication, until there is validation by other 
people--at least that is the theory. And in my testimony, I 
included two articles that are in some sense manifestos from 
computational scientists calling for greater reproducibility. 
The reason computational scientists are banding together and 
creating these manifestos is because there is a collective 
action problem. It does take time to make your data available 
and to make your software available. It is easier to hack 
things up on your machine and produce a paper and never really 
look at the code or the data in the sense of sharing it. That 
does take extra time. So what this means is that scientists who 
want to do reproducible research and sharing the code and data 
that replicates their results are at a disadvantage because 
they don't receive credit for this right now. They generally 
receive credit for the publications. So steps like what Science 
Magazine has taken with data-sharing requirements and code-
sharing requirements are extraordinary and laudable and very 
important. This is Science, though, our highest-impact journal, 
and it is much harder for lower-impact journals to demand that 
of the authors. But this is where the federal funding agencies 
come in as another lever that exerts pressure on scientists and 
what they are required to do.
    So in these manifestos that I included in my testimony, you 
will see computational scientist after computational scientist 
calling for help in a broad sense because people who stick 
their nose out get it cut off and we need the federal funding 
agencies to work in an integrated way to help overcome this 
collective action problem.
    Now, how does this happen? This happens through the 
creation of and financial support for repositories that can 
house code and can house data, and this is something that can't 
just happen, I don't believe, from added money on grants, on 
NIH grants and so on, that are supposed to fund these things in 
an ethereal way. I think this is more serious and this is 
something that needs to be directly confronted, more similar to 
a mandate when you take federal funds for your research.
    Now, standards, as Dr. Alberts mentioned, the protein data 
bank and these institutional repositories, other institutional 
repositories are very important for setting standards. They 
come from the community level. I don't believe they come from 
the federal level down. But this needs to be addressed and 
recognized. There is no point in saying we need to have 
reproducibility, we need to share data, we need to share code 
when they don't know where to put it and there aren't ways for 
people to share it and access it and curate it.
    So I will move to questions here.
    [The prepared statement of Dr. Stodden follows:]

    [GRAPHIC] [TIFF OMITTED] T9929.012
    
    [GRAPHIC] [TIFF OMITTED] T9929.013
    
    [GRAPHIC] [TIFF OMITTED] T9929.014
    
    [GRAPHIC] [TIFF OMITTED] T9929.015
    
    [GRAPHIC] [TIFF OMITTED] T9929.016
    
    [GRAPHIC] [TIFF OMITTED] T9929.017
    
    [GRAPHIC] [TIFF OMITTED] T9929.018
    
    [GRAPHIC] [TIFF OMITTED] T9929.019
    
    [GRAPHIC] [TIFF OMITTED] T9929.020
    
    [GRAPHIC] [TIFF OMITTED] T9929.021
    
    [GRAPHIC] [TIFF OMITTED] T9929.022
    
    [GRAPHIC] [TIFF OMITTED] T9929.023
    
    [GRAPHIC] [TIFF OMITTED] T9929.024
    
    [GRAPHIC] [TIFF OMITTED] T9929.025
    
    [GRAPHIC] [TIFF OMITTED] T9929.026
    
    [GRAPHIC] [TIFF OMITTED] T9929.027
    
    [GRAPHIC] [TIFF OMITTED] T9929.028
    
    [GRAPHIC] [TIFF OMITTED] T9929.029
    
    [GRAPHIC] [TIFF OMITTED] T9929.030
    
    [GRAPHIC] [TIFF OMITTED] T9929.031
    
    [GRAPHIC] [TIFF OMITTED] T9929.032
    
    [GRAPHIC] [TIFF OMITTED] T9929.033
    
    [GRAPHIC] [TIFF OMITTED] T9929.034
    
    [GRAPHIC] [TIFF OMITTED] T9929.035
    
    [GRAPHIC] [TIFF OMITTED] T9929.036
    
    [GRAPHIC] [TIFF OMITTED] T9929.037
    
    
    
    Chairman Bucshon. Thank you very much.
    I recognize Dr. Young for five minutes to present his 
testimony.

                TESTIMONY OF DR. STANLEY YOUNG,

             ASSISTANT DIRECTOR FOR BIOINFORMATICS,

          NATIONAL INSTITUTES OF STATISTICAL SCIENCES

    Dr. Young. Thank you for the opportunity of testifying.
    As an abstract principle, the sharing of research data is a 
noble goal and meets with little opposition. However, when data 
sharing is attempted in a particular circumstance, the 
conflicting interests of the parties can thwart the exchange. 
So said Joe Cecil of the Justice Department in 1985.
    What is the current status of science in general and data 
availability in particular? First, where are we with science 
claims? In 2005, John Ioannidis published two papers of 
interest. In one, he asserted that 90 percent of the claims 
made in science papers are wrong in the sense that they are not 
expected to replicate. In another, he noted that five out of 
six papers based on observational studies failed to replicate. 
I published a paper in 2011 and showed that of 52 hypotheses 
suggested from observational studies, none replicated in the 
expected direction and five were statistically significant, but 
in the opposite direction. Begley and Ellis reported that 47 
out of 53 claims made in major science journals failed to 
usefully replicate.
    Where are we on data sharing? John Ioannidis selected 10 
papers from each of 50 of the highest-impact journals--New 
England Journal of Medicine, Nature, Science, et cetera--and 
asked, is the data used in these papers publicly available? 
Overall, only 47 of 500 papers deposited full primary raw data 
online. None of the 149 papers not subjected to data 
availability policies made their full primary data publicly 
available.
    I report on two personal experiences. Dr. Beate Ritz of 
UCLA made a claim in Environmental Health Perspectives that air 
pollution in L.A. county leads to low birth weights. Dr. 
Frederica Perera of Columbia University asserted in the journal 
Pediatrics that air pollution decreased IQ in children. NIEHS 
provided funding for both studies. In both cases, I asked for 
the data sets from the authors. I also asked for help from 
NIEHS. I resorted to FOI. I received neither data set. 
Recently, I was informed that NIEHS does not have the legal 
authority to compel and an author to proved data that was 
funded by them. Operationally, NIH funding, the Shelby 
amendment, etc. mean very little with respect to data 
availability. Mostly, authors do not provide data sets used in 
their publications. It is technically easy to share data used 
in publications. Others will discuss reproducible Research, so 
I will leave that aside.
    Just why are we in this situation, where most claims do not 
replicate and authors will not make data sets available? In a 
long and illustrious career, Edwards Deming made the point that 
if a system is failing, it is not the workers' fault--that is 
the scientist--it is the fault with management, in this case 
funding agencies and journal editors. For over 30 years, 
workers have been admonished to do their work better and to 
make their data sets available. It was reported in Science in 
1988 that there were serious problems with observational 
studies. Nothing has changed in 25 years.
    Congress, funding agencies and journal editors need to step 
up and manage the scientific process. They should require 
authors to deposit data sets on publication of their papers. 
Funding of data set construction and analysis should be 
separate. They should require data analysis strategies that 
demonstrate reproducibility. For example, any claim should be 
replicated in a separate data set before publication. Remember, 
the reliability of current scientific claims is only 10 to 20 
percent. John Holdren's thing on the Office of Science and 
Technology Policy I think is a welcomed thing in this area.
    It is not enough to agree with sharing data. It is almost 
30 years since Joe Cecil stated the problem. Management should 
make the depositing of data sets on publication mandatory. This 
is a management problem; it is not a science worker problem.
    Thank you very much.
    [The prepared statement of Dr. Young follows:]

    [GRAPHIC] [TIFF OMITTED] T9929.038
    
    [GRAPHIC] [TIFF OMITTED] T9929.039
    
    
    
    Chairman Bucshon. Thank you.
    I now recognize Mr. Choudhury to present his testimony, 
five minutes.

               TESTIMONY OF MR. SAYEED CHOUDHURY,

          ASSOCIATE DEAN FOR RESEARCH DATA MANAGEMENT

        AT JOHNS HOPKINS UNIVERSITY AND HODSON DIRECTOR

          OF THE DIGITAL RESEARCH AND CURATION CENTER

    Mr. Choudhury. Chairman Bucshon, Ranking Member Lipinski, 
Members of the Subcommittee, thank you for the opportunity to 
be here today.
    I have been asked to address questions related to data 
sharing, access and preservation. I would like to do so from 
the perspective of infrastructure development. The other 
witnesses have already addressed the importance of persistent 
scientific data archives for reproducibility. I believe that 
strategic investments in data infrastructure also have 
important implications for our overall competitiveness.
    There are important lessons from our historical 
infrastructure development that are relevant as we consider 
data sharing, access and preservation. The development of 
railroads initially led to systems that served regional 
networks but eventually merged into a national network through 
a standard track gauge. With the development of automobiles, we 
adapted from early mistakes to adjust drivers' behavior through 
education, driving rules and seat belts. The development of the 
Internet reflects a layered approach of different technologies 
connected through a key component in the form of two protocols 
known as TCP and IP.
    Broadly speaking, successful infrastructure development has 
relied on a flexible balance of community and national 
approaches, social aspects relating to human behavior, and key 
components. In each case, as infrastructure evolved through 
community efforts, we reached the point where national 
coordination moved us to a more cohesive situation. In previous 
cases, the more cohesive infrastructure led to greater societal 
benefits from both the private and public sector. I believe we 
have reached a similar point with certain aspects of data 
infrastructure.
    From a policy perspective, the recent Executive Memorandum 
from the Office of Science and Technology Policy provides a 
useful framework for federal policies that would maximize data 
sharing, access and preservation. The memorandum acknowledges 
the need for flexibility by federal agencies for the 
communities they support balanced with the need for uniform 
guidelines when appropriate. There is one specific example that 
I will mention in my oral remarks. The memorandum outlines the 
need for appropriate data attribution and citation. The method 
for meeting this need is the persistent identifier, which is a 
long-lasting reference to data. You can think of persistent 
identifiers as an improved version of Web site addresses such 
as Congress.gov. It is a rough analogy, but the persistent 
identifier may be compared to having the same role as track 
gauge in the development of railroads.
    From an economics perspective, there is a greater need for 
understanding of costs. For example, some cost studies focus 
only on storage, ignoring related costs such as data center 
operations or longer-term costs related to preservation. 
Preservation of data ensures that we can extract value for the 
long term, noting that with data, preservation issues can arise 
in as little as five years. The development of data 
preservation infrastructure represents a case where effective 
partnerships could be formed between the public sector, private 
sector and university sector, in which I include libraries and 
national laboratories. It is possible that the private sector 
will not focus on data preservation because there are 
unresolved research problems, it is unlikely to be profitable, 
and it benefits from large-scale coordination. Federal agencies 
could provide the funding for research, prototypes and initial 
deployment of data preservation infrastructure. The university 
sector could then set up production systems that the scientific 
community and private sector could exploit for discovery and 
profit.
    From a technology perspective, it is important to remember 
that there are different types of data and different stages of 
scientific projects. Consequently, there is a need for a 
layered approach to diverse systems spanning individual 
researchers to large-scale national projects. Even with this in 
mind, it is possible to identify gaps that are common across 
this landscape. For example, today's storage systems work well 
for many purposes but they do not currently meet some 
preservation requirements. It is worth mentioning that some 
storage companies view this situation as an opportunity for 
code development with the university sector.
    From a non-technical perspective, scientists do their best 
to manage their data but they do not always have a full 
understanding. Raising awareness and reinforcing the importance 
of data sharing, access and preservation will be important. 
This type of awareness building and education is similar to the 
adjustment of automobile drivers' behaviors over time.
    In conclusion, I believe that we have an important 
opportunity to advance our data networks into more cohesive, 
large-scale infrastructure that will advance the scientific 
process and generate benefits for the public sector, industry 
and the scientific community.
    I thank you again for the opportunity to be here, and I 
look forward to answering your questions.
    [The prepared statement of Mr. Choudhury follows:]

    [GRAPHIC] [TIFF OMITTED] T9929.040
    
    [GRAPHIC] [TIFF OMITTED] T9929.041
    
    [GRAPHIC] [TIFF OMITTED] T9929.042
    
    [GRAPHIC] [TIFF OMITTED] T9929.043
    
    [GRAPHIC] [TIFF OMITTED] T9929.044
    
    [GRAPHIC] [TIFF OMITTED] T9929.045
    
    
    
    Chairman Bucshon. Thank you very much, and I thank all the 
witnesses for their testimony, reminding Members that the 
Committee rules limit questioning to five minutes. The Chair 
will at this point open the round of questions. The Chair 
recognizes himself for five minutes.
    As a cardiothoracic surgeon, I am very interested in this 
issue because I have to translate what is written into clinical 
practice, and so this type of issue really does affect real 
people. I can tell you the difficulty that people like me have 
in figuring out when to change your clinical practice, when you 
are doing something that turns out wasn't the right thing to 
do, it is a very difficult process that is ongoing, so I am 
very interested in this particular subject.
    I will start with Dr. Young. Could you give me some 
examples of where State and federal regulations were made 
without public release of data used to make those regulations?
    Dr. Young. Yes. I have taken an interest pro bono in air 
pollution questions, and an expert in the area worked with me 
and we developed 100 papers that are key papers in that area. 
Then being a statistician, I selected 50 of those papers at 
random and asked the authors for the data sets. I received no 
data sets at all. Many of these data sets were funded by the 
Federal Government and there are many regulations that are 
based on these data sets. They are key data sets. For the most 
part, these data sets are not available.
    Chairman Bucshon. Just so you know, I had the same problem 
getting the data out of the Federal Government. It can be an 
issue.
    Mr. Choudhury, could you give me what specific 
infrastructure technology requirements are required for the 
storage of scientific data research?
    Mr. Choudhury. There are several layers that are necessary 
to actually preserve scientific data. It begins with storage, 
which is basically just the bits residing on a hard disc or a 
tape or even in the cloud, but eventually we also need to do 
things to ensure data protection. We also need to have to then 
do things to ensure that we can migrate the data over time, so 
as we start to use new storage systems or if we have new file 
formats, we have to be able to move those data into those new 
environments. As Dr. Stodden mentioned, we also need to have 
access to the software or the tools that process the data 
because in many cases, it is not sufficient just to get access 
to the data alone. So the actual preservation of the data is 
this complex set of layers that go beyond storage. Storage is 
necessary but it isn't sufficient. So we have to do all these 
other things to understand the context and the reusability of 
the data as well.
    Chairman Bucshon. Do you think currently that university 
libraries or national laboratories are equipped for this type 
of infrastructure?
    Mr. Choudhury. At Johns Hopkins, we have taken an approach 
of looking at two stages. The first is prior to investigators 
submitting proposals--they need some sort of consultation and 
support to develop their data management plans. In this 
respect, I do believe that the university sector, and 
particularly university libraries, have stepped up very well. I 
think most research university libraries are providing that 
kind of consultation to their investigators.
    The second stage is that once an award is made, then we 
actually have to handle the data and we actually have to start 
preserving it for the long term. In this respect, there is a 
subset of that library community that has come forward to help 
provide that kind of support, and then there is the long-term 
preservation need, and even there, it is a smaller subset 
again. It is in the preservation of the data where I think 
there remains some research questions which ultimately when 
they are addressed they can migrate the support into the 
university library sector.
    Chairman Bucshon. Great. Dr. Alberts, on February 11, 2011, 
in a Science magazine editorial, you write, ``We will ask 
authors to provide a specific statement regarding the 
availability and curation of data as part of their 
acknowledgments requesting that reviewers consider this as a 
responsibility of the authors.'' Do you think this self-
policing policy works in practice?
    Dr. Alberts. We find that it has been working for Science 
magazine. Our senior author, deputy editor, Brooks Hanson, has 
been deeply involved in this. On rare occasions we have had to 
make authors do things that they should have done themselves 
but I guess we are fortunate we have the threat, which is, we 
are not going to publish any more papers from you, and they 
want to publish in Science magazine, and as Victoria said, not 
every journal can make that threat. So I think this is a very 
important issue to emphasize. We haven't talked about the 
fact--I am a biochemist, and I had lots of data from my 
laboratory when I was an active scientist. Not all of it should 
be preserved. I mean, if I tried to preserve everything, I 
couldn't find anything. So we also need different fields to 
decide what it is that we really need to preserve and make 
available. There is so much material being collected now that 
it is really important to get standards for different fields of 
what needs to be preserved and what needs to be put in your 
publication.
    Chairman Bucshon. Great. Thank you all. I now yield to Mr. 
Lipinski from Illinois.
    Mr. Lipinski. Thank you. I wanted to start out by saying I 
am sort of going back to my days as a social scientist and 
thinking about not just the research I did and the data that I 
had but also thinking about behavior, and it is--there are not 
rewards generally for having--someone had mentioned, I think 
Dr. Stodden, that you are rewarded for a result in a 
publication but you are not rewarded--the rewards aren't there 
to spend the time and the effort to have the data in a format 
even that is accessible to others, and if you are talking about 
going further than that, how exactly you went through and you 
analyzed the data. I can't tell you how much paper I had 
printed out of different ways, all these different models that 
I ran and trying to keep track of all that. So it is not simple 
to do and there has to be incentives. So somehow the culture 
has to be changed. And the question is, how do we change that 
culture? Now, the National Science Foundation requires that you 
have a data management plan when you are applying for a grant, 
so the NSF puts that in there.
    My question is, in a short period of time if you can do it, 
how do we change this, and should this be a situation where it 
is data available upon request or should it all be available? 
Should it be put out there published somewhere or put on a site 
that everyone can access? And how far do we go with the data? 
Is it, okay, this is how I analyze it, this is the statistical 
package I used, this is how exactly I did it. So let me start 
with Dr. Stodden. I mean, what is your quick sort of suggestion 
on it for your 30,000 foot? What would you do if you could?
    Dr. Stodden. So I think the efforts that have been taken so 
far are really this on request and so on, and there are a 
number of experiments and studies, and Dr. Young mentioned a 
couple, where that doesn't seem to work as well. You don't 
simply get the response. So I think it is time to move forward 
to this being a standard. Now, having said, as Dr. Alberts 
said, there are data sets and problems of different importance, 
and you can imagine investing a lot more time curating a data 
set that has broad use and applicability and might underlie 50 
or 100 studies and so on versus one one-off. But the changes 
really something that I believe scientists are willing to do 
and are working on standards. For example, in economics this is 
a very forward-thinking community and many of the journals have 
standards and they do engage in data sharing and code sharing 
but not even as much as they would like. And so I think the 
complexity of the problem means that it really is not a one-
size-fits-all solution. As you mentioned, it is something that 
comes from the field.
    But I would suggest that this is a standard that it should 
be understood that this code and the data go open for 
reproducibility and changing the culture is something 
scientists are talking about. There is a special issue I can 
point you to in Computing and Science in Engineering that is 
called Changing the Culture, and it is about giving these 
rewards. So as Dr. Choudhury mentioned, having these persistent 
identifiers allows citation for data and for code NSF steps 
towards allowing scholarly objects like data and code listed on 
the biosketch and not just publication is a real step in this 
direction, and I think the scientific community will sort out 
how it values data contribution and code contribution and 
publication contribution. They may not be all valued equally 
but we have a long history of doing this. Not all publications 
are valued equally. But I think that bringing this through 
citation and having citation standards is a way to really 
change the culture and reward people.
    And I will add one last point, which is there is a 
generational difference here because these changes in 
technology, young people and young scientists and people who 
want to go into research, it is very natural for them to share 
data and to share code, and it is discouraging for them to 
enter a situation where suddenly this is not the norm. So this 
is something where I think there is also this opportunity that 
the culture is changing naturally on its own just with time as 
younger people come in and have these expectations for sharing 
what they are doing digitally. And so that is also something to 
capitalize on. And again, I go back to the testimony in that 
there is this collective action problem because, as you 
mentioned, it takes time, and so something particularly from 
federal agencies that can help push through that is really very 
important.
    Mr. Lipinski. I thank you. My time is up. I yield back.
    Chairman Bucshon. I now yield to Mr. Stockman for five 
minutes.
    Mr. Stockman. I have a question for Dr. Alberts. My wife is 
a NASA privacy officer, and I want to follow up on something 
the Chairman related. In February in your editorial, you wrote, 
``We recognize that exceptions may be needed to these general 
requirements for sharing data, for example, preserve the 
privacy of individuals or in some cases when data materials are 
obtained from third parties and for security reasons but we 
accept those rare exceptions.'' Is this your view today?
    Dr. Alberts. For example, we had an experience with a 
Department of Energy lab where they weren't allowed to give us 
the code because presumably it had some security implications. 
So we do encounter those one-off occasions. But they have been 
rare. So we have to live with the law, and we try our best to 
do what we can.
    Mr. Stockman. Do you see other exceptions?
    Dr. Alberts. Not that--I don't know of any exceptions since 
that policy was made.
    Mr. Stockman. Okay. The other question I have is for all 
the witnesses. Many of you today also practice science. You are 
also members of the United States scientific community. You 
have been a world leader in producing first-class research. How 
do you envision the mechanism of enforcing the sharing of data 
without hindering the process of scientific discovery and 
simultaneously minimizing the administrative burden of a 
scientist? Because I know a lot of professors and everything a 
lot of time fill our more paperwork than they do research. If 
you could each just go quickly through the----
    Dr. Alberts. Well, I think Victoria said it right. We need 
to mobilize our communities. I mean, I am a cell biologist and 
the American Society of Cell Biology used to help us. What does 
it mean for our community, and we have to take responsibility 
for it, and it is going to be different for statisticians. 
Different people will have different requirements and it has to 
make sense, and I agree with you that it has gone way overboard 
now at universities. Every time I want to do anything, I have 
to fill out a form. So I think we should try to avoid 
legislating more flat requirements. You know, if I want to 
interview students, graduate students at UCSF about their 
career options, I have to fill out a 50-page human youth form. 
It drives me nuts. So this Committee might work on pushing back 
on some of the meaningless paper and get some requirements that 
are more meaningful.
    Dr. Stodden. That is a great question, and I think it goes 
back to these issues of reproducibility. If you are publishing 
a paper where you claim that data and code are out there and 
available for it to be reproducible, then that is in a sense 
the starting point of standards in a community. Now, as Dr. 
Alberts mentioned, this will change for different communities 
and different research problems and they can be quite 
different, but there needs to be this expectation that the 
results, the computational results will be reproducible and 
then when you go and get your hands dirty and you try and do 
the reproducibility, then if it doesn't work or it does work, 
then that is value too in the community, and I think that 
scaffolding and that framework is really there. It is a 
question of moving towards this default of openness rather than 
the default of being closed and then you request and so on, and 
as I was mentioning to Ranking Member Lipinski, the default 
needs to be open, and then as you mentioned, we have exceptions 
for confidentiality and so on but those are the exceptions, and 
then the standard is really about reproducibility.
    Dr. Young. The first thing to keep in mind is that many 
estimates say that 80 to 90 percent of the claims that appear 
in scientific papers are wrong in the sense that they will not 
replicate. So I would focus on cost per valid result. 
Additional costs can be put into reproducible research and 
things like that. The total number of claims that are checked 
will go down but the number of valid claims can easily go up if 
we do our research better. Thank you.
    Mr. Choudhury. I think one thing that is becoming clear is 
data management is a complex and demanding set of activities on 
its own. It may not be reasonable to expect scientists to 
conduct their own data management but rather work with a set of 
professionals who sit somewhere between the domain sciences, 
say, library information science. So I think there is a 
workforce development issue here. We don't expect scientists to 
be experts in IT systems or other kinds of systems. We provide 
support for them, and I think data management may be in that 
category.
    Mr. Stockman. Thank you. I yield back.
    Chairman Bucshon. I now recognize Mr. Bera from California.
    Mr. Bera. Thank you, Mr. Chairman.
    Now, to start off with, I would want to make sure we don't 
give the impression that our scientific community and our 
research institutions are producing faulty data. We maintain a 
competitive advantage. As a scientist myself, as someone who 
spent countless hours in the lab as a medical student and has 
spent time as a faculty member and associate dean at the 
University of California-Davis, working with our medical 
students and our resident physicians, we maintain a competitive 
superiority in our research institutions, and I think Dr. 
Alberts touched on the importance of the federal investment in 
our research institutions. We also need to recognize our 
journals and particularly our leading peer review journals. 
There is a rigorous process having again submitted articles and 
worked with countless students that you go through as you are 
submitting articles. Replicability is an important component 
but also putting the information out there so others can look 
at it and provide feedback is very important. So we want to be 
conscious of that as well.
    As we set up our research institutions, we often are doing 
it and our trials are in a very transparent way, you know, 
funding multi-center trials. When we look at major projects 
like the Human Genome Project, as we talk about brain mapping, 
we will set that up in as transparent a way as possible using 
multiple of our institutions. And it isn't always just about 
replicability. It is about sharing that data and working 
together, but at the same time--and my question is this--as we 
move into this era of wanting to share data, we also have to 
maintain our competitive advantage. We do have competitor 
nations that every day are trying to get to our data and get to 
the research institutions. We talk about cybersecurity on this 
Committee. We need to be very conscious of what we are putting 
out there as well.
    I would direct a question to Dr. Alberts. You talked about 
the importance of research funding as well as the threats to 
research funding in our academic institutions. Why don't you 
touch on that, and then if the rest of the panel wants to talk 
about how we move forward in kind of an open, transparent way 
but maintaining our competitive advantage and protecting those 
discoveries that we are making.
    Dr. Alberts. As I wrote in my written testimony, I referred 
to this major project from the National Academy of Sciences 
when I was president to explain to Congress and the public how 
fundamental knowledge produces breakthroughs. The first 
pamphlet we produced was on the global positioning system. 
Somewhere started with the fact that physicists invented atomic 
clocks. They won a Nobel Prize but everybody thought it was 
useless because it enabled us to keep time to a billionth of a 
second, and why should we want to do that. Well, you follow 
this progression, and I recommend that whole series. It is 
still up on the Web. That combined with many other findings of 
knowledge about the world enabled us to put up these 24 
satellites that produce this wonderful device that we all use 
and the military uses, and we did that over and over.
    And what has been true in the United States, remarkably, 
and I don't think people recognize this, we have been a magnet 
for the most talented people from all around the world coming 
here, and you just look at Silicon Valley and places like that. 
So if we don't keep our leading position as scientific 
research, a place to come to, our universities, then those 
people won't come here and they won't subsequently contribute 
their genius to the American economy and the American strength 
of our Nation. So I am quite worried right now because many 
other countries, China, for one, they see this very clearly. 
This is where we have our competitive advantage and they are 
trying to gain it, and if we don't pay attention to that, I 
think we are going to lose this game. We are taking it for 
granted that all these great people are going to come to this 
country but they are not going to do that anymore if we are not 
the best place to do research.
    Dr. Stodden. So I couldn't agree with your comments more, 
and also with Dr. Alberts that American science is absolutely 
superb, and as evidence of this, I believe our discussion today 
actually reflects the high integrity and the honesty of that 
community in trying to grapple with these problems. I mean, 
these manifestos and so on I put in the testimony here, these 
are scientists who are concerned about the quality of the 
science and trying to fix it. This is not anything other than 
the highest-integrity profession.
    I also want to make one quick comment about corollary 
benefits of open data, going back to your earlier point, which 
is, you probably gathered by now that I think reproducibility 
is important but there are also issues in terms of access to 
the technology. So if you have the ability, the software tools 
and the data to replicate those results and those findings, not 
only can you therefore build on them more easily as well as 
validating them but it also opens them to industry and to 
others who can then capitalize on this for commercial use. I 
mean, whatever they see as appropriate. So it opens all of 
these avenues towards economic growth that can't be overlooked 
that are extremely important.
    And to your point about, well, what if open data helps our 
competitors, I think that there is a long history in the United 
States of being able to capitalize on this and move ahead, and 
I don't think that maintaining a closure around our scientific 
enterprise does anything but restrict American enterprise and 
competitiveness internationally and also threaten the integrity 
of our results. I mean, science moves forward, as Dr. Alberts 
mentioned, through skepticism and through questioning and 
through transparency and openness, and being able to share 
those methods and giving others the tools to replicate and also 
build on, commercialize, capitalize on all of this, I think is 
an avenue towards economic growth and an avenue towards STEM 
understanding too. When it is open, you can imagine smart high 
school kids getting their hands on this stuff and figuring 
things out and playing with it, and that is very real.
    Chairman Bucshon. Thank you. I now yield to Ms. Lummis five 
minutes.
    Ms. Lummis. Thank you, Mr. Chairman.
    Now, my first question is for any of you who cares to 
answer. It is about OSTP guidance. My question is, do you think 
that the guidances provides appropriate flexibility to agencies 
in developing plans to improve access to federally funded 
research?
    Dr. Young. Stan Young. I read the guidelines very 
carefully. I think they are a major advance forward. The 
history is that if scientists are not compelled to make their 
data sets available, they generally don't make it available. 
The American Psychological Association, for example, just 
started a huge effort on reproducibility. Their journals, there 
are 50 of them, have the author sign a paper saying I will make 
my data set available. Studies have shown that two-thirds of 
the authors that have signed those statements do not make their 
data sets available, so I think there is--some scientists are 
great. In general, there is no data sharing.
    Mr. Choudhury. I do think the memorandum provides a good 
deal of flexibility for federal agencies and the communities 
they support. I do think it is also important to think about 
those opportunities where something may be uniform across 
different agencies. Another example that I would give is the 
memo talks very clearly about enforcing data management plans. 
Well, most reviewers in these early days don't even know what 
constitutes a good data management plan, so I think providing 
guidelines to reviewers about what constitutes a rigorous data 
management plan would be a very important thing that any 
federal agency could do, and it would, of course, be customized 
to their communities.
    Ms. Lummis. Well, I had an experience like you have 
mentioned with the greater Yellowstone interagency brucellosis 
committee where we trying to get data on elk and the 
transmission of brucellosis from elk to bison, bison to 
domestic livestock, and it was tremendously important because 
we finally have that disease pretty well isolated to the 
greater Yellowstone area after trying for, what, almost 100 
years now to isolate it because it does--it used to be 
prevalent in milk cows, but after years of destroying entire 
herds of dairy cattle, we finally have that disease isolated to 
the greater Yellowstone area. But it is raising havoc, and 
there was a woman who was an employee of Yellowstone National 
Park who gave her entire career paid by the taxpayers to 
studying elk and she would not share her data with us. I mean, 
she was taxpayer funded. So I have had personal experience with 
your frustrations here.
    Another question. Could you comment on the difference 
between what has been written in statute versus what is 
happening in practice regarding obtaining data in federally 
funded research, you know, any of you in your experience?
    Dr. Young. I have a lot of experience asking for data sets, 
and I will call out the country of Finland. Every time I ask a 
scientist in Finland to send me a data set, I get it in return 
email. Given the electronic age that we are in, it is 
reasonably easy to pass data sets around. My experience in the 
United States is not nearly so good. I mentioned requests for 
50 data sets in the area of air pollution, and I got none. The 
psychologists know very well that data sharing, even though it 
is compelled by their journals, it is not done there. There is 
a huge difference between what beautiful-thinking people say 
about sharing data, and then Joe Cecil is right. In practice, 
quite often it is to the advantage of the person that holds the 
data not to share it, and so there is a real problem and a 
difference. NIEHS or NIH, for example, has a wonderful data-
sharing policy. However, they have no legal authority to compel 
anyone to share data, and so many times I have gone all the way 
up through very high levels of the NIH asking for data sets and 
have not gotten them. So the practice is very different from 
the publicity.
    Dr. Stodden. I would like to just reiterate Stan's point 
there. Both NIH and NSF grant guidelines require data sharing, 
and even encourage software sharing, and these have been around 
for at least a decade, and it seems to be unenforceable. And so 
when the Executive Memorandum talked about mechanisms for 
enforceability, I found that very exciting because, like Stan 
says, things can be on paper and then without that enforcement, 
then things don't proceed, and that, I think, is a real bridge 
to breaking the collective action problem and providing those 
incentives for sharing and rewarding scientists to do this.
    Ms. Lummis. Thank you, panel. My time is up, so I will 
yield back to the Chairman.
    Chairman Bucshon. Thank you. I now yield to Mr. Palazzo for 
five minutes.
    Mr. Palazzo. Thank you, Mr. Chairman.
    Dr. Stodden, allowing open access to federally funded 
scientific data may also create new business opportunities. 
What are your thoughts on this issue?
    Dr. Stodden. I think the evidence is clear, and one of the 
reasons that scientific research is funded by the Federal 
Government is because we can discover scientific facts and 
inventions and so on that then can, among other things, 
undergird economic growth through these creations of 
opportunity for industry. So something like economic open data 
and open methods that allow reproduction of these discoveries, 
I don't think it can help but fuel economic growth in the sense 
that you can take these discoveries--scientists don't develop 
things for market. They don't do commercialization or full 
development, particularly not of software and so on. And then 
it is perfectly plausible that these can be taken out and 
developed into products and taken to market if that is viable, 
and I think that that is something that is a very compelling 
reason behind open data and open code.
    Mr. Palazzo. Do you have any examples of products and 
services that companies may be able to offer?
    Dr. Stodden. So, for example, some of my background is in 
image processing and working on standards like the JPEG 2000 
standard. So this came out of academic research on how to do 
image compression and then that is released openly with open 
code, and that is something that can be implemented and become 
standard in the Web for faster loading of Facebook or whatever 
it is or Flickr or whatnot, and it is these types of things 
that are done in the scientific labs and then sometimes, as Dr. 
Alberts said, you don't even see the end application. You are 
making these discoveries and then it takes ingenuity and 
industry to then turn it into different other applications, but 
this happens absolutely all the time.
    Mr. Palazzo. And I think you mentioned this in your 
testimony, that it is definitely a potential economic growth 
area for our country?
    Dr. Stodden. Absolutely.
    Mr. Palazzo. Now, on the flip side, allowing open access to 
federally funded scientific research and the impact, or what 
would be the impact on the intellectual property rights, which 
innovation and U.S. competitiveness and things of that nature?
    Dr. Stodden. That is a great question, and it has, 
unfortunately, a complex answer that I tried to touch on in my 
testimony. The intellectual property structure that affects 
scientists was not designed for science, and there is two 
principal ways that it touches scientific output, and one is 
copyright and the other is patents, and copyright is something 
that works against--in the scientific context that works 
against openness in the sense that a scientist who produces 
code or produces other copyrighted outputs like a paper, I 
actually would need to give you explicit permission to do this. 
The default is not openness. So this is something I mentioned 
in my testimony, that maybe this is something that we need to 
rethink how the intellectual property system interacts with 
scientists who have completely different normative structure to 
say, for example, a poet or someone creating a movie or 
something like this, it is a very different model.
    The other way that it interacts is through patents, and 
this is largely around inventions, not touching so much the 
computational work that we have been discussing today but 
software is patentable, and I can imagine--and this is actually 
increasing now, that patentable code is something that is 
coming out of the academic institution. So I think this is 
something that we need to think about very carefully. If you 
think back to 1980 and Bayh-Dole, this was something that was 
put into place to encourage transparency, the idea being that 
giving these intellectual property rights to institutions would 
then allow them to patent and give them this incentive, a 
financial incentive, to be open. Now if we have standards of 
reproducibility where code is open and data is open, it doesn't 
make sense to have that same incentive to patent because it 
actually becomes more of a barrier because in 1980, no one 
imagined you would just go to a repository or get hub or 
whatnot and click and get the code. It had to be this whole 
thing through a tech transfer and so on, which is completely 
different and now that is the barrier. So I think there is some 
careful thinking that needs to happen in terms of IP and also 
around how we collaborate with industry too. Industry has very 
fruitful collaborations with academia, and those need to be 
worked out in terms of what intellectual property remains over 
the scientific output so that industry has--essentially they 
can sort of get some return on their investment.
    Mr. Palazzo. I yield back, Mr. Chairman.
    Chairman Bucshon. Thank you very much. I would like to 
thank all the witnesses for their valuable very interesting 
testimony and the Members for their questions. The Members of 
the Committee may have additional questions for you, and they 
we will ask you to respond to those in writing. The record will 
remain open for two weeks for additional comments and written 
questions from Members.
    The witnesses are excused and the hearing is adjourned. 
Thank you, everyone.
    [Whereupon, at 11:06 a.m., the Subcommittee was adjourned.]


                               Appendix I

                              ----------                              


                   Answers to Post-Hearing Questions


Responses by Dr. Bruce Alberts



[GRAPHIC] [TIFF OMITTED] T9929.047

[GRAPHIC] [TIFF OMITTED] T9929.048

[GRAPHIC] [TIFF OMITTED] T9929.049

[GRAPHIC] [TIFF OMITTED] T9929.050

[GRAPHIC] [TIFF OMITTED] T9929.051

Responses by Dr. Victoria Stodden

[GRAPHIC] [TIFF OMITTED] T9929.052

[GRAPHIC] [TIFF OMITTED] T9929.053

[GRAPHIC] [TIFF OMITTED] T9929.054

[GRAPHIC] [TIFF OMITTED] T9929.055

[GRAPHIC] [TIFF OMITTED] T9929.056

[GRAPHIC] [TIFF OMITTED] T9929.057

Responses by Dr. Stanley Young

[GRAPHIC] [TIFF OMITTED] T9929.058

[GRAPHIC] [TIFF OMITTED] T9929.059

[GRAPHIC] [TIFF OMITTED] T9929.060

[GRAPHIC] [TIFF OMITTED] T9929.061

[GRAPHIC] [TIFF OMITTED] T9929.062

[GRAPHIC] [TIFF OMITTED] T9929.063

Responses by Mr. Sayeed Choudhury

[GRAPHIC] [TIFF OMITTED] T9929.064

[GRAPHIC] [TIFF OMITTED] T9929.065

[GRAPHIC] [TIFF OMITTED] T9929.066

[GRAPHIC] [TIFF OMITTED] T9929.067

[GRAPHIC] [TIFF OMITTED] T9929.068

[GRAPHIC] [TIFF OMITTED] T9929.069

[GRAPHIC] [TIFF OMITTED] T9929.070

[GRAPHIC] [TIFF OMITTED] T9929.071

[GRAPHIC] [TIFF OMITTED] T9929.072

[GRAPHIC] [TIFF OMITTED] T9929.073

                                 
