[Federal Register Volume 81, Number 244 (Tuesday, December 20, 2016)]
[Notices]
[Pages 92848-92851]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2016-30593]
-----------------------------------------------------------------------
DEPARTMENT OF THE INTERIOR
Bureau of Reclamation
[RR08100000, 17XR0680A1, RY.1541CH20.WA01701]
Announcement of Requirements and Registration for a Prize
Competition Titled: Sub-Seasonal Climate Forecast Rodeo
AGENCY: Bureau of Reclamation, Interior.
ACTION: Notice.
-----------------------------------------------------------------------
SUMMARY: The Bureau of Reclamation is announcing the following prize
competition, Sub-Seasonal Climate Forecast Rodeo. This Challenge seeks
to improve on existing sub-seasonal forecasts and asks Solvers (i.e.,
competitors in the Challenge) to develop systems that perform
demonstratively better than existing baseline forecasts for temperature
and precipitation over a 15-42-day time frame. Solvers will have
approximately 3 months to develop their system, at which point they are
asked to provide forecasts every 2 weeks over a 13-month period, with
the first month being a ``pre-season'' to become familiar with the
submission and evaluation processes.
DATES: Listed below are the specific dates pertaining to this prize
competition. Please note that times in meteorology are based upon a
world-wide 24-hour clock called Zulu time (Z). Additional detail on
Zulu time is available in the SUPPLEMENTARY INFORMATION section of this
notice.
1. First forecast due on or before 0Z (Zulu) March 21, 2017 (pre-
season), and 0Z April 18, 2017 (actual competition).
2. Final forecast and hind-cast due on or before 0Z April 3, 2018.
3. Final submission due on or before 11:59 p.m. (U.S. Eastern Time)
May 3, 2018.
4. Judging period ends on August 3, 2018.
5. Winners announced on or before September 5, 2018.
ADDRESSES: The Sub-Seasonal Climate Forecast Rodeo Prize Competition
will be posted on the following crowd-sourcing platforms where Solvers
can register for this prize competition:
1. The Water Pavilion located at the InnoCentive Challenge Center:
www.innocentive.com/water-pavilion/.
2. U.S. Federal Government Challenge Platform: www.Challenge.gov.
InnoCentive, Inc. is administering this challenge under a challenge
support services contract with the Bureau of Reclamation. Challenge.gov
will re-direct the Solver community to the InnoCentive Challenge Center
as the administrator for this prize competition. Additional details for
this prize competition, including background information, templates,
and the Challenge Agreement specific for this prize competition, can be
accessed through either of these prize competition web addresses. The
Challenge Agreement contains more details of the prize competition
rules and terms that Solvers must agree with to be eligible to compete.
Information pertaining to this competition will be posted to the
Bureau of Reclamation's current prize competitions Web page at
www.usbr.gov/research/challenges/current/.
FOR FURTHER INFORMATION CONTACT: Challenge Manager: Dr. David Raff,
Science Advisor, Bureau of Reclamation, (202) 440-1284, [email protected];
Ken Nowak (303) 445-2197, [email protected].
SUPPLEMENTARY INFORMATION: The Bureau of Reclamation (Reclamation) is
announcing the following prize competition in compliance with 15 U.S.C.
3719, Prize Competitions. The intent is to spur innovation toward
improved forecasts of temperature and precipitation using a real-time
competition and cash prizes as incentives.
Prize Competition Summary: Improved sub-seasonal forecasts for
weather and climate conditions (lead-times ranging from 15 to 45 days
and beyond) would allow water managers to better prepare for shifts in
hydrologic regimes such as the onset of drought or occurrence of wet
weather extremes. The challenge with sub-seasonal weather and climate
forecasting is that it encompasses the time frame where initial state
or condition information, such as coupled land-atmosphere processes
becomes less important, and
[[Page 92849]]
slowly varying long term states, such as sea surface temperature, soil
moisture, and snowpack, become more important to predictions. In
addition, the relative importance of the initial state or condition,
versus the longer term state, depends on the lead time, region of
interest, and time of year.
Accurate sub-seasonal weather forecasting has proven to be
particularly difficult to accomplish but is of great interest to water
managers tasked with predicting sub-seasonal streamflow and water
supply. Sub-seasonal forecasting, which spans approximately 15 to 45
days in the future, is difficult because it bridges short-term
forecasting, where initial conditions primarily determine upcoming
weather, and long-term forecasting in which slowly varying factors
become more important.
This Challenge is asking Solvers to develop systems that improve
upon existing sub-seasonal temperature and precipitation forecasts.
Solvers are not required to develop entire systems from scratch.
Methods could include, but are not limited to, approaches for improving
the accuracy of existing sub-seasonal forecasts, techniques that
leverage climate teleconnections, or statistical models. This Challenge
will be active for approximately 17 months, starting with a 3-month
development period followed by a 1-month ``pre-season'', and a 12-month
competition period. Following the competition period, Solvers will have
1 month to prepare final submissions. During the competition period,
Solvers will be required to upload sub-seasonal forecasts every 2
weeks. An online leaderboard hosted by the National Integrated Drought
Information System at drought.gov will track and display Solvers'
performance for the duration of the competition period. Please note
that InnoCentive usernames will be shared with the National Oceanic and
Atmospheric Administration (NOAA) as part of the forecast evaluation
process and leaderboard tracking. At the conclusion of the competition
period, a final submission is required.
Prizes total $800,000. Four categories are defined by two forecast
outlook periods and two forecast variables (temperature and
precipitation). In each category, prizes for eligible solvers are as
follows:
1st place--$100,000
2nd place--$50,000
3rd place--$25,000
In addition, one $25,000 prize per category may be awarded to an
eligible solver based solely on hind-cast performance, submission of
which is a requirement to be eligible for the above listed prizes.
Final submissions to the Challenge should include the following:
1. The detailed description of the proposed Solution addressing the
specific Technical Requirements that are presented in the Detailed
Description of the Challenge. This description should be accompanied by
a well-articulated rationale for the proposed Solution.
2. Locally executable application and corresponding source code to
implement the proposed Solution that are documented.
3. Proof-of-concept data obtained as outlined in the Detailed
Description of the Challenge.
The Challenge award is contingent upon theoretical evaluation and
operational validation of the submitted Solutions by the Seeker
(Reclamation).
To receive an award, the Solvers will grant to the Seeker a non-
exclusive license to practice their solutions and make the solution
available as open source software licensable under either Berkeley
Software Distribution or General Public License Open Source license.
Technical Requirements. Water managers and other users have
indicated that they want the existing forecasts to be better rather
than wanting the development of new forecast systems. Therefore, this
competition seeks new and novel forecast method(s) to improve the
forecasting of precipitation and temperature that can be incorporated
into existing frameworks relied upon for water management. Solvers may
leverage existing forecasts or ensembles in their solution, but must be
able to demonstrate appreciable value added by the solution relative to
any input or foundational framework. Specifically, the competition
desires solutions that can outperform current operational forecasts
(including forecasts from operational centers outside the U.S.) and a
damped persistence forecast at a 1x1 degree gridded resolution for the
western United States at two forecast outlooks: 15-28 days (weeks 3-4)
and 29-42 days (weeks 5-6) for temperature and precipitation. Overall
skill- spatially and across a range of weather/climate conditions- is
most important. The ability to skillfully forecast extreme conditions
is also very desirable. Any system that meets these criteria is sought.
Evaluation criteria. Forecast skill will be evaluated for
temperature and precipitation separately since the drivers responsible
for prediction of these variables are different and the subsequent
skill level is also expected to be different. Moreover, the 15-28-day
and 29-42-day periods will be evaluated individually for similar
reasons. Winning forecasts must outperform NOAA's sub-seasonal modeling
system, the Climate Forecast System Version 2 (CFSv2) and damped
persistence forecasts (see definitions below). Specifically, skill will
be evaluated individually for temperature and precipitation for weeks
3-4 and weeks 5-6 as the highest skill over the competition's
identified geographic area, averaged over the entire competition time
period. To be prize eligible, Solvers must also demonstrate historical
skill of statistical significance that is equal to or greater than that
of the CFSv2 through submission of a hind-cast analysis described
below.
Definitions.
Anomaly is defined as the difference between a given value
and climatology for a specific location (grid cell by grid cell) and
time.
Climatology is defined as average temperature and
precipitation for a specific 2-week period as computed from the Climate
Prediction Center's daily unified gauge data set for precipitation at
ftp://ftp.cpc.ncep.noaa.gov/precip/CPC_UNI_PRCP/GAUGE_GLB/, and the
Climate Prediction Center's global gridded temperature data set over
the 30 year period of 1981-2010 at ftp://ftp.cpc.ncep.noaa.gov/precip/wd52ws/global_temp/.
A damped persistence forecast will be represented using
seasonally developed regression coefficients based on the historical
climatology period of 1981-2010 that relate observations of the past 2
weeks to the forecast outlook periods on a grid cell by grid cell
basis.
Skill is defined as spatial anomaly correlation, averaged
over time. Final competition standings will be the average spatial
anomaly correlation of all 26 forecasts issued for the western U.S.
geographic area, as defined by the forecast submission template.
The CFSv2 forecasts at https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/climate-forecast-system-version2-cfsv2 will be
combined into a forecast baseline over the same time period as Solver
submissions using an ensemble mean of the 32 ensemble-member forecasts
leading up to each competition submission. The CFSv2 is run out to 45
days, 4 times daily, with four initial conditions per run for a total
of 16 forecasts per day. For example, for a forecast due March 1st, the
16 CFSv2 ensemble-member forecasts issued on February 27 and the 16
CFSv2 ensemble-member forecasts issued on
[[Page 92850]]
February 28 will be averaged together for each grid and over the two
forecast-periods to become the forecast baseline for the March 1st due
date. Further, the ensemble mean will be bias corrected using the
method employed by NOAA's Climate Prediction Center in developing their
operational forecasts. Specifically, this process involves comparing
the CFSv2 reforecasts for the period of 1999-2010 with observed data to
establish and correct daily bias on a grid cell by grid cell basis.
This is done for temperature and precipitation at both forecast outlook
periods.
Forecast submission requirements and instructions. Over the course
of this Challenge, Solvers will be required to submit 2 pre-season
forecasts and 26 forecasts during the year-long competition, for each
of the four categories detailed above and must meet the following
requirements:
1. Beginning at the start of the competition period and every 2
weeks thereafter (see below for required submission deadlines) for the
duration of the competition, Solvers will submit 1x1 degree gridded
forecasts corresponding to western U.S. competition geographic area as
defined by the forecast template available on Reclamation's current
competitions Web page listed in the ADDRESSES section of this notice
for:
a. Average temperature (degrees C) for days 15-28.
b. Cumulative precipitation (millimeters) for days 15-28.
c. Average temperature (degrees C) for days 29-42.
d. Cumulative precipitation (millimeters) for days 29-42.
2. To be eligible for an award, Solvers must submit all 26
forecasts on time. A grace of two missed forecast per category will be
allowed. For the purpose of computing skill, climatology will be
substituted for a missed forecast. More than two missed forecast in any
category will result in disqualification.
3. Forecast submissions will be made through a portal hosted by
NOAA and must utilize the supplied template, available on Reclamation's
current prize competitions Web page.
4. The first month of the 13-month competition period (i.e. the
first two forecasts) will be considered a `pre-season' for Solvers to
become familiar with the submission process and will not count toward
their standing in the competition or against the two missed forecast
grace described above.
Forecast submission deadlines. The forecast competition will run
for 1 year following the `pre-season' with new forecast submissions
required every 2 weeks. All forecasts need to be received by 0Z on the
day of the required forecast submission. All aspects of meteorology are
based upon a world-wide 24-hour clock called Zulu time (Z), more
commonly called Coordinated Universal Time (UTC). More information on
this topic is available on the National Weather Service Web site at
http://www.srh.noaa.gov/jetstream/synoptic/time.html, including
conversion to common U.S. time zones. Specific forecast submission
deadlines, including dates and times, will be available on
Reclamation's current prize competitions Web page listed in the
ADDRESSES section of this notice, the first of which is due
approximately 3 months following the announcement of this challenge.
In addition to requirements listed above for the forecast
submissions, proposed solutions must meet the following Technical
Requirements:
1. For first, second or third place, systems must outperform, based
on the formula described above, cumulatively over the competition time
period, the CFSv2 baseline and damped persistence forecasts for at
least one category: Week 3-4 temperature, week 5-6 temperature, week 3-
4 precipitation, or week 5-6 precipitation.
2. Must demonstrate historical skill of statistical significance
that is equal to or greater than that of the CFSv2 through submission
of a hind-cast analysis. The CFSv2 reforecasts at http://nomads.ncdc.noaa.gov/thredds/catalog/cfsr-hpr-ts45/catalog.html will
provide the baseline for the hind-cast, in the same way that the CFSv2
forecasts provide a baseline for the real-time competition. Two notable
distinctions between the CFSv2 forecasts and reforecasts are (1) the
reforecast baseline will be based on a mean of 8 ensemble members (only
one set of initial conditions were used, thereby producing 4
reforecasts per day) and (2) bias correction to the reforecast will be
developed and applied for each reforecast year independently, so as to
not leverage knowledge of that year's reforecast performance toward
bias correcting itself. Note that skill and prize eligibility will be
evaluated for each category individually--i.e. to be prize eligible in
a particular category, Solvers need only outperform the CFSv2 in that
category. To do this, Solvers will submit, hind-casts for the four
categories, issued every 2 weeks, for the period of 1999-2010, no later
than the Challenge final forecast due date. The hind-cast should be
performed as a ``leave one out cross validation.'' This is accomplished
by removing one year of observed historical data from the period of
1999-2010, calibrating the model based on the remaining years of data,
and forecasting that year. Hind-casts are to be issued at the month/day
combinations specified in the forecast submission deadlines referenced
above. Given the dates of the competition, each hind-cast will span 2
calendar years, thereby resulting in 11 1-year hind-cast periods for
the ``leave one out cross validation.'' This process should be repeated
until forecasts have been issued for all 11 years. Solvers will submit
one historical hind-cast per category for the 11 years. For more
information on this technique, see the discussion on cross-validation
by Clarke et al. in Principles and Theory for Data Mining and Machine
Learning (2009).
3. Solvers may utilize any available sub-seasonal forecast as the
starting point for their system, but must demonstratively improve upon
that forecast.
4. Must be written in C++, R, python, Fortran, or other widely
recognized programming languages and be licensable under either
Berkeley Software Distribution or General Public License Open Source
license.
5. Solvers with winning solutions are required to provide all code,
data, and other components of their forecast system necessary to run
the system and reproduce the forecasts issued in the competition.
Failure of code to reproduce performance during the competition or for
the hind-cast may result in disqualification. Furthermore, the Solvers
may be required to iterate with competition judges to ensure
documentation is sufficient.
Project Deliverables: In addition to the hind-casts, 2 pre-season
forecast submissions, and 26 forecast submissions during the year-long
competition, Solvers must submit a final proposed Solution by the
Challenge deadline. The submitted proposal should include the
following:
1. Detailed description of a sub-seasonal forecasting system that
meets the Technical Requirements listed in the Detailed Description and
Requirements section of the Challenge.
2. A well-reasoned rationale supporting the methodology of the
proposed system and addressing each of the Technical Requirements
described in the Detailed Description.
3. Locally executable application and corresponding documented
source code implementing the proposed Solution.
The Seeker may wish to partner with the Solver at the conclusion of
the Challenge. Solver should describe their expertise and include a
statement indicating their interest in this opportunity.
[[Page 92851]]
The proposal should not include any personal identifying
information (name, username, company, address, phone, email, personal
Web site, resume, etc.) or any information the Solvers may consider as
their Intellectual Property they do not want to share.
Judging: An online leaderboard hosted by the National Integrated
Drought Information System will track and display Solvers' performance
for the duration of the competition period. The Challenge award is
contingent upon theoretical evaluation and operational validation of
the submitted Solutions by the Seeker. If multiple proposals meet all
the Solution Requirements, the Seeker reserves the right to award only
the top three solutions per category which they believe are of sound
technical foundation. After the Challenge final submission deadline,
submissions will be identified as potentially prize eligible, as
determined by the quantitative forecast evaluation performed by NOAA
and described above. Of those potentially prize eligible solutions, the
Judging Panel will evaluate each with respect to the Solution
Requirements and make a decision on winning solution(s). The Judging
Panel may be composed of Federal and/or Non Federal scientists,
engineers, and other technical experts, including subject matter
experts from the listed collaborators for this Challenge. All Solvers
that submit a proposal will be notified on the status of their
submissions. Decisions by the Seeker cannot be contested.
Eligibility Rules: To be able to win a prize under this
competition, an individual or entity must:
1. Agree to the rules of the competition (15 U.S.C. 3719(g)(1));
2. Be an entity that is incorporated in and maintains a primary
place of business in the United States, or (b) in the case of an
individual, a citizen or permanent resident of the United States (15
U.S.C. 3719(g)(3)).
However, submissions can be entertained from all Solvers regardless
of whether they are U.S. citizens/entities. Meritorious submissions
from non-eligible persons and entities, if any, will be recognized in
publications issued by the Seeker announcing the results of the
competition, such as press releases. Non-U.S citizens/permanent
residents or non-U.S entities can also be included on U.S. teams.
However, prizes--whether monetary or otherwise--will only be awarded to
eligible persons and entities under the authority of the America
COMPETES Reauthorization Act of 2010 (15 U.S.C. 3719).
3. Not be a Federal entity or Federal employee acting within the
scope of their employment (15 U.S.C. 3719(g)(4)). A Federal entity is
defined by 5 U.S.C. Appendix 8G with a list of current Federal entities
periodically posted on the Federal Register.
4. Assume risks and waive claims against the Federal Government and
its related entities (15 U.S.C. 3719(i)(1)(B)); and,
5. Not use Federal facilities, or consult with Federal employees
during the competition unless the facilities and employees are made
available to all individuals and entities participating in the
competition on an equitable basis.
The following individuals or entities are not eligible regardless
of whether they meet the criteria set forth above:
1. Any individual or organization who employs an evaluator on the
Judging Panel or otherwise has a material business relationship or
affiliation with any Judge.
2. Any individual who is a member of any Judge's immediate family
or household.
3. The Seeker, participating organizations, and any advertising
agency, contractor or other individual or organization involved with
the design, production, promotion, execution, or distribution of the
prize competition; and all employees, and all members of the immediate
family or household of any such individual or organization.
4. Any individual or entity that uses Federal funds to develop the
proposed solution now or any time in the past, unless such use is
consistent with the grant award, or other applicable Federal funds
awarding document. Note: Individuals or entities that have been funded
by the Federal Government in the past to work within the technical
domain of the competition are eligible provided their specific
submission was not developed by them with Federal funds. Submissions
that propose to improve or adapt existing federally funded technologies
for the solution sought in this prize competition are also eligible.
Individuals are also encouraged to consult with their employer Ethics
Officer for additional guidance and considerations.
Consultation: Reclamation and collaborator scientists, engineers,
and technical specialists were consulted in identifying and selecting
the topic of this prize competition. Direct and indirect input from
various stakeholders and the broader water resources community of
practice were also considered.
Public Disclosure: InnoCentive, Inc. is administering this
challenge under a challenge support services contract with Reclamation.
Participation is conditioned on providing the data required on
InnoCentive's online registration form. Personal data will be processed
in accordance with InnoCentive's Privacy Policy which can be located at
http://www.innocentive.com/privacy.php. Before including your address,
phone number, email address, or other personal identifying information
in your proposal, you should be aware that the Seeker is under no
obligation to withhold such information from public disclosure, and it
may be made publicly available at any time. Neither InnoCentive nor the
Seeker is responsible for human error, theft, destruction, or damage to
proposed solutions, or other factors beyond its reasonable control.
Liability and Indemnification: By participating in this Challenge,
each Solver agrees to assume any and all risks and waive claims against
the federal government and its related entities, except in the case of
willful misconduct, for any injury, death, damage, or loss of property,
revenue, or profits, whether direct, indirect, or consequential,
arising from participation in this Challenge, whether the injury,
death, damage, or loss arises through negligence or otherwise. By
participating in this Challenge, each Solver agrees to indemnify the
federal government against third party claims for damages arising from
or related to Challenge activities
No Insurance Required: Based on the subject matter of the
Challenge, the type of work that it will possibly require, as well as
an analysis of the likelihood of any claims for death, bodily injury,
or property damage, or loss potentially resulting from competition
participation, Solvers are not required to obtain liability insurance
or demonstrate financial responsibility in order to participate in this
Challenge.
Dated: December 7, 2016.
Levi Brekke,
Acting Science Advisor.
[FR Doc. 2016-30593 Filed 12-19-16; 8:45 am]
BILLING CODE 4332-90-P