Piloting and Evaluating NSF Science Lottery Grants: A Roadmap to Improving Research Funding Efficiencies and Proposal Diversity

02.04.22 | 16 min read | Text byIshan Sharma&Daniel Correa&卡莱布·沃特尼&M. Anthony Mills

This memo was jointly produced by the Federation of American Scientists & theInstitute for Progress

概括

美国no longerleads the world in basic science. There is growing recognition of a gap in translational activities — the fruits of American research do not convert to economic benefits. As policymakers consider a slew of proposals that aim to restore American competitiveness withonce-in-a-generationinvestments into the National Science Foundation (NSF), less discussion has been devoted to improving our research productivity — which has been下降for generations. Cross-agency data indicates that this isnotthe result of a decline in proposal merit, nor of a shift in proposer demographics, nor of an increase (beyond inflation) in the average requested funding per proposal, nor of an increase in the number of proposals per investigator in any one year. As the Senate’s U.S. Innovation and Competition Act (USICA) and House’s America COMPETES Act propose billions of dollars to theNSF研发活动,有机会来支持研究刺激uctivity but it will require exploring new, more efficient ways of funding research.

NSF严格的优点审查过程长期以来一直被视为gold standard用于审查和资助研究。但是，自1950年代成立以来，新兴情况（例如主要研究人员总体人口的显着增长）已引入了传统的同行评估授予过程的许多挑战和效率低下：对研究生产力的税收税作为研究生产力的税收PIS提交2.3 proposals for every awardthey receive and spend an average of116小时赠款per NSF proposal(i.e., “grantsmanship”), corresponding to a staggering loss of nearly45% of researcher time;授予技巧的方向增量研究在高度竞争，共识驱动和基于积分的审查中幸存的可能性最高（与风险更大，小说, or investigator-driven research); rating bias againstinterdisciplinary research或者previously unfunded研究人员以及评论者疲劳。这种效率低下的结果令人不安：由于较少的申请人的资金是增加池的百分比，有些是economic analysissuggests that the value of the science that researchers forgo for grantsmanship may exceed the value of the science that the funding program supports.

我们国家支持新想法的方法应与我们的知识基础一起发展。

我们国家支持新想法的方法应与我们的知识基础一起发展。科学彩票（作为对传统同行评审进程的补充）时，可以通过随机选择一小部分已经执行的，高质量但无资金的赠款提案来提高系统的总体效率成本比率来提高系统的总体效率成本比率。测试多数积极的反馈从新西兰，德国和瑞士的参与者那里，科学彩票将引入随机性的元素，可以在代表性不足的人口统计和地理位置上解锁创新，破坏性的奖学金。

本文提出了一个实验性的NSF科学彩票试点，附录提供了说明性的立法文本。特别是，众议院和参议院科学委员会应考虑在《美国创新与竞争法》（参议院）和《美国竞争法》（众议院）中加入紧密的语言，该法案授权在所有NSF局中使用“授予彩票”，包括该局技术与创新。该语言应该具有扩大创新和循证评论的地理位置的精神，以测试有效的方法。

Challenge and Opportunity

A recentNSF reportpegged the United States as behind China in key scientific metrics, including the overall number of papers published and patents awarded. The numbers are sobering but reflect the growing understanding that America must pick which frontiers of knowledge it seeks to lead. One of these fields should be the science of science — in other words not just what science & technology innovations we hope to pursue, but in discovering new, more efficient ways to pursue them.

自1950年成立以来，NSF在推进美国的学术研究企业中发挥了关键作用，并加强了我们在全球科学研究中的领导地位。特别是，NSF严格的功绩审查过程被描述为gold standard用于审查和资助研究。但是，越来越多的证据表明，尽管值得称赞，但同行审查过程已被扩展到其极限。特别是growing overall populationof researchers has introduced a series of burdens on the system.

一份NSF报告将近70％的提案同样有功，而只有三分之一获得资金。With a surplus of competitive proposals, reviewing committees often face tough close calls. In fact, empirical evidence has found that award decisions change nearly a四分之一of the time when re-reviewed by a new set of peer experts. In response, PIs spend upwards of每个NSF提案的116小时符合授予期望，必须平均2.3 proposals to receive an award— a process known as “grantsmanship” that survey data suggests occupies nearly45% of top researchers’ time。更糟糕的是，这种授予技巧面向编写有关渐进研究主题的建议（versus风险更高，新颖或以研究者为导向的研究），其可能性更高，可以在共识驱动的基于积分的审查中幸存下来。在审阅者方面，数据支持明确的评级偏见interdisciplinary research或者previously unfundedresearchers PIs, while experts increasingly are下降invitations审查为了保护其奖励时间的利益（例如审阅者疲劳）的提案。

These tradeoffs in the current system appear quite troubling and merit further investigation of alternative and complementary funding models. At least oneeconomic analysis表明，随着申请人的资金较少，占池增加的百分比，这是研究人员由于助学士而放弃的科学价值often exceeds the value of the science that the funding program supports.In fact, despite dramatic increases in research effort, America has for generations been facing dramatic declines inresearch productivity。And empirical analysis suggests this isnotnecessarily the result of a decline in proposal merit, nor of a shift in proposer demographics, nor of an increase (beyond inflation) in the average requested funding per proposal, nor of an increase in the number of proposals per investigator in any one year.

随着参议院的《美国创新与竞争法》（USICA）和众议院的《美国竞争法》，向该法提出了数十亿美元NSF研发活动，其中约96％将是分散式通过同行审查，精英授予奖励流程，现在是时候将科学方法应用于我们的替代和补充机制来资助科学研究的实验。

Science lotteries, an effort tested in New Zealand, Switzerland, and Germany, represent one innovation particularly suited to reduce the overall taxes on research productivity while uncovering new, worthwhile initiatives for funding that might otherwise slip through the cracks. In particular, modified science lotteries, as those proposed here, select a small percentage of well-qualified grant applications at random for funding. By only selecting from a pool of high-value projects, the lottery supports additional, quality research with minimal comparative costs to the researchers or reviewers. In a lottery, the value to the investigator of being admitted to the lottery秤直接带有可用奖励的数量。

这些好处转化为经过科学彩票过程的PI的有利的调查数据。例如，在新西兰，大多数科学家支持的随机分配2％的总研究支出。新西兰健康研究委员会首席执行官Sunny Collings，叙述:

“Applications often have statistically indistinguishable scores, and there is a degree of randomness in peer review selection anyway. So why not formalize that and try to get the best of both approaches?”

通过确定进入彩票的条件（例如选择某些资金较低或代表的地区），NSF也可能会使那些申请人较少准备“赠款”。

我们建议的是modified“第二次机会”彩票，通过传统的同行评审过程被认为是有功的提案，但未选择用于资助的提案，作为资助过程中的第二阶段。这种修改的格式可确保彩票选择获得资金的项目的高质量，同时仍创建可以比较当前系统的随机基线。

The use of science lotteries in the United States as a complement to the traditional peer-review process is likely to improve the overall system. However, it is possible that selecting among well-qualified grants at random could introduce unexpected outcomes. Unfortunately, direct, empirical comparisons between the NSF’s peer review process and partial lotteries不存在。Through a pilot, the NSF has the opportunity to evaluate to what extent the mechanism could supplement the NSF’s traditional merit review process.

By formalizing a randomized selection process to use as a baseline for comparison, we may discover surprising things about the make up of and process that leads to successful or high-leverage research with reduced costs to researchers and reviewers. For instance, it may be the case that younger scholars who come from non-traditional backgrounds end up having as much or more success in terms of research outcomes through the lottery program as the typical NSF grant, but are selected at higher rates when compared to the traditional NSF grantmaking process. If this is the case, then there will be some evidence that something in the selection process is unfairly penalizing non-traditional candidates.

Alternatively, we may discover that the average grant selected through the lottery is mostly indistinguishable from the average grant selected through the traditional meritorious selection, which would provide some evidence that existing administrative burdens to select candidates are too stringent. Or perhaps, we will discover that randomly selected winners, in fact, producefewer比通过传统手段选择的候选人值得注意的结果，这将证明现有过程在过滤资金建议方面提供了切实的价值。通过提供比较的基准，彩票将提供基于证据的方法来评估当前同伴的效果- 浏览系统。因此，任何试点计划都应充分利用选择标准的菜单来切换结果，同时还接受了内部和外部科学社区的评估。

Plan of Action

建议1：国会应将NSF引导到通过美国竞争以及《美国创新与竞争法》等其他车辆进行试点实验彩票。

在协调美国竞争与参议院Usica的不同之家时，国会应添加授权“彩票”试点计划的语言。

We recommend opting for signaling language and follow-on legislation that adds textual specificity. For example, in latest text of the COMPETES Act, the responsibilities of the Assistant Director of the Directorate for Science and Engineering Solutions could be amended to include “lotteries”:

秒1308（d）（4）（e）。开发和测试各种绩效审查模型和机制，包括lotteries, for selecting and providing awards for use-inspired and translational research and development at different scales, from individual investigator awards to large multi-institution collaborations;

Specifying language should then require the NSF to employ evidence-based evaluation criteria and grant it the flexibility to determine timeline of the lottery intake and award mechanisms, with broader goals of timeliness and supporting the equitable distribution among regional innovation contenders.

The appendix contains one example structure of a science lottery in bill text (incorporated into the new NSF Directorate established by the Senate-passed United States Innovation and Competition Act), which includes the following key policy choices that Congress should consider:

Limiting eligibility to meritorious proposals;

Ensuring that proposals are timely;

Limiting the grant proposal size to provide the maximum number of awards and create a large sample to fairly evaluate the success of a lottery program;

Rigorous stakeholder feedback mechanisms from the scientific research community;

Fast-tracking award distribution following a lottery; and

Regular reports to Congress in accordance with the NSF’s Open Science Policy to ensure transparency; accountability; and rigorous evaluation.

推荐2：在新的NSF技术，创新和合作伙伴局中创建“转化科学”计划，该计划试行彩票与循证测试一起使用彩票：

First, the NSF Office of Integrative Activities (OIA) should convene a workshop with relevant stakeholders including representatives from each directorate, the research community including NSF grant recipients, non-recipients, and SME’s on programmatic implementation from New Zealand, Germany, and Switzerland in order to temperature- and pressure-test key criteria for implementing piloted science lotteries across directorates.

研讨会的最初目标应该是从PI社区收集有关此主题的反馈和衡量兴趣。为此，探索科学彩票建设中的各种要素是明智的，这是PI社区最支持的。例如，社区应参与提案质量质量的基线参数的开发以及及时，公平的过程，尽管董事会的申请截止日期有所不同。This might include applicants’ consented sign-off before entrance into the lottery, upfront and consistent communications of timelines, and randomization and selection from a pool with scores of at least [excellent/very good/good] during the peer evaluation process described in the NSF’s “提案和裁决政策和程序指南”。

这个车间的另一个目标将是th范围e process of an OIA inter-directorate competition to submit applications in order to receive an award from the Division of Grants and Agreements to pursue pilot science lottery. The workshop should therefore develop a clear sense of opportunities with respect to budget sizing for each directorate and could consider making recommendations about the placement of science lottery pilots across directorates based on willingness to devote experimental resources. To maximize the number of lottery recipients, the proposal must not exceed 200% of the median grant proposal to a given directorate;

Finally, a third goal of the workshop should be to explore standards and timeframe for evidence-based evaluation mechanisms as described above and in the bill-text below, including stakeholder feedback mechanisms, regular reports to Congress, and transparency requirements. Additional mechanisms might include detailed reports on grants and awardees like demographic and geographic information of awardees, comparison of outcomes from traditional awardees and lottery awardees, and a statistical picture of the entire pool of grant proposals entered into the lottery. If the workshop is based on competitive directorate applications, the General Services Administration’s Office of Evaluation Sciences (OES) should be invited for later-stage workshop convenings to provide technical assistance in designing evaluation criteria. Some unifying criteria include meeting the requirements of the NSF’s Open Science Policy, Public Access Policy, and making grant information public as soon as feasible to facilitate rapid evaluation from external stakeholders — a potential metric to judge directorate applications.

Appendix: Bill Text

Note: Please view attached PDF for the formatted bill text

H.______

为国家科学基金会授予彩票建立试点计划。

In the House of Representatives of the United States

2022年2月2日

______________________________

法案

标题：为国家科学基金会授予彩票制定试点计划。

无论是由美国国会美国参议院和众议院颁布的

秒_____。试点计划建立国家科学基金会赠款彩票

调查结果。国会做出以下调查结果：
- Over the past seven decades, the National Science Foundation has played a critical role in advancing the United States academic research enterprise by supporting fundamental research and education across all scientific disciplines;
- 国家科学基金会为科学进步，经济增长，人类健康和国家安全做出了巨大的贡献，其同行评审和优异审查过程已经确定和资助了科学和社会相关的基础研究；
- Every year, thousands of meritorious grant proposals do not receive National Science Foundation grants, threatening the United States’ leadership in science and technology and harming our efforts to lead translation and development of scientific advances in key technology areas; and
- While Congress reaffirms its belief that the National Science Foundation’s merit-review system is appropriate for evaluating grant proposals, Congress should establish efforts to explore alternative mechanisms for distributing grants and evaluating, objectively, whether it can supplement the merit-review system by funding worthwhile projects that otherwise go unawarded.
定义。在本节中：
- 董事会 - “董事会”一词是指在第二秒成立的技术和创新局。该法案的2102。
- 助理主任 - “助理董事”一词是指在第二节中描述的局长助理主任。该法案的2102（d）。
- 基金会。“基金会”一词是指国家科学基金会。
- PAPPG。“ PAPPG”一词是指题为“ OMB控制编号3145-0058”的文件，也称为“提案和奖励政策和程序指南”，由国家科学基金会出版，也出版为NSF 22-1。
- Program.—The term “program” refers to the program established in subparagraph (d) of this section.
- Grant request.—The term “grant request” refers to the amount of funding requested in an individual grant proposal to the National Science Foundation.
- 彩票获奖者。“彩票获奖者”一词是指本节建立的彩票期间选择的授予提案。
- 彩票年。—“彩票年”一词是指由助理董事确定的资格的日历年，该日历为根据该计划建立的彩票。
目的。本节的目的是为基于绩效的彩票制定试点计划，以授予科学研究赠款以：
- Provide grants to meritorious but unawarded grant proposals;
- 探索“第二种外观”的机制，以将赠款分配给有功但被忽视的赠款建议；和
- To evaluate whether alternative mechanisms can supplement the Foundation’s merit-review system.

Establishment.— No later than 180 days after establishment of the Directorate, the Assistant Director shall establish a lottery program to provide second-look grants for meritorious grant proposals that were declined funding by the Foundation.
要求。
- 资格 - 赠款提案应符合彩票的资格：
  - It did not receive funding from the Foundation;
  - 在同行评审过程中，赠款提案获得了被认为有功的整体评估评分；
    - Meritorious.—The Assistant Director determine a minimum score that a proposal must receive during the peer evaluation process described in Chapter III of the PAPPG to be deemed meritorious.
  - 在日历年中，赠款请求不超过给定董事的中位数赠款请求的200％；
  - 该赠款是向基金会中以下局之一提出的：
    - Biological Sciences;
    - Computer and Information Science and Engineering;
    - 工程;
    - Geosciences; or
    - Mathematical and Physical Sciences;
    - 社会，行为和经济科学；
    - Education and Human Resources;
    - 环境研究和教育；
    - 国际工程科学；
  - 基金会官员及时认为该赠款；和
  - Any other criteria deemed necessary by the Assistant Director
- 豁免。如果认为有必要或值得进一步的任务和目标，则助理董事可以：
  - Exempt grant proposals from the requirement in subparagraph (e)(1)(D); and
  - Determine an appropriate method to include such exempted proposals in a lottery.
- Stakeholder Feedback.—Prior to finalizing eligibility requirements, the Assistant Director shall, to the extent practicable, ensure that the requirements take into consideration advice and feedback from the scientific research community. The Federal Advisory Committee Act (5 U.S.C. App.) shall not apply whenever such advice or feedback is sought in accordance with this subsection.

执行。

政策和Procedures.—The Assistant Director shall:
- 制定程序和政策以确保每个授予彩票：
  - 是随机的，为所有参与者提供了平等的机会；和
  - 不易欺诈；
- 确保在第（e）（1）（d）项中所述的董事处公平地分配赠款金额；
- 确保相关外部各方应对参与彩票的义务有适当的通知；
- 确保基金会的相关人员和官员意识到他们对实施计划的职责和责任；
- Ensure that ranked alternative awardees are selected for each lottery in the event that:
  - 彩票获奖者撤回了申请；
  - 在上诉程序后，彩票获奖者获得了基金会资金；或者
  - is otherwise deemed ineligible for a Foundation grant.
- 授予批准。
  - It shall be submitted to the Division of Grants and Agreements for a review of business, financial, and policy implications and award finalization thereafter, as described in PAPPG Chapter III; and
  - It shall not be declined funding by the Division of Grants and Agreements unless granting the award would result in fraud, abuse, or other outcomes deemed egregious and antithetical to the mission of the Foundation.
- Lottery timeline.—For each directorate specified in subparagraph (e)(1)(D), the Assistant Director shall administer a lottery for each calendar year ending in the years [2022, 2023, and 2024].
- 利益相关者的反馈。—提出彩票实施以及进行每次彩票之后，助理主任应在可行的范围内确保彩票实施考虑科学研究界的建议和反馈。每当根据本小节寻求此类建议或反馈时，《联邦咨询委员会法》（第5届U.S.C. App。）不得适用。
提交给局的赠款的截止日期。）（1）（f）本节。

Authorization of Appropriation.— There is authorized to be appropriated to the Foundation [$—,000,000] to carry out this Section.

Evaluation and Oversight and Public Access.
- Evaluation.—The Assistant Director shall:
  - Ensure that awards are evaluated using the same methods and procedures as other grant programs of the Foundation, including as set forth by the Foundation’s Evaluation and Assessment Capability and the Foundation’s values of learning, excellence, inclusion, collaboration, integrity, and transparency; and
  - Establish a rapid, empirically-based evaluation program to determine the effectiveness of the lottery program.
- Reports to Congress.—
  - 周期性。—彩票完成后180天不晚，助理董事应向国会提交摘要报告，包括：
    - 所有授予的赠款清单；
    - 赠款获奖者的人口统计信息；
    - Geographic information of the grant awardees;
    - 有关获得赠款的机构的信息；
    - An assessment comparing lottery grant awardees with those awarded grants through the Foundation’s traditional review process;
    - Information and data describing the entire pool of grant proposals deemed eligible for the lottery.
    - Any other information deemed necessary or valuable by the Assistant Director;
  - Yearly.—Not later than [two years] following the first lottery, the Assistant Director shall submit comprehensive reports on a yearly basis, for a period of five years after the report submission, evaluating awards using the Foundation’s Evaluation and Assessment Capability or other assessment methods used to evaluate grants awarded through the traditional grant process;
  - 最终报告。—在最终彩票完成后[3年]，助理董事应向国会提交最终报告，以评估该计划的成功并评估国会是否应该使该计划永久性。
- Public Access.—The Assistant Director shall:
  - Ensure that the program meets the requirements of the Foundation’s:
    - Open Science Policy;
    - Public Access Policy; and
    - 学习，透明度和完整性的一般价值观。
  - 一旦可行，向公众提供赠款信息，以促进外部利益相关者的快速，基于经验的评估；