Research Project (RP) Application Reviews

General Review Information

Each specific review criterion (see below) is rated on a 5-point scale; higher scores indicate higher criteria strength. Reviewers consider strengths and weaknesses within each criterion when scoring. (1 = Poor, 2 = Fair, 3 = Good, 4 = Excellent, and 5 = Outstanding) A reviewer’s final recommendation concerning whether to invite or fund a proposal (depending on the Stage level) is not necessarily based on the average of the criterion scores. For example, a reviewer may give only moderate scores to some of the review criteria but still recommend inviting/funding because one criterion critically important to the project is rated highly; or a reviewer could give mostly high criterion ratings but still not recommend inviting/funding because one criterion critically important to the project being proposed is not highly rated.

Review Criteria Categories (see below for detailed information concerning each criterion)

1. Overall fit and potential impact

• Specific aims / scope of work and fit with BEF mission
• Collaboration
• Strength-based orientation
• Potential impact and dissemination plans
• Rationale

2. Design / methodology

• Research design
• Intended study sample
• Procedures and measures
• Analytic plan
• Feasibility

3. Research environment and team

• Research environment
• Expertise of the PI / research team
• Racial/ethnic composition of the leadership of the research team

4. Budget and budget justification

Detailed Information Concerning Review Criteria Specific to Research Project (RP) Applications

Below, the factors considered when scoring each of the criteria are described. The questions in the application most pertinent to each criterion are provided; however, responses throughout the application may be considered when scoring each criterion as well.

1. Overall fit and potential impact

BEF seeks to address disparities in educational opportunities (ages birth through 18) associated with race, ethnicity, and family income. The Foundation pursues its mission by funding research that has the potential to inform practice, private funders, and public policy. When determining the overall fit and potential impact of a proposal, the following criteria are considered:

• Specific aims / scope of work and fit with BEF mission
  • Factors considered when scoring these criteria include
    • what is/are the specific aim(s) of the proposed project (including primary and secondary aims)
    • clarity of aim(s)
    • the goodness of fit with the aim(s) of the mission of BEF
  • RP proposals that have clear and specific aims to provide research findings that will inform practice and/or policy to address disparities in educational opportunities (e.g., access and equity) score higher on these criteria.
  • Section 2, questions 1 and 2; Section 5 also considered if other current or potential funding partners exist and/or if the proposed research is part of a larger project.
• Collaboration
  • Scoring of this criterion is based on the extent to which the proposed project reflects strong collaborations between the research team and other stakeholders as appropriate (e.g., practitioners / service providers, parents / families, economists, policy makers, other community members, etc.) throughout the proposed project (e.g., developing research questions, recruitment, data collection, analyses and interpretation, dissemination).
  • Proposals that demonstrate authentic collaboration among the researchers and other stakeholders throughout the proposed research project score higher on this criterion.
  • Section 2, question 3
• Strength-based orientation
  • Scoring of this criterion is based on the extent to which the research is consistent with a strength-based perspective (i.e., the program and research design recognize the challenges and trauma experienced by different communities while also capitalizing upon each community’s cultural wealth and strengths (e.g., richness of language spoken, cultural norms, traditions).
  • Proposals that do either of the following score very low on this criterion:
    • adopt a color-blind approach (e.g., do not consider the sociohistorical contexts of the race of study participants)
    • reflect a deficit-model orientation (e.g., characterize populations only in terms of needs or lacking in certain respects, or define them as being “at risk” of deficits, thus holding individuals and communities responsible for disparities in outcomes rather than recognizing the systemic challenges and inequities that influence outcomes)
  • Section 2, question 4
• Potential impact and dissemination plans 
  • Factors considered when scoring these criteria include the potential for the research findings to
    • inform practice, major philanthropic giving, and/or policy 
    • be disseminated in 
      • high quality research journals 
      • other outlets that have to high probability of reaching practitioners, funders, and/or policy makers. 
  • Proposals that do the following score higher:
    • can identify concrete ways the findings could have an impact (e.g., how they might influence a practice in a specific setting, specific funders whose work would be informed by the findings, and/or specific policies concerning access and equity) 
    • have clear and specific plans for how results might be disseminated 
  • Section 2, questions 5 and Section 4, question 6
• Rationale 
  • Factors considered when scoring this criterion include 
    • the extent to which compelling rationale is provided to justify the aim(s) of the proposed project 
    • the strength of the empirical literature supporting these aim(s)
    • what “next steps” results from the proposed project might inform (e.g., providing pilot data that might lead to a larger study)
  • Section 3, questions 1 and 2

2. Design / methodology

Scoring of these criteria are based on the extent to which the proposed design and methodology are likely to provide high quality data that will address the specific aim(s) of the proposal. Section 4, questions 1-7; Section 7, timeline with benchmarks

• Research design
  • Scoring for this criterion is based both on 
    • the clarity 
    • the rigor of the evaluation design
  • Note: The Foundation does not view any type of design inherently more superior to another, rather whether the design proposed is appropriate for the specific aim(s) of the project is considered. 
  • Applications that propose the most rigorous design possible for addressing the aim(s) of the project score higher on this criterion. 
  • Section 4, question 1

• Intended study sample 
  • Scoring for this criterion is based on
    • the extent to which evidence is provided that the target sample size is adequate for all aims of the proposed project 
    • the extent to which the intended demographic characteristics are consistent with the aims of the study
    • whether proposed recruitment procedures (and retention over-time if applicable) are likely to recruit (and maintain) a sample representative of the intended population in sufficient numbers to conduct the proposed project
  • When appropriate, sample sizes should be supported by power analyses (stating assumptions made). 
  • Studies that intend to recruit high proportions of children and families in minoritized racial / ethnic groups and low-income communities tend to score higher on this criterion. 
  • Section 4, question 2

• Procedures and measures
  • Scoring for these criteria are based on 
    • whether the proposed procedures and measures are consistent with the aim(s) of the proposed project 
    • the likelihood that they will result in high-quality data to address the aim(s). 
  • Considered are 
    • the purpose of the data to be collected (e.g., to assess child outcomes)
    • the methods used to collect data (e.g., direct assessment, parent-report, administrative data)
    • the strength of the evidence provided concerning the psychometric properties of the methods proposed (including evidence of the validity of the proposed measures for use with the specific populations represented in the sample). 
  • If sources for data include administrative data sets or other existing data sets currently managed by other parties, be sure to attach the data sharing agreements to the application. (Section 7, required attachment) 
  • Proposals that describe clear and sound procedures and those that demonstrate that the measures proposed are reliable and valid for the populations represented in the sample score higher on these criteria. 
  • Section 4, questions 3 and 4

• Analytic plan
  • Scoring for this criterion is based on the level to which the analyses for each specific aim of the proposed project are clearly articulated and appropriate (e.g., accounting for clusters in modeling). 
  • Proposals that do the following score higher:
    • specify the statistical procedures that will be used to address the aims (e.g., descriptive statistics, regression analyses, growth curve modeling, path analyses, how qualitative data will be coded and analyzed)
    • explain proposed data analytic procedures using terms that educated individuals not familiar with the analytic methods can understand
    • articulate the rationale for selecting the analytic methods (providing references as appropriate) 
    • Section 4, question 5

• Feasibility 
  • Scoring of this criterion is based on 
    • the extent to which compelling evidence is provided that the proposed project can be completed as planned 
    • how likely it is that the study will obtain meaningful data and provide findings that can then be disseminated to inform funders, practice, and/or policy. 
  • Proposals that do the following score higher:
    • identify specific potential challenges that may be faced in attempting to conduct the project at each stage of the work 
    • articulate action plans for addressing each potential challenge should it arise score higher on this criterion. 
    • Section 4, question 7

3. Research environment and team

Scoring of these criteria are based on the extent to which the PI’s research environment can support the proposed research project and the evidence provided that the research team has a demonstrated record that provides confidence in their ability to do the project. When the target population of the program includes communities of color, the extent to which researchers of color will lead / co-lead the project are considered as well.

• Research environment
  • Scoring for this criterion is based on the strength of the evidence provided concerning the level of support the PI’s institution is able to provide for the proposed project. 
  • Considered is the research ranking of the institution in general (similar to the Carnegie Classification of Institutions of Higher Education in which universities are classified by research activity as measured by research expenditures, number of research doctorates awarded, number of research-focused faculty).
  • Resources that would be provided by the institution to support the research team while carrying out the proposed project are also considered (e.g., administrative support such as grant support services, office and lab space, personal computers and equipment, software, and /or technological support). 
  • Proposals that provide evidence of a high-quality research environment and identify specific resources provided by the institution to support the proposed work score higher on this criterion. 
  • Section 5, question 8

• Expertise of the PI / research team
  • Scoring for this criterion is based on the strength of the evidence provided (e.g., record of publications, prior grants) that provides confidence in the ability of the PI / Research Team to do the proposed project.
  • Considered when scoring this criterion are: 
    • evidence of expertise concerning the topic area of the proposed project
    • evidence of expertise concerning the research methods (including analytic methods) that would be used in the proposed project
    • the level to which the intended procedures have been previously pilot tested by the research team with the intended target population. 
  • Proposals which provide evidence that at least one member of the team (not necessarily the PI) has experience successfully leading projects of similar or greater scope score higher on this criterion.
  • Section 4, questions 9a; Section 7, CVs / Resumes of key personnel

• Racial/ethnic composition of the leadership of the research team: BEF is among a growing community of foundations that tracks the diversity of its grantees. In addition, proposed projects that intend to recruit individuals from communities of color to participate in the study must have at least one researcher of color in the leadership (PI / co-PI) level of the research team.
  • ALL proposals (regardless of the demographics of the target study sample) must provide the race/ethnicity of each of the key personnel of the research team. 
    • Each key member of the team must be listed with his/her/their racial/ethnic identity specified (i.e., do NOT provide just general descriptions of the diversity of the research team or the institution / organization). 
    • Section 4, question 9bi
  • When the leadership of the research team of the proposed project is diverse, the scoring of this criterion is based on the strength of the evidence provided that the collaboration is likely to be successful and that all voices on the team will valued and influence the project. 
    • If the leadership team has collaborated in the past, proposals that provide evidence that this collaboration was successful score higher on this criterion.
    • If this is a new collaboration and the team includes a researcher that identifies as white who has collaborated on diverse leadership teams in the past, proposals that provide evidence that this collaboration was successful score higher on this criterion. 
    • If this is a new collaboration and the team includes a researcher that identifies as white who has not collaborated on diverse leadership teams in the past, proposals that identify specific efforts that will be made to ensure the collaboration will be successful (e.g., antiracists education opportunities, protocols to address overt and implicit bias) receive higher scores on this criterion. 
  • Section 4, question 9bii

4. Budget and budget justification

Scoring for this criterion is based on the extent to which the proposed budget is in line with specific aim(s) / scope of work proposed and is reasonable and justified.

• Considered are
  • the total budget amount requested the total budget amount requested relative to the funding capacity of BEF
  • whether other support for the project has been secured and/or is pending
  • whether all activities proposed are represented in the budget
  • whether unjustified costs are included
  • the extent to which the FTE percent requested for each key personnel is reasonable given his/her/their role and responsibilities (i.e., is neither too high nor too low)
  • the extent to which estimates for supplies, equipment and other costs (e.g., incentives for participants, costs for assessment materials) are reasonable given the scope of work proposed.
• Proposals that are able to identify clear links between each cost and the specific tasks needed to complete the proposed work score higher.
• Section 1, questions 5; Section 5; Section 6; Section 7, budget justification