Guidelines for CIHR, NSERC and SSHRC Selection Committees

[ Table of Contents ]

3. The Review Process

The review process consists of the eight following phases which are further described below:

• Receipt and Assignment of Applications
• Identification of Conflicts by Members
• Review and Pre-Scoring of Applications
• Preliminary Ranking of Applications and Assignment of Readers
• Selection Committee Meeting in Ottawa
• Committee Recommendations to the Selection Board
• Selection Committee's Policy Discussion
• Preparation and Submission of the Chair's Report

3.1 Prior to the Selection Committee Meeting

3.1.1 Receipt and Assignment of Applications

Applications are received at the agency and the program administrator assigns them to committee members.

Each agency-specific Selection Committee is a multidisciplinary committee; nevertheless, members will be asked to review and score applications in a number of different research areas relative to the granting agency's mandate, including applications in areas which may not be their own. Each application will be reviewed by two committee members. One member will serve as primary reviewer and the other will serve as secondary reviewer.

Once applications have been received at the agency, the program administrator will assign a subset of applications to each member using ResearchNet - the electronic web portal used for review submission (www.researchnet.ca). The program administrator will seek to balance workload, taking into consideration potential conflicts, language capabilities and areas of expertise. Each member of the committee is responsible to review their assigned applications.

Prior to the Selection Committee meeting, each committee member will be granted access to ResearchNet to review their Banting Postdoctoral Fellowships assignments. Members will receive an email notification advising them once their assignments are available.

3.1.2 Identification of Conflicts by Members

Committee members identify additional conflicts with those applications to which they were assigned.

Using ResearchNet, members are asked to identify potential conflicts in reviewing applications to which they were assigned. The list below is not exhaustive – reviewers should consult the Confidentiality and Conflict of Interest Guidelines that appear as the first mandatory task in ResearchNet. Should a member feel unable, for any reason, to review an application which they accepted as an assignment, they should contact the program administrator immediately and the application will be re-assigned to a new reviewer.

Are you in conflict with the applicant? The following guidelines governing conflicts of interest apply to the evaluation of Banting Postdoctoral Fellowships applications. Guidelines of this nature cannot foresee all possible situations, and the granting agencies rely on the judgment of committee members when identifying conflicts. A member may not act as a reviewer for the application if: the member is affiliated with the host institution proposed by the applicant; the candidate is currently at the same department as the committee member; or the candidate is, has been, or will be under the direct or co-supervision of the committee member, or if there is a personal or professional link. When a committee member is in conflict with a particular application, the member will be asked to leave the room before the deliberation of that candidate's dossier begins. It is the responsibility of committee members to declare any conflicts of interest prior to the review of an application. In cases where the guidelines do not clearly describe a situation, or where the committee or Chair has difficulties making a decision about a particular situation, granting agency staff has the responsibility to rule.

3.1.3 Review and Pre-Scoring of Applications

Committee members review and pre-score their assigned applications using ResearchNet.

a) Introduction

Following the identification of any conflicts, committee members may begin to review and pre-score their assigned applications. Committee members are to pre-score all applications assigned to them using ResearchNet and submit their pre-scores electronically to the agency prior to the Selection Commitee meeting. A specific deadline date will be communicated to committee members in due course.

Are the applicant and the proposed research eligible for support? Agency staff is responsible for pre-screening all applications against eligibility requirements to determine if the proposed research falls within the agency's mandate and to ensure that the applicant is eligible to apply to the program. The assessment of mandate eligibility (and, to the extent possible, most eligibility issues depending on volume received) will be completed prior to assignment. Committee members are invited to bring any concerns or questions regarding the eligibility of an applicant  to the attention of agency staff, but should proceed with the evaluation of the application while the situation is assessed.

Is the proposed research contrary to the Tri-Council Policy Statement: Integrity in Research and Scholarship? If you judge that the proposed research may be contrary to the Tri-Council Policy Statement: Integrity in Research and Scholarship or if you have any ethical concerns with respect to the application, proceed with the evaluation as usual and inform the agency's Banting Postdoctoral Fellowships program administrator of your concerns as soon as possible.

b) Selection Criteria

Applicants will be evaluated and selected based on the following three criteria, which will be weighted equally by the Selection Committees:

1. Research excellence and leadership in the research domain;
   
2. Quality of the applicant's proposed research program; and
   
3. Institutional commitment and demonstrated synergy between applicant and institutional strategic priorities

APPENDIX A – Application Process "Step-by-Step Guide to Applying" as Communicated to Applicants
summarizes the steps to be followed by applicants in completing and submitting their applications, including information on how each application element relates to the selection criteria. Members should read this information carefully and refer to it when evaluating the applications assigned to them.

1. Research excellence and leadership in the research domain - demonstrated capacity for research excellence based on track record to date as defined by quality of applicant's research contribution, and demonstrated capacity for leadership in the research domain defined by the sphere of influence achieved to date by the applicant.

The candidate's research history and the impact of their activities in their area of expertise to date are important indicators of their potential as research leaders of tomorrow. Reviewers should consider the sphere of influence of the candidate relative to others along the following continuum of expanding impact:

• Research Program
• Institution
• Research Community
• International Research Community
• Society at Large

In the evaluation of this criterion, both the nature/prestige of this award and the stage and nature of the applicant's career should be considered. In the case of health professionals, consideration should be given to the standards in terms of research productivity, etc. for their level of experience/qualifications relative to those with a PhD. For applicants who have relevant work experience, scientific productivity prior to graduate school should be considered.

Supporting evidence to be evaluated for this criterion:

• 3 Referee Assessments (Research Excellence and Demonstrated Leadership section)
• Research Contributions of applicant (max 3 pages)
  • Part 1: List of contributions
  • Part 2: Choose up to three contributions and describe their significance
• Leadership Contributions of applicant (max 1 page)
  • Part 1: List of key contributions
  • Part 2: Choose up to three contributions and describe their impact and importance
• Applicant Banting CCV – to be completed via the Common CV
• Special Circumstances (max 1 page – optional)
  • To address eligibility timeline issues (bereavement period, parental leave, etc)
• Health professionals should include any post-degree training that is not research-focused (e.g., clinical training, residency, etc.)
• Only in exceptionally rare circumstances will a Banting Postdoctoral Fellowship be awarded to an applicant who is staying at the same institution where they completed their PhD, PhD- equivalent or health professional degree. Applicants who wish to undertake postdoctoral level research training in the same institution (or its affiliated hospitals, research centres and other laboratories) in which they received their PhD, health professional or equivalent degree must provide a strong justification in the Special Circumstances attachment. The more similar the proposed research environment is to that which existed during the applicant's PhD, PhD-equivalent or health professional degree, the stronger the required justification. This should be balanced against any indicators of a broad, cross-institutional outlook (e.g. inter-university collaborations, participation in international networks, etc.)

2. Quality of the applicant's proposed research program,- examined in terms of the potential of the proposed research program (taking into consideration feasibility, timelines, novelty of research), executed in the proposed institutional environment, to position the applicant for significant impact through a research-intensive career (potential for significant impact)

The potential contribution of proposed research to the advancement of knowledge in the field is dependent on the promise and quality of the proposal, the environment in which it will be conducted, and the aspirations of the candidate. Reviewers should consider the potential for significant impact presented by the above-noted combination of factors.

Supporting evidence to be evaluated for this criterion:

• 3 Referee Assessments (Merit of Proposed Research section)
• Career Aspirations and Choice of Institution (max 1 page)
• Description of career aspirations of applicant and rationale for decision to pursue a postdoctoral research experience at the proposed host institution and in association with the proposed supervisor.
• Research Proposal (max 4 pages)
• Lay Summary of Research Proposal (max ½ page)
• Bibliography, citations included in the "Research Proposal" (max 5 pages)

3. Institutional commitment and demonstrated synergy between applicant and institutional strategic priorities – examined in terms of the:

• Demonstrated commitment of the institution to support development of the applicant's research and leadership capacity through institutional support (funding, facilities, equipment, etc.) and professional development; and
• Demonstrated research capacity in the area of the applicant's proposed research which will enable the institution to provide an intellectually stimulating environment to position the candidate as a research leader; and
• Demonstrated alignment and synergy between the applicant's research ambitions and the Institution's potential to benefit strategically from engagement with the applicant (alignment with institution's strategic priorities).

This should not be an assessment of the institution per se, but rather to its commitment to the applicant, its capacity to enable the applicant to become a future leader in their chosen field and its potential to build upon its strategic priorities.

Supporting evidence to be evaluated for this criterion:

• 3 Referee Assessments (Suitability of the Research Environment section)
• Institutional Synergy and Letter of Endorsement (max 2 pages)
  • To be completed by the proposed host institution
  • Describe the institution's documented strategic directions and illustrate the synergy between that strategy and the applicant's proposed research program
  • In order to ensure that gender equity issues are considered in an institution's decision to support a given applicant, proposed host institutions will be required to confirm their commitment to gender equity and involve institutional equity officers (or equivalent) in the endorsement of applicants for these awards
  • The statement must be signed by the institution's President or their official delegate.
• Supervisor's Statement (max 3 pages)
  • To be completed by the proposed supervisor
  • Provide evidence that the supervisor is well positioned to provide the required support to the applicant in relation to the research proposed
  • Includes Supervisor's description of their academic and research background, key contributions/accomplishments to date (Note that a full CV for the supervisor is not requested)
• Research Environment (2 pages)
  • To be completed by the proposed host institution
  • Provide details concerning the applicant's proposed research environment. These details should clearly state the Institution's commitment, for example in terms of the funding, facilities/resources and personnel that will be available to support the applicant.
• Professional Development (1 page)
  • To be completed by the proposed host institution
  • Describe the institution's commitment to the applicant's professional development, clearly indicating the resources and/or mentoring activities that are available through the institution to support career development.

c) Pre-Scoring

Members are expected to assign a score between 0.1 and 9.0 (in increments of 0.1 with 9.0 being high and 0.1 being low) to their assigned applications for each of the three Selection Criteria. ResearchNet will automatically calculate the application's overall score by averaging the score assigned to each criterion.

In order to encourage members to differentiate between highly promising applicants, and to ensure that the full range of the scale is used, a forced binning system has been implemented. In order for this system to work effectively, it is essential that the entire range within a bin be used. Applications assigned to a member must fall within each of the three following bins according to the proportions in the table below:

Funding Recommendation	Score	Proportion
Recommended	7.5 – 9.0	10%
Could be recommended	5.1 – 7.4	20%
Not recommended	0.1 – 5.0	50%

The remaining 20% may be distributed between the three bins as members deem appropriate in order to compensate for a relatively strong or a relatively weak subset of applications. We recommend that members review a number of applications before entering scores as this will provide an adequate frame of reference for members to score all of their assigned applications.

As application volumes for this program will be difficult to predict, reviewers are encouraged to scale the level of effort for reviews of lower quality applications to the number of applications assigned (the higher the application volume, the lower the time spent on applications deemed to be in the "not recommended" range). Depending on application volume received, variations on the forced binning system and/or triage of lower quality applications may be implemented.

Note: Members will be required to bring to the meeting their personal notes on their assigned applications. All notes must be securely destroyed after the meeting.

Reviewer pre-scores must be submitted electronically to the agency via ResearchNet by prior to the Selection Commitee meeting (exact date to be confirmed by the agency's program administrator). Please respect the date provided, as staff cannot determine which applications will be reviewed at the Selection Commitee meeting until all pre-scores have been received. Members are encouraged to contact agency staff at any time if they need assistance during the pre-scoring process.

3.1.4 Preliminary Ranking of Applications and Assignment of Readers

The program administrator prepares a ranked list of applications, provides members with a list of applications being discussed at the Selection Committee meeting and assigns a reader to each application being discussed.

Once all the pre-scores have been received, agency staff will calculate the average of the two reviewers' overall scores and rank the applications from highest to lowest. The pre-score ranking of all applications will be made available to committee members prior to the Selection Committee meeting.

• It is expected that the 40 top-ranked applications will be discussed at the Selection Committee meeting held in Ottawa. A list identifying the applications being discussed will be provided to committee members by the program administrator.
• In addition, the intent is that those applications for which there is a discrepancy between the two reviewers' pre-scores will also be discussed at the meeting. For applications below the 40th ranked, a discrepancy consists of a difference of 2.0 or more between the two reviewers' pre-scores and where one of the pre-scores is higher than the overall score of the 24th ranked applicant.
• It is anticipated that those applications ranked 41 and below will not be discussed further except at a committee member's request¹.

Applications being discussed at the meeting will be assigned to a third member who will have the role of reader (exact number of applications per member will depend on the number of discrepancies). The reader will be responsible for reading the application prior to the meeting and must be ready to provide comments on the application's strengths and weaknesses during committee deliberations paying particular attention to the third selection criterion, institutional commitment and demonstrated synergy between applicant and institutional strategic priorities.

3.2 During the Selection Committee Meeting

3.2.1 Selection Committee Meeting in Ottawa

Committee members meet in Ottawa for the Selection Committee Meeting.

At the meeting the 40 top-ranked applications, discrepant applications and any others identified for discussion by committee members (the total number of files to be discussed will be dependant on the number of discrepancies) will be reviewed one at a time, in rank order from highest to lowest. The primary reviewer assigned to an application is responsible for verbally summarizing the strengths and weaknesses of the application by addressing each criterion in turn, as a lead-in to committee discussion. The secondary reviewer will provide comments to highlight agreement or discrepancies with the primary reviewer's analysis. Reviewers should summarize the relevant information (positive and negative) in approximately five minutes. The reader will then be given the opportunity to add any additional information which they feel is required by the committee in order to assess the application, focusing on the third selection criterion: institutional commitment and demonstrated synergy between applicant and institutional strategic priorities. Comments or questions may be raised by the other members in order to clarify any of the information presented by the reviewers and reader.

Following the comments from the assigned reviewers/reader and the committee's discussion, the committee will come to a consensus on the application's final score. (It is expected that the Selection Committee will decide on a consensus score for each of the three Selection Criterion and these will then be averaged to arrive at a final score for the application)
Once all the applications have been discussed and scored, the ranking of the applications will be reviewed to determine a cutoff for funding and to sort out any ties in scores between the 23/24 ranked application and that at funding cutoff. Additionally, all ties involving applicants proposing foreign host institutions ranked above the funding cutoff will be resolved.

The committee's overall rating on each criterion for an individual application will be made available to the applicant following the competition. This will provide feedback to the applicant on the relative strengths and weaknesses of their application and their relative ranking. Committee members will not submit written comments on individual applications.

Up to 25% of the Banting Postdoctoral Fellowships may be awarded annually to those who apply in collaboration with a foreign institution. No special consideration is required by the Selection Committees in ranking candidates who have proposed a foreign host institution. Members will review and rank all applications based on the selection criteria. The agencies will allocate these awards based on the relative ranking of the applications following the Committee review.

The final ranked list of all the applications will be submitted to the Selection Board for their endorsement.

Any comments related to the selection process should be brought to the attention of the agency's program administrator or a member of the agency's Banting Postdoctoral Fellowships team.

3.2.2 Committee Recommendations to the Selection Board

Each agency-specific Selection Committee finalizes their ranking list and establishes a cutoff for funding.

The final ranked list of all the applications identified by each agency-specific committee will be submitted to the Selection Board for their endorsement. This list will identify: (1) applications recommended for an award after staff's considering of the foreign host institution allocations and (2) he funding cutoff for alternate awards.

When offering awards to alternates, due either to declines or the 25% cap on foreign host institutions, program staff will not go below the funding cutoffs established by the Selection Committees.

The Selection Board's endorsement will result in the recommendation of 70 Banting Postdoctoral Fellowship recipients to the Steering Committee.

3.2.3 Selection Committee's Policy Discussion

Agency staff will lead the Banting Postdoctoral Fellowships Selection Committee's policy discussion.

At the end of the meeting, once the committee's recommendations have been made, members will be asked to participate in a policy discussion. The discussion generally includes comments on the quality of applications reviewed, possible improvements to the review process, future membership needs, competition and travel logistics, hotel accommodations, the agency's administrative services, policies, etc.

Note: Prior to the meeting members are invited to give some thought to individuals who would be appropriate to serve as future committee members. Any suggestions can be provided to the program administrator.

3.3 Following the Selection Committee Meeting

3.3.1 Preparation and Submission of the Chair's Report

The committee Chair, in consultation with the committee members and program staff, prepares and submits the Chair's Report to the agency.

Following the meeting, each committee Chair will be responsible for preparing a Chair's report. This report is presented to the Selection Board and is a means of communicating the Selection Committee's recommendations, comments and suggestions for improvements to the competition process.

The Chair, in collaboration with members and the program administrator, will prepare and submit the report to the agency prior to the Selection Board meeting. In addition to the final ranked list of all applications, the report should include, when relevant, comments on such matters as:

• the review procedures followed by the committee;
• specific difficulties experienced in assessing applications;
• recommendations for policy revisions;
• suggestions for information to be submitted by applicants (application materials);
• evaluation criteria;
• suggestions for modifications to the electronic application process; and
• suggestions for membership.

---

1. It should be noted that a maximum of 23 or 24 applications per committee can receive a Banting Postdoctoral Fellowship.