Selection committee guide

Foreword

This document is a guide for members of the Banting Postdoctoral Fellowships selection committees of the three federal granting agencies: the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC). It describes the adjudication process for members and chairs of these committees, as well as the policies, guidelines and deliverables that define each stage of the review process.

Members of the Banting Postdoctoral Fellowships selection committees are appointed from the Canadian and international academic communities and are familiar with the mandate, structures and programs of the federal granting agencies. For new members and those appointed from other sectors, please refer to Selecting the Appropriate Federal Granting Agency for more detailed information regarding federal granting agency mandates.

Although we have tried to cover most relevant topics, you may still have some questions. Please direct them to the program administrator responsible for your selection committee.

Table of contents

1. Overview of the Banting Postdoctoral Fellowships program

Announced in the 2010 Federal Budget and officially launched on July 6, 2010, the objective of the Banting Postdoctoral Fellowships program is to attract and retain top-tier postdoctoral talent, both nationally and internationally, to develop their leadership potential and to position them for success as research leaders of tomorrow, positively contributing to Canada's economic, social and research-based growth through leadership in a research-intensive career. Canadian and international candidates are eligible to apply for a Banting Postdoctoral Fellowship. Each Banting Postdoctoral Fellowship is valued at $70,000 (taxable) per annum for two years.

Prospective candidates wishing to apply for a Banting Postdoctoral Fellowship must do so in close collaboration with the host institution at which they seek to conduct their postdoctoral training. In light of the small number of awards in the program, and the demands of both the application process and the review process, institutions have been asked to be highly selective in the applicants they endorse, as Banting Postdoctoral Fellowships will ultimately be awarded to an elite group of researchers who have demonstrated both research excellence and leadership, and synergy with the institution's strategic priorities. Once in full operation, this program will support 140 successful international and Canadian awardees with highly prestigious fellowships. Banting Postdoctoral Fellowships awards must be taken up and held at the host institution that provided the initial endorsement.

The 70 Banting Postdoctoral Fellowships awarded each year are equally divided among the three federal granting agencies with the 70th award rotating annually.

The Vanier-Banting Secretariat, which is housed within the Canadian Institutes of Health Research, is responsible for the day-to-day administration of the program.

2. Roles and responsibilities

2.1 Selection committee members

Responsibilities applicable to all committee members include:

Specific responsibilities of assigned reviewers include:

Specific responsibilities of assigned readers include:

2.2 Selection committee chair

Selection committee chairs play an important oversight role on the committee and provide guidance to the Vanier-Banting Secretariat on the delivery of the program.

During the selection committee meeting, the committee chair ensures the orderly and complete evaluation of applications and the transmission of recommendations to the Vanier-Banting Secretariat. The evaluation process must ensure that all important aspects of an application are considered in light of the selection criteria. The chair leads the committee's efforts to maintain the highest quality of evaluation and to ensure consistency and equity of approach during the meeting.

The chair will represent the committee in dealings with the Vanier-Banting Secretariat on policy issues. This includes the submission of a report to the Secretariat following the meeting and the presentation of the report to the Tri-agency programs steering committee.

The Secretariat asks selection committee members not to make formal recommendations for a future chair. Suggestions on potential chairs should be given to the Banting Postdoctoral Fellowships program administrator.

2.3 Program administrator

The main responsibilities of the Banting Postdoctoral Fellowships program administrator during the competition include:

The program administrator is not a committee member and does not have voting rights on the committee.

2.4 Executive director, manager and team leader

Communication of secure documents

Throughout the competition cycle, Secretariat staff may need to provide you with documents which contain sensitive information. In most cases, these documents will be shared by posting them on the appropriate pages within ResearchNet – the electronic web portal used for review submission.

3. Review process

The review process consists of the eight following phases, which are further described below:

3.1 Prior to the meeting

3.1.1 Receipt and assignment of applications

Applications are received at the Vanier-Banting Secretariat and the program administrator assigns them to committee members.

Each agency-specific selection committee is a multidisciplinary committee; as such members will be asked to review and score applications in a number of different research areas related to the federal granting agency's mandate, including applications in areas which may not be their own. Each application will be reviewed by two committee members. One member will serve as primary reviewer and the other will serve as secondary reviewer.

Once applications have been received by the Secretariat, the program administrator will assign a subset of applications to each member using ResearchNet – the electronic web portal used to review submission. The program administrator will seek to balance workload, taking into consideration potential conflicts, language capabilities and areas of expertise. Each committee member is responsible for reviewing their assigned applications.

As soon as the assignment of applications is made, each committee member will be granted access to ResearchNet to review their Banting Postdoctoral Fellowships assignments. Members will receive an email notification advising them that their assignments are available.

3.1.2 Identification of conflicts by members

Committee members identify additional conflicts with those applications to which they were assigned.

Using ResearchNet, members are asked to identify potential conflicts in reviewing applications to which they were assigned. The list below is not exhaustive – reviewers should consult the Confidentiality and Conflict of Interest Guidelines that appear as the first mandatory task in ResearchNet. Should a member feel unable, for any reason, to review an application which they accepted as an assignment, they should contact the program administrator immediately and the application will be re-assigned to another reviewer as appropriate.

Are you in conflict with the applicant?

The following guidelines governing conflicts of interest apply to the evaluation of Banting Postdoctoral Fellowships applications. Guidelines of this nature cannot foresee all possible situations, and the federal granting agencies rely on the judgment of committee members when identifying conflicts. As both the applicant and the research environment are evaluated, a member may not act as a reviewer for an application if:

When a committee member is in conflict with a particular application, the member will be asked to leave the room before the deliberations on that application begins.

It is the responsibility of committee members to declare any conflicts of interest prior to the review of an application. In cases where the guidelines do not clearly describe a situation, or where the committee or chair has difficulties making a decision about a particular situation, Vanier-Banting Secretariat staff has the responsibility to rule.

3.1.3 Review and pre-scoring of applications

Following the identification of any conflicts, committee members may begin to review and pre-score their assigned applications. Committee members are to pre-score all applications assigned to them using ResearchNet and submit their pre-scores electronically to the Vanier-Banting Secretariat prior to the selection committee meeting. A specific deadline date will be communicated to committee members in due course.

a) Introduction

Peer review by nature is a subjective process. Bias may manifest in several ways and could be based on a school of thought, fundamental versus applied or translational research, areas of research or approaches (including emerging ones), types of research contributions, size or reputation of a participating institution, age, language, personal factors or gender of the applicant.

Reviewers are cautioned against any judgment of an application based on such factors, and asked to constantly guard against the possibility of implicit bias influencing the decision-making process.

Can we improve gender equity and reduce gender bias in the review process?

The Tri-agencies are actively engaged in increasing gender equity in their review processes to address some of the gender inequities that we have observed from recent funding competitions. In the last 5 years, the success rate for women has been about 3% lower than for men in the Banting PDF program.

Refer to these tips to reinforce standards of fairness, respect and equity.

Is the application eligible for support?

Vanier-Banting Secretariat staff is responsible for screening all applications against eligibility requirements to ensure that the applicant is eligible to apply to the program. Committee members are invited to bring any concerns or questions regarding the eligibility of an applicant to the attention of Secretariat staff, but should proceed with the evaluation of the application while the situation is assessed.

Is the proposed research contrary to the Tri-Agency Framework: Responsible Conduct of Research?

If you judge that the proposed research may be contrary to the Tri-Agency Framework: Responsible Conduct of Research or if you have any ethical concerns with respect to the application, proceed with the evaluation as usual and inform the Secretariat's program administrator of your concerns as soon as possible.

b) Selection criteria

Applicants will be evaluated and selected based on the following three criteria, weighted equally:

  1. Research excellence and leadership in the research domain
  2. Quality of the applicant's proposed research program
  3. Institutional commitment and demonstrated synergy between applicant and institutional strategic priorities

Note: The "Application guide" summarizes the steps to be followed by applicants in completing and submitting their applications, including information on how each application element relates to the selection criteria. Members should read this information carefully and refer to it when evaluating the applications assigned to them.

1. Research excellence and leadership in the research domain – demonstrated capacity for research excellence based on track record as defined by quality of applicant's research contributions, and demonstrated capacity for leadership in the research domain defined by the sphere of influence achieved to date by the applicant.

The candidate's research history and the impact of their activities in their area of expertise to date are important indicators of their potential as research leaders of tomorrow. Reviewers should consider the sphere of influence of candidates relative to others along the following continuum of expanding impact:

In evaluating this criterion, both the nature/prestige of this award and the stage and nature of the applicant's career should be considered.

All indicators of excellence of the candidate must be factored in the evaluation to allow for individual workstyles, contributions and commitments. For example, collaboration, teamwork, mentoring are important and valid contributions to research and training of highly qualified personnel. Each contribution should be valued equally regardless of gender of the candidate. For example, single-authored vs multi-authored publications should be valued similarly regardless of gender.

Evaluate excellence/productivity commensurate with the career stage of the candidate, work-family balance and other special circumstances that can lead to delays in research and dissemination of the results (e.g. career interruptions for child bearing and raising impacting publication records). These personal circumstances must be considered on a case-by case basis; careful consideration must be given to these gender-based factors in the evaluation process.

In the case of health professionals, consideration should be given to standards of research productivity, etc. for their level of experience/qualifications relative to those with a PhD. For applicants who have relevant work experience, scientific productivity prior to graduate school should be considered.

Supporting evidence to be evaluated for this criterion:

2. Quality of applicant's proposed research program – examined in terms of the potential of the proposed research program (taking into consideration feasibility, timelines and novelty of research), executed in the proposed institutional environment, to position the applicant for significant impact through a research-intensive career (potential for significant impact).

The potential contribution of the proposed research to the advancement of knowledge in the field is dependent on the promise and quality of the proposal, the environment in which it will be conducted, and the aspirations of the candidate. Reviewers should consider the potential for significant impact presented by the above-noted combination of factors.

Supporting evidence to be evaluated for this criterion:

3. Institutional commitment and demonstrated synergy between applicant and institutional strategic priorities – examined in terms of:

This should not be an assessment of the institution per se, but rather of its commitment to the applicant, its capacity to enable the applicant to become a future leader in their chosen field and its potential to build upon its strategic priorities.

Supporting evidence to be evaluated for this criterion:

c) Pre-scoring

Members are expected to assign a score between 0.1 and 9.0 (in increments of 0.1 with 9.0 being high and 0.1 being low) to their assigned applications for each of the three selection criteria. ResearchNet will automatically calculate the application's overall score by averaging the scores assigned to each criterion.

In order to encourage members to differentiate between highly promising applicants, and to ensure that the full range of the scale is used, a forced binning system has been implemented. Applications assigned to a member must fall within each of the three following bins according to the proportions indicated in the table below:

New for 2017-18 – change in binning proportions

Funding recommendation Score Proportion
Recommended 7.5 – 9.0 10%
Could be recommended 5.1 – 7.4 50%
Not recommended 0.1 – 5.0 20%

The remaining 20% may be distributed between the three bins as members deem appropriate in order to compensate for a relatively strong or a relatively weak subset of applications. We recommend that members review a number of applications before entering scores as this will provide an adequate frame of reference for members to score all of their assigned applications. An electronic worksheet is provided to members to assist with the task of scoring and binning.

In order for this system to work effectively, it is essential that the entire range within a bin be used. Therefore, every effort should be made to delineate applications within a bin. For example, if three applications are placed in the top bin, then one should be ranked highest and one lowest with the remaining application ranked in between, if appropriate. To the extent possible, there should be no tied scores.

Reviewer pre-scores must be submitted electronically to the Vanier-Banting Secretariat via ResearchNet prior to the selection committee meeting (the exact date to be confirmed by the Secretariat's program administrator). Please respect the date provided, as staff cannot determine which applications will be reviewed at the selection committee meeting until all pre-scores have been received. Members are encouraged to contact Secretariat staff at any time if they need assistance during the pre-scoring process.

3.1.4 Preliminary ranking of applications and assignment of readers

Once all the pre-scores have been received, Vanier-Banting Secretariat staff will calculate the average of the two reviewers' overall scores and rank the applications from highest to lowest. The pre-score ranking of all applications will be made available to committee members prior to the selection committee meeting.

Applications being discussed at the meeting will be assigned to a third member who will serve as reader (the exact number of applications per member will depend on the number of discrepancies). The reader will be responsible for reviewing and assigning scores to the application prior to the meeting and must be ready to provide comments on the application's strengths and weaknesses, including their scores on each criterion during the committee deliberations, paying particular attention to the third selection criterion, Institutional commitment and demonstrated synergy between applicant and institutional strategic priorities.

3.2 During the meeting

3.2.1 Meeting in Ottawa

Note: Members will be required to bring to the meeting their personal notes on assigned applications. All notes must be securely destroyed after the meeting.

Committee members meet in Ottawa for the selection committee meeting.

At the meeting, the 40 top-ranked applications, discrepant applications and any others identified for discussion by committee members (the total number of applications to be discussed will depend on the number of discrepancies) will be reviewed one at a time, in rank order from highest to lowest.

At the start of the review of an application the reader assigned to the application will inform the committee of their scores on each criterion. The primary reviewer will then verbally summarize the strengths and weaknesses of the application by addressing each criterion in turn including points raised in the three referee assessments. The secondary reviewer will provide comments to highlight agreement or differences with the primary reviewer's analysis. Reviewers should summarize the relevant information (positive and negative) in approximately five minutes. The reader will then be given an opportunity to add any other information which they feel is required by the committee in order to assess the application, focusing on the third selection criterion: Institutional commitment and demonstrated synergy between applicant and institutional strategic priorities. Comments or questions may be raised by other members in order to clarify any of the information presented by the reviewers and reader.

Following comments from the assigned reviewers/reader and the committee's discussion, the committee will come to a consensus on the application's final score. It is expected that the selection committee will decide on a consensus score for each of the three selection criterion and these will then be averaged to arrive at a final score for the application. To be considered eligible for funding, applicants must attain an average score of at least 5.1 in each of the three criteria.

Once all applications have been discussed and scored, the ranking of the applications will be reviewed to determine a cutoff for funding and to sort out any ties in scores between the 23/24 ranked applications and those at the funding cutoff. Additionally, all ties involving applicants proposing foreign host institutions ranked above the funding cutoff will be resolved.

The committee's final rating on each criterion for an individual application will be made available to the applicant following the competition. This will provide feedback to the applicant on the relative strengths and weaknesses of their application and its relative ranking. Committee members will not submit written comments on individual applications.

The final ranked list of all applications will be submitted to the Tri-agency programs steering committee for approval.

Any comments related to the selection process should be brought to the attention of the committee's program administrator.

3.2.2 Committee recommendations to the Tri-agency programs steering committee

Each agency-specific selection committee finalizes its ranked list and establishes a quality cutoff for funding.

The final ranked list of all applications identified by each agency-specific committee, including the funding cutoff for alternate awards, will be submitted to the Tri-agency programs steering committee for approval.

Program staff will not go below the funding cutoffs established by the selection committees when offering awards to alternates due to declines.

The Tri-agency programs steering committee's approval will result in the funding of 70 Banting Postdoctoral Fellowship recipients.

3.2.3 Selection committee's policy discussion

At the end of the meeting, once the committee's recommendations have been made, members will be asked to participate in a policy discussion. The discussion generally includes comments on the quality of applications reviewed, possible improvements to the review process and program policies, future membership needs, competition and travel logistics, hotel accommodations, and the Vanier-Banting Secretariat's administrative service etc.

Note: Prior to the meeting members are invited to give some thought to individuals who would be appropriate to serve as future committee members. Any suggestions can be provided to the program administrator.

3.3 Following the meeting

3.3.1 Preparation and submission of the chair's report

Following the meeting and in consultation with committee members and Secretariat staff, the chair will prepare and submit their report.

This report will be presented to the Tri-agency programs steering committee and is a means for communicating the selection committee's comments and recommendations for improvements to the competition processes and program policies.

In addition to the final ranked list of all applications, the report should include relevant comments on such matters as:

4. Policies and guidelines

In addition to the conflict of interest guidelines noted above, please keep in mind the following policies and guidelines when reviewing applications.

4.1 Confidentiality of application material

When logged onto ResearchNet, you will be asked to read and agree to the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers, which describes the expectations and requirements of the Tri-Agencies.

Application material is provided to committee members in confidence and should be used for review purposes only and kept in a secure location. For example, avoid saving confidential documents in shared networks and/or on cloud services. In addition, material that the committee member no longer requires should be destroyed in a secure manner (i.e., shredded). All binders, files, personal notes, assessments, etc. must be left with Vanier-Banting Secretariat staff at the conclusion of the meeting. If the Secretariat requires your assistance to provide additional information on a particular case after the competition, you will be provided with new copies of the relevant material.

4.2 Confidentiality of recommendations

All funding recommendations are subject to approval by the Tri-agency programs steering committee and may be changed for reasons of budget, administrative error, or lack of full compliance to program and agency-specific policies.

All matters discussed during selection committee meetings or teleconferences are confidential. Notifying applicants of the results of committee deliberations is the responsibility of the Vanier-Banting Secretariat, following the official approval by the Tri-agency programs steering committee.

Applications are confidential and must not be discussed with or divulged to others. Any release of information to an applicant must be done through the Secretariat.

Results must not be disclosed by committee members prior to the Secretariat's official release. If approached by an applicant and/or university representatives concerning a decision or any other matter, committee members should decline discussion and refer the individual to the Secretariat. Staff will act as the liaison between the Banting Postdoctoral Fellowships selection committee and the applicant.

4.3 Privacy Act

Canada's Privacy Act stipulates that personal information provided by applicants must be used only for the purpose of assessing applications and making funding decisions. The use or disclosure of such information for any other purpose is illegal.

The information gathered for this purpose must be collected directly from the individual. It may be collected from other sources only as part of the formal review process. For this reason, Banting Postdoctoral Fellowships selection committee members must not use or consider information about an applicant that has been obtained independently (e.g., a member by virtue of their involvement in other activities).

In view of the large number of applications, selection committee members will not be required to provide written assessments of individual applications. Applicants have a right to access information about their application that is held by the Vanier-Banting Secretariat; however, they do not have the right to access the names of persons who reviewed their application.

Selection committee membership will be published on this website approximately 60 days after funding decisions have been announced.

It is important that committee members adhere strictly to the guidelines set out in the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations.

4.4 Canadian Human Rights Act

The activities of federal granting agencies are subject to the Canadian Human Rights Act. The purpose of the Act is to give effect to the principle that every individual should have equal opportunity with other individuals to make the life that she or he is able and wishes to have, consistent with the duties and obligations of a member of society, without being hindered or prevented from doing so by discriminatory practices.

For all purposes of the Act, it is discriminatory practice to deny a service to an individual or to differentiate adversely in relation to any individual in the provision of that service based on that individual's race, national or ethnic origin, colour, religion, age, sex, marital status, family status, disability, or conviction for an offence for which a pardon has been granted. Where the ground of discrimination is pregnancy or childbirth, the discrimination is deemed to be on the ground of sex.

4.5 Official languages

The federal granting agencies, like all other federal institutions, have a key role to play in the implementation of the Official Languages Act. The federal granting agencies have an obligation to ensure that:

The federal granting agencies must ensure that their committees/panels and their staff are fully aware of their obligations and rights regarding official languages by providing documentation on official languages to employees and members and by including relevant guidelines in the instructions to selection committees.

In accordance with an active offer of bilingual service to the public, the Vanier-Banting Secretariat will try to appoint as many experts as possible with the appropriate language capabilities to serve on selection committees.

Selection committees must ensure that all applications receive a full evaluation (subject to the guidelines regarding level of effort for lower quality applications as noted above), regardless of the official language of presentation. On occasion, this may entail consultation with Secretariat staff to identify Banting Postdoctoral Fellowships selection committee members with adequate linguistic capability.

4.6 Ethical considerations

Committee members play an important role in alerting staff to any concerns they may have with respect to the ethics of the research being proposed. Examples of problems include the following:

If a member has any concerns with respect to ethical matters, these should be discussed immediately with Vanier-Banting Secretariat staff to determine if there is a means of resolving any apparent problems quickly or if the award of a fellowship should be delayed pending the resolution of the problem. The review of applications should continue without being influenced by this issue.

4.7 Responsible conduct of research

The three federal granting agencies have defined their expectations with respect to scientific integrity in the Tri-Agency Framework: Responsible Conduct of Research.

Research

The federal granting agencies expect the highest standard of integrity in the research they fund. The electronic submission of an application to the Vanier-Banting Secretariat commits the applicant to a number of principles, including compliance with the integrity policy. Should members identify during the evaluation process what appears to be a lack of integrity, they should bring it to the attention of Secretariat staff at the earliest opportunity. The Vanier-Banting Secretariat will refer any allegations to the Secretariat for the Responsible Conduct of Research for investigation. Such allegations should not be a consideration during the review process, nor should they be part of the committee's evaluation discussions.

A member may, during the evaluation process, encounter possible misconduct situations (e.g., apparent misrepresentation of publications and/or data, plagiarism and other problems such as a lack of appropriate control/monitoring within the university itself or undue restriction on the dissemination of research supported by federal funds). Members should alert the program administrator of these situations at the earliest opportunity.

Selection and review process

The Vanier-Banting Secretariat expects the highest standards of integrity in the review process that it manages. The Tri-Agency Framework: Responsible Conduct of Research indicates that information provided by applicants for the purpose of selection and review cannot be used by reviewers without the author's permission. In addition, a reviewer should reveal to the federal granting agencies any material conflict of interest, financial or other, that might influence the Secretariat's decision on whether the reviewer should review applications. Committee members are responsible for respecting the confidentiality of application material and for declaring material conflicts of interest. Should committee members become aware of a situation that violates the integrity of the review process, they should discuss this immediately with Secretariat staff.

5. Committee membership

5.1 Recruitment process

The federal granting agencies regularly solicit nominations for its selection committees from universities, industry and/or the government sector. Nominations are also put forward by previous committee members or by Vanier-Banting Secretariat staff.

The slate of nominees for each committee is prepared by program staff and is subject to final approval by Vanier-Banting Secretariat management. Chosen in a similar way, the committee chair is usually a member who has already served on a committee for one or more competitions.

5.2 Member selection criteria

The most important criterion governing committee membership is academic and research excellence. The value of the review process rests on the credibility of committee members and their recognized expertise and productivity in their field(s). Although each committee is representative of the community it serves, members themselves are not expected to act as representatives of any particular group, institution, region or country. Committees are structured to ensure:

5.3 Term of membership

The renewal of membership is essential to ensure the vitality of the selection committee process. Rotation of membership allows for broader representation of universities and complementary expertise. However, continuity is also desirable, since the presence of experienced members on a committee promotes consistency in the selection process and assists in the orientation of new members.

Selection committee members usually serve no more than three consecutive years. The extension of a term beyond three years for one additional year may be accepted in exceptional circumstances. Members who have completed their three-year term on the committee may be appointed as chair for an additional two years.

Under normal circumstances, approximately one-third of committee members retire each year.

Date modified: