F.R.S.-FNRS expert database: facts and figures
Analysis, evaluation and foresight unit, F.R.S.-FNRS
25 avril 2024,10:13
1 Introduction
1.1 Objective of the analysis
For more than ten years, the Fund has maintained an expert database to carry out its mission of evaluation. The present report is intended for research candidates and experts who wish to receive more information about this valuable tool, which will be presented here through various analyses and key figures.
1.2 The evaluation procedure
Since 2010 and the reform of its evaluation procedures1, the F.R.S.-FNRS has acquired an internationally recognized expertise in peer reviewing. Evaluation at F.R.S.-FNRS meets international standards and is based on the fundamental principles of excellence, transparency, confidentiality, and ethics in research.
The Fund organizes each year several calls for projects covering all scientific fields (Exact and Natural Sciences, Social Sciences and Humanities, Life and Health Sciences). Most of these calls are based on a bottom-up approach, inviting researchers to submit their research proposals without any particular thematic orientation. With the support of specialized associated Funds, other calls are structured according to a top-down approach and concern more specific research themes, linked to societal needs or explicit prospects for potential innovations.
The Fund organizes evaluation procedures that enable to identify the best applications among those submitted, according to pre-established criteria (known by the applicants) for each type of funding instrument. As shown in Figure 1.1, the evaluation generally takes place in two stages. First, applications are evaluated remotely by international experts (except for doctoral level instruments) who submit their evaluation via the F.R.S.-FNRS e-space web platform. Then, the scientific Commissions or Juries that meet at the F.R.S.-FNRS headquarters in Brussels establish a final ranking of the submitted funding applications, based particularly on the collected remote evaluations. Finally, the Board of Trustees of the F.R.S.-FNRS takes the final decisions concerning the grants by relying on the ranking given by the scientific Commissions or Juries and given the budgetary constraints. For the sake of transparency, all the evaluation reports are sent to the candidates at the end of the process. The names of the remote experts involved in the evaluation procedure remain however unknown to the candidates, and only the final grade awarded by the Commission is communicated to their attention.
Figure 1.1: The step by step evaluation procedure considered by the Fund
1.3 Identification and recruitment of remote experts: Principles
The Fund’s Analysis, Evaluation & Foresight unit is responsible for identifying and recruiting remote experts. The objective is to ensure that each funding application can be evaluated by an adequate number of legitimate experts. To this end, the unit has built up a database of experts since 2010 which includes more than 19.000 experts from all over the world. The F.R.S.-FNRS is committed to continually expand this database to cover a wide range of research topics and thus facilitate the selection of specialized experts for the assessment of the applications submitted to the Fund. It is also important to ensure a certain turnover among the remote experts involved at the first stage of the evaluation. The reviewers are guided in the different steps related to the registration on the e-space platform and the evaluation of proposals for the Fund, for which they receive a remuneration (for most calls). Satisfaction surveys are carried out on a regular basis among remote reviewers, enabling the Fund to assess their level of satisfaction with its evaluation procedures, and to propose improvements where necessary.
All remote experts as well as members of scientific Commissions or juries are registered in the expert database.
Experts’ selection is achieved by scientific officers of the Fund who process applications one by one and draw up a list of experts legitimate to provide an assessment, and whose profile matches with the expertise required for the evaluation of the application. For each scientific Commission, the President is in charge of validating the list of remote experts to whom the administrative staff of the F.R.S.-FNRS has assigned the submitted application files (fellowships and projects). The administration is then responsible (i) for checking that there are no conflicts of interest for the experts identified and (ii) for inviting them to carry out the assessment, in the order of priority established by the scientific officers.
Overall, the legitimacy of an expert to evaluate a funding application is carefully assessed with regard to his/her scientific experience and expertise and according to his/her ability to evaluate the application objectively and impartially (through the absence of conflicts of interest).
Due to the tight schedule constraints generally faced during calls, the choice of an expert is also influenced by secondary factors such as availability, which is closely monitored through a personal acceptation rate updated call by call, thus allowing to identify experts who are more likely to respond positively.
The database of experts is thus evolving dynamically through regular data updates, but not only. Indeed, new experts register annually in the database, either voluntary or following their new involvement in the evaluation procedures organized by the Fund, resulting in an annual increase of the number of created accounts. At the suggestion of Presidents or members of Scientific Commissions, some researchers are also invited to register in the F.R.S.-FNRS database. Finally, the Fund is particularly sensitive to privacy questions and is committed in this respect to respond to requests for deletion or withdrawal that are addressed by registered experts, free to submit such a request to the Administration at any time.
2 Methodology & Preliminary remarks
2.1 Data & Software
The present analysis is based on data collected from e-space, and more specifically its expert database. E-space is the F.R.S.-FNRS platform which was designed by the Fund to enter, store, process and share the information within the organisation and with the applicants (i.e., researchers) and stakeholders of the evaluation (experts, members of Commissions, research organisations and agencies). In particular, it is via this platform that evaluators create and connect to their F.R.S.-FNRS account and interact with the administration of the Fund regarding the assessments they are asked to provide.
As soon as an expert agrees to take on the role of evaluator (e.g., remote reviewer, member of Commission/Jury) for the first time, he or she is invited to create an account via the e-space platform, and to fill in a form enabling him or her to provide the information presented below.
INFORMATION AVAILABLE IN A RECORD OF THE F.R.S.-FNRS EXPERT DATABASE
Data followed by an asterisk* are mandatory.
Basic user details
- Last name*, Middle name, First name*
- ORCID iD
- Birthdate
- Gender (female/male)*
- Language in which the expert can perform the evaluation (French and/or English)*
- Awarding year of PhD thesis
Positions
- Country of affiliation
- Affiliation
- Position title
- Permanent position (yes/no)
Contacts
- Professional phone number
- Mobile professional phone number
- Private phone number
- Private mobile phone number
- Links to any personal or professional websites
Research fields
- Main research field (Exact and Natural Sciences, Social Sciences and Humanities, Life and Health Sciences)
- Expertise fields (description of the research interests/thematics)
- Selection of 1 to 6 descriptors2 that best describe the expert’s skills
- Free keywords
- Curriculum vitae (to attach)
Preferences
- Number of applications the expert wishes to evaluate on average during the calls for proposals that the Fund organises (about 2 major calls for proposals per year)
- Willingness to carry out unpaid evaluations (yes/no)
- Possible periods during which the expert is not available to carry out evaluations for the Fund
Further details
- Interdisciplinary approach in the scientific research conducted by the expert (yes/no)
- Details about outstanding professional achievement
- Expert’s peer-review experience (0 - none, 1 - evaluation of publications and/or internal projects, 2 - calls for proposals by agencies, 3 - calls for proposals by high-level agencies)
The first and last names, gender3, as well as the language for evaluation are the only information that experts are required to provide when they create their account. As a result, many experts do not complete their data (including details about their scientific expertise) or do so only partially due to lack of time, which results in a lot of missing data. That is why, for each call, the scientific officers of the F.R.S.-FNRS check new experts’ records and do their utmost to complete any missing data. They also update records with any new information that could concern experts already registered in the database (e.g., affiliation change, career development). Indeed, having a database that is as complete and up-to-date as possible is the goal to which the Fund aspires. This work is of particular importance, especially as it helps to refine the results of queries made in the expert database as part of the process of identifying potential evaluators for applications submitted during subsequent calls.
Despite the efforts made to update the F.R.S.-FNRS expert database, please note that data are certainly missing for some experts, for human or technical reasons. The figures provided in this report are therefore approximate.
2.2 Inclusion criteria
Only experts with active accounts are taken into account in this analysis: they are currently 19325 in the F.R.S.-FNRS database. Indeed, experts who no longer wish to act as evaluators for the F.R.S.-FNRS may, at any time, expressly ask not to be contacted for future calls for proposals. In this case, the accounts concerned no longer appear in search results as part of the expert selection process. An update is also required in the event of the notified death of an expert. Lastly, the scientific Commission may recommend that the Fund no longer invite certain reviewers because of the poor quality of the evaluations provided.
As the reader will notice later in this report, the F.R.S.-FNRS expert database is far from being fully exploited. Indeed, it includes 19325 active accounts for experts who are not all necessarily invited to act as evaluators, for several reasons. First, as registration may be achieved voluntarily and without approval procedures, a certain proportion of experts do not necessarily have the legitimacy to act as evaluators in the sense of the F.R.S.-FNRS rules (see Section 1.3). Secondly, some experts end up not accepting evaluation tasks after a first or several participations to the F.R.S.-FNRS evaluation processes, mainly due to unavailability. Furthermore, contact with some experts is sometimes lost following a change of institution and/or e-mail address that is not notified via an update to the expert record. At last, the needs in terms of research topics, the number of applications to be assessed during a call, and the schedule constraints also significantly influence the extent to which the expert database is used.
The scientific officers of the F.R.S.-FNRS regularly look for experts outside the F.R.S.-FNRS database during a given call. This helps to overcome the challenges mentioned above (i.e., finding more available, legitimate, and specialized experts), and to ensure a turnover among evaluators. In addition, the Funds pays particular attention to gender balance in the selection of reviewers. For some counts and graphs, the present report will make a distinction between experts registered in the database prior to a given year (former experts of the database) and those registered during the year in question (new experts of the database for the considered year).
An expert registered in the F.R.S.-FNRS database may take part in the evaluation as a remote expert (reviewer) and/or as a member of Commission or Jury. The present report will focus on remote experts, who will also be referred to interchangeably as international reviewers, remote reviewers, international evaluators or remote evaluators.
3 Analyses & Results
3.1 Summary statistics
As shown by Table 3.1, most of the experts are affiliated to institutions located outside the French-speaking community of Belgium, which is quite expected given that remote evaluations are mostly carried out by experts affiliated outside this area in order to limit conflicts of interests as far as possible.
| Affiliation | Total |
|---|---|
| Local | 2281 (11.8%) |
| International (outside the French-speaking community of Belgium) | 16099 (83.3%) |
| Missing data | 945 (4.9%) |
The database brings together evaluators providing expertise in the three main research domains, as shown by Table 3.2.
| Main research field | Total |
|---|---|
| Exact and Natural Sciences | 6183 (32.0%) |
| Life and Health Sciences | 4443 (23.0%) |
| Social Sciences and Humanities | 7649 (39.6%) |
| Missing data | 1050 (5.4%) |
| Gender | Total |
|---|---|
| Female | 5482 (28.4%) |
| Male | 13707 (70.9%) |
| Missing data | 136 (0.7%) |
Gender has been closely monitored by the F.R.S.-FNRS over the last years. In other respects, the Fund pursues its efforts to increase the proportion of female experts among its various evaluation panels and remote evaluators. This objective has been achieved concerning international scientific Commissions, as 54.7% of members involved in the Grants & Fellowships 2023 call were women. Regarding remote evaluations, the scientific officers of the F.R.S.-FNRS are particularly sensitive to this issue and do their utmost to increase the proportion of women available to act as remote evaluators. In this respect, it is important to remember that, since the selected experts are academic members from universities located in Flanders or internationally, the gender distribution of remote experts solicited by the F.R.S.-FNRS also indirectly reflects the proportion of women among academics in universities from different countries. Finally, in its gender equality plan (2022-2025), the F.R.S.-FNRS has formally undertaken the inclusion of female experts by encouraging their registration in its database5, through the explicit mention of the objective to recruit more women experts in its website. At present, 28.4% of the active F.R.S.-FNRS experts are women (see Table 3.3).
Figure 3.1: Annual evolution of the number of accounts over the whole database (purple), and concerning female experts more particularly (green)
Figure 3.1 shows the annual evolution of experts registered in the F.R.S.-FNRS database. Note that data on the year of creation is missing for 358 accounts, which could not therefore be considered in these counts. The number of F.R.S.-FNRS experts has significantly increased since 2010 and has almost been multiplied by 7. Meanwhile, the number of female experts has multiplied by almost 9. Figure 3.2 allows to look further at the increase in female experts over the whole database, and according to the main research domain.
Figure 3.2: Evolution of the proportion of female experts over the years. This proportion has been computed over the whole database (purple) and per main research field, when this data is available : Exact and Natural Sciences (blue), Life and Health Sciences (red), Social Sciences and Humanities (green). The total proportion of female accounts was computed over the accounts for which the year of creation is known, whether data on gender is available or not.
Fig. 3.3 shows the map of affiliation countries of the experts registered in the F.R.S.-FNRS database. The expert database now brings together a wide variety of institutions worldwide. Table 3.4 lists the most frequent countries of affiliation at the present time.
Figure 3.3: Evolutionary map of the affiliation countries of the experts registered in the F.R.S.-FNRS database (in absolute number)
| Affiliation country | Percentage of experts |
|---|---|
| France | 27.1 |
| Belgium | 15.5 |
| United States | 10.9 |
| United Kingdom | 8.7 |
| Italy | 4.9 |
| Canada | 4.8 |
| Germany | 4.7 |
| Spain | 3.2 |
| Switzerland | 3.0 |
| Netherlands | 2.6 |
3.2 Annual use of the expert database
As mentioned in Section 2.2, all experts of the database are not annually invited to provide a review for the Fund. Figure 3.4 shows the annual ratio of the number of experts invited to perform one remote evaluation (at least) to the total number of experts who were included in the database in a given year, including those who had just registered during the year in question.
Figure 3.4: As a percentage of the experts registered in the F.R.S.-FNRS expert database, proportion of those who were invited to perform an evaluation (2020-2023). It should be remembered that not all the experts who are invited agree to carry out the evaluation assignment.
From 2019 to 2023, on average 33.4% of the experts registered in the F.R.S.-FNRS expert database were invited each year to carry out an evaluation.
The overall use of the expert database seems to be decreasing over the years. This is mainly explained by the fact that the number of registered experts has significantly increased over the years (see Figure 3.1), while the number of assessments required has increased within a narrower range.
Figure 3.5: Of all experts registered in the F.R.S.-FNRS expert database prior to the year in question (former experts), proportion of those who were invited to perform an evaluation (purple) and those who accepted and finished one evaluation at least (green) over the last four years (2020-2023)
Many experts are invited through personalized emails to register in the F.R.S.-FNRS expert database call after call. In Figure 3.5, the proportions relate to the experts who were already registered in the database prior to the year of the evaluation in question. From 2020 to 2023, on average 27.4% of those experts were invited to carry out an evaluation (at least), and 15.7% of them accepted to undertake and completed the task. In other words, this means that among the experts already registered in the database before a given evaluation year, 27.4% are invited to take on a remote evaluation (at least) during the year in question.
Figure 3.6: Proportion of experts who performed at least one remote evaluation for a given year, aggregated by the number of evaluations (1, 2, 3, 4, or more than 5) performed. How to read this graph: 66.8% of experts who evaluated remotely applications submitted in 2019 performed only one review, while 19.1% of them performed two reviews and 3.4% were responsible for more than five evaluations.
Figure 3.6 shows the distribution of all the experts who acted as evaluators for the F.R.S.-FNRS for a given year, according to the number of evaluations carried out. As indicated by the results, some experts agree to evaluate more than one funding application, with a small proportion evaluating more than five proposals. On average, between 2019 and 2023, 69.3% and 17.9% of reviewers assessed respectively one and two funding applications, while 2.9% of them assessed at least 5 applications.
To sum up, despite the high number of 19260 experts registered in the F.R.S.-FNRS expert database, a limited number of them are contributing to the evaluation processes organized by the Fund each year. Some of them did not take part to remote evaluations for years, either because they voluntarily refused the task(s) or because they have not been invited to perform evaluations.
3.3 Are evaluations always performed by a sub-group of experts?
As suggested in the previous graph (see Figure 3.6), a certain number of experts carry out more than one evaluation for the F.R.S.-FNRS per year. The F.R.S.-FNRS can also count on the loyalty of many experts who continue to accept evaluations throughout the calls and the years.
Drawing on this observation, the question of how evaluations are concentrated among evaluators arises. In other words, what proportion of remote reviewers are involved in a given proportion of evaluations? Are evaluations always performed by a restricted sub-group of experts? The cumulative graph of Figure 3.7 shows the distribution of the evaluations carried out in relation to the proportion of experts who performed those remote reviews (over the last five years, all calls combined). The results have been broken down by major scientific domain.
Figure 3.7: Cumulative chart of the proportion of evaluations in relation to the proportion of experts who performed those over the last five years (2019-2023), per main research domain. How to read this graph: 83.8% of the evaluations of the LHS applications were carried out by 50% of all the experts who performed at least one evaluation of such applications.
Overall, the graph shows a certain concentration of assessments on a restricted group of reviewers, thus attesting to the fact that a subgroup of experts is responsible for most reviews: between 75% and 85% of the evaluations were provided by 50% of the experts who performed at least one evaluation between 2019 and 2023.
The results also show that Life and Health Sciences (LHS) experts have a slight tendency to carry out more evaluations per reviewer. Indeed, 83.8% of the evaluations of the LHS applications were performed by 50% of the evaluators who provided at least one remote review in this domain (see Figure 3.7, point A). On the other hand, the same proportion of evaluators who assessed Exact and Natural Sciences (ENS) and Social Sciences and Humanities (SSH) applications carried out respectively 80.5% and 76.4% of the evaluations made in these domains, during the same period (see Figure 3.7, points B and C).
Considering now the cumulative percentage of evaluators against the whole F.R.S.-FNRS expert database (all major research domains combined), the cumulative chart of the percentage of evaluations carried out over the last five years is given under Figure 3.8.
Figure 3.8: Cumulative chart of the proportion of evaluations in relation to the proportion of experts of the F.R.S.-FNRS database (registered by 2023) who performed those over the last five years (2019-2023). How to read this graph: 48.2% of all the experts registered to the F.R.S.-FNRS database by 2023 have evaluated at least one application over the last five years.
These results suggest that 48.2% of all the experts registered to the F.R.S.-FNRS database in 2023 have evaluated applications over the last five years (see point X, Figure 3.8). Among them, 10% carried out 58.2% of the evaluations made during this same period (see point Y, Figure 3.8).
Figure 3.9: Evolution of the proportion of evaluators carrying out evaluations for the F.R.S.-FNRS after the year of their first participation (denoted here as y0), aggregated by the number of evaluations (1, 2, 3, 4, or more than 5) performed. The evolution is presented over a 4-year time horizon. Only experts whose accounts were created between 2015 and 2019 are considered here; the figures were broken down according to the number of evaluations carried out.
Figure 3.9 shows the evolution of the contribution of the F.R.S.-FNRS experts to its evaluation procedures during the four years after their first participation as reviewers. As shown by the graph, one year after their first evaluation, 33.5% of the reviewers carry out assessment(s) for the agency. After 4 years, this proportion falls to 22.7%. It is worth noting, however, that a relatively constant proportion (on average, 4.1%) keeps providing more than five assessments one year after their first participation.
Note that the decrease observed here does not allow any conclusion to be drawn as to a refusal on the part of reviewers to carry out evaluations in the years following their first participation. Indeed, reviewers who carried out at least one evaluation in a given year are not necessarily invited to carry out assessments in subsequent years. It is possible that, in the meantime, none of the applications matches their expertise, or simply that a sufficient number of other reviewers have agreed to undertake the mission. However, it is also possible that experts are not selected because of outdated, missing and/or incomplete data in their records, making it more difficult to identify them in subsequent calls. That is why the Fund’s Analysis, Evaluation & Foresight unit aims, among its various objectives, to better maintain its expert database by regularly updating the experts’ profiles.
It is nevertheless important to stress that not all experts who agree to review a funding application are willing to continue to collaborate with the F.R.S.-FNRS. Indeed, some experts remain active for a few years but end up stopping to perform reviews for the Fund for personal and/or professional reasons. A major challenge for the Fund consists of retaining the collaboration of its former experts by listening to their needs and maintaining their satisfaction6.
Reviewer fatigue occurs when experts in a field are repeatedly called upon to evaluate research, both in their own country and internationally. This adds to the list of tasks that fall to researchers, which also includes the evaluation of scientific publications, editorial tasks and supervision. This may result in a reduction in the quality of the assessments provided (i.e., due to lack of time, assessments may be sloppier, shorter and less exhaustive) as well as a reduction in the reviewers' availability. Concerned by this issue, the Fund asked its remote experts to which extent they were affected by this phenomenon (independently of the COVID pandemic) in a survey conducted among them in 2020. In total, 54.4% of the respondents feel somewhat concerned by it, while 12.4% feel very concerned, and 33.2% do not feel concerned at all.
3.4 Experts’ long-term collaboration with the Fund
Figure 3.10 shows the distribution of the F.R.S.-FNRS experts who performed at least one evaluation over the last five years according to the year in which their account was created.
Figure 3.10: Distribution of the remote reviewers, for each evaluation year, according to the year their account was created (see colors in legend)
As mentioned before (see Section 1.2), the F.R.S.-FNRS regularly looks for experts outside the expert database, particularly in the context of the two F.R.S.-FNRS major calls7. This ensures a turnover among remote reviewers and avoids projects being systematically evaluated by the same pool of evaluators, as well as an over-solicitation of the experts in the Fund database. The effects of this approach can be seen in Figure 3.10, as a certain proportion of experts carry out their first evaluation for the F.R.S.-FNRS every year. This is the case for an average of 25.2% of the evaluators who provided at least one assessment for the Fund per year. Note that a slightly higher proportion of new experts were called in 2021 due to the occurrence of non-annual calls (such as the EOS - Excellence Of Science - call).
Although new experts are recruited every year, a number of former experts continue to act as remote reviewers for the Fund, including some who created their accounts at the creation of the F.R.S.-FNRS expert database (in 2010). These experts make a valuable contribution to the evaluation procedures of the Fund and have acquired a fairly broad overview of the applications submitted to it over the years, as well as an in-depth knowledge of the research groups of the French-speaking community of Belgium.
Finally, it should be noted that for some (but a very limited number of) accounts, creation year data are subject to inaccuracies, in particular due to accounts having been created by the same experts using different email addresses. In this case, accounts are merged, which may lead to record an inaccurate creation year. It also happens that for certain calls opened at the end of the year, experts are selected at the beginning of the following year, in which case their evaluation is recorded prior to the year in which their account was created.
3.5 Affiliation countries of the evaluators
The following tables show the affiliation countries of the experts who performed at least one evaluation for the F.R.S.-FNRS over the last five years (since 2019). We present here the top 15 countries contributing the most to the evaluation.
| Affiliation country | Percentage of evaluations |
|---|---|
| France | 30.9 |
| United States | 11.4 |
| United Kingdom | 9.4 |
| Italy | 7.3 |
| Germany | 5.0 |
| Spain | 5.0 |
| Canada | 4.7 |
| Belgium | 3.3 |
| Switzerland | 3.1 |
| Netherlands | 2.7 |
| Australia | 2.2 |
| Portugal | 1.8 |
| Sweden | 1.4 |
| Finland | 1.0 |
| Austria | 0.9 |
Over the last five years, most of the assessments were carried out by experts affiliated with an institution located in France, the United States, the United Kingdom and Italy (see Table 3.5).
As the results show, the top three positions in the ranking are therefore shared by France (30.9% of the evaluations carried out), followed by two English-speaking countries (20.8% of the evaluations carried out altogether).
| Affiliation country | Percentage of evaluations |
|---|---|
| France | 22.5 |
| United States | 11.6 |
| Italy | 9.8 |
| United Kingdom | 8.8 |
| Spain | 7.3 |
| Germany | 6.2 |
| Canada | 3.8 |
| Australia | 2.8 |
| Netherlands | 2.5 |
| Portugal | 2.2 |
| Switzerland | 2.0 |
| Belgium | 1.9 |
| Sweden | 1.8 |
| Austria | 1.2 |
| Denmark | 1.2 |
| Affiliation country | Percentage of evaluations |
|---|---|
| France | 29.5 |
| United States | 15.5 |
| United Kingdom | 7.4 |
| Italy | 6.5 |
| Germany | 5.8 |
| Spain | 5.0 |
| Canada | 4.5 |
| Belgium | 4.1 |
| Switzerland | 2.9 |
| Netherlands | 2.6 |
| Portugal | 1.9 |
| Finland | 1.6 |
| Australia | 1.5 |
| Japan | 1.4 |
| Sweden | 1.4 |
| Affiliation country | Percentage of evaluations |
|---|---|
| France | 42.4 |
| United Kingdom | 11.3 |
| United States | 8.0 |
| Canada | 6.1 |
| Italy | 5.5 |
| Switzerland | 4.5 |
| Belgium | 4.0 |
| Netherlands | 2.8 |
| Germany | 2.5 |
| Spain | 2.3 |
| Australia | 1.8 |
| Austria | 0.8 |
| Sweden | 0.8 |
| Denmark | 0.7 |
| Greece | 0.7 |
Note that applicants are free to submit their applications to the Fund in either French or English. The F.R.S.-FNRS however reserves the right to request an English translation from applicants for files submitted to the ENS and LHS Commissions, as well as one of the five SSH Commissions (“Sciences Humaines et Sociales-2”). This has unavoidably an influence on the selection of experts, as it means that French-speaking experts have notably to be called in for applications written in French. If the proportion of evaluators from France is the highest in all scientific domains (see Tables 3.6 to 3.8), it is particularly in the SSH domain that they have contributed most to the evaluation process over the last five years (42.4% of the evaluations made, see Table 3.8).
That being said, for each funding application, the scientific officers in charge of experts’ selection pay a great deal of attention to diversity among the experts that are invited to review a given application. More precisely, the list of reviewers likely to assess an application is drawn up to include experts from different institutions or different countries.
3.6 Do reviewers often evaluate applications submitted by the same promoter?
The results presented below show that a large number of experts act as evaluators for the Fund for more than one evaluation and for more than a call over the years.
As indicated in section 2.2, the scientific officers of the Fund do their utmost to ensure a turnover among the evaluators. Around 25.2% of the evaluators per year make their contribution for the first time (see Section 3.4). Such a turnover is all the more important as on the one hand, some applications may be resubmitted, and promoters are free to submit as many applications as they wish (while complying with the F.R.S.-FNRS eligibility and accumulation rules for each call). On the other hand, while it is important to vary the points of view, continuity of evaluation is an objective to be kept in mind; a compromise must therefore be reached.
It is thus here legitimate to ask about the diversity of the funding applications evaluated by the experts. More particularly, do they often evaluate applications submitted by the same promoters? To answer this question, we computed an index which corresponds to the ratio of the number of distinct promoters evaluated by an expert to the total number of evaluations carried out by that expert over the last five years, in the frame of the two F.R.S.-FNRS major calls (Grants & Fellowships, Credits & Projects). This index therefore tends towards unity when the expert is asked to evaluate applications submitted by different promoters. The index strictly equals one when the expert has evaluated applications from different promoters or has only evaluated one application up to now. Figure 3.11 shows the distribution of the computed indices (logarithmic scale).
Figure 3.11: Distribution of the index computed as the ratio of the number of distinct promoters evaluated by an expert to the total number of evaluations carried out by that expert. This index allows to assess the diversity of the applications reviewed by the evaluators. It is computed for evaluations made over the last five years (2019-2023), and for the two major F.R.S.-FNRS calls (Grants & Fellowships and Credits & Projects). If the index equals one, this means that an expert never reviewed twice proposals from the same promotor. An expert has an index equal to 0,5 if the number of promoters whose applications he or she assessed represent half of the number of evaluations he or she provided in total. How to read this diagram: 87 experts have an index included between 0.35 and 0.40, while 6092 experts have an index included between 0.95 and 1.00.
The results show that:
- 69.6% of the evaluators have an index equal to one, which means that they never evaluated twice the same promoter. Among them, 73.7% have assessed one application, while 26.3% have assessed more.
- 9.1% of the evaluators have an index equal to 0.5, which means that the number of promoters whose applications they assessed represent half of the number of evaluations they provided. Among them, 69% provided two evaluations, which thus were submitted by the same promoter.
- 13% of the evaluators have an overall index included between 0.6 and 0.8.
- Only 4.9% of the evaluators have an index inferior to 0.5.
In sum, although a few experts seem to evaluate mostly funding applications from the same promoter, this is not the case for all experts who acted as reviewers for the last years. This is quite expected given that on the one hand, experts are usually selected for their in-depth expertise in relation to the applications proposed to them. On the other hand, the Fund tends to, as explained before, make a compromise between making sure that promoters are not always evaluated by the same experts, while ensuring a certain continuity in evaluation.
4 Discussion & Conclusion
Since 2010, the F.R.S.-FNRS has set up a database of experts which has grown steadily since its creation. The aim of this report was to provide a context and figures for the use of the Fund expert database.
Overall, the results show a good distribution of experts by major scientific domain. Despite the increase in the proportion of women among the experts registered in the database, they are still in minority, representing a total proportion of 27.8%. The Fund continues its efforts to increase the proportion of women in the selection of evaluators, and also highly encourages female experts to register in its database. Finally, in the space of more than ten years, the database has become significantly more international with a greater diversity in the affiliation countries of the experts.
Though the database includes a large number of experts (19260 to date), only a limited proportion of them contribute concretely to the evaluation processes organized recently by the Fund. In this respect, the main results reveal that:
- 48.2% of the whole F.R.S.-FNRS experts have contributed to the Fund evaluation procedures over the last five years,
- 10% of the whole F.R.S.-FNRS experts have contributed to 58.2% of the evaluations made during this period,
- in average, 27.4% of the F.R.S.-FNRS experts have been invited annually to perform at least one evaluation (from 2020 to 2023), while 15.7% of them accepted to do so. This proportion only relates to former experts, i.e., those already registered in the database before the considered year of the evaluation,
- in average 25.2% of the experts providing assessments annually over the last five years were new experts, i.e., experts who were not initially registered in the F.R.S.-FNRS expert database,
- one year after their first evaluation, 33.5% of the experts carry out one or several new one(s) for the F.R.S.-FNRS. After four years, this proportion falls to 22.7%, which suggests that collaborations are not necessarily kept with all the evaluators in the long term,
- despite this, other experts keep evaluating funding applications more than ten years after their first registration in the database.
The Fund can count on a group of loyal experts who accept to undertake the role of reviewer through the calls and the years. As part of its commitment to organizing high-quality evaluation procedures, the Fund ensures that the same promoters are not systematically evaluated by the same experts. The present analysis shows that this approach is relatively successful.
The F.R.S.-FNRS sincerely thanks the more than 19.000 experts registered in its database for their precious collaboration over the last 13 years. Whether they evaluated only one project or annually keep accepting evaluation tasks, their work contributed to the quest for excellence that the Fund pursues.
A few challenges are ahead the Fund for the management of the database. The F.R.S.-FNRS is indeed dedicated to maintaining an up-to-date database that keeps growing every year, to keep recruiting new experts while staying in touch and maintaining nice relations with all the experts already registered to its database, some of whom have been collaborating with the Fund for more than ten years. The Fund pays particular attention to diversity among the experts recruited (notably in terms of scientific seniority, gender and institutions).
Another major challenge for the Fund is to ensure that the evaluators’ tasks are as simple and time-efficient as possible. This involves in particular: (i) ensuring the continuous improvement of the F.R.S.-FNRS e-space platform and its ergonomics, to make it easier to perform tasks such as creating an account, navigating and accessing documents, (ii) ensuring fair and prompt remuneration, (iii) providing personalized support from the F.R.S.-FNRS administrative staff (by aiming for short response times), (iv) providing clear guides and instructions for evaluators and (v) conducting regular satisfaction surveys.
Note at the destination of experts: your help is precious to us!
The selection of experts is a process that relies heavily on the data available in our records. Keeping an up-to-date database is a daily challenge for the F.R.S.-FNRS. We warmly invite you to update your data regularly. For any questions or remarks, do not hesitate to contact us, we would be happy to guide you through any steps (evaluation@frs-fnrs.be). Please also note that the F.R.S.-FNRS has plans to connect its e-space platform to ORCiD, which will make data updating an easier task for you in the near future.
For more information, consult the dedicated page on the F.R.S.-FNRS website.↩︎
The ERC/F.R.S.-FNRS classification includes a set of descriptor fields which are used to characterize the applications that the Fund collects, as well as the scientific themes of the experts in its expert database. These fields are primarily used for statistical purposes, as part of various analyses conducted with the Fund. They are selected by applicants when they submit their applications, by experts when they create their profiles, and also by the administration of the Fund (to characterize certain expert profiles, for example).↩︎
It is important to note that although we use the term “gender” throughout this report, the data analysed from the F.R.S.-FNRS expert database does not allow us to distinguish between “sex” (the term used for the platform in French) and “gender” (the term used for the platform in English). Experts invited to fill in their sex or gender data can only choose between two sexes (or two genders): male vs. female.↩︎
R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/↩︎
For more information, consult the evaluation page.↩︎
The Fund pays particular attention to the satisfaction of its remote experts, and regularly conducts surveys among them. The latest survey dates back to 2020, the results of which can be consulted online at this link.↩︎
Note that, among others, two major calls are organized by the Funds annually. Through its Grants and Fellowships call, the F.R.S.-FNRS provides financial support for individual researchers, in the form of grants (doctoral researchers), fixed-term fellowships (postdoctoral researchers) or open-ended fellowships (experienced researchers). On the other hand, through the Credits and Projects call, the Fund aims at supporting research teams through the granting of research credits, support for research projets, equipment funding, or by supporting young researchers wishing to develop their own research team.↩︎
Analyses & studies - FNRS
This report was written by the F.R.S.-FNRS Analysis, evaluation and Foresight unit.
The F.R.S.-FNRS Board of Trustees meeting of April 23, 2024 took note of these analyses.
The F.R.S.-FNRS, via its Analysis, evaluation and Foresight unit, carries out a certain number of analyses (mainly statistical ones): call analyses - following the closure of any call and making it possible to monitor its proper functioning; survey reports on former fellows and permanent researchers of the F.R.S.-FNRS and its associated Funds; survey reports on experts who have taken part in the various evaluation missions of the Fund; bibliometrics; report on the state of gender equality, etc.
For more information, please consult: FNRS Analyses & Studies page.
Contact data :
Sarah Itani (sarah.itani@frs-fnrs.be)
Analysis, Evaluation and Foresight
F.R.S.-FNRS
Rue d’Egmont 5, 1000 Bruxelles
Belgique
To cite this document: Fonds de la Recherche Scientifique-FNRS (2024). F.R.S.-FNRS expert database: facts and figures.