An article published in the journal Nature last month (July 5) puts forward a new technique for the evaluation of research on development. It marks a departure from conventional approaches that, according to the authors, have significant weaknesses.
The paper was written by Jean Lebel, president of Canada’s International Development Research Centre (IDRC) and Robert McLean, who is a senior program specialist at the same institution.
This new method for the evaluation of development research — known as RQ+ or Research Quality Plus — emphasises the crucial importance of context, local knowledge and the views of the populations whose lives the research aims to improve.
To better understand this new evaluation tool, SciDev.Net spoke to Jean Lebel who told us about its key dimensions…
What led you to embark on the study which resulted in this new technique for evaluating development research?
We wanted a way of measuring the quality of research that was more suited to the type of research that we support. For instance, good development research shapes policy and serves to improve people’s quality of life. It was therefore important for us to define, develop and put forward a new and broader evaluation framework that took contextual aspects into consideration, and that looked at how the project positioned itself in order to have an impact on the lives of ordinary people.
But we’re far from the only people asking questions of ourselves. Researchers have long debated the best ways of evaluating the quality of research. The article published in Nature will, I hope, serve to move forward the discussion. Research Quality Plus (RQ+) offers a solution that has the potential to be used by donors and bodies specialising in research into international development, as well as by the scientific community as a whole.
In what way does it fundamentally differ from existing evaluation systems?
Conventional approaches to evaluating scientific endeavours have a number of inbuilt constraints. For example, they focus primarily on peer assessment or bibliometrics but don’t explicitly pass judgement on the originality or usefulness of the research, nor do they look at the degree of respect for local knowledge. The RQ+ approach goes beyond an evaluation focused solely on the scientific merit of research outputs and includes other dimensions that are essential to measuring the value and quality of research.
For the IDRC, these dimensions include legitimacy, importance and the relationship to practical applications. But additional dimensions can vary from one institution to the next. The tool is designed to be adaptable.
In practice what does your method add to conventional approaches, and is anything lost in the process?
We are putting forward a tool that addresses the shortcomings of traditional approaches. RQ+ challenges the conventional wisdom that the quality of research is exclusively to do with the strength of its methodological approach. It incorporates additional dimensions of quality that are important when it comes to certain types of research.
For instance, [it] advocates a systematic examination of the context in which the work is being carried out. RQ+ also concerns itself with the efforts made in terms of social impact by, for instance, assessing how accessible the research findings are.
“Research entirely conducted in the global South is of better quality than research carried out in the global North. It is also of higher quality than research done in the context of a North-South collaboration”
You are critical of the fact that conventional approaches are heavily reliant on the opinion of evaluators and citations. Why is that?
Taken in isolation, the opinion of evaluators and counting citations give an idea of the popularity of scientific research, but tell us little about how original or useful it is and what social impact it could have. As a result, some development research does not get the credit it deserves, even when it is of high quality and has a positive effect on the lives of people in the global South.
RQ+ takes account of what evaluators have to say, but their views should be evidence-based, rather than a simple opinion. Those carrying out the evaluation should take into consideration external points of view — for example those of users targeted by the research or of the communities it is supposed to benefit — as well as the perspectives of other researchers working in the same field.
RQ+ also takes gender into consideration. How does this work?
The evaluator must specifically look at the degree to which the research takes account of questions affected by gender. We [also] assess to what extent the results of the research are the product of a process involving the priorities and opinions of stakeholders, including women.
You also believe that African PhD students should be encouraged to complete their theses at their home institutions. What implications would this have for the quality of research?
In the Nature article we mention a fellowship programme for African PhD students that we supported for many years. The evaluators found it was an exemplary programme in terms of equality between men and women. This is because it took the gender dimension into consideration in the course of the recruitment process, which resulted in greater take-up by female students, enabling the training and retention of a critical mass of female researchers in the region.
One of the findings of our meta-analysis was that local knowledge was of great importance. Building the capacity of young researchers in global South countries that value local knowledge tends to result in innovative solutions. It is those men and women who are most directly affected by an issue who are most likely to come up with an innovative way of addressing it.
RQ+ also challenges the conventional wisdom that global North researchers help to build the capacity of their partners in the global South. What alternative approach would you like to see?
What we are challenging is the notion that research placing an emphasis on training and capacity building is necessarily of lower quality. On the contrary, this kind of research needs to be taken into consideration because capacity building and research quality go hand in hand. When we broaden our definition of research quality, this enables us to highlight the value of the perspectives and the contribution of researchers from the global South.
How have you trialled your method and what results have you obtained?
By focusing on 170 studies funded by the IDRC across seven research areas in various parts of the world, we have generated a number of interesting insights. For instance, we have learned that research entirely conducted in the global South tends to be more scientifically rigorous. It is also more important with respect to development. To be more precise, development research entirely conducted in the global South is of better quality than research carried out in the global North. It is also of higher quality than research done in the context of a North-South collaboration.
Your article mentions a study on climate change in Peru, which is shown to have been credible and useful but was never published in a high-impact journal. What is the significance of this example?
The message we want to get across is precisely that it is important to go beyond conventional methods when evaluating the quality of development research. The research in Peru helped us gain a better understanding of climate change in the central region of the country and contributed to shaping policy and interventions in terms of adaptation in that area. It was rated highly by recourse to a more holistic evaluation process such as RQ+, particularly in terms of its integrity, legitimacy and positioning with a view to practical applications. It is valuable in spite of not being published in high-impact journals but rather disseminated in publications that use local languages, thus allowing its immediate use by local actors seeking to resolve urgent issues.