Dear Denis and Team, 
 
Thank you for revising your work and engaging with the reviewers. 
 
We consider that the manuscript is more streamlined now, and some sections make a valuable contribution. 
 
However, there are a few things we ask you to consider before we can make the manuscript recommendation ready. Please find attached the PDF of your submission with our edits. Kindly note the following: 
 
1. The manuscript has a heavy number of typos and grammatical errors. At this point, it severely interferes with readability. We request the team carefully edit the document, avoid typos or have the paper professionally copy-edited. 
Note that we highlighted many typos, but not all of them. Some words are regularly misspelled (practices, assessment, potentially, additionally, activities, etc.). 
 
While some edits and suggestions may appear stylistic, we consider that attention to these details is necessary, so that the reader is not distracted from the worth and strength of your core argument. 
 
2. We make certain recommendations in-text on the pdf document; please address them.
3. Christophe Dejours – please provide a few lines to justify their body of work and why you draw heavily on their work in this paper. 
4. Relatedly, ‘Beauty judgment’: This is an important bridge between the sociology of work and organization studies. It would help if you could briefly summarize Dejours’s conceptual development of beauty judgment. While utility can be understood, beauty is not an everyday reference in this context of academic evaluations. This fresh and novel perspective you integrate has had little traction in organization studies thus far and can ignite some exciting conversations. This becomes particularly important given that many of Dejour's writings are in French and may not be immediately accessible by reference to the interested reader. An elaboration from your side would be helpful. 
5. Subjectivity: This paragraph, pages 10-11, must be better justified or unpacked. Either develop it or please remove the whole paragraph, as its contribution to the argument needs to be clarified. 
 
Overall, this is a valuable contribution. We look forward to taking this forward. 
Thank you for working with us on this.

Sincerely, 

Devi and Heloise

Dear Denis Tagu, 

Thank you for revising your manuscript in response to the reviewers' suggestions. We continue to find the manuscript of interest and relevant to PCI Organization Studies. 
 
Based on our readings and that of the two peer reviewers, we have some recommendations for you to revise this manuscript. We invite you to carefully consider the detailed comments provided by Reviewer 1 and Reviewer 2’s suggestions to advance this manuscript as a perspectives article. We also agree with Reviewer 1 for the need for a clear and coherent narrative arc across the manuscript. 
 
We look forward to receiving the revised manuscript. 
 
Sincerely,
Devi and Heloise. 

Thank you for submitting this work to PCI Organization Studies. The article's topic is highly aligned with PCI Organization Studies’ interests. 

Thank you for your patience. Based on our readings and those of the two peer reviewers, we have some recommendations for you to revise this manuscript. 

First and foremost, we consider that this manuscript may be best shaped as a perspectives piece or a commentary rather than a conventional research article that advances theory. This is primarily because a traditional research article would need a deeper engagement with extant theory in the field. Currently, the paper anchors primarily, and limitedly, around Christophe Dejours. That being said, we do see value in the case of INRAE. 

With this in mind, we have the following suggestions regarding introduction, context of INRAE and France, analysis of INRAE’s alternative practices, and transferability. 

Introduction

1.         As a background, could you further develop the trend of rankings and problematization of rankings by coalitions like DORA?

a)         For example, could you develop more of what is the “restrictive view of work”?

Page 3: Nowadays, and since only a few years, these quantitative parameters (number of publications, impact factor, H index, quartile rank of the journal, …) are less in used and even banned from several organisations (DORA, the San Francisco Declaration on Research Assessment2), because of their restrictive view of the work.

b)         Similarly, can you tell us more about the “hidden activities” and “real work”. While these arguments may be familiar for those already immersed and familiar with a critique of the ranking system, this hidden and real work can be fleshed out for an interested reader.  

c)         Could you connect with broader institutional changes, beyond DORA (e.g. what’s being done at COARA). You may wish to start the introduction or the section after the introduction with a broader picture of what these changes look like, what is the dominant paradigm. This will allow you to better substantiate the originality and specificity of INRAE. There are other more or less critical works on research evaluation methods (impact factor, citation counts, rankings, UK research excellence framework, bibliometrics etc.) in other countries and disciplines. We are not insisting that you must review all of them. But bringing in some elements from these texts will help the reader gain elements for comparison (e.g., Brankovic et al., 2022; Dougherty & Horne, 2022; Gingras, 2016; Martin, 2011; Vasen et al., 2023). 

Here, Table 3 is interesting. Can you integrate/ explain some of these elements in the text? 

Context of INRAE and Higher Education in France

2.         It would help the reader outside France to (even if briefly) understand the context of higher education employment practices in France/Europe and INRAE. The civil-servant scientist category may not be immediately relatable in other contexts. We encourage you to add more elements of context, and reflect upon what is transferable to other disciplines, institutions and/or countries.

Analysis of INRAE’s Alternative Assessment Practices

3.         It would help to understand the context in which INRAE developed this assessment form. Why? What were the triggers? Did they begin with this form of assessment? Did they tweak it? What are the challenges they face?

After the backdrop of neoliberal ranking system in academia, an initial section could chronologically describe the evolution of INRAE assessment framework. 

4.         Is INRAE one of the earliest in France to do so? In Europe? Are there other models? Is INRAE a distinctive model that others emulate or learn from? These are important for readers to understand the canvas within which INRAE stands out. 

5.         Both qualitative and quantitative assessments inhere a certain degree of subjectivity – page 8. Page 8: So i) indicators are necessary to limit the risk that subjectivity operates during qualitative assessment, and ii) the use of a multicriteria approach - as performed at INRAE - might limit this risk by buffering each criteria to - at the end - define the profile of each scientist by the distribution of her/his types of activities

There is a need for more explanation or substantiation through relevant citations. 

6.         A critical engagement with the assessment practice: You may wish to discuss what happens when there are conflicts in assessment. How are they resolved? 

An important parameter here is how different social groups fare in this kind of assessment. How do gender, race, and class differences in how people are assessed play out in the organizational context? How would these situations play out? 

For example, how many women are promoted to research director? Is this INRAE method securing models of domination and inequalities, or does it in fact, help challenge them? Critical engagement with these insights would be persuasive. 

What are the costs of conducting such modes of evaluation at INRAE? Who bears the cost? Peers as they spend time on this? Those evaluated, as they need to spend more time on their evaluation file?

Another critique could be that when evaluation committees abandon ranking as a management tool, they actually still use, they simply do not say it (cf. Berry’s work on management tools as invisible technologies (Berry, 1983)). How do we ensure that career advancement is not still based simply on bibliometrics? How does INRAE approach the more informal aspects of research assessment, individual/collective biases we have towards journals/grants, etc.?

7.         This manuscript can unpack the notion of “beauty” in further detail. How is this concretely implemented? For example, can you share a case with the concerned stakeholders sufficiently anonymized? 

There appears to be a dichotomy between the front-end of the paper on beauty, and the rest of the paper on the analysis of career assessment, categories, etc.

Please see also the reviewers’ questions regarding the beauty assessment. 

Similarly, regarding interdisciplinarity, you raise some of the difficulties of assessing inter/pluridisciplinarity while functioning by disciplines. Research shows that interdisciplinary researchers are penalized as they threaten social boundaries (Fini et al., 2023). Can the beauty assessment help here? 

Overall, if the main claim is about beauty, the paper needs to be more coherent throughout its different parts (including in the categories identified at the end on evolutions)

Transferability

8.         One big issue we have when questioning the system and especially developing PCI Org Studies is evaluation. How do we interrupt the prevailing assessment system so a PCI-recommended preprint can be valued in the career of a researcher. INRAE provides an inspiring approach. But to what extent is this transferable or usable in other fields, institutions and countries? How does this connect with bigger issues like rankings of universities. 

(Please see announcements of Utrecht University, in its decision to abandon uses of impact factors and to be removed from international rankings)

Minor: 

1.         Please avoid the acronym OS, which is both Open Science and Organization Studies, and can interfere with readability. 

Overall, we consider the case of INRAE as worthy of documentation. We suggest that the manuscript is revised as a perspectives piece. While there is no imperative to develop a theoretical contribution, we agree with reviewer 1 for more “situatedness”, and R2 for more “critical reflection” and critical distantiation in presenting the case.

When you submit your revisions, also submit a letter that replies to our and the reviewers’ comments, point by point, clarifying what changes you have made.

Please submit your revisions within the next four months. If you need an extension, please get in touch with us 

Devi Vijay and Héloïse Berkowitz 

References

Berry, M. (1983). Une technologie invisible ? L’impact des instruments de gestion sur l’évolution des systèmes humains. Centre de recherche en Gestion de l’Ecole Polytechnique.

Brankovic, J., Ringel, L., & Werron, T. (2022). Spreading the gospel : Legitimating university rankings as boundary work. Research Evaluation, rvac035. https://doi.org/10.1093/reseval/rvac035

Dougherty, M. R., & Horne, Z. (2022). Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences. Royal Society Open Science, 9(8), 220334. https://doi.org/10.1098/rsos.220334

Fini, R., Jourdan, J., Perkmann, M., & Toschi, L. (2023). A New Take on the Categorical Imperative : Gatekeeping, Boundary Maintenance, and Evaluation Penalties in Science. Organization Science, 34(3), 1090‑1110. https://doi.org/10.1287/orsc.2022.1610

Gingras, Y. (2016). Bibliometrics and research evaluation : Uses and abuses. The MIT Press.

Martin, B. R. (2011). The Research Excellence Framework and the ‘impact agenda’ : Are we creating a Frankenstein monster? Research Evaluation, 20(3), 247‑254. https://doi.org/10.3152/095820211X13118583635693

Vasen, F., Sarthou, N. F., Romano, S. A., Gutiérrez, B. D., & Pintos, M. (2023). Turning academics into researchers : The development of National Researcher Categorization Systems in Latin America. Research Evaluation, rvad021. https://doi.org/10.1093/reseval/rvad021