Yesterday, in a casual conversation with a friend, a professor of economics, I brought up the problem with performance evaluations of scientists who work at universities. I made a comment at some point, saying that if teaching performance was assessed similar to research performance, we should consider if a graduate of our courses were hired in some great company, e.g., McKinsey & Co. That I thought highlights the absurdity of using an outcome metric as a performance variable, when these outcomes are not comparable. Great graduates may have totally different careers and work at different companies, some certainly making more income. Yet, we have no right to say that a high-paying firm is of higher quality than any other firm. There are numerous cases of some extremely well paying firms that went bankrupt a decade later… It is extremely difficult to forecast the long-term outcome to scientific endeavors, in both teaching and research.
Effort is indeed a better measure of performance than the outcome to assess performance of a scientist.
Universities must provide opportunities and support for scientists with sufficient academic training (doctorate degrees) to contribute within their fields of expertise, as well as for enthusiastic amateur researchers in the areas they are interested in. The return on the resources allocated should not be evaluated over a short-term period of one or two years, but rather over decades. We can interpret the concept of academic freedom this way. The lifespan of universities must be much longer than that of companies. Therefore, when a university assesses the performance of the scientists, simpler and longer-term criteria.
Let me be more specific with a proof by contradiction, i.e., reductio ad absurdum. Often research performance evaluation is based on the number of publications in certain scientific journals considered as high-quality research-output venues. Although that may appear as an objective criterion with moderate to high reliability, validity is questionable with systematic errors. For example, we may use the UK-based Chartered Association of Business Schools (ABS) Academic Journal Guide (AJG) 2024 list, for an assessment of research performance in Marketing. There are various validity issues.
Concerns About the Validity of Hiring and Promotion Criteria
Distribution of journals by fields illustrates the first validity issue. In Economics, there are 67 journals classified with a rating of “3”, 32 in Finance, 22 in Accounting and 12 in Marketing. In Economics, the number of journals classified with a “4” rating drops to 21, yet in Marketing there are only 2 journals in that class, 10% of Economics. A criterion that demands a publication in a “3” or “4” journal clearly generates inconsistency by design. This is the most obvious validity issue. The probability of achieving promotion criteria is much higher in Economics, in Finance and in Accounting, compared to Marketing.
| Field | 1 | 2 | 3 | 4 | 4* | Grand Total |
| Grand Total | 789 | 565 | 323 | 98 | 47 | 1822 |
| ECON | 144 | 117 | 67 | 21 | 6 | 355 |
| FINANCE | 47 | 39 | 32 | 5 | 3 | 126 |
| SOC SCI | 38 | 34 | 30 | 8 | 5 | 115 |
| ETHICS-CSR-MAN | 60 | 30 | 13 | 3 | 5 | 111 |
| SECTOR | 57 | 36 | 10 | 5 | 108 | |
| INFO MAN | 48 | 36 | 11 | 5 | 3 | 103 |
| ACCOUNT | 27 | 34 | 22 | 2 | 4 | 89 |
| MKT | 39 | 19 | 12 | 2 | 6 | 78 |
If we apply another criterion to trust ratings in some of the fields, e.g., because they are within business administration such as Accounting, then we should question why the list includes other fields. That demands questioning of categorization of journals into fields. There are 23 categories, which gives the impression of a somewhat arbitrary process. There are numerous categories with only one (1) 4* rating journal. The category STRAT has only 27 journals in total. We may compare that with FINANCE, which has 126 journals in total. Furthermore, one journal corresponds to one category in the ABS-AJG list. How should we think about interdisciplinary journals, e.g., Technological Forecasting and Social Change? That is yet another issue. We may consider interdisciplinary journals based on their weaknesses in fitting in, “neither that nor this” instead of “both this and that”. Since ABS-AJG does not list any journal in more than one category, or I failed to find one, the list probably does not recognize interdisciplinary research. Lastly, the difference between ratings “3” and “4*” is highly significant for the hiring and promotion criteria we will apply. The Journal of Business Ethics is scored with a “3” in ABS-AJG, yet is Financial Times 50 list of journals, which includes a combination of 4*, 4 and 3 rating journals. There is some areas of improvement for external validity as well.
Conclusion
From personal experience, I find setting minimum criteria counter-productive. In my first academic appointment, I had to teach 4 courses per semester for a few years, although my agreement was 2 courses per semester initially. I am not sure if it is related but I repeatedly declined the request by the Dean of Faculty to send one of my working papers to a journal listed in Q1/ Q2 of Social Sciences Citation Index (SSCI) so that I could meet the minimum criterion for having a publication as a validation of my appointment as a professor.
Here is my take-away from that experience. Good to have minimum targets to achieve high quality output, are not as useful as providing the necessary opportunities to achieve high quality output. Research, services and teaching easily take too much effort. Faculties are definitely in better positions to know about their specific needs (and may develop the capacity to make these assessments in-house) so that they may be more transparent about their decision making. We may then more effort, not less to report reasons for hiring/ promotion decisions with the evaluated candidates.