On JPP, Wiley, Prestige Production, and Modeling Quality:

A Weekend Bonus post.

As Leiterreports (and here; here) and Dailynous reported, Wiley (a commercial publisher), got rid of Robert Goodin (a prominent political philosopher) as editor of the Journal of Political Philosophy (JPP), which is (one of) the leading journals in political philosophy in good part thanks to Goodin's long stewardship.  It is no surprise that most of Goodin's fellow editors and board resigned in protest. There has been a lovely closing of ranks.

Because I am going to be rather (sternly) critical of the ranks, I want to stress that Wiley's explanation was ridiculously and scandalously vague. More important, the manner in which they fired Goodin also suggests unwarranted editorial interference that actually risks endangering academic freedom. So, resignation was the right thing to do.

But as the resigning editorial board members' comments started to appear in public, it seems that folk in the know believe that the cause of Wiley's decision is that it wanted to increase the number of published papers rather rapidly up to a tenfold more (see here; here). Immediately one started to read on social media that pursuit of profit is undermining quality. (This show up in the threads I have already linked to above, but also in discussions elsewhere.)

However, a major reason why Wiley is nudging many presses to expand journal space is that universities, libraries, and grant agencies have contracted with them to do so in the context of expanding open access. From the perspective of grant agencies and universities this makes a lot of sense: they are funding research that, if of sufficient quality, needs an outlet. (For those who work in a grant system the absence of slots often becomes visible only after acceptance when one discovers that a journal has no more slots available in a given year!) I am not a fan of these contracts because they basically show what happens when well-connected corporations take advantage of well-intentioned policy shifts: they capture the financial rents. Yet, it doesn't follow that open access or expanding journal space (and so increasing acceptance rates) is a mistake. And in what follows I focus on the latter.

For, as is well known, academic philosophy has incredibly low acceptance rates in its flagship journals (often under 5% acceptance rates).* I couldn't find a recent number for JPP, but it seems to be around 5% and I wouldn't be surprised if it were lower. A quick count suggests, JPP publishes 24 articles a year (give or take). Another leading journal, P&PA "has historically published 12-16 articles per year." And it does not get much better at a journal like CRISPP (about 20). [I make the relationship between slots and acceptance rates more precise below.] The situation is not much better in most areas of professional philosophy (but see below).

I have noted in the past that these acceptance rates only make sense, epistemically, if philosophical activity had enormous inductive risk to society. Nobody believes that as Tim Crane argued in an important essay in TLS (back in 2018; see here for ungated version). Tim helpfully suggests that the real explanation is that "publication in one of these journals may no longer be that sole decisive achievement that will get you that job or grant." That's clearly true, but not wholly right not just because I would add promotion to his list. As Anna Stilz, whose comments were highlighted by Justin Weinberg, notes, publication in these journals is a "signalling device that validates the importance of someone’s work when they go up for tenure and promotion."

However, as should be well known, compared to other disciplines, including the sciences, professional philosophy's acceptance rates is especially low (recall Justin Weinber's discussion). And other disciplines also hire and promote people. Thus, I suggest the real function of such very low acceptance rates is the felt need for prestige production in professional philosophy among the circle entrusted to run leading journals.+ Before I get to the attempt to conflate such 'prestige' or 'importance' with 'quality' that insiders often make, it is worth noting that the needs of such signaling are not evidently co-extensive with the pursuit of truth (leaving wisdom aside) or progress in the field. In fact, they clearly can come apart.

For, there are some (quite predictable and also widely discussed) side-effects of the very low acceptance rates: very fine papers need to be shopped around to many journals before they find a home. This has overburdened the supply of referees, and it has put tremendous pressure on editorial process at many journals. We are a discipline that is familiar with moratoria on submission and backlogs from acceptance to publication at quite a few journals. There is also often a huge delay between submission and verdict. And consequently overall a huge delay between research and actual publication for too many papers. Thus, as organized, prestige generation slows down whatever rate of progress one might hope for in the field. (Yes, I am a skeptic about philosophical progress, so I use 'progress'  here relative to a research frontier or relative to how competent judges in a field would describe it.)** 

The problem with this state of affair is not merely epistemic. It also generates huge additional anxiety for people who are or expect to be on the job market and who are up for promotion. (This is also true in other fields, of course.) With acceptance rates so low, they basically face a (weighted) lottery whether their papers are accepted in a time frame needed. This state of affairs probably reinforces all kinds of structural injustice, again not only relative to other disciplines. The fact that many leading journals have, sometimes for decades on end, been entrusted to a 'safe pair of hands' (a distinguished, often male, philosopher) probably increases the bottlenecks and the risks of such structural injustice, but -- like the fact that we see so little experimentation with journal form (and kinds of review) [something I strongly advocate] -- I leave that aside here.

Now, I am not claiming editors are gaming acceptance rates directly. But they do clearly view number of issue/volume slots or pages as a hard constraint (perhaps regulated by an implicit willingness to let publication not be delayed for more than a year or two after official acceptance). But notice that restricting supply of slots is not the same as selecting on quality. If one really selected on quality alone one would allow the supply of slots/pages to be flexible. So, in what follows (and above) I use 'low acceptance rates' as a proxy for limiting the number of publication slots, or selectivity.

Above I noted that prestige generation trade-offs with progress at the low rates of acceptance in professional philosophy. We don't need to model that. (It gets more complicated as acceptance rates rise significantly because there may well be an inflection point where too much ease of publication undermines progress.) But we need some modeling to help understand that prestige generation or selectivity and quality can come apart significantly.

Luckily for me (and the quality of discussion) I was sent a pre-print of a paper, "Academic journals, incentives, and the quality of peer review: a model," by K. J. S. Zollman, J. Garcıa, and T. Handfield conditionally accepted at Philosophy of Science that has a game-theoretic model exploring possible trade offs between aiming for quality and (you guessed it) for low acceptance rates, in contexts where selectivity is taken to be a measure or proxy for quality. The paper should be publicly available fairly soon. But my discussion won’t turn on technical details and the paper's possible lack of robustness. (I call incentive-alert philosophy of science, ‘public choice philosophy of science.’) In their model quality is an effect of peer review which is costly. And somewhat worrisome they conclude that "journals incentivized by selectivity have strong incentives to maintain worse peer review than those incentivized by their quality." I claim this nicely describes what happens in professional philosophy. 

Now, they note that aiming for low acceptance or selectivity creates an incentive among editors to discourage "self-selection because a paper which is not submitted cannot be rejected. This results in a strange process whereby journals make peer review worse in an attempt to induce bad papers to submit, but maintain sufficiently good peer review to ensure that a large proportion of those bad papers will probably be rejected." And somebody may well object that referee quality at our leading journals is -- despite the borish incivility and rudeness -- actually qualitatively great. Let's assume there is a lot of correct testimony to this effect.

One can explain such testimony in light of Zollman/Garcia/Handfield model, if we pay attention to the role of desk rejects (something they ignore.) Let's return to JPP; it is especially famous for two interrelated features: it offered quick (even helpful) desk-rejects (often/always by Goodin) to the vast-majority of initial submissions and then stellar review process on the papers sent out to reviewers which help improve the final publications. (I know important scholars who almost only review for JPP.) And this creates (now quoting Zollman et al) "the appearance of high standards."

One may think that JPP has, thus, hit on a fine equilibrium between balancing peer review costs and quality; and that this approach is a good way to aim for selectivity and quality at once. And people do like to point to the quality of the papers appearing in JPP. (In 2021 its impact factor was according the journal's website 1.881--quite respectable for philosophy standards, but not especially noteworthy. Something fans of selectivity tend to ignore.) Fair enough.

But, of course, desk-reject even by an impartial and distinguished and unrushed editor is a very coarse-grained (and very cheap) peer review system. And the Zollman et al. model explains that this does reduce quality. Obviously, this presupposes that among the desk-rejects are sufficiently many papers that with (costly) peer review would have been as good as the papers that JPP did publish. That's obviously true, but that's because among the papers published in very selective peers of JPP and even slightly less selective journals there is very little difference in overall quality. But the nice thing about working with a game-theoretic model like Zollman et al.'s is you don't have to trust my judgments, it's a predictable effect of the incentives -- artificially limiting supply! -- shaping decisions. Of course, the model may not be robust so here I have used it solely to help think through the issues. But notice that when multiple journals aim to be selective it becomes much costlier to provide adequate peer review because referees get overworked (and perhaps bored by seeing the same papers multiple times).

Now, I am not against prestige in professional philosophy or higher education (although one does wonder if excessive selectivity in admissions to top colleges actually is conducive to society's needs). But from the perspective of the epistemic (and social) needs of the discipline and profession, it is a mistake to have excessive selectivity in philosophy journal publication help produce or coincide with that prestige. It slows down research and, as the model suggests, plausibly hurts quality. (It probably does so by reducing intellectual diversity and increasing risk aversion.) 

So, from my perspective, while the publishers should respect editorial independence and not use heavy-handed practices, nudging our leading journals to accept more papers is the right thing to do even if the motive -- profit -- or honoring flawed open access contracts need not be so. We need such a regime change.

Of course, the increase should be gradual and not tenfold at once. The critics of Wiley have a point! As should be clear most arguments that conflate incredible selectivity and quality in defense of the status quo seem bogus and, I say this politely (these are my friends, after all), seem generally self-serving not the discipline's interest.

Tom Hurka suggested that such selectivity allows one to "keep abreast of important new developments and contributions in the field without having to wade through dozens or hundreds of published pieces." This was very true before the age of scholar.google and philpapers. But less so now. I do think we need more outlets like (as Ingrid Robeyns suggested in context) Philosophy Compass that (especially with the disappearance of review chapters in dissertations) give opinionated surveys of recent developments, but that's orthogonal to the present discussion.

Finally, I don't want to claim that increasing, even doubling slots or reducing/halving selectivity (to say 10-20% acceptance rates) solves the profession's social problems. Undoubtedly, if journal publication in particular venues is seen as key, then we cannot prevent an arms race among job candidates and tenure seekers to publish in such journals. And maybe the gains to the referee process will be ephemeral or temporary--only time and experience will tell. But my argument for the epistemic gains from increasing slots remain.

Academic disciplines do need high quality signals for quality in publication for institutional and epistemic reasons; excessive selectivity is a bad mechanism to produce such a signal and it perniciously confers prestige that reinforces all the bad effects of the very steep disciplinary prestige hierarchies. I hope the scholars who nobly and correctly resigned in protest also work to change this bad status quo.

---

*I am going to ignore the pervasive editorial unaccountability and deviations of best scholarly practices that also show in these journals.

**Of course, the salience of significant work seeps out to well connected insiders, but that just strengthens my argument.

+Some field specific journals within philosophy that actually presuppose considerable technical expertise (in the sciences or ancient languages) have much higher acceptance rates without corresponding loss of prestige. I think this is due to the fact that the prestige is already sufficiently accorded to the expertise (and that editors in these fields can focus more on keeping the discussion going aright).