Monday, November 22, 2010

The Place of Peer Review

by BONNIE WHEELER

[we are pleased to bring to you for discussion this pre-publication section of an essay to appear in the Journal of Scholarly Publishing, "What is a Scholarly Journal? The Place of Peer Review" by  Bonnie Wheeler]

Credentialing / Peer Review
Journal publication provides several forms of credentialing. The first is simply the credit that redounds to an author by being published in a particular journal. The second is the “value added” by careful checking and editing of the article prior to publication. Each editor can list articles that might accurately cite the editor as co-author. The third is the crux: peer review. The credentialing provided by peer review is, I think, the bedrock “value added” in scholarly journals, even though we are as diverse in its definitions as we are about all other aspects of journal editing.

Academic editors think about peer review constantly.(1) Because of petitions and protests, I was alerted to the fact that several science journals still practice single-anonymous review (the reviewer remains unknown but the names of the authors are revealed). I suspect, but have no reliable empirical proof, that most of us in the humanities are accustomed to requiring standard double-anonymous peer reviews even though we are all aware, as bitter colleagues note, with Foucault in the foreground, that peer review is often viewed as “a form of censorship,” an exclusionary, disciplining mode that functions to inhibit or delay new work. I am told that the German system requires a fully open review process in which all parties are revealed to each other. There are strong reasons and feelings on all sides of these arguments.(2)

As an editor, I always feel guilty about requesting double-anonymous peer reviews, knowing that the reviewer will never be fully credited either in print or in profit for a penetrating, helpful peer review. Perhaps I am once again too na├»ve: do scholarly journals increasingly pay for peer review, or does peer review count in salary negotiations? (3) Probably not. I’ve recently received several complaints (one recently in a powerfully argued editorial from the Journal of Hydrology) over the past years about the increasing unwillingness of specialists to provide peer review precisely because it is “unrewarded activity.” The generations of junior faculty we “protected” from committee work and taught to concentrate on their own rewarded activity have now moved into senior positions. Though many of these colleagues have benefitted mightily from fine peer review, they aren’t necessarily acculturated to accepting reciprocal responsibility. We made ’em this way, we socialized them this way, but now we need them to adopt a different professional training and sociology and “ante up” if peer review is to survive in its current form. Some even suggest a “point system” by which reviewers would earn points they could redeem when they next request peer review for their own work.

Journals that our colleagues recognize as most prestigious will still get their attention (flattery has its uses), and many of you, my fellow editors, are skeptical that there is a problem at all. But on the evidence that I see, an occasional irritation is quickly becoming a dominant force. The pressures increase as the number of journals and thus journal submissions increase. In my experience, most schools offer some stipend when they approach you for tenure and promotion reviews; most presses reward you for manuscript reviews. But for peer review of essay submissions in the humanities, we depend upon the generosity of others.

We depend upon a commitment to our profession that transcends private gain. We used to call it collegiality. Collegial peer review, in all its manifestations, depends upon a fading notion of noblesse oblige. Freely produced peer review may be one of the last “class” vestiges we retain in our profession, an appendix remaining from the positive side of the otherwise punitive (to outsiders) Old Boys’ Network. Yet, in revised formats, it now might provide a transparent activity that reflects one’s obligation to aid the development of vivid intellectual work in one’s field.

How can we remake our systems to interest our young in participating in the credentialing process? In this issue of JSP, James J. O’Donnell suggests that journals of academic societies change their mission:
The traditional publishing peer review ... is seriously flawed and depends too heavily on the willingness of scholars to participate. What vehicles can be found for bringing together scholars around a common purpose and exposing what they do to evaluation of its own? ... Could we imagine a [learned] society that suspended publication of a journal per se but focused on a panel of reviewers who evaluated and recognized the best articles published by members wherever they should appear? A society's web page that contained links to twenty of the best articles published this year by its members might well do more for the discipline than one that spent proportionally more effort on soliciting and editing twenty articles of its own.”(4)
This is an attractive but exclusionary tactic: the notion of “twenty best” blockades the intellectual opulence and openness that our generation of academics has worked hard to achieve.

NIH grant directives insist upon and provide funding for peer review in the sciences. The formal assessment of proposed research begins, according to the NIH, “at a sufficiently early stage to influence the course of that research, the nature of its outputs, and ultimately even whether it takes place at all (or is made available to a wider audience).” In humanities publications, peer review typically takes place at a medial point, when an author or authors submit to a journal an essay that has been shaped to the point that it is considered “ready for publication.” Kathleen Fitzpatrick, whose rich intervention in this question I mentioned earlier, uses an e-publication format that embodies the desire for credentialing that she advocates—a kind of 1960s intellectual e-commune in which everyone participates in the work of production and of evaluation. In her complex analysis of peer review, especially on-line credentialing, she models a new best practice: open peer review, in which scholars post the penultimate version of their work to a website that invites responses. Those responses are open in every sense: the reviewers are known (and thus can be credited), the author can respond, and the final “product” in hard or e-copy acknowledges all participants in the process. She argues that:
the time has come for us to consider whether, really, we might all be better served by separating the question of credentialing from the publishing process, by allowing everything through the gate, and by designing a post- publication peer review process that focuses on how a scholarly text should be received rather than whether it should be out there in the first place. What if peer review ... became peer-to-peer review?
Fitzpatrick’s process may become a new norm, though it is notable that her work, posted now for many months, has not received heavy e-commentary. So far, like our CELJ blog,few are answering her call for comments, contradictions, and suggestions. Her book is slated for hardcopy publication, after which it will presumably receive the prominence it deserves in book reviews.

Whether peer review will continue to fulfill its highest goal of aiding authors to achieve the originality, clarity, and authority they seek depends upon what individual and collective roles we play as editors and publishers. How much of this will remain under editorial control is an open question. But among all elements of scholarly editing, peer review is the most likely to persist in some form. At its best, this form of credentialing extends our mission as teachers to authors who are our most intensely interested audiences. Most fearfully, however, credentialing suits larger political mandates for evaluation and control.

Bonnie Wheeler, Director of the Medieval Studies Program at Southern Methodist University, is executive editor of Arthuriana and past president of CELJ, for whom she collected and edited the essays in this issue of JSP.
--------
(1) For thoughtful studies of peer review, see David Shatz, (Lanham, Md: Rowman & Littlefield, 2004) and Kathleen Fitzpatrick, Planned Obsolescence: Publishing, Technology, and the Future of the Academy, on-line pre-print.

(2) Thatcher comments: “I would hazard the generalization that peer review of monographs carried out by university presses generally fits the single-anonymous review model, though reviewers are always given the option of revealing their identities to the authors.”

(3) The book world, in general, operates differently, and pays at least a nominal amount to reviewers, though I’ve been watching recently as some mega-corporations have begun to compensate journal reviewers with come-ons worthy of carnival hawkers: see Literature Compass.

15 comments:

Jeffrey J. Cohen said...

A response worth reading here.

Eileen Joy said...

Per your link, Jeffrey, to the comments at the "Progressive Geographies" weblog, I agree with Stuart Elden that peer-reviewing is essentially an exchange economy, and while I think I understand Bonnie W.'s concerns about younger faculty not being as willing as their predecessors to always assent to reviewing their peers' scholarship [and I think this simply has a lot to do with what I don't think is an arguable point: that internal institutional pressures--in the form of more and more demands for self-assessment, participation in institutional governance, increasing expectations for spending more time on campus, and the like--as well as various forms of hyper-professionalization, has led to a state of affairs where faculty find they have less and less time to do everything they need to do just to keep their heads above water, get tenure, etc.], BUT: Elden's point is still an important one to reflect on further here: aside from the salaries we receive from our institutions [and the extra monies that some of us might luckily receive in the way of grants and fellowships, guest-speaking engagements, guest-teaching, consulting, and the like], there will simply be no advancement in ANY of our research agendas if we do not all tacitly agree that we are obligated to review each other's work [otherwise, how would our own work ever get reviewed and published and thereby have a chance of actually influencing the field?].

There is still, and always will be, a question of information or work overload, as Bonnie aptly points to with the example of the low web-traffic at Kathleen Fitzpatrick's website where she has made her book manuscript fully open for review by any and all comers [and yet, where are they?]. This is partly why I favor the *partial*/HYBRID peer review experiment recently engaged in by The Shakespeare Quarterly because in order for any structure of so-called "open" review to work, it has to be initiated and managed/curated by an editorial team that would work, ahead of time, to bring a certain number of "experts" and other interested parties on board and to keep them there for a certain sustained period of time. The "crowds" will vary in size from piece to piece, and special issue to special issue, and book to book, but this again goes back to Elden's point that we have to all kind of agree up front that engaging in this process, whether closed and blind, or open and "crowded," is part and parcel of our professional obligations and this is not necessarily *only* pro bono work, because we have the hope/right to expect reviews in return for our own reviewing [and editing] labors.

[to be continued]

Eileen Joy said...

[continuing]

I also think it is a little misleading, from a historical perspective, to believe [as some do and say often] that our current digital age brings with it an information overload that has heretofore never existed and which overwhelms our waking senses to the point where we're all exhausted before we can even begin leading to various states of emotional and labor entropy and "giving up"]. As any scholar of bibliography studies knows, starting with the ancient Library [Museum] at Alexandria, when it comes to books and manuscripts and the scholarship attendant thereupon, information overload has always been the problem--just ask Callimachus, one of the first bibliographer-cataloguers at Alexandria. So, we'll admit up front that information overload has always been a problem, while at the same time, there have never been more economic threats to what might be called the "business" of the university library and academic publishing, which I think also calls for more collective and creative labors on the part of the intelligentsia, situated both within and outside of the university proper. And let's reflect, too, that academic publishing does not *have* to be undertaken at a financial loss [it turns out that medievalists actually buy a lot of books], while at the same we have to continue to convince university administrators that supporting university presses at break-even returns is a critical part of their mission.

Information overload has always been a problem--okay--and there is the question of faculty in the current moment perhaps being squeezed to the nth degree [while also watching in horror as some humanities programs at more than several schools are being "de-funded"--something we should not neglect to see as a *common* concern, no matter our particular specialty, period, or language], and this is why I think we need to especially concentrate our efforts now on more collective labors pointed toward the realm of what I would call the enlargement ans re-tooling of the domains of our scholarly development [and if the digital age is also the age of information overload--which is not new--at the very least, it provides us with more avenues of dissemination of information that are, let's say, "easier" to get to than a hard-structure library in Turin or Paris]. I also think it will be important to open up more than just "review"--meaning, I think we have to diversify the modes and genres that are available and *possible* for our work in order to [pleasurably] re-wire our dispositions and affects *toward* our work: in other words, if we can conceive of our work as more pleasurable, more exciting, more personally fulfilling, more conducive to individual and collective happiness, then the question of labor in mitigated somewhat in relation to these dispositions and affects. This is also something we have to work at collectively.

[to be continued]

Eileen Joy said...

[continuing]

Perhaps it goes without saying, then, that I am in favor of more open review processes and even more *collective* review processes whereby the burden(s) on each individual reviewers can be lessened somewhat [and where we can still provide various channels for those who might want to remain "anonymous"--on both sides of the equation] and where the communal nature/gift-exchange economy of our profession that Elden is at pains to remind us of is made more clear and transparent. There are ethical considerations at stake here as well, and I've always been struck by Jane Bennett's argument, in "The Enchantment of Modern Life," that ethical actions require affective propulsions: we must be enchanted with the world [a little in love with it, struck by wonder at it] in order to want to act on its behalf. I think the same can be argued for a profession, such as medieval studies. Of course we're all a little "in love" with our field [however we want to describe or characterize our dispositions toward our field of study], while we all have very different and I'm sure complicated reasons as to why we work on the subjects we do, and these reasons will always impel us, regardless of the economies of surplus or scarcity within which we work [and regardless of whether or not some of us would say "I *love* what I do"], but again, I would just ask us to consider how we might "open up" not only the avenues through which we develop and seek professional legitimation for our scholarship, but also how we might diversify and "open up" the types and genres of work that is possible and the spaces [virtual, digital, and otherwise] within which it is permissible to "play/work," alone and together. There are issues of "control" with the way our scholarship is authorized/legitimated, which I feel strongly blind peer-review has [wittingly or unwittingly] contributed to and which has possibly narrowed the horizons for what is permissible to conceptualize/execute in our work, and I think the time has come for some serious dismantling of these traditional systems in favor of some wild experimentations that would be both productive *and* enjoyable. I will also always side with the idea that more of everything is never a bad thing: despite all the bad news regarding publishing profits, the end of the book/reading, an over-stretched academic labor base, and the like, we should be working toward more journals, more book series, and even short-term publishing ventures that would exist to fill a certain niche/void, then disband as soon as certain objectives were accomplished. The trick, of course, would also be in getting "points" on one's c.v. for participating in all of these ventures. But just look at entities like Open Humanities Press--the thing is, I'm really not worried at all. Indeed, I'll be traveling to NYC the first week of December to draw up plans for a new *print* and e-book series for BABEL. Stay tuned.

KF said...

Thanks so much to you, and to Bonnie, for this thoughtful post and discussion. I do however want to raise a slight objection to the characterization of the review process of my book manuscript as either having received too-few comments or having low traffic. In the nine months after the manuscript was posted, I had over 31,000 pageloads, with over 12,700 unique visitors, over 3370 of whom made multiple return visits. 44 unique commenters left a total of 295 comments. On the one side, hold that up alongside the fact that the average scholarly press monograph in the humanities sells well fewer than 400 copies over its lifetime; on the other side, compare it with the usual two or three anonymous outside reviewers' reports, reports that neither respond to one another nor are in dialogue with the author. Given both the quantity and the quality of dialogue held in the margins of my manuscript, I do not understand how it can come to be characterized as under-visited or under-commented. Perhaps the comments might be worth another look.

Eileen Joy, your point about the SQ open review's advance management is well to the point; that decision came both from the SQ board's desire to ensure that the same folks who would have been anonymous reviewers participated in the open review process, and from the lessons that we learned from my own review. Which is to say that facilitating discussion requires real work. But that work paid off, both for the SQ issue and for my own book. Take another look -- and perhaps, next year, look at how the final print version has been influenced by the open review process.

KF said...

Thanks so much to you, and to Bonnie, for this thoughtful post and discussion. I do however want to raise a slight objection to the characterization of the review process of my book manuscript as either having received too-few comments or having low traffic. In the nine months after the manuscript was posted, I had over 31,000 pageloads, with over 12,700 unique visitors, over 3370 of whom made multiple return visits. 44 unique commenters left a total of 295 comments. On the one side, hold that up alongside the fact that the average scholarly press monograph in the humanities sells well fewer than 400 copies over its lifetime; on the other side, compare it with the usual two or three anonymous outside reviewers' reports, reports that neither respond to one another nor are in dialogue with the author. Given both the quantity and the quality of dialogue held in the margins of my manuscript, I do not understand how it can come to be characterized as under-visited or under-commented. Perhaps the comments might be worth another look.

Eileen Joy, your point about the SQ open review's advance management is well to the point; that decision came both from the SQ board's desire to ensure that the same folks who would have been anonymous reviewers participated in the open review process, and from the lessons that we learned from my own review. Which is to say that facilitating discussion requires real work. But that work paid off, both for the SQ issue and for my own book. Take another look -- and perhaps, next year, look at how the final print version has been influenced by the open review process.

Eileen Joy said...

OOPS! Kathleen: please accept my deepest apologies, as I did not check my facts! You might like to know that, at BABEL's inaugural conference, held at the Univ. of Texas at Austin a few weeks ago, that we had a round-table session on the future of scholarly/digital publishing in which your chapter on Authorship was one of our "core" discussion texts, and I've visited your site more than once, so again, my apologies!

Eileen Joy said...

I also want to second something Kathleen highlights here, which has to do with readership and the ways in which electronic and digital media have increased the traffic of those who read our work, which can also be brought to a "mixed" [academic and non-academic] audience much faster in a myriad of ways not afforded by traditional scholarly publishing. I will take the daily readership of this weblog, for instance, over the #s of people who will likely read my article on the Old English "Seven Sleepers," which I think is the bets thing I have ever written, and which I am still waiting, almost 4 years after completing it, to see come into print. That is why I also regularly flout copyright laws and self-publish/pre-print + re-print everything I write on my website. I want readers more than I want traditional "publication."

Notorious Ph.D. said...

I'd be interested in hearing what Bonnie and others like her think of legitimate reasons for declining to do a peer review (other than being frightfully overextended, which is a perennial problem). I've declined to review article MSS on topics that I'm an "expert" in because I knew the person involved and more or less knew their work (though perhaps not the particular article under review). When this has happened, I've generally declined, stated my reasons, and suggested alternate possibilities. But as our "specialty" fields are always smallish, that might pose a problem: if I receive a request to review, say, an article on monastic architecture in the low countries, chances are that the community of potential experts is quite small, and eventually we're all going to know each other, for good or for ill.

So the question: where do we find a balance between impartial review and review done by experts in a particular area?

KF said...

No problem, of course! One thing that this does point to is, with a text the size of a book manuscript, how difficult it is for a discussion to feel as though it's attained a kind of critical mass -- and now if we multiply that by everyone putting drafts online in this fashion... well, it's clear that we're going to have to find new ways to reward active reviewing, as there's going to be a lot of work to do.

Thanks for mentioning the BABEL roundtable; I hope that it was fruitful -- the entire conference sounded fantastic, from what I was catching over Twitter. I'll look forward to following more of how things develop here.

Another Damned Medievalist said...

jeebus, EIleen!

I think that one thing that could help, at least at SLACs like mine where service both on campus and to the profession count, is a regularization of commission- and thank-you letters from editors that could be placed in one's files. Peer review of articles might not count any more than book reviews, but they might count abou the same ...

Another Damned Medievalist said...

Oh -- two more things: as a person who has so little time to do my own writing, I actually love reviewing my friends' and colleagues' work, formally and informally. This is because it gives me a "legitimate" excuse to work on something in my field and short-term. It's a great way to keep my head in the game when teaching four courses in a semester.

Also, though, I wonder... if you're like me and have already had friends look at your ms, and your field is fairly small, how does one find a peer reviewer who is appropriate?

tenthmedieval said...

... if we can conceive of our work as more pleasurable, more exciting, more personally fulfilling, more conducive to individual and collective happiness, then the question of labor in mitigated somewhat in relation to these dispositions and affects...

We're all in trouble when our work starts to feel like work, surely. At that rate we could be earning better elsewhere, the love is what covers the gap. I think one of the last posts at Acadamnit had the right approach: witness here. It speaks to a sense of defeat but surely the writer ultimately has the Right Idea.

Katherine Rowe said...

I'm really enjoying this thoughtful exchange and looking forward to the whole article, Bonnie.

I wanted to underscore something that's happening here, implicitly, that strikes me as important to the viability of our scholarly processes over the long term: which is that we now have multiple humanities disciplines engaged in thinking about how peer review can and should evolve, and that seems to me a crucial condition for effective experimentation and for validating what works (outside of the alchemy of a single experiment or field).

Bonnie, I don't know if your article will map some of this evolving conversation among different humanities fields. But precisely because this thinking is being tested in a distributed way, it would be tremendously helpful to have some kind of starting place, annotated bibliography, etc. So far I count this ongoing conversation at In the Middle; Bonnie's essay; Kathleen's book; SQ's pages; Larry Cebula over at Northwest Historian peer-reviewing the SQ process itself; the next SQ special issue on performance... What should be added to this list?

(Here's Larry's blog post, for reference:
http://northwesthistory.blogspot.com/2010/09/peer-review-20.html)

Bonnie W said...

I am so grateful to Jeffrey for posting my piece-in-progress. I wrote it almost a year ago but it is only now going to press. Your posting saved me from being utterly out of date: I have revised to take account of the snowball-effect of Kathleen Fitzpatrick's PLANNED OBSOLESCENCE and some other concerns. This is such a generous forum. Bonnie Wheeler