Month: January 2008

Check out The Chronicle of Higher Education‘s coverage of ELO board member Noah Wardrip-Fruin’s latest project.

___________________________________

Tuesday, January 22, 2008

Blog Comments and Peer Review Go Head to Head to See Which Makes a Book Better

By JEFFREY R. YOUNG

What if scholarly books were peer reviewed by anonymous blog comments rather than by traditional, selected peer reviewers?

That’s the question being posed by an unusual experiment that begins today. It involves a scholar studying video games, a popular academic blog with the playful name Grand Text Auto, a nonprofit group designing blog tools for scholars, and MIT Press.

The idea took shape when Noah Wardrip-Fruin, an assistant professor of communication at the University of California at San Diego, was talking with his editor at the press about peer reviewers for the book he was finishing, The book, with the not-so-playful title Expressive Processing: Digital Fictions, Computer Games, and Software Studies, examines the importance of using both software design and traditional media-studies methods in the study of video games.

One group of reviewers jumped to his mind: “I immediately thought, you know it’s the people on Grand Text Auto.” The blog, which takes its moniker from the controversial video game Grand Theft Auto, is run by Mr. Wardrip-Fruin and five colleagues. It offers an academic take on interactive fiction and video games.

Inviting More Critics

The blog is read by many of the same scholars he sees at academic conferences, and also attracts readers from the video-game industry and teenagers who are hard-core video-game players. At its peak, the blog has had more than 200,000 visitors per month, he says.

“This is the community whose response I want, not just the small circle of academics,” Mr. Wardrip-Fruin says.

So he called up the folks at the Institute for the Future of the Book, who developed CommentPress, a tool for adding digital margin notes to blogs (The Chronicle, September 28, 2007). Would they help out? He wondered if he could post sections of his book on Grand Text Auto and allow readers, using CommentPress, to add critiques right in the margins.

The idea was to tap the wisdom of his crowd. Visitors to the blog might not read the whole manuscript, as traditional reviewers do, but they might weigh in on a section in which they have some expertise.

The institute, an unusual academic center run by the University of Southern California but based in Brooklyn, N.Y., was game. So was Mr. Wardrip-Fruin’s editor at MIT Press, Doug Sery, but with one important caveat. He insisted on running the manuscript through the traditional peer-review process as well. “We are a peer-review press—we’re always going to want to have an honest peer review,” says Mr. Sery, senior editor for new media and game studies. “The reputation of MIT Press, or any good academic press, is based on a peer-review model.”

So the experiment will provide a side-by-side comparison of reviewing—old school versus new blog. Mr. Wardrip-Fruin calls the new method “blog-based peer review.”

Each day he will post a new chunk of his draft to the blog, and readers will be invited to comment. That should open the floodgates of input, possibly generating thousands of responses by the time all 300-plus pages of the book are posted. “My plan is to respond to everything that seems substantial,” says the author.

The institute is modifying its CommentPress software for the project, with the help of a $10,000 grant from San Diego’s Academic Senate, to create a version that bloggers can more easily add to their existing academic blogs.

A Cautious Look Forward

Mr. Wardrip-Fruin’s friends have warned him that sorting through all those comments will take over his life, or at least take far more time than he expects. “It’s been said to me enough times by people who are not just naysayers that it is in the back of my mind,” he acknowledges. Still, the book’s review process “will pale in comparison to the work of writing it.”

He expects the blog-based review to be more helpful than the traditional peer review because of the variety of voices contributing. “I am dead certain it will make the book better,” he says.

Mr. Sery isn’t so sure. “I don’t know how this general peer review is going to help,” the editor says, except maybe to catch small errors that have slipped through the cracks. Traditional peer review involves carefully chosen experts in the same subject area, who can point to big-picture issues as well as nitpick details. He bets that the blog reviews might merely spark flame wars or other unhelpful arguments about minor points. “I’m curious to see what kind of comments we get back,” he says.

That probably “depends on what you’re writing about,” says Clifford A. Lynch, executive director of the Coalition for Networked Information, a group that supports the use of technology in scholarly communication. “If, God help you, you’re writing about current religious or political issues, you’re going to get a lot of people with agendas who aren’t interested in having a rational discussion. Some of them are just psychos.”

Even without flame wars, Mr. Sery equates the blog review with the kind of informal sharing of drafts that many academics do with close friends. It’s useful, but it’s still not formal peer review, he argues. Carefully choosing reviewers “really allows for the expression of their ideas on the book,” he says. Scholars can say with authority, for instance, that a book just isn’t worth publishing.

Ben Vershbow, editorial director at the Institute for the Future of the Book, concedes that comments on blogs are unlikely to fully replace peer review. But he says academic blogging can play a role in the publishing process.

“The conversational modes of reading and writing on the Web in things like blogs and wikis really chime well with the essential idea of peer review, which is putting out work in development to a peer group and refining the work,” he says. But he hopes that Mr. Wardrip-Fruin’s project demonstrates that the scholarly communities that have formed around many academic blogs “can to a large extent take care of their own review processes.”

Whether it does or does not, Mr. Wardrip-Fruin expects the experience will be interesting enough to write up in an academic essay, or maybe in the preface to the book, when it finally comes out in the old-fashioned printed form.

Copyright (c) 2008 by The Chronicle of Higher Education

This article, “Blog Comments and Peer Review Go Head to Head to See Which Makes a Book Better” is available online at this address:

http://chronicle.com/article/Blog-Comments-vs-Peer-Review-/22268

This article will be available to non-subscribers of The Chronicle for up to five days after it is e-mailed.

The article is always available to Chronicle subscribers at this address:

http://chronicle.com/article/Blog-Comments-vs-Peer-Review-/22268

______________________________