• Welcome to OGBoards 10.0, keep in mind that we will be making LOTS of changes to smooth out the experience here and make it as close as possible functionally to the old software, but feel free to drop suggestions or requests in the Tech Support subforum!

Is America the most brainwashed country on Earth?/least scientific belief

PhDeac said:
Good post, BigTree. I'm definitely not saying we need to eliminate peer review. Not at all. I do think we need to streamline it in a few ways.

First, the editorial staffs need to be an effective line of defense. Over the last year, I've gotten a few papers that had very obvious methodological flaws that would be immediately picked up by anybody with reasonable knowledge in the field. Hire a few people to just be that line of defense and they can provide initial suggestions before the editor has to go begging people to review it. I like reviewing to be honest. But if a journal I don't usually review for sends me very suspect work, I may not be willing to review for them again.

Second, your enthusiasm for reviewing gets to what I'd like to see. Instead of editors trying to hunt down experts, journals should just put submissions online and let potential reviewers select what they want to review. There would be some level of screening of course and editors can decide if they want to accept reviews with potential conflicts of interest. Of course, I've reviewed papers written by people I know and vice versa, so I don't think it's that big a deal except in some cases.

I submitted an article for a special edition of an prominent education journal back in October. We were expecting a decision by February at the latest. Mid-Feb, I got an email from the editors saying they'd have reviews in in early March. Heard nothing in March. Heard nothing by the national conference in April. I emailed the editors again and one responded that they've had a tough time getting reviewers for the main issue and the special issue. My system avoids that issue.

I'll leave it at that. I would be in favor of far more radical changes than that, however. Townie, I assume you're in the biz. I'd like to discuss some ideas with you further. PM me if you're interested.
Click to expand...

Not to toot the horn of my spot, but our slowest journal, in terms of turnaround time, is 6 weeks, and that journal publishes twice a month, well over 500 pages per issue. And they have to assign reviewers on the traditional system.

I think you run into problems at huge commercial journals, especially high impact/high profile journals that get just insane numbers of submissions, and try to run a lean staff to maximize profits. We're lucky in the sense that we're nonprofit and disseminating good science fast is in the mission.

Feel free to PM!
 
I have a grant pending with a multidisciplinary research team that will hopefully generate research that would be a good fit for computer science and engineering journals that require shorter articles and have quicker turnaround times. Co-authors papers for those will be a nice change.

Will PM later.
 
Also I know we're clogging up the thread with sci-comm/publishing stuff, but this is a good read about collaboration in biomedicine - http://www.nature.com/nbt/journal/v31/n4/full/nbt.2555.html

It's from last year, nothing altogether striking, but it does get into some structural changes that could be helpful to enhance and encourage collaboration in science.
 
Paper on peer review in the Journal of Public Economics - http://obs.rc.fas.harvard.edu/chetty/referee_experiment.pdf

Written by Harvard Economist and Editor of the journal, it's a look at whether cash incentives, coupled with shorter deadlines, led to faster and better reviews.

The results suggest, basically, if you pay them, and give them a shorter deadline, they'll meet it and do a good job:

chetty.png


I think there are pretty much only Economics journals that do this now, but it seems to me that paid peer review might not be the worst idea in the world. I'd love to hear from others publishing in the sciences/social sciences on their thoughts on this.
 
Where the money comes from is critical. You can't have them being paid from industry sources.

Many bar associations have arbitration and other boards that are non-paid but scheduled.
 
RJKarl said:
Where the money comes from is critical. You can't have them being paid from industry sources.

Many bar associations have arbitration and other boards that are non-paid but scheduled.
Click to expand...

They'd be paid by publishers.
 
Interesting study. I have many thoughts about this but will try to keep it brief unless somebody like Townie wants more info

For starters, I think it is unfortunate they found no appreciable increase in the level of agreeance, quality of review, etc by adding incentives to the process. Those are the real big issues that plague peer review right now and it would be awesome to be able to kill two birds with one stone. As it stands, I would be happy to get equal levels of reviews back quicker (assuming an equally simple system cannot be identified that improves overall quality).

Depending on how the compensation worked (and how much it was for), you could theoretically see an increase in the quality of reviews over time since more top level scientists may be willing to devote time to reviews rather than sending them down the pipeline to their post-docs (or even graduate students if it is a low tier journal). A continued want (or need) to get articles to review could also increase the quality of the reviews over time since the number one thing you can do to ensure you don't get papers for review, even now, is to do crappy reviews. Such a shift wouldn't necessarily be evident in a short term study since the participants likely new, or at least deduced, that further payments would not be coming at the conclusion of the study.

That all assumes, of course, that such payment would still be considered "consulting" by the granting institutions and thus not subject to additional red-tape such as effort reporting. If the NIH changed that stance (in light of more time being devoted to reviewing for instance), then the whole system could get thrown for a loop. Suddenly you would rely on that compensation to support a % of your salary, rather than it serving as additional income. I think it can be easily seen where that could cause problems both for quality of review (more pressure to produce more reviews) and overall quality of reviewers (top guys with funded grants aren't going to want to go through the hassle involved).

One, perhaps unintended, side effect of paying reviewers could be increased tutelage at the doctoral level on how to actually review a paper properly. Investigators would be motivated to increase the capacity for those under them to do reviews (since they may still pass some on to these individuals even when compensated). And on the level of a student/post-doc, they would also be highly motivated to hone their reviewing skills so if they ever reach the faculty level, they will be able to take advantage of the additional income. As I have said, one of the biggest issues with Ph.D. programs (other than there being too many churning out too many doctors) is that they have lost the necessary emphasis on the thinking aspect of the sciences and have become far too bench/research focused. Anything that could help tip the balance of the scales back towards the logical and critical analysis side is a win in my book.
 
I especially liked your last paragraph on graduate education and early career mentoring.

If there were a way to link compensation to quality of the review, which is hard to gauge without someone else looking at the quality of the review, it's hard to tell why we should expect quality to improve. In other words, if advancing science isn't enough incentive, why should pay be? The other approach to explore is open reviews, such as the EMBO journals. If others are going to be reading the reviews, quality will certainly improve.
 
You must log in or register to reply here.
Back
Top