When the refereed becomes the referee...

Last week I talked about how much fun I was having (NOT!) responding to a peer review of one of my papers. Well, now the tables are turned, and I am working on a review of someone else's paper.

It's a bit nerve-wracking. I am supposed to look over the science and make judgements on the quality of another person's work. But more than that, the author usually knows more about what she/he is writing about than I do. So, if I want to do a good job as a peer reviewer, I need to read not only the paper that I'm reviewing, but a lot of background literature.

There are lots of things that need to be considered in a peer review. These include:

• Is the research new and useful? It doesn't need to be the most exciting astronomy ever, but if a paper has little use to anybody else, then the big astronomy journals tend not to want it.
• Is the research convincing? Do I believe the results? Even if I don't believe the conclusions, do I at least believe that the data and the methods are sound?
• Have the authors references enough of the literature? When we write a paper, we don't need to re-derive all of 500 years of astronomical research. It is okay to use the results of other people (like Newton did in his famous quote about "standing on the shoulders of giants."), but we have to give those people credit. And, as a referee, I need to make sure that the paper is not based on old results that have been discredited.
• Did the authors do the math right? Astronomy involves a lot of math. If I were to be a super referee, I would re-calculate their numbers and re-derive their equations to make sure I get the same answers. I tend not to do that. Instead, I spot check numbers and see if the results make sense. This is not an ideal method! I'm being a bit lazy here, and I am giving the referee the benefit of the doubt, assuming that they've checked their own work for mistakes. But, generally, the only way to truly check the work would be for me to re-do their project. That would take time, and delay the publication of the paper. If I suspect a problem, it is okay for me to tell the authors that I think there may be an error, and for them to check the results.
• Does the organization of the paper make sense?
• Do tables and graphs and figures convey the information the authors claim, and do so in a clear manner?

In the end, as a reviewer I get large sway over whether or not a paper eventually gets published. If I reject a paper, the authors can ask the editor of the journal to send the paper to another reviewer, and that decision lies with the editor. If I say a paper is fine, it will almost always appear in print, but the editor would have the power to request changes or even send it to another reviewer, if he/she had reason to doubt the quality of my report. So far, I have never suggested that a paper be rejected.

I do feel a little bad for the people who get my reports. I tend to be long-winded, ask lots of questions, and make lots of suggestions. I just hope that the authors on the receiving end of my reports find my long litany of comments useful. After all, it is science that needs to be served, not my whims or the authors' own desires.

The referee report

In addition to my travels last weekend, another reason that I've been so quiet is because I've been working on revisions to a paper that I have submitted for publication in one of our professional journals. It's been a very intense process, and dries up all of my writing skills for the day.

In astronomy (and every science), when we submit a paper to a professional journal, it has to go through a process called "peer review," or "refereeing." The draft of the paper is sent to one or more experts in the field. These experts, who are expected take a fair and unbiased look at a paper, are given three options: (1) approve the paper, (2) send the paper back for revisions, or (3) reject the paper.

In astronomy, option (2) is by far the most common, especially if it is the first time the paper has been sent in. Very few papers are approved without any revisions. And outright rejection is rare, with only a few percent of papers never being published.

So, when I submit a paper, I usually wait a month and then get an email from the editor of the journal with the referee's report, a list of suggestions, questions, and criticisms. I am then expected to make a good faith effort to address the referee's report in my revisions. After I revise the report, I re-submit the paper to the journal, who then passes it on to the referee (almost always the same person as before). In most cases where the authors have made a good effort, the referee will accept the paper, often contingent on a few minor edits. Sometimes the referee decides it needs some more work and sends it back for more substantial revisions. And, rarely, the referee will decide the paper is just not up to snuff and reject it.

This process is designed to try and ensure that all papers published in a professional journal are scientifically rigorous and meet certain standards of quality. At least, this is the theory. The peer review process is not perfect. Sometimes bad articles get through. Sometimes the referee tries to hold up a good article for purely personal reasons. Sometimes a different or additional referee has to be brought in to resolve conflicts. But while the process isn't perfect, it generally works pretty well.

I've found that most referee's reports I get can be ranked on three scales: length, positive/negative, and helpfulness. Some reports I've received have been short and positive with a couple of useful suggestions. I've gotten reports that are short and positive, and only later did I find out that they weren't useful, because both the referee and I missed a glaring problem. I've gotten long, thoughtful reports that have really helped a paper. And I've gotten long reports that are fairly critical, with the criticisms making me wonder if the person actually read the paper. (As a fictional example, the referee's report might include the comment, "I am shocked that the authors did not discuss the price of tea in China," when the paper has a section headlined, "Our star and the price of tea in China.")

These latter types of reports are a pain. I feel I have to make at least one good-faith effort to address the report, but it is hard for me (a naturally sarcastic person) to avoid putting snarky remarks in my response. Once I had a long, negative report that, after a lot of thought, I realized came down to the referee misunderstanding a single word. I did a search and replace of that word in the paper, and the referee's report came back saying, "this paper is vastly improved." Alas, if they were all that simple.

As for my most recent paper, the revision process wasn't exactly fun. But I finished it, and, after my co-authors review the paper, I will be able to send it back in for another round with the referee.

writing about nothing

These days I am working on a fairly boring paper that I intend to publish in one of our professional astronomy journals. The reason it is boring is because it is about nothing. Or, at least, it is about us looking for something and not finding it.

A couple months ago, my colleagues and I announced that we has seen variations in the light of a special type of white dwarf star. This represented the discovery of a new type of variable star, and we continue to work on understanding that star.

That new variable star was discovered as part of a targeted search. We were looking for variations in the light coming from stars such as the one we observed. And we found those variations in one star. But we didn't see them in other stars, and now we need to write a paper describing these non-detections.

There may be many reasons why we didn't see variations in the other stars we targeted:

1. These other target stars aren't varying.
2. The other targets are varying, but at a very low level that we can't detect.
3. We messed up in our data analysis.

And there may be other reasons. Number 3 is fairly straightforward to check; I'm doing that now. I make sure that we were pointed at the right star, and that the other stars around it are acting pretty normal. I make sure that the weather wasn't too bad, and that the clouds weren't too thick. These checks are time consuming, but important. The hard part is choosing between possibilities one and two, and this is crucial for the science! If we claim (and we do) that these other target stars should not be varying, then we have to rule out variations to pretty low levels, better than one percent. And even that may not be good enough; one colleague of mine states that, "if you don't see a star varying, you haven't looked hard enough." And we have other reasons to thinking that variations smaller than the percent level would be something altogether different. But there is a big difference between saying, "it doesn't vary" and "it doesn't vary at a level larger than 1 percent."

A lot of astronomers don't bother publishing non-detections (or "null results," as we often call them). The papers are boring to write, because we usually aren't sure if we should have seen something or not, and it's not as fun as claiming to find something new. But, when it comes to testing theories, a null result can be just as important as finding something. If a theory were to predict that all of our stars should vary in brightness, but only one of a dozen does, than that theory can be ruled out.