The referee report

In addition to my travels last weekend, another reason that I've been so quiet is because I've been working on revisions to a paper that I have submitted for publication in one of our professional journals. It's been a very intense process, and dries up all of my writing skills for the day.

In astronomy (and every science), when we submit a paper to a professional journal, it has to go through a process called "peer review," or "refereeing." The draft of the paper is sent to one or more experts in the field. These experts, who are expected take a fair and unbiased look at a paper, are given three options: (1) approve the paper, (2) send the paper back for revisions, or (3) reject the paper.

In astronomy, option (2) is by far the most common, especially if it is the first time the paper has been sent in. Very few papers are approved without any revisions. And outright rejection is rare, with only a few percent of papers never being published.

So, when I submit a paper, I usually wait a month and then get an email from the editor of the journal with the referee's report, a list of suggestions, questions, and criticisms. I am then expected to make a good faith effort to address the referee's report in my revisions. After I revise the report, I re-submit the paper to the journal, who then passes it on to the referee (almost always the same person as before). In most cases where the authors have made a good effort, the referee will accept the paper, often contingent on a few minor edits. Sometimes the referee decides it needs some more work and sends it back for more substantial revisions. And, rarely, the referee will decide the paper is just not up to snuff and reject it.

This process is designed to try and ensure that all papers published in a professional journal are scientifically rigorous and meet certain standards of quality. At least, this is the theory. The peer review process is not perfect. Sometimes bad articles get through. Sometimes the referee tries to hold up a good article for purely personal reasons. Sometimes a different or additional referee has to be brought in to resolve conflicts. But while the process isn't perfect, it generally works pretty well.

I've found that most referee's reports I get can be ranked on three scales: length, positive/negative, and helpfulness. Some reports I've received have been short and positive with a couple of useful suggestions. I've gotten reports that are short and positive, and only later did I find out that they weren't useful, because both the referee and I missed a glaring problem. I've gotten long, thoughtful reports that have really helped a paper. And I've gotten long reports that are fairly critical, with the criticisms making me wonder if the person actually read the paper. (As a fictional example, the referee's report might include the comment, "I am shocked that the authors did not discuss the price of tea in China," when the paper has a section headlined, "Our star and the price of tea in China.")

These latter types of reports are a pain. I feel I have to make at least one good-faith effort to address the report, but it is hard for me (a naturally sarcastic person) to avoid putting snarky remarks in my response. Once I had a long, negative report that, after a lot of thought, I realized came down to the referee misunderstanding a single word. I did a search and replace of that word in the paper, and the referee's report came back saying, "this paper is vastly improved." Alas, if they were all that simple.

As for my most recent paper, the revision process wasn't exactly fun. But I finished it, and, after my co-authors review the paper, I will be able to send it back in for another round with the referee.

writing about nothing

These days I am working on a fairly boring paper that I intend to publish in one of our professional astronomy journals. The reason it is boring is because it is about nothing. Or, at least, it is about us looking for something and not finding it.

A couple months ago, my colleagues and I announced that we has seen variations in the light of a special type of white dwarf star. This represented the discovery of a new type of variable star, and we continue to work on understanding that star.

That new variable star was discovered as part of a targeted search. We were looking for variations in the light coming from stars such as the one we observed. And we found those variations in one star. But we didn't see them in other stars, and now we need to write a paper describing these non-detections.

There may be many reasons why we didn't see variations in the other stars we targeted:

1. These other target stars aren't varying.
2. The other targets are varying, but at a very low level that we can't detect.
3. We messed up in our data analysis.

And there may be other reasons. Number 3 is fairly straightforward to check; I'm doing that now. I make sure that we were pointed at the right star, and that the other stars around it are acting pretty normal. I make sure that the weather wasn't too bad, and that the clouds weren't too thick. These checks are time consuming, but important. The hard part is choosing between possibilities one and two, and this is crucial for the science! If we claim (and we do) that these other target stars should not be varying, then we have to rule out variations to pretty low levels, better than one percent. And even that may not be good enough; one colleague of mine states that, "if you don't see a star varying, you haven't looked hard enough." And we have other reasons to thinking that variations smaller than the percent level would be something altogether different. But there is a big difference between saying, "it doesn't vary" and "it doesn't vary at a level larger than 1 percent."

A lot of astronomers don't bother publishing non-detections (or "null results," as we often call them). The papers are boring to write, because we usually aren't sure if we should have seen something or not, and it's not as fun as claiming to find something new. But, when it comes to testing theories, a null result can be just as important as finding something. If a theory were to predict that all of our stars should vary in brightness, but only one of a dozen does, than that theory can be ruled out.