Paper reviews
DraftA paper review records what a paper claims, how it argues it, and whether the argument holds. The last part is the point. A summary is available from the abstract; an assessment is not.
Why write one
Reading a paper and moving on retains very little. Writing down the claim, the method and the weaknesses forces the reading to be active, and produces something that can be returned to years later when the paper is cited by something else.
It also builds calibration. Reviewing enough papers in a field develops a sense of which claims are robust and which rest on a single assumption, and that judgement transfers to new papers far better than any amount of passive reading.
Structure
| Field | Content |
|---|---|
| Citation | Authors, title, venue, year, and a DOI or link |
| Claim | The central assertion, in one sentence, in your own words |
| Why it matters | What changes if it is right |
| Method | How the claim is argued: observation, experiment, simulation, derivation |
| Assumptions | What must be true for the method to be valid, including unstated assumptions |
| Result | The quantitative finding, with its stated uncertainty |
| Assessment | What is convincing, what is not, what would change your mind |
| Relation to other work | Whether it agrees with, extends or contradicts what is already recorded |
| Follow-up | What to read next, and any question it opened |
Reading for assessment
A few specific things are worth checking on every paper, because they are where weak results usually fail.
- Does the conclusion follow from the result?
- Papers routinely state a conclusion broader than the data supports. Compare the abstract claim against the actual measurement.
- What is the uncertainty, and is it complete?
- A stated statistical uncertainty with no systematic budget is incomplete, and in observational fields systematics usually dominate.
- What would falsify it?
- A claim with no conceivable disconfirming observation is not an empirical claim.
- Who has tried to reproduce it?
- A single result is a hypothesis. Check for independent confirmation before treating it as established.
- What is not shown?
- Which figures are absent, which cases were excluded, which parameter ranges were not explored.
The last is the hardest and the most useful. A paper presents the analysis that worked, and the interesting question is frequently what else was tried.
Two worked cautions
Both are recorded elsewhere on this site and are useful calibration examples.
The Venus phosphine claim, discussed in Venus, turned on data reduction choices that were not visible in the headline result. Reanalysis of the same data with different baseline fitting produced weaker or absent signals. The lesson is that an extraordinary claim resting on a single measurement should be assessed on its reduction methodology, and that this is usually in a supplement rather than in the paper.
The Hubble tension, discussed in Cosmology, is the opposite case: two careful measurements disagreeing, with extensively examined systematics on both sides. Here the correct assessment is not to pick a side but to record that the disagreement is real, significant, and unresolved, and that it may indicate the model rather than either measurement.
Rating
A crude rating is worth attaching, because it makes the collection scannable and forces a judgement.
| Rating | Meaning |
|---|---|
| Foundational | Establishes something the field now depends on |
| Solid | Convincing within its stated scope |
| Suggestive | Interesting, insufficiently supported to rely on |
| Unconvincing | The argument does not support the conclusion |
| Superseded | Was right or useful, now overtaken |
A rating is a judgement and may be wrong. Record the reasoning, so that a later disagreement with your own earlier assessment can be understood rather than merely noticed.
Promotion
A review that changes understanding of a topic should update the corresponding subject page, with the paper added to that page’s references. The review stays here as the record of the assessment.
Reviews that accumulate around one topic without changing anything are a signal worth noticing: either the topic is settled, or the reading is not reaching the work that matters.
References
Reviews cite the papers they review. This page is a convention.