House manual
How we do it
How we test each myth and how we reach a verdict, explained without formulas.
Bet first, look later
Every issue starts with a question and a vote. That way, before seeing the data, you already know what you thought. Get it right: great. Get it wrong: you’ve learned something.
Two things moving together doesn’t mean one causes the other
In summer more ice cream is sold, and more people drown at sea. They go together, but ice cream drowns nobody: both rise with the heat.
So before comparing anything, we remove what we already know changes the data: the weekday, the time of year, holidays. What’s left is what the myth could explain.
Things wobble even when nothing happens
Flip a coin 100 times and you’ll rarely get exactly 50 heads. Data work the same way: any group of days you pick lands a bit above or below normal, purely by chance.
So the question isn’t “is there a difference?”, because there almost always is. The question is: is it bigger than what chance produces?
The draw
To answer it we run a draw. In the full-moon issue, for example, we took one ordinary day from each lunar month, as fake full moons, and checked how far they strayed from normal. We did it 2,000 times.
Having the fake moons land once a month, like the real ones, matters: if we picked loose days at random, some draws would grab lots of holidays and others none, and chance would look stronger than it is.
Issue 01: in 26 out of 100 draws, random days match the full moon
Each square is a draw of randomly picked days
- 26draws where random days stray as much as the full moon, or more
- 74draws where they stray less
In 26 out of 100 draws, randomly picked days strayed from normal as much as full-moon days did, or more.
Source: CDC/NCHS and SSA, via FiveThirtyEight
Show the data as a table
| Squares | |
|---|---|
| draws where random days stray as much as the full moon, or more | 26 |
| draws where they stray less | 74 |
If the effect were real, random days would hardly ever catch up. The usual rule is that it should happen fewer than 5 times in 100.
“Nothing shows up” isn’t enough
Not seeing an effect doesn’t prove it isn’t there: with little data, hardly anything shows up. So the draw gives us something more useful: how big the real effect could be, at most and at least. For the full moon, between -3 and 13 babies per 1,000.
And before giving a verdict we set a bar: the smallest effect for the myth to “be right”, meaning someone would notice it in real life. For the full moon we set it at 20 babies per 1,000.
How we reach a verdict
With those two things, the effect’s range and the bar, the verdict follows. The analysis itself works it out; we don’t pick it while writing.
Myth when we can rule out the effect reaching the bar: even in the best case nobody would notice. Also if it goes the opposite way to the myth.
Fact when the effect is surely there (its whole range is above zero) and it reaches the bar.
Half true when the effect is surely there but falls short of the bar: real, but smaller than people claim.
Unclear when the data fall short: the range is so wide we can neither confirm nor rule out the myth.
The bar is our call, which is why every issue states it. If a smaller effect matters to you, you have the numbers to judge.
Open data, no tricks
- We only use public data, linked in every issue. If Spanish data exist, they come first; if not, we explain why we used others.
- The figures on the site come straight from the analysis; none is typed by hand.
- Every issue also says what the analysis can’t tell you. A fact without its limits is just another myth.