The BS Detector: How to Tell if a Health Claim Is Actually True

Every week there's a new miracle and a new menace, and they often contradict last week's. You don't need a PhD to see through it — you need a checklist. This is ours, and there's a free prompt at the bottom that runs any claim through it for you.

By Scott Covert · 60, skeptic, not a physician

I built this checklist for myself before I built it for anyone else. At 60, I was tired of being whipsawed by health headlines that flatly contradicted each other every week — so I wrote down the questions that actually sort the signal from the noise.

Here's the trap the whole longevity industry depends on: most people treat a headline, a study, a pattern, and a personal decision as if they were the same thing. They're not. A headline is an ad for a study. A study is one brick. A pattern is a wall built from many bricks. And a decision about your body is something only you can make once you know which of those you're actually looking at.

The BS Detector is just a handful of questions that sort the signal from the noise. Run any claim through them — “NMN reverses aging,” “fasting cures everything,” “this supplement adds 10 years” — and the truth usually sorts itself out in about three minutes.

1. Who was actually studied — and are they anything like you?

This is the question almost nobody asks, and it's the most important one. Every result comes from a specific group of real subjects. Before you borrow that result, find out who they were:

• Species. A huge share of “breakthrough” longevity findings are in mice, worms, or cells in a dish. Fascinating — but you are not a mouse, and most things that work in mice never pan out in people.
• Age and sex. A result in 25-year-old male athletes tells you little about a 62-year-old woman. Aging bodies respond differently to almost everything.
• Health status. Were the subjects already sick? A drug that helps diabetics may do nothing for someone with normal blood sugar — or vice versa.
• How many. Twelve people for six weeks is a hint. Thousands for years is evidence.

You can only borrow a finding to the degree that you resemble the people it came from. That's not a technicality — it's the whole game.

2. Which way does the arrow point?

A study finds that people who do X are healthier. The headline says X makes you healthy. But there are three completely different things that could be going on, and only one of them is the headline's version:

• X really causes the benefit (what they want you to assume).
• Reverse causation — the health causes the X, not the other way around. Healthy people take more supplements, exercise more, get more checkups. The supplement may be a marker of someone already doing well, not the cause.
• A common cause — some third thing (wealth, education, overall conscientiousness) drives both the behavior and the good outcome. This is called confounding, and it fools more studies than anything else.

Only a randomized controlled trial — where chance, not the person, decides who gets X — can cut through this. If a claim rests on “people who do X tend to…,” the arrow is still unproven, no matter how confident the narrator sounds.

3. Who paid, and what got published?

Money doesn't automatically make a study wrong, but it bends the odds, and you should know where it came from:

• Funding & conflicts. A study showing a supplement works, funded by the company selling it, deserves a harder look. So does an influencer reviewing a product they get paid to sell.
• Publication bias. Studies that find an exciting effect get published and promoted. Studies that find nothing quietly disappear. So the literature you see is tilted toward positive results that may not hold up.
• Cherry-picking. One supportive study pulled from a messy, contradictory body of evidence isn't evidence — it's selection. Ask: what does the whole field say, not the one study in the video?

4. How big is the effect — really?

This is where headlines do their worst work. A claim can be technically true and still functionally meaningless:

• Relative vs. absolute risk. “Doubles your risk!” sounds terrifying. But if it's a jump from 1% to 2%, the absolute change is one percentage point. Headlines almost always quote the scary relative number and hide the boring absolute one.
• Surrogate vs. real outcomes. A pill that “improves a biomarker” has not been shown to help you live longer or better. Moving a number on a lab sheet is not the same as avoiding a disease or adding a good year. Many things move markers and change nothing that matters.
• Significant vs. meaningful. “Statistically significant” just means the effect probably isn't zero. It can still be far too small to feel or care about. Big sample sizes can make trivial effects look impressive.

5. Why do the headlines keep flipping?

Because attention is the product, and novelty is what sells it. “The thing you already heard is still true” gets no clicks. “Everything you knew about X is WRONG” gets millions. So the incentive — on social media and in mainstream outlets alike — is to find the one new study that flips the consensus and shout it, then flip again next month.

The contradictions you see usually aren't science failing. They're the reporting swinging between cherry-picked extremes while the actual evidence sits, mostly unchanged, in the boring middle. When you feel whiplash, that's the signal to go find the body of evidence — not to believe whichever take you saw most recently.

The shortcut: Three Ways a Claim Can Be True

When something sounds true and still feels off, it's almost always one of these:

Most “contradictory” claims aren't lies. They're one of these three, sold to you as if they were the fourth thing: true, for you, in a way that matters. That fourth kind is rare and worth a lot — which is exactly why everything wants to look like it.

Run any claim through the BS Detector yourself

You don't have to do this in your head. Copy the prompt below, paste it into ChatGPT or Claude, drop in the claim you're wondering about and a couple of facts about yourself, and it will walk the whole checklist — honestly, without the hype.

You are my skeptical evidence analyst. I'll give you a health or anti-aging claim. Do NOT cheerlead, and do NOT default to either "it works" or "it's a scam." Walk it through this checklist and tell me what is actually known.

THE CLAIM:
[paste the exact claim, headline, or quote here — include the source if you have it]

ABOUT ME (so you can tell me whether this even applies to me):
- Age:
- Sex:
- Relevant health context (conditions, meds, family history, or "none"):

Analyze in this order, and label each section:

1. WHAT IS THE CLAIM, EXACTLY?
   Restate it plainly. Separate the specific, testable claim from the vague marketing wrapper around it.

2. WHAT KIND OF EVIDENCE BACKS IT?
   Name the strongest real evidence — and call out anything weak being passed off as strong. Rank it on this ladder (weakest to strongest): cell/test-tube | animal | human observational (association) | randomized controlled trial | meta-analysis of trials. If it's animal- or test-tube-only, say so loudly.

3. WHO WAS ACTUALLY STUDIED — AND ARE THEY LIKE ME?
   Describe the real study population: species, age, sex, health status, sample size, length. Then tell me honestly how much I can borrow the result given MY age and context above.

4. WHICH WAY DOES THE ARROW POINT?
   Is this cause or just correlation? Check (a) reverse causation — could the outcome cause the "cause"? and (b) confounding — could a third factor (wealth, overall healthiness, age) drive both? Say plainly whether causation is established or only implied.

5. WHO PAID, AND WHAT GOT PUBLISHED?
   Note funding or conflicts of interest if known, and whether this looks cherry-picked from a larger, messier body of evidence (publication bias).

6. HOW BIG IS THE EFFECT, REALLY?
   Translate any scary or exciting number into ABSOLUTE terms (e.g., "doubles risk" may mean 1% to 2%). Distinguish a surrogate marker (a number moved) from a real outcome (lived longer, felt better, avoided disease). Flag effects that are statistically significant but too small to matter.

7. DOSE, FORM, DURATION, REPLICATION.
   Does the claim match the exact form and dose that was studied? Has it been replicated, or is this a single study?

8. VERDICT — for someone like me.
   Give one of: WORTH IT / PROMISING BUT UNPROVEN / UNPROVEN / SKIP. State your confidence (low / medium / high), the single biggest reason, and what one piece of new evidence would change your mind.

Be concise and specific. Where you're uncertain, say so instead of guessing.