AI School · Level 1 · Practical case · First steps, no risk

The health story going round on WhatsApp: checking it without believing it

Léelo en español →

Reading time: about 12 minutes

The situation. A regular customer holds her phone out to you: a message forwarded many times saying that "a study from an American university" has shown that taking a certain supplement every day halves the risk of dementia. She wants to know whether to start today and which one you recommend.

This will happen to you every week, and AI can help you a great deal or make you look foolish, depending on how you use it. If you ask it "is this true?", it will most likely reply with a convincing summary of the story, because it has read a thousand texts like it. If you use it to find the original study and read what it actually says, it saves you half an hour. The whole difference is in the question, and in accepting nothing as a source that cannot be opened.

What you need to hand: The text of the message (copied without the name of whoever forwarded it), a browser and the lesson on hallucinations and checking, which is the basis for all of this. Nothing about the customer goes into the chat: not her age, not her history, not why she is worried. You do not need any of that to know what a study says.

Step by step

  1. Separate the claim from everything else

    A health message on WhatsApp nearly always has the same structure: a strong claim, a vague authority ("a Harvard study", "doctors do not want you to know") and a call to action. The only thing you can check is the claim, so the first job is to isolate it in one sentence.

    In our example: "taking supplement X every day halves the risk of dementia". Look at everything that sentence already takes for granted: that there is a study, that it measured dementia and not something else, that the reduction is by half, that it is due to taking it and not to something else the people taking it have in common, and that it applies to anyone.

    Each of those pieces can fail on its own. Writing them down helps you know what to look for, and protects you from the temptation to accept the whole story because one part of it is true.

  2. Ask it to find the study, not to give an opinion on the story

    The question that works is not "is it true?" but "where does it come from?". A model is far more useful helping you locate something than ruling on whether it is true, and when it searches it forces you to have something to open.

    This health message is going round (copied as it is, with nobody’s data):

"text of the message"

I do not want you to tell me whether it is true. I want you to help me
FIND the original study:
1. What kind of study would be needed to justify that claim.
2. If you know a specific study that fits, give me its full reference
   with a DOI or PubMed identifier (PMID).
3. If you do NOT know a specific one, say so plainly. Do not give me a
   reference that "might" exist.

    Point 1 is the one that teaches most. It will tell you, for instance, that to claim something reduces a risk you need a trial where some people take it and others do not, chosen at random. With that in mind you already know what to look for when you find the study.

  3. Open the reference, and if there is none, look for it yourself

    If it gives you a DOI or a PMID, open it. It may lead to a real article on the subject, a real article on something else, or nothing. All three happen, and the second is the most misleading: a DOI that exists, from a serious journal, about a similar supplement or a different outcome.

    If it gives you no reference — or honestly says it does not know a specific one — search PubMed yourself with the keywords from the claim. If the study exists and has the impact the message suggests, it usually appears in the first results.

    And if it appears nowhere, that is also an answer. A story that claims to cite "a study" and cannot be traced back to any study is not a story about a study.

  4. Once you have the study, ask it what it really says

    With the article summary in hand — the abstract, which is public — AI is genuinely useful: it helps you read it quickly. Paste the abstract in and ask what matters.

    I am pasting the abstract of a scientific article.

Answer ONLY with what this text says, without filling in from what you know:
1. What type of study it is (randomised trial, observational, review…).
2. How many people, what age and for how long.
3. What exactly they measured as the outcome.
4. The main result, in ABSOLUTE risk if it appears; if it only appears
   as relative risk, tell me.
5. What the authors themselves say about the limitations.

Abstract:
paste the abstract here

    The instruction to answer "only with what this text says" is what stops it filling the gaps with what it has read elsewhere. If the abstract does not say how many participants there were, you want to know that, not to be given a plausible number.

  5. Compare the story with the study and prepare your answer for the customer

    Now you can answer. The distance between the message and the study is almost always in one of four places: the study was observational (it saw that people who take the supplement have less dementia, not that the supplement reduces it), the result was a relative risk (half of a very small risk is still very small), it measured something else (a drop in a test score, not diagnosed dementia), or it was done in a specific population that does not resemble the person asking.

    What you tell the customer is neither "it is a lie" nor "it is true". It is what the study says, at its real size, and whether it makes sense for her to take anything. That last part is no longer the AI’s job: it depends on her medication, her history and her doctor, and it is precisely your job.

    And if the supplement could interact with what she already takes, run it through the supplement interaction checker, which works on a reviewed dataset and tells you where each piece of information comes from.

A full example, and what is wrong with it

This is what a model replied when asked directly "is it true that taking this supplement every day halves the risk of dementia?", without the prompt from step 2. It is shortened and not polished.

What it replies

"Yes, there is scientific evidence supporting this claim. A study published in a prestigious scientific journal and carried out by researchers at an American university followed thousands of older adults for several years and found that those who took this supplement regularly had around a 50 % lower risk of developing dementia.

According to the World Health Organization, nutritional factors play an important role in cognitive health. Taking this supplement daily can therefore be a good preventive strategy, especially from the age of 60."

And now, what is wrong with that reply:

  1. It confirms the story by repeating it. "A study published in a prestigious journal" and "an American university" are the same vague words as the WhatsApp message, rewritten with better grammar. There is no title, journal, year or DOI: it found nothing, it paraphrased.
  2. It turns an association into a cause. "Those who took it had a lower risk" is what an observational study finds, and it is compatible with people who take supplements looking after themselves better in every other way. By the next paragraph it has become "taking it can be a good preventive strategy", which is a different claim with nothing new supporting it.
  3. The 50 % is relative and it does not say so. Half of what? Without the starting risk, the figure does not let you tell whether the benefit is large or imperceptible — and it is exactly the figure that will get repeated at the counter.
  4. The WHO citation does not support what it says. That nutrition influences cognitive health is a general sentence nobody disputes; putting it behind a specific recommendation for a supplement is using an authority as decoration. It is the "according to the WHO" pattern the hallucinations lesson teaches you to recognise.
  5. And it ends up recommending it from the age of 60, knowing nothing about the person. That recommendation, repeated at the counter, is yours, not its.

Five problems in a reply that would have reassured the customer and sold her a supplement. And none of them is visible if what you are after is "does it confirm the story?", because it confirms it very fluently.

With the prompt from step 2, the same query usually ends in "I do not know a specific study that matches that claim", which is a much less satisfying answer and a much more useful one: it tells you the search is yours to do and protects you from repeating a figure that does not exist.

And if your pharmacy is not like that

If the story is about a medicine, not a supplement. Even more care. If someone has stopped a treatment because of a message, the priority is not to refute the story but to make sure they do not stop their medication without talking to their doctor. You do the checking afterwards, calmly, and tell them next time they come in.
If your pharmacy gets a lot of these. Save the prompts from steps 2 and 4 in a note on your phone. And keep a simple record of the stories you have already checked and what you found: many come back every few months with a different university’s name, and the second time the answer is already done.
If the person asking is a colleague, not a customer. The same technique works for internal training. A story checked in ten minutes is a good topic for Monday’s meeting: it teaches the whole team to read an abstract and to tell relative from absolute risk, which is what the counter needs most.

When it does not work first time

It gives me a DOI, I open it, and the article is about something else.
That is common, and it is exactly the failure you were looking for. Tell it so, but do not expect the next DOI to be the right one: search PubMed yourself. A model that got one reference wrong can get the next one wrong with the same confidence.
The abstract is technical and hard going.
AI is excellent for this. Ask it first to rewrite the whole thing in plain language, without summarising, and then apply the prompt from step 4 to that version. That way you know the answers come from the text and not from a summary of a summary.
The customer is not reassured by "the study is observational".
Understandable: it is a technical phrase. Translate it into something that makes sense, such as "they saw that people who take it have less dementia, but those people also tend to look after themselves better in general; it has not been shown that it is down to the supplement". If you like, ask the AI for several ways to say it and pick the one that sounds like you.
I cannot find the study anywhere.
Then the answer for the customer is exactly that: you looked and found no study that says what the message says. It is an honest answer and it is usually more convincing than a speech about evidence.

Before you use what it gave you

It paraphrases the story as if confirming it. It has read many texts that say the same thing, so it repeats it with confidence. The match between the message and the reply is not a confirmation: they both come from the same place.
Decorative authorities. "According to the WHO", "a prestigious journal", "Harvard researchers". If there is nothing to open, there is no source, however much weight the name carries.
It jumps from "is associated with" to "prevents". It is the most repeated mistake in health reporting, and a model trained on that reporting reproduces it. Always ask what type of study it is before you believe a verb.
Now, with your own. Look on your own phone for the last health story someone forwarded to you — few people do not have one — and go through the whole route: the claim in one sentence, the prompt from step 2, open or find the study, and the prompt from step 4. Note how long it takes. The first time it is twenty minutes; the third time, five.

← All practical cases Back to the school →