Anecdotal evidence

{{logic}}[[File:Anecdotal Evidence.gif|thumb|right|Also known as the meta-anecdote. ([http://smbc-comics.com/index.php?id=2159 Source])]]{{cquote|<nowiki>[</nowiki>[[Bloodletting]]<nowiki>]</nowiki> removes coma. Mr. Henry Clymer was suddenly relieved of this alarming symptom, in the fever of 1794, by the loss of twelve ounces of blood.|||[[Benjamin Rush]], ''A Defense of Blood-letting''<ref>[https://web.archive.org/web/20101114110251/http://bobarnebeck.com/defence.html ''A Defence of Blood-letting, as a Remedy for Certain Diseases'']</ref>}}
'''Anecdotal evidence''' (also '''proof by [[Cherry picking|selected instances]]''', or, more pejoratively, '''anecdata''') is use of one or more anecdotes (specific instances of an event; stories) to either support or refute a claim. The use of anecdotal evidence to draw a conclusion is like using the NBA all-star teams to estimate the average height of [[American]]s.

Whereas anecdotal evidence is sometimes the ''starting point'' of a proper scientific investigation, it is all too often the ending point and every point of a pseudoscientific investigation. In the world of [[pseudoscience]], an anecdote is treated as the equivalent of a [[peer-review]]ed, [[double-blind]], repeatable [[scientific]] experiment with consistent results. 

Anecdotal evidence is often used in [[politics]], [[journalism]], blogs, and many other contexts to make or imply generalisations based on very limited and [[Cherry-picking|cherry-picked]] examples, rather than reliable statistical studies. A classic instance was [[Ronald Reagan]]'s story of a "[[welfare queen]]" who was abusing the system and whom Reagan attempted to portray as [[Stereotype|indicative of the average welfare recipient]]. It turned out she ''didn't even exist'' when some reporters finally decided to look for her (even if she had, this would not actually prove anything about welfare recipients generally).

Anecdotal evidence is especially vulnerable to [[confabulation]] or outright [[deceit]].

Remember: the plural of "anecdote" is ''not'' "data".<ref group=note>Unless, of course, you gather ''so many'' anecdotes that it's possible to construct a representative statistical model. And even then, a [[double-blind]] study would be preferable.</ref>

==Form==
{{syl
|2=X happened once when Y.
|4=(''[[Enthymeme|unstated]]'') Things that happen once will happen every time.
|5=C|6=X happens every time when Y.
}}

For example, "Billy-Bob ate 3 pounds of apples a day for a year and his cancer disappeared – so can you!"

It should be clear that P2 is invalid. Take the following syllogism:

{{syl
|2=Abby choked once when she ate some [[goat]].
|4=(''unstated'') Things that happen once will happen every time.
|5=C|6=Everyone will choke when they eat goat.
}}

Consume that goat, and face the wrath of the Goat Overlord!

Another form:

{{syl
|2=Horatio started drinking orange juice after being diagnosed with cancer and then his cancer went into remission.
|4=(''[[Enthymeme|unstated]]'') There is no other plausible explanation for Horatio's cancer going into remission than the OJ.
|5=C|6=Orange juice can cure cancer.
}}

It's clear from the above that fallacies like [[post hoc ergo propter hoc]] and [[argument from incredulity]] can play supporting roles in the perception of anecdotes as convincing.

===Legitimate use===
In two instances, it is possible to use anecdotes non-fallaciously:

*If you use one or more anecdotes to refute the claim that there are [[Argument from omniscience|no instances of the event that the anecdote describes]]. This is not fallacious because one counterexample is all it takes to prove a universal rule false, or an existential rule true.

*If you use one or more anecdotes as an ''example'' of a general rule which is ''already supported'' by a broad, comprehensive investigation (i.e., your evidence/argument does not ''rely'' on the anecdotes, they are just used to ''illustrate'' the point).

==Problems==
{{cquote|If you are basing your claims on anecdotal experience, then any treatment will seem to work for [[Panacea|anything and everything]].|||[[Steven Novella]], ''The [[CAM]] Worldview''<ref>[http://www.sciencebasedmedicine.org/the-cam-worldview/ ''The CAM Worldview''], [[Science-Based Medicine]], October 23, 2013.</ref>}}
While anecdotes — when believed to be true, at least — are nice illustrative stories, they do not constitute good repeatable scientific [[evidence]]. This is because anecdotes only ever apply to individuals or individual experiences and are subject to the [[bias]]es that this brings with it. Even without bias, the small scope of an anecdote means that it cannot be generalized to a larger group, due to variations between, say, the physiology of individual humans (and of individual pathogens), the structural integrity of rocks, or whatever the anecdote is about meaning that what works to cure an illness in one person or shatter one stone may or may not work to cure the same illness in another person or to shatter another stone of the same kind. And even if those differences ''could'' be accounted for, that still leaves the ''circumstances'' of an anecdote unaccounted for; the influence of environmental factors on a given procedure's result should not be overlooked. Anecdotes also tend to begin when a problem is first noticed, without any awareness of how long it may have been festering quietly without being observed;<ref group=note>For instance, many diseases have an "{{wpl|incubation period}}" — a length of time between a {{wpl|pathogen}} or toxin entering the body and the onset of its symptoms, during which a pathogen might quietly proliferate without the host being consciously aware of its presence. Given that most people do not have {{wpl|Photographic memory|photographic memories|,}} different diseases have different incubation periods (and these are just diseases with external causes; genetic disorders such as {{wpl|Huntington's disease}} or {{wpl|multiple sclerosis}} can have "incubation periods" of ''multiple decades''), and that a person generally does not instinctively know what disease they are suffering from without the aid of a trained doctor's diagnosis, it can often be difficult or impossible for a person to deduce when and how a given pathogen entered their body and started causing an illness.</ref> thus, the root cause of the problem is often unaccounted for. For all of these reasons and more, it is impossible to say that an individual anecdote is representative, and it is also impossible to actually detect the real cause of the anecdote.

For instance, with life-saving [[medical]] treatments (say, pills that reduce heart disease and subsequently lower the death rate), there are some deaths that occur whether or not the medication was taken (and not all of these deaths are even caused by heart disease). Therefore, if someone who is on the medication dies, you cannot tell if they would have died ''anyway'' without it — you can't prove that the medical intervention worked, or not, from the one case study.

It is very rare for an intervention to be, by itself, a ''sufficient'' cause of something. Rather, they tend to change the ''probability'' of a given event occurring. This means, obviously, that one can cherry pick examples that show something does or does not work, regardless of what effect it actually has. For instance, if the municipal government of some city enacts a law to reduce [[crime]], one can find anecdotes "demonstrating" the exact opposite of the law's actual effect, whatever it may be. If it's effective: "Before the law I never had any problems with crime, but [[Post hoc ergo propter hoc|since its enactment]] I've been mugged once and had my house burglarized! This law is useless!" If it's not: "Before the law, I was robbed twice, but ever since the law, I haven't been robbed once! It must be working!"

Ultimately, anecdotal evidence is very prone to {{wpl|false positive||s}} (and is also prone to false negatives).

===Regression to the mean===
{{main|Regression to the mean}}
If someone with a rash gets better after taking a [[homeopathic]] remedy, that doesn't mean the remedy caused the improvement, as it could simply have been an [[Improbable things happen|exception to the rule]]. Maybe it would have gotten better anyway — for example, it could have just been an allergic reaction, or a sweat-rash or anything along those effects. The anecdote doesn't tell you anything about all the people who did not get better; many other people with rashes could have taken homeopathic remedies to no effect.<ref group=note>The anecdote also says nothing about other people who took the treatment and got better for reasons that [[Correlation does not imply causation|may or may not be related to the treatment]] or the cause of our given anecdote subject's recovery.</ref>

===Adulteration===
Not to mention that these remedies can contain actual active substances, which are misleadingly listed under "non-medicinal ingredients". A "homeopathic" anti-acne cream may contain [[List of medicinal plants#T|tea tree oil]], which is an effective antifungal.

In addition, some people selling alternative medicines sometimes adulterate them with drugs in order to give them the desired medicinal effect and thus generate return customers. For instance, a "herbal" cold remedy may be adulterated with paracetamol and antihistamines.<ref>[http://www.medsafe.govt.nz/hot/media/media2003.asp Media Releases for 2003], MedSafe.</ref> In these sorts of cases, the improvements may very well be caused by the drugs rather than the herbs.

===Improbability===
There are also anecdotes in which a person suffered from a disease for many years, and no conventional treatment was able to cure them, but they improved after applying some alternative remedy for a few days/weeks/months. For instance:

{{quotebox|I having ministered [tobacco] to many people as well men as women, in great number, and being grieved of ten, and of twenty years they have healed old rotten sores in legs, and other parts of the body, with only this remedy to the great admiration of all men.<ref>[https://web.archive.org/web/20120607233337/http://archive.tobacco.org/History/monardes.html ''Of the Tabaco and of His Greate Vertues'']</ref>}}

The implication is that the improvement cannot ''possibly'' have been a [[coincidence]], because it was so unlikely. Of course, since ''millions'' of people use alternative medicines ''every day'', such coincidences are certain to happen all the time to many people, even if the odds of it happening in one specific instance are low. The odds of a particular person winning the [[lottery]] are extremely low, but somebody always does, regardless. Not to mention that most people get sick many times in the course of their life, and the number of such events may number in the thousands or even tens of thousands, if one includes such mundane things as pain, cuts, nausea, or blemishes. Some improbable coincidence [[Littlewood's law|is bound to happen]] during at least one, if not several, of these many illnesses. It is worth also considering a practitioner's situation. Over the course of her career, she will see countless thousands of patients. Even if her treatments are effective, many of her patients' recoveries (and possible website testimonials) will be unrelated to her treatments, and the large number of cases almost certainly means that some of these coincidental recoveries will be extremely unlikely.

While "case studies" exist within medicine that are essentially anecdotal, these are treated mostly as curiosities or communications between doctors, rather than hard evidence on par with real trials.

===Selection bias===
{{main|Selection bias}}
[[Selective reporting]] is the main reason that popularly disseminated anecdotes will be almost certainly unrepresentative. People tend to focus on the interesting, the unusual, or the powerfully emotive events while ignoring the mundane. After all, no one ever sold a newspaper where the front-page headline read "girl receives new [[vaccine]]; suffers zero complications and is now alive and well". People who believe in alternative medicine may, due to [[confirmation bias]], selectively remember stories that support their belief (which they will, of course, tell everybody about), and ignore those that don't.

Selection bias can have an impressive effect in terms of [[Public relations|PR]]. Let's assume 1 million people (1,000,000) decide to take some ineffective remedy to cure their [[cancer]]. Let's further assume (for the sake of argument) that only 0.1% of this million will experience [[spontaneous remission]] (the actual remission rate, for breast cancer and basal cell carcinoma at least, is closer to 20%<ref>[http://jnci.oxfordjournals.org/content/93/14/1047.full Spontaneous Regression of Melanoma May Offer Insight Into Cancer Immunology], Carrie Printz, JNCI ''J. Natl. Cancer Inst.'' (2001) 93 (14): 1047-1048.</ref><ref>[http://dartmed.dartmouth.edu/spring09/html/disc_remission.php Study reveals surprising cancer remission rate], ''Dartmouth Medicine Magazine''.</ref>), and that 0.3% were [[Misdiagnosis|misdiagnosed]] and so do not actually have cancer. This makes for a total of 0.4%, or 4000 people. Now, 4000 people translates into a lot of testimonials, "CAM cured my cancer" blogs, Internet comments, newspaper articles, and real-life word-of-mouth, so this makes an extremely positive impression for the therapy. But the other 99.6% [[Survivorship bias|died and so are not around to leave any testimonials,]] positive or otherwise. Thus, even in a hypothetical scenario that assumes statistically very low false positive rates, the quantity of false positives is nevertheless numerically quite large. In reality, the number of past and present cancer patients relying on alternative medicine is much larger than one million, and the false positive rate is significantly higher than 0.4%, making the possibility of a given supposedly successful "cancer cure" anecdote being a coincidence even more likely.

Even if a given treatment were effective, anecdotal evidence would still be useless, because many of the cases would be the result of natural improvement, and it would be impossible to tell which anecdotes are cures and which ones are flukes. There is no way to separate the signal from the noise with only single data points suffering from selection bias.

=== Incompleteness and lack of follow-up ===
Another issue arises: anecdotes are usually not complete — they do not necessarily tell the full story. They often cover only the period of the improvement, and anything happening after the punchline or after the publication of the anecdote is not included. In addition, the anecdote-narrator may omit important factors relating to the "cure" (such as, for instance, a patient using conventional medicine together with an alternative remedy). [[Mark Twain]]'s essay ''[http://en.wikisource.org/wiki/Page:In_defense_of_Harriet_Shelley,_and_other_essays.djvu/345 A Majestic Literary Fossil]'', which describes various forms of old-fashioned [[quack]]ery, notes an anecdote:

{{quotebox|Then follows a long account of how a dying woman, who had suffered nine hours a day with an ague during eight weeks, and who had been [[Bloodletting|bled]] dry some dozens of times meantime without apparent benefit, was at last forced to swallow several wads of "Spiders-web," whereupon she straightway mended, and promptly got well. So the sage is full of enthusiasm over the spider-webs, and mentions only in the most casual way the discontinuance of the daily bleedings, plainly never suspecting that this had anything to do with the cure.}}

If, in this case, the fact that the bloodletting was discontinued gets "mentioned only in the most casual way", it is easy to imagine that many anecdotes leave out important facts like this altogether.

A cancer patient named [[Robert O. Young#Kim Tinkham|Kim Tinkham]] went on [[Oprah|''The Oprah Winfrey Show'']] with a proponent of the [[alkaline diet]] and claimed the diet had cured her, but she died of the cancer a few years later. Now, the eventual fate of Tinkham was revealed in news sources because she was famous, but if anonymous (or at least non-famous) authors of online testimonials die of a supposedly "cured" illness, it is not likely that this fact will get published on the website of the original testimonial, or possibly anywhere.

In a case-study involving spontaneous remission, a woman's {{wpl|diabetes}} went away completely, with no treatment, shortly after she woke up from a coma. In a [[CAM]] anecdote, that would be that — cure proven. But since this was a case-study rather than a testimonial, the person's post-recovery situation was observed, and two years later, a mild form of diabetes reappeared.<ref>[http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1578128/?page=1 "Spontaneous Remission of Diabetes"], R. B. Haining and R. G. Haining.</ref> Similarly, in an anecdotal case of a "cured" disease, the disease may later turn out to have been [[Psychosomatic|illusory]].

==Advantages of anecdotal evidence==
My grandma heeded anecdotal evidence and she lived to 106.

{{cquote|When a cook in [Jean] Nicot's household nearly cut off his thumb with a chopping knife, the steward ran for the [[tobacco]] plant and bound the thumb back on; after five or six dressings of the same sort, the wound healed.|||''Medicinal uses of tobacco in history''<ref>http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1079499/</ref>}}

Also, Jeanne Calment, the person with the longest confirmed lifespan (122 years), [[Smoking|smoked]] for almost her entire adult life and nothing bad happened to her.<ref>https://web.archive.org/web/20110418091515/http://extremelongevity.net/supercentenarians/worlds-top-ten-oldest-people-ever/oldest-person-ever-jeanne-calment/</ref><ref>https://web.archive.org/web/20111003152711/http://www.endlesshumanpotential.com/jeanne-calment.html</ref> And besides Calment, there are many centenarians who are heavy smokers.<ref>[http://healthland.time.com/2011/08/05/even-the-long-lived-smoke-drink-and-dont-exercise/ Even the Long-Lived Smoke, Drink, and Don't Exercise]</ref> The composer Jean Sibelius was an alcoholic and smoker for most of his life, and he lived to be 91, even saying: "All the doctors who wanted to forbid me to smoke and to drink are dead."<ref>[http://www.therestisnoise.com/2007/07/sibelius-chapte.html The Rest Is Noise], Alex Ross.</ref> By [[alternative medicine]] logic, this should be indisputable proof that tobacco is actually good for you. As noted by Dan King in his 1861 book ''Tobacco: What it is, and What it does'':

{{cquote|Some are ready to contend that tobacco can have no tendency to shorten life because there are many old persons who use it. This is the drunkards' standing argument, and if it proved anything it would prove quite too much: if we are to conclude that tobacco is harmless because old men are found who use it, then by the same course of reasoning we must conclude that intoxicating liquors are harmless because drunkards sometimes live to old age; and by the same rule it might be shown that nobody had been slain or wounded in battle because some old soldiers still survive. Out of fifty persons bitten by a mad dog several may escape the [[Rabies|hydrophobia]], but such rare instances do not show that there is no danger from the bites of rabid animals. Peculiarity of temperament or idiosyncrasy may enable some constitutions to withstand for many years the pernicious effects of a virulent poison, but such exceptions to a general rule do not prove that poisons are harmless.|||}}

==The power of anecdotes==
Anecdotal evidence is not accepted as proper proof or scientific evidence for good reason, yet they still dominate people's thinking. This is because while facts and figures may be ''true'', they are difficult for people to relate to. Anecdotes have convincing and strong narratives that resonate with people on a key individual level — someone telling an anecdote will lead the listener to think that it just could happen to them too. There are further [[Cognitive bias|biases]] at work. Anecdotes, even implausible ones, can contain a high level of detail. It is this detail that causes people to become involved in the story and to imagine it more clearly; a [[Urban legend|story about a young girl]] won't be nearly as widely spread and accepted as a story about a five year-old girl called Sally from [[Wisconsin]] who was cycling to school one day.

The biases at work in causing people to add disproportionate weight to anecdotes cause major problems during health scares. A major case in point is [[vaccine hysteria]].   
*Hard, scientific and proven (yet dry and difficult to grasp) evidence about the value of vaccines exists on the one side.
*Strong narrative stories told by concerned parents competes on the other side. 
It is often advised in such circumstances that those on the side of [[science]] should also use their own anecdotes to take advantage of their persuasive power. For example, instead of giving statistics of illnesses increasing when vaccination uptake dropped, tell — in great detail — stories about specific deaths that have occurred from vaccine-preventable illnesses. Respectable scientists, however, may be against [[appeal to emotion|such manipulative tactics]].

==See also==
*[[What's The Harm?]] — Anecdotal evidence against alternative medicine
*[[Essay:List of CAM-free anecdotes]] — "Miraculous" recoveries without alternative medicine.
*[[Pragmatic fallacy]]
*[[Magical thinking]]
*[[Exception that proves the rule]]
*[[Credulity]]
*[[Miracle]]
*[[Trepanation]]
*[[Shoehorning]]

==External links==
*[http://www.csicop.org/SI/show/why_bogus_therapies_seem_to_work/ Why Bogus Therapies Seem to Work], Barry Beyerstein
*[http://www.sciencebasedmedicine.org/why-we-need-science-i-saw-it-with-my-own-eyes-is-not-enough/ Why We Need Science: “I saw it with my own eyes” Is Not Enough], [[Harriet Hall]]
*[https://web.archive.org/web/20150412233008/http://rationalblogs.org/rationalwiki/2013/08/10/the-futility-of-anecdotal-evidence/ The Futility of Anecdotal Evidence], The "What's going on at [[RationalWiki]]" blog
*[http://www.gutenberg.org/files/2700/2700-h/2700-h.htm#link2H_4_0004 Homeopathy and Its Kindred Delusions], by Oliver Wendell Holmes Sr., which dwells at some length on the topic of anecdotal evidence
*[http://www.quackwatch.org/01QuackeryRelatedTopics/Cancer/miracles.html Cancer: "Miraculous Recoveries"], [[Quackwatch]]
*[http://oracknows.blogspot.ca/2004/12/understanding-alternative-medicine.html Understanding alternative medicine "testimonials" for cancer cures], [[Orac]]
*[http://scienceblogs.com/insolence/2012/04/16/a-homeopath-lectures-scientists-about-an/ A homeopath lectures scientists about anecdotal evidence], Orac
*[http://scienceblogs.com/insolence/2013/08/20/the-key-difference-between-alternative-medicine-and-evidence-based-medicine/ The key difference between alternative medicine and evidence], Orac
*[http://saveyourself.ca/articles/popular-but-dangerous-quackery.php Popular but Weird and Dangerous Cures], Paul Ingraham
*[https://yourlogicalfallacyis.com/anecdotal Your logical fallacy is anecdotal], [[YLFI]]
*[https://web.archive.org/web/20150515145743/http://www.theonion.com/article/sociologist-considers-own-behavior-indicative-of-l-421 Sociologist Considers Own Behavior Indicative Of Larger Trends], The Onion

==Notes==
{{reflist|group=note}}

==References==
{{reflist|2|80%}}

{{logfal}}

[[Category:Imprecision fallacies]]
[[Category:Language]]
[[Category:Pseudoscience]]
[[Category:Rhetoric]]

Categories: Imprecision fallacies, Language, Pseudoscience, Rhetoric