Too many people rely on anecdotal evidence (personal experience or cherry picked examples) to assess if something is true, and I don't like it.
To me, everything is a model. All the ways in which we view the world, or our explanations for various phenomena like behaviour, are merely models. The techniques we use to estimate weather patterns are models. The techniques we use to estimate group dynamics are models. All estimates are models. There are a number of ways to consciously build models. You could use anecdotal evidence. You could also use critical reasoning.
There's a famous phrase that goes, "all models are wrong, some are useful". I like this because it feeds into what scientists do. Science is not about finding the 'truth'. It can be about the pursuit of the truth, but the truth might never be known. Therefore, all you do is continue to build better approximations of the truth, or better models to explain and predict phenomena, for both academic and practical purposes. This is what science does. Science is essentially a mix of critical reasoning and research techniques combined with domain knowledge. The sciences - Biology, Psychology, Chemistry, Physics - are merely fields of knowledge, domains that revolve around certain interest areas. Of course there is overlap. But these are not sciences because they encompass domains of study. That's half of it. They are sciences because they use critical reasoning techniques to investigate and build models that approximate the truth.
There's a famous phrase that goes, "all models are wrong, some are useful". I like this because it feeds into what scientists do. Science is not about finding the 'truth'. It can be about the pursuit of the truth, but the truth might never be known. Therefore, all you do is continue to build better approximations of the truth, or better models to explain and predict phenomena, for both academic and practical purposes. This is what science does. Science is essentially a mix of critical reasoning and research techniques combined with domain knowledge. The sciences - Biology, Psychology, Chemistry, Physics - are merely fields of knowledge, domains that revolve around certain interest areas. Of course there is overlap. But these are not sciences because they encompass domains of study. That's half of it. They are sciences because they use critical reasoning techniques to investigate and build models that approximate the truth.
Where does anecdotal evidence come in? Anecdotal evidence is a first step towards building a model, but not evidence for the model. Anecdotal evidence is the presence of something interesting that requires further study. You see a ghostly white figure at night. You have no idea what you are looking at. You investigate, you make a hypothesis and attempt to verify it. Things can get a bit shaky if you skip the investigation and rush to make a claim, because anecdotal evidence could be due to a number of causes, not just the one you have in mind. False positives abound. This is why it is important to treat anecdotal evidence as a first step only. It would be disastrous to claim something as fact based on personal observation, and then find out that your claim is wrong because you didn't properly investigate the matter.
Let's take some examples. The claim that God is real. There are various types of evidence for this claim. One is prayer, a type of anecdotal evidence. I pray for something, something happens, therefore God is real. Anecdotal evidence like prayer cannot be evidence for the existence of God till it is verified. For every anecdotal claim of prayer working, there could be another for it being useless. To verify if prayer works, you would have to experimentally demonstrate its effectiveness. This is called falsifiability. Note that this is neither proving nor disproving the existence of God. This is not the question at hand for the scientist. It might be the question at hand for the person claiming God's existence and using prayer as an example, but for the scientist the investigation only concerns the effectiveness of prayer. A scientist who demonstrates that prayer is useless is not proving or disproving the existence of God. He or she is merely verifying a specific claim. This is important to remember. Science is not always concerned with the big questions. It is merely a tool to verify claims or existing models. After all, prayer is a model of how the world works. A scientist can spend his or her entire life falsifying such claims. This would get us nowhere if the claims were spurious to begin with. This is why anecdotal evidence should not be used to claim something. Because there are more reliable ways to build models.
[This is why proving or disproving the existence of God is a futile activity. No one knows exactly why the concept of God came about. We have theories. But nothing that seems to be founded in verifiable evidence. There is a lot of anecdotal evidence, but upon verification, a lot of it does not hold up to scrutiny. This is not to say that any of the thousands of Gods do not exist, or that people are wrong in believing in them. Science cannot falsify something that was made up to begin with, or is currently too difficult to verify. It can only analyse the evidence and show over time how improbable something is, using existing methods. True falsifiability is impossible. Which is why we will never be able to disprove the existence of the Loch Ness monster either.]
Here's another example. Psychometric tests like MBTI. HR professionals love them. But the data from meta analyses picks holes in the test's reliability and validity. But HR professionals who have used these tests swear by them. One person I spoke to even compared it to the accuracy of a horoscope while praising it (I doubt he was trying to be ironic). This kind of reliance on anecdotal evidence to back something, is used as a model by a lot of people, just like people use prayer as a model. Why do they use it when there are scientific techniques that discredit these models? I have no idea. Maybe people are ignorant. Maybe they find it easier to act on someone else's recommendation or 'try it yourself first' advice rather than doing personal research. Maybe they think that discrediting one model will mean discrediting a larger model that they have more of an emotional investment it. Maybe they already choose to believe in something to make themselves feel better. Maybe creating a faulty but useful model works for them. Maybe the model's degree of usefulness wins over the fact that it is wrong.
Which is interesting because of what I said earlier - all models are wrong, some are useful. Let's say human sacrifice to appease the weather Gods is supported by anecdotal evidence i.e. a group of people practice human sacrifice and choose to notice only when the weather changes for the better, convincing themselves of a correlation between the two. They of course ignore instances when sacrifice does not affect the weather, attributing it to human fault or God being angry with them, or it all being a part of God's larger plan. Now let's say hypothetically that this model/belief is the only thing keeping this society stable.
Note that science isn't always concerned whether the effect is real or not, or if belief in it should continue. Yes, assumptions are faulty. Correlations abound in large amounts of data. They're a function of statistical noise. Experimentation should verify the probability of the correlation. But even if it finds that the correlation/belief/model of human sacrifice for better weather is wrong, it doesn't erase the fact that it is useful. Now replace human sacrifice with belief in God, or MBTI. These models might work in certain contexts. Belief in God helps people in certain contexts. Belief in aliens might just help society. I have no idea. MBTI might be useful in certain contexts. Neither of these models might be correct, but they can be useful. If MBTI works for you, then great, use it. But that doesn't mean it does what it claims to do, which is why you wouldn't be right in recommending it to me. Which is why people need to look at the evidence to verify if a model is good for them, and not rely solely on anecdotal evidence, or else risk disappointment.
In summary,
1. Anecdotal evidence can be a good first step to further research.
2. If you notice something interesting, collect data, find patterns, make a hypothesis and verify it. Then make a claim.
3. Your claim is your model. It can only be built on the elements in point 2.
4. Anecdotal evidence by itself cannot be used to build models.
5. If someone builds a model that approximates what they think is the truth, question their assumptions and verify the evidence.
6. If their model is build on anecdotal evidence (personal or cherry picked examples), reject the model for being incomplete.
7. Their model is not necessarily completely wrong but it is pointless to consider something correct if it hasn't been verified, even if it is useful.
8. A model's usefulness does not necessarily reflect its correctness.
