Critical thinking
How to call Bullshit
This article is a summary of strategies and tips from the Calling Bullshit course by University of Washington professors Carl T. Bergstrom and Jevin West which you can do online here for free on YouTube Calling Bullshit website Calling Bullshit videos Most of the calling bullshit course focuses on spotting bullshit within data, graphs and […]
This article is a summary of strategies and tips from the Calling Bullshit course by University of Washington professors Carl T. Bergstrom and Jevin West which you can do online here for free on YouTube
Most of the calling bullshit course focuses on spotting bullshit within data, graphs and statistics, however in this article I’ll simply be focusing on how to spot bullshit, fake news, lies etc. from advertisers, politicians and the media.
Why the calling bullshit course began
There is so much bullshit everywhere. We are drowning in bullshit.
Politicians are unconstrained by facts. Science is conducted by press release. Higher education rewards bullshit over analytic thought. Startup culture elevates bullshit to high art. A generation ago advertisers fed us bullshit straight up, now they wink conspiratorially and invite us to join them in seeing through all the bullshit — and take advantage of our lowered guard to bombard us with bullshit of the second order.
Bullshit pervades all aspects of our lives.
What is bullshit?
Bullshit can present itself as highbrow nonsense couched in big words and fancy rhetoric, but more and more we see it presented instead in the guise of big data and fancy algorithms, quantitative, statistical, and computational forms of bullshit.
Alberto Brandolini, a computer programmer, gives the Bullshit Asymmetry Principle:
“The amount of energy necessary to refute bullshit is an order of magnitude bigger than to produce it.” – Alberto Brandolini
It’s not just that bullshit is hard to refute, it also spreads faster. Falsehoods travel faster than truth.
“Falsehood flies, and the truth comes limping after it” – Jonathon Swift, 1710
Lies vs bullshit
“One of the most salient features of our culture is that there is so much bullshit. Everyone knows this. Each of us contributes his share. But we tend to take the situation for granted.” – Harry Frankfurt, On Bullshit
What’s the difference between a liar and a bullshitter?
According to American Philosopher Harry Frankfurt, in his essay, On Bullshit:
A liar knows the truth and is trying to convince us of something different.
A bullshitter either doesn’t know the truth, or doesn’t care – and is just trying to be persuasive.
In other words:
The liar knows what’s going on, and is deliberately trying to deceive you.
The bullshitter either doesn’t know the truth, or just doesn’t care. The bullshitter isn’t trying to convince you of a lie, they’re just trying to convince you that they know what they’re talking about. They’re trying to be persuasive, impressive.
However, philosopher Gerald Cohen calls bullshit on On Bullshit
A claim is either bullshit, or it isn’t. The mental state of the person that made it should be irrelevant.
How to spot bullshit
Ask yourself these 3 questions
Start by asking yourself these three questions:
- Who is telling me this?
- How do they know it?
- What’s in it for them? What’s their motivation?
If a claim seems too good to be true it probably is
If a claim seems too good (or too bad) to be true, it probably is, so really dig in and question.
Beware of confirmation bias
When you hear claims that agree with things you already believe, challenge yourself to question them. Dig in and be skeptical of those too. Just as much as you would if the claim said something you don’t believe.
Have multiple working hypotheses
Try to come up with alternative explanations, multiple working hypothesis. This is one of the most important habits of mind you can have.
Think about orders of magnitude
You might read an article or headline that says that $70 Million dollars a year is lost to food stamp fraud, but when you do the math and see it in the context of the big picture, you might find that it turns out to be only 0.2% of expenditures (typical losses in retail would be in the order of 1.5% to 3%)
Correlation vs Causation
Two variables are correlated when knowing the value of one gives you information about the likely value of the other.
Two states are causally related when one state influences the other via a cause-and-effect process.
Correlation does not imply causation
These graphs from Tyler Vigen Spurious Correlations illustrate the point nicely
Post hoc propter ergo hoc fallacy
We want to avoid the post hoc propter ergo hoc fallacy (Latin: “after this, therefore because of this”)
Just because B followed A, that doesn’t mean A caused B.
If correlation isn’t causation, how do we determine causality?
You want to manipulate your experiments, change thing A to see if it affects thing B.
Garbage in, garbage out
Even if you don’t know how an algorithm or statistical test works, you can spot bullshit by simply looking carefully at what goes in and what comes out.
- Are the data and comparisons reasonable?
- How did they get that data?
- Is it pertinent to the claims being made?
- What’s the output?
- Does the output even make sense?
- If it does make sense, does it support or refute the claim that someone is making?
Assignment: Bullshit created vs Bullshit encountered
- Track bullshit for 1 week:
- What you encounter
- What you create
- What you debunk
- Represent it as you see fit (drawings, charts, text)
Think about all the bullshit you’re fed.
Think about all the bullshit you create.
- When am I bullshitting others and putting bullshit out there?
- When am I debunking bullshit?
- When am I successful in debunking bullshit?
- Am I debunking as much bullshit as I’m creating?
Science is amazing, but there’s a lot of bullshit
Beware of “news” stating that something is “scientifically proven”.
There are lots of bullshit articles out there pretending to be science:
8 Scientific studies that prove beer is good for you
Eating bogies is good for teeth and overall health scientists conclude
Questions to ask about scientific studies:
- How were the individuals/study subjects chosen?
- How was the experiment designed?
- How was the analysis done?
- In what condition?
How to know if a paper is legit
“Any scientific paper can be wrong. No matter where it is published, no matter who wrote it, no matter how well supported its arguments, any scientific paper can be wrong. Every hypothesis, every set of data, every claim and every conclusion is subject to re-examination in light of future evidence.” – Carl T. Bergstrom and Jevin West
- Consider the source. If there was a substantial peer review process the paper went through, it’s more likely to be legit
- Peer review, while an important part of the scientific process, doesn’t guarantee that published papers are correct. Peer reviewers carefully read a paper to make sure its methods are reasonable and its reasoning is logical. They make sure a paper accurately represents what it adds to the literature, and that its conclusions follow from its results. They suggest ways to improve a paper, and sometimes recommend additional experiments. But peer reviewers can make mistakes, and more importantly, peer reviewers cannot possibly check every aspect of the work.
- Where has the paper been published? A quick way to evaluate the legitimacy of a published paper is to find out about the journal in which it is published. A number of websites purport to rank journal quality or prestige, typically ascertained based on citations. Highly cited journals are thought to be better than their seldom-cited competitors. If the paper is such a big deal, why is it in a low tier journal?
- Has the paper been retracted or seriously questioned? While retractions are uncommon, it can be a good idea to check for a retraction or correction before staking too much on the results of a scientific paper. The easiest way to do this is simply to check the paper on the publisher’s website or better yet, if it is a biomedical paper, on PubMed. If a retraction has occurred it will be clearly marked as such in these places.
- Is the publisher legitimate or “predatory”? A journal should meet at least one of the following criteria:
- Published by a known, reputable publisher
- Sponsored by a known reputable scholarly society
- Listed in the JCR or Scopus
- Peter Burns has developed a good infographic for Allen Press’s FrontMatter, highlighting the features of a journal’s website that offer clues as to whether the journal is legitimate or predatory
- Who are the authors? In principle, science is utterly egalitarian. It doesn’t matter who has an idea, it only matters whether that idea offers a better representation of nature. In this sense, the identity of a paper’s authors should not matter in the least. In practice, however, there is no reason why we should not approach a paper as a good Bayesian would: taking in account all prior information when making judgements.
- Are the authors well-established? On one hand, we believe that the best ideas in science often come from graduate students and postdocs, and we believe that people pay too much attention to famous names. On the other hand, and as much as it pains us to say this, with all else equal a paper from a researcher with an extensive publication record and strong reputation is somewhat more credible than a paper from authors who have not published other scholarly work.
- Are the authors experts in the specific area treated in the paper? While many good papers have been written by newcomers to a field, we feel that all else equal, a paper is more likely to be reliable when written by authors with substantial experience in the area.
- Do the authors have a vested interest in the results they are publishing?
Carl Bergstrom and Jevin West: How to know if a paper is legit
Fake news
“If a nation expects to be ignorant and free, in a state of civilization, it expects what never was and never will be.” – Thomas Jefferson
How to spot fake news
Carl and Jevin recommend the following for spotting fake news:
Politifact: 8 steps to spotting fake news
Fact Check: How to spot fake news
Note: Don’t spread articles until you’ve done some digging!
Four Rules for Calling Bullshit
- Be correct
- Be charitable
- Be clear
- Admit fault
Rule #1: Be correct
Have all the information you need. Don’t just read the headline, actually read and understand the article or paper, before you comment on it or call it bullshit. Go to the original cited article, read the original study.
Double-check your facts. Make sure you’re getting your information from at least two reliable sources, that aren’t just copied from one another. Make sure your data and facts are right.
Run it by a friend or colleague before you go public with a criticism because we all miss things.
Rule #2: Be charitable
Consider the possibility that you are the one who is confused. Even if you’re right, you don’t have to be a jerk about it. Don’t call bullshit until you’re sure that somebody is not you.
“When I read something, and I think, “Boy, somebody is an idiot!” I’m usually right. But the “somebody” is usually me. Because I don’t understand what’s being said the first time through.” – Carl T. Bergstrom
Don’t impute malice when stupidity is sufficient. There are a lot of things that corporations and the news media do that aren’t necessarily malicious, it’s not that they’re necessarily trying to trick you, or force you into some agenda. A reasonable fraction of it is simply that somebody made a mistake, done or said something stupid. It’s not that they’re necessarily trying to do this bad thing to you.
Don’t impute stupidity when understandable error is sufficient. We all make mistakes. When someone writes something wrong, they may not even be an idiot, they may be a perfectly intelligent person that’s written something wrong. Attack the claim not the person.
Rule #3: Be clear
Think as hard about how to present your argument, as you did about how to prove your claim in the first place.
You want to put at least as much effort into designing, shaping and crafting your argument, and making it as simple, clear and persuasive, as you have into having figured out that it was bullshit.
Rule #4: Admit fault
If you’re calling bullshit, arguing about things, writing scientific papers, or anything else, you’re going to make mistakes.
We all do.
When it happens, own your mistakes.
Don’t double down on your stupidity. It blows your credibility. It wastes everybody’s time.
Debunking myths
You can’t just debunk the bullshit and myths that others believe by shoving more information into their brains.
If you’re not delicate when you address your friends and family about myths, you’ll encounter the backfire effect. You can try harder and harder, and push more information and complex arguments at them, but even when you correct bullshit and misinformation, once a belief has settled in the brain, it’s really hard to overcome it.
How to debunk myths
- Start with core facts. Don’t start with the myth. If you start with the myth, and then you give the alternative explanation and the facts, and then you ask people 30 days later what they remember, you’ll find that they remember the myth even more
- Don’t refer to the myth. If you need to reference it, precede it with explicit warnings. If you have to refer to the myth because of the way you’re delivering your argument, precede it with an explicit warning e.g. “Beware…X could be misleading information”
- Fill in the gap with alternative explanations.
- Find counterexamples. If someone claims that A implies B, find a case in which A is true, but B is not
- Be clear and concise with your alternative explanation/s. More is not always better.
- Use graphics and visualizations where possible. They’re more likely to be remembered by your audience.
Calling Bullshit Resources
Calling Bullshit tools and tricks