Thursday, March 28, 2013

Critique of Politifact's Truth-o-meter

Politifact.org uses the following ratings in its Truth-o-meter:

True – The statement is accurate and there’s nothing significant missing. 

Mostly True – The statement is accurate but needs clarification or additional information. 

Half True – The statement is partially accurate but leaves out important details or takes things out of context.

Mostly False – The statement contains some element of truth but ignores critical facts that would give a different impression. 

False – The statement is not accurate. 

Pants on Fire – The statement is not accurate and makes a ridiculous claim. 

The reason for having these gradations is that "- especially in politics - truth is not black or white". 

I agree with this. And I appreciate the research that PolitiFact does. It is all I need in order to gain clarity for myself on things that are said in the media. But these categories don't seem to be helping much. 

How can you decide between Mostly True and Mostly False? What is the difference between needing clarification or additional information and ignoring critical facts that would give a different impression? When Romney was accused of being "director of a company that stole millions from Medicare", the claim was judged as "Mostly True": "needing additional clarification". According to PolitiFact, "His firm bought the company that was later found guilty of fraud, but Romney was not running the show while crimes were being committed."  Why doesn't that count as ignoring critical facts that would give a different impression? These criteria seem rather bogus.

Furthermore, some of the examples that are "Mostly False" are merely unknown, such as the claim that 40% of gun sales are done without a permit -- a number that is based on old research. We don't really know whether it contains some element of truth, and it's not like critical facts are being ignored here. We have uncertainty about the facts in this case.

Some of the cases are ambiguous, and true under one interpretation and false under another, such as the claim that there are twice as many gun stores as McDonald's -- it depends on what counts as a "gun store". No critical facts are being ignored here; it's just a matter of what the statement means. We have uncertainty about the interpretation of the statement in this case.

The interesting cases for a linguist such as myself are the ones where the statement itself is true, but it carries a conversational implicature that is false. A conversational implicature is an inference that a hearer can draw based on the fact that a speaker said something, crucially using the assumption that the speaker is adhering to certain norms of conversation. Read Grice (1975), "Logic and Conversation" if you don't know what I'm talking about. This is a class of misleading statements that should have its own category label, something like "True statements with false implicatures".

Here's an example with a conversational implicature.  There was a Republican attack ad that said that Chuck Hagel advocates nuclear disarmament for the US, and wants to "back down", despite a growing nuclear threat from our scary enemies. FactChecker.org fact-checked that claim, and found that indeed, Chuck Hagel had co-authored a document for an organization whose mission includes a reduction in nuclear stockpiles worldwide. So yes, at least in a sense, the claim is true. But they also point out that the document does not advocate unilateral disarmament, and says in fact that this would be a "less good option". The ad suggests that Chuck Hagel would be in favor of unilateral disarmament, and that is not true. So technically, what the ad says is true, but it implies something that is not true through a conversational implicature. You could say it's true, but misleading.

But the term "misleading" is still too vague. Here is another example of somebody saying something misleading. Obama said that "there are millions of women all across the country who rely on Planned Parenthood for not just contraceptive care; they rely on it for mammograms, for cervical cancer screenings." FactChecker.org assessed the statement as Half True, saying "The critics have a point that Planned Parenthood does not actually provide mammograms; instead, the organization refers patients who need them offsite. Supporters of Planned Parenthood also have a point that the organization serves as an important link between some female patients and the mammograms a doctor determines they need." I would analyze this case as follows. Obama's statement could be interpreted in two ways. Either it means that Planned Parenthood enables people to get mammograms (in which case it is true), or it means that Planned Parenthood provides people with mammograms (in which case it is false). The latter interpretation might be somewhat more likely, and therefore the statement is misleading.

But Obama's statement is misleading in a different way from the Republican attack ad. In Obama's case, the statement is ambiguous, and true or false depending on how you interpret it. In the case of the attack ad, it's relatively clear what the statement is saying, and that it is true, but there is a conversational implicature going along with it that is false. So "misleading" is too vague because it doesn't distinguish between these two cases.

So I would propose the following "shades of grey" between True and False:

  • Unverifiable (needs further research)
  • Ambiguous (true on one interpretation, false on another)
  • Technically true but misleading (due to a false implicature)