From Leiter, and it's comedy gold

Eric Schwitzgebel: "Qualitative Research Reveals a Potentially Huge Problem for Standard Methods in Experimental Philosophy"

Mainstream experimental philosophy aims to discover ordinary people's opinions about questions of philosophical interest. Typically, this involves presenting paragraph-long scenarios to online workers. Respondents express their opinions about the scenarios on simple quantitative scales. But what if participants regularly interpret the questions differently than the researchers intend? The whole apparatus would come crashing down.

Kyle Thompson (who recently earned his PhD under my supervision) has published the central findings of a dissertation that raises exactly this challenge to experimental philosophy. His approach is to compare the standard quantitative measures of participants' opinions -- that is, participants' numerical responses on standardized questions -- with two qualitative measures: what participants say when instructed to "think aloud" about the experimental stimuli and a post-response interview about why they answered the way they did.

Kyle's main experiment replicates the quantitative results of an influential study that purports to show that ordinary research participants reject the "ought implies can" principle. According to the ought-implies-can principle, people can only be morally required to do what it is possible for them to do. Thompson replicates the quantitative results of the earlier experiment, seeming to confirm that participants reject ought-implies-can. However, Thompson's qualitative think-aloud and interview results clearly indicate that his participants actually accept, rather than reject, the principle. The quantitative and the qualitative results point in opposite directions, and the qualitative results are more convincing.

It was always obvious what was going on.