How Scientific Can Philosophy Be? or, Scientific Progress Really Does Go Boink On Occasion

This was originally going to be a comment; it only seemed to be fair to talk directly to someone when giving feedback. But since the site wouldn't let me post it (no idea why)...here it comes.

A study done recently claims that people tend not to assume an objective standard in morality. The writers of the study argue that, contrary to the view of many philosophers, most normal people have a fairly relativistic view of ethics. The conclusion isn't what I have a beef with; it's the method of arriving at that conclusion that irks me.

The experiment is as follows. Subjects were asked to imagine two people who are asked to evaluate a situation involving killing. One of these hypothetical people is a college student, the other is (a) also a college student, (b) from an Amazonian culture, or (c) a cruel extraterrestrial race. In this imaginary scenario, one person thinks the act is morally permissible, the other thinks it's morally wrong. The subjects were then asked whether the hypothetical evaluators would agree or disagree in terms of their moral judgment.

Depending on how closely related the two speculative people are, the subjects responded variously. In the case of (a) two college students from the same campus, they were imagined to agree. In (b) a college student and an Amazon, they were imagined to be less in agreement. In the case of (c), a college student and Evil ET, the imagined level of agreement was even lower.

In other words, a few college students in philosophy courses believe that between two individuals, the level of agreement is inversely proportional to the closeness of the similarity of those two individuals. The greater the difference between cultures, the greater the difference of agreement on ethical issues.

I'm afraid the experiment seems quite flawed. Leaving aside the more general problem of the extent to which statistics should determine how we actually think about ethical questions (doesn't every logic textbook consider the argumentum ad populum to be a fallacy? Har-har); leaving aside the even more serious problem of the extent that those numbers should influence how we ought to think about such problems, there are serious difficulties with the experiment presented here.

Basically, the actual sample is not representative - of anything, let alone of "ordinary folks." All the participants were students taking philosophy courses at one college. Hardly the average Joe. And it was a small sample at that: only 223. Let's put that into perspective. In 2006-7 alone there were 11,969 philosophy BAs awarded. This comes from a total number of 1,524,092 BAs awarded in that year. (Figures taken from the National Center for Educational Statistics.) The study doesn't specify whether all the subjects were philosophy majors, but for the sake of perspective and generosity, let's assume they are. And because we're feeling extra generous today, let's assume that all the undergrads completed their studies.

In other words, the sample consisted of 1.45 x 10^-4 of the total undergrad population. This out of an estimated general population of 301.6 million. Assuming the number didn't rise between then and now, this means the sample would only account for less than 2% of the philosophy undergrad population, which itself constitutes less than 1% of the total undergrad population. In the US. (The Census Bureau's estimate for 2007 can be found here; world population figures taken from the Population Reference Bureau.)

That means the study yields a sample of 7.4 x 10^-7 of the US population, or 3.38 x10^-8 of the world's population - in a snapshot of the world's history. Representative it ain't.

Now I confess to knowing little about what experimental philosophy is up to. But if it's trying to be more scientific about philosophy by collecting data, it has to be more scientific in its data collection than is presented here. If a physicist published such dismally designed research, she'd be laughed out of court. Philosophers have been criticized for their "unscientific armchair theorizing." Experimental philosophy is attempting to use scientific methods of gathering information. Put simply, it's doubly important for philosophers to do the work right: to curry favor with scientists and the broader community that esteems natural science so highly. Take a tip from them - get bigger, more representative samples.