What Social Science Does and Doesn't Know

Manzi's article on "What Social Science Does or Doesn't Know" explained the limitations of experimentation in the social sciences.

I found this article very interesting because of my background with social science experimentation. Last year in my psychology class, I had to perform an experiment. And while psychology is arguably more testable than criminology or business, the results of psych experiments still encounter the same problems that Manzi describes in all social sciences. I had to deal with false positive claims and the inability to reproduce my results in my psych experiments. Manzi's ability to describe experimentation through several fields in the social sciences supports his argument and allows for very little wiggle room. I was amazed how Manzi's evidence piled up and covered every social science from medicine to business.

I found that Manzi explained perfectly the errs in social science experimentation and how physics is so much easier to state claims. Although Aristotle made wrong philosophical claims, much like economists do today, there are experiments in physics that prove them wrong. There are no experiments that prove really anything for economics, or any other social science.

(to see what I thought was counterintuitive, see the questions at the end)

Manzi used examples of sick sailors, Capital One, and domestic violence in order to give value to his arguments. He showed that experimentation is difficult, but sometimes does work. However, we must be careful of the claims for effectiveness in positive results.

Manzi explains that there are limitations of experimentation in medicine, criminology, and business, because they lack control groups. Manzi shows how "few programs can be shown to work in properly randomized and replicated trials." RFT's, randomized field trials, allow scientists to use random groups as the experiment or control group because the differences in genetics or other factors will cancel out between the groups. Manzi then says that we it is more effective to change the incentives, not the people, in social experiments. And finally Manzi says that there is "no magic" and positive results do not happen by accident.

Manzi ends his article by talking about a future Experimental Revolution, but it seems far away.

We read this article because it is important for economic students to see the problems with social science experimentations and why debates get so heated with no way to solve any of the important questions. Also, we will be the ones probably taking part in this future experimental revolution.

      There were several points where I had to raise an eyebrow to Manzi. First, he seems to contradict himself when he says we must stay skeptical of claims for effectiveness in positive results, but then goes on to say there is "no magic" and positive results lead to improvements. What is Manzi saying and why is he contradicting himself? How does he expect our world to be weary and to support positive claims of effectiveness at the same time? Also, despite his random claim about magic at the end of the article, Manzi shoots down every positive result (that he brings up) and claims that the results are not reproducible, and thus we should keep experimenting. I'm not saying that we should stop experimenting, but this seems ridiculous.

      Also, where is the argument of the person who believes in the accuracy of the positive experiments? How would they support their claims? Obviously Manzi sees a problem with social experiments, despite his random magic claim at the end, so where is this other view point?

      Finally, will we ever find a way where we will be able to have a control and experimental group to find accurate results for the social sciences? Manzi talks about this experimental revolution, but when is it coming and will it even work if we are supposed to be so skeptical of positive experimentation?