Tuesday, June 26, 2012

The Fundamental Uncertainty of Science

While I have not had much time for the mini data-gathering/research projects that I usually try to post on this blog, I found the recent flurry over Professor Jacqueline Stevens' New York Times editorial "Political Scientists are Lousy Forecasters" (and the follow-up on her blog) worth commenting on a bit more.

The political science blogosphere has since responded in full-force (and snark). I agree entirely with the already stated criticisms and will try not to repeat them too much here. The editorial is at best a highly flawed  and under-researched critique of quantitative political science and at worst a rather cynical endorsement of de-funding all NSF political science programs on the grounds that the NSF tends to fund studies using a methodological paradigm that Professor Stevens does not favor. I'll err on the side of the former.

But one quote from the piece did irk me quite a bit:
...the government — disproportionately — supports research that is amenable to statistical analyses and models even though everyone knows the clean equations mask messy realities that contrived data sets and assumptions don’t, and can’t, capture. (emphasis mine)
This statement is on-face contradictory. The entire point of statistical analysis is that we are uncertain about the world. That's why statisticians use confidence levels and significance tests. The existence of randomness does not make all attempts at analyzing data meaningless, it just means that there is always some inconclusiveness to the findings that scientists make. We speak of degrees of certainty. Those who use statistical methods to analyze data are pretty clear that none of their conclusions are capital-T truths and the best political science tends to refrain from any absolute statements. Indeed, this is a reason for why a gap tends to exist between the political science and the policymaking communities. Those who enact policy want exact and determinate guidance while political scientists are cautious about making such absolute and declarative statements. It is depressing to see these sorts of caricatures of quantitative methods being used to denounce the entire field. Simply put, just because physics is very quantitative and physics appears to describe very clean and determinate relationships does not mean that all uses of math in social science result in only simple, exact and absolutely certain conclusions.

But putting aside that highly inaccurate picture, Professor Stevens' definition of what constitutes scientific knowledge is remarkably limiting. Prof. Stevens is staking out a very extreme position by implying that the existence of randomness - "messy realities" as she calls it - makes all attempts at quantification meaningless.  She argues for a very radical version of Popper's philosophy of science, positing that any theory should be considered falsified if it is contradicted by a single counter-example. It's unfortunate that Prof. Stevens glosses over the extensive philosophical debate that has followed in the eight-or-so decades after Popper, but this is inevitable given the space of a typical NYT column. Nevertheless, it is very disappointing that the OpEd gives the impression that Popperian falibilism is the gold standard of scientific method and the philosophy of science, when in fact, the scientific community has moved far beyond such a strict standard for what constitutes knowledge. While I won't go into a full dissection of Popper, Kuhn, Lakatos, Bayesian probability theory, and so on, it suffices to say that Stevens' reading of Popper would discount not only political science, but most modern sciences. Accounting for and dealing with randomness is at the heart of what so many scientists in all disciplines do.

By rejecting the idea that probabilistic hypotheses could be considered "scientific," Professor Stevens is perpetuating another caricature - one of science as a bastion of certitude. It's a depiction that resonates well with the popular image of science, but it is far from the truth. I'm reminded of a quote by Irish comedian Dara O Briain:
"Science knows it doesn't know everything, otherwise, it would stop."
All science is fundamentally about uncertainty and ignorance. Knowledge is always partial and incomplete. There was actually an interesting interview with neuroscientist Stuart Firestein on NPR's Science Friday on this topic a few weeks back, where he offered this valuable quote:
...the answers that count - not that answers and facts aren't important in science, of course - but the ones that we want, the ones that we care about the most, are the ones that create newer and better questions because it's really the questions that it's about.
Ultimately, I would argue that probabilistic hypotheses in the social sciences still have scientific value. Events tend to have multiple causes and endogeneity is an ever-present problem. This does not automatically make systematic, scientific and quantitative, inquiry into social phenomena a futile endeavor. Making perfect predictions the standard for what is "science" would dramatically constrain the sphere scientific research. (See Jay Ulfelder's post for more on predictions). Climate scientists constantly debate the internal mechanics of their models of global warming - some predict faster rates, some slower. Does this mean that the underlying relationships described by those models (such as between CO2 concentration and temperature) should be ignored because the research is too "unsettled"? While deniers of climate change would argue yes, the answer here is a definite no. 

Or to take an example from the recent blog debates about the value of election forecasting models. Just because Douglas Hibbs' "Bread and Peace" model (among other Presidential election models) does not perfectly predict President Obama's vote percentage in November, does not mean that we can learn nothing from it. One of the most valuable contributions of this literature is that systemic factors like the economy are significantly more relevant to the final outcome than the day-to-day "horserace" of political pundits.

What should be said, then, about Prof. Stevens concluding suggestion that NSF funds be allocated by lottery rather than by a rigorous screening process? Such an argument could only be justified if there were no objective means to distinguish what is and is not "scientific" research. If the criteria for what passes for real political science is simply the consensus of one group of elites, then from the standpoint of "knowledge," there is no difference between peer review and random allocation. This in fact would be the argument made that Thomas Kuhn, Popper's philosophical adversary, made about all science. But while Kuhn's criticism of a truly "objective" science was a useful corrective to 20th century scientific hubris, it too goes too far in this case, justifying an anything goes attitude towards scientific knowledge that is all too dangerous. Penn State Literature Professor Michael Bérubé' wrote a rather interesting article on this exact topic as applied to science at-large, noting the worrying congruence of the highly subjectivist approach to "science studies" adopted by some in leftist academia and the anti-science rhetoric of the far-right.
But now the climate-change deniers and the young-Earth creationists are coming after the natural scientists, just as I predicted–and they’re using some of the very arguments developed by an academic left that thought it was speaking only to people of like mind. Some standard left arguments, combined with the left-populist distrust of “experts” and “professionals” and assorted high-and-mighty muckety-mucks who think they’re the boss of us, were fashioned by the right into a powerful device for delegitimating scientific research. For example, when Andrew Ross asked in Strange Weather, “How can metaphysical life theories and explanations taken seriously by millions be ignored or excluded by a small group of powerful people called ‘scientists’?,” everyone was supposed to understand that he was referring to alternative medicine, and that his critique of “scientists” was meant to bring power to the people. The countercultural account of “metaphysical life theories” that gives people a sense of dignity in the face of scientific authority sounds good–until one substitutes “astrology” or “homeopathy” or “creationism” (all of which are certainly taken seriously by millions) in its place.  
The right’s attacks on climate science, mobilizing a public distrust of scientific expertise, eventually led science-studies theorist Bruno Latour to write in Critical Inquiry:
Entire Ph.D. programs are still running to make sure that good American kids are learning the hard way that facts are made up, that there is no such thing as natural, unmediated, unbiased access to truth…while dangerous extremists are using the very same argument of social construction to destroy hard-won evidence that could save our lives. Was I wrong to participate in the invention of this field known as science studies? Is it enough to say that we did not really mean what we meant? Why does it burn my tongue to say that global warming is a fact whether you like it or not?  
Why can’t I simply say that the argument is closed for good? Why, indeed? Why not say, definitively, that anthropogenic climate change is real, that vaccines do not cause autism, that the Earth revolves around the Sun, and that Adam and Eve did not ride dinosaurs to church?
In the end, Bérubé calls for some sort of commensurability between the humanities and sciences, and I think this kind of coming together that is actually becoming the norm in political science academia, particularly as political theorists and quantitative political scientists still tend to fall under the same departmental umbrella:
So these days, when I talk to my scientist friends, I offer them a deal. I say: I’ll admit that you were right about the potential for science studies to go horribly wrong and give fuel to deeply ignorant and/or reactionary people. And in return, you’ll admit that I was right about the culture wars, and right that the natural sciences would not be held harmless from the right-wing noise machine. And if you’ll go further, and acknowledge that some circumspect, well-informed critiques of actually existing science have merit (such as the criticism that the postwar medicalization of pregnancy and childbirth had some ill effects), I’ll go further too, and acknowledge that many humanists’ critiques of science and reason are neither circumspect nor well-informed. Then perhaps we can get down to the business of how to develop safe, sustainable energy and other social practices that will keep the planet habitable.

Tuesday, June 5, 2012

Is There a Role for International Institutions in Regulating "Cyberweapons"?

David Sanger's extensive New York Times piece about the United States and Israel's covert cyberwarfare operations on Iran's nuclear facilities is the first article I've seen that explicitly confirms the two countries' involvement in Stuxnet's development. But this revelation isn't particularly surprising. Given the virus' complexity and purpose, the list of possible developers was rather short. Rather, what I found most interesting was this section towards the end:
But the good luck did not last. In the summer of 2010, shortly after a new variant of the worm had been sent into Natanz, it became clear that the worm, which was never supposed to leave the Natanz machines, had broken free, like a zoo animal that found the keys to the cage. It fell to Mr. Panetta and two other crucial players in Olympic Games — General Cartwright, the vice chairman of the Joint Chiefs of Staff, and Michael J. Morell, the deputy director of the C.I.A. — to break the news to Mr. Obama and Mr. Biden. 
 An error in the code, they said, had led it to spread to an engineer’s computer when it was hooked up to the centrifuges. When the engineer left Natanz and connected the computer to the Internet, the American- and Israeli-made bug failed to recognize that its environment had changed. It began replicating itself all around the world. Suddenly, the code was exposed, though its intent would not be clear, at least to ordinary computer users.  
 The question facing Mr. Obama was whether the rest of Olympic Games was in jeopardy, now that a variant of the bug was replicating itself “in the wild,” where computer security experts can dissect it and figure out its purpose.  
 “I don’t think we have enough information,” Mr. Obama told the group that day, according to the officials. But in the meantime, he ordered that the cyberattacks continue. They were his best hope of disrupting the Iranian nuclear program unless economic sanctions began to bite harder and reduced Iran’s oil revenues. 
Within a week, another version of the bug brought down just under 1,000 centrifuges. Olympic Games was still on.
The excerpt highlights one of the unique and troubling aspects of "cyberweapons" - their use against adversaries permits their proliferation. Despite all of the effort at keeping Stuxnet both hidden and narrowly tailored, the virus escaped into the public and its source code is open to be analyzed by pretty much everyone. While competent coding can make it difficult to reverse engineer and re-deploy the virus against other targets without a significant investment of time and resources, it's still a distinct possibility. Cyberweapons create externalities - side-effects that don't directly affect the militaries using them, but can have spill-over consequences on other sectors of society. For example, a SCADA worm like Stuxnet which targets industrial control systems could theoretically be re-targeted at civilian infrastructure like power or manufacturing plants.

Certainly most governments using cyberwarfare will likely want to limit these externalities since they do create an indirect threat (such as non-state actor attacks on critical infrastructure). This is evidenced by the fact that the U.S. and Israel not only tried to designed Stuxnet and its ilk to be difficult to detect, but also to have very tailored aims. The virus was designed to work on the specific reactor designs possessed by Iran, thereby somewhat limiting the initial damage of a leak (imagine what would have happened were Stuxnet to deploy its "payload" on all computer systems that it lands on). Nevertheless, these externalities exist so long as governments with the capacity to do so continue to use cyber espionage and attacks. The logic of collective action suggests that governments are also unlikely to unilaterally refrain altogether from utilizing these technologies since a blanket ban would be both impossible and entirely unverifiable due to the dual-use nature of the weapons.

This got me thinking a bit about what sorts of institutions could help mitigate some of the consequences of "leaks". Proposals for an international cyberweapons convention have been thrown around, but most have been very vague and poorly defined. Kaspersky Labs founder Evgeny Kaspersky recently suggested a treaty along the lines of the Biological Weapons Convention or the Nuclear Non-proliferation Treaty (the Russian government has also floated similar proposals). However, an out-right ban on "cyberweapons" would be highly unlikely and generally impractical. As I mentioned, verifying compliance would be substantially more difficult than it has been for either the BWC or the NPT. Given that both have been violated by a number of states party to them via clandestine programs, a "cyberweapons" ban would be toothless, even if it only banned particular types of attacks (such as those on SCADA systems). Moreover, states find cyber-capabilities significantly more versatile and useful than either biological or nuclear weapons. The category of "cyberweapon" is broad enough to include highly-developed viral sabotage (Stuxnet) to simple distributed denial of service (DDOS) and these sorts of technologies are useful not only to militaries, but also to intelligence services. Finally, the dual-use nature of information technology and its globalization make locking-in a "cyberwarfare oligopoly" a-la the nuclear monopoly of the NPT near-impossible. The "haves" cannot credibly promise disarmament to the "have-nots" and the "have-nots" face significantly lower barriers to developing basic cyber-espionage or warfare capabilities.