AMERICAN DISTORTION OF DUTCH DRUG STATISTICS
Robert J. MacCoun

In early 1998, I published an essay in the Annual Review of Psychology on bias in the interpretation and use of social science research evidence by researchers and research consumers. In many ways, it was a discouraging essay, documenting the numerous psychological, sociological, and economic processes that can produce such biases, and the increasing evidence that they can occur unintentionally as a result of otherwise adaptive cognitive mechanisms.

But the idea that such biases are pervasive is already widely held— if anything, the public probably errs on the side of excessive skepticism of experts. So the essay offered some arguments against cynicism. First, under standard decision theory analyses, “biased” interpretation of evidence is sometimes normatively justifiable. Second, recent research indicates that credible counter- evidence does tend to curtail extreme claims in favor of one’s position— we do not simply see whatever we want in the data. And third, we should avoid jumping to quick conclusions about experts’ motives. It is devilishly difficult to establish that someone else is being biased— indeed, the bias is often in the eye of the beholder. The “hostile media phenomenon” provides evidence for this last point. In many conflicts, it has been shown that each side of a dispute tends to think that the media is biased in favor of the other side.

Ironically, after completing that essay, but before it appeared in print, I was confronted with two examples of what seemed to be gross distortions of my own recent research on Dutch cannabis policies and outcomes. In my gut, all my earlier talk of normative justification, data- constrained assertions, and biased beholders seemed like so much hooey. Two years later I am still trying to reconcile my intellect and my gut.

Dutch cannabis policy is a staple of the U. S. drug policy debate, because it the closest thing to drug legalization in a modern industrialized nation— for drugs other than tobacco and alcohol. Since 1976, the Dutch have maintained a formal legal prohibition on cannabis products (marijuana and hashish), while tolerating the commercial sales of up to 30 grams of cannabis (reduced to 5 grams in 1995), mostly in coffee shops and bars. “Toleration” does not mean discretionary non- enforcement, like the casual way Americans police prostitution or illicit gambling. It means that there are formal written policies instructing the police and prosecutors not to enforce cannabis prohibition for small quantity transactions.

(Drug) War Story
In October 1997, Peter Reuter and I published an article in Science examining Dutch “de facto” cannabis legalization and its consequences. The article was motivated by our frustration with the grossly discrepant “factual” comparisons of U. S. and Dutch drug statistics routinely encountered in the mass media and on the internet. According to recent clippings in our files, the lifetime prevalence of marijuana use (the percentage who have ever used marijuana) among Dutch teens had either fallen from 15 to 2 percent or risen from 5 to 14 percent. Readers were told that respective marijuana rates for Dutch and U. S. adolescents were either 14 vs. 38 percent or 30 vs. 11 percent.

The sources of these factoids appear authoritative on their face, and in fact each is technically accurate. It all depends on which statistics one cites— in particular, the year of the estimate and the age group of the respondents. Even under the best of circumstances, cross- national comparisons are problematic. But at the very least, one ought to compare rates for the same year, and the same age groups.

Reuter and I set out to describe more accurately Dutch policies and outcomes and assess what lessons, if any, they plausibly provided for the U. S. drug debate. Our article presented time- series data on cannabis prevalence over a 25 year period in the Netherlands, and 15 static comparisons of Dutch and non- Dutch (American, Danish, and or German) cannabis prevalence rates, each matched by age group and year.

Our interpretation of the Dutch data was rather nuanced, so we crafted a carefully worded press release, designed to minimize potential misunderstandings. On 3 October 1997, the day Science lifted the embargo, we were pleased to discover that our study was widely covered. Many papers, including USA Today and the San Francisco Chronicle, ran an Associated Press story by Paul Recer. Our delight turned to dismay when we read the following passages: “… the percentage of 18- year- olds who had tried marijuana rose from 15 percent to 44 percent” in the Netherlands, but “by contrast, teenage use of marijuana in the United States was estimated at about 12 percent in 1992.”

Since 1976, the Dutch have maintained a formal legal prohibition on cannabis while tolerating sales of up to 30 grams.

The Recer article implied a 32 percent difference between the two nations, clearly implying that there are a lot more stoned Dutch teens than U. S. teens. In fact, according to our paper, there were three possible comparisons between matched national estimates from the Netherlands and the United States, and they had “an average Dutch- U. S. difference of 1%, well within the sampling error of the surveys.”

We immediately submitted a correction letter to the Associated Press and to each paper that ran the Recer story. The key passage explains how Recer’s selective facts distorted our article: “This comparison, which is not taken from our article... is quite misleading, for two reasons. First, it compares the lifetime experience of Dutch 18- year- olds to that of all U. S. teens. But in each country, 18- year- olds are much more likely to have tried marijuana than other teens, because they have lived longer, and because drug experimentation is more common in late adolescence. Second, it compares a 1996 Dutch rate with a 1992 U. S. rate, yet U. S. rates skyrocketed between 1992 and 1996.” Our account was precise, but tedious, which may explain why it was only published in one paper, the Honolulu Advertiser. We were gratified by that correction, but had aspired to bring truth to light in all 50 states, or at least some of the lower 48.

Significantly, Recer’s Dutch figure appeared in our press release; the U. S. figure did not. The correct U. S. figure for 12 to 17- year- olds in 1996, 10.6 percent, was described by Recer as “around 12 percent,” suggesting he got it from some different source. Indeed, we suspect he never saw our article, basing his story entirely on the press release. A call to Recer produced neither a correction nor any clarification. He seemed not to see why there was a problem, arguing that 18- year- olds are in fact teenagers. (I asked rhetorically: Would he compare the average height of 18- year- olds to that of 12- to17 year- olds?)

Almost a year later, in July of 1998, we were cited in a Los Angeles Times op- ed essay by General Barry McCaffrey, the Director of Office of National Drug Control Policy (ONDCP). He wrote: “A 1997 study by Robert MacCoun and Peter Reuter… notes that the percentage of Dutch 18- year- olds who tried pot rose from 15 percent to 34 percent from 1984 to 1992, a time when the numbers weren’t climbing in other European nations. By contrast, in 1992 teenage use of marijuana in the United States was estimated at 10.6 percent.” It appeared that the Associated Press story had taken on a life of its own.

We were at least heartened to see the 32 percent gap reduced to 24 percent, though still well above the correct 1 to 2 percent difference we had reported. Apparently, on the basis of a phone conversation with my co- author, the ONDCP staff corrected the 1992- 1996 discrepancy but failed to correct the 18 vs. 12- 17 age discrepancy.

We wrote a correction letter to the Los Angeles Times, and faxed a copy to ONDCP as a courtesy. They immediately contacted us to apologize, and we negotiated an arrangement whereby we would withdraw our correction letter and ONDCP would correct the error themselves. We received a copy of that letter but it never appeared in the Times. Some months later, the Houston Chronicle ran the McCaffrey essay in its uncorrected form. A call to ONDCP elicited another agreement that they would send in a correction. Again, no correction letter was ever published.

This was not the first time General McCaffrey had mischaracterized Dutch policy. Earlier that same month, in a critique of Dutch tolerance toward hard drug users, he asserted that the Dutch homicide rate was over twice that of the U. S., when in fact the U. S. rate (8 per 100,000) is four times larger than the Dutch rate (1.8 per 100,000). The erroneous figures were based on an apparent failure to realize that the Dutch statistics included unsuccessful homicide attempts.

My essay on biased interpretation relied heavily on a key distinction in contemporary social and cognitive psychology— hot vs. cold cognition. Cold cognition is abstract and dispassionate, but not necessarily “rational”— it is nearly as vulnerable to distortion, but due to the “quick and dirty” mechanical shortcuts of mental processing. Hot cognition is cognition infused with emotion and motivation. For several months after these media incidents, my own interpretations were positively scalding. With the passage of time, I’m now able to reflect on these events more coolly.

War stories like mine are all too common in the public debate on American drug policy, as they are in many other areas of social policy. Are they inevitable? Are they defensible?

Interpreting Misinterpretations
Arguably, there are at least two normatively justifiable mechanisms for “biased” interpretation of evidence. First, from the standpoint of the Bayesian induction framework, disagreements about thea posteriori probability of a hypothesis, conditioned on the available data, are justifiable when judges differ in their “priors”— their subjective estimate of the a priori probability of the hypothesis. Second, because evidence strength is a matter of degree, the dichotomous decision to accept or reject a verdict requires a decision threshold or standard of proof. Decision theory suggests that perceivers can and should apply different thresholds depending on their relative aversion to false positive errors (accepting a hypothesis when it is false) vs. false negative errors (rejecting it when it is true).

But it is unclear how one might exonerate the distortions in the AP story and the McCaffrey essay on these normative grounds. These sources did not simply differ from our Science essay in their judgment of the likely effect of policy differences on Dutch vs. U. S. drug rates, or on the question of whether the weight of the evidence favored one policy approach over the other. Rather, they simply juxtaposed correct facts in a manner that is charitably described as meaningless, and less charitably described as patently misleading.

What of the argument that one should be wary of labeling others as biased? Admittedly, I am hardly capable of cool neutrality in my assessment of these uses of my research. But to assert, as I do, that Recer and McCaffrey distorted the evidence is not to impute motives or assert that they acted fraudulently. Recer’s error might be viewed as deceptive, but it might simply reflect confusion or carelessness. For General McCaffrey’s essay, a plausible account might invoke an overworked staff, a recent public history of skepticism if not outright hostility toward the Dutch approach to drug problems, and an AP story that seemed to meet their rhetorical needs.

Cynicism and Hope
After some decades debating hot vs. cold accounts of various judgmental biases, most psychologists have come to the conclusion that “warm cognition” is the norm. Warm cognition is motivated— it acts in the service of furthering one’s desired ends. But it acts the way lawyers are supposed to act when they are properly fulfilling their advocacy role. Evidence constrains warm cognition; we cannot simply claim to see whatever we want to see. Social psychologists call this “constrained directional bias”— we push our interpretation as far in the desired direction as the evidence will permit, but not further. Evidence against our positions gets scrutinized with a fine- toothed comb; evidence that can plausibly be construed as favorable is immediately flaunted.

For many if not most Americans, the use of into the use of intoxicating drugs, at least illegal ones, is a profoundly immoral act and must be judged and policed on moral terms.

A corollary to the “constrained directional bias” idea is the suggestion by Harry Kalven and Hans Zeisel, in their 1966 The American Jury, that jurors’ personal sentiments are most likely to emerge as a factor in deliberations when the evidence was equivocal or ambiguous. Similarly, physicist and science fiction author Gregory Benford has offered a “Law of Controversy” in which “passion is inversely proportional to the amount of real information available.”

Drug policy fits these models perfectly. For many if not most Americans, the use of intoxicating drugs, at least illegal ones, is a prof oundly immoral act and must be judged and policed on moral terms. In principle, illicit drugs might be regulated like any other risky activity, and one can plausibly argue that doing so might produce more effective policies than the current approach. But the fact of the matter is that the science of drug policy is still remarkably crude. At present, there is little serious prospect of the sort of demonstrably effective technical expertise that might trump simple moral intuitions— as has largely happened in the domains of medicine, cosmology, and (for the most part) natural history.

Will good drug data eventually drive out misleading claims? There are some grounds for optimism. In various domains, policy- relevant indictors— macroeconomic, agricultural, educational— are now routinely collected in a standardized fashion by highly trained technicians using state- of- the- art methods for establishing reliability and validity. Gradually, slowly, American drug data collection and analysis is moving in this general direction. There are various efforts in Western Europe to standardize the collection and reporting of statistics on drug use and drug- related outcomes. And a favorable sign is that even the most extremely partisan drug warrior and drug reformer web sites now provide links to the web pages of the major government drug data sources.

Relatively good data are already widely available on American crime and criminal sanctioning rates, at least in comparison to the poor state of drug statistics. Local politicians take credit for falling crime rates, and blame others for rising crime rates, but they rarely assert that rates are falling when they are rising. Nevertheless, distortion is still rampant in criminal justice discourse. Systematic content analyses show that the media disproportionately reports the most heinous, atypical crimes, and the “cops” on live- action shows are disproportionately white and the offenders are disproportionately black.

Science and Conflicting Values
The misrepresentations of our research resulted in part from a questionable assumption— the notion that one can assess the relative merits of American vs. Dutch approaches to the drug problem by comparing current marijuana rates in each country. This exemplifies the “horse race” reflex in American journalism and politics.

Recent levels of Dutch and American marijuana use are roughly equal, a fact that drug war doves find congenial because drug war hawks believe tolerance should raise drug rates. But we found the changes in Dutch use over time more informative. In the early 1970s, Dutch cannabis rates were considerably lower than those in the U. S., and they remained at that low level for at least a decade after the 1976 Dutch drug law. But Dutch use rose steeply during the 1980s, coinciding with a rapid increase in the number and visibility of the cannabis coffee shops. We argued that the experience of the first decade suggests that depenalizing possession, per se, produces little or no measurable increase in drug use. On the other hand, we hypothesized that commercial promotion, not surprisingly, produces a significant increase in the market for cannabis.

Thus, even without distortion, our article already offered something for each side of the debate. As a result, most subsequent citations of our article have been accurate, but selective. For example, in February 1998, an article in New Scientist correctly cited our conclusion that “reductions in criminal penalties have little effect on drug use, at least for marijuana.” The April 1998 issue printed a letter from a British government official objecting to this “seriously misleading quotation from the editorial summary of [MacCoun and Reuter’s] article,” which, he noted, actually said that “growth in commercial access to cannabis, after de facto legalization, was accompanied by steep increases in use, even among youth.” The editors replied by quoting our methodological caveat that the correlation between rising marijuana use and the increasing number of coffee shops “may not be causal.”

Technically, each statement in this exchange is accurate. Taken separately, each draws a different lesson from the Dutch experience; each is incomplete. Improvements in drug statistics seem unlikely to eliminate this sort of selective emphasis. Accuracy will invariably breed consensus. When accurate portrayals depict a complex world, conf licting values can always yield conflicting simplifications.

Robert J. MacCoun is professor of public policy and professor of law at the University of California at Berkeley, and co- author (with Peter Reuter) of the forthcoming Drug War Heresies: Learning from Other Vices, Times, and Places.