Official Statistics

National Public Radio’s On the Media program ran an interview two weeks ago with Sir Michael Scholar, chairman of the UK Statistics Authority. The authority was created in 2008 as a watchdog over official government statistics. (They even have flags.)

On the Media summarizes the need for such a watchdog this way:

The creation of bad statistics and the distortion of good ones for political purposes is enough to cause the public to lose faith in government numbers. The U.K. lost so much faith in them that, in fact, the government decided they had to do something. In 2007, Parliament approved a new agency, the U.K. Statistics Authority, whose job would be to hold government agencies accountable for the numbers they released to the public.

The interview focused on the relationship between the UK Statistics Authority and the rest of the government. The authority is a government agency, but is insulated from politics in a few ways:

• The authority reports directly to parliament;
• It’s budget is set separately from the normal government budgeting process and is adjusted by objective formula; and
• Authority board members are appointed by Her Majesty the Queen, with parliamentary approval, but after an open competitive application process.

These measures address the problem of political influence and corruption, but they do not address a more fundamental problem: what is a statistic in the first place?

To see why that question is important, consider the opening sentences of the On the Media interview:

Last week, the White House sent out a press release saying the Obama administration has created or saved more than 150,000 jobs in the first hundred days and will create or save another 600,000 in the next hundred days. But when reporters looked for the basis of these authoritative-sounding numbers, they found they were based on—not much. It turns out it was a guess, based on a 16-page White House memo.

“Estimates” of jobs created or saved are not statistics, they are predictions. A statistic is a number that summarizes a set of data. For example, a list of students with their individual heights can be summarized by reporting the average height of the students (*note). Governments produce volumes of statistics—summaries of sets of data—on topics such as incidence of disease, accidents, demographics, employment, and education.

The difference between summaries of data sets (statistics) and predictions about the future is critical. There are well developed, objective, and verifiable standards for gathering data sets and producing statistics. Most universities have an entire department that specializes in that very question. It is therefore relatively easy for a watchdog to say when a government agency has produced misleading statistics. It is also easy for a watchdog to give guidance to agencies to use when preparing statistics. (See the UK Statistics Authority’s Code of Practice and other reports for examples.)

By contrast, there are no objective standards for predicting the future. (If you think there are, why aren’t you rich?)

To me, the problem with the Obama administration’s job estimates is not that they fail to follow some code of statistical practice, but that there is no code of practice for predicting job impacts. There will never be such a code.

There I go, predicting the future.

---

* To split hairs, a statistic is really an estimate. If we have accurate height measurements of every student in the class, we can calculate the so-called population average height, the actual average of all students in the class. In reality, we will be missing data on some students. Furthermore, data on students will be inaccurate due to measurement errors. So the best we can do is calculate the sample average height and hope that it is close to the population average height. The science of statistics is all about finding ways to make statistical estimates more accurate. (return)

Green Jobs

A colleague sent me the following paper:

Green jobs myths. Andrew P. Morriss, William T. Bogart, Andrew Dorchak, and Roger E. Meiners. Social Science Research Network Working Paper Series, March 2009. URL http://ssrn.com/abstract=1358423.

As you may guess from the title, the paper has an agenda, namely to attack the hype surrounding green job claims and green job proposals.

If you read it, be warned about two things. First, the paper is long. Three of the four authors are law professors and legal style calls for long papers and long footnotes. (Perhaps law professors are paid by the word.) Second, the paper blends objective criticisms with political criticisms. If your politics matches the authors’ you’ll have no trouble. Otherwise, you might be repelled and miss the gems.

To me, the most important part of the paper is section II. D. “The inappropriate use of input-output analysis.” Input-Output models are the source of most claims about job creation and economic impacts. When you read a news report about a new plant creating x thousands of jobs, chances are there is an I-O model lurking in the background.

The authors note the political appeal of I-O models’ job creation claims:

[A] common thread among advocates of renewable energy and related programs is that they will create new jobs. No doubt that promise has political appeal to help generate support from voters who hear that the programs will create clean energy and many new employment opportunities. Who can be opposed to jobs, especially green jobs?

There are several problems with using I-O models to analyze job impacts. The problems have a common root: Input-Output models explicitly assume that nothing changes except the particular policy being studied. For instance, an I-O model of a new baseball stadium assumes that only the stadium changes. No people move in or out of the neighborhood; no businesses form or fold; no traffic patterns change; nothing else changes.

Put another way, I-O models are—by design—snapshots in time. They tell us how the economy is organized now but they cannot tell us how it will change. When a proposal is very small relative to the surrounding economy, like a new restaurant in a million-person city, the assumption that nothing changes is acceptable. But when the proposal is relatively large, I-O models are useless.

It should be obvious that retooling the entire U.S. economy from fossil fuels to renewables is a relatively large change.

The authors use the problems with I-O models, along with 91 other pages of argument, to criticise the goal of shifting to renewable energy. I think that is overreaching, but I agree with their other point: Any policy as large as renewable energy deserves much better analysis than the job claims we have gotten so far.

Idolizing Economic Development

Politicians like to support “economic development” and “job creation”. They may tell us we can’t remove a certain tax break because it will cost jobs. Or they may tell us that we have to give grants to a particular business or industry because it will create jobs or grow our economy.

Of course, such statements resonate with the public, and interest groups have learned to frame their arguments in economic development and job creation terms. There is also a cottage industry that will run models purporting to show how many jobs and how much economic activity will be created by a particular (favored) proposal or reduced by a particular (opposed) proposal.

In my opinion, the key weakness with such claims is that they never include comparisons. If job creation and development were really our goals, we would gather a bunch of proposals, estimate their job and economic activity impacts, and choose the top proposals. The fact that we do not make such comparisons indicates that job creation and economic development are used as rhetorical terms, not analytical ones.

If one takes the rhetoric of economic development seriously, without making comparisons among proposals, one gets policies like this new one, from China:

Chinese ordered to smoke more to boost economy. Telegraph, May 4, 2009. Local government officials in China have been ordered to smoke nearly a quarter of a million packs of cigarettes in a move to boost the local economy during the global financial crisis.

The edict, issued by officials in Hubei province in central China, threatens to fine officials who “fail to meet their targets” or are caught smoking rival brands manufactured in neighbouring provinces.

Even local schools have been issued with a smoking quota for teachers, while one village was ordered to purchase 400 cartons of cigarettes a year for its officials, according to the local government’s website.

The move, which flies in the face of national anti-smoking policies set in Beijing, is aimed at boosting tax revenues and protecting local manufacturers from outside competition from China’s 100 cigarette makers….

This policy will certainly increase revenue to China’s cigarette manufacturers and enable them to hire or retain employees. The problem is that the money that goes to cigarette purchases could have been used for something else more productive and less damaging.

The point is obvious in this example, but it applies just as strongly to “economic development and job creation” proposals in the U.S. When we give money to a business to “create jobs,” we should ask what alternative uses there were for the money. How do the benefits and costs of each alternative compare to one another?

(Photo “Panda Cigarettes China” by Flickr user hakaider. Used under a Creative Commons license. I have no idea whether the pictured cigarettes are from one of the protected manufacturers, or are even really from China.)

The Low Price Consensus

Wisconsin has a law regulating the minimum price sellers can charge for gasoline. I should say “had” a law, since it was recently found to be unconstitutional and Attorney General Van Hollen will not appeal the ruling. (Ruling, Milwaukee Journal-Sentinel article, Fox11 story on Van Hollen.) The public debate over that law, while heated, conceals a strong consensus. In my opinion, both sides of the debate oppose the law’s original purpose—though they don’t realize it.

Social Science and the Environment

Jon Gertner, writing in the New York Times Magazine, explores the important role of social science in responding to climate change and other environmental problems.

He points out that physical scientists neglect social science issues. As measured by funding, social science is an after thought. (Gertner says that “about 98 percent of the federal financing for climate-change research goes to the physical and natural sciences, with the remainder apportioned to the social sciences.”)

It isn’t immediately obvious why such [social science] studies are necessary or even valuable. Indeed, in the United States scientific community, where nearly all dollars for climate investigation are directed toward physical or biological projects, the notion that vital environmental solutions will be attained through social-science research — instead of improved climate models or innovative technologies — is an aggressively insurgent view.

The counterargument is simple, environmental problems are caused by people so their solutions will involve changes in how people behave. Futhermore, since people are complicated social animals, understanding and influencing human behavior is an exceptionally difficult problem.

You might ask the decision scientists, as I eventually did, if they aren’t overcomplicating matters. Doesn’t a low-carbon world really just mean phasing out coal and other fossil fuels in favor of clean-energy technologies, domestic regulations and international treaties? None of them disagreed. Some smiled patiently. But all of them wondered if I had underestimated the countless group and individual decisions that must precede any widespread support for such technologies or policies. “Let’s start with the fact that climate change is anthropogenic,” Weber told me one morning in her Columbia office. “More or less, people have agreed on that. That means it’s caused by human behavior. That’s not to say that engineering solutions aren’t important. But if it’s caused by human behavior, then the solution probably also lies in changing human behavior.”

If you doubt that influencing human behavior vis à vis climate change will be difficult, consider how hard it is to influence much simpler human behavior, say drunk driving.

Accessible Numbers on Energy

Recommended reading: Sustainable Energy—Without the Hot Air.  David JC MacKay, 2009.

(Electronic version free under a Creative Commons license. Hardcover available for purchase.)

This is one of those clear-headed books that takes complicated numbers and makes them accessible to a general audience…and dispenses with twaddle along the way.

As Cory Doctorow puts it:

This is to energy and climate what Freakonomics is to economics: an accessible, meaty, by-the-numbers look at the physics and practicalities of energy. (Full review here)

Here is my favorite quote so far:

In a climate where people don’t understand the numbers, newspapers, campaigners, companies, and politicians can get away with murder.
We need simple numbers, and we need the numbers to be comprehensible, comparable, and memorable.

Voting Machines and Fraud

Disturbing follow-up news on electronic voting machines, via BoingBoing. (There’s not much point in talking about public policy if the foundation of the system is untraceably corrupt.)

Diebold Admits Audit Logs in ALL Versions of Their Software Fail to Record Ballot Deletions

Posted by Dan Gillmor, March 21, 2009 2:55 PM

Brad Friedman at the Brad Blog has been keeping up on the latest too-real news about the nation’s voting machines and the people who sell, buy and operate them. Two recent postings send the outrage meter way into the red.

First is California’s continuing effort to clean up the mess it’s made over the last few years. It’s going to be harder than anyone imagined. As we learn in this post:

Even the audit log system on current versions of Premier Election Solutions’ (formerly Diebold’s) electronic voting and tabulating systems — used in some 34 states across the nation — fail to record the wholesale deletion of ballots. Even when ballots are deleted on the same day as an election. That’s the shocking admission heard today from Justin Bales, Premier’s Western Region manager, at a State of California public hearing on the possible decertification of Diebold/Premier’s tabulator system, GEMS v. 1.18.19.

New About Page

I have changed the About page to better reflect my intentions for this blog.

Visualizing Media

Media Cloud is a nascent effort to help ordinary people visualize the flow of news coverage (including blogs) worldwide. I’d call it version 0.2, but the idea is promising.

The system indexes media reports from a wide range of sources. Users then choose up to three news sources and query the index in one of three ways:

• Get a simple list of the top ten words appearing in three sources.
• Specify a search term, and get a list of the top ten words associated with that term.
• Get a map showing the attention devoted to different countries by each source.

For instance, the top four terms for the New York Times (as of this posting) are:

1. United States
2. Washington
3. Barack Obama
4. California

The top for for the Milwaukee Journal Sentinal:

1. Wisconsin
2. Green Bay
3. Green Bay Packers
4. National Football League

At least we have our priorities straight in Wisconsin.

(Hat tip: Boing Boing)

Scientists and Public Policy

This essay is about how scientists think about public policy. I should say “mis-think” because that is my argument: Scientists have a naïve view of the public policy process that makes their advice less than helpful. (Public policy people have a corresponding misunderstanding of science, but that is a topic for another time.)

This topic came to me during a recent meeting of the Wisconsin Initiative on Climate Change Impacts (WICCI). WICCI is a joint venture of the Nelson Institute at the University of Wisconsin and the Wisconsin Department of Natural Resources. WICCI is composed of a science council (on which I sit), an advisory committee, and an operations and outreach unit.

The purpose of WICCI is to study the effects of climate change in Wisconsin and to develop strategies by which Wisconsinites can adapt. WICCI is expressly and self-consciously aimed at public policy. This is reflected in its structure. The science council is filled with scientists; the advisory committee with representatives of “agencies, business interests, non-governmental organizations and other stakeholders.”

At the meeting that prompted this essay, the science council working groups split up to discuss their topics: human health, fisheries, storm water and so on. The groups were to discuss possible climate change effects, data needs, and—in particular—strategies by which people and governments might adapt.

The human health group, which I attended, spent time talking about infrastructure. For example, cities might respond to increased summer heat waves by creating more public cooling sites. Or they might respond to increased rainfall by building larger storm water facilities or buying and preserving wetlands.