Special Issue on Survey Non-response

Introduction: New Challenges to Social Measurement
Douglas S. Massey and Roger Tourangeau
Abstract | PDF

Facing the Nonresponse Challenge
Frauke Kreuter
Abstract | PDF

Explaining Rising Nonresponse Rates in Cross-Sectional Surveys
J. Michael Brick and Douglas Williams
Abtract | PDF

Response Rates in National Panel Surveys
Robert F. Schoeni, Frank Stafford, Katherine A. Mcgonagle, and Patricia Andreski
Abstract | PDF

Consequences of Survey Nonresponse
Andy Peytchev
Abstract | PDF

The Use and Effects of Incentives in Surveys
Eleanor Singer and CongYe
Abstract | PDF

Paradata for Nonresponse Adjustment
Kristen Olson
Abstract | PDF

Can Administrative Records Be Used to Reduce Nonresponse Bias?
John L. Czajka
Abstract | PDF

An Assessment of the Multi-level Integrated Database Approach
Tom W. Smith and Jibum Kim
Abstract | PDF

Where Do We Go from Here? Nonresponse and Social Measurement
Douglas S. Massey and Roger Tourangeau

Abstract | PDF

The Twitter paper from PAA’s “social media” session

Using Twitter for Demographic and Social Science Research:
Tools for Data Collection

T. McCormick, H. Lee, N. Cesare and A. Shojaie | CSSS/University of Washington
April 8 2013
This is a proof of concept paper. The researchers searched through tweets for phrases that indicated an intention to “not vote” in the 2012 election. They used Amazon’s Mechanical Turk to identify the profile pictures of their sample (age, gender, race).

Folks interested in other examples of “wild data” like Google searches, Twitter, etc. should check these posts:

Wild Data: Expanding Social Science Research
Big Data: Google Flu
Using Wild Data to Estimate International Migration

The Unauthorized Immigrant Population: Two Technical Excercises

This blog entry has two nice technical pieces. The first describes how PEW Hispanic (and others) estimate the undocumented population in the US. The second is a life-table exercise, which shows how many of the undocumented population will die waiting for citizenship – assuming a 13 year wait time.

Unauthorized Immigrants: How Pew Research Counts Them and What We Know About Them
Interview with Jeff Passel | Pew Hispanic
April 17, 2013
In this interview, Passel describes how he estimates the undocumented population in the US – including the other characteristics of this population, e.g., occupation, current residence, family composition, etc. using data from the Current Population Survey.

As a note, most of the reports PEW Hispanic writes on the undocumented population have an appendix, which provides a more technical description of the methodology. See page 25 of the following report for an example: Cohen, D’Vera and Jeff Passel. 2011. “Unauthorized Immigrant Population: National and State Trends, 2010″ Pew Hispanic: February 1, 2011.

The life-table exercise is from Philip Cohen’s Family Inequality blog.

How many people should die waiting for citizenship? 319,462?
Philip Cohen | Family Inequality Blog
April 24, 2013

This is a life-table exercise, taking the current age distribution of the undocumented population in the US and applying a life-table for Hispanics to the numbers. He describes his assumptions and invites folks to re-calibrate the numbers.

Note that Cohen takes a dig at Reinhart and Rogoff [previous PSC Infoblog entry] by making his spreadsheet available. And, he notes “If you don’t like the way Excel does the maths, by all means, fix it in R.”

Living Apart Together: Data & Research

Living Apart Together: Uncoupling Intimacy and Co-Residence
S. Duncan, M. Phillips, S. Roseneil, J. Carter & M. Stoilova | NatCen Social Research Policy Brief
Winter 2013
Major conclusions from the research are (a) some “singles” are in LAT relationships; (b) living alone doesn’t always means being alone; and (c) intimacy doesn’t always imply co-residence

Note, a similar policy brief for the Canadian LAT population is in an earlier PSC-Info blog entry.

The Census Reform Act of 2013

This proposed legislation is really radical [H.R. 1638]. It would eliminate all surveys collected by the Census Bureau: Economic Census, Census of Governments, Census of Agriculture and a non-existent mid-decade census. Furthermore, it would limit the census to a population count.

In short:

(a) Notwithstanding any other provision of law–
(1) the Secretary may not conduct any survey, sampling, or other questionnaire, and may only conduct a decennial census of population as authorized under section 141; and
(2) any form used by the Secretary in such a decennial census may only collect information necessary for the tabulation of total population by States
(b) Repeal of Survey, Questionnaire, or Sampling Authority- Sections 182, 193, and 195 of title 13, United States Code, are repealed.

The Census Project Blog discusses this in more detail:
What We Don’t Know Can’t Hurt Us (Right?)
Teri Ann Lowenthal | The Census Project Blog
April 23, 2013

If Congress only wants a head-count census, will they fund a ‘mandatory population register?’ This is something New Zealand is considering:

National Census Could be Scrapped
National News | TVNZ
April 23, 2013

Microsoft Excel: The Ruiner of Global Economies?

This is a series of articles on the news that a well-cited and influential paper by Carmen Reinhart and Ken Rogoff had an Excel error in it, which led to an overstating of the association between debt and growth. There are other more fundamental problems with the paper – see comments by economists below.

From a training viewpoint, it is relevant to note that this was discovered by a graduate student, working on a class assignment: find a famous study and replicate it.

This entry has four sections: (a)the student; (b)comments by other economists; (c)replication & programming; and (d)coverage from the press.

The Story of the Student
Meet the 28-Year-Old Grad Student Who Just Shook the Global Austerity Movement
Kevin Roose | The New York Magazine
April 18, 2013

How a student took on eminent economists on debt issue – and won
Edward Krudy | Reuters
April 18, 2013

‘They Said at First That They Hadn’t Made a Spreadsheet Error, When They Had’
Peter Monagham | Chronicle of Higher Education
April 24, 2013
My favorite Q & A from this interview with Thomas Herndon is:
Q. This is more than a spreadsheet error, then?

A. Yes. The Excel error wasn’t the biggest error. It just got everyone talking about this. It was an emperor-has-no-clothes moment.

Comments/Analysis by Economists
Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogo ff
Thomas Herndon, Michael Ash and Robert Pollin | Political Economy Research Institute
April 15, 2013
easier to read pdf of paper, but link above includes data, code, etc.

Researchers Finally Replicated Reinhart-Rogoff, and There Are Serious Problems
Michael Konczal | Next New Deal (blog of the Roosevelt Institute)
April 16, 2013

Reinhart and Rogoff are wrong about austerity
Robert Pollin and Michael Ash | Financial Times
April 17, 2013

Reinhart/Rogoff and Growth in a Time Before Debt
Arindrajit Dube | Next New Deal (blog of the Roosevelt Institute)
April 17, 2013

Reinhart, Rogoff, and How the Macroeconomic Sausage Is Made
Justin Fox | Harvard Business Review
April 17, 2013

The Excel Depression
Paul Krugman | New York Times
April 19, 2013

Replication & Programming
The Mysterious Powers of Microsoft Excel
Colm O’Regan | BBC News Magazine
April 20, 2013

What the Reinhart & Rogoff Debacle Really Shows: Verifying Empirical Results Needs to be Routine
Victoria Stodden | The Monkey Cage Blog
April 19, 2013

What Reinhart-Rogoff Means for the Replication Debate
Political Science Replication Blog
April 19, 2013

Microsoft Excel: The ruiner of global economies?
Peter Bright | Ars Technica
April 16, 2013
This piece describes the Excel error, but also discusses other issues with the paper, including the interesting tidbit that the original Reinhart-Rogoff paper was published in the American Economic Review proceedings issue(May), which are not peer reviewed.

Two clever economists have looked to see if researchers pad their resumes by hiding their AER proceedings publications. The University of Michigan economics department was included in their sample.

Research: Bad math rampant in family budgets and Harvard studies
Jeremy Olshan | Wall Street Journal (Market Watch blog)
April 17, 2013
88% of spreadsheets have errors

On the accuracy of statistical procedures in Microsoft Excel 2007
B.D. McCullough and David A. Heiser | Computational Statistics and Data Analysis
March 2008
These authors criticize Excel for its use in statistical analysis because of its failures in statistical distributions, random number generation, and the NIST StRD(Statistical Reference Datasets). I suspect most users of Excel are using the simpler tools: summation, product, etc., but on occasion faculty have used Excel as a rudimentary statistical analysis tool.

What We Know about Spreadsheet Errors
Raymond Panko | Journal of End User Computing
May 2008

Come to Jesus Slides: Use Script-Based Analysis, not Excel
Matt Frost | Charlottesville, Virginia
The author is recommending R or more specifically R Studio, but his point applies to any script-based statistical package.

The Press
Too many to link to for the moment, but here’s a sampling:
[Search Link]

Income Mobility and Welfare

International Monetary Fund Working Paper
By: Tom Krebs, Pravin Krishna, and William Maloney

Abstract:

This paper develops a framework for the quantitative analysis of individual income dynamics, mobility and welfare. Individual income is assumed to follow a stochastic process with two (unobserved) components, an i.i.d. component representing measurement error or transitory income shocks and an AR(1) component representing persistent changes in income. We use a tractable consumption-saving model with labor income risk and incomplete markets to relate income dynamics to consumption and welfare, and derive analytical expressions for income mobility and welfare as a function of the various parameters of the underlying income process. The empirical application of our framework using data on individual incomes from Mexico provides striking results. Much of measured income mobility is driven by measurement error or transitory income shocks and therefore (almost) welfare-neutral. A smaller part of measured income mobility is due to either welfare-reducing income risk or welfare-enhancing catching-up of low-income individuals with high-income individuals, both of which have economically significant effects on social welfare. Decomposing mobility into its fundamental components is thus seen to be crucial from the standpoint of welfare evaluation.

Free full text (PDF, 700KB)

Essay: Linking, Exploring and Understanding Population Health Data

This is a nice data essay by former PSC trainee Michael Bader. He discusses multiple sources of data that one might use to understand population health. I especially like his point about the need to archive neighborhood conditions – after all neighborhoods change. But he also touches on the range of data available for analysis from focus groups to big data.

Linking, Exploring and Understanding Population Health Data
Michael Bader | Human Capital Blog (RWJ)
June 25 2012

The opening paragraph deserves a highlight, but read the entire entry. It is worth it:

Data are the sustenance of population health research, and like the food that sustains us, it comes in many forms, shapes and sizes. Also like food, it’s best appreciated in combination. A single data source in the absence of context is unfulfilling; but combining datasets that are rich with information and contours — now that’s a meal!

War on Science: Canada

While Political Science has had its major funding source cut for the remainder of this fiscal year [see synopsis] and Congressional Republicans want to tinker with the American Community Survey (ACS) or completely cut its funding [see synopsis], Canada’s experience with governmental interference into scientific research is more drastic. The trend has been to fund “applied” research – sort of like funding MRI machines and not the science that developed the technology. Likewise, government scientists and even librarians are muzzled – not able to speak to the press without clearance.

Some of this has been covered in Nature and Science Insider, but most of the details require reading some Canadian news.

[Muzzling Scientists]
Harper government’s muzzling of scientists a mark of shame for Canada
Jeffrey Hutchins | thestar.coom
March 15, 2013
Notable Quotations
Since 2006 the federal government has been shortening the leash on its scientists. In some departments researchers are now not allowed to speak about their studies without ministerial (meaning political) permission. And in several documented instances that permission has been refused. In February, Fisheries and Oceans Canada raised additional non-science barriers to the publication of scientific research.

Let’s be clear. When you inhibit the communication of science, you inhibit science. The legitimacy of scientific findings depends crucially on unfettered engagement, review, and discussion among interested individuals, including members of the public.

Refreshingly, a Scandinavian with impeccable credentials provides an enlightened perspective. Gro Harlem Brundtland, three times Prime Minister of Norway and chair of the renowned Brundtland Commission on sustainable development, argues that:

“If we compromise on scientific facts and evidence, repairing nature will be enormously costly – if possible at all. Politics that disregard science and knowledge will not stand the test of time.”
If politics that diminish and devalue science should not stand the test of time, then neither should politically motivated barriers to the communication of science.

The Canadian government’s current communication controls are clearly not the hallmark of a confident, mature, and progressive society. We can and should do much, much better.

[Political Interference]
Canada puts commercialization ahead of blue-sky research
Brian Owens | Nature
March 22, 2013
Notable Quotations
But the government’s relentless focus on business innovation does not represent a coherent science strategy, says Paul Dufour, director of Paulicy Works, a science-and-technology consultancy in Gatineau, Quebec. He notes that the budget makes no reference to a national science-and-technology strategy that Harper released in 2007. “We have to assume that it’s dead, and that the government has no strategy,” Dufour says.

Instead, Dufour says, there is a piecemeal approach, with the government “picking winners” and providing new money to the automotive, aerospace, forestry and aquaculture sectors. “It’s very short-term thinking,” he says.

Canadian Budget Targets Industrial Applications
Wayne Kondro | Science Insider
March 22, 2013
Notable Quotations
The new budget promises stiffer competition for a smaller pool of research grants. What little new money is made available will again be funneled into targeted “industry-academic” partnerships.

Program after program [within the councils] is becoming company specific,” he says. “This is all money that’s being squeezed out of what should be going for discovery research. Previous budgets had signaled a shift of priorities from basic research to various collaborations with industry. This budget confirms that.”

The budget reaffirms plans by Minister of State for Science and Technology Gary Goodyear to revamp the National Research Council, the government’s primary in-house research arm. The goal is to create a “concierge” service that provides one-stop shopping and solutions for industrial needs

The Canadian war on public science, basic research and the free and open exchange of scientific information
John Dupuis | Science Blogs
March 22, 2013
This blog entry provides a synopsis of a resolution voted on by the Canadian Parliament:

That, in the opinion of the House: (a) public science, basic research and the free and open exchange of scientific information are essential to evidence-based policy-making; (b) federal government scientists must be enabled to discuss openly their findings with their colleagues and the public; and (c) the federal government should maintain support for its basic scientific capacity across Canada, including immediately extending funding, until a new operator is found, to the world-renowned Experimental Lakes Area Research Facility to pursue its unique research program.

Which was defeated 157 to 137 – every conservative voted Nay, including Prime Minister Harper.

Closure of Experimental Lakes Area part of assault on science
Stephen Scharper | the star.com
March 29, 2013
Notable Quotations
Last May, scientists were told that the federal government intended to stop funding the Experimental Lakes Area (ELA) research facility, a site encompassing 58 lakes which, for more than 40 years, has provided cutting-edge findings on myriad ecological issues, including phosphate and mercury pollution, acid rain and aquatic effects of climate change. They were also told, according to some sources, not to talk about it with the media or other colleagues.

The government claims the move will save $2 million annually, and says it is willing to allow another operator to take over. As of now, no alternative agency has come forward to assume operation of the facility.

According to Cynthia Gilmour, senior scientist with the Smithsonian Institution, the ELA is “the only place in the world” where you can do controlled experiments within a lake ecosystem.

Research and Politics

The field of Political Science has been hit hard by an amendment to the Continuing Approriations Act of 2013, which pretty much axes the NSF political science funding mechanism. The money remains with NSF rather than being shifted to the National Cancer Institute and political science research can still be funded, but only if their research is useful for “national security” or “the economic interests” of the United States.

This amendment only applies until the end of this fiscal year, but NSF funding for political science has been on Tom Coburn’s radar for years. Expect more of the same and perhaps even for the rest of the softer sciences.

The links are in presented in order of publication – oldest first:

First, the prequel
New Attempt to Cut NSF Funding for Political Science
March 15, 2013

NSF’s political science program siphons valuable resources away from higher priority research that will yield greater applied benefits and potential to stir further innovation. This amendment does not aim to hinder science, but rather to allocate more support for research that will save lives.
Tom Coburn’s Fact Sheet

The amendment sets up a false dichotomy between medical research and research in
the social sciences that we emphatically reject
Hunter R. Rawlings III, president of the Association of American Universities

Senate Delivers a Devastating Blow to the Integrity of the Scientific Process at the National Science Foundation
PRNewswire
March 20, 2013
Notable Quotes
Adoption of this amendment is a gross intrusion into the widely-respected, independent scholarly agenda setting process at NSF that has supported our world-class national science enterprise for over sixty years.

The amendment creates an exceptionally dangerous slippery slope. While political science research is most immediately affected, at risk is any and all research in any and all disciplines funded by the NSF. The amendment makes all scientific research vulnerable to the whims of political pressure.

Adoption of this amendment demonstrates a serious misunderstanding of the breadth and importance of political science research for the national interest and its integral place on the nation’s interdisciplinary scientific research agenda.

Singling out any one field of science is short-sighted and misguided, and poses a serious threat to the independence and integrity of the National Science Foundation.

And shackling political science within the national science agenda is a remarkable embarrassment for the world’s exemplary democracy.

Money for Military, Not Poli Sci
Libbie A. Nelson | Inside Higher Education
March 21, 2013
Notable Quotes
The amendment defunding political science was adopted in a voice vote that surprised many observers. Ending federal funding for political science research has been a longtime cause for some Republicans in Congress, including the measure’s sponsor, Oklahoma Senator Tom Coburn, and the effort has failed many times in the past.

Senate Moves to Limit NSF Spending on Political Science
Paul Basken | Chronicle of Higher Education
March 21, 2013
Notable Quotes
The amendment was proposed by Sen. Tom Coburn, a Republican of Oklahoma who has sharply criticized the foundation’s spending priorities.

Mr. Coburn sent a letter last week to the NSF’s director, Subra Suresh, listing a series of agency-financed projects he considered a waste of taxpayer money. His list included several involving political science, including studies of voter attitudes toward the Senate filibuster and of the cooperation between the president and Congress.

Projects likely to be affected, he said, include the American National Election Studies, a landmark series of studies and polls dating to 1948. Its current principal investigators are at the University of Michigan at Ann Arbor and Stanford University.

Political Science Research: Singled Out
R.D.N. | The Economist
March 21, 2013

Tom Coburn Doesn’t Like Political Science
Henry Farrell | Chronicle of Higher Education
March 22, 2013
Notable Quotes
The NSF pays for 61 percent of basic research in the social sciences. Publicly supported academic research is, and should be, democratically accountable. Yet politicians have wisely delegated the particulars of funding lines to the scientific community. Politicians are not scientists, and do not have the expertise to judge which research areas and questions are promising and which are not.

The Coburn amendment changes that. It imposes crude political criteria on scientific grant making, arbitrarily decreeing that social scientists cannot get funds for studying key aspects of politics. It is clear that Coburn’s ambitions stretch far beyond the social sciences. In previous reports he has attacked the NSF for purportedly useless research in robotics, biology, and other areas of the natural sciences.

If this precedent is not reversed, it will probably be expanded in unhappy ways. Politicians will attach ever-more-onerous conditions to NSF funds, in order to make sure that research they like gets money, while research that they dislike does not. Politicians should not micromanage the grant-making process. They are likely to not only misunderstand the science but use their influence to mischaracterize good research in attempts to score political points.

In the worst-case scenario, Coburn’s amendment could also set a dangerous precedent for academic research in general. Introducing political micromanagement into a system that should be governed by scientific criteria would essentially politicize science. The NSF finances important research in politically controversial areas such as climate science, biology, and evolutionary science. To date, the NSF has been able to shield grant-making decisions in those areas from broader political acrimony. Politicians who deny global warming and evolution have not wanted to seem overtly anti-science, and have refrained from direct attack.

That delicate balance may be upset, as it becomes more acceptable to interfere with the inner workings of decision making at the NSF. Research on global warming, evolution, and biology may become fair game. The Coburn amendment is a tragedy for both political science and public debate. Its broader legacy may be a tragedy for the basic process of scientific discovery, if it is not swiftly reversed. Tom Coburn may not like political science. It’s important to remember that many of his colleagues don’t like science at all.

What We Need to Know about the NSF Funding Vote
Seth Masket | Mischiefs of Faction Blog
March 26, 2013
This blog entry is mostly a call to action among political scientists. He starts out by commenting on some of the technical details of the legislation from his history as a Congressional staffer.

That Time Where Tom Coburn Didn’t Believe in Micro-Managing Scientific Research
John Sides | The Monkey Cage Blog
March 27, 2013
In this post, Sides finds a time when Tom Coburn argues against micromanaging scientific research:

Coburn told Nature Medicine that he will continue to oppose any disease-specific legislation because he doesn’t think Congress should micromanage the leaders of the NIH. “If you’re going to do a disease-specific bill, you ought to tell them what mass spectrometer to buy,” he quips.

Tom Coburn Flip-Flops on NSF Funding of Political Science Research
John Sides | The Monkey Cage Blog
December 20, 2011
This piece gives some nice examples of Coburn deriding political science research and then using some NSF-funded political science research to support a point he was making about the decline in Congressional oversight.