Hints on How to Read Summary Level data Efficiently

This sample program shows how to read summary level data efficiently.

The first step for an efficient program is to use a data_null line. This keeps SAS from creating a permanent file. This will have an important effect on the amount of /usr/tmp space the program needs.

The second step is to discard all records that one does not need. The program reads a line of data and decides if it meets the selection criteria or not. In this example the program looks to see if the record contain data for a county. If it does, it keeps it. If it does not, it deletes it. This has a huge effect on the amount of data one will manipulate, as in Michigan, and most states, county level records, are less than 1% of all records in a summary file (97/308725 for Michigan).

Finally, the program only writes out a file that includes the information needed (selected variables for a single summary level).

Sample Program 

filename pl94 PIPE 
 'zcat /usr/data/public/us/government/stf/2000/PL94-171/mi.dat.Z';
data _null_;
  infile pl94 lrecl=2992;
  input @9 s $char3. @;
if s eq ' 50' then keep=1; else keep=0;
if keep eq 1 then do;
input state $ 30-31 county $ 32-34 p0010001 401-409  p0010009 473-482;
file mi_county;
put state county p0010001 p0010009;
   end;
else if keep eq 0 then input @9 s $char3.;
run;

This example didn't show that one could use a wildcard and pull off county level data for the entire nation. One would replace 

 'zcat /usr/data/public/us/government/stf/2000/PL94-171/mi.dat.Z';

with 

 'zcat /usr/data/public/us/government/stf/2000/PL94-171/*.dat.Z';

NEW PSC blog

Recent resources, events, news

New Publications

Knodel et al. "Gendered Housework in Vietnam." PSC Research Report.

Couper. Designing Effective Web Surveys

Next Brown Bag

Monday, Oct 13
Jamie Jones
"Fertility & Risk Aversion on Utah Frontier"

W A R N I N G

If you are reading this, it may be that you are using rather old web browsing software that does not support modern international Web technology standards. For a better experience of the Web and this site in particular, please upgrade your web browser software today. The following are good choices: Firefox 2; Opera 9; Safari 3.