Hints on How to Read Summary Level data Efficiently

This sample program shows how to read summary level data efficiently.

The first step for an efficient program is to use a data_null line. This keeps SAS from creating a permanent file. This will have an important effect on the amount of /usr/tmp space the program needs.

The second step is to discard all records that one does not need. The program reads a line of data and decides if it meets the selection criteria or not. In this example the program looks to see if the record contain data for a county. If it does, it keeps it. If it does not, it deletes it. This has a huge effect on the amount of data one will manipulate, as in Michigan, and most states, county level records, are less than 1% of all records in a summary file (97/308725 for Michigan).

Finally, the program only writes out a file that includes the information needed (selected variables for a single summary level).

Sample Program 

filename pl94 PIPE 
 'zcat /usr/data/public/us/government/stf/2000/PL94-171/mi.dat.Z';
data _null_;
  infile pl94 lrecl=2992;
  input @9 s $char3. @;
if s eq ' 50' then keep=1; else keep=0;
if keep eq 1 then do;
input state $ 30-31 county $ 32-34 p0010001 401-409  p0010009 473-482;
file mi_county;
put state county p0010001 p0010009;
   end;
else if keep eq 0 then input @9 s $char3.;
run;

This example didn't show that one could use a wildcard and pull off county level data for the entire nation. One would replace 

 'zcat /usr/data/public/us/government/stf/2000/PL94-171/mi.dat.Z';

with 

 'zcat /usr/data/public/us/government/stf/2000/PL94-171/*.dat.Z';

PSC blog

Recent resources, events, news

New Publications

Bingenheimer & Geronimus, "Behavior & HIV"

Wildeman, "Imprisonment & Infant Mortality," PSC Research Report

Next Brown Bag

Tues, Dec 1
Arland Thornton & Barb Koremenos
Mobilizing for Human Rights
For live stream
LINK HERE

W A R N I N G

If you are reading this, it may be that you are using rather old web browsing software that does not support modern international Web technology standards. For a better experience of the Web and this site in particular, please upgrade your web browser software today. The following are good choices: Firefox 2; Opera 9; Safari 3.