PINGO

First PINGO steps

contents: Getting started, How to call, How to get informations about my files, How to change the header, Fieldmanipulation, Split and merge, Missing data, From ascii to PINGO, Pipes, Area weights, GrADS, Example, Grids,Contact, Hint

Getting started

This introduction is not a complete manual but a short overview to find out how the PINGOs work. For a complete and detailed description see DKRZ Technical Report No. 11, which can also to be found on our PINGO Website as pdf-file.
The PINGOs work with 2-dimensional fields. Each field occupies one record of a file. A file contains an arbitrary number of records (upper limit for the time being: 2 GByte filesize). A file contains records of one meteorological parameter, i.e. surface temperature, over several decades, or lots of parameters in different levels over a few months. Therefore I need informations about my file and perhaps must split it into pieces. The PINGOs give me informations and are able to split a file but do not check if it is physical correct what I'm doing. For example I can add pressure fields to temperature fields without warning!
The PINGOs read ascii, SIMPLE, EXTRA, SERVICE, LOLA und GRIB formated data. The most common ones to read in are ascii or grib files. The other formats will be used during work with the PINGOs or for example if I need FORTRAN-input (EXTRA).
The lola and grib formats also contain informations about the grid of my field. You will find the grid-description files in the directory $GRIDS. If the PINGOs are to be find in "/usr/prog/PINGO/bin" then the variable $GRIDS contains "/usr/prog/PINGO/bin/grids" and should be declared in .cshrc (for cshell-users).

How to call

The call of the PINGOs depends on what I want my output file to contain:

• ext - if I need EXTRA format
• smp - for SIMPLE format
• pure - if I need my data in IEEE format (GrADS)
• srv - for SERVICE format
• lola - for LOLA format and
• grb - if I will save my data in grib format.

It is a good idea to start with the format my input file is in, mostly grb (for grib format). The PINGOs can't use pure format saved data as input.
In second place I must tell what the PINGOs have to do with the data. So a call to the PINGOs looks like:

ext add ifile1 ifile2 ofile

The data from ifile2 will be added to the data of ifile1 and the result is written as EXTRA formatted data to ofile. You will see examples of what is to do if the program needs further information on the following pages.

And what's about the DKRZ-Servers?
PINGOs are install in afs, i.e. there links in /client/bin pointing to the PINGOs and the grid and weight descriptionfiles are to be found in /client/bin/grids.

How to get informations about my files

Examples on how to get informations about the data file:

grb info ifile

Writes date, code, level, size and minimum, maximum and average for each single record in file ifile to standard output. Standard output is the display. For more information use linfo and redirection, if you need that infos in a file.

grb linfo ifile > ofile

longinfo writes the whole data set to standard output.

grb nrec ifile

displays the number of records.

grb nlevel ifile

the number of levels for each code
more possible commands working the same way:
gribinfo, formatinfo, pipeinfo, nyear, nmon, ndate, ntime, ncode, showyear, showmon, showtime, showcode, showlevel

How to change the header

All commands to set or to change the header begin with ch(ange):
chdate, chtime, chyear, chmon, chday, chcode, chcodes, chlevel, chgriddes

grb info zzz

produces the following output
REC : DATE COD LEVEL SIZE : MINIMUM MEAN MAXIMUM : MISS
1 :18600101 11 2 128* 64 : 2.310e+02 2.767e+02 3.111e+02 : 0
2 :18600201 11 2 128* 64 : 2.276e+02 2.762e+02 3.121e+02 : 0

use chyear, chmon, chday to change the date in all records of the file to one value:

grb chmon,5 zzz zzzmon

see the output of the info command to see what has changed:

grb info zzzmon

REC : DATE COD LEVEL SIZE : MINIMUM MEAN MAXIMUM : MISS
1 :18600501 11 2 128* 64 : 2.310e+02 2.767e+02 3.111e+02 : 0
2 :18600501 11 2 128* 64 : 2.276e+02 2.762e+02 3.121e+02 : 0

To get a consecutive date use chdate:

grb chdate,19700101,00010000 zzz1 zzz3

The following is what PINGOs display to your monitor. If no data following the command chdate seperated with colons, the chdate command asks for them.

grb chdate: Enter start date and date step!
grb chdate> 19700101 00010000
OK
grb chdate: Processed 10 records.

grb info zzz3

See the info command output again to see what happened:

REC : DATE COD LEVEL SIZE : MINIMUM MEAN MAXIMUM : MISS
1 :19700101 11 2 128* 64 : 2.310e+02 2.767e+02 3.111e+02 : 0
2 :19710101 11 2 128* 64 : 2.276e+02 2.762e+02 3.121e+02 : 0
3 :19720101 11 2 128* 64 : 2.172e+02 2.770e+02 3.093e+02 : 0

I key in the start date (19700101) and the increment (00010000) and the record date changes year by year. (increment 00000100 for monthly and 00000001 for daily changes).
chcode and chcodes change one or more codes and chlevel one or more level. See the Big Manual for further information.

grb chcode,3 zzz zzc3

grb chcode: Enter code!
grb chcode> 3
OK
grb chcode: Processed 10 records.

grb info zzc3

REC : DATE COD LEVEL SIZE : MINIMUM MEAN MAXIMUM : MISS
1 :18600101 3 2 128* 64 : 2.310e+02 2.767e+02 3.111e+02 : 0
2 :18600201 3 2 128* 64 : 2.276e+02 2.762e+02 3.121e+02 : 0
3 :18600301 3 2 128* 64 : 2.172e+02 2.770e+02 3.093e+02 : 0

Fieldmanipulation

Enlarging, shrinking, masking and thinning out of fields is possible with the PINGOs. The commands are named melt, break, enlarge, thinout, mask und normalize and the experts will find informations about it in the DKRZ TR 11.
If you have to compare data fields from different models, you will see that the fields have different grids, or one grid points from north to south and the other vice versa. In this case the field must be swapped:

lola swaptopbottom ifile ofile

lola swapbottomtop ifile ofile

A

lola swapleftright ifile ofile

lola swaprightleft ifile ofile

rotates the field round the vertical axis.
Attention! After such a rotation you must add the new grid description:

lola chgriddes ifile ofile < $GRIDS/gitter.grid.asc

gitter stand for something like t42, t106 oder r360x180 (see Getting started).

Split and Merge

With the PINGOs I can change the order of records in a file or cut one big file into lots of small ones, or merge several small files to a big one:

grb reverse ifile ofile

changes rec(1) .... rec(n) to rec(n) ... rec(1).

grb transpose ifile ofile

for the transpose command ifile looks like a big matrix, with every record as a row. The matrix ifile(x,t) becomes ofile(t,x).

grb split<n> ifile ofile1 .... ofile<n>

splits one big file to <n> small ones, so that rec(n+1) becomes the second record of ofile1, usw. Attention! For the time being there is an error in the split command. The number of outputfiles must be a divisor of the number of records in the inputfile.

splitmon and splitseas split one file in 12 i.e. 4 pieces.
merge, mergetime<n>, mergedate2 merge several ifiles to one ofile.
Using mergetime and mergedate the records will be stored in ofile sorted by date. With merge rec1(ifile1) becomes rec1(ofile), rec1(ifile2) becomes rec2(ofile), til rec1(ifile<n>) becomes recn(ofile); then rec<n>+1(ofile) is rec2(ifile1) and so on. .

merge<n> ifile1 ifile2 .... ifile<n> ofile

As I told you in the first section, the PINGOs do not check if your work is right. So sometimes I must extract times series of codes or all levels of a code for further manipulation. Two shell-scripts exist for selecting months and levels. They called onlevel and onmonth and are part of the package. They use the split - command so be careful that the number of outputfiles is a divisor of the numbers of records of the inputfile. "By hand" I will select as follows:

grb selrec,1,10 ifile ofile

the first 10 records are written to ofile.

grb seldate,19700101,19791201 ifile ofile

all records starting with 1970, january, 1. to 1979, december, 1. will be written to ofile.
Other commands for timeselection: selyear, selmon, selseas, selday.

grb selcode,3 ifile ofile

all records of the code 3 are written to ofile.

grb sellevel,50,500 ifile ofile

all records with levels between 50 and 500 hP are written to ofile. Modell output files cover the whole earth. If I only need a part from that, maybe europe, I can select these data and reduce the filesize noticeable. So I like to cut out europe. My file contains fields in t21 solution (64 x 32 gridpoints, see picture). I can select with gridpoints:

grb selindexbox,62,12,1,10 ifile ofile

with 62,12 describing the most west and east and 1,10 the most north and south spot of my rectangle. If I know the longitudes and lattitudes of my box I work with

grb sellonlatbox,340,60,85,35 ifile ofile

The picture:

Missing Data

Not all records are complete. Sometimes some values are missing so that calculations will lead to wrong results. Die PINGOs contain commands that will find these "missing values" , mark them and/or replace them with an other value. Mostly these missing values will be signed with a constant value (9.9999E99) These value I can replace with a marker.

grb setctomiss,<konstante> ifile ofile

with

grb countmiss ifile ofile

I will prove if it worked. In ofile you will find 1 where a mv is and 0 else:

ext linfo zzmo |more

Record 1: Date=19540327 Time=0 Code=143 Level=0hPa Size=128*64
1, 1: 1.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00
7, 1: 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00
13, 1: 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00

In my record the mv's are replaced by -. Now I can replace these - with new values:

grb setmiss ifile1 ifile2 ofile

See also:
setmissc, setnotmiss, setnotmissc, countmissr, countnotmiss, countnotmissr, packr,unpackr.

From Ascii to PINGO

The PINGOs not only will read its own formats but also ascii-formated files. These files must contain floating point numbers or exponetial numbers, where the exponent will be preluded by e or E, seperated with blancs or tabulars. Allowed examples:

1.7 -33.4765 0.99E-01 0.11e09

A PINGO accepted file must also have a header with elementary informations: the size of the field, date and date step, time and time step, code, number of levels and level. This informations will be entered after the command seperated by colons or asked interactive. The input for a t21 solution field starting with 1.1.1999, date step one year, no time steps, code 167, 1 level, level 0, looks like:

ext input,2048,19990101,00010000,0,0,167,1,0 ofile.ext < ifile.asc

If the outputfile should contain grid informations change to lola output:

lola copy ifile.ext ofile.lola < $GRIDS/t21.grid.asc

The commands above in one step with pipes:

echo 19990101 00010000 0 0 167 1 0 64 32 | cat $GRID/t21.grid.asc\ - ifile.asc | lola input ofile.lola

If there is no grid description but another lola file with the same solution I can seperate the grid description from this file:

ext griddes ifile.lola > t21.grid.asc

Area Weights

You will find lots of informations to the concept of area weight in the big manual. The grids used are rectengular and 2 dimensional, so I have to have a compensation for the different distance of two longitudes near the equator and the poles by computing means. I can use these area weight files. For the known grids these files are in the directory $GRIDS. lola or grip files know the grid and the PINGOs automatically use weights by computing means. Else I must expand the command with the weight file using

ext meanrw ifile $GRIDS/t21.weights.lola ofile

If the weight file is missing I will compute one:

lola weight0 r72x36.weights.lola < r72x36.grid.asc

Instead of a weight file I can use a constant weight. This will also make sense sometimes.?:

ext meanrw ifile -const,1 ofile

Pipes

A unix-user knows ( or should know) pipes, a way to use the output of a command directly as an input for another command. An example:

echo 19990101 00010000 0 0 167 1 0 64 32 | cat $GRID/t21.grid.asc\ - ifile.asc |lola input ofile.lola

The pipe character is the |.
The PINGOs also allow to pipe commands:

lola info -selrec,1,10 ifile

The command info uses the output of the command selrec as input. Only the first 10 records will displayed.

ext info -div ifile -sum ifile

I calculate the sum over all records of ifile, substract this sum from every record and display the result.
Those who do not like pipes, because it's not exactly predictable, use shell scripts.

GrADS

After all my computings are done I would like to have a picture of my results. The PINGOs compute the necessary data and description files with one command:

pure grads ifile ofile.grads > ofile.ges

The ofile.grads contains the data and the ofile.ges the description of the grid and the data. GrADS uses the descriptionfile .ges if I use 'open ofile.ges' after starting GrADS (The left part of the following table shows a ".ges" file.). See the right part of the table for a looklike of the output:

#!/bin/csh -evx # set LIBS = "/pf/k/k204016/paletten" set WORK = "/mf/k/k204016/IPCC/SRES/work" set WER = $1 set COL = temk cd $WORK # cat > REIN.gs <<ENDE 'reinit' 'set vpage 0 11 0.5 8.5' 'set display color white' 'clear' 'set gxout shaded' 'run $LIBS/palette.joerg3.gs' * 'open $WER.ges' 'set parea 1 10 0.5 8' 'run $LIBS/${COL}_col_input.gs' 'set lon -180 180' 'display c3' * 'run /client/grads-1.6/lib/cbar.gs' 'draw string 1.5 0.5 $WER' * 'enable print ./$WER.mf' 'print' 'disable print' *'quit' * ENDE # grads -lc 'run REIN.gs' # rm REIN.gs gxps -c -i ./$WER.mf -o ./$WER.ps # exit

(example)

Example

We want to calculate the mean value about 3 temperature fields (ie the difference between time slot to compare and reference time slot). We have a correct MP* file, HC* has a different grid and NP* data are ascii.

-rw-r--r-- 1 k204016 k20400 28120 Oct 26 08:42 HC_temk.hc
-rw-r--r-- 1 k204016 k20400 33104 Oct 26 08:37 MP_temk.t42
-rw-r--r-- 1 k204016 k20400 82944 Oct 26 08:36 NC_temk.asc

The NC* file must be converted: Read ascii data and fille the header (ext input), include grid (lola copy), the NC-file stores the data south to north and must be swapped (lola swaptopbottom) and after that the grid must be newly included ( lola chgriddes):

ext input,8192,20700101,00010000,01,01,3,1,0 NC_temk.ext < NC_temk.asc

lola copy NC_temk.ext NC_temk.uo < $GRIDS/t42.grid.asc

lola swaptopbottom NC_temk.uo NC_temk.ou

lola chgriddes NC_temk.ou NC_temk.t42 < $GRIDS/t42.grid.asc

The HC* file becomes a new grid:

lola interpolate HC_temk.hc HC_temk.t42 < $GRIDS/t42.grid.asc

Make one file out of three files:

lola cat3 NC_temk.t42 MP_temk.t42 HC_temk.t42 SUM_temk.t42

lola chlevel,1,0 SUM_temk.t42 SUM_temk.l0.t42

lola chdate,20700101,00000000 SUM_temk.l0.t42 SUM_temk.l1.t42

lola info SUM_temk.l1.t42

REC : DATE COD LEVEL SIZE : MINIMUM MEAN MAXIMUM : MISS
1 :20700101 3 0 128* 64 : -6.884e-01 2.839e+00 9.997e+00 : 0
2 :20700101 3 0 128* 64 : -1.615e+00 3.606e+00 1.191e+01 : 0
3 :20700101 3 0 128* 64 : 9.032e-01 3.735e+00 9.660e+00 : 0
lola info: Processed 3 records.

lola mean SUM_temk.l1.t42 SUM_temk.mean.t42

lola info SUM_temk.mean.t42

REC : DATE COD LEVEL SIZE : MINIMUM MEAN MAXIMUM : MISS
1 :20700101 3 0 128* 64 : 5.830e-01 3.393e+00 9.128e+00 : 0
lola info: Processed 1 record.

pure grads SUM_temk.mean.t42 SUM_temk.mean.grads > SUM_temk.mean.ges

The first two commands manipulate the header of the file. All records are changed to the same date and same level. The lola mean command produces the mean value for each gridpoint over the three records and 'pure grads' creates two files to use with GrADS. The result is to be seen on the

Grids

The assignment of grids to fields was mentioned in this 'How to get started' before. I need this grids for weighted means and grads output.
If I have a Extra, Simple or Service file I can assign a grid to my data with

lola copy ifile.ext ofile.lola < $GRIDS/t42.grid.asc

If I have to compare fields with different grids i must interpolate:

lola interpolate ifile ofile < $GRIDS/t42.grid.asc

The PINGOs really change the data field by interpolating. That means that this command takes some time. A copy or chgriddes command on the other hand does only change the header, not the data.

lola chgriddes ifile ofile < $GRIDS/t42.grid.asc

If a file contains a grid description I can extract it and use it for further files.

ext griddes ifile.lola > t42.grid.asc

Contact

The PINGOS will be maintained and distributed by

Jörg Wegner
Max-Planck-Institut für Meteorologie
Bundesstraße 55
Telefon 040 41173 178
email: wegner@dkrz.de

For the time being the PINGOS are available for SUN-Solaris and Linux. The distribution take place via anonymous ftp server and internet.
ftp ftp.dkrz.de
The files on our ftp-server will only survive for about a week. Therefore You have to send me a mail and I will place a .tar file in the directory
/pub/Outgoing/wegner
, which contains the latest version of the PINGO package, an install file, documentaion, grid and weight files.

Hint

This Program is not bugfree! For the time being some routines are not working like they should! Neither the creators of this program nor the DKRZ or the Max Planck Institut for Meteorology or myself, make warranties, expressed or implied that the PINGOs software package runs free of error. Like for all software the user has to question the results. He himself is responsible for his work with the PINGOs and the results.