The HERMES software was written around a database concept: ADAMO. ADAMO is a CERNLIB programming library that provides such a database concept, and utility routines for its management. You must understand something about ADAMO if you want to interact with any of the HERMES data files.
In addition, an extension was written to ADAMO by famous HERMES programmer Wolfgang Wander. This extension package is called DAD, for Distributed ADAMO Database. DAD is a very powerful library which enables server and client programs to talk to each other through pipes, using ADAMO. DAD also provides its own native output format, or driver, as you will see below.
It might be a good idea to scan the manuals first. You can find the manuals at the ADAMO Home Page. As you will see, both a Users Guide and a Reference Manual are available. They are available directly in HTML form, or you can download the postscript versions and print them. Please note: printed versions of both manuals are available in the computing center at DESY.
You might like to read quickly through the Users Guide for now, to get an idea of the ADAMO database concept and pick up some of the jargon. But the idea is not to perform any in-depth reading now ... and anyway I will summarize the main concepts in the next section.
After you have finished all the tutorials here, and become a bona fide ADAMO expert, you will find the following web page incredibly useful: Alphabetic List of ADAMO Routines. Bookmark this page! It is basically a jumptable to the Reference Manual. If you forget the syntax of any particular command, just click and there you go.
Here is a summary of the principal concepts and jargon of ADAMO:
Programming Steps
Here are the basic components of programming in ADAMO. We will go through
each step in the tutorials.
makeddl
to interact with MAD.
There are tons of these little scripts throughout the HERMES software
suite. The tutorials below will provide you with an example.
partap.h
(or partap.inc
in F77)
Drivers
A note about ADAMO drivers. ADAMO GAFs come in a variety of formats,
because ADAMO can use one of several drivers
to create these files. The informtion contained is the same, only the
output format is different. The filename extension of a GAF usually
tells you what driver was used to create it.
Writing a GAF
In this tutorial, we will create a simple toy Monte Carlo program,
which generates 10 simulated events and writes them out to a GAF in
FZ format.
Start a new directory somewhere, and download the tutorial files as they appear. This tutorial consists of all the source code you need to create your application. The source codes intentionally go through every command you are likely to need during your ADAMO lifetime. They are jam-packed with comments: the whole point of the tutorial is for you to go through the source codes themselves very carefully and read the comments to understand what each command does. Also, the example codes are available in both C and FORTRAN (F77), just select the language you prefer. (Thanks to Cynthia Chiang for the C conversion!)
Download: toyMC.ddl
This is the complete DDL structure for your toy Monte Carlo. It contains examples of all common data structures and relationships. Go through the file carefully, and make sure you understand the data structures we will be creating. You might find it helpful to make a diagram of the tables and dataflows, and the relationships between them.
Processing the DDL file with makeddl
Download: makeddl (C) or makeddl (F77)
This is an example of amakeddl
script. It is just a shell
script. It creates for you a file called mad.kumac
and then runs MAD on your DDL file using this kumac file.
After you have looked through the script, and read the comments,
run the script (i.e. just type makeddl).
Did UNIX come back with a 'Permission denied' error message? If so, you forgot to do chmod a+x makeddl, to give yourself execute permission on the file. Remember the scripting section of your UNIX survival training ...
Now examine the output files that were created by makeddl.
COMMON /mcEvent/ mcEvent, + mcEvent_ID, + mcEvent_Nu, + mcEvent_Q2, + mcEvent_mcBeam, + mcEvent_9999These are the variables you will to access the mcEvent table. This common block is called a Window Common Block or WCB. The WCB is also sometimes referred to as the WIN (window). It basically provides a holding area for one row of information from your table at a time. Some features common to all the WCB's:
Note that all of your table include files appear at the top, along with partap.h (.inc).
Download: write-gaf.c or write-gaf.F
There you are, all the ADAMO commands you will likely ever need to create your own GAF files. Go through the source code in detail. If you need more information on a particular command, consult the ADAMO manual.
Download:
configure
Download:
Makefile.in (C)
or
Makefile.in (F77)
configure make write-gafand your executable file should be created without problems.
Now run the program: just type write-gaf. You will see that your output GAF, called toyMC.events, has been created. In addition, a lot of information got dumped to screen ... this is the output of the various pedagogical print statements that were littered throughout the code. I suggest you sent this screen output to a file:
write-gaf > write-output.txtNow look through the output along with the source code, to see what some of the commands actually did.
Examining a GAF
So wonderful, you have just written a beautiful GAF. But how do you
look at it? We used the FZ driver, so just like most of the HERMES data
files this GAF is in unreadable binary format. Here is one of the most
potentially irritating things about ADAMO: you do you look at your output
easily?
Using the Pink Browser
The pink browser pb is a window-driven visual interface
to ADAMO files. This nifty program understands all GAF driver formats,
and even knows what to do with gzipped files.
The browser is very easy to use: just type pb.
The pink browser main window will appear. I'm sure you can figure out
very quickly how to read in and display your GAF file.
The basic button-pushing sequence was already described during your
Reconnaissance training:
The pink browser supports a few command line options, which can be listed by typing pb -usage. Among them are various ways to specify your input file directly on the command line. The option you use depends on the format of your file.
Using hexe
One of the most useful, and most ignored pieces of code in the HERMES
software suite is a little program called hexe. This program
is part of DAD, and is amazingly powerful. I won't go into much detail
here ... just point out a few tricks.
First, hexe can convert GAFs between all available formats. Why is that helpful? Because converting to the IE format will make our new GAF toyMC.events readable with a text editor! Here's what you do:
hexe --driver 'FZ,FILFOR=EXCH' toyMC.events --driver IE --outfile toyMC.ieor if you prefer
hexe -d 'FZ,FILFOR=EXCH' toyMC.events -d IE -o toyMC.ieYou can now examine your GAF directly. Please do so! (Now that the GAF is in IE format, you can even edit it directly ...)
For big files like the micro DST's, you probably don't want to convert the entire thing to IE format (which would be a huge file!). Instead, you may want to pull out only one event or burst and look at it in detail. (For example, you might be doing an event-by-event comparison with another analyzer, and want to investigate a handful of cases where you disagree). Hexe's filtering capability makes this easy:
hexe -z -d RFIL run12345.smdst.gz -e 'g1DAQ:1:iBurst=26255' \ -d IE -o run12345.ie hexe -z -d RFIL run12345.smdst.gz -e 'g1Track:1:iEvent=25579528' \ -f 'g1Track:iEvent<>25579528' -d IE -o run12345.ieThe first command performs record-level filtering with the -e option: only those records where at least 1 row of the table g1DAQ has burst number 26255 are kept. The second command, with the -f option, also filters out certain table rows, keeping only the tracks from event 25579528. Note that the -e and -f conditions work in the opposite sense: records are kept when the -e condition is true, but table rows are thrown away when the -f condition is true. Finally, the -z option in each example informs hexe that the input file is compressed.
As you see, hexe has some very powerful features ... you might want to become familiar with this package! To learn more, consult the Hexe Documentation. This very readable manual contains helpful examples. You can also type hexe --help to obtain a summary of command line options. Also, the filtering expressions you saw above are written in dad parser syntax. This powerful syntax includes common C operators and even ADAMO commands! This syntax is described on the Dad Parser page (which is linked off the hexe page).
Here is another useful application of hexe: quickly extracting certain pieces of information from a GAF or DAD file. Hexe allows you to print out individual entries from individual tables! Suppose you want to find out: what was the minimum energy cut placed on photon clusters in a certain uDST production? It used to be 0.8 GeV ... recently it changed to 0.5 GeV, and the set of release notes for the productions is sadly incomplete :-/. Here's a fast way of finding out. Let's check one run of the 97b4 production:
cd ~opa/udstprod/udst_97b4/smlinks hexe -z -d RFIL -n -p 'smCluster::%f:rE' run20000.smdst.gz | sort | headHah! The lowest cluster energy is set to the old value of 0.8 GeV. If you ran this command on a run from the 00b1 production, we would see that the new cutoff of 0.5 GeV was used. The new options that appeared in this example are:
hexe -z -d RFIL -n -p 'g1Track::%X:(iMethod&3):%d:iEvent:%f:rEnergy' run20000.smdst.gzThe first column of your output contains the two lowest-order bits of g1Track.iMethod, which tells you the tracking method used (0=STD, 1=NOVC). You will see that most tracks were found by both the STD and NOVC tracking methods, with slightly different reconstructed momenta ... but some tracks were found by only one of the two methods.
Again, the value and condition strings can be any Dad Parser expression. As I mentioned above, common C operators are available, so you can e.g. add two values from a table before printing. You can even use ADAMO commands to navigate from one table to another! Some impressive examples are provided on the Hexe Documentation page. With data and Monte Carlo files in event-level HRC format, you can practically perform an entire analysis like this. However, the burst-level / track-level structure of the uDST files makes this more difficult.
A little note: I have occasionally seen hexe experience difficulties when converting Monte Carlo DST files to GAF format (FZ, IE, etc ...): hexe complains about missing dataflows or tables and dies. This can be fixed by first removing the key table from your uDST file:
hexe -z -d RFIL file.smdst.gz -f g1Key:1 -d WFIL -o newfile.smdst.gzYou should now be able to convert the new DST file to any format.
Using tip
Finally, there is a program called tip which is part of
ADAMO. This is a third way to view your GAFs. It is basically an extension
of PAW to include ADAMO commands, so if you get good with TIP, you can
even do some amount of analysis on your GAF directly. The TIP documentation
can be found within the ADAMO manuals themselves.
Please note, since tip is a native part of ADAMO, it only understands ADAMO-native drivers: it cannot read GAFs in DAD format. Since most HERMES data files are in DAD format, this program has fallen into disuse. If you find you like tip, however, you can get around this limitation by using hexe to convert GAFs from DAD format to FZ format. Please note: if you specify -d FZ -o file.fz in the hexe command line, your output file will be in regular FZ format ... but if you leave off the -d FZ part entirely, you will get hexe's default output driver, which is the more familiar 'FZ,FILFOR=EXCH' driver. (You can also specify this driver explicitly, with -d 'FZ,FILFOR=EXCH').
I personally like tip, so I'll just show you a few commands to get started. These are enough to read in a GAF and display the information in the tables, which is usually all you want to do.
TIP> gaf/open 1 'name=toyMC.events,driver=FZ,filfor=EXCH' TIP> gaf/read 1 ! ! 2 TIP> tab/list TIP> tab/print ALL mcTrack TIP> tab/print 2 mcTrack TIP> gaf/read 1 TIP> gaf/rewind TIP> gaf/read 1 ! ! 1 TIP> gaf/close 1Here's what these commands did:
Reading a GAF
Those tools are certainly useful for viewing the contents of your GAFs.
But finally, you will want to do serious analysis on the data and for
that, we need more ADAMO commands and another example program.
Download: read-gaf.c or read-gaf.F
Once you have downloaded the file, you have all the tools you need to create the executable and run the program. Just type:make read-gaf read-gaf > read-output.txtNow, with the output file in hand, go carefully through the source code as before. Between the two source files read-gaf.F and write-gaf.F, we have literally covered all ADAMO commands in normal use.
Let me make one important remark about ADAMO. The structure of an ADAMO database is defined by the DDL file used to build it. Thus, when you write a program to access such a database, you usually need the DDL files that were used to build it. For example, if you write a program to read in uDST's from the 98b4 production, you will need the file g1_98B4.ddl, so that you can run MAD on it and get the source codes you need. However, our DDL files at HERMES sometimes change slightly over time. As things progress, columns may be added to tables, or removed from them. But here is where ADAMO is really clever: if you read in a GAF using a DDL file that is slightly different from the one used to create it, no problem! ADAMO is smart enough to project the GAF's actual structure onto the one you used to write your program.
This is not infallible of course: if your 98b4 uDST files contain a new variable mystery_pressure in table g1DAQ, and you are using an old DDL file that does not contain this new mystery variable, well then you have some problems. But everything is ok, as long as the variables you are interested in are present in both the GAF you are reading, and the DDL structure you are using to read it.
It is actually possible to write DDL-free code. Recall the programs hexe and pb that we discussed above: they are able to process and retrieve information from any ADAMO file you give them ... without knowing anything about the corresponding DDL file! This is an example of DDL-free coding. It is possible to do, and a few users have written all their programs in this way. But it requires significant work, and I will not describe the necessary techniques. The reason it is possible is that each GAF actually starts with an 'invisible' initial record, that contains metatables describing the entire structure of the database. This is also the reason that ADAMO is able to project one database structure onto another. If you have a look at the IE file you produced earlier, you will see all these metatables at the top of the file, before your actual data tables begin.
If you are interested in DDL-free coding, ask an expert, and have a look at the source code for an existing DDL-free program. A little clue: the routine FETCOL is useful when you do not have a WCB, and so cannot retrieve entire tables ...
Reading and Writing in DAD Format
Finally, since so much of the HERMES software reads and writes GAFs
in DAD format, let's learn how to do that. All we need to do is
replace a few of the ADAMO commands
from write-gaf.c (.F) and
read-gaf.c (.F) with their DAD library
equivalents. You should go through this exercise and see that it works.
LIBS = @std_library_path@ -ldad -ltap -lpacklib @extralibs@ @forlibs@When you rerun configure, this change will appear in your Makefile and will cause the loader to take the library libdad.a from $HERMES_ROOT/lib.
#include "dad.h" (C code) #include "dad/command.inc" (F77 code)The preprocessor will find the file in directory $HERMES_ROOT/include.
int dadflags;Then remove the CREGAF command from your code, and replace it with these lines:
dadflags = CAC_WRITE + CAC_KEYTABLE; dadDoConnect ( &igaf, /* identifier for the new file */ "toyMC.devents", /* file name */ "WFIL", /* driver */ 0, /* port (only needed for servers) */ "mcEvents", /* dataflow name */ "mcEvKey", /* name of key table */ dadflags, /* sum of flags */ &iok ); /* error code */(In F77, omit the & address symbols in front of the first and last arguments, add a CALL statement in front of dadDoConnect, change the double quotes to single quotes, and of course change the general syntax. The same applies to all the other code changes described below.) I wrote the code this way to indicate what each argument to dadDoConnect means. By the way, if you want to write compressed DAD output, add this line when you are setting up the dadflags option variable.
dadflags = dadflags + CAC_COMPRESS;
dadFillRecordKey (igaf, &mcEvKey, &iok); dadWrite (igaf, &iok);
dadDisConnect (igaf,&iok);
int dadflags;Then, replace the OPEGAF command with this:
dadflags = CAC_READ + CAC_KEYTABLE; dadDoConnect ( &igaf, /* identifier for the new file */ "toyMC.devents", /* file name */ "RFIL", /* driver */ 0, /* port (only needed for servers) */ "mcEvents", /* dataflow name */ "mcEvKey", /* name of key table */ dadflags, /* sum of flags */ &iok ); /* error code */Compared to the previous section, CAC_WRITE becomes CAC_READ and driver WFIL becomes driver RFIL. CAC_COMPRESS can again be used to indicated compressed dad format.
dadRead (igaf, &iok);Now just remove the FETGAF statements completely, DAD has already read in the whole record.
dadGetRecordKey (igaf, &mcEvKey, &iok);
dadDisConnect (igaf,&iok)
Pipes, Servers, and dadinit.cnf
This section can probably be skipped if you have no plans to do
software development.
You have now learned how programs can communicate with DAD-format ADAMO files. However, DAD provides more I/O methods than that. Recall that the whole point of DAD is to extend ADAMO so that clients and servers can exchange database information with each other directly. DAD thus supports several connection types:
In DAD, the communication method can be changed very easily. Remember the command dadDoConnect that we saw earlier? We used connection types WFIL and RFIL, for communication with files, but we could have used one of the other types. As far as DAD is concerned, it is just connecting to a host. This can be a file, a pipe, or a server ... all DAD needs is the host specification, which consists of these pieces of information:
Code-wise, DAD offers several ways to make a host connection:
I would like to briefly point out the last method. It allows the host definitions to be made outside the program, using an input file called dadinit.cnf. I mention this because you will find such files scattered around the HERMES software directories. They are typically used for making socket connections to the standard servers which are running on our machines. Have a look at the file /hermes/pro/lib/dadinit.cnf on the SGI, for example. You will see a number of calibration-type servers running on geordi, which is one of the machines on our production farm. The HDC program needs the information from these servers. If you chose to run HDC yourself on the SGI, you would need to connect to them, using this dadinit.cnf file. As an example, here is an excerpt from the official file on the SGI, showing the two host definitions that comprise the GServer:
#$ Geometry 19990 /tmp/dadgs.00a GServer # geordi INET 19990 dgGeometries r geordi INET 19990 SLTtable r #The order of the specification words for each host is:
Download: dad-io.tar.gz (C) or dad-io.tar.gz (F77)
The tar file above contains two little programs write-dad and read-dad, along with a configure script (same one as before), a Makefile.in template, a ddl file, and makeddl script. The programs are stripped down versions of write-gaf and read-gaf that include working examples of the different types of DAD connections (except for socket connections). Option 5 allows you to explore the use of a dadinit.cnf file. Option 6 illustrates pipes: try it! If you start the program write-dad with this option, it will open a pipe but then wait for another program to connect to it. So, start read-dad ... poof! The two programs will then run, communciating via pipe. Cool :)
Exercise: Read the geometry file
Here's the mission, Cadet: write a program to read
in the HERMES geometry file and extract
two pieces of information from it.
First, you will have to find the geometry file hmcdg.ie, and the DDL files that define its structure. As you learned in your HERMES Reconnaissance training, the ddl files are named GEOMETRY.ddl and GEOGEANT.ddl.
Let me tell you enough about the DDLs to complete this exercise. You will only have to interact with a handful of tables.
(dgDETINFO [1,1] -> [0,1] dgVOLU | -> [0,1] dgDVN | -> [0,1] dgDVN2 BY Volume) : 'detectors have a certain kind of volume'; (dgPOS [1,1] -> [0,*] dgVOLU) : 'volume shapes are positioned'; (dgPOS [1,1] -> [0,*] dgVOLU | -> [0,*] dgDVN BY Mother) : 'volumes are pos. inside mothers';
All clear? Good. Write your program, Cadet ...
If you did everything correctly, your program should produce something like this as output:
>>> Detector BRLV (or BRUV) is furthest downstream, at z = 1120.00 cm >>> The mother volume of V1U4 is called V1UA
Download: makeddl Makefile.in read-gaf.c (C)
Download: makeddl Makefile.in read-gaf.F (F77)