Pipeline Operation

[Tom Droege <tdroege2@earthlink.net>]

The operation of the pipeline has become quite routine here.  I am quite 
happy with the results (well, the low mag end is not what I would like).

I have one Windows 98 computer running each telescope.  I have everything 
down to a routine.  Early evening if the weather looks promising (using 
several weather maps), I go up and check and refill if necessary the camera 
desiccant and the water in the wet bulb thermometer.  I then open things 
up, and start the cameras running warm with the dry air circulation on.

At dusk, I start the cameras taking data.  Since they are warm (and 
hopefully dried out) it takes a while to cool down.  This works just fine 
since there are 20 or so image pairs taken before it really gets 
dark.  During this time the cameras cool down.

When it gets dark enough to get good images, I go into my office and start 
looking at the images with DS9.  I make sure the focus is OK, and that the 
systems are really running and taking images.  Often not the case as 
something goes wrong.  Yesterday it was TOM3 Windows 98 requiring a 
reload.  There is something like that almost every day.  I still have a 
noise problem with the TOM3 cable.  An early project is to make a better 
shielded cable.

If every thing is OK, I go off and enjoy life with Jennifer.  I come back 
every so often and look at images to make sure they are OK.

I wake up at dawn.  This is no big deal.  At 73 I never sleep through the 
night.  This just happens as you get older.  So I have no problem getting 
up and shutting down the three systems.  Last night clouds moved in early 
AM but there were later clear spots.  The cloud detector did it's thing on 
TOM1 and wrote darks when it was too cloudy for a good image.  I has this 
set just about right.

After I shut down the systems, I start the processing of the data.  I have 
3 linux computers set up, one for each TOM.  This helps prevent using the 
wrong flats and the like.  I just point the pipelines at the Windows 
computers for each TOM and transfer the files to the local processing 
computer.  After waiting to see that this process has actually started, I 
go back to bed.

When I get up, I check on the progress.  This usually involves looking at 
the images with DS9.  I check to see that the pipeline throws away the 
images that I think should be thrown away, and that the images that it 
keeps are even enough that the errors in processing should be within the 
noise.  The pipeline now does what I think is a very efficient job of 
this.  Last night was a typical partly cloudy night.  We got a lot of data 
in spite of the clouds.  The software just throws the right images away.

The make_list.out file is very useful to check on the progress of the 
processing.  I use it and DS9 mostly.  Sometimes along the way I make an 
intermediate list from the .ast files and plot them with gnuplot.  This 
will show when there were good periods and when there were clouds.

It is now 12:30 PM and TOM1 and TOM2 have finished processing.  There were 
55,000 pairs from TOM1 and 59,000 pairs from TOM2.  I expect a similar 
number from TOM3 which is almost done as I can tell from make_list.out.

About half the night was clear enough for data.  When the runs complete, I 
will make plots of the data just to check for things like the -90 
declination points.  Then the program stashes the results in a file.  I 
always check that it really did so before starting the next run which would 
otherwise erase the results.

Now I write all the corrected .fits files to CD by machine and JD.  At the 
moment, I must mark by hand each .fits file that has a matching .ast 
file.  This means that the pipeline successfully matched the file to the 
sky.  Eventually I will write a script that finds the right .fits files and 
writes them to sub files in the directory in blocks of 80.  Roughly what a 
CD will hold without compression.  This is a pain, but I am not ready to 
ask for help yet.  Eventually I will either have to write a script to do 
this, or ask for one of you to do it.  This would be no big deal for most 
of you, but it is a challenge for me.  But I can do it now, thanks to 
taking a Unix course last spring.

By the time I finish writing the CDs, it is time to start all over again 
with turning on the machines and opening up the dome if I have not left it 
open.  Sometimes the weather gives me a vacation night.  When I picked 
astronomy as a retirement project in Illinois, I figued I could not work to 
hard.  But I was wrong.  There is always something to do to fill up my days 
"off".

I really like the way the pipeline works.  Michael has done a wonderful job 
creating a practical pipeline.   I also think the results are good.  I 
think it will be hard to improve.  But we shall see.  I await one of you to 
send me some data to run through the pipeline for comparison.

So goes life in Batavia.

Tom Droege