Re: New data

[Tom Droege <tdroege2@earthlink.net>]

While the .lst files contain entries for all the images processed, I think 
that none of the bad ones have made it to the final output.  I don't think 
there were any stars closer than about 5 degrees to the pole that made it 
to the output in the test data I sent, but I could be wrong.  The list 
entry has to have a 1 at the end, and a real number in the stars found 
column, and it still might not end up in the final output.  Note that the 
star list only contains stars that were collated.  This means that they 
were found in V and I and met the limits set on sky and skysig.

Because of the memory board problem, there are images taken at +92 that I 
think are in the list.  These are absolute garbage.  The processing nicely 
eliminates such bad frames.

There are also darks mixed in with the object files.  If the clear sky 
detector thinks the sky is not clear it takes a dark.  I just write 
everything to CD and process it through the pipeline.   The pipeline just 
ignores the darks, though it will process a dark from them if you ask it 
(nicely).

Using the .lst and .config files, one can determine whether stars from an 
image made it into the output star list.  That is the main purpose of this 
exercise.  Get Michael to consider how to handle this in the data base.

Suppose that we want to pick a set of measurements made under the best 
conditions.  One can go through the .lst files and pick out images where 
the sky was exceptionally dark and/or the skysig was low.  I am not sure 
how to do this in a data base, but one can search for say low skysig and 
get a .lst line.   This tells you the image, the telescope, and the time 
the data was taken.  Can one now efficiently crank out a list of 
measurements taken under the best conditions?   This is a practical 
question for data base construction, I think.

Having done this exercise, one then goes through and picks out a set of 
stars where the sky was exceptionally bad.  Then one compares the two data 
sets.

I will bet a Nickle that there is not a big difference in photometric 
accuracy between the two data sets.   I have spent a lot of time running 
images with various cuts and think that Michael's pipeline does a good job 
until the sky gets pretty bad.  I have set the cuts at "pretty bad".  But 
all this needs to be tested.

In fact, data quality is one of the things to be investigated with this 
data.  This disk was sent as test data - really as data base construction 
test data.  OK, it was really sent so that Michael could see the lists that 
are now related to the data.  (Albeit I don't plan to process it again so 
it is really final data - but a test set of the final data) Put it in a 
data base and compare the stars that are close to the pole with a catalog 
and test the data!  I think the current process is quite successful in 
weeding out bad data.

So instead of setting aside measurements just because they are close to the 
pole, I suggest that you test the measurements and see if they are 
valid.  If the astrometry is good, then I see no reason why the photometry 
should be any different close to the pole.

I would say that a good test would be to select random stars out of the 
data I sent, look them up in VizieR and see if there is a star there with a 
match.  I would hope that this can be done with some automation so that 
some process sits and runs getting random stars out of the data I sent, 
looking it up in VizieR, and then computing some measure of error.  Then 
you get to decide whether VizieR is right or tass is right.

I think it is absolutely necessary that we do something like this.  Either 
Michael S. can do it, or he can put up another test data base, and one of 
the people on this list can have a go at it.

We certainly don't want bogus data in the final data base.  One of the 
things I try to do in setting up the pipeline is to weed out the bogus 
data.  We now need some validation tools to check on the data 
quality.  Lots of data is coming for this purpose.  I am hoping that some 
of you will look at it and then give direction -  like set the V sky level 
lower or higher or some such based on measurement and tests.

When you start to do this, I think you will find that more data is 
needed.  Sigh!  More data, more data, and more data are the three things 
needed.   It is like real estate.

Tom Droege

At 07:29 PM 5/16/03 -0400, Stupendous Man wrote:

>   Michael Sallman wrote:
>
> > I just received the sample of the new data from Tom.
> >
> > There appears to be a problem with images that include the pole.
> > I noticed in the *.list files that there were many images taken with
> > declination of +88. All (almost all, see below) of them had zeropoints
> > and color terms of 99.000.
> > In Mjra2761909.list, Mjra2762576.list and Mjra2764697.list there are
> > multiple images with declination of +92. They also had zeropoints and
> > color terms of 99.000 and most had negative sky values and/or skysig
> > value greater than sky value.
>
>   Tom and I have been discussing this issue via E-mail.  I have not
>yet had a chance to run these high-Dec images through the pipeline
>myself and watch each step of the procedure.  It does not surprise
>me that the astrometry routines fail this close to the pole.
>
>   I suggest strongly that any images which are centered closer than,
>say, 5 degrees from the pole be set aside for the present time.
>I suggest that they NOT be placed into Michael S.'s database.
>In my opinion, a database with completely reliable data is much
>more useful than one with even a little bogus data.
>
>   Other opinions?
>
>                                     Michael Richmond