[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Data sets over time
Chris and all,
At the present, all the data is in piles of CD cake boxes labeled by month
and camera. So there are about 50 cake boxes at the moment. 2000-2500
CDs. Growing by 3-6 cake boxes per month. Over 200 CDs from September so
far.
To set up a data base, the only way to get the information is to read each
fits header from each disk. So this involves loading 2500 or so CDs,
reading the headers and making up the information for the data base. While
at it, one might as well invest in a stack of hard disks and load the
images. Then one could extract images from such a file. There are a some
of tests I would like to do by processing at one time all the images
containing a particular star or region of the sky.
I probably don't want to load all the data. But loading one camera might
be reasonable. It might fit on 2-3 large drives. I worry that if I try to
do the whole thing then I will have to go to a raid structure to make sure
that everything works long enough to be useful.
So I am looking for a volunteer from this group to write a script that:
1) Prompts for loading disks and then reads the fits header and makes
appropriate tables.
2) Sets up a simple data base that say allows requests like "give me all
the images containing <region of the sky> and put them in a directory.
I suspect that this is only a few hours work for some of you that write
scripts for a living. I would do the work of setting it up and buy the
appropriate hardware. I assume there is a data base in the Mandrake
distribution that could be used for this.
Given such code, I would load up a camera worth of images and respond to
the group with requested data sets on CD. Not quite the coming data base
at DAO, but it could get us started on trying new types of analysis.
Tom Droege
> [Original Message]
> From: Chris Albertson <chrisalbertson90278@yahoo.com>
> To: <tdroege2@earthlink.net>
> Cc: <tass@listserv.wwa.com>
> Date: 9/15/2004 11:22:23 AM
> Subject: Re: Data sets over time
>
>
> Tom,
>
> I assume all your data is in a "database". The database would
> have pointers the FITS file each point came from and the
> parameters of that FITS file. Getting a list of the files that
> "saw" a given point of the sky can be done with a single SQL
> statement The SQL querry would ask for the set of images that
> intersect the study region in both space and time.
>
> The current database only provides point and
> click access to remote users and no button for ths kind of
> querry but someone
> with ability to type in direct SQL could extract the information
> you need in a minute or so. With more effort a new button could
> placed in the interface.
>
> Of course all the database will
> be able to tell you is the filesnames ofthe FITS images. I assume
> you have some way to physically locate the associated CD.
>
> The next question is why would you want FITS image data of the sky
> when you can have the reduced photometry? Are you working on an
> improved reduction pipeline?
>
> For those who don't know what is SQL and what it can do, here is the
> 60 second summary:
>
> SQL was invented by mathematicians, not by computer programmers.
> SQL does NOT say HOW to perform an operation, it only specifies
> the result. Basically you define a set and some constraints on
> that set that will pproducethe desired subset.
> An example SQL statment transliterates like this:
> "Give me the subset of ( a, d, f(y))
> such that a > d and f(y) < g(a,b)
> SQL is worth learning. The above is MUCH easier then writting
> a program to compute the result with loops and if statments.
>
>
> --- tom <tdroege2@earthlink.net> wrote:
>
> > Robert,
> >
> > One more thing. It is one of my projects to make just such data
> > sets. I will
> > need some software, possibly trivial for you but not so for me. The
> > software
> > needed would read disks full of images and look at the fits headers.
> > Then it
> > would load the fits images in such a way that one could select the
> > type of
> > data set you desire.
> >
> > It is always a big project to go through even one camera. However,
> > if it were
> > all set up I could just load disks when prompted and could get
> > through my
> > stash in a couple of weeks.
> >
> > The three telescope systems cover different areas of the sky. TOM1
> > covers -6
> > to +18, TOM2 +18 to +50, TOM3 +50 to +90. I could set up enough
> > storage on
> > one computer and load the disks from one camera, about 700 or so.
> >
> > TOM1 does overlap ASAS, don't know where OGLE is pointing.
> >
> > Tom
> >
> > On Wednesday 15 September 2004 07:28 am, you wrote:
> > > Tom,
> > >
> > > How difficult would it be for you to put together one
> > > or a few CDs (more than a few CDs I could not handle
> > > currently) of a specific region of sky on an extended
> > > time basis? I.e. months (years?) of observations of
> > > just a single (or a few) frames with approx. the same
> > > position?
> > >
> > > Also, for general analysis purposes it might be
> > > interesting to be able to construct such a
> > > time referenced data set with several points
> > > of view. I can think of perhaps 6 though I'm
> > > not sure you can capture them all.
> > > a) Directly towards the galactic center;
> > > b) Directly away from the galactic center;
> > > c) Directly forward of the sun's orbit around
> > > the galactic center;
> > > d) Directly aft of the sun's orbit around the
> > > galactic center;
> > > e) A view above the galactic bulge;
> > > f) A view below the galactic bulge.
> > >
> > > In particular, (e/f) may overlap with the OGLE III
> > > observations (though it is taking pictures from Chile
> > > and it isn't clear to me how much overlap may be available).
> > >
> > > This might feedback into the question as to whether
> > > the pipeline analysis needs to be adjusted by general
> > > background (star density, dust density, etc.).
> > >
> > > Thanks,
> > > Robert
> >
> >
>
>
> =====
> Chris Albertson
> Home: 310-376-1029 chrisalbertson90278@yahoo.com
> Cell: 310-990-7550
> Office: 310-336-5189 Christopher.J.Albertson@aero.org
> KG6OMK
>
>
>
> __________________________________
> Do you Yahoo!?
> Yahoo! Mail - 50x more storage than other providers!
> http://promotions.yahoo.com/new_mail