Hi Tom, You might want to consider using something like "pvm" or "GNU Queue" which allows you to run code on any number of disparate machines. http://www.csm.ornl.gov/pvm/pvm_home.html http://sourceforge.net/projects/queue/ Clearly the smallest "atomic" unit would be reduction of a VI pair of images. The next available CPU could just ask what image was next in the queue. (It is fairly straightforward to do this without either package, too, as you have already shown.) Cheers, Doug On Thu, 2004-08-26 at 12:15, Thomas Droege wrote: > Robert, > > I use 9 computers to process the data stream from the 3 dual telescopes. > > 3 are used to take and store the night's data. > > 6 are used to process the data. The scheme is to break the night's run > down the middle and process one telescope with two computers. > > It takes about 6 hours to process the data from one night. > > The telescopes are started at dusk, and at dawn programs are started that > copy the resulting images from the 3 data taking computers to the 6 > processing computers. > > This scheme has several advantages. The raw data can sit on the data > access computers until the next clear night. This means there is always 24 > hours to complete the processing. Since there are several processing > computers, if one dies, it's work can be transferred to one of the other > computer. I also keep a hot running spare which is loaded with the > pipelines so that it can take on the job if a computer fails. The > processing computers are mostly 1600 to 2600 Athlons. The data taking > computers need only 400 MHz or so capacity but they need ISA slots. These > are getting hard to find though I just found a modern 2600 Athlon that I am > saving as a spare. The data taking computers run Windows 98 while the > others run linux. I will switch over to linux everywhere as soon as I work > out how to do it. > > I seem to be losing a computer every couple of months. > > The results are written to the processing computer and two other computers > on the net to try to avoid loss of processed data. The dark subtracted and > flat fielded images are saved to CD. Just writing CDs takes several hours. > I will believe in other media when it has been proven to last for 10 years > or so. This will take 10 years and I doubt I will be around to make the > changeover. > > As I bring two more telescopes on line, I am adding to the CPU list. > > Tom Droege > > > > [Original Message] > > From: Robert J. Bradbury <bradbury@aeiveos.com> > > To: TASS <tass@listserv.wwa.com> > > Date: 8/26/2004 9:11:30 AM > > Subject: Storage breakthrough? > > > > > > You may be interested in the development of holographic > > data storage devices by Optware [1,2]. The media may be a bit > > pricey due to the dichroic layer but at 200-1000 GB/disk > > it will probably be worth it. > > > > At least in a couple of years it looks like one will be able > > to store the TASS database on a much smaller stack of "CDs". > > It would also appear likely that libraries are going to get > > significantly smaller... > > > > On a different topic... can someone send me information > > on the current data reduction hardware and CPU load > > requirements. > > > > Robert > > > > 1. http://www.optware.co.jp/english/what_040823.htm > > 2. http://slashdot.org/article.pl?sid=04/08/25/1639224 > > > > -- ================================================================== Douglas L Welch | Res office/voicemail (905) 525-9140 x23186 Physics & Astronomy | FAX (905) 546-1252 McMaster University | Hamilton, Ontario | Canada L8S 4M1 | E-mail welch@physics.mcmaster.ca ==================================================================
This is a digitally signed message part