"Rice-compression" utilities for 16-bit integer FITS images Michael Richmond March 28, 1996 ----------------------------------------------------------------------------- Contents: Credits Introduction How to Build the programs Performance on various platforms Bonus: Free Bit I/O routines ----------------------------------------------------------------------------- CREDITS: These programs are based upon algorithms probably created by Robert Rice and Pen-Shu Yeh, and definitely developed by Nan Ellman and Robert Lupton. Nan and Robert L. have created versions for use in dealing with Sloan Digital Sky Survey images. All the afore-mentioned did the hard part. All I did was to put the algorithms into practice. INTRODUCTION: The files rice.c bits.c bits.h rice.1 Makefile README can be used to make the executable programs rice unrice These two programs make a small, standalone package for compressing and uncompressing 16-bit integer FITS images. They will NOT work on 8-bit, or 32-bit integer FITS images, or 32-bit floating-point FITS images. They will NOT work on FITS binary tables, or images with extensions. The programs will fail to work if run on a machine which does not represent 2-byte integers with the most-significant-byte first. For example, the code fails to compress data on a Dec Alpha running Digital Unix. The algorithm used for compression is described in the man page (rice.1). It works pretty well for "typical" astronomical images, as long as - there is a lot of "empty" sky - the noise in the sky is less than, say, 200 ADU For the few images I tested, using both real and simulated astronomical data, I found a typical best performance of about 5-7 bits per pixel. That means that a large image, one in which the FITS header is a negligible fraction of its size, will shrink to 5 - 7 bits ------------- = 31 - 44 percent 16 bits of the original file size. This is, I hope, a significant enough improvement over the standard "compress" or "gzip" programs that it might be of use to astronomers with lots of images. These programs compress the data section of the FITS file, but leave the FITS header intact. Thus, one can examine the FITS header of a compressed file in the same way that one would a normal FITS file (i.e. via a simple "more" command). Note that rice/unrice does NOT remove the orginal/compressed file! Instead, the originals are left on disk for the user to remove, if he desires. Users can ask "rice" to look for the best choice of "noise_bits" when compressing (see the man page), they can supply a value by hand, or they can allow the program to use its default value of noise_bits = 5. There is a special keyword, "unsigned", to allow one to handle the 16-bit unsigned pseudo-FITS images used in the SDSS collaboration. HOW TO BUILD THE PROGRAMS You should be able to to copy the file "rice.shar" to your local machine, and place it into a (preferably empty) directory. Then type sh rice.shar which should create files Makefile README bits.c bits.h rice.1 rice.c If you don't have the "sh" shell (or any substitute), it isn't too painful to edit the "rice.shar" file by hand to create these files. The only file you may have to edit is "Makefile". If necessary, edit the "CC" and "CFLAGS" lines as needed to invoke your ANSI C compiler. Also, check to see if you need to define the macros "BSTRING" or "NOBCOPY" in the Makefile. When the compiler lines are correct, simply type make and watch as the files rice unrice are created. Note that "unrice" is simply a link to "rice". The programs depend upon being named "rice" and "unrice", so if you want to change their names, you may have to edit the "main" routine in "rice.c". See the man page for usage. PERFORMANCE ON VARIOUS PLATFORMS I have built and run the code successfully on several platforms (and seen it fail on another). As a test, I used a 16-bit integer FITS image taken from Tom Droege's TASS camera. This image, "orion11", has 800 rows and 768 columns. Below is the time taken to compress, and uncompress, this image on various machines; in each case, I compiled with the -O2 flag. I made some of the test runs on machines which were serving several other users at the same time. machine OS Mflops compress uncompress ------------------------------------------------------------------ Sun 4/50 IPX Solaris 2.4 4.2 4 sec 7 sec Sun SS1000MP Solaris 2.4 14*2 1 2 SGI Indy R4400 IRIX 5.2 24 1 1 SGI Challenge IRIX 5.2 3 7 PC (AMD DX4-120) Linux 1.2.8 8 8 I suspect that disk access via NFS slowed down the SGI Challenge considerably. BONUS: FREE BIT I/O ROUTINES As a bonus, you can use the C functions in "bits.c" and "bits.h" with other programs that you may wish to write. There are several functions that make up a not-quite-complete set of bitwise I/O routines. Feel free to copy and modify as desired; be careful to keep a copy of the original for use in building "rice" and "unrice", though.