Intro.

Arun has made a list of the files sent to TACC. This page uses the file: NAS_arun_listing dated 6sep22 (more files have been sent since then).

8712505 entries in NAS_arun_listing

most of these entries have a file size attached (the last few 100,000 don't)
I created a new file removing the 0 length files (433817 had 0 length)

8278688 entries in ./phil/NAS_arun_listing_nz (not zero).

I used this file to build the tables below.

Locations:

/share/s3453g1/keysha/Arun_Files/phil/NAS_arun_listing_nz.
I've also included some scripts, idl code in ../phil/ to break the files into various categories.

listing file	fileSize	#file entries	Notes
NAS_arun_listing_nz	832MB	8278688	includes path and basenames a copy of arun's original list with: zero length files removed rsync errors removed

Be carefull clicking on the large files, it can take a long time to display it

atm files

scanned arun's file looking for files

that had /tnnnn (atm proj usually started with t)
all files ending .shs (echotek files)
files started with hf_

**Atm related files**
listing file	fileSize	#file entries	Notes
		atm
atm.flist -->large file	61MB	669,690	all file that include ./tnnnn_ .shs files, gio ri files, atmspectra from plasma line processing does not include the hf files. Large file..clicking to display can take a long time. better to right click and download.
atmshs.flist	1.4MB	15133	echotek files. all files in NAS_arun.. ending in .shs (includes /hfxxx files)
atmgio.flist	90KB	1391	ri files written via gio. Mostly the ion line data. (leftout: substorm_21sep2005.nnn.. 14 files)
atmspc.flist -->large file	59MB	653,624	the .hdr,.dcd files holding the plasma line spectra. 127202 usrp spectra 526422 echotek spectra Large file..clicking to display can take a long time. better to right click and download.
		HF
hf.flist	8.8MB	101,662	All files with ./hf_yyyy this turns out to be all the hf files ending in .dcd or .hdr. It does not include the raw usrp hf files (sent after arunslist file created.

Notes:

the routine atm.sc was used to generate the file lists (/share/s3453g1/keysha/ARUN_FILES/phil/)

File list by arun's file prefix types:

Arun used a prefix subdir name for all data sent to tacc.
The table below lists these names with a link to a file containing a list of all file names in this group (be careful, some links point to very large files).

file prefix	# of files	Size in MBytes
AO327	85	0.004
AODATA	18733	0.960
AOPULS	658	0.038
AOUSRP	120138	5.499
CLPSPEC	283608	13.778
ECHOTEK	5162	0.232
GALFACTS	1847875	101.631
GALFAHI	59225	2.673
MOCKSPEC0	603894	29.100
MOCKSPEC1	1151902	55.712
P2722	560	0.036
P3218	95	0.004
PALFA	45552	2.683
PUPPI	7182	0.414
Planetary	2908759	192.271
S3043	927	0.043
S3453	54262	3.662
USRP	294410	19.373
USRPRAW	244583	12.072
VLBI	130	0.004
WAPPDATA	34492	1.578
ao	68189	6.099
arun	8279	0.486
clp	471668	20.752
p3184	509	0.023

probably want to split links above into subdir that have a list for each proj.

processing: /share/s3453g1/keysha/Arun_files/phil/doit.pro

Fits files.

The files in NAS_aruns_list_nz (non zero) were scanned to find the fits files.
My first attempt was to require files to end in .fits (this ended up skipping galfacube fits files that had been compressed).

There have been different types of fits files generated at ao:

The keywords can vary.
Some use the fits heap, others don't
The type of data stored can vary.

**Some types of fits files generated at AO**
Fits source	used for
mock spectral line fits files	regular spectral line observing with the mock spectrometer. the galfacts survey fits files have a small modification to this type. filenames are: projid.yyyymmdd.bnsmgp.nnnnn.fits yyyymmdd is the date the file was generated (opened) n=0..6 (which mock box m=0,1 (high,low mock bands) p =0,1 group 0,1 (if piggy back observing)
mock psrfits files	pulsar observing with the mock spectrometer these are psrfits files. the filenames are: projid.yyyymmdd.SSSSSSSS.bnsmgp.nnnnn.fits the name format is similar to the spectral line files with SSSSSS (the pulsar name) included in the file name.
puppi files	pulsar observing with puppi backend. puppi uses 8 gpu's. Each processes 100 MHz bandwidth. The initial output files from puppi are labeled xxx.raw Fits files with gpu in the path are the separate 100Mhz fits files there can be 8 duplicate basenames (each with a different gpu number in the path. eg: PUPPI/P3379/gpu01/puppi_58697_J1713+0747_0597_0001.fits PUPPI/P3379/gpu02/puppi_58697_J1713+0747_0597_0001.fits puppi files without gpu in the name may be the fits file after combination, or they could have been taken in the mode that doesn't use the gpu's (search) filenames: puppi_58099_B2303+30_1458_0003.fits 58099 is the mjd 1458 increments for each scan 0003 is a sequence number within a scan (if it takes more than 1 file puppi_58099_B2303+30_1457_cal_0001.fits Is an example of a calibration scan file (_cal_ gets added).
wapp spectral line	spectral line observing with the wapp backend. (I think the wapp pulsar observing is not a fits file).
galfa HI files	the galfa backend was used for galactic Hi surveys. It generated its own fits file.
galfa cubes	these are data cubes in fits file format generated offline. I noticed that these have normally been compressed (so they don't end in .fits.)
misc fits formats	radar processing generated fits files offline to store some of the generated maps.

I went through and tried to split the file list by fits type.

For each fits type a generated a subdirectory.
Within that subdirectory i generated an ascii file (by project) containing all the fits files (of this type) for the proj.
Before clicking on a project link make sure the files size is not too large (a2130_xxx are about 50MB each)
the file path includes everything that was include in the NAS_aruns_list

fitsType	# projects	Notes
galfa	24	each filename in a proj file has the file base name this does not include the galfa cube fits files
galfacube	1	fits cubes made from the galfa data 48413 fits files (mostly compressed (.fits.bz2) all sitting in one file. It contains just the base names.
puppi (no gpu in name)	19	each proj file contains the path and base name.
puppi (with gpu)	8	each proj file contains the path and base name.
wapp	183	each proj file contains the file basename.
mock	140	a2130_dfrq,a2130_dtm are galfacts files after decimation in freq and time. i found 784 duplicate files (basenames) in the mock list. They have not yet been removed. each proj file has the path and base name.
planetary	684	this is the number of sources.. sources includes thinks like:ehowel,irtfdata,nolan,tobemoved,... filenames have path and base name. some links are still present.

processing: /share/s3453g1/keysha/Arun_files/phil/dofits.pro

Files sent to tacc (arun's list)

Intro.

atm files

File list by arun's file prefix types:

Fits files.