Terminology

• scan: set of data taken without changing the configuration
• eg: on,off pattern: on scan,off scan,calOnScan,calOffscan
• Scan number:
• Used to identify a scan. format: ydddnnnnn
• y  -last digit of yr
  
• To resolve 10yr ambiguities..look at the julian date in the header.
  
• ddd daynumber of year
• nnnnn .sequence number that starts over each day..
• 
  
• Pattern: a set of scan used to make an observation:
• eg: position switched onoff
• mapping
• PatternId:
• unique number given to all scans in a pattern. It usually matches the scan number of the first scan in the pattern.
  

---

Intro.

    The interim correlator was used from around 1996 to 2020. There were 4 separate boards that could be sent 4 separate freq bands,. Each board accepted 2 pols (A,B). Sampling was either  3level or 9 level (depending on the requested configuration). For  9 level sampling you could have bandwidths of 25MHz/ (2^N) where N was 0..7.  3 level sampling added a 50MHz bw. Full Stokes measurements were also available.

    Data was recorded to disc files in a proprietary format (hdr followed by binary floating point data).
Data was grouped by scans where a scan was an integration period with a fixed setup .eg:

• on,off position switching followed by a cal on,off would have:
• on position : a scan
• off position : a scan
• calOn         : a scan
• cal  Off       : a scan

   Unlike more recent backends (eg mock spectrometer) multiple scans could be written to a single disc file.

    People could you different observing modes depending on what they wanted to do. Some examples:

• Position switching, on position ,off position, calOn, calOff..
  
• This was used to try and remove any standing waves in the spectra.
• Mapping: You would drive in ra, then step in dec to cover an area

Each of the observing modes is called a pattern. Newer data sets (mock data) had a pattern Id that would have the same header value for all of the scans within one pattern (or observation).

•  eg. position switching onScan,offScan,calOnscan,calOff scan would have different scan numbers but they'd have the same patternid.

To   process data, you need to use all of the scans within a pattern. For the interim correlator i've tried to generate patternId's after the fact by assuming a certain order of scans taken. This has not been done for all pattern types yet (and i can't promise that these pattern id's are 100% correct :).

---

Keeping track of the data taken

What was recorded:

When ever a datafile was closed by the interim correlator, the filename and size was written to a file:

• /share/olcor/archive.dat (ascii file)
  
• /share/olcor/archive.datmiss (ascii file) .. file that may now have made it into archive.dat

At the end of each month the datafiles written were scanned and meta data was stored in an idl save file for each scan taken.

• /share/megs/phil/x101/archive/
• f010101_010131.dat  ascii filelist for yymmdd
  
• sl/
• f010101_010131.sav  .  short {sl} form of the metadata
• slcor/
• f010101_010131.sav  . long form (slcor) of the meta data. Includes everything in {sl}
• This was done for 7jul2000 thru 8aug2020.

Creating structures that contain meta data for the entire period.

• interim cor data jul2000 ->aug2020
• 2773273 scans
• 19780 disc files.
  
• /share/megs/phil/x101/closeout/archive/intcor/
• combine all of the slcor save files jul2000 thru aug2020
• generates
  
•  SLARC  STRUCT    = -> CORSL Array[2773273]
• IDL> help,slarc,/st
  ** Structure CORSL, 28 tags, length=200, data length=189:
     SCAN            LONG         100100001
     BYTEPOS         LONG                 0
     FILEINDEX       LONG                 0
     STAT            BYTE         0
     RCVNUM          BYTE         5
     NUMFRQ          BYTE         4
     RECTYPE         BYTE        11
     NUMRECS         LONG                 5
     FREQ            FLOAT     Array[4]
     JULDAY          DOUBLE           2451910.8
     SRCNAME         STRING    '09448+1139'
     PROCNAME        STRING    'one'
     PROJID          STRING    'a1312'
     PATID           LONG         100100001
     SECSPERREC      FLOAT           60.0000
     CHANNELS        INT       Array[4]
     BWNUM           BYTE      Array[4]
     LAGCONFIG       BYTE      Array[4]
     LAG0            FLOAT     Array[2, 4]
     BLANKING        BYTE         0
     AZAVG           FLOAT           295.823
     ZAAVG           FLOAT           14.8701
     RAHRREQ         DOUBLE           9.7926034
     DECDREQ         DOUBLE           11.429160
     RADELTA         FLOAT           0.00000
     DECDELTA        FLOAT           0.00000
     AZERRASEC       FLOAT         0.0310815
     ZAERRASEC       FLOAT          0.38251
• One entry for each scan taken
• fileIndex is an index into slFileAr[] (see below
• SLFILEARC       STRUCT    = -> SLIND Array[19780]
  
• IDL> help,slfilearc,/st
  ** Structure SLIND, 6 tags, length=48, data length=48:
     PATH            STRING    '/proj/a1312/'
     FILE            STRING    'corfile.01jan01.a1312.1'
     SIZE            ULONG          1028608
     TAPENUM         ULONG                0
     I1              LONG                 0
     I2              LONG                11
• One entry for each disc file
• I1 is the index into slarC[] for the first scan of this file
• I2 is the index into slarC[] for the last scan of this file.
  
• These two structures are stored in the idl save file:
  
•  /share/megs/phil/x101/closeout/archive/intcor/slarc.sav (631MB)

    How to find data using the  structures (from idl)

• If you had the scan number scanNum
  
• ii=where(slarc.scan eq scanNum,cnt)
• Get the file for this scan
• indFile=slarc[ii].fileindex
• fileNm=slFileArc[indFile].path + slfileArc[indfile].file
• openr,lun,fileNm,/get_lun
• istat=corget(lun,b,brecs,scan=slarc[ii].scan)
• corplot,b
• Of course you'll have to probably change the path since the datafiles may be loaded in a different location.
  

---

Data by observing mode

    The table below lists the scans taken by observing pattern:

ind	pattern	# scan	#scans with PatId	# scans without patId	#projId's using pattern
0	calonoff	1802653	1590236	212417	348
1	x111auto	418865	418864	1	21
2	corcrossch	191012	191012	0	130
3	chzm3	77823	2	77821	26
4	onoff	76336	76336	0	186
5	chzm2	61343	0	61343	36
6	one	23638	23637	1	21
7	corcross	18345	0	18345	1
8	onoffbml	11316	11216	100	11
9	measocal5	10197	0	10197	1
10	frqswtest	9215	0	9215	3
11	cormap1	9046	0	9046	32
12	cordrift	6459	6459	0	18
13	on	4945	4944	1	43
14	doon	4818	4818	0	13
15	ondop	4420	4420	0	2
16	meascal5	4417	0	4417	1
17	measocal9	4368	0	4368	1
18	measocal7	4259	0	4259	1
19	hexmap2s	3762	0	3762	26
20	meascal9	2921	0	2921	1
21	lbnfbcal	2733	0	2733	1
22	cormapdec	2627	0	2627	13
23	measocal11	2440	0	2440	1
24	lbndrift	2070	0	2070	1
25	dps	1976	0	1976	14
26	meascal7	1577	0	1577	1
27	dooff	1304	1304	0	3
28	measocal8	1200	0	1200	1
29	lbwdrift	1032	1032	0	1
30	meascal11	931	0	931	1
31	sitting	550	0	550	1
32	meascal8	480	0	480	1
33	chcross	468	0	468	2
34	ephemmap	388	0	388	26
35	meascal6	384	0	384	1
36	lbnfbtest	369	0	369	1
37	meascal12	320	0	320	1
38	chzmfsw	278	0	278	1
39		268	0	268	3
40	cormapmer	265	3	262	2
41	327drift	237	0	237	1
42	cormapbm	188	0	188	2
43	sbcycle	153	0	153	11
44	meascal10	144	0	144	1
45	corstarotf	138	0	138	3
46	'testcals'	129	0	129	1
47	drift	96	0	96	1
48	calonoffbl	76	0	76	1
49	corcrosschL	66	0	66	1
50	pntmod	36	0	36	6
51	430drift	34	0	34	1
52	bmapazc	27	0	27	1
53	meascal3	20	0	20	1
54	azswing	19	0	19	1
55	track	14	0	14	1
56	longdrift	12	0	12	1
57	trking	9	0	9	1
58	tiltaz	9	0	9	1
59	mvtd	9	0	9	1
60	hexmap2	9	0	9	2
61	sit	8	0	8	1
62	test	6	0	6	1
63	ž—øŸ7ë™ßŸ	3	0	3	1
64	testing	3	0	3	1
65	rimapdw	3	0	3	1
66	430test	3	0	3	1
67	alfascan	2	0	2	1
68	ÿ7˟ë³^÷ÿ;	1	0	1	1
69	'on'	1	0	1	1

Notes:

•  calonoff: this is often included in a different patterns
• eg . onoff pattern includes a calonoff set
• patterns with very few scans were probably just used for testing.
  
• the number of project id's with each pattern may be inaccurate.
• The project id had to be entered by the observer (or operator). It was occasionally mistyped
• eg.. project x111 had project id's of x1 x11 x1111 111...
  

ToDo:

• If the actual disc files were available, we could do a more complete job of generating the patternId's
• the hdr.proc data helps deterimine which scans belong together (this info was unfortunately not included in the .sav file meta data).
• Need to map idl processing routines to observing modes.
  
• Need to repeat this for the wapp and mock data.
  

processing: x101/closeout/archive/intcor/archgetcor.pro , genpatid.pro.