• august 2018
• 2019-2020

Software

•  idl routines to process,plot the ips data
• ipscatlist: script to list a subset of the ips catalog for a given date, time range.
  
• It also evaluates the ips parameters for the center of the time range.
  
• try ipscatlist -h    for help

Intro: (top)

How the data is taken: (top)

• on, off position switch patterns are used   to acquire the data (180 second on src, 180 sec off src).
• This is normally done for a set of receivers,  then move to the next source.
• The mock spectrometers are used to acquire spectral density data (spectra).
• There are 7 mock spectrometer, each can record up to 160Mhz Bw in polA and polB simultaneously. This allows up to 7*160=1120Mhz  of total bandwidth.
  
• The mocks typically average the spectra for 2 milliseconds (for lband we switched to 1 millisecond so the radar ipps don't alias). This gies  90,000 (180,000) spectra for each on,off pattern.
• The raw spectra are dumped to disc in fits format.

The table shows typical configurations for various receivers.

Processing the data: (top)

    The spectral density data  allow us to remove rfi from frequency channels prior to computing the total power. The steps are

• For the explanation i'll use the lband setup:
• 7  80Mhz bands covered the lband receiver bandwidth
• 2048 freq channel spectra were dumped every millisecond.
• For 180 seconds there were 180,000 spectra of 2048 channels (for polA and polB) times 7 80 Mhz frequency bands.
  
• Input the onsrc and offsrc data. for each of the 80Mhz bands..  process 10 seconds of data at a time.
• compute the median spectra  for the 180 seconds of off spectra.
  
• to a robust harmonic fit (11 terms) to the median spectra to generate the bandpass correction.
• Divide both the on and the off spectra by this bandpass. This removes the filter shape and lets you use points closer to the edges
  
• For a 10 second junk, compute the rms by frequency channel over the entire band 80 Mhz band. 10000 spectra each of 2048 channels.
  
• do a robust linear fit to the rms vs freq and flag any freq channel greater than 3 sigma as  contaminated.
• split the 80Mhz band into 8 10Mhz subbands .. so  256Chan,10Mhz 8 subbands.
  
• compute the total power within each of these 10Mhz subbands..
• Only use the points that are not flagged as contaminated. Scale by the number of additions.
  
• The bandpass correction  ensures that all points have equal weight.
  
• It also scales the units to Tsys. (off source value).
  
• We end up with tpOn,tpOff of [180000,2,8] [points,pols,subbands]
• We end up with 18 10 second blocks of tpOn(180000,2,8) and tpOff(180000,2,8)
• For 327 we noticed very  strong, short  spikes in the total power (not resolved in the 2ms sampling of the 327 data).
• These may be coming from power lines.
  
• We try to remove these narrow spikes  with:
• for each tpOn or tpOff dataset  we subtract off a median filtered version (with filter length=3 ,,3*1ms for lbw, 3*2ms for other receivers).
• convolve a [-.5,1.,-.5] kernel to try and isolate the narrow total power spikes
• compute the rms of the convolved data then remove  any points above (4 sigma.).  this is only removing narrow spikes...
• For removal we interpolate across the channel using the adjacent points.
  
• After removing spikes set tpDif[180,000,2,8]= tpOn-median(tpoff) .. (use median so we don't degrade the statistics).
  
• The spectra were then computed in 10 second junks (.1 hz resolution).
• reform tpdDif to [10000,18,2,8] .. [10secsOfdata,18 10sec blocks,2pols,8 10 Mhz bw chunks)
• for each 10 secs of tpdif:
• process each 10Mhz band junk of the 180 sec dataset:
  
• compute the spectra as: spc[*,i,j,k]=(abs(fft[tpDif[*,i,j,k]]))^2
  
• keep 5000 of the 10000 channels (since it is a real xform)
• compute  a scaling factor by summing channels[1.. thru 55Hz].
  
• this skips dc and excludes 60hz and above
• this is a problem when we are in the strong scattering region since scintillating power can go above 55hz.
  
• Multiply the spectra by 1/scaling factor. This is the scintillating power
• multiply dc by .5 (to put dc and the no dc channels in the correct scale.. we could have left dc alone and multiplied all of the non-dc channels by 2 to compensate for negative freq channels that we through out .. (real xform).
• Averaging the spectra:
• the average and the median for the 18 10 sec spectra were computed (separately for each 10 mhz band, and pol) ending with:
• spcAvg[5000,2,8] and spcMedian(5000,2,8) for each 80Mhz section.
• The spectra were scaled to the area under the spectrum (0 to 55 hz).. this started on 15may19.
• When computing the modulation index:
• compute the median white noise value (median of spectra above 65hz)
• sum the values( spc[i] - whiteNoiseMed) (where i runs 1 channel above dc to 2 channels below 60hz)
• this will leave out lots of scintillating power if we are in the strong scattering regime.
  
• divide this by spc[0] .. dc. (which is the source deflection).
  
• The routines to process various datasets are in
• x101/projects/ips/    yyyymon 
  
• trying to remove lightning
  

Movies: (top)

• movies are made with the routine: x101/projects/ips/plotall_yymmdd.pro
• This needs to be copied to a subdir for the dataset you're working on
• change the dates yymmdd
  
• create the output directory in /share/megs2_u1/phil/ips/
• run the plotall_... file
• goto /share/megs2_u1/phil/ips/
• edit mkmov.sc to include the directory you just did,
• run mkmov.sc (a shell script)
• to create the .mov files, edit then run quicktime.sc
• finally move the files to /share/phildat/ips  by editing cpfiles.sc and mov/cpfiles.sc
• 
  
• the image order in the movie:
• day order, then
• source order (increasing ra)
• within each source, order by rf center freq.
  
• Each frame is a 180 sec on/off pattern for an 80Mhz band within a receiver
• The first 2 columns show the polA (left) and polB (center) data.
• Each color is  the spectra from a separate 10Mhz band.
• The frequencies are listed on the left side, followed by the modulation index computed from the data in the frame.
  
• The top row averages the 18 10 second spectra, the bottom row takes the median of the 18 10 second spectra.
• The rightmost column is the average (white trace) , and median (red trace) of all of the data.
• The bottom right frame plots the modulation index by frequency (color) for the different computation modes
• +,*  using the spectral average.. polA,B
• Diamond,Triangle: using the  spectral Median (polA,B)
• box,x : using the  robust mean/rms of the total power data.
• The movies are stored in /share/megs2_u1/phil/ips   .. the routine mkmov.sc will make the movies
  

processing:x101/projects/ips/pro/ ips_dorcvr.pro  plotrcvr.pro plotspc.pro  and  ips/ips_yymmdd_template.pro to do a daily set