• height vs frequency images
• Average ipp over time and height
• Signal to noise with rfi in and outside of the plasma line frequency range
• Mike's plots
  

•  clp coding spreads out the ion and plasma line. so removing  spikes in the spectral domain does not affect them
• When using uncoded pulses (or tpsd data), the ion line (and maybe a strong plasma line) will be clipped by the rfi removal (since they stand out in the frequency domain).
  

     rfi removal in clp data  can take advantage of the decoding step when trying to remove frequency dependent rfi.
clp processing for each 10 second block of data (1000 ipps) consists of:

• For each ipp:
  
• code the transmitted signal with a 2 usec pncode (2^39 code length). The 2 usec baud defines the height resolution.
  
• sample the 440 usec coded transmitter pulse at .2 usecs (2200 samples). Use this for decoding the returned echo.
  
• 440 usecs is pushing the duty cycle of the pulsed transmitter.
  
• sample the height data at .2 usecs (5 MHz bandwidth). keep 22000 samples for each ipp
• for each of the 3961 heights (spaced by 1 usec):
• multiply  2200 height samples by the conjugate of the code, then 0 extend to 4k.
  
• do a 4k xform
  
• compute power
• accumulate this spectrum in the corresponding height of the average power buffer (4k,nhghts)
• continue this average the 3961 height spectra for the 1000 ipps (10 secs).
  

• multiplying by the code in the time domain is the same as convolving the voltage spectrum by the transform of the code.
• This will spread narrow band rfi to 1/baudlen or .5 MHz. It will also alternate the phase of these spread out samples
• this will decrease the strength of the rfi.
• very strong rfi will still appear in the spectrum spread out over .5 MHz

How the rfi is removed: (top)

• transform the 22000 height samples to the frequency domain spcV.
• compute the power (spcV*conj(spcV))
• find rfi in this power spectrum
• find the rfi outliers
• replace these outliers in the spcV  (voltage spectrum)
• Replace the values with 0's or maybe with noise
• The notch that is put in the spcV by the 0's (or noise)  will get convolved by the transform of the code (during the decoding step).
• This will smear out this discontinuity in the final averaged spectrum.
  
• transform back to the time domain.
• proceed with regular clp decoding.

Looking at the results:  (top)

    Two 5 MHz bands were accumulated for 10 seconds centered at 435.5 and 439.5 MHz. The 10sec block chosen has:

• rfi in the lower block that sits on top of the plasma line
• rfi in the lower block sits above the plasma line

Height vs frequency images:

Average spectra over all heights and 10 seconds. (top)

• Page 1 is channel one (centered at 435.5 MHz). Page 2 is channel two centered at 439.5 MHz.
• Top:  spectrum of undecoded data.
• The spectra of the 1000 ipps were averaged
• The rfi is narrow since the decoding has note been applied yet
• The vertical scale is in units of Tsys
• Middle: blowup overplotting raw and decoded spectra.
• black. Average of undecoded spectra (same as top plot)
• red: average spectra of decoded data
• The decoding has spread the rfi in frequency
• You can see sidelobes of the code. These  show up in the images as vertical stripes (they wrap around on page 2).
  
• Not sure why the decoded rfi is not centered at the undecoded frequency.
• Green. average of decoded spectra. Replace outliers with 0's before decoding.
• blue: average of decoded spectra. Replace outliers with noise before decoding
• Bottom: Blowup of average spectra with rfi outliers removed
• green: remove outliers with zeros
• blue: remove outliers with noise
• The two plots track each other pretty well.
• On page two the dip around the rfi is deeper for the spectra with zero's removed.

Signal to noise with rfi in and outside of the plasma line freq range (top)

    The below plots show a 10 second average spectrum at a particular height that includes the plasma line. Some have rfi outside of the plasma line, other plots have rfi on top of the plasma line. The data was taken from   t1193_03feb2014_015_001

plots comparing the 10 second average spectrum at a given height  (.ps) (.pdf)

• Page 1: band 2 (439.5MHz) Plasma line cut off: height: 282.2 km, freq=441.12
• Top: 440.5 - 441.2 MHz
• black: no rfi removal
• The rfi starts around 441.4 MHz.
  
• red: replace rfi with zeros
• green: replace rfi  with noise.
• The solid line is the same data filtered by a 71 channel median filter
• The plasma line is at  441.2 MHz
• Middle: normalize the bandpass  by the median filtered data.
• overplot the 3 spectra
•  green overlays the red... There is no difference between the rfi zero fill, and rfi with noise fill.
• bottom: blowup in frequency, scale   to sigmas.
• 440.5-441.0 was used to compute the rms. This was then used to scale the plot.
• spectra with and without rfi removal are the same.
• Page 2: band 1 (435.5MHz) rfi in plasma line. Height: 558.5 Km, 432.28MHz. upshifted line
• top: data scaled to Tsys
  
• black: no rfi removal
• The large bump is the rfi at 434.18 spread by the code.
  
• red,green:( remove with zeros, noise)
• The solid lines are the data filtered by a 71 channel long median filter
• the dotted red line is the location of the plasma line
• .bottom: data scaled to sigmas.
• the amplitude of the line is larger in the unfiltered data, but it has many more noise outliers,.
• Page 3: band 1 (435.5 MHz).downshifted plasma line at hght:192.3Km, 432.28MHz
  
• Top: the 3 spectra scaled to Tsys
• bottom: the 3 spectra scaled to sigmas.
• In this case, the filtered data is larger than the unfiltered data.
• Page 4: Mean,rms values of the undecoded spectra for each ipp.
• black is chan 1 (435.5), red is chan 2(439.5 MHz)
• Top: the mean value of the spectra for each ipp
• 
  
• 2nd: the rms value of the spectra for each of the 900 ipps.
• rfi in channel 1 jumped up around ipp 730
• rfi in channel 2 jumped down around ipp 850
• 3rd: rms/mean for spectra of each ipp.
• Only 1 spectra has been accumulated, so rms/mean should equal 1 (the actual value is 1.2 .. this may be from the channel filter width.. a little larger than bandwidth/nchannels).
• ipps with rfi have rms/mean > 1/
• 4th % of points discarded in an ipp because of rfi
• This used a 4 sigma clipping level to identify rfi
• black chan1:
• .7% of the points were discarded until ipp 730 when it dropped down to close to .1%
• When the rfi appeared, the sigma increased, so points > 4sigma were fewer.
• red: chan2:
• .3% of the points were excluded until ipp 
• bottom: how many times iterated the rms.
• If the rms decreased by 2 after throwing out all points > 4sigma, then the rms was recomputed using those points < 4sigma.
• This occurred for most ipps with strong rfi.

Mike's plots (top)

• compare plasma line freq vs time:
• horizontal is time, vertical is altitude, color is plasma frequency,.
  
• without rfi excision (.pdf)
• with rfi excision (.pdf)
  

Summary:  (top)

• replacing points in the ipp spectrum with values > 4sigma by zeros or noise reduces the decoded plasma line images.
• Replacing the points with zeros, or noise showed no difference in the results.
• .2% to .7% of the points were modified in each ipp.
• the lower value .2% occurs with strong rfi.
• This is not getting rid of all of the rfi in the ipp.
• for Gaussian noise, 4 sigma should have only 1 or 2 points in 22000 samples.
• We find 44 to 176 points. part of this may be the bandpass correction .. although i think the noise is probably not Gaussian...
  
• Decoded rfi is not centered at the rfi freq (off by a .1-.2 MHz). Mike thinks this could be from the transmitter pulses used to decode the data.
• I'll implement this in the multi threaded code, with an option so specify the sigmas to use as replacement.
  

 
up to aeron
home_~phil