17may17: Investigating the problem.

    Project p2030 reported problems with alfa beams 5 and 6 during observing runs in apr/may17. When processing the runs, the rfi excision was excluding most of the beam 5 and beam 6 data.

    The p2030 setup is:

• alfa  using both bands. one centered at 1450 MHz, the other centered at 1300 MHz (each 172 MHz bw).
• parallactic  angle tracking used.
• they typically look at a know pulsar (for 300 secs, no cal) at the start.
  
• They then do 268 sec scans tracking each search position. The hcorcal is fired for the first 5 seconds of each of these scans.
• Data is taken with the mock spectrometers:
• group 0 - psrfits files. 65 usec sampling 15270 spectra/row, 512  freq channels, polarization adding.
  
• group 1 - regular fits files, 1 second integrations for a2152 (commensal..).

Known problems with alfa beams:

• during the 07mar17  thermal cycling of the alfa dewar:
• beam1a which had been bad, started to work again
• beam5a which had been good, went bad.
• The lna's for alfa bm5a were turned off
• beam5a now has noise from the if/lo system
• We should split alfa 5b and inject it into alfa 5a and 5b (making sure we have no reflections).
• This would improve the current signal to noise by sqrt(2). This can be done downstairs in the if/lo output to the wapps/mocks.

Looking at the p2030 data.

Image of 1 second of data from 10may17 (.gif)

• beam 6, 1450 band. 512 frequency channels by 15270 spectra (sampled at  65usecs)
• The dark lines are dropouts in power. They span the entire frequency range (it was also seen in the 1300 MHz band).
• counting the dark lines, there are 12 in .2 secs .. or .01667 ms .. this is 60 Hz.

bm 6 total power plots vs time (10may17)

    10may17  1450 MHz band (.ps) (.pdf)
    10may17  1300 MHz band (.ps) (.pdf)

• This came from file: p2030.20170510.FRB1105238B.b6s0g0.00000.fits
• The total power was computed across the 172MHz band for each 65 usec sample
• The top frame plots 10 seconds of data.
• The jump at 6.3 secs is the hcorcal going off.
• The spectra was normalized to the cal off median.
• The bottom 2  frames plots .2 seconds around the cal transition.
• The dashed green lines are spaced by 16.667 millisecs (60 Hz)
  
• middle frame: the plot is normalized to the cal on data
• The red lines show a 1 db step
• The bottom frame normalizes to the cal off median.
• The red lines show a 1 db step.
• The dropout last for about 6 millisecs.
  
• the falling edge is slow. The return is within 1 sample (65 usecs).
• This looks like an attenuator stepping by 1db when it shouldn't.
  
• the only variable attenuator in the signal path is the 32 db attenuator that we change in 1db steps (i think it might have a resolution of .5db).
• actually i think there is also a 10db attenuator when switching between narrow band (100 mhz) and wide band (300 mhz).
  
• This is a pin diode attenuator, so it can probably change this fast.

total power plots vs time from all  7 beams.

    I looked at p2030 from apr17 thru may17 to see what has been going on.

• for each date I looked at the start of a single source
  
• The total power for the first .2 seconds of the source was plotted for all 7 beams.
• The data was normalized to the median value of the first second of data
• Dashed red lines were plotted showing a 1 db step.
• The first page is the 1450 band, the 2nd page is the 1300 MHz band (lots of radar rfi).
• 
  

• date	plots	notes
  14apr17	.ps  .pdf	clean
  17apr17	.ps  .pdf	clean
  18apr17	.ps  .pdf	60 Hz jumps bms 5,6 group 0. < .5 db
  19apr17	.ps  .pdf	60Hz jumps bms 5,6 group 0.
  20apr17	.ps  .pdf	60Hz jumps bms 3,4,5,6 of group 1 looks like pulsar search was taken with both groups rather than grp 0 65usec, grp 1 1 sec..this is an frb scan The jumps were not seen in bms 5,6 of group 0 (where they had been seen previously.
  23apr17	.ps  .pdf	60Hz jumps bms 5,6 group 0 .. about .5 db
  25apr17	.ps  .pdf	clean
  26apr17	.ps  .pdf	clean
  27apr17	.ps  .pdf	60Hz jumps bms 3,4,5,6 of group1 looks similar to the 20apr17 data.
  29apr17	.ps  .pdf	60 Hz jumps bms 5,6 group 0.  < .5 db
  03may17	.ps  .pdf	60 Hz jumps bms 5,6 group 0.   1db jumps
  10may17	.ps  .pdf	60 Hz jumps bms 5,6 group 0   1 db jumps
  16may17	.ps  .pdf	60Hz jumps bms 5,6 group 0.  .5 to 1db jumps

  
  

Taking data 16may17.

    I took some data on 16may17 (after the p2030 run).

• my setup was similar to p2030 (i used their config file).
• I did not see any 60 Hz jumps.
• i looked in the output data.
• We also looked at the output of the mock baseband mixers for beams 5,6..

What is going on:

• 60 Hz usually means a power supply problem.
  
• When first looking at the 10may17 data  with their 1db steps, i thought we were looking at an attenuator problem.
• Looking at the earlier data, the jump was not always 1 db...So maybe it isn't an attenuator...
  
• but it could be that the fall time was slow enough that we didn't get down to 1db before the 60Hz cycle caused it to jump back
• The  programmable attenuator is in the mock mixer chassis.
  
• for each pol of each beam
• there is an attenuator for the high and a separate attenuator for the low band
• Each attenuator is driven by 5 bits coming from a micro controller in the bbm chassis. (there is also a atten select bit).
• There are two power supplies that drive the mixer chassis.
• 1 for beams 1..2
• 2 for beams 3..6
• luis looked at the ripple from these power supplies.. (millivolt level).
  
• He found that one of the power supplies did have a dead fan.. so maybe things are heating up..
• The fan will be replaced..
  
• We need to continue taking data until  we can duplicate the problem.
  

170609: resolving the problem.

• On 7jun17 jason queried whether any progress had been made on the alfa bm5,6 problem.
• Later that day i looked at alfa's output with a square law detector/scope in the control room. This time i could see the 60 Hz dropouts. The problem was seen as the alfa signal entered the control room.. so the problem was probably upstairs.
• Felix and crew went up and first inspected the rf to 250 Mhz mixer chassis, but didn't find anything.
  
• After some more digging, it turned out that the problem was in the bias cards of alfa.
• a 15 volt supply that fed beams 5 and 6 was bad.
  
• The bad supply was repaired, and replaced on 09jun17

After the repairs, i took some data with alfa to check things.

The setup was;

• center the 2 172 Mhz bands of alfa at 1300 and 1450 Mhz.
• use the mock spectrometers to take data:
• 172 Mhz bw on each band, 65 usec sampling, 16 bit spectra, keep polA, B separate (p2030 adds the pols).
• take data for 60 seconds with the telescope stationary.

Processing the data:

• There were 918000 spectra each integrated for 65.3595 usecs.
• I only processed the 1450 Mhz band (the 1300 Mhz band has lots of radar...)
• for each beam and pol I computed the total power for each spectra, ending up with a total power time series of 918000.
• I normalized each time series to the median value (so tsys   =1.)
• y= abs(fft(totalPower))*2was computed and then plotted (*2 since i only kept 1/2 the points since a transform of real data).
  

the plots show the residual 60 hz in the data (.ps) (.pdf)

• The top frame is polA, to bottom frame is polB
• Each beam is plotted in a separate color. I added an amplitude offset to  each beam for display.
• The beams were also shifted by bmNumber*1hz  so that 60 hz from adjacent beams would not overlay each other.
  
• The amplitude of 60 Hz is about ,0005 to .0008 times Tsys..
  
• i checked this by processing a sine wave of unity amplitude through the same code. It came out with an amplitude of 1.
  

Summary:

• the 15 volt supply in the bias chassis feeding beams 5 and 6  had a 60 hz problem.
  
• It was replaced on 09jun17.
• after replacing it, the residual 60 hz through the mock spectrometer was about .0006 Tsys.
• Note:
• When first testing on 09jun16, i inserted the 100Mhz rf filters in alfa.. to help with the rfi.
• I then saw strong 60 Hz on all beams of alfa.....
• so either the 100Mhz filter bank upstairs, or the extra amps downstairs have a 60hz problem. Luckily no one uses the 100 Mhz filters anymore (they were initially installed when we only had the wappsfor data taking.. with their 100Mhz bandwidth limitation on alfa).

processing: x101/170609/alfa60hzfix.pro