aug10 spectral channel jumps (top)

• single pixel observations over wide bandwidths. Many different receivers used.
  
• Groups g0,g1 were used to increase spectral resolution
• Digital filter and digital mixer used with 8192 channels each spectra.
  
• the adjpwr parameter was set so 1sigma was 120 counts.
• For xband observations:
• decimate by 2: 172->86 Mhz bw each band
• offset bands by +/- 37 Mhz
  

• b0s1g1(pdev-119), b1s1g1(pdev-108), b5s1g1(pdev-117).
• The jumps were only seen in polA. polB was clean.
• The jumps were only in group 1, group 0 showed no jumps.

05aug10 test data on telescope. (top)

• All bands group0,group 1 were centered at the same frequency:
• rf center: 9500, if2 center : 9200, digital mixing center: 9237.46 Mhz
  
• This matched b1s1g1 of a2463 which often had the problem.
• an 86 mhz band (decimation=2) was used in the digital filter.
  
• about 30 1 second integrations were taken and written to disc

• Red is pola, green is polB
• 5 1 second spectra have been plotted (with offsets for display).
• Top: The full band. The dropouts in polA (red) are visisble. PolB (green) is clean.
• center: blowup 9215->9225
• bottom: for each dropout in polA, 2^24 (b23) was added to the dropout value then replotted. The spectra now look normal. This implies that the dropouts in 9215-9225 are 1 bit error in the spectra.
• During this scan only b1s1g1 had jumps. In other data taken, jumps were also seen in:
• b0s1g1,b5s1g1

Running the pdev diagnostics:  (top)

• 06aug10
• 09aug10

06aug10 results: (top)

• ltest - pncode from fpga checked on spectrometer. all spectrometers ok
• rtest - pncpde from fpga check on fileservers.
• all passed.
• most spectrometers ran at 35-40 Mb/sec (combined hi/low bands --> 70mb/sec)
• b5sXg1 could only run at 25 mb/sec before i/o failures.
  
• Since ltest ran ok at 40mb/sec, the fpga to spectrometer dma is ok.
• must be problem in spectrometer to fileserver ethernet (or maybe cpu is busy??)..
• rcomp- pncode with all spectrometers merged to pdevs1 and compared. ran same as rtest
• rdcomp- noise source on fpga piped through fpga spectral processor then on to pdevs where all spectrometer values are compared.
• 1 second, noise level=max 65535 gives mean spectral level on output of 2.6e8
• Using 172Mhz bw there were no failures on the full bandwidth ??
• 86Mhz bandwidth
• group 0 no failures
• group 1 failures
• tried changed hr_shift from 12 to 11 .. still got errors.
• tried rebooting all grp 1 spectrometers still had problems...

09aug10 test results:  (top)

summary of problems  (top)

Looking at data on the sky: 

• the problem only occurs in polA. This is the first fpga of a board.
• the problem does not occur in group0 only group 1. Mainly in b0s1g1,b1s1g1,b5s1g1
• the data on sky does not use bxs0gx bands.
• The problem has not  been seen when hi-res mode is turned off (full 172 Mhz)
• This is also seen with the test patterns.
• a2463 saw the problem with decimation=2 (86 Mhz). It did not see the problem with decimation=4 43 Mhz.
• The error has been seen as a dropout in some cases, and a jump in others.
• test pattern data mostly look like dropped bits.
  
• test data at a single frequency showed jumps in the same boards that a2463 saw jumps
• it's not some large frequency spike coming in thru the telescope.
• Looks like a bit error:
• many of the jumps have the same size.
• the 9215-9225 jumps were B24 (count from 0).
  
• Other jumps were a different bit.
• If the jump is in a lower order bit, it would probably be lost in the noise.
• The bit jumps are in the spectra.
  
• This --> that the error needs to be after the  fourier transform:
• A bit jump in the digital filter (time domain) would cause a sign wave in the spectra, not a spike.
• There are dropouts in B24 (16,000,000) --> the dropout is after the accumulation:
• individual spectra have average values close to 12000 so bit24 is always 0. There is no possibility of a bit flip from 1 to 0 in the individual spectra. A flip of the sign bit would not cause a single bit dropout.
  

Test pattern data

• The pncode test show no errors. This --> that the dma from fpga to spectrometer, spectrometer to file server is not causing the problem. Also spectrometer to fileserver is tcp/ip so it would catch any dropped bits in the xfr.
• I'm not sure what the amplitude levels for the pncode are? maybe B24 is always 0?
  
• the test pattern noise data never shows an error when full bandwidth is used (hi-res mode is off).
  
• The test pattern failures are normally a single bit difference in a byte. It can jump up or jump down.
• The test pattern noise source has trouble starting with decimation=4 (43 mhz).
  
• If only one box is used, then it always starts ok, if more than one box is used then it take many retries before it   runs for awhile. Looks like a phasing problem between master clock and decimated clock for different boxes.
• The bit switched tends to occcur in the upper most 2 bytes of the 4 byte integer.
• Let B0-B3 be the 4 bytes of the integer value with B0 the least significant and B3 the most significant byte.
• The error is ever seen in B0 and very rarely in B1. most of the jumps are B2 and B3.
• dropouts in B2,B3 would not be seen if the integrated value is less then 32768.
  

both

• The occurence of an error looks like it depends on how things start.
  
• All of the above starts have used the 1pps external start.
  
• If you get an error after say 20 seconds, you will continue to get errors spaced intermittantly
• If you don't  get an error after say 30 seconds, then it will run forever without an error.
• aborting then restarting a few times will eventually give an error.
  
• I wonder if this problem is related to the "starting problem" that galfacts has seen with the winking cal.
• occasionally the first winking cal duration will be different. Looks like it has started with the duration cal for the start different than all the other cal transitions. This causes the first cal transition to be bad. All the others are ok ,except for the phase shift at the beginning.
  

Resolution (top)

<- page up
home_~phil