Intro

• for each fpga there is a 64 bit x 2^20 deep fifo that connects the real time data of the fpga with the dma controller that moves data into the main memory of the power pc. This is called the PI FIFO.
• The data in power pc main memory is then transferred to the fileserver via gigabit Ethernet.
  
• Two errors are common when the box can't keep up with the data rate:
• FPGAx:DMA buffer overflow.
• this might be that a dma wa requested to start but there is not memory available in the powerpc memory to receive the data?
• FPGA0: PI FIFO overflow
• This must occur when the SP processor tries to push more data into the PI fifo, but it is full (since the dma hasn't been emptying it fast enough.

• I think both of these errors can occur if the i/o from box to fileserver, or fileserver to disc can not keep up.

• The disc i/o on the fileserver slows down.
• There is a 1 gbyte disc queue in fileserver memory. If the disc slows down, a timeout will only occur  after this queue fills (usually about 16 seconds at 64mb/sec i/o rate).
• You can check the disc queue usage with the bpmon data. This samples the number of free buffers in the fileserver  buffer memory every 5 seconds and then writes the info to disc.
• If there is an Ethernet problem between the spectrometer and the file server
• Check this with ifconfig eth1 on the file server
• If the cpu becomes so busy that the pserv program on the file server does not get enough cpu cycles to read the Ethernet data from the spectrometer.

24jan13: b4sXg0 PI FIFO overflow (pdevs5)

  a2130 (galfacts) was taking data on 24jan13  when a PI FIFO  overflow occurred from b4sxg0 fpgas. :

• 04:37:10 From MOCK: infook pnetg0:b4s0g0: FPGA0: PI FIFO overflow
• 04:37:10 From MOCK: infook pnetg0:b4s1g0: FPGA1: PI FIFO overflow
• Both fpga's had the same problem.
  

Their setup was:

• alfa, group 0 in use.
  
• 65mbytes/sec i/o rate

Checking if it was a disc i/o problem.

• There are 50 20mbyte buffers in the freelist of the  fileserver buffer queue
• The plots show the number of buffers in the freelist.
• Page 1 shows midnight to 8am
• Page 2 is a blowup 4.61 to 4.63 (72 seconds)
  
• Each frame is a different pdev file server
• The error occurred on b4s[0/1]g0 (the fifth frame).. This is pdevs5
• The red dashed lines show where the error occurred.
• Looking at page 2:
• about 18 seconds before the error, there were about 48 free buffers
• The number of free buffers then starts to decrease linearly to 0 free buffers. Then the error occurs.

Conclusions:

• The data was getting from the spectrometer box to the fileserver ok
• The data rate from the file server 1 gb memory buffer to the disc started to slow down.
• Looks like there was a disc problem

07oct10 b0s1g0 dma overflow (pdevs1)

• alfa, group 0 in use.
  
• 65mbytes/sec i/o rate

• 18:19:45 mock started for freqsw measurements
  
• 18:25:08: finished freqsw dataa files (000-004).
• 18:31:42: start regular datatking (file 00100)
• 18:32:42 NOTE    from_mock: From MOCK: infook  pnetg0:b0s1g0: FPGA1: DMA buffer overflow
•                              from_mock: From MOCK: infook  pnetg0:b0s1g0 prun socket close
  

Checking if it was a disc i/o problem.

• There are 50 20mbyte buffers in the freelist of the  queue
• The plots show the number of buffers in the freelist between 18 and 19 hrs.
• Each frame is a different pdev file server
• The error occurred on b0s1g0 (the topmost frame).. This is pdevs1
• The red dashed lines show where the error occurred.
• There were still 47 20mbyte buffers available when the error occurred.
• The problem was not a disc i/o problem.

Checking for Ethernet problems:

    On 08oct11 ifconfig eth0,eth1 was done on each fileserver to see if there were any Ethernet errors:

• pdevs1 has 161746 dropped packets on eth0. This is the Ethernet that connects to aonet (not to the spectrometer)
• This is 80 times more than pdevs1 (who had more receive packets).
• The spectrometer to fileserver connection has 0 errors and 0 dropped packets.
• Looking at  /var/log/messages for pdevs1-7 there are a number of messages:
• Oct  7 16:27:40 pdevs1 avahi-daemon[3554]: Received response with invalid source port 47422 on interface 'eth0.0'
• This error is present in all of the spectrometers, and all of the messages.n files.. so it is not unique to this error.
• processing: svn/pdev/pdev/scripts/pdevifconfig

Summary:

• on 07oct12 at 18:32:42 there was an fgpa dma overflow from fpga b0s1g0.
• The disc queue freelist still had 47 free buffers to output, so it wasn't a disc problem
• eth0 on pdevs1 has 80 times more dropped packets than the other fileservers (with similar number of receive packets).
• The only other option would be if there was too much cpu activity for psrv to read the Ethernet data into the fileserver.
  

<- page up
home_~phil