Timeouts from disc i/o slowdown


The problem

    12meter data taking with the mock spectrometers  was occasionally having an error and causing the datataking to stop.

On 29mar23 at 13:23:56  one of these error occurred.

The mock datataking programs consist of:

Logging info around the error

The tcl log file had:

2023-03-29_13:22:48  13 308800075  offsets:  -400.691     0.000 start:132252
2023-03-29_13:22:49 Mock scan starting filenumber:07300
2023-03-29_13:22:52 Observation begins Wed Mar 29 13:22:52 2023. scan:308800075
2023-03-29_13:23:53 Scan 308800075 completed.
2023-03-29_13:23:56 mockpset:error mockpset grp g1 (0). stat:er retVal:pseterr pdevs2.pdev.naic.edu psetReply:.. retrying
2023-03-29_13:23:56 mockpset:error mockpset grp g1 (0). stat:to retVal:.. failed retry..quitting

message log of pnetctl

The message log on the mock spectrometer cpu

Plotting the buffer pools status around the error.

    I had started the bufpool monitoring routine around 13:00 on 29mar23 to see what was happening with the buffers.

The plots show the number of buffers in the various queues around the time of the failure (.ps) (.pdf)


processing:x101/230329/pseterr.info, x101/pdev/bpmon/bpmonplot_230329.pro

<- page up