The user starts the perl script pnet. Pnet starts prun and then psrv:

Starting datataking:

• The configuration files are read by pnet.
• The config info is sent to prun (pnet_setup)
• pnet_dump is called to start the datataking:
• The first available file sequence number is found by querying all psrvs (O cmd).
• Filename  with seq number is sent to all psrv's (P cmd). This opens the files.
• pnet makes the headers and then sends the 4K block to each psrv (Q cmd). The command includes the number of dumps that are to be taken. At this point psrv is in data acquistion mode (just waiting for data blocks from prun).
  
• dma is engine is started on prun (pnet_startdma sends L command to prun).
• pnet_sync() will wait for the next 1 PPS from the first spectrometer box (assuming not start immediate). This uses the H cmd.
• pnet sends the M cmd to prun to take data forever.
  
• While taking data:
• psrv will send back a status message every 1 second (w command). This is passed back thru prun.
• prun does a select on: read: psrv, fpga, and pnet. It does a select on write for psrv.
• pnet is waiting on any messages from prun.

Finishing datataking:

• psrv is counting the blocks of data it reads from prun. The number it needs to complete the run was sent during the writeheaders command.
• When psrv reads the final block of data, it sends a message back to pnet (x cmd) saying that it has completed the run. In the meantime psrv will continue reading from prun but the data will be discarded.
• pnet receives the x (done) message from each psrv. When the number of end messages equals the number of spectrometers running, pnet exits(0).
• pnet exitting causes the socket to prun to close.
• When prun sees the socket from pnet returning an error, it exits (after stopping the datataking).
• When psrv sees the socket from prun return an error, it exits.
  

    The datataking uses a large block of shared memory to buffer the input data. Two sets of fixed length buffers are allocated: large buffers of 20 Mb each and small buffers of 4kb each. Queues (linked lists in shared memory) are used to keep track of the free buffers, and various stages of processing. Programs request  a free buffer from the free list, fill/process the buffer, and then place the buffer on the next queue for processing. When done the with a buffer, it will be put back on the free list.
    Buffers can hold data (usually the large buffers) or commands for the program (the small buffers). The data processing is queued with a possibly large pipeline delay. Commands to a program go thru the same queueing system as the data buffers to insure that the commands are  executed in the correct order.
    The buffers system is centered around a Node. There is 1 node for each buffer in the system. The first part of each node contains pointers that can be used to link the node into a linked list. Nodes also contain some space to specify what they contain (data, header, command, etc..). The final elements are the pointers to the buffer for this node. The node to buffer mapping is setup by bufpoolD at boot time and does not change. The pointers/addressing are done using offsets from the start of shared memory rather than absolute addresses since different processes can map shared memory into their local address space at different locations. The routines make the addressing transparent to the user (when you want it, you get a ptr to the buffer).

     The ao programs will be multi threaded. One thread will wait on the input queue for that program. When a buffer becomes available, it will process it and then pass it on to the next step. Once a second this thread will wake up to see if there is anything else to do. A second thread will wait on communications from the outside world (probably a tcp socket?). When a request arrives (say from the gui)  it will process it and probably place it on the input queue for the first program. This guarantees that the commands are executed in the correct order.
    Some things that need to be worked out with the queueing system are:

1. When a command gets processed from a queue, there may need to be a reply going back somewhere. This requires some type of address in the queued command for the reply.
   

• bufpoolD: This is a daemon that gets started at boot time. It allocates the shared memory block, fills in the data structures in shared memory, and then sits around forever. It needs to be run by root to be able to allocate more than 32k of locked shared memory. Since it never exits, the shared memory will not be freed.
• psrv: A modified version of jeff mocks psrv. It requests a free buffer , fills it with data from the spectrometer, and then places this buffer on the bpInp list in shared memory for further processing.
• bpInp: Takes data off of the bpinp queue, bit flips the data, does any other type of processing, and then places the data on the bpout queue.
• bpOut: Takes buffers off of the bpout queue and then outputs them to wherever the user has requested. If fits files are being written it will probably merge the header info from the scramnet before writing here.

• bpfrlL: Buffer Pool Free List of Large buffers. These are 20 Mb each. The data gets put here.
  
• bpfrlSt: Buffer Pool Free List of Small buffers. Headers and command can be placed in these buffers.
• bpinp:   Buffer Pool INPut buffers. psrv fills buffers and places them here for further processing. The program bpInp will take the buffers off of this list and process them.
• bpout: Buffer Pool OUTput buffers. The program bpInp will place the processed buffers here. The program bpOut will take the buffers from here an output them.

Psrv

• Inputs data from prun.
• It reads data from prun via a socket. It places this data in buffers requested from shared memory. When the buffer is full, psrv places the buffer on the bpinplist in shared memory. It then continues to read from the input socket.
  
• I've tried to minimize the changes in jeffs routine. Most of the processing has been moved to the bpInp and bpOut programs.
  

bpInp 

• processes the data in the buffers
• It waits for buffers on the bpInp queue. Once a second it wakes up to check if it has been signaled to exit.
  
•  It will process the buffer and then place it on the bpOut queue so it can be written out.
•  The processing depends on the type of buffer:
• Cmd buffer: 
• If this is a comand buffer than it will do something with the command. For now it will pass filenames on to the output program.
• Hdr buffer:
  
• When a header buffer arrives, it is stored in a data structure for later use. There is room to store 2 separate headers (one from each 170 Mhz band). The bit flip index array is recomputed if the fftlen or the start/stop channels have changed.
• For now the header is also put on the output queue so it can be written out.
• Data Buffer:
  
• For each integration in the data buffer, the spectra are bit reversed to put them in the correct fft order. The  start/stop bins option means that this can not be done in place. At 65 mb/sec this take about 30% of 1 cpu.
  
• Any other processing that needed to be done could be inserted in this routine.
  
• The buffer is then placed on the output queue so it can be written out.
• The fits generating routines could be here or in the bpOut program. It may be more convenient to put them in the bpOut routine since they could then write each record to disc as it was processed. If it was done here, we'd have to merge the header and data in a shared memory buffer, pass it to bpOut and then write it to the header and stack locations of the data file.
  

bpOut:

• Take the processed buffers and output them somewhere.
• It waits for buffers on the bpOut queue. Once a second it will wake up to check if it should exit.
• Cmd buffers. These might be:
  
• Filename  and max file size. Store this for later use.
• enable/disable file formats. I can think of allowing  a .pdev file format and a .fits file format for now.
  
• enable/disable i/o destinations. This command can turn on/off recording to a particular device (disc, socket, etc..). I'm not sure whether this should be the same for all file formats? maybe you want to do online monitoring with .pdev file formats while disc i/o with fits format?
  
• any other commands that we can think of.
• hdr buffers:
• For now store the hdr buffer in case we need it for later processing
• If fits file processing is enabled we may want to build up the fits header info from the scramnet info when we get a new pdev hdr.
  
• If .pdev output enabled, output the header to all enabled output devices.
• Data buffers:
• If .pdev output is enabled, output to all enabled devices. Keep track of max file sizes so we can switch when we hit the max size.
• If .fits file output is enabled, grab get the field info for this rec and output it as well as the raw data.
• Some things to thing about:
• The data buffers can hold many integrations. For slow integration times there may be a large lag between when the record was taken and when we process it for output. This means that we will have to buffer the scramnet info and have a timestamp for each integration in the buffer (or at least 1 timestamp for the start of the buffer).
  
• For slow data taking it may be better to not use the entire 20 Mb buffer. Just put one integration (128Kb) in a 20mb buffer and pass it on. Then the delay from data taking to output won't be so long. At least there should be a command to limit how many integrations we put in a single buffer.