plots:
az1 encoder failed and the pointing errors from using az2 encoder (.ps) (.pdf)
pointing errors before/after the encoder board failure(.ps) (.pdf)

The Failure.  (top)

    Debugging the azimuth 1 encoder failure:

• We saw  "az1 encoder failure" as a yellow (non fatal) fault on the ocu display. This error signals that encoder 1 has failed and the software has switched to the az2 encoder (on carriage house side) for pointing.
• On the ocu the az1 encoder reading would be correct for awhile and then jump by 100's of degrees.
• We replaced the az1 encoder with a spare. The problem persisted.
• We replaced the encoder interface board. The problem persisted. The interface board has 4 ports. Each port reads one of the 4 encoders:
• Port 1: az1 encoder. Used for pointing the azimuth. (top position of board).
• Port 2: az2 encoder. Not used for pointing. It's main purpose is to measure the azimuth arm bending.
• Port 3: gregorian encoder.
• Port 4: the carriage house encoder.
• We plugged az1 encoder into encoder interface port 2. Az1 encoder began to read ok (through port 2). So the az1 encoder was ok.
• We moved az2 encoder from port 2 to port 1. When it was plugged into port1, az2 encoder failed. So the problem was the encoder interface port 1.
• We had the same problem with port 1 on both encoder interface boards (it took us awhile to realize that both boards were bad).
• We looked at the clock and data lines  for encoder board port 1 on the breakout connector behind the lcu computer. The clock lines were fine (from interface board to encoder). There were no data line signals (encoder to interface board).
• When we looked at encoder interface port 2, the clock and data lines were fine. This was true for all encoders plugged into port 2.
• The encoder documentation (vertex 3rd party vendor documentation) stated that the encoder was short circuit and reverse polarity protected. The data lines disappearing meant that encoder interface  port 1 has a short on the data lines. This has to be at the board input or in the sn75175 differential line receiver chip. The failure must be on both encoder boards.
• The software switched to encoder port 2 for azimuth pointing on 14oct06 15:56:03. At 1:00 am 19oct06 we plugged az1 encoder into port 2 of the interface board. This got rid of the pointing errors caused by using the az2 encoder. The software was still using port 2 of the encoder board, but we now had the correct  "az1 encoder)" plugged into it.
• We did replace az1 encoder in the debugging process. This lost the az encoder position. We used the index at 270 degrees to realign the new encoder. After doing this we needed to check the azimuth pointing (since this alignment is hard to do down to the arc second level).
• We looked at the spare encoder interface board in the lab. The sn75175 differential receiver chip for port 1 had bad data line inputs. The resistance between vcc,ground to the data line inputs was not the same as the other ports.
• We ordered new chips and socketed the line receiver chip.
• On 26oct06 (around noon) we installed  the  spare encoder board (with the new line receiver chip). The system worked correctly. Az1 encoder was reinstalled in encoder board port 1. The encoder board that had been running also had a bad line receiver chip (which is being fixed).
• On 27oct06 we did a pointing check to verify that the pointing was ok.

What probably happened.

• Port 1 on both interface boards were bad.
• The cable to az1 encoder is always plugged into port 1.
• The lightning is probably coming down the az1 encoder cable and blowing out the receiver chip. The cable to the az2 encoder is a lot shorter (right next to the vertex shelter). The gr and ch encoder cables are real long so length may not be the problem. It may be that the shield for the az1 encoder cable is no longer well grounded at the vertex shelter.
• The spare encoder board was probably blown out and replaced some time in the past.  In hind sight, the 11nov05 encoder failures  was probably when the spare encoder board port 1 was damaged. My guess is that there was an encoder and an encoder board failure. When they installed a new encoder board, the problem did not go away. When they then installed a new encoder the problem was solved. This may have led them to think that only the encoder was bad.

What we should do:

• Check the shield on az1 encoder. Make sure it has a good ground to the vertex shelter.
• Have a preventative maintenance that looks at the data lines on all 4 encoder boards. We might see a degradation in time if smaller strikes are affecting the receiver chips.

---

Pointing errors caused by running on az2 encoder:  (top)

• The az1 encoder is normally used to point the azimuth. It is the encoder that is used when the pointing models are made.
• The difference between the az1 and az2 encoders is not a constant 180 degrees.
• As we drive in one direction or another, the azimuth arm bends causing the encoder difference to vary.
• The encoder rack gear is also not perfectly round (probably not even close to perfect .. see az encoder rack gear runout). The encoders are measuring arc length rather than angle. So if the rack gear is not round, the two encoder will run at a different rate causing their angular difference to vary.

• Page1:  Azimuth position on day of failure. This uses az1 encoder. The position started jumping around 16:00 AST. The bottom plot is  a blowup in time  when the failure occurred. The encoder read the correct value for a few seconds and then jumps away.
• Page2: The az1-az2 encoder difference for 01oct06 thru 14oct06.
• Top: az1-az2 encoder  versus azimuth in encoder units (.52 asecs per encoder unit). The two tracks in azimuth correspond the clockwise and counter clockwise motion. The difference comes from the bending of the azimuth arm.
• Middle: (az1-az2)*sin(za). This is the error on the sky. The units are arc seconds.
• Bottom:(az1-az2)*sin(za) versus za. This shows the sin(za)  dependence. At  za=20 the pointing error on the sky can differ from -60 to +10 arcseconds.

---

Pointing check after reinstalling encoder board.  (top)

    To verify the alignment, data was taken on 27oct06. These were the same sources taken on 14oct06 prior to the failure. The plots show the pointing errors before/after the encoder board failure(.ps) (.pdf):

• The black * are taken 14oct06 prior to the failure. The red * are taken 27oct06 after everything was replaced.
• The top two plots show the za error versus za for the two sources.
• The bottom two plots show the azimuth error (great circle ) vs za for the two sources.