Intro

    We have had problems with the turret timing out waiting to be on position. The turret positioning sequences is:

• it sends the turret to the position and then calls turwaitpos()  to wait for the turret to be in position. 
• The routine is called from within setupall().proc
• see /home/online/Tcl/Proc/pnt/turwaitpos.proc
  
• By default this routine declares the turret to be on position when it is within .1 turret degrees of the requested position.
• The turret plate scale is 45 asecs great circles on the sky for each turret degree.
• turwaitpos() times out in 200 seconds..
  
• I'm not sure if cima times out before this.

If the turret times out, cima log file has:
2017-Feb-19 03:16:12 LOG4    got_cormsg: From DATATAKING: waiting for turret position:26.64
2017-Feb-19 03:16:53 ERROR   end_wait_seconds: ERROR timed out while waiting for configuring IF/LO path!
2017-Feb-19 03:16:53 LOG1    task_failed: Task 'waiting on configuring IF/LO path' failed!

When timeouts occur, they have been traced to the zero velocity offset of the turret dac, It drifts over time until it is outside the allowable region. This drift can take  days.. So we need to monitor this value daily.. and update the dac before it gets too far out..

12jan16: turret not arriving on position.

    This is a similar to problems we had back in aug14. At that time i thought the problem  was from the backlash board being misadjusted.. but  it turned out to be a problem with the zero velocity offset of the dac that drives the motor amplifier (in velocity mode).

    To find out when this problem started i went back through the turret log files and plotted the turret position and the different (actual - requested turret position):

• The plots show:
• top frame turret position vs hour of day.
• 2nd frame: Actual - Requested turret position vs hour of day.
• The red lines show the limit of +/- .1 degrees when cima thinks the turret is on position.
• 3rd,bottom frames: same info as 1,2.. except a different day.
• The errors should be small when the turret is sitting at one position (while it is moving, the error will be large).
  
• plots of turret position and position offset for 1st day of each month for 2015 (.ps) (.pdf)
• each day has info from the first day of two months.
• You can see that the errors were small on 01Nov15 and then large on 01dec15.
  
• The daily plots for Nov15 show when the errors increased.
• plots of turret position and position offset for each day of Nov15 (.pdf)
• each page has the position and position error for two days.
• the turret was not moving 151109 to 151115
• this was when we had problems with the main contactor being driven incorrectly by some digital outputs.
• when the turret started moving again on 151116, the turret error had increased to up to .05 degrees.
• They probably put a new dac in the turret during the 09-15th testing.
• The error continued to degrade through Nov and dec15.
• It's interesting to note that when moving to a new turret position
• the turret would move at 3deg/sec and then arrive at a position (that would be off by some error)
• It would then creep toward the correct position over a few hours.
• The PID loop integrator must be working very slowly...
  
• On 12jan16 the error had gotten so large that it no longer arrived at the .1 degree threshold of turwaitpos().
• I used the procedure (found here)   to update the zero velocity offset for the dac.
  
• Once this was done, the turret came rapidly to the requested position .. to within .005 degrees.

Summary:

• If the turret doesn't come within .1 degrees of the requested position, you need to update the zero velocity offset (see procedure)
• The PID loop does a very slow (hours) approach to the correct position when there is a zero velocity error in the dac.
• This is probably a bug in the code...

processing: x101/160112/turerryr.pro, turerryrmon.pro

170219 turret not arriving on position.

    Timeline of the problem:

• 18Feb17
• 23:30 a power dip occurred. The system was reset and the observation continued.
• 19Feb17:
• 01:38  observation finished. Tsysall run and completes successfully,.
• 01:40  next observer tries to run. Has trouble when turret does not get within .1 degree of the requested position.
• 09:00  luis and phil change the zero velocity offset from 57 to 80 counts, and the turret now reaches the correct position.

The plots shows the turret position and the (Actual - requested) position for Feb 1 to 19 2017. (.ps) (.pdf)

• Each page has 2 days of data.
• upper frame: turret position
• 2nd frame (actual - requested) turret position (in deg),.
• The red horizontal lines are the +/- .1 deg turret on position limits.
• We first thought that the power dip caused a problem with the turret. Looking at the data for the month of Feb17, you can see that the  error was slowly increasing for about 11 days.
  
• things were ok until 08Feb17 when the actual -req error started to increase
• on 18Feb17 it was getting close to -.1 deg.

We should monitor this error. When it starts to increase, the zero velocity offset should be recomputed.

processing: x101/170219/turerryrmon.pro

180529: may18 data shows zeroVelErr from dac/signalCond board

    We've had a problem with the turret not arriving at the final requested position (for the last few years). Debugging showed that this occurred when the zero velocity offset for the dac had drifted. This has been occurring more regularly. During 2018 the zero velocity offset had to be updated on:

• 09Feb18 zeroVelcnt: 2038
• 09Apr18 zeroVelCnt: 2094
  
• 04may18 zeroVelCnt:2143
  
• 16may18 zeroVelCnt:2000
  

    The turret data for 01may18 - 29may18 was used to look into this problem.

What happens when commanding the turret to move:

• the PI loop generates a digital commanded velocity to use.
•  This is a number from 0 to 4095 with   0 turret velocity around 4095/2
• the zero velocity offset is added to this value and becomes the Vel_command sent to the dac board
  
• the zero velocity offset is measured by commanding the turret to a position, and seeing what commanded value is needed to hold it stationary (once it gets in position).
• This is needed when we cross one of the turret encoder pre-limits. In this case, the zero velocity value is sent to the dac (rather than using the PI loop)
• The vel_command is sent to the dac.
• commanded values 0..4095 are mapped to 0 to 10 volts dac output
  
• The dac output is sent to the signal conditioning board.
• This offsets the signal by -5 volts and then multiplies it by 2 (the amplifier wants +/- 10 volts)
• the signal conditioning output is sent to the torque bias board and then the motor amplifiers.
• there are actually two motors (used for backlash compensation).
  
• between the signal conditioning board and the torque bias board, the voltage is measured by the a/d converter. This is called the velocity feedback signal.
• The velocity of the turret floor is measured in two ways:
• Master amp speed readback
• This is a voltage output from the motor amplifier that is sent to the a/d. It is "proportional"  to the motor speed.
• Encoder position. An absolute encoder is read once a second to get the floor position
• Floor velocity, and acceleration can be computed by differencing the position values.

Plotting the may18 turret data

(Warning.. the .pdf files are 1-2MBytes. they may take 10-20 secs to load.)

Velocity command vs velocity feed back, Master amp speed  vs velocity feedback (.pdf)

• Top frame: velocity command vs  velocity feedback.
• The vel_command is sent to the dac board
• the velocity feedback is the output of the signal conditioning board (input to the torque bias board),.
• The velCmd,velFdback  
  
• changing the zero velocity offset would cause a jump in the velocity command.
• Bottom frame:
• Master amp speed vs velocity feedback.
• This remains a linear  function for the entire month.
• So the jumps are not caused by the amplifier, torque bias board, or motors.
• So changes in the vel_command vs motor speed are occurring in the dac or signal conditioning board.

Velocity command for 0 vel vs date (.pdf)

• Top frame: vel_command (for zero velocity) vs date
• data points with velocity feedback within 2 counts of zero velocity were plotted vs date.
• Black: before 04may18 zeroVelocity change
• Red: between 04may18 and 16may18 zeroVelOffset changes
• green: after 16may18 zeroVelOffset change
• The hour for the changes were not recorded, so i guesstimated the time.
• bottom:
• vel_command vs velocity feedback.
• The different colors show that the different relationships go with the 3 sections of time.

•  the 24 volt power supply value is plotted vs date.
• This is sampled by the a/d converter
• The red points are when the motor is energized.
• The voltage drops about .3 volts when the load is applied.
• You can also see a slight day,night variation of voltage (probably from temperature changes).
• There is no systematic power supply voltage change that could have caused the jumps.
  

• the encoder position was used to compute the encVelocity and encAcceleration
• encVel(t+.5) =( encPos[i+1] - encPos[i])/dt[i] 
  
• then encVel was then interpolated from (t+.5) back to (t)
• The encAccel was computed the same way (but using encVel)
• top: cmd_velocity vs encoder velocity
• there are multiple patterns.
• 2nd: velocity feedback vs encoder velocity
• there is a single pattern here.
• the figure 8 is probably from  acceleration vs de-acceleration
  
• The smaller pattern is limited by the maxVel of 3deg/sec we normally run at
• the larger pattern is when the sband transmitter is used.
  
• Looks like they are upping the maxTurVel to 12deg/sec.
• bottom:  encoder acceleration vs encoder velocity
• these are both computed from the encoder position
• you probably need to flip the bottom plot to compare it with the 2nd plot .
  
• the constant velocity feedback (around 200 counts) probably occurs when de-accelerating.

SUMMARY:

• The turret will occasionally not arrive at the final requested position.
• This is caused by the zero velocity changing.
  
• There is a jump in the command velocity vs output to amp voltage .
  
• The changes are occurring in the dac or signal conditioning board.
• We should probably change the dac and signal conditioning board.
• It would also be  a good idea to write a diagnostic program to see the dac output vs input (this should be done on the bench).
  

23jul20 turret does not arrive on position during sbtx test

    On the afternoon of 23jul20 the sband transmitter was tested. They moved between the tx and rx position  at the requested velocity of 12deg/sec
During the test, the turret was not arriving on position in time for the receive or  transmit cycle.

The first set of plots show the turret operation during this test (.pdf)

• Page 1: position and velocity
  
• top: turret actual and requested position.
• black: current position
• red: last requested position (may be 1 sec  behind the actual position)
• You can see an offset between the 2.
• 2nd: turret velocity
• computed from the turret encoder
• The spikes are when the turret moved from sbtx to sbrcv positions
• The moves are requested to be 12 deg/sec.
• We measure about 10 deg/sec.
  
• part of the difference may be the sampling rate. We sample at once a second and it takes 4-5 secs to move..So it doesn't sit at the maximum velocity for very long.
• So the turret is probably moving at the correct velocity.
• 3rd: position error
• Last requested position vs current position
• The turret overshoots the rx to tx move and then slowly moves closer to where it is supposed to be (over 30 minutes or more).
• This slow recovery to the actual position has been seen before when the zero velocity offset for the dac's had changed.
• bottom: velocity command sent to dac board
• this sits around 1999 counts. The current zero velocity offset setting was 2000 cnts, so the value is close to what it is supposed to be.
• Page 2: motor current and speeds
• top: turret position
• middle: master and slave motor currents. these are proportional to the applied torques.
• When sitting still , the master torque is moving about a bit, while  slave torque is 0. I think this shows the absolute value of the torque.
• Bottom: Motor amp speed
• this is read from the amplifiers. black:master, red: slave
• There is a -12 count offset in the readback from both motors.
  
• This may contribute to the problem.
• Page 3: how often did this problem occur during 23jul20
• top: histogram of turret position (by second) for the 86400 seconds of 23jul20
• the vertical scale is a log scale.
• the spikes are the locations of various receivers.
• bottom: Position error when turret velocity < 2 cnts/sec.. (
• The  sbtx position did not arrive at position the most (although it also sat at this position the most time.)
• the sbrx position also showed counts when it didn't arrive at position.
• this shows that the problem is not rfi from the sbtx getting into the turret (since the sbtx is off when it's at the rcv position).
• The other receivers only show a few counts when there was a position error.
• I probably should have normalized this to the fraction of time we sat at each receiver.

The 2nd set of plots shows the turret motion during the entire day (.pdf)

•  I've limited the points to when the  abs( turret encoder velocity) was less than .003 deg/sec (pretty much stationary within 1 encoder count).
• top: turret torque counts when encvel < .003deg/sec
• black is the master, red is the slave board.
• 2nd: amp motor speed readback when encVel < .003deg/sec
• the -13 count offset was constant for the entire day.
• 3rd: the position error when encVel < .003deg/sec
• the large value at 15.5 was the sband test.
• bottom: blowup of position error when encvel < .003 deg/sec
• the overshoot and slow return looks to have happened at a lower level during the day,

SUMMARY:

• during sband tx test the turret came to within about .75 degrees of the required position, and then took about 30 minutes to slowly get to where it was supposed to be.
• This has typically signaled a problem with the zero velocity offset of the dacs
• But the zero velocity of the dacs was close to what was later used to hold the turret stationary.
• There was a 10 count offset in the amplifier motor speed readback.. this may have contributed to the problem.
• looking at the entire day, there were other periods where the turret overshot and then took awhile to return to the correct position (although the overshoot was much smaller)

processing: x101/200723/turspd.pro