Crossing midnight at the end of the year.

 Background:

/home/online/vw/etc/initDat/year

    This file is read by the pointing program and the irigServ program when they are started. Both of these programs are running on the vme pointing crate.

---

Procedure for crossing midnight:

The instructions for crossing midnight  at the start of a new year are:

Note: i changed the instructions  on 01jul15

• . Instead of rebooting the pointing crate, we just stop the pointing program and restart it.. this  gets around a race condition when the pnt clock is being syncronized and the pointing program is asking for the utcToUt1 info..
  
•  When is midnight:
• Leap seconds are transmitted 0 hr 01jan UTC. This is 20:00 hours at arecibo:AST.
• The clocks that we run at the AO (TRAK, old eeco, etc) are running on AST time. They will perform operations (addition of leap second, leap year) when they read 0 hrs (in our case AST ).
  
• The UTC to UT1 conversion (/home/online/vw/etc/Pnt/utcToUt1.dat) uses a linear approximation to the drift. The file contains many sets of timestamped polynomials that get updated when the linear approximation is large or a leap second is added. Pointing searches for the most recent set of coefficients (the one whose start time has passed and is closest to the current time). The timestamp in the file is mjd but the comparison uses AST time so that leap second additions will occur at 0hrs AST (to be in sync with our clocks).
• So midnight in this document refers to 0hrs AST.
• There is a  potential problem when leap seconds are added. Some of the computers in the datataking are using ntp (network time protocal). This  will insert the leap second at  0hrs UTC (20hrs ast).
  
• between 20hrs and 24:00 on the day of a leap second, the start times in the headers will be without the leap second. Pulsar observers need to take this into account.
• The pointing should be ok since it runs off of our irig that switches with our clocks.
  
• Make sure that the file /home/online/vw/etc/initDat/year  has the new year in the first line of the file (eg. 2006).
• You can update this file before midnight since the pointing program only reads it when it starts.
• I wouldn't worry about the pointing crate rebooting prematurely  from a power dip since it is on a UPS.
• Shut off the tiedowns using the tiedown disable switch.
• When you reboot the pnt crate (below), the tiedowns will lose communications with the pnt crate.  This causes the tiedowns to release their tensions. When you disable the tiedowns with the switch, they do not release (since you've cut the power).
• Stop any data taking that is running.
• If the gui is running, it's probably best to exit it.
• You don't have to exit the other vxWorks (vw%) windows.
• pre jul15 verion.. (no longer used).
• At the  operators terminal in the window with the vw% prompt enter: exit
• This window is where the operators enter the pointing commands. It is running on the pnt crate that we are going to reboot below.
• At the laser ranging pc (on the wall to the right of the observer2 workstation) update the dos time using the date command.
• New instructions (after jun2015)
• from vx% prompt
• rsh pnt pntMProgStop
• loop
• rsh pnt i
• continue doing this till  the programs pntMProg  pntXform  are no longer in the list
• rsh pnt pntMProgStart
• pre jul15 instructions.. (no longer used) this has a race condition .. see things to think about below..
  
• Go to the pointing crate in the clock room and push the red reset button. This will reboot the pointing (pnt) crate.
• It takes about 1 minute for the crate to reboot.
• Wait for the pointing crate reboot to complete.
• You can monitor this by looking at the error message in the lower part of the telescope display (next to the operators and observers windows). While the crate is rebooting, there will be an error message saying:  agcProg Error or scramNet error. When this error goes away, the reboot is complete.
• Re-enable the tiedowns by turning the tiedown disable switch back to enabled.
• Restart the operators vxWorks window.
• At the operators terminal (in the same window you typed exit in above) enter: pointing  This will restart the operators vw% program.
• In the operators vw%  window enter:
• pnt td reset
• pnt td time 0
• pnt tur time 0
• pnt reset 15
• If you want the tiedowns to track the platform height then enter: tietrk
• Make sure that the other vxWorks windows are connected to the pointing crate after the reboot.
• In the receiver room on the cpu by the oscilloscopes enter connectall  in the window with the vw% prompt.
• If the correlator vw% window is also in use, enter connectall in its window.
• If you want to restart the gui, go ahead.

---

Things to think about:

•  It is possible to go through this sequence without rebooting the pointing crate but you would have to also worry about some other things. The tiedowns and turret have their own little star cpus. These computers receive time stamped requests to move to different positions. They keep track of time by counting 1 second ticks. When they start (and when the pnt xx time 0 command is sent), they resync their time to that of the pointing program. This is not really necessary during a midnight crossing unless a leap second has been inserted. In that case you would have to:
• Have the pnt program and irigServ program reread the year file.
• After the one second tick has been inserted, execute the pnt td time 0 , pnt tur 0 commands to update the second count.
• If programs are taking data while this is occurring then you need to worry about race conditions where datataking does something in the middle of the above sequence.
• It's simpler just to reboot the pnt crate.
• You don't have to reboot the cor or da vme crates. Each crate is counting 1 second ticks. Every 10 seconds they talk to the irigServ program and ask the time for the current tick. When a leap second occurs, all of the crates will jump to the current time (within 10 seconds after midnight).
• 01jul15: found race condition in pntProg startup and synchronizing with irig clock.
  
• i rebooted the pointing crate 30 seconds after midnite. We had the new time, but the point program did not get the latest utcInfo (offset,rate) from the utcToUt1.dat file. The next morning i  stopped, restarted pointing and it got the correct value.
  
• the routine is passed year,daynum,dayNum .. these come from extTickGetYear(),
• the dayno,,secs come from the irig server via syncClock (queries irig server every 10 secs). the problem is:
  
• in the startup script for the cpu we do:
• extTickInit  (ext tick running, but dayno, sec probably 70,1 (initial values) with year correct (comes from file)
• irigServStart (on pnt computer)
• syncClStart(10) this start the synchronizing.. this is a daemon that stays running. so we return immediately...
• more startup commands
• start point program.. if the syncClock has not yet finished synchronizing with the irig, the call to utcInpInfo(_) will have the correct year but daynum will probably be 11mar (guess who's birthday that is...).
• This is a possible race condition..
• tfor leap seconds A solution would be to not reboot the crate, but just restart the pointing program:
• rsh pnt pntMProgStop
• rsh pnt i ... till pntMProg has gone away
• rsh pnt pntMProgStart
• This way we will not be rsynchronizing the clocks. you should wait for say 20 secs after midnite before doing this...
• This race condition exists any time we reboot the pnt crate. The changes i made to get around this are:
• when pntMProg starts, i check to see if the syncClock() has done at least 1 successful synchronize with the irig server.. If not, i initialize the year,daynum, secmid from observer2.   This just needs to have the correct year,daynumber.. eventually the syncClock will succeed, and the seconds will be correct....