LCU MAINTENANCE LOG File: \hagen\wpfiles\vertexdoc\lculog.wpd

#1 (orig lcu)	6ES5 581-1Ea11 (Intel 386) failed Oct 15, 2001; Siemens couldn't repair it

#2 (1st spare) 6ES5 581-OED14 (Intel 486) always flaky, never saw platform service

100MHz	finally sent for repair, never returned,

Siemens kept the Si Disk and mem module.

#3 6ES5 581-OED14 (Intel 486) ran months in the lab, failed after 3 days on platf.

returned w#1 to Siemens who couldn't repair it.

#4	6ES5 581-OED14 (Intel 486) running on platform since Oct 15, 2001

#5	6ES5 581-OEE15 (Pentium)

#6	6ES5 581-OEE15 (Pentium)

Week of June 10-15, 2001

Phil Perillat and Luis Murray followed up the Vertex problem of last week, leaving only the LCU computer as the probable culprit. They have left the spare LCU installed and running and we will resurrect our test setup to deal with the suspicious LCU.

Week of Nov. 19, 2001

This short week was really a continuation of last week - trying to clarify the work schedule and priorites for the Alaska klystron trip and working on the circuit board and chassis layout for the 430 MHz transmitter timing generator. This can now be turned over to the technicians for fabrication and I will turn once again to the problem of having a full complement of working Vertex spares - in particular the ever troublesome LCU computer card.

Subject:Week of Dec. 10, 2001

Except for Thursday (a vacation day) I continued this working with the Vertex development system. At the week's end, I have learned to navigate through the debugging features of the system: breakpoints, dynamic memory display, interupt stack tracing, etc. I alternate between using pointing program ("lab" version, left by Klaus) and a very simple program that simply turns one of the digital i/o bits on and off and services a 1-second interrupt. While the simple program contains almost no code, it requires all the operating system "machinery" needed for the actual pointing program. It seems that, indeed, the LCU (communications CPU, a 386 pc) module is failing to establish communications with the PLC. Phil is totally familiar with the LCU software, so we should be able to combine PLC and LCU knowledge and finally get a handle on the system.

Subject: Week of Jan 14,27,18 2002 (two days vacation)

I have the PLC crate and spare boards working in the digital lab - a better set up than I had in the screen room. I seem to have learned how to get the PLC and LCU computers started together and synchronized. The Lab Test version of the PLC code runs correctly. The next steps are to test and troubleshoot the IRIG clock card and the ethernet card. Phil keeps up to date with progress on this front.Ray Jurgens at JPL found that the person who is in charge of maintaining (modifying?) the PLC code that drives the subreflector on the 70m telescopes is:

Harlow (Hal) Ahlstrom phone 818 354 1847. ahlstrom@mail1.jpl.nasa.gov

He will be back at work next week and I will contact him to see what strategy JPL has used to deal with the PLC/LCU code written by Vertex for their system and then modified by Vertex for our pointing system.

Jan 21, 2002

As I work with the development system, I've been able to make a much better block diagram/flow chart of the PLC software. With help from Phil on the C code in the LCU, I'm optimistic that we will succeed in mastering ("reverse engineering") the Vertex system.

Jan 28, 2002

Work continues on the Vertex computer system. We have learned more about the procedures needed to start the system so that the two computers synchronize before either times out. The disk drive card is now working with a working spare LCU. Next to install and troubleshoot are the IRIG time code card and the Ethernet card.

Feb 4, 2002

In the lab test setup for the Vertex pointing control system, the IRIG time code reader is now working along with the PLC and spare LCU. The remaining and final step is to add the Ethernet card. Both Phil and Jeff are helping to advance this work, and I hope we will be ready to test the spare PLC and LCU package by the end of next week.

Feb 11, 2002

Got the Ethernet card working in the lab test setup for the Vertex pointing control system. The difficulty in this final step was an address conflict; the LCU computer makes internal use of the port needed to configure to LAN card. Finally located instructions on how to change this default port assignment and configured two spare cards. With all the components now working in the test setup, we should formalize our maintenance strategy. I suggest the following:

We can consolidate the "lab test" versions of the PLC and LCU programs with the working versions, so no EPROMs have to be swapped after the spares pass the lab tests and so that only a single PLC program and a single LCU need to be maintained. We can also buy a Siemens utility program that will allow us to load the LCU program directly from an external PC so that the hard disk storage modules can be removed from the LCUs (eliminating any moving parts from the upstairs control computer). We will also write a maintenance manual describing all the steps needed to prepare and install the PLC and LCU programs and the sequences needed to get the PLC and LCU started and properly synchronized. Finally, we can improve the test set up. We should add an expansion box to the spare (lab) crate, as this is an important spare part we lack. (The expansion unit is more than just a chassis; it contains undocumented bus extension electronics). The test setup can contain the spare digital input, digital output, analog input, and analog output cards. All of these would be constantly exercised in the test program, producing simple blinking patterns on the indicator lamps. The test setup should also include our spare encoder interface card , hooked to a spare encoder, and also the spare PCU (portable control unit) interface card , hooked to a spare PCU. (At present we do not have a spare PCU, only a spare single-board computer that is the heart of the PCU. It would not take much to put together a complete spare PCU).

Feb 198, 2002

Have been trying to get a quote from Siemens on a software produce, CPLINK, that would allow us to load the on-board silicon disk in LCU computer in the pointing system directly from an external PC. Using this software we could eliminate the hard disk units now present on our working LCU and on our spare LCU. This would improve reliability and make it easier to maintain "configuration control" with a single master copy of the program, kept and archived on the SUN.

May 6, 2002

Got the test rack working with the spare PLC and LCU computers for the Vertex pointing system. The rack will normally be left running in the digital lab so that anyone passing by it can see at a glance that both spare computers are working properly. The monitor display is the same as the Vertex display in the control room; the time advances and the positions of the the Carriage House and the Gregorian Dome change (in the test program, the encoder positions are simulated from the time clock). If the test rack is off, it boots up running the test program when the power is turned back on, so cycling the power tests whether these computers will boot up properly when removed from the test rack and installed upstairs. Wrote a five-page memo with detailed procedures for configuring and replacing the computers in the upstairs system and in the test rack. This puts us a big step ahead in not being dependent on Vertex for emergency maintenance.

May 13, 2002

A final step in preparing the test/storage rack for the spare pointing system computers was to provide a test of the Ethernet interface card in the LCU computer. A small program will be written to provide a one-step process to change beetween the configurations needed for the lab test program and the operational program.

Week of May 20, 2002

Finished and distributed memo with procedures to test and swap the LCU and PLC computers in the Vertex pointing system. The LCU now has just one configuration option: the choice of booting and running from the silicon disk (non-volatile RAM) or from the hard disc. The silicon disk is the normal choice, while the hard disc can be a backup. During the boot process, the operator is prompted to specify if the system is to run the lab test version of the program or the operating version. The latter is the default, so that the LCU can be installed even without a keyboard.

The spare LCU and spare PLC will be taken to the platform and the swap process will be tested and practiced.

Week of May 21, 2002 (four days)

Arranged to order a spare IRIG time code card for the Vertex platform system. Will add some lines of code to the lab test version of the LCU code to display the time code error counter (a counter installed two years ago by Klaus and Phil during a week-long troubleshooting marathon).

: Week of June 3, 2002 (four days)

In a successful exercise in engineering managment, I got Phil to add code to the lab test version of the LCU program to display the time code error counter (a counter by Klaus and Phil during the December '91 troubleshooting marathon). This counter accumulates the number of jumps, i.e. instances of too large a difference between consecutive time readings. The new code also displays information not recorded or even tested in the working program: number of times the IRIG card failed respond and number of times that the IRIG card was unable to lock onto the time code signal. The IRIG card installed in the spare LCU test package is passing all these tests very well, so we feel this package is ready to test on the platform.

Aug. 19, 2002

Swapped the LCU computer in the Vertex pointing system with the "hot spare" which had run flawlessly for several months in the test rack in the digital lab. The original LCU had not failed or misbehaved; we simply needed to verify the correctness of the swapping procedure and to verify the correct operation of the spare unit when installed in the working system. No problems were encountered. Took photos of the interior of the control system racks in order to update the Vertex drawings to include the part designation numbers.

After two days of running, the pointing system failed, indicating a problem in the LCU. We put back the original unit and returned the spare to the lab, where, as before, it ran normally. However, after one hour in the lab, it hung up and had to be rebooted. Many more hours have passed in the lab without a second failure. This leaves us once again in the position of not having a satisfactory spare LCU. We may eventually have to contract Vertex or Seimens to solve this problem

Week of Aug, 26, 2002

Installed a new cpu board in the spare Vertex LCU processor. The unit seems to work perfectly in the downstairs test rack. We will try this unit on the platform next week.

Week of Oct. 15, 2002

Monday morning Luis Murray noticed that the LCU pointing computer in the platform equipment shelter had its red "fault" light lit. According to the Siemens manual, this indicates a CPU failure detected by a boot time diagnostic routine - "contact your Siemens representative". We attempted to reboot this computer ( LCU #1) to avail; it would not come up running the operating program. We did get it to boot up in our laboratory test rack, but the "fault" light remain lit and the unit would not synchronize with the PLC, its co- processor. We installed our only spare LCU (#4) at the platform and it has been running all week.. (Yet LCU #3 was installed under similar circumstances about two months ago and failed after three days). We have sent LCU#1 and LCU#3, which was a replacement for the never satisfactory LCU#2, to Siemens for repair. They quoted four weeks turnaround. During this period, we will again be without a spare. This board is absolutely vital for our pointing system. It is built by Siemens for their proprietary control rack and there are no second sources.

Week of Oct. 21, 2002

We are trying to buy a new spare LCU , one of two CPUs in the Vertex pointing system, in order to have a spare while Siemens repairs our failed units. Our local Siemens rep, Felix Encarnacion at Prime Controls, referred us to the Siemens office in Tennessee to check part numbers and availability. I spoke with Tony Sizemore in that office, who told me that the CP581 is now a discontinued product, even the last version, which came out in 1997, and that we may not be able to buy any version of the CP581. This is bad news; we had expected that as new versions of the Pentium chip are introduced, Siemens would continue to produce new versions of their CP581. Sizemore also was doubtful that Siemens would repair our failed units. He is going to trace the units we submitted for repair and also see if there may be a CP581 available from Germany, as none remain in stock in the U.S. He said that the Siemens S5 system, while now 30 years old, is still a current product, and mentioned several current network interface units that might be made to substitute for the CP581 - although he doesn't know that our CP581 does quite a bit of processing.

Mar. 28, 2003

Inspected the recently assembled spare LCU processor package for the Vertex pointing system. Luis Murray did the mechanical work while Jorge Rodríquez installed the software. The unit is in the test rack in the digital lab, running the Lab Test version of its program. Everything looks correct, though we will also test it at the platform in the actual system. Luis is now putting together a second package, so that we will have two spare processors. This processor is the only component in the Vertex system that has given us continuing problems. As it is now no longer available, we feel we must have two spares.