rebuilding the system software

03aug07

The pdev user
Important files and their locations locations
Building the software

A few different ways to build the software
Things to do before building the software
Build the host tools cfitsio and fftw
Making the tools
Making the pdev software
Before running with a new system.
Changes to run on tdevs0
Installing the linux software on the N (14 or 1) pdev servers

History:

09sep08: update installing the software for changes to install on pdevs1 set or pdevs0 set.

The pdev user: (top)

We've added the user pdev to the pdevs0 system. The different versions of the software should be made here. The files in this directory are:

bin - empty so far
local - directory holding code for file processing: cfitsio, fftw in ./local/src.. lib in local/lib
pdev - current working version of software built from jeff mocks svn repository.
PDEVR - (pdevroot) in the code belowis actually pdev
pdev-cd: holds the dump files of the various releases. Use these to create the various versions. T
pdev.sh : shell variables needed for building
Readme : this file

Important files and their locations: (top)

After the system is built the following files and locations are used (note the pdev.xxxx image has not yet been implemented). If you are running multiple boxes/servers only one of them is the dns server. You should also only have to make the system on one of them, we should have an rdist script to load the rest of them.

Linux os files:

/etc/services :

added port pdev 1421 . This is used by run to connect to psrv.
added port bpcmd 1422. This starts the buffer pool cmd program.

/etc/dhpcd.conf 1 entry for each spectrometer box. Maps Ethernet address to dns name. Allows for dhcpd to supply ipaddress and boot host.
/etc/hosts maps ip addresses to host names for the file servers and spectrometers. It also defines the host master. psrvs0 is running named (dns server) but it does not include pdev.naic.edu in the /etc/resolv.conf so the file servers need to use the hosts file to map server/spectrometer names to ipaddresses. The names in the hosts file look like the dns names, but they are just regular host names.
/etc/syslog.conf: the ao versions of psrv, bpinp,bpout use syslog local0. I've added a separate line local0.* /var/log/messages so that debug messages are included. You should add local0.none to the *.info line so you don't get double messages.
/etc/xinetd.d/pdev tells what program to start (/usr/local/pdev/psrv) when someone connects to port 1421. This is how prun connects to psrv (via port 1421).
/etc/xinetd.d/tftp You need to switch the disable keyword from yes to no so that tftp booting can work.
/etc/xinetd.d/bpcmd To start the buffer pool command program.
/etc/sysconfig/network-scripts/ifcfg-eth1: Configuration script for eth1 (where spectrometer box connects to psrvxxx). This maps the ipaddress to the Ethernet port, disables the bootprotocol (default is dhcpd).
/var/named/chroot/etc/named.conf Defines the domain pdev.naic.edu and associates it with 192.168.7.0/24.
/var/named/chroot/var/named/pdev.naic.edu Lists the ipaddress mapping for servers and spectrometers for pdev.naic.edu. This is only used for the spectrometer boxes. The file servers use host files.
/var/named/chroot/etc/named.rfc1912.zones, /var/named/chroot/etc/named.caching-nameserver.conf These are used by the named.conf. They need links from /etc to here..
The .ssh key files for logging into the spectrometer boxes are stored on pdevs0:/root/.ssh/id_rsa id_rsa.pub. You should grab these and put the in the /root/.ssh and maybe ~pdev/.ssh/ directories so you can login to a spectrometer box without a password.
Make sure that /etc/rc3.d has: S65dhcpd (it defaults to K65dhcpd).

Booting of the boxes, dhcp and tftpboot

The spectrometer boxes use dhcp to get their boot info. It is configures use the file:/etc/dhcpd.conf. The file contains:
- The dns domain name for the clients (pdev.naic.edu).
- ntpservers, routers, dns servers,logservers,boot server (the host master aliased to psrvs0).
- The Ethernet address for each board. The last digits are the box number: aa:bb:cc:01:01:00 for box 101
- the fixed address and optional hostnames: pdev-102.pdev.naic.edu for box 102
- the boot file (pdev) for tftp.
- dhcpd.conf should only be on pdevs0 (or whoever is the boot server).
dhcpd maps the Ethernet address to the spectrometer boxes ip address. It returns this as well as the boot server (192.168.1.1 (pservs0)) and the boot file (/tftpboot/pdev) to the spectrometer. The spectrometer then boots the image in /tftpboot/pdev from psrvs0.
For tftp booting to work, the file /etc/xinetd.d/tftp needs to be edited so that the disable keyword is no (the default is yes).

Creating the boot image:

PDEVR /sw/make Makefile will create the file:PDEVR /sw/image/build/bootimg and then try to copy it to /tftpboot/pdev (this usually fails so you have to copy it manually).
The copy command in the make file is: scp ${BOOT_IMG} ${PDEV_BOOT} where boot_img is bootimg and pdev_boot is set in the pdev.sh file to be something like: pdev@pdevs0.naic.edu:/tftpboot/pdev

DNS name service:

The communications between the spectrometers and the file servers use the domain name pdev.naic.edu with an ipaddress of 192.168.7.0/24.

The domain is defined in /var/named/chroot/etc/named.conf.
The individual servers and spectrometer boxes are listed in /var/named/chroot/var/named/pdev.naic.edu.
You also need:

from /var/named/chroot/etc/ named.rfc1912.zones and named.caching-nameserver.conf. These need symbolic links from /etc.
From /var/named/chroot/var/named/ : localdomain.zone, localhost.zone, named.broadcast, named.ca, named.ip6.local, named.local, named.zero. These are needed by named.rfc1912.zones.

The hostname master.pdev.naic.edu is aliased to pdevs0 (in the hosts file, dhcpd.conf, and in pdev.naic.edu). spectrometer boxes use this name when booting or mounting filesystems.
Dns is only used by the spectrometer boxes, the file servers use the hosts file. There is no entry in /etc/resolv.conf for pdev.naic.edu on the file servers.
If the other file servers do not run named, they will not know about the domain pdev.naic.edu. This is probably ok since only the spectrometer boxes use dns and they all talk to pdevs0.

pdev software files:

/home.local/pdev (~pdev):

holds the release files (see above). The build source sits in this directory.

/home.local/pdev/linuxfiles

directory holding the Linux files needed by pdev system (see above).

/home.local/pdev/PDEVR

each version build will reside in a subdirectory of ~pdev. PDEVR should be replace with pdev0701,pdev0706, or pdevcur ...
pdev is the version that is updated directly from jeffs svn repository.

~pdev/pdev.sh: environment variables needed for build
/usr/local/pdev

location for tools and programs
make tools stores info here, make sw accesses tools from here and stores programs here.
This directory name is defined in pdevxx/tools/make/common.mk, pdevxx/sw/common.mk. Both of these define the make variable PDEV_TOOLS=/usr/local/pdev. The make files then used $(PDEV_TOOLS).
The shell script pdev/pdev.sh adds /usr/local/pdev to the $path so you can find programs like pnet and psrv.
I've made this into a link so it can point to whichever version is being used: pdev0701,pdev0706, pdevcur, etc..

/home.local/pdev/local/

./lib holds libciftsio.a and libfftw3.a. these are used when building spcvt offline.

~pdev/PDEVR/sw/src: source code

The following get copied to /usr/local/pdev/bin on psrvsX:

pnet: perl script to run datataking
psrv.c-> psrv: program to take data from prun and write to disc.
spcvt.c->spcvt: offline program to look at .pdev files. (this also goes to the spectrometer box??).

The following get copied to ~pdev/sw/root/bin and eventually end up in the image for the box:

prun.c->prun: program that runs on the spectrometer. includes plinth.c

pdiag.c->pdiag:
pmon.c->pmon. monitoring that goes to lcd display (and web?).

The filter coef for the polyphase filter:

pfb.nnnn.hamming. nnnn = 64 to 8192 in power of two
They are created by the make file using the perl script pdev/gx/jfft/src/mkcoef
The coef are copied to /usr/local/pdev/etc for the server.
The coef are copied to sw/root/etc for the boxes. The sw/root is eventually copied to image/root

Building the software:

A few different ways to build the software: (top)

You are starting from scratch with a new pserver and spectrometer box. You need to setup everything.
You already have a psrv setup and you want to rebuild a new version of the pdev software. Some options are:

You want to just overwrite the current version with the new version (this is jeffs default).
You want to build a test version leaving the current version as the the production version. To do this you need to:

Change the default destination /usr/local/pdev -> /usr/local/pdevxxx where xxx is some version number. You can do this in the pdev.sh (after editing some of jeffs common.mk files
Change where the boot image ends up.. (now done in the pdev.sh file)
After building the new system. You need to switch some links /usr/local/pdev -> and /tftpboot/pdev -> so they point to the new system. You can't run both current and test versions together, but it is pretty easy to switch between the 2 (just 2 links).

Things to do before building: (top)

The build needs to be done a 64 bit version of Linux (eg. uname -a )

Linux tdevs0.naic.edu 2.6.20-1.2952.fc6 #1 SMP Wed May 16 18:18:22 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

Assume the pdev server names are pdevsN
Setup the ssh keys on each pdevsN:

run /share/megs/phil/svn/pdev/sshkeys/rdist.cmd. You'll need to enter the root passwd 7 times.

Run script to install linux software:

cd /share/megs/phil/svn/pdev/linuxfiles
If you're not phil, get the root keys for the spectrometer and put them in your .ssh/ (see the Readme)
Check that input Data files are correct:

install.dat .. ip,mac addresses for pdev servers. used to create hosts file, ifcfg-eth0/1
hosts.tmpl .. template. holds hosts portion of the file that is the same for each server.

Don't bother with hosts, ifcfg-eth0/1 these files are generated on the fly..
install.sc will do all the work.

Get jeffs pdev data. Your options are:

checkout a current version from jeffs website to get the most recent updates(a few hundred mb) .

as pdev on pdevs0. cd ~pdev
svn checkout https://www.mock.com/svn/pdev/trunk PDEVR
After you've done this the first time (which takes awhile) you can then get the updates via:

cd ~pdev; cd ;svn update

Create your own svn repository from one of the dump files that jeff created:

cd /tmp ; sftp pdev@pdevs0.naic.edu

cd /home.local/pdev/pdev-cd
get pdev.dump.20070624 .. get the dump file

as root or pdev create svn repository, put data in the repository, and then check the source out.

svnadmin create /tmp/pdevsvn
svnadmin load /tmp/pdevsvn < /tmp/pdev.dump.20070624
cd /home.local/pdev
svn checkout file:///tmp/pdevsvn/trunk PDEVR .. revision 975 checked
get rid of the files in /tmp.

rm -rf /tmp/pdevsvn
rm /tmp/pdev.dump.20070624 .. unless you want to keep it around for later reference.

Get the script to set the environment variables used in building:

you may want the .bashrc from pdevs0:/home.local/pdev/.bashrc .
get pdev.sh from pdevs1:/home.local/pdev/pdev.sh and put it in ~pdev

Edit script file ~pdev/pdev.sh:

edit export PDEV_TOOLS=/usr/local/pdev to PDEV_TOOLS=/usr/local/PDEVR
change PDEV_BOOT=pdev@pdevs1.naic.edu:/tftpboot/pdev to ..tftpboot/pdev.cur . This will be the bootimg name in the tftpboot directory.

edit the common.mk files (eventually these may be updated..)

in PDEVR/sw/make/common.mk.. comment out the line PDEV_TOOLS=
in PDEVR/tools/make/common.mk change:

TOOLS_DIR = /usr/local/pdevcur TO TOOLS_DIR = ${PDEV_TOOLS}

temp edits:

revision 990 and before. sw/src/Makefile.. spcvt: change -I.. to -I${PDEV_HOSTINC}

Build the host tools cfitsio and fftw (top)

If you are making a new version of the software, you can probably just use the old tools in pdev/local. If you are on a new server you'll have to do the following:

Get the local directory things from pdev/local

cd ~pdev
rsync -av pdev@pdevs0:/home.local/pdev/local/ /home.local/pdev/local
build cfitsio lib (if you don't want the binary that is already in local/lib

cd ~pdev/local/src
tar -xvf cfitsio3030.tar.gz
cd fitsio

.configure --prefix=/home.local/pdev/local ... probably then need to do make

build fftw lib; cd ~pdev/local/src

tar -xvf fftw-3.1.1.tar.gz
./configure --prefix=/home.local/pdev/local ... probably then need to do make

Making the tools: (top)

become root (since it will try to write to /usr/local/PDEVR/)
If it doesn't exist, create the directory /usr/local/PDEVR
Here is how to grab an already built version of the tools:

cd /tmp
~sftp pdev@pdevs0.naic.edu
cd pdev-cd/e
get pdev-tools-bin-64bit.tgz ; bye
tar -xzvf /tmp/pdev-tools-bin-64bit.tgz -C
mv /tmp/pdev /usr/local/PDEVR
You are now done making the tools

Here is how to compile the tools:

make sure you've executed . pdev/PDEVR/pdev.sh

cd pdev/PDEVR/tools; make clean

make 2>&1 |tee makefile.log
There is a bug in the /usr/local/PDEVR/bin/mkimage program when built with the 64 bit Linux. It will create images but the tftpboot will not work. To fix this:

cd /usr/local/PDEVR/bin
mv mkimage mkimage.64
cp mkimage.32 from patches.. cp -p /home.local/pdev/patches/mkimage.32 /usr/local/PDEVR/bin/
ln -s mkimage.32 mkimage..
this will use the 32 bit version of mkimage that does work..

when done the tools are installed in /usr/local/PDEVR

Making the pdev software: (top)

After the tools are installed you can make the pdev software and image for the boxes.

need to be root since it writes to /usr/local/PDEVR
cd ~pdev ; . pdev.sh
cd ~pdev/pdev/sw
make clean
make 2>&1|tee makefile.log

at the end the makeimg script will ask for the pdev passwd so it can scp bootimg to pdevs0:/tftpboot/pdev.xxxx
If this fails, just do it manually...

The programs pnet, psrv are now in /usr/local/PDEVR and the image for the boxes is in /tftpboot/pdev.cur
You should probably copy the files PDEVR/sw/src/ boxes.conf cal.conf pnet.conf sp1def.conf to /usr/local/PDEVR/etc so that pnet can find them if the user doesn't provide them.

Be sure and edited teh boxes.conf and change the rightmost field from ao to pdevs0 (jeff's file server is called ao).

Before running with a new system: (top)

Before running with a new system you need to do the following:

Make sure that the system files have been configured correctly (Linux os files).
Make the following links:

ln -s /usr/local/PDEVR /usr/local/pdev
ln -s /tftpboot/pdev.xxx /tftpboot/pdev

make sure your path has /usr/local/pdev/bin.. the pdev.sh script in the top ~pdev/ directory is set for this.

Changes for tdevs0 test machine: (top)

These are the changes to get a spectrometer to run on a file server that is not called pdevs0. In this example i'm calling it tdevs0.

In pdev/linuxfiles:

pdev.naic.edu edit changing /^pdevs/s//tdevs/ to change the file server names to tdevsxxx

Copy the following files from pdev/linuxfiles/ (as root):

cp -p named.conf /var/named/chroot/etc/
cp -p pdev.naic.edu /var/named/chroot/var/named/
cp -p dhcpd.conf /etc/ .. overwrite the current file.
cp -p pdev /etc/xinetd.d/
cp -p tftp /etc/xinetd.d/
cp -p named.caching-nameserver.conf /var/named/chroot/etc/
cp -p named.rfc1912.zones /var/named/chroot/etc/
cp -p the following files: localdomain.zone, localhost.zone, named.broadcast, named.ca, named.ip6.local, named.local, named.zero .. to /var/named/chroot/var/named .. make sure you use the -p since the group has to boe named. These files also have symbolic links from /var/named. (i'm not sure the links are necessary).

edit /etc/services:

add the line: pdev 1421/tcp # PDEV spectometer

Create the links in /etc for the named conf files:

ln -s /var/named/chroot/etc/named.caching-nameserver.conf named.caching-nameserver.conf
ln -s /var/named/chroot/etc/named.rfc1912.zones named.rfc1912.zones

edit pdev/linuxfiles/hosts file:

change pdevs to tdevs
change the ipaddr fir tdevs0 to 192.65.176.19 (or whatever it should be).
after editing, cp -p pdev/linuxfiles/hosts /etc/hosts .. overwrite the old file. (you might want to cp /etc/hosts /etc/hosts.orig first).

edit /etc/sysconfig/network_scripts/ifcfg-eth1.. leave the hw address for the port..

DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
NETMASK=255.255.255.0
IPADDR=192.168.7.1
USERCTL=no
IPV6INIT=no

sep08: Installing the linux software on the N (14 or 1) pdev servers

Update so you can install the software on the pdevs1 network (astronomy/spectral line) and pdevs0 (radar, aeronomy).
on pdevs1: uname -a is:

Linux pdevs1.naic.edu 2.6.20-1.2962.fc6 #1 SMP Tue Jun 19 18:50:05 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
This is a newer version of linux than pdevs0

Wrote script to install the pdev specific linux software on the servers:

Setup the ssh keys on each pdevsN:

Go to /share/megs/phil/svn/pdev/sshkeys. Look at Readme then run rsync_pdevs[10].cmd. You'll need to enter the root passwd N times. This only needs to be done once.

Run script to install linux software:

cd /share/megs/phil/svn/pdev/linuxfiles. Look at Readme file
If you're not phil, get the root keys for the spectrometer and put them in your .ssh/ (see the Readme)
Check that input Data files are correct:

install.dat .. ip,mac addresses for pdev servers. used to create hosts file, ifcfg-eth0/1
hosts.su .. template. holds hosts portion of the file that is the same for each server.
pdev.naic.edu.pdevs[01]. hosts dns info for the two networks

Don't bother with hosts, ifcfg-eth0/, pdev.naic.edu these files are generated on the fly..
install.sc or install.sc -s 0 will do all the work.

Move jeff's pdev software to pdevs[10]. (pdevs1 will be the new master)

used svn checkout of jeff's svn server.. see above. installed in ~pdev/pdev

used the ./local tools that were already brought over from pdevs0 (didn't recompile them)

made the tools (see above)..

cp mkimage.32 from patches directory to /usr/local/pdev/bin then linked to mkimage

Making jeffs pdev software:

In pdev.sh had to change PDEV_BOOT from pdevs0 to pdevs1.. This depends on which network you are on.
did the steps above to make jeffs software
The last step scp build/boot_img pdev@pdevs[01]:/tftpboot/pdev.cur fails because pdev can not write to /tftpboot... do the copy manually

trouble with dns on pdevs1

ping master from pdev-110 was failing.
reverse lookup was going pdev-110 to pdevs1 then to outside internet.
/etc/nsswitch.conf did not have yp included, pdevs0 did have yp
to monitor the problem.

pdevs1: tcpdump -n -s 128 -i eth0 -port 53
pdevs1: tcpdump -n -s -128 -i eth1 -port 53
Then watch from pdev-110 nslookup master

Problem fix: added reverse lookup domain pdev.rev and added forwarders to named.conf