This file explains the typical administrative tasks to be performed on the SAP Sailing server. For an explanation of what this server does, please refer to the README file stored next to this ADMIN file. Emergency Startup After Reboot ------------------------------ When for some reason the server re-booted, a number of manual steps need to be performed to get back up and running. Proceed as follows: - "ssh -A trac@sapsailing.com" - "screen" Open 10 screen sessions, using the "Ctrl-a c" keyboard shortcut in the screen session. Name them, using "Ctrl-a A", according to the following scheme: 0$ dev 1$ test 2$ prod1 3$ prod2 4$ wind 5$ mongo 6$ ssh tunnel 7-$ build 8$ logs 9*$ storeandforward Additional sessions may be used for the race committee app server. The first four sessions are used to run the four server instances. Session 4 is used to run the udpmirror script that receives wind data through UDP and forwards to the four server instances. Session 5 hosts a test MongoDB instance. Session 6 hosts an ssh tunnel making sure port 12347 forwards to local port 22 to create homogeneous SSH access to the machine for build and deploy scripts. Session 7 is used for build processes. Session 8 can be used to monitor logs and server load. Session 9 is used to record and forward inbound SwissTiming data. In each screen session it is necessary to run the /home/trac/.bash_profile script by entering ". .bash_profile" This, in particular, adjusts the JAVA_HOME variable as needed. Perform the steps "Launch MongoDB" and "Launch Python Server" described below. Launch the four server instances by cd'ing to /home/trac/servers/ and invoke "./start" there. If you additionally want to upgrade from a fresh build, call "./install" before calling "./start". Follow the step "Launch UDP Mirror" described below. Follow the step "Launch SwissTiming forwarder" described below. Launching Readily-built Java Server Instances --------------------------------------------- We usually launch, manage and observe the Java server instances using a screen session. See below for how to update and restart an already running instance. - ssh -A trac@sapsailing.com - screen -rd - Connect to the screen session for your server, e.g., session 0 for dev: Ctrl-a 0 - "cd servers/" where is one of dev/test/prod1/prod2 Then launch the server: - ./start This should start an OSGi console with the latest version of the bundles deployed to ~trac/servers//plugins. Launch MongoDB -------------- The MongoDB is used to store wind information and general leaderboard-related information required by the Python web application. - log in as user "root" - "cd /opt/mongodb/bin" - "./supervisord" This will launch the MongoDB servers. If not, check for old mongod.lock files in the server directories and remove them, then try again. Launch UDP Mirror ----------------- The two production instances shall share the same wind information. Therefore, it is necessary to launch a "UDP mirror" that distributes UDP packets received on the single inbound UDP port for the production instances to the two Java server instances. In screen session 4 (Ctrl-a 4), do the following: - "cd ~/servers/prod1" - "./udpmirror -v 2012 localhost 2013 localhost 2014 localhost 2010 localhost 2011" The "-v" argument ensures that UDP packets received will be dumped to the console. The process will listen for inbound UDP packets on port 2012 and forward anything received on that port to localhost:2013 as well as localhost:2014. See also the list of ports as documented in the README file. Launch SwissTiming forwarder ---------------------------- Go to screen session 9 (Ctrl-a 9), then do the following: - "cd ~/servers/prod1" - "./swisstiminglistener 3500 3501" Apache HTTP Server configuration -------------------------------- To define the different URL's to access the java servers you need to configure the corresponding virtual hosts of the Apache Http server. The configuration file can be found at /etc/httpd/conf.d/kiel.conf Re-Building the Java Server --------------------------- If, e.g., after a bug fix, it is necessary to re-build and re-deploy the Java server, you should consider doing this first on the "dev" branch of the git versioning system. The master branch contains the sources for the "dev" server instance. The git branch to server instance mappings are as follows: git branch server instance ---------- --------------- master (internal development inside SAP network) dev dev test test prod1 prod1 prod2 prod2 First, check out the branch you want to re-build: - log in as user "trac" - ensure that your SSH key is loaded / unlocked, either by logging in using SSH with the -A argument, or by invoking the alias "sa" and entering the password for the trac user's SSH key - "cd git/java" - "git checkout -f " where is either one of dev, test, prod1 or prod2 - ensure the "ulimit -n" output is at least 4096 to enable the GWT compiler to assemble the resource sets which consist of many files that all need to be opened concurrently. Currently, the maximum value for "ulimit -n" is configured in /etc/security/limits.conf and is set to 16384. This specified the maximum amount to which a user's shell can set this value. The ~trac/.bash_profile contains a "ulimit -n 4096" command, but when running "screen" the shells usually are no login shells. You need to make sure you run ~trac/.bash_profile in the build shell to set the limit of open files to at least 4096. - rm -rf com.sap.sailing.gwt.ui/com.sap.sailing.gwt.ui.*; mvn -fae -s /home/trac/git/settings.xml -P debug.without-proxy clean install 2>&1 | tee log This will compile the sources, bundle them up in to OSGi bundle JAR files, run all tests which required online access to the TracTrac servers, and, if successful, creates an eclipse product in the directory /home/trac/git/java/com.sap.sailing.feature.p2build/bin/products/raceanalysis.product.id/linux/gtk/x86_64 which contains all files required to install and run the build result. Note that the running server(s) are not updated automatically. See next topic. OSGi Server Instances --------------------- We usually have four OSGi consoles running with a terminal window each, visible at the VNC server running on por 5902. The title bar shows the current working directory which should be something like "~/servers/" where is one of dev, test, prod1 and prod2. At the "osgi>" prompt of the OSGi console you can type - "exit" to terminate the particular Java server instance. You can either install a complete new server instance or just update an existing one. Install a new OSGi Server Instance ---------------------------------- To install a complete new server instance you can issue - "./install" This command will delete the old installation and copy all required files from the build result to the right directories of the server instance. To start the server instance again, type - "./start" and the server should welcome you again with the "osgi>" prompt after a successful start. Update an existing OSGi Server Instance ----------------------------------------- To update the server you can issue - "./update" This command will NOT delete the old installation but copy the updated plugins from the build result to the right directories of the server instance. To start the server instance again, type - "./start" and the server should welcome you again with the "osgi>" prompt after a successful start.