eclipse-sailing-analytics/sailing-analytics
1
0 Fork 0
mirror of https://github.com/eclipse-sailing-analytics/sailing-analytics.git synced 2026-11-04 01:59:35 +00:00
Code Activity

Merge branch 'main' into bug5774

Browse Source
This commit is contained in:
Axel Uhl committed 2024-08-13 17:23:49 +02:00
2 parents 1a1ffe1905 86dd97ccf1
commit e2433bbc76
530 files changed
+2757 -1774
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/release.yml
+1 -1
View File
@@ -57,7 +57,7 @@ jobs:
APP_PARAMETERS: "-Daws.region=eu-west-1 -Dgoogle.maps.authenticationparams=${{ secrets.GOOGLE_MAPS_AUTHENTICATIONPARAMS }} -Daws.s3.test.s3AccessId=${{ secrets.AWS_S3_TEST_S3ACCESSID }} -Daws.s3.test.s3AccessKey=${{ secrets.AWS_S3_TEST_S3ACCESSKEY }}"
JAVA8_HOME: ${{env.JAVA_HOME_8_X64}}
run: |
./configuration/buildAndUpdateProduct.sh -x 60 ${{ github.event.inputs.skip_tests == 'true' && '-t' || '' }} build 2>&1 | tee build.log
./configuration/buildAndUpdateProduct.sh -x 60 ${{ github.event.inputs.skip_tests == 'true' && '-t' || '' }} build 2>&1
- name: Upload build log
uses: actions/upload-artifact@v4
if: always()
README.md
+11
View File
@@ -145,6 +145,17 @@ In this example, find your web application at http://localhost:32779 which is wh
```
to connect to the server's OSGi console.
Alternatively, use an "environment" definition that sets useful defaults, e.g., like this:
```
docker run -P -it --rm -e "SERVER_NAME=test77" -e "USE_ENVIRONMENT=live-master-server" -e "REPLICATE_MASTER_BEARER_TOKEN=BRxGpF0nr68Z4m/f13/MgiYhdRB3xoDCYd+rLc17rTs=" ghcr.io/sap/sailing-analytics:latest \
bash -c "rm env.sh; echo \"
SERVER_NAME=test77
USE_ENVIRONMENT=live-master-server
REPLICATE_MASTER_BEARER_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\" | ./refreshInstance.sh auto-install-from-stdin; ./start fg"
```
This will download and install the latest release and set up a primary/master instance, here as an example for server name "test77" and synchronize security data from ``security-service.sapsailing.com`` using the bearer token specified as ``REPLICA_MASTER_BEARER_TOKEN``.
The default Docker image built by ``docker/makeImageForLatestRelease`` uses the ``eclipse-temurin:8-jdk`` base image. If you'd like to use the SAP JVM 8 which gives you great profiling and reversible on-the-fly debugging capabilities, you may build a Docker image for it, using ``docker/Dockerfile_sapjvm``. Using it to build an image will accept version 3.2 of the SAP tools developer license on your behalf. Build the image, e.g., like this:
```
cfg/VERSION
+1 -1
View File
@@ -1 +1 @@
1.4.118
1.4.119
configuration/archiveDb.sh
-1
View File
@@ -1 +0,0 @@
environments_scripts/central_mongo_setup/files/var/lib/mongodb/archiveDb.sh
configuration/buildAndUpdateProduct.sh
+4 -1
View File
@@ -667,10 +667,13 @@ if [[ "$@" == "build" ]] || [[ "$@" == "all" ]]; then
echo "ANDROID_HOME=$ANDROID_HOME"
PATH=$PATH:$ANDROID_HOME/tools/bin
PATH=$PATH:$ANDROID_HOME/platform-tools
SDK_MANAGER="$ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager"
SDK_MANAGER="$ANDROID_HOME/cmdline-tools/8.0/bin/sdkmanager"
if [ \! -x "$SDK_MANAGER" ]; then
SDK_MANAGER="$ANDROID_HOME/tools/bin/sdkmanager.bat"
fi
echo "SDK_MANAGER=${SDK_MANAGER}"
echo "cmdline-tools:"
ls -l "$ANDROID_HOME/cmdline-tools/"
BUILD_TOOLS_VERSION=`grep "buildTools = " build.gradle | cut -d "\"" -f 2`
echo "BUILD_TOOLS_VERSION=$BUILD_TOOLS_VERSION"
configuration/cp_root_mail_properties
-1
View File
@@ -1 +0,0 @@
environments_scripts/sailing_server/files/usr/local/bin/cp_root_mail_properties
configuration/cp_root_mail_properties_sudoers
-1
View File
@@ -1 +0,0 @@
environments_scripts/sailing_server/files/etc/sudoers.d/cp_root_mail_properties_sudoers
configuration/crontab
-3
View File
@@ -1,3 +0,0 @@
* * * * * export PATH=/bin:/usr/bin:/usr/local/bin; sleep $(( $RANDOM * 60 / 32768 )); update_authorized_keys_for_landscape_managers_if_changed $( cat /root/ssh-key-reader.token ) https://security-service.sapsailing.com /root 2>&1 >>/var/log/sailing.err
# NOTICE: Please try to reference the customised crontabs at $GIT_HOME/configuration/crontabs or use
# the build_crontab_and_setup_files command in imageupgrade_functions.sh. This file has been maintained for continuity, but is deprecated.
configuration/crontabs/README
+23 -3
View File
@@ -1,7 +1,27 @@
This is the crontab repo and it contains one line crontabs for all the different environments.
These files are concatenated by the build-crontab-and-cp-files script. Any time the crontab should contain
a user's home directory, instead write PATH_OF_HOME_DIR_TO_REPLACE; if the crontab should
contain the path to the git directory, instead write PATH_OF_GIT_HOME_DIR_TO_REPLACE.
These are replaced by the build-crontab-and-cp-filesscript.
a user's home directory, instead write PATH_OF_HOME_DIR_TO_REPLACE.
These are replaced by the build-crontab-and-cp-files script found under
configuration/environments_scripts.
Further replacements in the crontab snippets can be controlled by the names used for the
files or symbolic links used in the users/ directories, as follows:
crontab-snippet@{PARAM1}={VALUE1}@{PARAM2}={VALUE2}...
will replace all occurrences of PARAM1 in the snippet by VALUE1, etc. The values can be
empty. If a value shall contain an @ sign, use @@ instead. For a / (slash) character
use an underscore _ character, and to produce a single underscore, double it (__), as
illustrated by the following example:
crontab-snippet@EMAIL_ADDRESS=john.doe@@example.com@HOME_DIR=_home_john__doe_
It is good practice to name a reusable crontab snippet such that its parameters are
listed in its name, separated from the name and from each other with @-signs, as in
crontab-snippet@USER@EMAIL_ADDRESS
which would indicate that two placeholders exist in the snippet that a name for a symbolic
link referencing this snippet shall configure.
Note: theses files are symbolically linked to, so beware of the ramifications of changes.
configuration/crontabs/crontab-download-new-archived-trac-trac-events
-1
View File
@@ -1 +0,0 @@
15 12 * * * PATH_OF_GIT_HOME_DIR_TO_REPLACE/configuration/downloadNewArchivedTracTracEvents.sh PATH_OF_HOME_DIR_TO_REPLACE/static/TracTracTracks "PATH_OF_GIT_HOME_DIR_TO_REPLACE">PATH_OF_HOME_DIR_TO_REPLACE/downloadNewArchivedTracTracEvents.out 2>PATH_OF_HOME_DIR_TO_REPLACE/downloadNewArchivedTracTracEvents.err
configuration/crontabs/crontab-syncgit
-1
View File
@@ -1 +0,0 @@
*/10 * * * * export PATH=/bin:/usr/bin:/usr/local/bin; syncgit PATH_OF_GIT_HOME_DIR_TO_REPLACE
configuration/crontabs/crontab-update-authorized-keys
-1
View File
@@ -1 +0,0 @@
* * * * * export PATH=/bin:/usr/bin:/usr/local/bin; sleep $(( $RANDOM * 60 / 32768 )); update_authorized_keys_for_landscape_managers_if_changed $( cat PATH_OF_HOME_DIR_TO_REPLACE/ssh-key-reader.token ) https://security-service.sapsailing.com PATH_OF_HOME_DIR_TO_REPLACE # 2>&1 >>/var/log/sailing.err
configuration/crontabs/crontab-update-authorized-keys-ec2-user → configuration/crontabs/crontab-update-authorized-keys@USER_HOME_DIR
+1 -1
View File
@@ -1 +1 @@
* * * * * export PATH=/bin:/usr/bin:/usr/local/bin; sleep $(( $RANDOM * 60 / 32768 )); update_authorized_keys_for_landscape_managers_if_changed $( cat PATH_OF_HOME_DIR_TO_REPLACE/ssh-key-reader.token ) https://security-service.sapsailing.com /home/ec2-user # 2>&1 >>/var/log/sailing.err
* * * * * export PATH=/bin:/usr/bin:/usr/local/bin; sleep $(( $RANDOM * 60 / 32768 )); update_authorized_keys_for_landscape_managers_if_changed $( cat PATH_OF_HOME_DIR_TO_REPLACE/ssh-key-reader.token ) https://security-service.sapsailing.com USER_HOME_DIR # 2>&1 >>/var/log/sailing.err
configuration/downloadNewArchivedTracTracEvents.sh
-1
View File
@@ -1 +0,0 @@
environments_scripts/central_reverse_proxy/files/usr/local/bin/downloadNewArchivedTracTracEvents.sh
configuration/environments_scripts/README
+16 -6
View File
@@ -1,14 +1,24 @@
The environments_scripts directory contains more directories, which map to environment types, each holding scripts and files useful to that environment. Each environment type also contains a "users" folder, which contains subfolders for each user on the system.
The build-crontab-and-cp-files script uses the contents of these subfolders to create and install a customised crontab for each user in the folder. It does this by concatenating one liner crontabs into an uber crontab.
Any time the crontab should contain a user's home directory, instead write PATH_OF_HOME_DIR_TO_REPLACE; if the crontab should
contain the path to the git directory, instead write PATH_OF_GIT_HOME_DIR_TO_REPLACE.
These are replaced by the build-crontab-and-cp-files script with the correct paths
The environments_scripts directory contains more directories, which map to environment
types, each holding scripts and files useful to that environment.
Each environment type also contains a "users" folder, which contains
subfolders for each user on the system.
The build-crontab-and-cp-files script uses the contents of these subfolders to
create and install a customised crontab for each user in the folder.
It does this by concatenating one liner crontab files/links following the naming
pattern "crontab*" into an uber crontab.
For crontab snippet placeholder substitutions, please see
${GIT_ROOT}/configuration/crontabs/README.
environment_scripts (directory)
|
|_environment_type (directory)
|_{environment_type} (directory)
|
|_usefulScripts
|_files (directory)
| |
| |_{contents to copy to environment, starting from / folder}
|
|_users (directory)
|
|_user1 (directory)
configuration/environments_scripts/build-crontab-and-cp-files
+9 -17
View File
@@ -5,9 +5,8 @@
# file for each user. It also takes the name of the user containing a copy of the git repo as well as the name of the
# dir within that user to go to. The created crontab for each user go into that user's home dir and are installed for
# that user too.
# Within these crontab stubs, are certain string literals, which are replaced: PATH_OF_GIT_HOME_DIR_TO_REPLACE becomes
# the absolute path to the git home dir, and PATH_OF_HOME_DIR_TO_REPLACE is changed to the path of the home directory
# in which the crontab will be installed (ie. that user's home).
# Within these crontab stubs, are certain string literals, which are replaced: PATH_OF_HOME_DIR_TO_REPLACE is changed
# to the path of the home directory in which the crontab will be installed (ie. that user's home).
# Furthermore, within the users folder optional uid, gid and groups files an exist. uid should store the user's uid,
# gid the group id and groups should have secondary groups (one per line). The user is added to the secondary groups
# after all users are created.
@@ -17,10 +16,10 @@
# will be automatically enabled.
# For options, see the help below.
# This script should NOT be installed into /usr/local/bin. See the README for more.
. repo/usr/local/bin/imageupgrade_functions.sh
INSTALL_CRONTAB="true"
COPY_FILES="true"
CREATE_CRONTABS="true";
CREATE_CRONTABS="true"
options='fnc'
while getopts $options option
do
@@ -33,18 +32,14 @@ do
esac
done
shift $((OPTIND-1)) # shift the arguments along so there are no options in the arguments anymore.
if [[ "$#" -ne 3 ]]; then
echo "$0 [ -f ] [ -n ] [ -c ] <ENVIRONMENT_TYPE> <USER_WITH_COPY_OF_REPO> <RELATIVE_PATH_OF_GIT_DIR_WITHIN_USER>"
if [[ "$#" -ne 1 ]]; then
echo "$0 [ -f ] [ -n ] [ -c ] <ENVIRONMENT_TYPE>"
echo ""
echo "Where USER_WITH_COPY_OF_REPO is a user that contains a checked out copy of the main git."
echo "And where RELATIVE_PATH_OF_GIT_DIR_WITHIN_USER is the path to the git repo from the USER_WITH_COPY_OF_REPO's home directory."
echo "Use the f(iles) flag to disable file copying. Use the c(rontab) flag to disable the creation of users and the creation of crontab files for those users."
echo "If the c flag is not used then the n(o install) flag can be used to setup the crontabs but not install them."
exit 2
fi
ENV_TYPE="$1"
GIT_USER="$2"
RELATIVE_GIT_DIR_NAME="$3"
cd "$(dirname "$0")/${ENV_TYPE}"
if [[ -d "groups" ]]; then
cd "groups"
@@ -61,7 +56,6 @@ if [[ -d "groups" ]]; then
fi
if [[ -d "users" ]]; then
cd "users"
GIT_PATH="$(eval echo $(printf "~%q" "$GIT_USER"))/${RELATIVE_GIT_DIR_NAME}" # The absolute path to a git repo in the environment type.
for dir in *; do
[[ -d "$dir" ]] || continue
USERNAME="$dir"
@@ -96,15 +90,14 @@ if [[ -d "users" ]]; then
# Clear the crontab file before assembling it from the snippets:
> $HOME_DIR/crontab
echo "# Note that this file should not be edited manually. Please instead make use of the build_crontab_and_setup_files command in imageupgrade_functions.sh">>"$HOME_DIR"/crontab
echo "# which concatenates the crontabs found at GIT_HOME/configuration/crontabs into this file. Please edit these crontab stubs in the git repo to make permanent changes.">>"$HOME_DIR"/crontab
echo "# which concatenates the crontabs found at users/*/* into this file. Please edit these crontab stubs in the git repo to make permanent changes.">>"$HOME_DIR"/crontab
echo "">>"$HOME_DIR"/crontab
for crontab in ${USERNAME}/crontab*; do
[[ -e $crontab ]] || continue
cat "${crontab}">> $HOME_DIR/crontab
cat "${crontab}" | eval $( get_replacements_from_crontab_filename_as_sed_command "${crontab}" ) >> $HOME_DIR/crontab
echo "">> $HOME_DIR/crontab
done
chown "$USERNAME":"$USERNAME" $HOME_DIR/crontab
sed -i "s|PATH_OF_GIT_HOME_DIR_TO_REPLACE|${GIT_PATH}|g" "$HOME_DIR"/crontab # Sets correct path to the git repo within the crontab.
sed -i "s|PATH_OF_HOME_DIR_TO_REPLACE|${HOME_DIR}|g" "$HOME_DIR"/crontab # Sets the correct path to the home dir of the user whose crontab will be installed.
sed -i '/^$/d' $HOME_DIR/crontab # purges random empty lines.
echo "">>$HOME_DIR/crontab # Adds a newline
@@ -113,7 +106,6 @@ if [[ -d "users" ]]; then
fi
fi
fi
echo "****** finished user ******"
done
# Add to groups after all the groups exist.
@@ -149,4 +141,4 @@ if [[ "$COPY_FILES" == "true" && -d "files" ]]; then
systemctl reenable /"${unit}" # to recreate all the dependency links.
done
cd ..
fi
fi
configuration/environments_scripts/build_server/setup-hudson-server.sh
+1 -1
View File
@@ -23,7 +23,7 @@ else
sudo chown root:root /root/secrets
sudo chmod 600 /root/secrets
. imageupgrade_functions.sh
if ! build_crontab_and_setup_files build_server sailing code; then
if ! build_crontab_and_setup_files build_server; then
exit 1
fi
# Make eu-west-1 the default region for any aws CLI interaction:
configuration/environments_scripts/build_server/users/root/crontab-update-authorized-keys
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-update-authorized-keys
configuration/environments_scripts/build_server/users/root/crontab-update-authorized-keys@USER_HOME_DIR=_root
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../crontabs/crontab-update-authorized-keys@USER_HOME_DIR
configuration/environments_scripts/central_mongo_setup/files/etc/mongodb/mongodb-archive.conf
+2 -2
View File
@@ -1,5 +1,5 @@
net:
port: 10202
port: 10201
bindIp: 0.0.0.0
storage:
dbPath: "/var/lib/mongodb/archive"
@@ -11,5 +11,5 @@ systemLog:
destination: file
path: "/var/log/mongodb/archive.log"
replication:
replSetName: "slow"
replSetName: "archive"
configuration/environments_scripts/central_mongo_setup/files/etc/mongodb/mongodb-archive-winddb.conf → configuration/environments_scripts/central_mongo_setup/files/etc/mongodb/mongodb-slow.conf
+4 -4
View File
@@ -1,15 +1,15 @@
net:
port: 10201
port: 10202
bindIp: 0.0.0.0
storage:
dbPath: "/var/lib/mongodb/archive-winddb"
dbPath: "/var/lib/mongodb/slow"
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 2
systemLog:
destination: file
path: "/var/log/mongodb/archive-winddb.log"
path: "/var/log/mongodb/slow.log"
replication:
replSetName: "archive"
replSetName: "slow"
configuration/environments_scripts/central_mongo_setup/files/etc/systemd/system/ephemeralvolume.service
-1
View File
@@ -1 +0,0 @@
../../../../../repo/etc/systemd/system/ephemeralvolume.service
configuration/environments_scripts/central_mongo_setup/files/etc/systemd/system/imageupgrade.service
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../../repo/etc/systemd/system/imageupgrade.service
configuration/environments_scripts/central_mongo_setup/files/etc/systemd/system/mongo-replica-set.service
-1
View File
@@ -1 +0,0 @@
../../../../../repo/etc/systemd/system/mongo-replica-set.service
configuration/environments_scripts/central_mongo_setup/files/etc/systemd/system/mongod-archive.service
+36
View File
@@ -0,0 +1,36 @@
[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=mongod
Group=mongod
ExecStart=/bin/mongod --config /etc/mongodb/mongod-archive.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
PIDFile=/var/run/mongodb/mongod-archive.pid
RuntimeDirectory=mongodb
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings
[Install]
WantedBy=multi-user.target
configuration/environments_scripts/central_mongo_setup/files/etc/systemd/system/mongod-hidden-live-replica.service
+36
View File
@@ -0,0 +1,36 @@
[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=mongod
Group=mongod
ExecStart=/bin/mongod --config /etc/mongodb/mongod-hidden-live-replica.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
PIDFile=/var/run/mongodb/mongod-hidden-live-replicaslow.pid
RuntimeDirectory=mongodb
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings
[Install]
WantedBy=multi-user.target
configuration/environments_scripts/central_mongo_setup/files/etc/systemd/system/mongod-slow.service
+36
View File
@@ -0,0 +1,36 @@
[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=mongod
Group=mongod
ExecStart=/bin/mongod --config /etc/mongodb/mongod-slow.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
PIDFile=/var/run/mongodb/mongod-slow.pid
RuntimeDirectory=mongodb
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings
[Install]
WantedBy=multi-user.target
configuration/environments_scripts/central_mongo_setup/files/etc/systemd/system/patch-mongo-replicaset-name-from-ec2-metadata.service
-1
View File
@@ -1 +0,0 @@
../../../../../repo/etc/systemd/system/patch-mongo-replicaset-name-from-ec2-metadata.service
configuration/environments_scripts/central_mongo_setup/files/usr/local/bin/add-as-replica
-1
View File
@@ -1 +0,0 @@
../../../../../repo/usr/local/bin/add-as-replica
configuration/environments_scripts/central_mongo_setup/files/var/lib/mongodb/archiveDb.sh → configuration/environments_scripts/central_mongo_setup/files/usr/local/bin/archiveDb.sh
+3 -2
View File
@@ -12,7 +12,8 @@ else
fi
DB=$1
DUMPBASEDIR=/mnt/mongodb-slow
DUMPBASEDIR=/var/lib/mongodb/slow/.dump
mkdir -p "${DUMPBASEDIR}"
if [ "$CONFIRMATION" = "" ]; then
echo -n "This will drop the database $DB in the archive DB before importing. Are you sure (y/n)? "
@@ -44,4 +45,4 @@ else
echo "Hashes of old and new are empty of differ: ${ORIGINAL_HASH} and ${ARCHIVE_HASH}"
exit 1
fi
rm -rf ${DUMPBASEDIR}"
configuration/environments_scripts/central_mongo_setup/files/usr/local/bin/ephemeralvolume
-1
View File
@@ -1 +0,0 @@
../../../../../repo/usr/local/bin/ephemeralvolume
configuration/environments_scripts/central_mongo_setup/files/usr/local/bin/imageupgrade
Executable
+22
View File
@@ -0,0 +1,22 @@
#!/bin/bash
# Upgrades the AWS EC2 MongoDB instance that this script is assumed to be executed on.
# The steps are as follows:
. imageupgrade_functions.sh
clean_mongo_pid() {
rm -f /var/run/mongodb/mongod.pid
}
LOGON_USER_HOME=/home/ec2-user
METADATA=$( get_ec2_user_data )
echo "Metadata: ${METADATA}"
if echo "${METADATA}" | grep -q "^image-upgrade$"; then
echo "Image upgrade..."
run_yum_update
build_crontab_and_setup_files central_mongo_setup
clean_startup_logs
clean_mongo_pid
finalize
fi
configuration/environments_scripts/central_mongo_setup/files/usr/local/bin/imageupgrade.sh
-24
View File
@@ -1,24 +0,0 @@
#!/bin/bash
# Upgrades the AWS EC2 MongoDB instance that this script is assumed to be executed on.
# The steps are as follows:
. imageupgrade_functions.sh
run_git_pull_root() {
echo "Pulling git to /root/code" >>/var/log/sailing.err
cd /root/code
git pull
}
clean_mongo_pid() {
rm -f /var/run/mongodb/mongod.pid
}
LOGON_USER_HOME=/home/ec2-user
run_yum_update
build_crontab_and_setup_files central_mongo_setup root code
clean_startup_logs
clean_mongo_pid
finalize
configuration/mongohash → configuration/environments_scripts/central_mongo_setup/files/usr/local/bin/mongohash
View File
File renamed without changes.
configuration/environments_scripts/central_mongo_setup/files/usr/local/bin/patch-mongo-replicaset-name-from-ec2-metadata
-1
View File
@@ -1 +0,0 @@
../../../../../repo/usr/local/bin/patch-mongo-replicaset-name-from-ec2-metadata
configuration/environments_scripts/central_mongo_setup/files/usr/local/bin/remove-as-replica
-1
View File
@@ -1 +0,0 @@
../../../../../repo/usr/local/bin/remove-as-replica
configuration/environments_scripts/central_mongo_setup/setup-central-mongo-instance.sh
Executable
+82
View File
@@ -0,0 +1,82 @@
#!/bin/bash
# Usage: Launch an Amazon EC2 instance from an Amazon Linux 2 AMI with
# 16GB of root partition size and the "Database and Messaging" security group
# using an SSH key for which you have a working private key available.
# Make sure to launch it in the same AZ as your current "MongoDB Central"
# instance, or you won't be able to attach the existing data volumes!
# Then, run this script on your local computer, using the external IP address
# of the instance you just launched in AWS as only argument. This will then
# prepare the instance as a central MongoDB instance.
# When the script is done you may log in to look around and check
# things. When done, stop the running "MongoDB Central" instance and detach
# its MongoDB data volumes as instructed below; attach them to the new instance,
# then run "mount -a" and start the MongoDB processes using the service units:
# - mongod-archive.service
# - mongod-hidden-live-replica.service
# - mongod-slow.service
# as per the instructions printed to the console at the end of this script.
# Name your instance "MongoDB Central" and assign the Route53 entry for
# dbserver.internal.sapsailing.com to its internal IP address.
if [ $# != 0 ]; then
SERVER=$1
scp "${0}" ec2-user@${SERVER}:
ssh -A ec2-user@${SERVER} ./$( basename "${0}" )
else
if ec2-metadata | grep -q instance-id; then
echo "Running on an AWS EC2 instance as user ${USER} / $(whoami), starting setup..."
# Install standard packages:
sudo yum -y update
sudo yum -y install nvme-cli chrony cronie cronie-anacron jq mailx
# Copy imageupgrade_function.sh
scp -o StrictHostKeyChecking=no -p root@sapsailing.com:/home/wiki/gitwiki/configuration/environments_scripts/repo/usr/local/bin/imageupgrade_functions.sh .
sudo mv imageupgrade_functions.sh /usr/local/bin
# build-crontab
. imageupgrade_functions.sh
build_crontab_and_setup_files central_mongo_setup
# obtain root SSH key from key vault:
setup_keys "central_mongo_setup"
# Create some swap space for the case mountnvmeswap hasn't created any
setup_swap 6000
# Install MongoDB 4.4 and configure as replica set "live"
setup_mongo_4_4
# Disable default mongod service unit derived from /etc/init.d/mongod:
sudo systemctl disable mongod.service
# Prepare for the MongoDB volume mounts:
sudo mkdir -p /var/lib/mongodb/hidden-live-replica
sudo mkdir -p /var/lib/mongodb/slow
sudo mkdir -p /var/lib/mongodb/archive
sudo chown -R mongod:mongod /var/lib/mongodb
sudo su - -c "cat << EOF >>/etc/fstab
UUID=d59156c5-7833-425a-9143-3a83b01f32b9 /var/lib/mongodb/hidden-live-replica xfs defaults,relatime 1 1
UUID=5586bc6e-d399-42e8-a0a0-bf2d65f8c703 /var/lib/mongodb/archive xfs defaults,relatime 1 1
UUID=0c73fde1-4719-405a-ab0e-bffd0aaf8d84 /var/lib/mongodb/slow xfs defaults,relatime 1 1
EOF
"
scp root@sapsailing.com:ssh-key-reader.token /tmp
sudo mv /tmp/ssh-key-reader.token /root
sudo chown root:root /root/ssh-key-reader.token
sudo chmod 600 /root/ssh-key-reader.token
setup_sshd_resilience
echo "Part 1 has completed. Now carry out the following steps in the AWS Console:"
echo " - stop the \"MongoDB Central\" instance"
echo " - detach the following volumes from it:"
echo " * Hidden MongoDB Live Replica encrypted"
echo " * MongoDB Archive winddb encrypted"
echo " * MongoDB Archive slow encrypted"
echo " - attach these volumes to the new instance"
echo " - issue the following commands on the new instance:"
echo " mount -a"
echo " chmod -R mongod:mongod /var/lib/mongodb"
echo " systemctl start mongod-archive.service"
echo " systemctl start mongod-hidden-live-replica.service"
echo " systemctl start mongod-slow.service"
echo " - assign the Route53 DNS record \"dbserver.internal.sapsailing.com\" to this"
echo " instance's internal IP address $( ec2-metadata --local-ipv4 | sed -e 's/^local-ipv4: //' )"
echo " - name the new instance \"MongoDB Central\" and terminate the old instance"
echo " - tag the instance with mongo-replica-sets = live:10203,archive:10201,slow:10202"
else
echo "Not running on an AWS instance; refusing to run setup!" >&2
echo "To prepare an instance running in AWS, provide its external IP as argument to this script." >&2
exit 2
fi
fi
configuration/environments_scripts/central_mongo_setup/users/root/crontab-update-authorized-keys
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-update-authorized-keys
configuration/environments_scripts/central_mongo_setup/users/root/crontab-update-authorized-keys@USER_HOME_DIR=_home_ec2-user
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../crontabs/crontab-update-authorized-keys@USER_HOME_DIR
configuration/environments_scripts/central_reverse_proxy/files/etc/awstats/awstats.www.sapsailing.com.conf
+6 -2
View File
@@ -48,8 +48,12 @@
# If there are several log files from load balancing servers :
# Example: "/pathtotools/logresolvemerge.pl *.log |"
#
# TODO the following line is for testing only; comment again when done testing and uncomment the next LogFile line
#LogFile="/var/log/httpd/access_log"
LogFile="/usr/share/awstats/tools/logresolvemerge.pl /var/log/httpd/access_log /var/log/old/access_log-???????? /var/log/old/access_log-????????.gz /var/log/old/*/elb-origin-access_log* /var/log/old/*/*/elb-origin-access_log* /var/log/old/*/access_log* /var/log/old/*/*/access_log* | /bin/grep -v HealthChecker |"
# TODO the following line is the "production" version; commented for debugging with a single file only
#LogFile="/usr/share/awstats/tools/logresolvemerge.pl /var/log/httpd/access_log /var/log/old/access_log-???????? /var/log/old/access_log-????????.gz /var/log/old/*/elb-origin-access_log* /var/log/old/*/*/elb-origin-access_log* /var/log/old/*/access_log* /var/log/old/*/*/access_log* | /bin/grep -v HealthChecker |"
# The following ocllectes only access logs for years 202X:
LogFile="/usr/share/awstats/tools/logresolvemerge.pl /var/log/httpd/access_log /var/log/old/access_log-202????? /var/log/old/access_log-202?????.gz /var/log/old/*/*/access_log-202?????.gz /var/log/old/*/*/access_log-202????? | /bin/grep --text -v HealthChecker |"
# Enter the log file type you want to analyze.
@@ -467,7 +471,7 @@ DefaultFile="index.php index.html"
# Example: "localhost REGEX[^.*\.localdomain$]"
# Default: ""
#
SkipHosts="127.0.0.1"
SkipHosts=""
# Do not include access from clients with a user agent that match following
configuration/environments_scripts/central_reverse_proxy/files/etc/dnf/vars/releasever
+1
View File
@@ -0,0 +1 @@
latest
configuration/environments_scripts/central_reverse_proxy/files/etc/tmux.conf
+1 -3
View File
@@ -1,6 +1,4 @@
# C-b is not acceptable -- Vim uses it
set-option -g prefix C-b
bind-key C-b last-window
# Start numbering at 0
set -g base-index 0
@@ -35,4 +33,4 @@ setw -g monitor-activity off
# Highlight active window
set-window-option -g window-status-current-style bg=red
set-window-option -g default-shell /bin/bash
set-window-option -g default-shell /bin/bash
configuration/environments_scripts/central_reverse_proxy/setup-central-reverse-proxy-part-2.sh
+1 -3
View File
@@ -17,8 +17,6 @@ if [[ "$#" -ne 2 ]]; then
fi
IP=$1
IMAGE_TYPE="$2"
GIT_COPY_USER="wiki"
RELATIVE_PATH_TO_GIT="gitwiki" # the relative path to the repo within the git_copy_user
TEMPORARY_HOME_COPY_LOCATION="/root/temporary_home_copy" # home nested within this.
ssh -A "root@${IP}" "bash -s" << EOF
sudo systemctl start crond.service
@@ -27,7 +25,7 @@ sudo systemctl start tmux-management-panel.service
cp -r "$TEMPORARY_HOME_COPY_LOCATION"/home /
rm -rf "$TEMPORARY_HOME_COPY_LOCATION"
# Localhost works here as we are logged on as root and are using ssh agent forwarding.
if ! build_crontab_and_setup_files -h localhost -f "${IMAGE_TYPE}" "${GIT_COPY_USER}" "${RELATIVE_PATH_TO_GIT}"; then # files have already been copied so -f is used.
if ! build_crontab_and_setup_files -h localhost -f "${IMAGE_TYPE}"; then # files have already been copied so -f is used.
exit 1
fi
setup_keys -p "${IMAGE_TYPE}"
configuration/environments_scripts/central_reverse_proxy/setup-central-reverse-proxy.sh
+2 -4
View File
@@ -44,8 +44,6 @@ IMAGEUPGRADE_FUNCTIONS_IP="$3" # can be a domain name, such as sapsailing.com
IMAGEUPGRADE_FUNCTIONS_PATH_ON_INSTANCE_TO_GIT="$4"
IMAGE_TYPE="central_reverse_proxy"
HTTP_LOGROTATE_ABSOLUTE=/etc/logrotate.d/httpd
GIT_COPY_USER="wiki"
RELATIVE_PATH_TO_GIT="gitwiki" # the relative path to the repo within the git_copy_user
# This authorized keys copying is essential as we rely on the ability to log into the root user.
ssh -A "ec2-user@${IP}" "bash -s" << FIRSTEOF
# Correct authorized keys. May not be necessary if update_authorized_keys is running.
@@ -55,7 +53,7 @@ FIRSTEOF
ssh -A "root@${IP}" "bash -s" << SECONDEOF
# update instance
yum update -y
yum install -y httpd mod_proxy_html tmux nfs-utils git whois jq cronie iptables mailx nmap icu mariadb105-server tree #icu is a c/c++ library that provides unicode and globalisation support for software development.
yum install -y python httpd mod_proxy_html tmux nfs-utils git whois jq cronie iptables mailx nmap icu mariadb105-server tree #icu is a c/c++ library that provides unicode and globalisation support for software development.
# docker setup
yum install -y docker
sudo curl --silent -L "https://github.com/docker/compose/releases/download/v2.26.1/docker-compose-\$(uname -s)-\$(uname -m)" -o /usr/local/bin/docker-compose
@@ -128,7 +126,7 @@ scp -o StrictHostKeyChecking=no -p root@"$IMAGEUPGRADE_FUNCTIONS_IP":"$IMAGEUPGR
. imageupgrade_functions.sh
setup_cloud_cfg_and_root_login
# setup files
if ! build_crontab_and_setup_files -c -n "${IMAGE_TYPE}" "${GIT_COPY_USER}" "${RELATIVE_PATH_TO_GIT}"; then # -c & -n mean only files are copied over.
if ! build_crontab_and_setup_files -c -n "${IMAGE_TYPE}"; then # -c & -n mean only files are copied over.
exit 1
fi
setup_swap 5000
configuration/environments_scripts/central_reverse_proxy/users/root/crontab-update-authorized-keys
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-update-authorized-keys
configuration/environments_scripts/central_reverse_proxy/users/root/crontab-update-authorized-keys-httpdConf-user
-1
View File
@@ -1 +0,0 @@
* * * * * export PATH=/bin:/usr/bin:/usr/local/bin; sleep $(( $RANDOM * 60 / 32768 )); update_authorized_keys_for_landscape_managers_if_changed $( cat PATH_OF_HOME_DIR_TO_REPLACE/ssh-key-reader.token ) https://security-service.sapsailing.com /home/httpdConf # 2>&1 >>/var/log/sailing.err
configuration/environments_scripts/central_reverse_proxy/users/root/crontab-update-authorized-keys@USER_HOME_DIR=_home_httpdConf
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../crontabs/crontab-update-authorized-keys@USER_HOME_DIR
configuration/environments_scripts/central_reverse_proxy/users/root/crontab-update-authorized-keys@USER_HOME_DIR=_root
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../crontabs/crontab-update-authorized-keys@USER_HOME_DIR
configuration/environments_scripts/central_reverse_proxy/users/wiki/crontab-download-new-archived-trac-trac-events
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-download-new-archived-trac-trac-events
configuration/environments_scripts/central_reverse_proxy/users/wiki/crontab-download-new-archived-trac-trac-events
+1
View File
@@ -0,0 +1 @@
15 12 * * * export PATH=/bin:/usr/bin:/usr/local/bin; downloadNewArchivedTracTracEvents.sh /home/trac/static/TracTracTracks "/home/wiki/gitwiki" >/home/wiki/downloadNewArchivedTracTracEvents.out 2>/home/wiki/downloadNewArchivedTracTracEvents.err
configuration/environments_scripts/central_reverse_proxy/users/wiki/crontab-syncgit
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-syncgit
configuration/environments_scripts/central_reverse_proxy/users/wiki/crontab-syncgit
+1
View File
@@ -0,0 +1 @@
*/10 * * * * export PATH=/bin:/usr/bin:/usr/local/bin; syncgit /home/wiki/gitwiki
configuration/environments_scripts/hudson_slave/README
+2
View File
@@ -3,3 +3,5 @@ Deploy the imageupgrade script to /usr/local/bin,
furthermore the ../imageupgrade_functions.sh has to go to /usr/local/bin, best as symbolic link to files in git.
See the script in wiki/howto/development/ci.md that shows how to set up such an instance from scratch.
With us moving our builds to Github, maintaining Hudson build slaves is DEPRECATED.
configuration/environments_scripts/mongo_instance_setup/README
-16
View File
@@ -1,16 +0,0 @@
Deploy the .mount and .service units to /etc/systemd/system.
Deploy the ephemeralvolume and the patch-mongo-replicaset-name-from-ec2-metadata script to /usr/local/bin,
furthermore the ../imageupgrade_functions.sh has to go to /usr/local/bin.
Deploy mongod.conf to /etc and make sure that /root has a+r and a+x permissions because
otherwise the mongod user won't be able to read through the symbolic link
Link mongodb to /etc/logrotate.d
Utilise the build-crontab-and-cp-files script or the build_crontab_and_setup_files (in imageupgrade_functions.sh)
to create the crontab file and install it in the root user.
Run with optional EC2 user detail, e.g., as follows:
REPLICA_SET_NAME=archive
REPLICA_SET_PRIMARY=dbserver.internal.sapsailing.com:10201
This will automatically patch /etc/mongod.conf such that the replSetName property
is set to the value of REPLICA_SET_NAME. Then, the instance will be added to
the REPLICA_SET_PRIMARY's replica set.
configuration/environments_scripts/mongo_instance_setup/files/etc/mongod.conf
-2
View File
@@ -16,7 +16,6 @@ storage:
enabled: true
directoryPerDB: true
# engine:
# mmapv1:
# wiredTiger:
# how the process runs
@@ -29,7 +28,6 @@ processManagement:
net:
port: 27017
# bindIp: 127.0.0.1 # Listen to local interface only, comment to listen on all interfaces.
# bindIp: 172.31.33.146
bindIp: 0.0.0.0
#security:
configuration/environments_scripts/mongo_instance_setup/files/etc/systemd/system/imageupgrade.service
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../../repo/etc/systemd/system/imageupgrade.service
configuration/environments_scripts/mongo_instance_setup/files/etc/systemd/system/mongo-replica-set.service
-1
View File
@@ -1 +0,0 @@
../../../../../repo/etc/systemd/system/mongo-replica-set.service
configuration/environments_scripts/mongo_instance_setup/files/etc/systemd/system/mongo-replica-set.service
+16
View File
@@ -0,0 +1,16 @@
[Unit]
Description=If REPLICA_SET_NAME EC2 user data is provided, add this node to the replica set of REPLICA_SET_PRIMARY
Requires=mongod.service
After=mongod.service
Requires=cloud-init.service
After=cloud-init.service
[Install]
WantedBy=multi-user.target
[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/usr/local/bin/add-as-replica
ExecStop=/usr/local/bin/remove-as-replica
TimeoutStopSec=120s
configuration/environments_scripts/mongo_instance_setup/files/etc/systemd/system/patch-mongo-replicaset-name-from-ec2-metadata.service
-1
View File
@@ -1 +0,0 @@
../../../../../repo/etc/systemd/system/patch-mongo-replicaset-name-from-ec2-metadata.service
configuration/environments_scripts/mongo_instance_setup/files/etc/systemd/system/patch-mongo-replicaset-name-from-ec2-metadata.service
+15
View File
@@ -0,0 +1,15 @@
[Unit]
Description=Check EC2 metadata for MongoDB Replica Set Name and patch /etc/mongod.conf accordingly
Requires=ephemeralvolume.service
After=ephemeralvolume.service
Requires=cloud-init.service
After=cloud-init.service
Before=mongod.service
[Install]
RequiredBy=mongod.service
[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/usr/local/bin/patch-mongo-replicaset-name-from-ec2-metadata
configuration/environments_scripts/mongo_instance_setup/files/usr/local/bin/imageupgrade
Executable
+22
View File
@@ -0,0 +1,22 @@
#!/bin/bash
# Upgrades the AWS EC2 MongoDB instance that this script is assumed to be executed on.
# The steps are as follows:
. imageupgrade_functions.sh
clean_mongo_pid() {
rm -f /var/run/mongodb/mongod.pid
}
LOGON_USER_HOME=/home/ec2-user
METADATA=$( get_ec2_user_data )
echo "Metadata: ${METADATA}"
if echo "${METADATA}" | grep -q "^image-upgrade$"; then
echo "Image upgrade..."
run_yum_update
build_crontab_and_setup_files mongo_instance_setup
clean_startup_logs
clean_mongo_pid
finalize
fi
configuration/environments_scripts/mongo_instance_setup/files/usr/local/bin/imageupgrade.sh
-24
View File
@@ -1,24 +0,0 @@
#!/bin/bash
# Upgrades the AWS EC2 MongoDB instance that this script is assumed to be executed on.
# The steps are as follows:
. imageupgrade_functions.sh
run_git_pull_root() {
echo "Pulling git to /root/code" >>/var/log/sailing.err
cd /root/code
git pull
}
clean_mongo_pid() {
rm -f /var/run/mongodb/mongod.pid
}
LOGON_USER_HOME=/home/ec2-user
run_yum_update
build_crontab_and_setup_files mongo_instance_setup root code
clean_startup_logs
clean_mongo_pid
finalize
configuration/environments_scripts/mongo_instance_setup/setup-mongo-instance.sh
Executable
+44
View File
@@ -0,0 +1,44 @@
#!/bin/bash
# Usage: Launch an Amazon EC2 instance from an Amazon Linux 2 AMI with
# 8GB of root partition size and the "MongoDB Replica Set" security group
# using an SSH key for which you have a working private key available.
# Then, run this script on your local computer, using the external IP address
# of the instance you just launched in AWS as only argument. This will then
# turn the instance into a MongoDB replica set node.
# When the script is done you may log in to look around and check
# things. When done, shut down the instance (Stop, not Terminate) and create
# an image off of it, naming it, e.g., "MongoDB Live Replica Set NVMe 2.0" and
# also tagging its root volume snapshot as, e.g., "MongoDB Live Replica Set NVMe 2.0 (Root)".
# If you want to use the resulting image in production, also tag it with
# tag key "image-type" and tag value "mongodb-server".
if [ $# != 0 ]; then
SERVER=$1
scp "${0}" ec2-user@${SERVER}:
ssh -A ec2-user@${SERVER} ./$( basename "${0}" )
else
if ec2-metadata | grep -q instance-id; then
echo "Running on an AWS EC2 instance as user ${USER} / $(whoami), starting setup..."
# Install standard packages:
sudo yum -y update
sudo yum -y install nvme-cli chrony cronie cronie-anacron jq mailx
# Copy imageupgrade_function.sh
scp -o StrictHostKeyChecking=no -p root@sapsailing.com:/home/wiki/gitwiki/configuration/environments_scripts/repo/usr/local/bin/imageupgrade_functions.sh .
sudo mv imageupgrade_functions.sh /usr/local/bin
# build-crontab
. imageupgrade_functions.sh
build_crontab_and_setup_files mongo_instance_setup
# obtain root SSH key from key vault:
setup_keys "mongo_instance_setup"
# Install MongoDB 4.4 and configure as replica set "live"
setup_mongo_4_4
scp root@sapsailing.com:ssh-key-reader.token /tmp
sudo mv /tmp/ssh-key-reader.token /root
sudo chown root:root /root/ssh-key-reader.token
sudo chmod 600 /root/ssh-key-reader.token
setup_sshd_resilience
else
echo "Not running on an AWS instance; refusing to run setup!" >&2
echo "To prepare an instance running in AWS, provide its external IP as argument to this script." >&2
exit 2
fi
fi
configuration/environments_scripts/mongo_instance_setup/users/root/crontab-update-authorized-keys
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-update-authorized-keys
configuration/environments_scripts/mongo_instance_setup/users/root/crontab-update-authorized-keys-ec2-user
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-update-authorized-keys-ec2-user
configuration/environments_scripts/mongo_instance_setup/users/root/crontab-update-authorized-keys@USER_HOME_DIR=_home_ec2-user
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../crontabs/crontab-update-authorized-keys@USER_HOME_DIR
configuration/environments_scripts/mysql_instance_setup/setup-mysql-server.sh
+1 -1
View File
@@ -47,7 +47,7 @@ else
sudo systemctl enable crond.service
sudo systemctl start crond.service
. imageupgrade_functions.sh
if ! build_crontab_and_setup_files mysql_instance_setup ec2-user no_local_copy; then
if ! build_crontab_and_setup_files mysql_instance_setup; then
exit 1
fi
setup_sshd_resilience
configuration/environments_scripts/mysql_instance_setup/users/ec2-user/crontab-update-authorized-keys
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-update-authorized-keys
configuration/environments_scripts/mysql_instance_setup/users/ec2-user/crontab-update-authorized-keys@USER_HOME_DIR=_home_ec2-user
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../crontabs/crontab-update-authorized-keys@USER_HOME_DIR
configuration/environments_scripts/rabbitmq_instance_setup/setup-rabbitmq-server.sh
+1 -1
View File
@@ -23,7 +23,7 @@ else
scp -o StrictHostKeyChecking=false -r root@sapsailing.com:/home/wiki/gitwiki/configuration/environments_scripts/repo/usr/local/bin/imageupgrade_functions.sh /home/admin
sudo mv imageupgrade_functions.sh /usr/local/bin
. imageupgrade_functions.sh
if ! build_crontab_and_setup_files 'rabbitmq_instance_setup' admin environments_scripts; then
if ! build_crontab_and_setup_files 'rabbitmq_instance_setup'; then
exit 1
fi
setup_sshd_resilience
configuration/environments_scripts/rabbitmq_instance_setup/users/admin/crontab-update-authorized-keys
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-update-authorized-keys
configuration/environments_scripts/repo/etc/systemd/system/mongo-replica-set.service
-16
View File
@@ -1,16 +0,0 @@
[Unit]
Description=If REPLICA_SET_NAME EC2 user data is provided, add this node to the replica set of REPLICA_SET_PRIMARY
Requires=mongod.service
After=mongod.service
Requires=cloud-init.service
After=cloud-init.service
[Install]
WantedBy=multi-user.target
[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/usr/local/bin/add-as-replica
ExecStop=/usr/local/bin/remove-as-replica
TimeoutStopSec=120s
configuration/environments_scripts/repo/etc/systemd/system/patch-mongo-replicaset-name-from-ec2-metadata.service
-15
View File
@@ -1,15 +0,0 @@
[Unit]
Description=Check EC2 metadata for MongoDB Replica Set Name and patch /etc/mongod.conf accordingly
Requires=ephemeralvolume.service
After=ephemeralvolume.service
Requires=cloud-init.service
After=cloud-init.service
Before=mongod.service
[Install]
RequiredBy=mongod.service
[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/usr/local/bin/patch-mongo-replicaset-name-from-ec2-metadata
configuration/environments_scripts/repo/usr/local/bin/ephemeralvolume
+3 -3
View File
@@ -6,8 +6,8 @@
METADATA=$( /bin/ec2-metadata -d | sed -e 's/^user-data: //' )
echo "Metadata: ${METADATA}"
if echo "${METADATA}" | grep -q "^image-upgrade$"; then
echo "Image upgrade; not trying to mount/format ephemeral volume; calling imageupgrade.sh instead..."
imageupgrade.sh
echo "Image upgrade; not trying to mount/format ephemeral volume"
exit 0
else
echo "No image upgrade; looking for ephemeral volume and trying to format with xfs..."
. imageupgrade_functions.sh
@@ -29,4 +29,4 @@ else
chown mongod:mongod /var/lib/mongo # Added for robustness
fi
fi
fi
fi
configuration/environments_scripts/repo/usr/local/bin/imageupgrade_functions.sh
+73 -29
View File
@@ -7,59 +7,64 @@ REBOOT_INDICATOR=/var/lib/sailing/is-rebooted
LOGON_USER_HOME=/root
run_yum_update() {
echo "Updating packages using yum" >>/var/log/sailing.err
echo "Updating packages using yum"
yum -y update
}
run_dnf_upgrade() {
echo "Upgrading using dnf"
dnf -y upgrade --releasever=latest
}
run_apt_update_upgrade() {
echo "Updating packages using apt" >>/var/log/sailing.err
echo "Updating packages using apt"
apt-get -y update; apt-get -y upgrade
apt-get -y install linux-image-cloud-amd64
apt-get -y autoremove
}
run_git_pull() {
echo "Pulling git to /home/sailing/code" >>/var/log/sailing.err
echo "Pulling git to /home/sailing/code"
su - sailing -c "cd code; git pull"
}
download_and_install_latest_sap_jvm_8() {
echo "Downloading and installing latest SAP JVM 8 to /opt/sapjvm_8" >>/var/log/sailing.err
echo "Downloading and installing latest SAP JVM 8 to /opt/sapjvm_8"
vmpath=$( curl -s --cookie eula_3_2_agreed=tools.hana.ondemand.com/developer-license-3_2.txt https://tools.hana.ondemand.com | grep additional/sapjvm-8\..*-linux-x64.zip | head -1 | sed -e 's/^.*a href="\(additional\/sapjvm-8\..*-linux-x64\.zip\)".*/\1/' )
if [ -n "${vmpath}" ]; then
echo "Found VM version ${vmpath}; upgrading installation at /opt/sapjvm_8" >>/var/log/sailing.err
echo "Found VM version ${vmpath}; upgrading installation at /opt/sapjvm_8"
if [ -z "${TMP}" ]; then
TMP=/tmp
fi
echo "Downloading SAP JVM 8 as ZIP file to ${TMP}/sapjvm8-linux-x64.zip" >>/var/log/sailing.err
curl --cookie eula_3_2_agreed=tools.hana.ondemand.com/developer-license-3_2.txt "https://tools.hana.ondemand.com/${vmpath}" > ${TMP}/sapjvm8-linux-x64.zip 2>>/var/log/sailing.err
echo "Downloading SAP JVM 8 as ZIP file to ${TMP}/sapjvm8-linux-x64.zip"
curl --cookie eula_3_2_agreed=tools.hana.ondemand.com/developer-license-3_2.txt "https://tools.hana.ondemand.com/${vmpath}" > ${TMP}/sapjvm8-linux-x64.zip
cd /opt
rm -rf sapjvm_8
if [ -f SIGNATURE.SMF ]; then
rm -f SIGNATURE.SMF
fi
unzip ${TMP}/sapjvm8-linux-x64.zip >>/var/log/sailing.err
unzip ${TMP}/sapjvm8-linux-x64.zip
rm -f ${TMP}/sapjvm8-linux-x64.zip
rm -f SIGNATURE.SMF
else
echo "Did not find SAP JVM 8 at tools.hana.ondemand.com; not trying to upgrade" >>/var/log/sailing.err
echo "Did not find SAP JVM 8 at tools.hana.ondemand.com; not trying to upgrade"
fi
}
clean_logrotate_target() {
echo "Clearing logrorate-targets" >>/var/log/sailing.err
echo "Clearing logrorate-targets"
rm -rf /var/log/logrotate-target/*
}
clean_httpd_logs() {
echo "Clearing httpd logs" >>/var/log/sailing.err
echo "Clearing httpd logs"
service httpd stop
rm -rf /var/log/httpd/*
rm -f /etc/httpd/conf.d/001-internals.conf
}
clean_startup_logs() {
echo "Clearing bootstrap logs" >>/var/log/sailing.err
echo "Clearing bootstrap logs"
rm -f /var/log/sailing*
# Ensure that upon the next boot the reboot indicator is not present, indicating that it's the first boot
rm "${REBOOT_INDICATOR}"
@@ -69,15 +74,6 @@ clean_servers_dir() {
rm -rf /home/sailing/servers/*
}
#DEPRECATED
update_root_crontab() {
# The following assumes that /root/crontab is a symbolic link to /home/sailing/code/configuration/crontabs/<the crontab appropriate
# to the environment or user>
# which has previously been updated by a git pull:
cd /root
crontab crontab
}
build_crontab_and_setup_files() {
if [[ "$#" -eq 0 ]]; then
echo "Number of arguments is invalid. Please use the options and arguments as follows."
@@ -89,8 +85,6 @@ build_crontab_and_setup_files() {
echo " -f means no files are copied over, which is useful if you have already copied files accross or don't want to override existing files"
echo "Then there are the arguments, where the order matters:"
echo " ENVIRONMENT_TYPE - the directory name in environments_scripts which will be used."
echo " USER_WITH_COPY_OF_REPO - a user which will exist on the environment type, which has a checked out copy of the git workspace."
echo " RELATIVE_PATH_OF_GIT_DIR_WITHIN_USER - the relative path within the USER_WITH_COPY_OF_REPO to get to the git workspace."
else
TEMP=$(getopt -o fnch: -n 'options checker' -- "$@")
[[ "$?" -eq 0 ]] || return 2
@@ -130,20 +124,54 @@ build_crontab_and_setup_files() {
# user of the scp command (as seen in the second command below).
scp -o StrictHostKeyChecking=no -pr wiki@"$HOSTNAME":~/gitwiki/configuration/environments_scripts/* "${TEMP_ENVIRONMENTS_SCRIPTS}"
[[ "$?" -eq 0 ]] || scp -o StrictHostKeyChecking=no -pr root@"$HOSTNAME":/home/wiki/gitwiki/configuration/environments_scripts/* "${TEMP_ENVIRONMENTS_SCRIPTS}" # For initial setup as not all landscape managers have direct wiki access.
sudo chown root:root "$TEMP_ENVIRONMENTS_SCRIPTS"
sudo chown root:root "${TEMP_ENVIRONMENTS_SCRIPTS}"
sudo chmod 777 "${TEMP_ENVIRONMENTS_SCRIPTS}"
cd "${TEMP_ENVIRONMENTS_SCRIPTS}"
# Add all args to array, otherwise, if PASS_OPTIONS is empty, and we also pass $@ then argument $1 is in fact null, which would cause errors.
for option in "$@"; do
PASS_OPTIONS+=( "$option")
PASS_OPTIONS+=( "$option" )
done
if ! sudo ./build-crontab-and-cp-files "${PASS_OPTIONS[@]}"; then
if ! sudo ./build-crontab-and-cp-files "${PASS_OPTIONS[@]}"; then
return 1
fi
cd ..
rm -rf "$TEMP_ENVIRONMENTS_SCRIPTS"
sudo rm -rf "$TEMP_ENVIRONMENTS_SCRIPTS"
fi
}
# Assumes a filename following the pattern "crontab*[@{param}={value} ...]"
# It prints an "sed" command on the standard output that callers can use to
# obtain a piping command through which to send the file's contents to achieve
# the text pattern replacements requested by the @-parts in the file/link
# name.
# If a value shall contain the @ literal, it needs to be doubled, as, e.g., in
# crontab-some-mail-thing@EMAIL_ADDRESS=john.doe@@example.com
# Multiple replacements may be requested by adding more @-phrases:
# crontab-something@A=x@B=y
# Example usage:
# echo "The user USER has e-mail address EMAIL_ADDRESS and home directory HOME_DIR" | eval $( get_replacements_from_crontab_filename_as_sed_command crontab-humba@USER=uhl@EMAIL_ADDRESS=axel.uhl@@sap.com@HOME_DIR=_home_axel__uhl_ )
# will output
# The user uhl has e-mail address axel.uhl@sap.com and home directory /home/axel_uhl/
get_replacements_from_crontab_filename_as_sed_command() {
LINK_NAME="${1}"
AT_SIGN_REPLACEMENT=`mktemp -u XXXXXXXXXXXXXXXX`
UNDERSCORE_REPLACEMENT=`mktemp -u XXXXXXXXXXXXXXXX`
LINK_NAME_WITH_UNDERSCORES_REPLACED=$( echo "${LINK_NAME}" | sed -e 's|__|'${UNDERSCORE_REPLACEMENT}'|g' )
LINK_NAME_WITH_AT_LITERALS_REPLACED=$( echo "${LINK_NAME_WITH_UNDERSCORES_REPLACED}" | sed -e 's|@@|'${AT_SIGN_REPLACEMENT}'|g' )
IFS='@' read -r -a BASENAME_AND_SUBSTITUTIONS <<<"${LINK_NAME_WITH_AT_LITERALS_REPLACED}"
# Now, BASENAME_AND_SUBSTITUTIONS[0] is the link's base name, 1..n are the replacements
SED_COMMAND="sed -e 's| | |'" # start with a no-op so that in case of no replacements we still have a valid sed command
for REPLACEMENT in ${BASENAME_AND_SUBSTITUTIONS[@]:1}; do
IFS='=' read -r -a PARAM_AND_VALUE <<<"${REPLACEMENT}"
PARAM="${PARAM_AND_VALUE[0]}"
VALUE_WITH_SLASHES=$( echo "${PARAM_AND_VALUE[1]}" | sed -e 's|_|/|g' )
VALUE_WITH_UNDERSCORES=$( echo "${VALUE_WITH_SLASHES}" | sed -e 's|'${UNDERSCORE_REPLACEMENT}'|_|g' )
VALUE=$( echo "${VALUE_WITH_UNDERSCORES}" | sed -e 's|'${AT_SIGN_REPLACEMENT}'|@|g' )
SED_COMMAND="${SED_COMMAND} -e 's|${PARAM}|${VALUE}|g'"
done
echo "${SED_COMMAND}"
}
setup_keys() {
# Installs the necessary aws and ssh keys for a given environment type, by copying from the key vault.
# $1: Environment type.
@@ -248,7 +276,7 @@ __setup_keys_using_local_copy() {
}
clean_root_ssh_dir_and_tmp() {
echo "Cleaning up ${LOGON_USER_HOME}/.ssh" >>/var/log/sailing.err
echo "Cleaning up ${LOGON_USER_HOME}/.ssh"
rm -rf ${LOGON_USER_HOME}/.ssh/authorized_keys
rm -rf ${LOGON_USER_HOME}/.ssh/known_hosts
rm -f /var/run/last_change_aws_landscape_managers_ssh_keys*
@@ -356,7 +384,8 @@ myorigin =${subdomain_of_sender_address}.sapsailing.com
setup_sshd_resilience() {
sudo su -c "echo 'ClientAliveInterval 3
ClientAliveCountMax 3
GatewayPorts yes' >> /etc/ssh/sshd_config && systemctl reload sshd.service"
GatewayPorts yes
MaxStartups 100' >> /etc/ssh/sshd_config && systemctl reload sshd.service"
}
identify_suitable_partition_for_ephemeral_volume() {
@@ -422,3 +451,18 @@ setup_swap() {
sudo swapon -a
popd
}
setup_mongo_4_4() {
# Install MongoDB 4.4 and configure as replica set "live"
sudo su - -c "cat << EOF >/etc/yum.repos.d/mongodb-org.4.4.repo
[mongodb-org-4.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2/mongodb-org/4.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc
EOF
"
sudo yum -y update
sudo yum -y install mongodb-org-server mongodb-org-shell mongodb-org-tools
}
configuration/environments_scripts/reverse_proxy/files/etc/dnf/vars/releasever
+1
View File
@@ -0,0 +1 @@
latest
configuration/environments_scripts/reverse_proxy/files/usr/local/bin/imageupgrade
+2 -7
View File
@@ -1,8 +1,5 @@
#!/bin/bash
# Script to deploy on an instance that has an ephemeral volume as /dev/nvme0n1 (adjust env var PARTITION if different)
# Ensures the partition is xfs-formatted, any existing partition contents will be overwritten if formatted otherwise.
# An existing xfs partition will be left alone.
. imageupgrade_functions.sh
get_ec2_user_data() {
@@ -13,12 +10,10 @@ METADATA=$( get_ec2_user_data )
echo "Metadata: ${METADATA}"
if echo "${METADATA}" | grep -q "^image-upgrade$"; then
echo "Image upgrade..."
GIT_COPY_USER="trac"
RELATIVE_PATH_TO_GIT="gitcopy"
LOGON_USER_HOME="ec2-user"
ENVIRONMENT_TYPE="reverse_proxy"
run_yum_update
build_crontab_and_setup_files "${ENVIRONMENT_TYPE}" "${GIT_COPY_USER}" "${RELATIVE_PATH_TO_GIT}"
run_dnf_upgrade
build_crontab_and_setup_files "${ENVIRONMENT_TYPE}"
finalize
fi
configuration/environments_scripts/reverse_proxy/setup-disposable-reverse-proxy.sh
+4 -6
View File
@@ -1,6 +1,6 @@
#!/bin/bash
# Setup script for Amazon Linux 2. May need to update macro definitions for the archive IP.
# Setup script for Amazon Linux 2023. May need to update macro definitions for the archive IP.
# Parameter 1 is the IP and parameter 2 is the bearer token to be installed in the root home dir.
# Ensure that the security for requesting the metadata uses IMDSv1
if [[ "$#" -ne 2 ]]; then
@@ -11,8 +11,6 @@ IP=$1
BEARER_TOKEN=$2
IMAGE_TYPE="reverse_proxy"
HTTP_LOGROTATE_ABSOLUTE=/etc/logrotate.d/httpd
GIT_COPY_USER="trac"
RELATIVE_PATH_TO_GIT="gitcopy" # the relative path to the repo within the git_copy_user
ssh -A "ec2-user@${IP}" "bash -s" << FIRSTEOF
# Correct authorized keys. May not be necessary if update_authorized_keys is running.
sudo su - -c "cat ~ec2-user/.ssh/authorized_keys > /root/.ssh/authorized_keys"
@@ -24,8 +22,8 @@ mkdir /var/log/old
echo "logfiles.internal.sapsailing.com:/var/log/old /var/log/old nfs tcp,intr,timeo=100,retry=0" >> /etc/fstab
mount -a
# update instance
yum update -y
yum install -y httpd mod_proxy_html tmux nfs-utils git whois jq cronie iptables nmap
dnf upgrade -y --releasever=latest
dnf install -y httpd mod_proxy_html tmux nfs-utils git whois jq cronie iptables nmap
sudo systemctl enable crond.service
# setup other users and crontabs to keep repo updated
cd /root
@@ -35,7 +33,7 @@ scp -o StrictHostKeyChecking=no -p "root@sapsailing.com:/home/wiki/gitwiki/confi
setup_keys "${IMAGE_TYPE}"
setup_cloud_cfg_and_root_login
# setup files and crontab for the required users, both dependent on the environment type.
build_crontab_and_setup_files "${IMAGE_TYPE}" "${GIT_COPY_USER}" "${RELATIVE_PATH_TO_GIT}"
build_crontab_and_setup_files "${IMAGE_TYPE}"
# setup mail
setup_mail_sending
# setup sshd config
configuration/environments_scripts/reverse_proxy/users/root/crontab-update-authorized-keys
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-update-authorized-keys
configuration/environments_scripts/reverse_proxy/users/root/crontab-update-authorized-keys@USER_HOME_DIR=_root
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../crontabs/crontab-update-authorized-keys@USER_HOME_DIR
configuration/environments_scripts/sailing_server/files/etc/logrotate.d/mongodb
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../repo/etc/logrotate.d/mongodb
configuration/environments_scripts/sailing_server/files/etc/security/limits.d/filedescriptorlimits.conf
+2
View File
@@ -0,0 +1,2 @@
* hard nofile 1000000
* soft nofile 1000000
configuration/environments_scripts/sailing_server/files/etc/systemd/system/imageupgrade.service
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../../repo/etc/systemd/system/imageupgrade.service
configuration/environments_scripts/sailing_server/files/usr/local/bin/imageupgrade
Executable
+20
View File
@@ -0,0 +1,20 @@
#!/bin/bash
# Upgrades the AWS EC2 instance that this script is assumed to be executed on.
# The steps are as follows:
. `dirname $0`/imageupgrade_functions.sh
METADATA=$( get_ec2_user_data )
echo "Metadata: ${METADATA}"
if echo "${METADATA}" | grep -q "^image-upgrade$"; then
echo "Image upgrade..."
run_yum_update
download_and_install_latest_sap_jvm_8
clean_logrotate_target
clean_httpd_logs
clean_servers_dir
clean_startup_logs
build_crontab_and_setup_files sailing_server
finalize
fi
configuration/environments_scripts/sailing_server/files/usr/local/bin/imageupgrade.sh
-15
View File
@@ -1,15 +0,0 @@
#!/bin/bash
# Upgrades the AWS EC2 instance that this script is assumed to be executed on.
# The steps are as follows:
. `dirname $0`/imageupgrade_functions.sh
run_yum_update
download_and_install_latest_sap_jvm_8
clean_logrotate_target
clean_httpd_logs
clean_servers_dir
clean_startup_logs
build_crontab_and_setup_files sailing_server sailing code
finalize
configuration/environments_scripts/sailing_server/files/usr/local/bin/refreshInstance.sh
+1 -1
View File
@@ -153,7 +153,7 @@ install_environment ()
load_from_release_file ()
{
if [[ ${INSTALL_FROM_RELEASE} == "" ]]; then
INSTALL_FROM_RELEASE="$(wget -O - https://releases.sapsailing.com/ 2>/dev/null | grep build- | tail -1 | sed -e 's/^.*\(build-[0-9]*\).*$/\1/')"
INSTALL_FROM_RELEASE="$(wget -O - https://releases.sapsailing.com/ 2>/dev/null | grep main- | tail -1 | sed -e 's/^.*\(main-[0-9]*\).*$/\1/')"
echo "You didn't provide a release. Defaulting to latest master build https://releases.sapsailing.com/$INSTALL_FROM_RELEASE"
# Alternatively, to download a release from Github, here is how the latest Java8/main-release can be obtained:
# curl -L -H 'Authorization: Bearer ***' https://api.github.com/repos/SAP/sailing-analytics/releases 2>/dev/null | jq -r 'sort_by(.created_at) | reverse | map(select(.name | startswith("main-")))[0].assets[] | select(.content_type=="application/x-tar").id'
configuration/environments_scripts/sailing_server/files/usr/local/bin/sailing
+19 -27
View File
@@ -8,8 +8,9 @@
# Source function library. This alters the PATH variable.
. /etc/init.d/functions
if [ -f /etc/init.d/functions ]; then
. /etc/init.d/functions
fi
RETVAL=0
SERVERS_DIR=/home/sailing/servers
@@ -26,36 +27,34 @@ REBOOT_INDICATOR=/var/lib/sailing/is-rebooted
mkdir --parents $( dirname ${REBOOT_INDICATOR} )
SSH_KEY_READER_BEARER_TOKEN=/root/ssh-key-reader.token
echo "Executing with $1 at `date`" >>/var/log/sailing.err
echo "Executing with $1 at `date`"
start_servers() {
/usr/local/bin/update_authorized_keys_for_landscape_managers $( cat ${SSH_KEY_READER_BEARER_TOKEN} ) https://security-service.sapsailing.com /root 2>&1 >>/var/log/sailing.err
chown root /usr/local/bin/cp_root_mail_properties
chgrp root /usr/local/bin/cp_root_mail_properties
/usr/local/bin/update_authorized_keys_for_landscape_managers $( cat ${SSH_KEY_READER_BEARER_TOKEN} ) https://security-service.sapsailing.com /root
chown root:root /usr/local/bin/cp_root_mail_properties
chmod 755 /usr/local/bin/cp_root_mail_properties
if which $EC2_METADATA_CMD && $EC2_METADATA_CMD -d | sed "s/user-data\: //g" | grep "^image-upgrade$"; then
echo "Found image-upgrade in EC2 user data; upgrading image, then probably shutting down for AMI creation depending on the no-shutdown user data string..." >>/var/log/sailing.err
/usr/local/bin/imageupgrade.sh
echo "Found image-upgrade in EC2 user data; not configuring/starting sailing server processes"
else
echo "No image-upgrade request found in EC2 user data $($EC2_METADATA_CMD -d); proceeding with regular server launch..." >>/var/log/sailing.err
echo "Servers to launch: ${JAVA_START_INSTANCES}" >>/var/log/sailing.err
echo "No image-upgrade request found in EC2 user data $($EC2_METADATA_CMD -d); proceeding with regular server launch..."
echo "Servers to launch: ${JAVA_START_INSTANCES}"
if [ -f "${REBOOT_INDICATOR}" ]; then
echo "This is a re-boot. No EC2 user data is evaluated for server configuration; no server configuration is performed. Only configured applications are launched." >>/var/log/sailing.err
echo "This is a re-boot. No EC2 user data is evaluated for server configuration; no server configuration is performed. Only configured applications are launched."
for conf in ${JAVA_START_INSTANCES}; do
su - sailing -c "cd ${SERVERS_DIR}/${conf} && ./start" 2>>/var/log/sailing.err >>/var/log/sailing.err
su - sailing -c "cd ${SERVERS_DIR}/${conf} && ./start"
done
else
echo "This is a first-time boot. EC2 user data is evaluated for potential application deployment and configuration, and applications are launched." >>/var/log/sailing.err
echo "Initializing local MongoDB replica set \"replica\"..." >>/var/log/sailing.err
echo "This is a first-time boot. EC2 user data is evaluated for potential application deployment and configuration, and applications are launched."
echo "Initializing local MongoDB replica set \"replica\"..."
while ! echo "rs.initiate()" | mongo; do
echo "MongoDB not ready yet; waiting and trying again..." >>/var/log/sailing.err
echo "MongoDB not ready yet; waiting and trying again..."
sleep 5
done
FIRST_SERVER=$( eval $( ${EC2_METADATA_CMD} -d | sed -e 's/^user-data: //' ); echo $SERVER_NAME )
if [ "${FIRST_SERVER}" = "" ]; then
echo "No SERVER_NAME provided; not configuring/starting any application processes" >>/var/log/sailing.err
echo "No SERVER_NAME provided; not configuring/starting any application processes"
else
echo "Server to configure and start: ${FIRST_SERVER}" >>/var/log/sailing.err
echo "Server to configure and start: ${FIRST_SERVER}"
configure_and_start_server "${FIRST_SERVER}"
fi
echo 1 >"${REBOOT_INDICATOR}"
@@ -74,26 +73,19 @@ configure_and_start_server() {
chgrp sailing "${SERVERS_DIR}/${conf}"
# If there is a secret /root/mail.properties, copy it into the default server's configuration directory:
/usr/local/bin/cp_root_mail_properties "${conf}"
su - sailing -c "cd ${SERVERS_DIR}/${conf} && refreshInstance.sh auto-install; ./start" 2>>/var/log/sailing.err >>/var/log/sailing.err
pushd ${SERVERS_DIR}/${conf}
./defineReverseProxyMappings.sh 2>>/var/log/sailing.err >>/var/log/sailing.err
popd
RETVAL=$?
[ $RETVAL -eq 0 ] && success || failure
su - sailing -c "cd ${SERVERS_DIR}/${conf} && refreshInstance.sh auto-install; ./start"
}
stop_servers() {
for conf in $JAVA_START_INSTANCES; do
echo "Stopping Java server $conf" >> /var/log/sailing.err
echo "Stopping Java server $conf"
su - sailing -c "cd $SERVERS_DIR/$conf && ./stop"
RETVAL=$?
[ $RETVAL -eq 0 ] && success || failure
sync_logs
done
}
sync_logs() {
echo "Executing logrotate followed by a sync to ensure that logs are synchronized" >> /var/log/sailing.err
echo "Executing logrotate followed by a sync to ensure that logs are synchronized"
logrotate -f /etc/logrotate.conf
sync
}
configuration/environments_scripts/sailing_server/setup-sailing-server.sh
+2 -20
View File
@@ -32,10 +32,7 @@ else
sudo mv imageupgrade_functions.sh /usr/local/bin
# build-crontab
. imageupgrade_functions.sh
# The 2nd argument references a user home to navigate to and the 3rd argument is the path to navigate to within that user to find a checked
# out git workspace. This dependency will be removed in the future, when we no longer have any dependency on a checked out
# workspace on a sailing server.
build_crontab_and_setup_files sailing_server sailing code
build_crontab_and_setup_files sailing_server
# Create an SSH key pair with empty passphrase for ec2-user, deploy it to trac@sapsailing.com
# and then move it to the sailing user's .ssh directory
setup_keys "sailing_server"
@@ -47,11 +44,6 @@ else
# Install SAP JVM 8:
sudo mkdir -p /opt
sudo su - -c "source /usr/local/bin/imageupgrade_functions.sh; download_and_install_latest_sap_jvm_8"
# Configure SSH daemon:
sudo su - -c "cat << EOF >>/etc/ssh/sshd_config
MaxStartups 100
EOF
"
# Increase limits
sudo su - -c "cat << EOF >>/etc/sysctl.conf
# number of connections the firewall can track
@@ -61,17 +53,7 @@ EOF
sudo systemctl daemon-reload
sudo systemctl enable mountnvmeswap.service
# Install MongoDB 4.4 and configure as replica set "replica"
sudo su - -c "cat << EOF >/etc/yum.repos.d/mongodb-org.4.4.repo
[mongodb-org-4.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2/mongodb-org/4.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc
EOF
"
sudo yum -y update
sudo yum -y install mongodb-org-server mongodb-org-shell mongodb-org-tools
setup_mongo_4_4
sudo su - -c "cat << EOF >>/etc/mongod.conf
replication:
replSetName: replica
configuration/environments_scripts/sailing_server/users/root/crontab-update-authorized-keys
-1
View File
@@ -1 +0,0 @@
../../../../crontabs/crontab-update-authorized-keys
configuration/environments_scripts/sailing_server/users/root/crontab-update-authorized-keys@USER_HOME_DIR=_root
Symlink
+1
View File
@@ -0,0 +1 @@
../../../../crontabs/crontab-update-authorized-keys@USER_HOME_DIR
configuration/finduserbyemail
-1
View File
@@ -1 +0,0 @@
environments_scripts/central_mongo_setup/files/usr/local/bin/finduserbyemail
configuration/github-download-release-assets.sh
+3 -3
View File
@@ -16,12 +16,12 @@
BEARER_TOKEN="${1}"
RELEASE_NAME_PREFIX="${2}"
RELEASES=$( curl -L -H 'Authorization: Bearer '${BEARER_TOKEN} https://api.github.com/repos/SAP/sailing-analytics/releases 2>/dev/null )
RELEASE_NOTES_TXT_ASSET_ID=$( echo "${RELEASES}" | jq -r 'sort_by(.created_at) | reverse | map(select(.name | startswith("'${RELEASE_NAME_PREFIX}'")))[0].assets[] | select(.content_type=="text/plain").id' 2>/dev/null)
RELEASE_NOTES_TXT_ASSET_ID=$( echo "${RELEASES}" | jq -r 'sort_by(.published_at) | reverse | map(select(.name | startswith("'${RELEASE_NAME_PREFIX}'")))[0].assets[] | select(.content_type=="text/plain").id' 2>/dev/null)
if [ "$?" -ne "0" ]; then
echo "No release with prefix ${RELEASE_NAME_PREFIX} found. Not trying to download/upload anything." >&2
else
RELEASE_TAR_GZ_ASSET_ID=$( echo "${RELEASES}" | jq -r 'sort_by(.created_at) | reverse | map(select(.name | startswith("'${RELEASE_NAME_PREFIX}'")))[0].assets[] | select(.content_type=="application/x-tar").id' )
RELEASE_FULL_NAME=$( echo "${RELEASES}" | jq -r 'sort_by(.created_at) | reverse | map(select(.name | startswith("'${RELEASE_NAME_PREFIX}'")))[0].assets[] | select(.content_type=="application/x-tar").name' | sed -e 's/\.tar\.gz$//')
RELEASE_TAR_GZ_ASSET_ID=$( echo "${RELEASES}" | jq -r 'sort_by(.published_at) | reverse | map(select(.name | startswith("'${RELEASE_NAME_PREFIX}'")))[0].assets[] | select(.content_type=="application/x-tar").id' )
RELEASE_FULL_NAME=$( echo "${RELEASES}" | jq -r 'sort_by(.published_at) | reverse | map(select(.name | startswith("'${RELEASE_NAME_PREFIX}'")))[0].assets[] | select(.content_type=="application/x-tar").name' | sed -e 's/\.tar\.gz$//')
# For backward compatibility with deployed versions of java/target/refreshInstance.sh
# which may be looking for a default release named "build-...", in case the release
# is a "main-..." release, additionally create a corresponding folder and symbolic links named "build-..."
configuration/github-download-workflow-artifacts.sh
+5 -1
View File
@@ -12,8 +12,12 @@
# an exit status of 2 is returned.
BRANCH="${1}"
BEARER_TOKEN="${2}"
UNIX_TIME=$( date +%s )
UNIX_DATE=$( date --iso-8601=second )
UNIX_TIME_YESTERDAY=$(( UNIX_TIME - 2*24*3600 )) # look back two days in time
DATE_YESTERDAY=$( date --iso-8601=second -d @${UNIX_TIME_YESTERDAY} )
# Get the artifacts URL of the last workflow run triggered by a branch push for ${BRANCH}:
LAST_WORKFLOW_FOR_BRANCH=$( curl --silent -L -H 'Authorization: Bearer '${BEARER_TOKEN} https://api.github.com/repos/SAP/sailing-analytics/actions/runs 2>/dev/null | jq -r '.workflow_runs | map(select(.status == "completed" and .name == "release" and ((.head_branch | startswith("'${BRANCH}'")) or (.head_branch | startswith("releases/'${BRANCH}'"))))) | sort_by(.created_at) | reverse | .[0]' )
LAST_WORKFLOW_FOR_BRANCH=$( curl --silent -L -H 'Authorization: Bearer '${BEARER_TOKEN} https://api.github.com/repos/SAP/sailing-analytics/actions/runs?created=${DATE_YESTERDAY/+/%2B}..${UNIX_DATE/+/%2B}\&per_page=100 2>/dev/null | jq -r '.workflow_runs | map(select(.status == "completed" and .name == "release" and ((.head_branch | startswith("'${BRANCH}'")) or (.head_branch | startswith("releases/'${BRANCH}'"))))) | sort_by(.updated_at) | reverse | .[0]' )
ARTIFACTS_URL=$( echo "${LAST_WORKFLOW_FOR_BRANCH}" | jq -r '.artifacts_url' )
CONCLUSION=$( echo "${LAST_WORKFLOW_FOR_BRANCH}" | jq -r '.conclusion' )
ARTIFACTS_JSON=$( curl --silent -H 'Authorization: Bearer '${BEARER_TOKEN} "${ARTIFACTS_URL}" )
configuration/imageupgrade.sh
-1
View File
@@ -1 +0,0 @@
environments_scripts/sailing_server/files/usr/local/bin/imageupgrade.sh
configuration/imageupgrade_functions.sh
-1
View File
@@ -1 +0,0 @@
environments_scripts/sailing_server/files/usr/local/bin/imageupgrade_functions.sh
configuration/init.d/mongodb
-58
View File
@@ -1,58 +0,0 @@
#!/bin/bash
#
# mongodb Starts sailing services like MongoDB
#
# chkconfig: 2345 95 10
# description: Sailing contains all sailing services
# Source function library.
. /etc/init.d/functions
RETVAL=0
MONGODB_BINARY=/opt/mongodb-linux-x86_64-1.8.1/bin/mongod
MONGODB_CONF_DIR=/etc/mongodb
MONGODB_START_INSTANCES="default dev live archive racecommittee_dev"
start_mongodb() {
for conf in $MONGODB_START_INSTANCES; do
echo "Starting MongoDB server $conf"
rm -f /var/lib/mongodb/$conf/mongod.lock
su - mongodb -c "$MONGODB_BINARY --quiet --fork -f $MONGODB_CONF_DIR/mongodb-$conf.cfg"
RETVAL=$?
[ $RETVAL -eq 0 ] && success || failure
done
# give database servers some time to start
sleep 2
}
stop_mongodb() {
killall mongod
sleep 2
}
# See how we were called.
case "$1" in
start)
start_mongodb
;;
stop)
stop_mongodb
;;
start-mongo)
start_mongodb
;;
start-wiki)
start_wiki
;;
status)
status mongod
RETVAL=$?
;;
*)
echo $"Usage: $0 {start|stop|status}"
RETVAL=3
esac
exit $RETVAL
configuration/mongo_instance_setup/README
-1
View File
@@ -1 +0,0 @@
../environments_scripts/mongo_instance_setup/README
configuration/mongo_instance_setup/add-as-replica
-1
View File
@@ -1 +0,0 @@
../environments_scripts/mongo_instance_setup/files/usr/local/bin/add-as-replica
configuration/mongo_instance_setup/chownvarlibmongo.service
-1
View File
@@ -1 +0,0 @@
../environments_scripts/mongo_instance_setup/files/etc/systemd/system/chownvarlibmongo.service
configuration/mongo_instance_setup/crontab
-3
View File
@@ -1,3 +0,0 @@
* * * * * export PATH=/bin:/usr/bin:/usr/local/bin; sleep $(( $RANDOM * 60 / 32768 )); update_authorized_keys_for_landscape_managers_if_changed $( cat /root/ssh-key-reader.token ) https://security-service.sapsailing.com /home/ec2-user 2>&1 >>/var/log/sailing.err
# NOTICE: Please try to reference the customised crontabs at $GIT_HOME/configuration/crontabs or use
# the build-crontab-and-cp-files script. This file has been maintained for continuity, but is deprecated.
Loaded 100 of 530 files, more files were not shown because too many files have changed in this diff. Show more