eclipse-sailing-analytics/sailing-analytics
1
0
Fork 0
mirror of https://github.com/eclipse-sailing-analytics/sailing-analytics.git synced 2026-09-16 02:38:44 +00:00
Code Activity

Added Wiki article on MongoDB node OS upgrade;

Browse Source 
still to do: description of MongoDB major version upgrade
This commit is contained in:
Axel Uhl
2026-02-10 10:10:51 +01:00
parent dd4aaae623
commit fa76250473
2 changed files with 65 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Home.md
+1
View File
@@ -59,6 +59,7 @@ SAP is at the center of today’s technology revolution, developing innovations
* Amazon
* [[Amazon EC2|wiki/info/landscape/amazon-ec2]]
* [[Upgrading ARCHIVE server|wiki/info/landscape/archive-server-upgrade]]
* [[Upgrading MongoDB Nodes|wiki/info/landscape/mongo-cluster-upgrade]]
* [[EC2 Backup Strategy|wiki/info/landscape/amazon-ec2-backup-strategy]]
* [[Creating an EC2 image from scratch|wiki/info/landscape/creating-ec2-image-from-scratch]]
* [[Upgrading an EC2 image|wiki/info/landscape/upgrading-ec2-image]]
wiki/info/landscape/mongo-cluster-upgrade.md
+64
View File
@@ -0,0 +1,64 @@
# MongoDB Cluster Upgrades
In our production environment on AWS, we currently (2026-02-10) run three MongoDB replica sets:
- ``live``: holds all databases for live operations and consists of three nodes: two i3.large instances with fast NVMe storage used for the ``/var/lib/mongo`` partition, and a hidden instance with an EBS volume that is backed up on a daily basis
- ``archive``: holds the ``winddb`` database used for the ARCHIVE server
- ``slow``: used for backing up databases when removing them from the ``live`` replica set, e.g., when shutting down an application replica set after an event
The ``archive`` and ``slow`` replica sets usually have only a single instance running on ``dbserver.internal.sapsailing.com``, and this is also where the hidden replica of the ``live`` replica set runs. The other two ``live`` nodes have internal DNS names set for them: ``mongo[01].internal.sapsailing.com``.
Upgrades may affect the packages installed on the nodes, or may affect the major version of MongoDB being run. Both upgrade procedures are described in the following two sections.
## Upgrade Using Package Manager
With Amazon Linux 2023, ``dnf`` is the package manager used. When logging on to an instance, a message like
```
A newer release of "Amazon Linux" is available.
Version 2023.10.20260202:
Run "/usr/bin/dnf check-release-update" for full release and version update info
```
may be shown. In this case, run
```
dnf --releasever=latest upgrade
```
and watch closely what the package manager suggests. As soon as you see a kernel update about to install, displayed in red color (if your terminal supports colored output), a reboot will be required after completing the installation. This can also be checked using the following command:
```
needs-restarting -r
```
It will output a message like
```
No core libraries or services have been updated since boot-up.
Reboot should not be necessary.
```
and exits with code ``0`` if no reboot is required; otherwise, it will exit with ``1`` and display a corresponding message.
To avoid interrupting user-facing services, rebooting the MongoDB nodes shall follow a certain procedure:
- Ensure that no ARCHIVE candidate is currently launching; such a candidate would read from the ``archive`` replica set, so that rebooting the ``dbserver.internal.sapsailing.com`` node would interrupt this loading process. If an ARCHIVE candidate is launching, wait for the launch to finish.
- Ensure that no application replica set is currently being shut down with backing up its database. This backup would fail if the ``dbserver.internal.sapsailing.com`` node were restarted as it hosts the ``slow`` replica set used for the backup.
- ssh into ``ec2-user@dbserver.internal.sapsailing.com``
- There, run ``sudo dnf --releasever=latest upgrade`` and confirm with "yes"
- Assuming an update was installed that now requires a reboot, run ``sudo reboot``
- Wait until the instance is back up and running, you can ssh into it again, and ``pgrep mongod`` shows the three process IDs of the three running ``mongod`` processes
- ssh into ``ec2-user@mongo0.internal.sapsailing.com``
- run ``mongosh`` to see if ``mongo0`` is currently primary or secondary in the ``live`` replica set
- if you see "secondary", you're all set; if you see "primary", enter ``rs.stepDown()`` and see how the prompt changes from "primary" to "secondary"
- use ``quit()`` to exit the ``mongosh`` shell
- run ``sudo dnf --releasever=latest upgrade`` and confirm with "yes"
- if a reboot is required, run ``sudo reboot``
- wait for the instance and its ``mongod`` process to become available again; you may probe, e.g., by ssh-ing into the instance and checking with ``mongosh``
- repeat the process described for ``mongo0`` for ``mongo1.internal.sapsailing.com``
Hint: You can choose the order between ``mongo0`` and ``mongo1`` as you wish. If you start with the "secondary" instance, you will save one ``rs.stepDown()`` command.
## MongoDB Major Version Upgrade