mirror of
https://github.com/eclipse-sailing-analytics/sailing-analytics.git
synced 2026-09-21 13:15:35 +00:00
This is a short readme file outlining the architecture of the SAP
Sailing server. For hints for how to perform typical administrative
tasks, please see the ADMIN file which should be stored right next to
this README file.
The architecture mainly consists of
- A 64-bit Linux server running the following services.
- Apache web server handling inbound HTTP requests on port 80 (see
hostname configurations in /etc/httpd/conf.d/kiel.conf)
- dev/test/prod1/prod2 OSGi server instances running the Java
component
- UDP mirror process that duplicates UDP packets received for "the"
production instance and forwards them to the two inbound UDP ports
for prod1 and prod2, respectively.
- dev/test/prod Python servers running the web application for
browser, iPhone and iPad that connect to a Java OSGi server to
complete its requests
- a Pound load balancer that does alive checks for the prod1/prod2
Java instances to enable quick fail-over
- a number of MongoDB instances for dev/test/prod1/prod2 that store
leaderboard information for the Python application as well as wind
information for the Java component
We have a VNC server running on port 5902 with a KDE window
manager. There, we operate console windows for the various Java
instances and the UDP mirror.
Inbound Port Assignments
------------------------
- TCP
22: ssh
80: apache
5902: vnc
8026: Python "dev" instance
....: Python "test" instance
....: Python "prod" instance
8886: servlet container of "dev" Java instance
8887: servlet container of "test" Java instance
8888: servlet container of "prod1" Java instance
8889: servlet container of "prod2" Java instance
10200: MongoDB "dev" instance
10201: MongoDB "test" instance
10202: MontoDB "prod" instance
12347: SSH forward to localhost:22 required for "mvn deploy"
- UDP
2010: wind input for "dev" Java instance
2011: wind input for "test" Java instance
2012: wind input for "prod" (mirrored to 2013/2014)
2013: wind input for "prod1" Java instance
2014: wind input for "prod2" Java instance
Java
----
We use Equinox as an OSGi container into which we deploy a set of
bundles that implement TracTrac connectivity, Expedition wind
connectivity, sailing domain knowledge, race tracking capabilities as
well as servlets with RESTful APIs for use by web clients.
Most data is managed on the TracTrac servers. The only data currently
stored persistently by the Java component is the wind data received
through the supported channels. This helps in re-playing a race later
and understanding the wind conditions at the time which should support
fine-grained after-race analysis.
* Jetty
A Jetty servlet container runs embedded in Equinox. Servlets are
registered (currently in the com.sap.sailing.server bundle) using the
org.eclipse.equinox.http.registry.servlets extension point. See
com.sap.sailing.server/plugin.xml for the currently registered
servlets.
Static content to be provided by Jetty is also registered as an
"extension" for the org.eclipse.equinox.http.registry.resources
extension point. We have a few very simple forms registered that aid
in testing the web application provided by Jetty.
The port on which Jetty listens for incoming connections is configured
using the system / bundle property org.osgi.service.http.port.
* Binary Libraries
The Java component as of this writing has two bundles that wrap
binary libraries: com.tractrac.clientmodule and com.mongodb.driver.
* External Communications
The com.tractrac.clientmodule bundle is used to obtain data about
races / regattas and register for the corresponding live data
feeds. These feeds contain information about boat positions, race
course changes such as marks being moved, replaced or added, mark
positions, mark passings and race start/stop events.
To improve rankings and predictions, wind data is tracked by the Java
component. Wind can be fed through wind sources. Currently supported
are "WEB" and "EXPEDITION." The WEB wind source is a REST API by which
a single wind datum with optional position specification, time point,
wind bearing and speed can be provided. EXPEDITION uses a connector
bundle (com.sap.sailing.expeditionconnector) to receive wind data via
UDP from one or more Expedition instances.
The Expedition connector can listen on inbound UDP ports for messages
sent point-to-point or using UDP broadcast by the Expedition tool
which in turn receives its data from boat-mounted sensors and other
devices, such as the Nexus on-board system. Expedition enriches this
data and sends selected attributes over the network.
The connector understands the Expedition data feed format, supports
Expedition checksum verification and knows about the most important
attribute IDs such as true wind bearing, GPS time, latitude and
longitude.
* Storing Wind in MongoDB
The Java component piggy-backs on the MongoDB persistence solution
installed mainly for the Python server. Storing wind is helpful
particularly for post-race analysis and to support the case of a
re-start of the Java component(s) on the server, e.g., in case an
update or a quick-fix needs to be installed.
Since all other data can be reloaded from the TracTrac server, we
don't store that redundantly. But the wind information isn't retrieved
from TracTrac and hence needs to be stored on our server.
By default we use the database named "winddb". The MongoDB settings
can be configured using system / bundle properties:
mongo.port (default is 27017)
mongo.hostname (default is 127.0.0.1)
mongo.dbname (default is winddb)
* List of REST APIs and their parameters
The distribution of services across the /admin and /moderator servlet
is currently not well sorted out and requires some clean-up. The
general idea is to have administrative tasks accessible through the
/admin servlet whereas things that the moderator will see are accessed
through the /moderator servlet. We may want to consider merging both
into one in the future.
/admin
action=addevent
2014: wind input for "prod2" Java instance
Java
----
We use Equinox as an OSGi container into which we deploy a set of
bundles that implement TracTrac connectivity, Expedition wind
connectivity, sailing domain knowledge, race tracking capabilities as
well as servlets with RESTful APIs for use by web clients.
Most data is managed on the TracTrac servers. The only data currently
stored persistently by the Java component is the wind data received
through the supported channels. This helps in re-playing a race later
and understanding the wind conditions at the time which should support
fine-grained after-race analysis.
* Jetty
A Jetty servlet container runs embedded in Equinox. Servlets are
registered (currently in the com.sap.sailing.server bundle) using the
org.eclipse.equinox.http.registry.servlets extension point. See
com.sap.sailing.server/plugin.xml for the currently registered
servlets.
Static content to be provided by Jetty is also registered as an
"extension" for the org.eclipse.equinox.http.registry.resources
extension point. We have a few very simple forms registered that aid
in testing the web application provided by Jetty.
The port on which Jetty listens for incoming connections is configured
using the system / bundle property org.osgi.service.http.port.
* Binary Libraries
The Java component as of this writing has two bundles that wrap
binary libraries: com.tractrac.clientmodule and com.mongodb.driver.
* External Communications
The com.tractrac.clientmodule bundle is used to obtain data about
races / regattas and register for the corresponding live data
feeds. These feeds contain information about boat positions, race
course changes such as marks being moved, replaced or added, mark
positions, mark passings and race start/stop events.
To improve rankings and predictions, wind data is tracked by the Java
component. Wind can be fed through wind sources. Currently supported
are "WEB" and "EXPEDITION." The WEB wind source is a REST API by which
a single wind datum with optional position specification, time point,
wind bearing and speed can be provided. EXPEDITION uses a connector
bundle (com.sap.sailing.expeditionconnector) to receive wind data via
UDP from one or more Expedition instances.
The Expedition connector can listen on inbound UDP ports for messages
sent point-to-point or using UDP broadcast by the Expedition tool
which in turn receives its data from boat-mounted sensors and other
devices, such as the Nexus on-board system. Expedition enriches this
data and sends selected attributes over the network.
The connector understands the Expedition data feed format, supports
Expedition checksum verification and knows about the most important
attribute IDs such as true wind bearing, GPS time, latitude and
longitude.
For converting magnetic bearings to true bearings the
com.sap.sailing.declination bundle can talk to a web page at
http://www.ngdc.noaa.gov/geomagmodels/struts/calcDeclination. It is
possible to download and store declinations in a file shipped with the
bundle. The resources/ folder in the bundle's source holds files named
"declination-yyyy.txt" where "yyyy" represents the year of which the
file holds declination values. For 2011 there is a file that holds
declination values and annual change rates for a 1x1 degree grid. If
values of greater precision are requested, this may result in online
requests. After a timeout, interpolation/extrapolation based on the
known values and annual change rates is tried.
* Storing Wind in MongoDB
The Java component piggy-backs on the MongoDB persistence solution
installed mainly for the Python server. Storing wind is helpful
particularly for post-race analysis and to support the case of a
re-start of the Java component(s) on the server, e.g., in case an
update or a quick-fix needs to be installed.
Since all other data can be reloaded from the TracTrac server, we
don't store that redundantly. But the wind information isn't retrieved
from TracTrac and hence needs to be stored on our server.
By default we use the database named "winddb". The MongoDB settings
can be configured using system / bundle properties:
mongo.port (default is 27017)
mongo.hostname (default is 127.0.0.1)
mongo.dbname (default is winddb)
* List of REST APIs and their parameters
The distribution of services across the /admin and /moderator servlet
is currently not well sorted out and requires some clean-up. The
general idea is to have administrative tasks accessible through the
/admin servlet whereas things that the moderator will see are accessed
through the /moderator servlet. We may want to consider merging both
into one in the future.
/admin
action=listracesinregatta: outputs a JSON object describing the
races and their paramURLs available for tracking in
the following form:
[ { "name": ..., // race name
"ID": ..., // a UUID for the race
"paramURL": ..., // the paramURL to use for addrace
"replayURL": ... } ...]// URL to use for TracTrac 2D view
eventJSONURL: the URL from where metadata about the entire regatta
can be obtained; the resulting document includes the
parameter and simulation URLs for the individual
races that are part of the event/regatta.
action=addevent: starts tracking all races described in the document received
from the eventJSONURL to the server instance.
eventJSONURL: the URL from where metadata about the entire regatta
can be obtained; the resulting document includes the
parameter and simulation URLs for the individual
races that are part of the event/regatta.
liveURI: the live data URI for the TracTrac server
storedURI: the stored data URI for the TracTrac server; usually
the port number of this URI is that of liveURI + 1
action=stopregatta: stops tracking of all races of the regatta; this
also stops all wind tracking for the races of the regatta
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
action=addrace: starts tracking a single race; if other races of
the same event/regatta are already being tracked,
the race is added to the regatta
paramURL: the URL from where metadata about the race
can be obtained; the resulting document includes
UUIDs of waypoints, competitors, etc.
liveURI: the live data URI for the TracTrac server
storedURI: the stored data URI for the TracTrac server; usually
the port number of this URI is that of liveURI + 1
action=stoprace: stops tracking of a single race, also stopping the
wind trackers that were started for this race; other
races of the same regatta that were being tracked will
continue being tracked.
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
racename: name of the race, as returned by the "listevents"
action on the /moderator servlet
action=receiveexpeditionwind: starts a wind tracker that receives
wind data through a UDP port from an Expedition
client; the wind is fed into a wind track for the
race specified by regattaname/racename.
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
racename: name of the race, as returned by the "listevents"
action on the /moderator servlet
correctexpeditionwindbearingbydeclination: optional; if "true",
the wind bearings received from Expedition will be
corrected by the magnetic declination for the
time and location specified by the Expedition
message.
action=stopreceivingexpeditionwind: stops an expedition wind tracker
for the race specified by regattaname/racename
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
racename: name of the race, as returned by the "listevents"
action on the /moderator servlet
action=listwindtrackers: lists all wind trackers currently
registered in the server instance. The output
structure is an array of {regattaname,racename,windtrackerinfo}
objects where "windtrackerinfo" is a tracker type-specific
string field which may contain, e.g., the tracker type
name or, in case of a UDP port-based tracker,
the UDP port number the tracker is receiving
wind information on.
action=setwind: feeds a single wind measurement into the WEB
wind track of the race specified. If no time is
specified, the measurement is assumed to have been
taken at the current time.
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
racename: name of the race, as returned by the "listevents"
action on the /moderator servlet
truebearingdegrees: as the name suggests; can have decimal places
knotspeed: optional; as the name suggests; can have decimal
places. If omitted, 1kn is used as a default
latdeg/lngdeg: optional; latitude and longitude in degrees,
optionally with decimal places
time: optional; time stamp for the wind measurement
in ISO format in arbitrary precision; takes
precedence over a "timeasmillis" parameter.
Example values:
2011-05-27T13:27:19.283
2010-06-30
2011-08-01T15:44:00
timeasmillis: optional; time stamp for the wind measurement
in milliseconds since 1970-01-01T00:00:00 UTC
action=removewind: removes a single wind measurement with the
millisecond-exact time stamp again from the
specified race's specified wind source. A time
should be provided, although the current time
will be used as the default. However, it's
unlikely that this will match the time point
of the wind to remove by the millisecond.
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
racename: name of the race, as returned by the "listevents"
action on the /moderator servlet
sourcename: name of the wind source, such as WEB or EXPEDITION
time: time stamp for the wind measurement
in ISO format in arbitrary precision; takes
precedence over a "timeasmillis" parameter.
Example values:
2011-05-27T13:27:19.283
2010-06-30
2011-08-01T15:44:00
timeasmillis: time stamp for the wind measurement
in milliseconds since 1970-01-01T00:00:00 UTC
action=showwind: shows the wind tracks that exist for the race
specified. The output is a JSON object with the
following structure:
{ [ "WEB": { "truebearingdeg": ...,
"knotspeed": ...,
"meterspersecondspeed": ...,
"timepoint": ... (optional, if time point is known),
"dampenedtruebearingdeg": ... (optional),
"dampenedknotspeed": ... (optional),
"dampenedmeterspersecondspeed": ... (optional)
"latdeg": ... (optional, if position known),
"lngdeg": ... (optional, if position known)
}
"EXPEDITION": { ... }
}
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
racename: name of the race, as returned by the "listevents"
action on the /moderator servlet
action=addwindtomarksforonehour: Can be used to translate one or
more positions in the direction and distance that
the wind blows at the position's location during
one hour. One use case is drawing the wind lines
through the mark positions. If no time point
is specified, the current time is used for the
wind lookup. The result is a JSON array with
objects of the following structure:
{ "markLatDeg": ..., // position provided
"markLngDeg": ..., // position provided
"windTrueBearingDeg": ..., // wind info
"windKnotSpeed": ..., // wind info
"toLatDeg": ..., // translated position
"toLngDeg" } // translated position
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
racename: name of the race, as returned by the "listevents"
action on the /moderator servlet
time: optional; time stamp for the wind measurement
in ISO format in arbitrary precision; takes
precedence over a "timeasmillis" parameter.
Example values:
2011-05-27T13:27:19.283
2010-06-30
2011-08-01T15:44:00
timeasmillis: optional; time stamp for the wind measurement
in milliseconds since 1970-01-01T00:00:00 UTC
latdeg/lngdeg: can occur multiple times with an equal number of
latdeg and lngdeg parameter occurrences. Specifies
the positions to be translated.
/moderator
action=listevents: shows the events with their races that are
currently being tracked by this server instance. The
result is a JSON array with entries describing the
events like this:
{ "name": ..., // event name
"boatclass": ..., // optional; name of boat class
"competitors": [ // the event's competitors
{ "name", ..., // competitor's name
"sailID": ... // sail ID if known
"nationality": ..., // 3-letter IOC country code
"nationalityISO2": ..., // 2-letter ISO country code (may be empty)
"nationalityISO3": ..., // 3-letter ISO country code (may be empty)
"team": [ { // the sailors forming the team
"name": ...,
"description": ... } ... ] } ... ]
"races": [ // list of races in the regatta
"name": ...,
"boatclass": ...,
"start": ..., // race start time as millisecond
// timestamp, Integer.MAX_VALUE
// if not started yet
"legs": [ { // legs in the course's order
"start": ..., // name of the leg start mark
"end": ... } // name of the leg end mark
... ]
... ]
action=showboatpositions: outputs a JSON object describing the boats
with their positions for the race specified. Output
format:
{ "name": ..., // race name
"competitors": [ {
"name": ...,
"track": [ {
"timepoint": ..., // millisecond timestamp
"latdeg": ..., // boat position
"lngdeg": ...,
"truebearingdeg": ..., // true boat bearing in degrees
"knotspeed": ..., // boat speed in knots
"tack": [STARBOARD|PORT] } ... ] // side on which sails are
} ... ] }
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
racename: name of the race, as returned by the "listevents"
action on the /moderator servlet
since/sinceasmillis: optional; ISO / milliseconds timestamp
describing the earliest time point for boat
positions to be returned; if not specified,
known boat positions are listed from the
"beginning of time"
to/toasmillis: optional; optional; ISO / milliseconds timestamp
describing the latest time point for boat
positions to be returned; if not specified,
known boat positions are listed until the
"end of time"
action=showwaypoints: shows the positions of a course marks
at a specified point in time; if no time/timeasmillis
is provided, the race start time is used; if that is
not available, the start of tracking is used;
if that is not available either, the time point
of the newest event is used; if that is not
available, the current time is used. The output
is an array of JSON objects with following format each:
{ "name": ..., // waypoint name
"marks": [ { // usually one mark or two marks in case of a gate
"name": ..., // mark name
"lat": ..., // optional; if mark pos known
"lng": ... } ... ] } // optionsl; if mark pos known
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
racename: name of the race, as returned by the "listevents"
action on the /moderator servlet
time: optional; time stamp for the wind measurement
in ISO format in arbitrary precision; takes
precedence over a "timeasmillis" parameter.
Example values:
2011-05-27T13:27:19.283
2010-06-30
2011-08-01T15:44:00
timeasmillis: optional; time stamp for the wind measurement
in milliseconds since 1970-01-01T00:00:00 UTC
action=showrace: shows lots of tracking information about a
tracked race at a given point in time. If no
time point is provided (time / timeasmillis),
the time point of the event received last is
used. If no such time point exists, the current
time is used. The output has the following
structure:
{ "name": ..., // race name
"startoftracking": ..., // millisecond timestamp
"start": ..., // millisecond timestamp
"timeofnewestevent": ...,// greatest millisecond timestamp
// of any event received so far
"timeoflastevent": ..., // millisecond timestamp
// of the event received last
"updatecount": ..., // integer number telling
// the number of events
// received so far; a bit
// like a "transaction time"
"wind": { // optional, if wind set
"truebearingdeg": ...,
"knotspeed": ...,
"meterspersecondspeed": ... },
"legs": [ { // leg tracking info per leg
"from": ..., // name of from-waypoint
"fromwaypointid": ..., // a unique integer for the
// from-waypoint
"to": ..., // name of to-waypoint
"towaypointid": ..., // a unique integer for the
// from-waypoint
"upordownwindleg": ...,// true or false
"competitors": [ { // competitor list
"name": ...,
// current SOG is optional
"currentSpeedOverGroundInKnots": ...,
"currentSpeedOverGroundInMetersPerSecond": ...,
// average SOG is optionsl
"averageSpeedOverGroundInKnots": ...,
"averageSpeedOverGroundInMetersPerSecond": ...,
// distance traveled is optional
// next comes the speed over ground if the competitor is still in the
// leg, or the average speed over ground if the competitor has already
// finished the leg, but only if the respective value is defined
"currentSpeedOverGroundOrAverageSpeedOverGroundIfLegFinishedInKnots": ...,
"currentSpeedOverGroundOrAverageSpeedOverGroundIfLegFinishedInMetersPerSecond": ...,
"distanceTraveledOverGroundInMeters": ...,
// VMG info is optional
"velocityMadeGoodInKnots": ...,
"velocityMadeGoodInMetersPerSecond": ...,
"averageVelocityMadeGoodInKnots": ...,
"averageVelocityMadeGoodInMetersPerSecond": ...,
"rank": ..., // competitor's rank in this leg
"gapToLeaderInSeconds": ..., // optional, only if wind is known
"estimatedTimeToNextMarkInSeconds": ..., // optional, only if wind is known
"windwardDistanceToGoInMeters": ..., // optional, only if wind is known
"started": ..., // true or false, tells if competitor has started leg
"finished": ... // true of false, tells if competitor has finished leg
} ... ] // competitor's rank in leg
} ... ],
"ranks": [ { // the overall total ranks, across legs
"competitor": ..., // name of competitor
"rank": ... } ... ] }
regattaname: name of the regatta, as returned by the "listevents"
action on the /moderator servlet
racename: name of the race, as returned by the "listevents"
action on the /moderator servlet
time: optional; time stamp for the wind measurement
in ISO format in arbitrary precision; takes
precedence over a "timeasmillis" parameter.
Example values:
2011-05-27T13:27:19.283
2010-06-30
2011-08-01T15:44:00
timeasmillis: optional; time stamp for the wind measurement
in milliseconds since 1970-01-01T00:00:00 UTC
sinceupdate: optional; an integer number that if provided,
causes the call to block on the server
until the update count returned will be greater
than the value provided for sinceupdate.
* Server Directory Structures
git/:
This is our git "hub" directory with checked-out workspace.
We use it also as the directory to check out a specific branch
that we want to build using Maven.
servers/:
The four Java server instances dev, test, prod1 and prod2 have their
directories here.
[dev|test|prod1|prod2]/
configuration/
config.ini: contains the set of initial bundles to launch upon
server start as well as bundle properties such as the
HTTP port to listen to and the mongo.port property
start: the start script used to launch the respective server instance
update: used to copy the built artifacts from the corresponding
maven-repositories/ subdirectory to the plugins/ subdirectory
plugins/: contains the JAR files for the server instance
maven-repositories/
[dev|test|prod]/: the target repositories for the "mvn deploy" tasks
for the respective git branch (dev|test|prod)
* Maven Environment
Our Java OSGi bundles are built and tested using the Apache Maven build system together
with the Tycho plugin for Maven. The bundles carrying the JUnit tests (typically ending
with *.test in their bundle name) are built using eclipse-test-plugin as packaging
type. The regular OSGi bundles are built using eclipse-plugin as packaging type.
When invoking "mvn install" in the ~trac/git/java directory, all bundles for the Java
server will be built and the tests will start to run. A report is printed at the end. If
it decides for "SUCCESS" then all builds and tests finished successfully. However, this
hasn't had any effect on any of the running servers, not even the maven repository.
For that, see the respective instructions in the ADMIN file located next to this README
file (search for "mvn deploy").
Python
------
TODO Simon to write a few warm words here...