sailing-analytics/wiki/basic-architectural-principles.md

Basic Architectural Principles

[[TOC]]

##Domain Model We have created a set of interfaces that represent the concepts of the domain of sailing races. Typical abstractions captured by these interfaces are, e.g., Regatta, Series, Course, Leg, Waypoint, Buoy and Competitor. There are also interfaces describing more general concepts not necessarily specific to sailing, such as GPSTrack, Position and Distance.

Instances of classes implementing these domain interfaces are generally created using a DomainFactory instance. However, in some special cases constructors of domain classes implementing the domain interfaces may also be invoked directly.

We have tried to split the domain interfaces into such describing "master data" and others describing "transactional data." The master data interfaces can be found largely in the com.sap.sailing.domain and com.sap.sailing.domain.common packages. "Transactional" data is largely obtained through tracking devices which is why most of those interfaces are found in the com.sap.sailing.domain.tracking package. For some of the master data domain interfaces, we have corresponding "Tracked..." counterparts that capture tracking data that pertains to the respective master data object. For example, TrackedRace describes the tracking data for a race. It has TrackedLeg as one of its constituents, corresponding to the Leg objects contained by a race's Course object.

Where possible we have tried to keep objects immutable. This has several advantages, particularly when caching such objects and when replicating and propagating and displaying in a user interface. However, this rigid approach also leads to problems, particularly if the underlying assumption that an object doesn't change turns out not to be true in all cases. For example, we have modeled the Competitor interface as immutable. However, if the competitor data was obtained through a tracking service provider who made a mistake during capturing the competitor data which is later corrected, having the Competitor as immutable is a real problem.

For some of those concepts that allow for changes to be applied, we have distinguished between the reading interface and the modifying interface. For example, DynamicTrackedRace is the modifiable counterpart to TrackedRace which offers only reading operations.

The domain model is independent of tracking service provider specifics. When the system receives data from a tracking provider, the specific messages are converted into domain objects. To avoid duplicate domain objects for the same entity (e.g., a Competitor), the connectors (see also Tracking and Wind Sensor Connectors) typically keep mappings that allow the connector to retrieve and use an existing domain object instead of creating new ones. The challenge with these mappings is to avoid memory leaks. Currently, the existing mappings only maintain domain objects that have a small memory footprint. We need a good strategy for releasing objects from these mappings. Ideally, the life cycle of the objects in such caches should be coupled to the life cycle of other domain objects. For example, a competitor object may be released if all races and regattas and leaderboards possibly using this competitor have been garbage collected.

In-Memory Architecture and the Database

The application generally deals with two sorts of data. One is the sensor data originally created by a set of physical sensors such as GPS trackers, wind measurement devices or the sensors included in smart phones and other mobile devices. The other is master data and meta-data captured and maintained by administrators and users, such as the leaderboard configuration data, connectivity data for the tracking providers, or official scoring results imported from external sources.

For the sensor data, the time at which the data is received is never the same as the time at which the sensor data was valid. Sensors don't predict the future but measure some present value. The transmission from the sensor to the server adds a rather unpredictable latency. In the worst case, the sensor's transmission unit fails, and the data can only be imported into the server once the sensor is back on shore. With this in mind, the server's view of reality is partial and lagging, and history may be re-written at any point in time if a sensor decides to deliver its data later than most other sensors. There is no precise synchronization across the set of sensors used at an event. At best, there is a pre-configured maximum delay at which trackers make an effort to deliver their data. However, this may also fail, for example, if trackers lack network connectivity when they would actually be due to send their data. Tracking providers may also decide to re-compute some derived data which our server receives. For example, some tracking providers send data about when they think which boat passed which mark. The provider may change this at any time, sending an updated list of mark passing times. This can, e.g., happen if the course layout was changed on short notice, and the course update didn't make it into the server in time. Once the course layout change is then updated to the system, the mark passing times will be re-evaluated, and updates to the previous mark passing times will result.

These circumstances suggest an architecture which basically records the sensor facts and dynamically aggregates all derived information on the fly. This is how we started. The more complex the rules for deriving interesting figures from the sensor data grew, the more computational resources the on-the-fly aggregation required. In particular, three algorithms turned out to be quite expensive to carry out: maneuver analysis based on the recursive Douglas-Peucker algorithm; wind estimation based on the boat tracks, assuming that boats on different tacks use roughly the same beat angle to the wind; and the average cross-track error which computes a projection of each boat's position to the wind direction. In addition to those, aggregations of distances traveled incur lots of duplicate work if done naïvely. Usually, GPS fixes are appended to the end of a track. Therefore, the distance traveled would re-compute the distances traveled from the start to the last but one fix which is obviously unnecessary.

The role of the database in all of this is currently largely for recovery purposes only. It stores the administrative information such as which leader boards have been defined, the structure of regattas, explicit score corrections, and the association of tracked races with leaderboard columns, to name a few. Additionally, it stores the wind measurements because those would be hard to recover when the server needs a re-start. For the GPS tracking data, however, we currently try to recover them from the tracking provider where possible.

While this works well for the TracTrac connector, in case of the SwissTiming connector this is not possible at all. Therefore, we developed an additional component (see also SwissTiming Connector) that can record the SwissTiming messages received in the database so they can be retrieved from there after a server restart.

One key consequence of this architecture is that if two server instances share a common database then usually the second instance doesn't see the first instances writes to the database before the second instance is restarted and recovers from the joint database.

Caching where Necessary

Based on the performance implications explained above, we decided to carefully introduce caching where it was absolutely necessary. We added a cache for the results of the maneuver analysis (see TrackedRaceImpl.maneuverCache). Another cache exists for the wind estimations derived from the boats' GPS tracks (see TrackBasedEstimationWindTrackImpl.cache). Yet another cache was introduced for the average cross track error (XTE, see TrackedRaceImpl.crossTrackErrorCache) and for the distances traveled (see GPSFixTrackImpl.distanceCache) as well as for the maximum speeds (see GPSFixTrackImpl.maxSpeedCache). Another trivial cache in TrackedRaceImpl.directionFromStartToNextMarkCache is used to speed up the query for the direction of the first leg which is frequently used in races starting upwind.

While those caches speed up requests on TrackedRace, even larger performance gains can be achieved by caching the higher-level aggregates requested by many clients. In particular, many clients request the same leader board in live mode. Then, having the server determine what the "live" time is instead of using the client's clock, and just returning the latest contents for the leader board requested from a cache has increased the application's scalability significantly. The leaderboard caching is implemented in the classes LeaderboardDTOCache and LiveLeaderboardUpdater.

Implementation Patterns for Caches

A cache is like a materialized view in a database. The greatest challenge with a cache is to keep it consistent. The challenge becomes greater if concurrency is added to the mix, and tens or even hundreds of sensor fixes update the data structures that provide the input for the calculations filling the caches.

In the SAP Sailing Analytics, when introducing a cache we usually need to consider the following questions:

• When do which parts of the cache need to be invalidated?
• Should invalidated cache contents be re-calculated straight upon invalidation, or should this wait until the next relevant request is received?
• Can clients tolerate stale cache contents, permitting to update the cache contents in a background thread?

The invalidation logic depends directly on the computation rules for the value being cached. The cache needs to act as observer of those values so that it receives a notification in case a value changes that the cache content depends on. A useful observer pattern is offered by TrackedRace which offers the registration of RaceChangeListener objects as observers. There are many more such patterns in place, e.g., the RaceColumnListener, CourseListener, RaceListener and RegattaListener.

Whether or not to re-calculate cache contents right after their invalidation usually depends on the access patterns and frequencies compared to the cost and duration of re-calculating. Especially in live mode, we pro-actively keep the LiveLeaderboardUpdater running as long as there are requests for live leaderboards received. Using a timeout (see LiveLeaderboardUpdater.UPDATE_TIMEOUT_IN_MILLIS) the updater stops re-calculating cache contents if for that time no request has been received. Other cache contents, such as for the cache for the direction from the start to the next mark, are only re-calculated when the next request is received because a single re-calculation doesn't take long, compared to the number of times it can then be used from the cache and compared to the overall round-trip time of the request to which this single value contributes.

For some values a client may be able to tolerate a certain lag relative to the latest data received. This also has to be evaluated against the background of the delay in the sensor data transmission, starting at the time point when the sensor picks up a value until the value has arrived in the SAP Sailing Analytics. If this delay is significantly longer than the time required to re-calculate a cache entry based on a new sensor value received, then it is not all that bad to keep pretending for a while that the last cached value is still the best we have until a short time later a cache update is performed, taking into account the new value.

If a cache can be used by clients not worrying too much about slight staleness of data, cache updates can be performed in background threads, and cache contents that we know already are outdated can be continued to be served until the re-calculation has completed. The SmartFutureCache class serves as a base for implementing caches that can be updated by a background task. Values in those caches are only updated, never removed. Clients can choose whether they want to wait for the latest ongoing calculation or if they just want to get the last value available (which may be a bit stale).

Approaches to Locking: "synchronized" vs. ReentrantReadWriteLock

User requests, receiving sensor events and updating caches all happen concurrently, using many threads. Additionally, some user requests are split into many tasks, each of which can be assigned to a separate thread. It is instrumental that no inconsistencies occur due to this high level of concurrency. The usual approaches to locking have to be considered; only that the architecture of the SAP Sailing Analytics is not primarily one using a database with support for transactions, but an in-memory architecture with competing reads and updates.

We started out by using the Java built-in object monitors ("synchronized") to protect code regions from concurrent access of readers and writers or multiple readers. Being a basic language construct, the tool support for synchronized is great. The Eclipse debugger can visualize deadlocks right away, once they occur (and they did of course occur).

There are two well-known problems with using synchronized as an approach to locking:

• Locking is bound to the lexical code structure. Code regions using synchronized have to be specified as a block of Java code. While this ensures that locks are properly released once the execution leaves this block of code, it makes hand-over-hand locking impossible, and it makes locking a variable set of objects before executing some block of code impossible.
• It is not possible to distinguish between readers and writers so that multiple readers could read concurrently.

This fact reduces the amount of concurrency possible to only one concurrent reader per lock object. For some situations this is just not sufficient. To increase the amount of concurrency across multiple readers, we decided to use ReentrantReadWriteLock instead of synchronized where we had identified bottlenecks. This helped to increase concurrency a lot. However, we had to sacrifice two things that were so convenient with sycnhronized, namely the tool support (tools don't understand deadlocks between readers and writers which hold technically distinct primitive locks) and the implicit correctness of releasing the lock again. The latter boils down to establishing a discipline of always releasing each lock obtained in a finally clause. Although no longer enforced by the lexical structure, it isn't too difficult to execute this discipline.

The lack of tool support, particularly for deadlock detection, hurts worse. Some deadlocks that occurred with the use of ReentrantReadWriteLock were obvious and easy to find. Others, however, were very hard to find, particularly if they happened in a Java VM that was not running in debug mode and hence didn't allow attaching an external tool that would have helped in identifying the various call stacks and lock ownership relationships.

To alleviate this problem, we at least introduced excessive tracing in case a lock cannot be obtained for some period of time. The trace output contains all reader and the writer thread (if any) including their full stack traces. This has already helped identify one further deadlock which occurred since the introduction of the traces.

Generally, one of the most important locks in the system is that of a Course object. When a course changes, e.g., by the addition or removal of waypoints, many dependent data structures will need to be updated. Therefore, in turn, operating on those data structures requires the caller to hold the Course's read lock. This will hold back a course update until no reader is working with the course or any derivatives. As a general hint, note that if a thread tries to obtain the read lock of a ReentrantReadWriteLock, if may seem that it's only failing for another thread holding the read lock. However, there may be another thread trying to obtain the write lock which cannot be granted because of the reader currently holding the read lock. But being a fair lock, the ReentrantReadWriteLock will not let the next reader get the lock if a writer has been waiting to obtain the lock before the next reader arrived. Therefore, this may look like a reader-reader deadlock, but a writer is actually involved. Given the timeouts that LockUtil applies, this won't deadlock forever, but it will lead to an unpleasant waiting time until the writer times out and before it retires lets the next reader obtain the lock.

Scale-Out through Replication

Scalability becomes an issue as the number of concurrent users increases and the requirements regarding availability grow. We think that replicating a server instance so that the replica can respond to client requests will help towards both goals: handle more concurrent users and increase availability.

Master/Replica Distinction

There are various ways to implement server replication. Among the easiest is a landscape that distinguishes a "master" from a "replica" side. In such a set-up, changes are injected into the master first, from where they are replicated to all replicas. No other changes are permitted on the replicas in this easy set-up, leading to eventual consistency once the flow of events on the master stops.

Operational Transformation

With a distinction between master and replica it also becomes possible to apply a technique called Operational Transformation (OT). It is an approach leading to eventual consistency between one master and multiple replicas, supporting changes injected on both, master and replicas. This may become an interesting option when discussing a dying master and the replicas negotiating a new master.

The code base already contains an OT implementation (see package com.sap.sailing.server.operationaltransformation). The operations used for replication are prepared to interact with this framework, but the implementation of the transformation rules are largely not yet implemented except for a few tests in the area of leaderboard-related operations.

Implementation of Operations, Services, and Events

The operations used for replicating changes are so far all located in the package com.sap.sailing.server.operationaltransformation. There are three ways in which these operations are being used, two of which need urgent consolidation.

GWT Service Constructing and Applying an Operation

In some cases, an instance of the class SailingServiceImpl which handles incoming GWT servlet requests constructs an operation object and applies it to the RacingEventService directly by calling getService().apply(operation). This way of using operations should probably better be encapsulated and hidden behind the RacingEventService interfaces, as we have done for many operations already, as described in the following section.

RacingEventService Constructing and Replicating an Operation Triggered by a Client Request

Several RacingEventService methods that are relevant for replication do their job and afterwards produce an operation object submitted to the replicator. The receiving replica applies the operation to its own RacingEventService. Most of these operations are implemented such that they call the same method that produced the operation on the master. While this again constructs an operation on the replica and submits it to the replica's own replicator, no transitive replication happens usually because a replica has no further replicas.

It would probably be better to consolidate the two ways in which explicit requests use the replication architecture. It would be nice if the same operation implementation was used on the master and the replica side to carry out the actual change on the RacingEventService instance. Ideally, the RacingEventService would offer dedicated methods, not for external use but for use by the operation implementations. Those dedicated methods then wouldn't have to worry about replication. Methods exposed by RacingEventService should then guarantee replication by producing the operation locally, applying it (which uses the dedicated, non-exposed methods) and replicating it. The receiving replica would then again just apply the operation, with no further replication being implicitly triggered.

RacingEventService Constructing and Applying an Operation Triggered by an Event

The system receives event notifications that lead to a state change requiring replication. For example, when a GPS fix arrives from a boat tracker, the fix needs to be recorded locally and has to be propagated to all replicas. This currently happens by letting the RacingEventService observe the data structures that change due to such events, for example, the GPS and wind tracks. The RacingEventService maintains a RaceChangeListener for each tracked race. Whenever a change worth replicating occurs on any of these races, the RacingEventService forms an operation and passes it to the replicator.

When thinking about the necessary consolidation of the different approaches to replication, we may even want to go as far as having these types of low-level, fine-granular changes be funneled through the RacingEventService interface in a consistent way, such that the service doesn't need to listen to low-level data structures for replication purposes anymore but instead can use the explicit calls to exposed methods to assemble an operation which, when applied locally, updates the respective data structures and can be replicated consistently.

Replication and the Database

So far we have tested the replication only with replicas that maintain their own database. This database, however, is not read from after the node has been defined to be a replica. When becoming a replica, a RacingEventService clears all its internal state and requests an initial load of data from its assigned master node.

When operations arriving at the replica are executed, many of them call features of the RacingEventService which also update the replica's local database. So far, we have not carried out any tests regarding the completeness or consistency of the resulting database image. Keep in mind that the replica may have started out as a system with its own particular database content. When receiving the initial load, this database content is not removed. Probably, a partial update and new inserts may result. Note also that the initial load does not at all update the replica's database. It is probably safe to assume that a replica's database will be in a questionable state that the replica should no longer rely on. It may serve as an accidental back-up in case the leaderboard score corrections have been lost on the master and happened to be replicated through an operation to the replica so that they updated the database there. But this is nothing planned or anything to rely on.

A larger discussion about this topic has to be started if we start to roll out replication into our production systems at a larger scale.

Open Issues

Besides the database issues on the replica which were discussed in the previous section, two more urgent issues exist with the current form of replication.

Dead Master

When a master with one or more active replicas dies (gets stopped, crashes, ...), the replicas will still be able to handle client requests. However, the replicas won't receive further updates from the master anymore.

When the master is started again, we currently have to load the races into the master's main memory again. This currently works by means of the regular connectors to the tracking providers, as if a live race were being tracked. We currently don't have a persistent local copy of the tracking data received from the tracking providers.

When a master starts, it has forgotten about its replicas (which is something we may consider to change in the future). Therefore, the replicator is not configured to listen to operations emitted by the RacingEventService yet. Only once a replica is explicitly added again to the master will the master start to emit the operations again to the message queuing system which then distributes them to the replicas again.

Therefore, when loading all tracking data again into the master, it depends on whether a replica was registered with the master again before starting the loading process. If not, the tracking data recorded between the master's death and the time when one replica is registered is lost for the replicas unless they choose to do a full initial load again.

Solution approaches may include a persistent representation of the replication topology as well as a heartbeat check performed by the replicas to see if the master is still alive, including some sort of probing for the master's recovery, followed by an automatic re-registration for the operation stream.

Dead Client

If a replica dies, the master won't know. This is no problem as long as there are still replicas surviving. But if the last replica has died, the master will not stop pushing the operations into the message queuing system. The master loses track of which replicas are still actively registered. We haven't yet measured the overhead of pushing the operations into the message queuing system, but it seems good to avoid it if it's not needed.

Therefore, a heartbeat mechanism performed by the clients to check for the master's liveness can also be used by the master to see which clients are still alive.

Preference for Immutable Value Objects

It is a general pattern in the application to try to keep objects immutable where it reasonably makes sense. Offering a setter for a field is a huge and often underestimated commitment. It allows clients to modify the object more or less at all times if no locking precautions are taken. Modifying an object also needs to be aligned with the replication concern, making sure that if the modification is relevant for the replicas, an operation needs to be implemented that carries this change on to the replicas consistently.

Object immutability has other advantages as well. The application is highly concurrent, using many concurrent threads. Setters always need to ensure that the object is still in a consistent state when the setter returns. If multiple setter calls are necessary to bring the object into a consistent state again, this needs to be hidden behind a common operation that carries out all necessary changes and takes the locking steps necessary to avoid concurrency issues.

A typical pitfall with mutable objects is to assume that an object is not visible to other threads when starting to modify it. Guaranteeing this is very hard. Mostly, it's not even the case, so other threads may be able to see an object that is under modification by one thread. In such a situation it is impossible for one thread to rely on its changes to prevail because other threads may perform competing changes using the setter on this object at all times.

Therefore, if it is by any means possible, making all fields final and passing all values necessary to the constructor is to be preferred over offering setters. This lets the object pass around freely without the risk of race conditions and transactional inconsistencies.