sailing-analytics/wiki/general-information.md

General Project Information

The Situation

Sailing races can be difficult to understand and follow for an audience as they happen live, out on the water. The trouble starts with figuring out who's in the lead and doesn't stop with understanding the impact of the current race's ranks on the overall regatta standings which are subject to non-trivial scoring rules.

Project Description

Using GPS trackers and wind sensors, it is possible to collect real-world data about a sailing race and analyze and visualize the race live. These visualizations can be used by a commentator while talking to an audience, directly by the audience over online channels on various devices, as well as by the sailors who can assess their performance, also relative to their competitors, in replay mode.

Objective

We're doing this in the scope of SAP's sport sponsorship efforts, also in order to show technology leadership and to combine an exciting technology showcase with our sponsorship.

Approach

The project started shortly before Kiel Week 2011, in May, leaving little time for planning and developing a first prototype to show in Kiel. The SAP Center of Excellence (CoE) supported (and still supports) the project by skilled development resources and good connections to flexible external hosting providers assisting in the operations of the application. We started from scratch with a Java application using the Equinox OSGi implementation as its runtime infrastructure. The first bundles to be developed were the sailing domain abstractions (boats, boat classes, sailors, courses, races, events, wind and the geometry model) and the connectors for GPS and wind data connectivity. The web front end was initially developed using Python, connecting to the Java backend through HTTP. We chose a NoSQL database (MongoDB) to store data not otherwise retrievable from the tracking provider later, such as the wind data and administrative information. JUnit tests for the back end, a central Maven build with Hudson/Jenkins integration, a GIT repository for change management and the use of Bugzilla laid the technical ground for our development environment. Given our microscopic team size that varied between one and three developers, we skipped daily scrums and used on-demand ad-hoc chats, also through IM systems, as a replacement. Our deadlines for a release are rigid: the sailing regattas at which we present the solution.

After successfully demonstrating the prototype at Kiel Week, the project entered a second phase with major architecture clean-ups. We removed the Python server and replaced it by the use of the Google Web Toolkit (GWT). We were particularly impressed by the client-side performance and flexibility of its powerful controls such as tables and list displays. Web designers, though, need some time to find their way into the world of GWT. We ported the leaderboard functionality from Python to GWT and successfully presented it in its new shape at the International German Championship 2011 in Travemuende and at the Championship of the Champions later that year in Hamburg.

What followed were improvements in the analytical treatment of GPS data and the integration of a flexible Google Map-based race visualization. We added algorithms to identify and count the various sailing maneuvers for the competitors (tacks, jibes, penalty circles, etc.) and implemented a wind estimation algorithm that analyzes the boats' courses and by means of clustering them by tack and leg type infers the most probable wind direction together with a confidence value. A first pragmatic approach to replicating the so far UDP-provided wind data through an HTTP push channel to additional server instances running on spectator boats and serving the solution to mobile devices on these boats rounded off the release presented at an SAP-internal sailing event. When this release with only minor extensions was presented at a training regatta in S'Arenal, Mallorca, our server got under heavy load because the URL had virally spread, and sailors showed high interest in the solution. We saved the event with some ad-hoc performance patches. The event taught us the limitations of dynamically re-computing various key figures each time a tiny bit of raw data changed and brought us to think more about smarter cache invalidation strategies. Additionally, we now started to work on server instance replication techniques that will help us leverage the cloud to scale our solution out during the peak loads experienced during live events.

The Java In-Memory Challenge

More or less co-incidentally we chose an architecture in which we keep all relevant race data in main memory. We applied a few techniques to avoid excessive amounts of object headers to consume too much heap space while preserving the API "look & feel" of individual, fine-grained objects for things like speed, bearing, time points, distances or positions. With this, all tracking data of Kiel Week and several other events easily fit into the 16GB of heap space we allocate for our production instance. Key technical challenges remain the performance of live calculations of complex key figures with quickly changing tracking data because it makes caching and cache invalidation tough topics scaling out the server with its in-memory state while allowing client sessions to run modifying transactions on each server instance achieving great performance with a Google Maps integration when the number of boats to display and the expectations regarding the quality of the boat displays rise.