Updated dataconsistency (markdown)

wiki/planning/businesscases/dataconsistency.md

@@ -17,7 +17,89 @@ In the "Data Consistency" project, we take the new approach to gain higher stabi
 
 ### Features
 
+* Start Phase
+  * Check correctness of start time based on several data sources: tracker, race committee, start detection
+  * Stabilize start detection algorithms
 
+* Completeness Checks
+  * Considered levels: leg, race, regatta, regatta series
+  * Add completeness state "complete/incomplete" on all levels
+  * Refined calculation-behavior of measures dependent on completeness on all levels
+  * Refined display-behavior in leaderboards and comparison charts dependent on completeness on all levels
+  * Visualize completeness state
+
+* Connected Checks
+  * Considered entities: GPS-track, mark-track, wind measurement
+  * Add connected state "connected/disconnected" on all entities
+  * Log state changes
+  * Visualize connected state
+  
+* Data Properties
+  * Visualize (or make accessible) data properties
+  * Sampling frequency of GPS-tracker (per competitor, on average)
+  * Number of active GPS-satellites
+  * Accuracies of measurements, especially GPS-positions and wind bearing & speed
+  
+* Outlier detection
+  * Considered entities: GPS-track, mark-track, wind measurement
+  * Add outlier state "in/out" on all entities
+  * Detect faulty jumps in GPS-tracks and mark-tracks by checking physical limits
+  * Detect faulty noise in wind measurements
+  
+* Correction of faulty raw data
+  * Focus: GPS-tracks, mark-tracks, wind measurements
+  * Apply interpolation and extrapolation where possible
+  
+* Robustness of Aggregate Calculations
+  * Remove outliers from calculation
+  * Remove incomplete entities from calculation (if possible, otherwise return "n/a")
+  * Adapt to missing data or changed sampling rate
+ 
+### Description
+
+The features listed above each address problems that have occurred frequently in the past while using the SAP sailing software during public events.
+
+#### Start Phase
+
+During the start phase of a race, it is a common problem to determine the actual start time of a race accurately. This is partly caused by time shifts of the various involved clocks, e.g. SAP server time, GPS time, tracking provider time, race committee time. Moreover, the manual start time input via the race committee app is subject to inaccuracies due to the user's reaction and software handling time. An automated start time detection based on GPS-tracks that takes into account other start time values shall lead to higher stability to avoid recording a race with a wrong start time.
+
+#### Completeness Checks
+
+There are various reasons why in terms of raw data a competitor does not finish a race, e.g. tracker lost power, tracker lost network, tracker lost satellites, SAP server lost connection to tracking provider server, competitor capsized, competitor decided to finish the race earlier. From the raw data most these cases cannot be distinguished. But based on comparisons of GPS-tracks, mark-tracks and times it can be determined whether the current leg has been finished. Based on the finished legs it can be decided whether the race has been regularly finished. And in the same manner, it can be derived whether regatta and regatta series have been regularly finished by a competitor.
+
+The completeness state of legs, races, regattas and regatta series has to be taken into account for calculated scores and measures. Moreover, the completeness state has to be taken into account and visualized in leaderboards and comparison charts so that the user is able to easily understand that scores and measures may have been influenced by "incomplete" state.
+
+#### Connected Checks
+
+Since raw data is transmitted using various network technologies that are vulnerable to connection dropouts, it is of interest for data processing and also the user of SAP sailing software to know when a connection is or was lost. The connection to GPS-trackers can unfortunately not be tested with alive-pings but needs to be monitored based on average transmission rate and time-out threshold. The connected state of GPS-trackers and wind measurement units shall be logged and visualized in the application so that equipment failures as a reason for non-availability or changes of data and/or measures becomes apparent to the eye of the user.
+
+#### Data Properties
+
+The base properties of data like GPS sampling rate & number of active satellites are interesting to expert users and shall be accessible via the UI. Furthermore, accuracies of measurement shall be optionally visualized (or at least accessible via the UI) to avoid misinterpretations of the visualized races and belonging measures, e.g. the general accuracy of GPS positions is up to 20 meters which should be at least optionally be visualized especially when trying to clarify race protests of sailors.
+
+#### Outlier detection
+
+GPS-tracks sometimes contain intermittent jumps (caused by changes in the GPS satellite reception of the tracker and shadowing from other boats) that should be detected in order to ignore them because such jumps can drastically change the aggregate measures and lead to severely wrong data display. Similarly, wind measurements can suffer from very strong noise that should be detected to be able to display plausible wind information and construct a meaningful advantage line.
+
+#### Correction of faulty raw data
+
+For time series data, like GPS-tracks and wind measurements, detected faulty raw data may be corrected using interpolation or extrapolation. The essential ingredient is assuming a numerical model for boat movement or wind pattern which is calibrated by live raw data (or recorded raw data) and used to bridge detected intermittent raw data faults. The duration of time that may be bridged shall be limited by a time threshold above which equipment failures should be visualized, as described in paragraph "Connected Checks".
+
+#### Robustness of Aggregate Calculations
+
+Aggregate calculations depend on a set of raw data and are influenced by intermittent faulty raw data as determined by outlier state, as well as, incomplete raw data as determined by the completeness state. In order to do automatically identify constellations in which parts of the raw data due to faultiness or incompleteness should be ignored, all aggregate calculations have to incorporate outlier state and completeness state to compute plausible results also in special raw data constellations.
+
+### Synergies
+
+The data consistency features belong closely to the calculation procedures implemented in the sailing analytics server.
+
+### Risks
+
+Since this project targets to increase the robustness of SAP sailing solutions, the risk of failures should be reduced as an outcome.
+
+### Prototype
+
+n/a
 
 
 ## Minutes from Initial Brainstorming Meeting (December 2, 2013)
@@ -61,7 +143,7 @@ In the "Data Consistency" project, we take the new approach to gain higher stabi
   * cleansing of races, if possible
   * include race states, e.g. DNF, into aggregate calculations, e.g. average speed
   * take into account #{common races} of competitors in aggregate calculation
-* Robustness of Aggergate Calculations
+* Robustness of Aggregate Calculations
   * on all levels: leg, race, regatta, regatta series
   * aggregate calculation can be corrupted by filtered outliers, lost connection
 * Quality based on Outliers and/or Dropouts