improved Igtimi Wiki docs

java/com.sap.sailing.domain.igtimiadapter.gateway/src/com/sap/sailing/domain/igtimiadapter/gateway/impl/RiotResourcesResource.java

@@ -61,7 +61,6 @@ public class RiotResourcesResource extends AbstractRiotServerResource {
                 : serialNumbers;
         final Iterable<DataAccessWindow> daws = getDataAccessWindowsReadableBySubject(MultiTimeRange.of(TimeRange.create(null, null)), serialNumbersToUse);
         final Map<String, MultiTimeRange> multiTimeRangesPerDeviceSerialNumber = getTimeRangesByDeviceSerialNumber(daws);
-
         for (final String serialNumber : serialNumbersToUse) {
             final Device device = getRiotService().getDeviceBySerialNumber(serialNumber);
             if (device != null && subject.isPermitted(

wiki/info/landscape/igtimi-riot.md

@@ -77,12 +77,21 @@ Enhancing security for device authentication is described in [bug 6074](https://
 
 ### Replication
 
-The ``RiotServerImpl`` class is a ``Replicable`` object. A Sailing Analytics replica will replicate this by default. The processes in the replica set shall be registered with the Network Load Balancer (NLB) target group to which the listener for the Riot port (usually 6000) forwards. The replicated state consists of the ``Device`` and ``DataAccessWindow`` entities. Creating, updating and deleting these entities happens through replicable operations, with the usual pattern of implementing these operations as lambda expressions that comply with the ``OperationWithResult`` interface and as such are serializable.
+The ``RiotServerImpl`` class is a ``Replicable`` object. A Sailing Analytics replica will replicate this by default. The processes in the replica set shall be registered with the Network Load Balancer (NLB) target group to which the listener for the Riot port (usually 6000) forwards. The replicated state consists of the ``Device`` and ``DataAccessWindow`` entities. Creating, updating and deleting these entities happens through replicable operations, with the usual pattern of implementing these operations as lambda expressions that comply with the ``OperationWithResult`` interface and as such are serializable. The replicable state does *not* contain the actual messages received from the devices. Those are stored in the database, and as usual, only the database of the primary/master process is to be trusted.
 
 Messages received from a device are turned into a replicable operation invoking the ``internalNotifyListeners`` method. This way, messages from the devices are replicated across the replica set. If a message is received by the primary/master process, it is stored in the replica set's database and propagated to all replicas. While replicas will also store the message received from the primary/master in their "phony" database, the key point on the replicas is that they, like the primary/master will also forward the message to any live web socket connection that is currently known to the replica.
 
 If a message is originally received by a replica, it will be stored in the replica's phony database and will be forwarded to the primary/master process for persistent storage in the replica set's "real" database, as well as for forwarding to all other replicas and the primary/master's live web socket connections.
 
+REST API requests can be of two types
+
+- those that work with the ``Device`` and ``DataAccessWindow`` entities
+- those that work with stored data
+
+The first type of requests can be handled on either a primary or a replica process. The replication architecture will ensure eventual consistency and persistence on the primary.
+
+The second type of request will have to use the database, and that can be trusted only on the primary/master process. Therefore, when such a request hits a replica, the replica will turn around and become a REST API client itself and request the data from the primary/master, making sure to set the ``X-SAPSSE-Forward-Request-To`` HTTP header to the value ``master`` which will instruct a load balancer to send this request to the primary and not a replica. The response is then returned by the replica to its client.
+
 Taking this together with the connection handling that uses hand shakes and automatic re-connects in case of broken connections (both, for the devices, as well as for client-issues live data web socket connections), we get a highly available set up. Should the primary/master process become unavailable, devices and web socket clients will fail over to any remaining replica. The devices will re-connect to the NLB and will get connected to one of the replicas. Likewise, live web socket connections will fail over through the regular public target group of the ``wind`` replica set and re-connect to one of the replicas. If there is exactly one replica in the replica set, this will lead to quick recovery because devices will send to the one replica to which the live web socket clients have then also failed over to. In case of multiple replicas, in the worst case all devices send to one replica, and all live web socket connections use different replicas. The replication messages on the replica receiving data from the devices will be queued until the primary/master becomes available again; only then will the data be forwarded to the other replicas.
 
 The failure of one out of multiple replicas is less problematic. Devices and live web socket connection clients will fail over to another replica or the primary/master, and all replication will still succeed.