High availability for Event Hubs processors
Published: 2020-11-10 16:23:04

<p><strong>Azure</strong> <strong>Event Hubs</strong> with <strong>Stream Analytics</strong> is a powerful combination for quasi real time high throughput data processing.</p>
<p>It’s a great solution if you want fast reporting on live data and save your architecture from extra complexity.</p>
<p>Use case: Your service receives real time high volume user data I.E. website tracking or application telemetry or IoT.</p>
<p>The traditional process for generating reports would involve slow and painful steps such as:</p>
<ul>
<li>Sensors/source sending data to a queue</li>
<li>Service A constantly fetching messages from the queue</li>
<li>Service A storing data</li>
<li>Service B querying the data persistence layer</li>
<li>Service B processing/aggregating the data</li>
<li>Service B storing the result of the processor</li>
</ul>
<p>And, of course, the process involves development resources for the implementation and several machine resources.</p>
<h2 style="text-align: center;"><strong>The solution: Event Hub, Stream Analytics, Service Bus</strong></h2>
<p><img class=" size-full wp-image-38 aligncenter" src="https://rocketdotsciencedotblog.files.wordpress.com/2016/12/1.png" alt="1" width="124" height="780" /></p>
<p>This can be simplified by letting <strong>Azure</strong> take care of most of the steps:</p>
<ul>
<li>Sensors sending data to <strong>Event Hub</strong></li>
<li><strong>Stream Analytics</strong> aggregating/processing the data in time windows and storing the result or handing it over to a service.</li>
</ul>
<p>In diagram the result is sent to a <strong>Service Bus queue</strong> to be stored/processed by a service later, but <strong>Stream Analytics</strong> has the capability of storing directly to many data layers.</p>
<p><strong>Event Hub</strong> is a chronological stream storage, there is no concept of message locking or deletion and stores data in partitions allowing parallel access for multiple consumers. We don’t have to worry about removing messages once processed as our progress can be stored into <strong>checkpoints</strong> which record the point in time where we stopped processing to resume later.<a name="_GoBack"></a></p>
<p>Data is safely stored there and resilient and it will only be deleted after it passed the expiration set in the retention option, no manual/accidental deletion.</p>
<p>The architecture described in the diagram is the cheapest and simplest but it doesn’t take into account <strong>high availability</strong>; if the <strong>Azure Event Hub</strong> in our region has an outage we will lose all the messages during that period.</p>
<p>We can setup a failover stream in a different region to accept messages in case the primary hub is unavailable:</p>
<p><img class=" size-full wp-image-39 aligncenter" src="https://rocketdotsciencedotblog.files.wordpress.com/2016/12/2.png" alt="2" width="523" height="708" /></p>
<p>In this example our web application has to implement a failover logic to try and send messages to the primary and failover to the secondary during downtimes (I.E. in case of exceptions on sends).</p>
<p>We now have a good solution to prevent data loss from the web application side as the event will find its way in one or another hub and, assuming we set up the secondary in a different <strong>Azure</strong> region, we achieved geo-redundancy and we also qualify for the <strong>Azure</strong> 99.9% SLA making our messages resilient to alien invasions (I do not take responsibility if they target your two <strong>Azure</strong> clusters specifically).</p>
<p>Is it over? Are we happy our customers will never contact our support again? Of course not...</p>
<p>What happens then if the <strong>Stream Analytics</strong> service has an outage? Messages will be safe and sound but their journey will be delayed for the duration of the outage.</p>
<p>It is true that looking at the history of outages there has never been one involving <strong>Stream Analyitics</strong>, but we cannot rely on hope when it comes to the danger of pissing off customers.</p>
<p>Let’s try and solve this, too. On a Microsoft blog, the recommended solution is similar to the following:</p>
<p><img class=" size-full wp-image-40 aligncenter" src="https://rocketdotsciencedotblog.files.wordpress.com/2016/12/3.png" alt="3" width="566" height="658" /></p>
<p>Now we have a geo-redundancy of all the resources, but what happens when one <strong>Stream Analytics</strong> only has an outage? It’s unlikely that the whole region will be down, and we have no guarantee the services will be in the same fault/update domain. It might only happen in case a bomb lands on the whole data centre set, but in that eventuality we have bigger fish to fries so let's go back to it later.</p>
<p>If only <strong>Stream Analytics</strong> is down, the primary <strong>Event Hub</strong> and our web application will be unaware and will continue to store messages in the stream that will not be processed down in the chain; we will be unaware until we hear customer shouting.</p>
<p>We need to make sure a <strong>Stream Analytics</strong> works as failover and we could do so by checking the status of the service before sending messages... but:</p>
<ul>
<li>There is no public <strong>Azure</strong> API to check the status of the service</li>
<li>Our <strong>high throughput and performance critical</strong> application receiving millions of requests would have to delay the completion to make a new check request to a service on every call.</li>
</ul>
<p>So, even if we had the API, it would not be the best way to go. Maybe a separate background service checking the status, but we still risk of losing messages in the delay for our service to give us the status after the real outage occurred.</p>
<p>So let's try something else:</p>
<p><img class=" size-full wp-image-42 aligncenter" src="https://rocketdotsciencedotblog.files.wordpress.com/2016/12/x.png" alt="X.png" width="514" height="666" /></p>
<p>But now, both the SA will process the same messages and generate an output which is likely to be similar, but extremely unlikely to be exactly the same (Resources contention, time windows unaligned, et cetera).</p>
<p>This is still a good solution if you do not care about having potential duplicates in the data.</p>
<p>But if this is not your case, how do we know what’s in the primary and what’s in the secondary? How do we avoid duplication?</p>
<p>At the time I’m writing this there is no solution.</p>
<p><strong>Event Hubs</strong> works with checkpoints and <strong>Stream Analytics</strong> stores its checkpoint somewere and a good solution would be to share checkpoints across the two <strong>Stream Analytics </strong>as they will work in synergy and never overlap each other; but <strong>Azure</strong> does not support this feature at the moment.</p>
<p>So we go back to this original solution if we care about the precision of our data and we have to hope the weak ring (<strong>Stream Analytics</strong>) will not fail (which is unlikely as it is also based on <strong>Service Fabric</strong>.</p>
<p>I will stress the fact that we never lose data in this configuration so we might only delay the processing for the duration of the outage:</p>
<p><img class=" size-full wp-image-39 aligncenter" src="https://rocketdotsciencedotblog.files.wordpress.com/2016/12/2.png" alt="2" width="523" height="708" /></p>
<p>But yet we have another problem… What happens when the output <strong>Service Bus</strong> is down? I verified empirically that <strong>Stream Analytics</strong> has an internal cache for messages, but there is no guarantee the message will land eventually and the retention can be days, minutes or seconds (source:Microsoft). So we can send message identical copies to two geo-redundant <strong>Service Bus</strong> queues to prevent loss.</p>
<p>We would also like our services not to process twice the same message though, so what can we do?</p>
<p>My first idea was to add a unique identifier to the messages so our service can cache it and make sure it doesn’t process it twice (in a distributed cache/database if we want to make the service scalable over multiple instances). Great, so let’s ask <strong>Stream Analytics</strong> to generate a <strong>GUID</strong> for us and attach it to the two messages… failed. <strong>ASA</strong> cannot generate <strong>GUIDs</strong>.</p>
<p>OK, not a problem, I will just get the current date and time which should be unique enough if we have time windows… failed. <strong>ASA</strong> doesn’t have a <strong>GETDATE()</strong> or any time function so we need to rely on some data in our message to generate a sort of “hash” or unique identifier for the message.</p>
<p>I choose to use the combination of the first event date and the last event date so it could precisely define our window.</p>
<p><img class=" size-full wp-image-41 aligncenter" src="https://rocketdotsciencedotblog.files.wordpress.com/2016/12/4.png" alt="4" width="529" height="666" /></p>
<p>Now our architecture is as highly available as possible (still with a chance of delaying the results in case of a <strong>ASA</strong> outage) so we should be able to sleep reasonably well knowing our service is (almost) always available (bombs/aliens apart).</p>