The goal is to provide an API for high availability I/O that can transparently provide recovery to clients, especially in an environment where the servers must also be highly available. The recovery is configurable to tailor specific client needs; we provide examples of ways to develop more complicated recovery mechanisms.
The main principle of the HA library is to provide drop-in replacements for all the “transmission” functions (e.g. MsgSend*()). The API lets a client choose specific connections that it would like to make highly available — all other connections will operate as ordinary connections.
Normally, when a server that the client is talking to fails, or if there's a transient network fault, the MsgSend*() functions return an error indicating that the connection ID (or file descriptor) is stale or invalid (EBADF).
In an HA-aware scenario, these transient faults are often recovered from almost immediately (on the server end), thus making the services available again. Unfortunately, clients using a standard I/O offering might not be available to benefit from this to the maximum unless they provide mechanisms to recover from these errors, and then retransmit the information/data, which often might involve a nontrivial rework of client programs.
By providing/achieving recovery inside the HA library itself, we can automatically take advantage of the HA-aware services that restart themselves or are automatically restarted or of the services that are provided in a transparent cluster/redundant way.
Since recovery itself is a connection-specific task, we allow clients to provide recovery mechanisms that will be used to restore connections when they fail. Irrecoverable errors are propagated back reliably so that any client that doesn't wish to recover will get the I/O library semantics that it expects.
The recovery mechanism can be anything ranging from a simple reopen of the connection to a more complex scenario that includes the retransmission/renegotiation of connection-specific information.
Normally, the MsgSend*() functions return EBADF/ESRCH when a connection is stale or closed on the server end (e.g. because the server dies). In many cases, the servers themselves return (e.g. they're restarted) and begin to offer the services properly almost immediately (in an HA scenario). Rather than merely terminate the message transmission with an error, in some cases it might be possible to perform recovery and continue with the message transmission.
The HA library functions that “cover” all the MsgSend*() varieties are designed to do exactly this. When a specific invocation of one of the MsgSend*() functions fails, a client-provided recovery function is called. This recovery function can attempt to reestablish the connection and return control to the HA library's MsgSend*() function. As long as the connection ID returned by the recovery function is the same as the old connection ID (which in many cases is easy to ensure via close/open/dup2() sequences), then the MsgSend*() functions can now attempt to retransmit the data.
If at any point the errors returned by MsgSend*() are anything other than EBADF/ESRCH, these errors are propagated back to the client. Note also that if the connection ID isn't an HA-aware connection ID, or if the client hasn't provided a recovery function or that function can't re-obtain the same connection ID, then the error is allowed to propagate back to the client to handle in whatever way it likes.
Clients can change their recovery functions. And since clients can also pass around “recovery/connection” information (which in turn is passed by the HA library to the recovery function), clients can construct complex recovery mechanisms that can be modified dynamically.
The client-side recovery library lets clients reconstruct the state required to continue the message transmission after reconnecting to either the same server or to a different server. The client is responsible for determining what constitutes the state that must be reconstructed and for performing this appropriately while the recovery function is called.
In addition to the cover functions for the standard MsgSend*() calls, the HA library provides clients with two “HA-awareness” functions that let you designate a connection as being HA-aware or similarly remove such a designation for an already HA-aware connection:
Remove a previously specified association between a recovery function and a connection. This makes the connection no longer HA-aware.
The HA library also provides the following cover functions whose behavior is essentially the same as the original functions being covered, but augmented slightly where the connections are also HA-aware:
Open a connection and attach it to the HA lib. These functions, in addition to calling the underlying open calls also make the connections HA-aware by calling ha_attach() automatically. As a result, using these calls is equivalent to calling open() or open64() and following that with a call to ha_attach().
Create a connection and attach it to the HA lib. These functions, in addition to calling the underlying creat calls also make the connections HA-aware by calling ha_attach() automatically. As a result, using these calls is equivalent to calling creat() or creat64() and following that with a call to ha_attach().
Create a connection using ConnectAttach() and attach it to the HA lib. These functions, in addition to calling the underlying ConnectAttach calls also make the connections HA-aware by calling ha_attach() automatically. As a result, using these calls is equivalent to calling ConnectAttach() or ConnectAttach_r() and following that with a call to ha_attach().
Detach an attached fd, then close the connection using ConnectDetach(). These functions, in addition to calling the underlying ConnectDetach calls also make the connections HA-aware by calling ha_attach() automatically. As a result, using these calls is equivalent to calling ConnectDetach() or ConnectDetach_r() and following that with a call to ha_attach().
Open a file stream and attach it to the HA lib. This function, in addition to calling the underlying fopen() call also makes connections HA-aware by calling ha_attach() automatically. As a result, using this call is equivalent to calling fopen() and following that with a call to ha_attach().
Detach an attached HA fd for a file stream, then close it. This function, in addition to calling the underlying fclose() call also makes connections HA-aware by calling ha_attach() automatically. As a result, using this call is equivalent to calling fclose() and following that with a call to ha_attach().
Detach an attached HA fd, then close it. This function, in addition to calling the underlying close() call also makes connections HA-aware by calling ha_attach() automatically. As a result, using this call is equivalent to calling close() and following that with a call to ha_attach().
Duplicate an HA connection. This function, in addition to calling the underlying dup() call also makes connections HA-aware by calling ha_attach() automatically. As a result, using this call is equivalent to calling dup() and following that with a call to ha_attach().
In addition to the covers, the library also provides these two convenience functions that reopen connections for recovery:
Here's a simple example of a client that has a connection open to a server and tries to read data from it. After reading from the descriptor, the client goes off to do something else (possibly causing a delay), and then returns to read again.
During this window of delay, the server might have died and returned, in which case the initial connection to the server (that has died) is now stale.
But since the connection has been made HA-aware, and a recovery function has been associated with it, the connection is able to reestablish itself.
In the following example, in addition to reopening the connection to the server, the client also reconstructs the state of the connection by seeking to the current file (connection) offset.
This example also shows how the client can maintain state information that can be used by the recovery functions to return to a previously check-pointed state before the failure, so that the message transmission can continue properly.
Cake was small and soft. Sweetness just nice. Able to choose delivery on the same day. Even though there were delay on delivery, they called me to inform for the delay.
Unfortunately We do not have at this moment. Do check at a later date as we are doing r&d for eggless cake at present. Thank you !	
Hi, can i request to arrange to deliver the cake on Monday (28/08/17) at 7pm or after 7pm if i would like to order the cake?
Hi...would like to check regarding my order today. The cake will be delivered on 7pm as per my request right? Thank you very much!	
I really cant promise your cake can reach your place by 7pm . As QX will collect your cake ard 5.30pm to 6.00pm. We will inform the delivery guy upon collection.