# rdma_cm

## NAME



           rdma_cm - RDMA communication manager.
    


## SYNOPSIS



           **#include <rdma/rdma_cma.h>**
    


## DESCRIPTION



           Used to establish communication over RDMA transports.
    


## NOTES



           The RDMA CM is a communication manager used to setup reliable,
           connected and unreliable datagram data transfers.  It provides an
           RDMA transport neutral interface for establishing connections.
           The API concepts are based on sockets, but adapted for queue pair
           (QP) based semantics: communication must be over a specific RDMA
           device, and data transfers are message based.
    
           The RDMA CM can control both the QP and communication management
           (connection setup / teardown) portions of an RDMA API, or only the
           communication management piece.  It works in conjunction with the
           verbs API defined by the libibverbs library.  The libibverbs
           library provides the underlying interfaces needed to send and
           receive data.
    
           The RDMA CM can operate asynchronously or synchronously.  The mode
           of operation is controlled by the user through the use of the
           rdma_cm event channel parameter in specific calls.  If an event
           channel is provided, an rdma_cm identifier will report its event
           data (results of connecting, for example), on that channel.  If a
           channel is not provided, then all rdma_cm operations for the
           selected rdma_cm identifier will block until they complete.
    
           The RDMA CM gives an option to different libibverbs providers to
           advertise and use various specific to that provider QP
           configuration options. This functionality is called ECE (enhanced
           connection establishment).
    


## RDMA VERBS



           The rdma_cm supports the full range of verbs available through the
           libibverbs library and interfaces.  However, it also provides
           wrapper functions for some of the more commonly used verbs
           funcationality.  The full set of abstracted verb calls are:
    
           rdma_reg_msgs  - register an array of buffers for sending and
           receiving
    
           rdma_reg_read  - registers a buffer for RDMA read operations
    
           rdma_reg_write - registers a buffer for RDMA write operations
    
           rdma_dereg_mr  - deregisters a memory region
    
           rdma_post_recv  - post a buffer to receive a message
    
           rdma_post_send  - post a buffer to send a message
    
           rdma_post_read  - post an RDMA to read data into a buffer
    
           rdma_post_write - post an RDMA to send data from a buffer
    
           rdma_post_recvv  - post a vector of buffers to receive a message
    
           rdma_post_sendv  - post a vector of buffers to send a message
    
           rdma_post_readv  - post a vector of buffers to receive an RDMA
           read
    
           rdma_post_writev - post a vector of buffers to send an RDMA write
    
           rdma_post_ud_send - post a buffer to send a message on a UD QP
    
           rdma_get_send_comp - get completion status for a send or RDMA
           operation
    
           rdma_get_recv_comp - get information about a completed receive
    


## CLIENT OPERATION



           This section provides a general overview of the basic operation
           for the active, or client, side of communication.  This flow
           assume asynchronous operation with low level call details shown.
           For synchronous operation, calls to rdma_create_event_channel,
           rdma_get_cm_event, rdma_ack_cm_event, and
           rdma_destroy_event_channel would be eliminated.  Abstracted calls,
           such as rdma_create_ep encapsulate several of these calls under a
           single API.  Users may also refer to the example applications for
           code samples.  A general connection flow would be:
    
           rdma_getaddrinfo
                  retrieve address information of the destination
    
           rdma_create_event_channel
                  create channel to receive events
    
           rdma_create_id
                  allocate an rdma_cm_id, this is conceptually similar to a
                  socket
    
           rdma_resolve_addr
                  obtain a local RDMA device to reach the remote address
    
           rdma_get_cm_event
                  wait for RDMA_CM_EVENT_ADDR_RESOLVED event
    
           rdma_ack_cm_event
                  ack event
    
           rdma_create_qp
                  allocate a QP for the communication
    
           rdma_resolve_route
                  determine the route to the remote address
    
           rdma_get_cm_event
                  wait for RDMA_CM_EVENT_ROUTE_RESOLVED event
    
           rdma_ack_cm_event
                  ack event
    
           rdma_connect
                  connect to the remote server
    
           rdma_get_cm_event
                  wait for RDMA_CM_EVENT_ESTABLISHED event
    
           rdma_ack_cm_event
                  ack event
    
           Perform data transfers over connection
    
           rdma_disconnect
                  tear-down connection
    
           rdma_get_cm_event
                  wait for RDMA_CM_EVENT_DISCONNECTED event
    
           rdma_ack_cm_event
                  ack event
    
           rdma_destroy_qp
                  destroy the QP
    
           rdma_destroy_id
                  release the rdma_cm_id
    
           rdma_destroy_event_channel
                  release the event channel
    
           rdma_freeaddrinfo
                  release the list of rdma_addrinfo structures
    
           rdma_set_local_ece
                  set desired ECE options
    
           An almost identical process is used to setup unreliable datagram
           (UD) communication between nodes.  No actual connection is formed
           between QPs however, so disconnection is not needed.
    
           Although this example shows the client initiating the disconnect,
           either side of a connection may initiate the disconnect.
    


## SERVER OPERATION



           This section provides a general overview of the basic operation
           for the passive, or server, side of communication.  A general
           connection flow would be:
    
           rdma_create_event_channel
                  create channel to receive events
    
           rdma_create_id
                  allocate an rdma_cm_id, this is conceptually similar to a
                  socket
    
           rdma_bind_addr
                  set the local port number to listen on
    
           rdma_listen
                  begin listening for connection requests
    
           rdma_get_cm_event
                  wait for RDMA_CM_EVENT_CONNECT_REQUEST event with a new
                  rdma_cm_id
    
           rdma_create_qp
                  allocate a QP for the communication on the new rdma_cm_id
    
           rdma_accept
                  accept the connection request
    
           rdma_ack_cm_event
                  ack event
    
           rdma_get_cm_event
                  wait for RDMA_CM_EVENT_ESTABLISHED event
    
           rdma_ack_cm_event
                  ack event
    
           Perform data transfers over connection
    
           rdma_get_cm_event
                  wait for RDMA_CM_EVENT_DISCONNECTED event
    
           rdma_ack_cm_event
                  ack event
    
           rdma_disconnect
                  tear-down connection
    
           rdma_destroy_qp
                  destroy the QP
    
           rdma_destroy_id
                  release the connected rdma_cm_id
    
           rdma_destroy_id
                  release the listening rdma_cm_id
    
           rdma_destroy_event_channel
                  release the event channel
    
           rdma_get_remote_ece
                  get ECe options sent by the client
    
           rdma_set_local_ece
                  set desired ECE options
    


## RETURN CODES



           =  0   success
    
           = -1   error - see errno for more details
    
           Most librdmacm functions return 0 to indicate success, and a -1
           return value to indicate failure.  If a function operates
           asynchronously, a return value of 0 means that the operation was
           successfully started.  The operation could still complete in
           error; users should check the status of the related event.  If the
           return value is -1, then errno will contain additional information
           regarding the reason for the failure.
    
           Prior versions of the library would return -errno and not set
           errno for some cases related to ENOMEM, ENODEV, ENODATA, EINVAL,
           and EADDRNOTAVAIL codes. Applications that want to check these
           codes and have compatibility with prior library versions must
           manually set errno to the negative of the return code if it is <
           -1.
    


* * *