# ovn-architecture

## NAME



           ovn-architecture - Open Virtual Network architecture
    


## DESCRIPTION



           OVN, the Open Virtual Network, is a system to support logical
           network abstraction in virtual machine and container environments.
           OVN complements the existing capabilities of OVS to add native
           support for logical network abstractions, such as logical L2 and
           L3 overlays and security groups. Services such as DHCP are also
           desirable features. Just like OVS, OVN’s design goal is to have a
           production-quality implementation that can operate at significant
           scale.
    
           A physical network comprises physical wires, switches, and
           routers. A _virtual network_ extends a physical network into a
           hypervisor or container platform, bridging VMs or containers into
           the physical network. An OVN _logical network_ is a network
           implemented in software that is insulated from physical (and thus
           virtual) networks by tunnels or other encapsulations. This allows
           IP and other address spaces used in logical networks to overlap
           with those used on physical networks without causing conflicts.
           Logical network topologies can be arranged without regard for the
           topologies of the physical networks on which they run. Thus, VMs
           that are part of a logical network can migrate from one physical
           machine to another without network disruption. See **Logical**
           **Networks** , below, for more information.
    
           The encapsulation layer prevents VMs and containers connected to a
           logical network from communicating with nodes on physical
           networks. For clustering VMs and containers, this can be
           acceptable or even desirable, but in many cases VMs and containers
           do need connectivity to physical networks. OVN provides multiple
           forms of _gateways_ for this purpose. See **Gateways** , below, for more
           information.
    
           An OVN deployment consists of several components:
    
                  •      A _Cloud Management System_ (_CMS_), which is OVN’s
                         ultimate client (via its users and administrators).
                         OVN integration requires installing a CMS-specific
                         plugin and related software (see below). OVN
                         initially targets OpenStack as CMS.
    
                         We generally speak of ``the’’ CMS, but one can
                         imagine scenarios in which multiple CMSes manage
                         different parts of an OVN deployment.
    
                  •      An OVN Database physical or virtual node (or,
                         eventually, cluster) installed in a central
                         location.
    
                  •      One or more (usually many) _hypervisors_. Hypervisors
                         must run Open vSwitch and implement the interface
                         described in **Documentation/topics/integration.rst** in
                         the Open vSwitch source tree. Any hypervisor
                         platform supported by Open vSwitch is acceptable.
    
                  •      Zero or more _gateways_. A gateway extends a tunnel-
                         based logical network into a physical network by
                         bidirectionally forwarding packets between tunnels
                         and a physical Ethernet port. This allows non-
                         virtualized machines to participate in logical
                         networks. A gateway may be a physical host, a
                         virtual machine, or an ASIC-based hardware switch
                         that supports the vtep(5) schema.
    
                         Hypervisors and gateways are together called
                         _transport node_ or _chassis_.
    
           The diagram below shows how the major components of OVN and
           related software interact. Starting at the top of the diagram, we
           have:
    
                  •      The Cloud Management System, as defined above.
    
                  •      The _OVN/CMS Plugin_ is the component of the CMS that
                         interfaces to OVN. In OpenStack, this is a Neutron
                         plugin. The plugin’s main purpose is to translate
                         the CMS’s notion of logical network configuration,
                         stored in the CMS’s configuration database in a CMS-
                         specific format, into an intermediate representation
                         understood by OVN.
    
                         This component is necessarily CMS-specific, so a new
                         plugin needs to be developed for each CMS that is
                         integrated with OVN. All of the components below
                         this one in the diagram are CMS-independent.
    
                  •      The _OVN Northbound Database_ receives the
                         intermediate representation of logical network
                         configuration passed down by the OVN/CMS Plugin. The
                         database schema is meant to be ``impedance matched’’
                         with the concepts used in a CMS, so that it directly
                         supports notions of logical switches, routers, ACLs,
                         and so on. See ovn-nb(5) for details.
    
                         The OVN Northbound Database has only two clients:
                         the OVN/CMS Plugin above it and **ovn-northd** below it.
    
                  •      ovn-northd(8) connects to the OVN Northbound
                         Database above it and the OVN Southbound Database
                         below it. It translates the logical network
                         configuration in terms of conventional network
                         concepts, taken from the OVN Northbound Database,
                         into logical datapath flows in the OVN Southbound
                         Database below it.
    
                  •      The _OVN Southbound Database_ is the center of the
                         system. Its clients are ovn-northd(8) above it and
                         ovn-controller(8) on every transport node below it.
    
                         The OVN Southbound Database contains three kinds of
                         data: _Physical Network_ (PN) tables that specify how
                         to reach hypervisor and other nodes, _Logical Network_
                         (LN) tables that describe the logical network in
                         terms of ``logical datapath flows,’’ and _Binding_
                         tables that link logical network components’
                         locations to the physical network. The hypervisors
                         populate the PN and Port_Binding tables, whereas
                         ovn-northd(8) populates the LN tables.
    
                         OVN Southbound Database performance must scale with
                         the number of transport nodes. This will likely
                         require some work on ovsdb-server(1) as we encounter
                         bottlenecks. Clustering for availability may be
                         needed.
    
           The remaining components are replicated onto each hypervisor:
    
                  •      ovn-controller(8) is OVN’s agent on each hypervisor
                         and software gateway. Northbound, it connects to the
                         OVN Southbound Database to learn about OVN
                         configuration and status and to populate the PN
                         table and the **Chassis** column in **Binding** table with
                         the hypervisor’s status. Southbound, it connects to
                         ovs-vswitchd(8) as an OpenFlow controller, for
                         control over network traffic, and to the local
                         ovsdb-server(1) to allow it to monitor and control
                         Open vSwitch configuration.
    
                  •      ovs-vswitchd(8) and ovsdb-server(1) are conventional
                         components of Open vSwitch.
    
                                             CMS
                                              |
                                              |
                                  +-----------|-----------+
                                  |           |           |
                                  |     OVN/CMS Plugin    |
                                  |           |           |
                                  |           |           |
                                  |   OVN Northbound DB   |
                                  |           |           |
                                  |           |           |
                                  |       ovn-northd      |
                                  |           |           |
                                  +-----------|-----------+
                                              |
                                              |
                                    +-------------------+
                                    | OVN Southbound DB |
                                    +-------------------+
                                              |
                                              |
                           +------------------+------------------+
                           |                  |                  |
             HV 1          |                  |    HV n          |
           +---------------|---------------+  .  +---------------|---------------+
           |               |               |  .  |               |               |
           |        ovn-controller         |  .  |        ovn-controller         |
           |         |          |          |  .  |         |          |          |
           |         |          |          |     |         |          |          |
           |  ovs-vswitchd   ovsdb-server  |     |  ovs-vswitchd   ovsdb-server  |
           |                               |     |                               |
           +-------------------------------+     +-------------------------------+
    
       **Information Flow in OVN**
           Configuration data in OVN flows from north to south. The CMS,
           through its OVN/CMS plugin, passes the logical network
           configuration to **ovn-northd** via the northbound database. In turn,
           **ovn-northd** compiles the configuration into a lower-level form and
           passes it to all of the chassis via the southbound database.
    
           Status information in OVN flows from south to north. OVN currently
           provides only a few forms of status information. First, **ovn-northd**
           populates the **up** column in the northbound **Logical_Switch_Port**
           table: if a logical port’s **chassis** column in the southbound
           **Port_Binding** table is nonempty, it sets **up** to **true** , otherwise to
           **false**. This allows the CMS to detect when a VM’s networking has
           come up.
    
           Second, OVN provides feedback to the CMS on the realization of its
           configuration, that is, whether the configuration provided by the
           CMS has taken effect. This feature requires the CMS to participate
           in a sequence number protocol, which works the following way:
    
                  1.  When the CMS updates the configuration in the
                      northbound database, as part of the same transaction,
                      it increments the value of the **nb_cfg** column in the
                      **NB_Global** table. (This is only necessary if the CMS
                      wants to know when the configuration has been
                      realized.)
    
                  2.  When **ovn-northd** updates the southbound database based
                      on a given snapshot of the northbound database, it
                      copies **nb_cfg** from northbound **NB_Global** into the
                      southbound database **SB_Global** table, as part of the
                      same transaction. (Thus, an observer monitoring both
                      databases can determine when the southbound database is
                      caught up with the northbound.)
    
                  3.  After **ovn-northd** receives confirmation from the
                      southbound database server that its changes have
                      committed, it updates **sb_cfg** in the northbound
                      **NB_Global** table to the **nb_cfg** version that was pushed
                      down. (Thus, the CMS or another observer can determine
                      when the southbound database is caught up without a
                      connection to the southbound database.)
    
                  4.  The **ovn-controller** process on each chassis receives the
                      updated southbound database, with the updated **nb_cfg**.
                      This process in turn updates the physical flows
                      installed in the chassis’s Open vSwitch instances. When
                      it receives confirmation from Open vSwitch that the
                      physical flows have been updated, it updates **nb_cfg** in
                      its own **Chassis_Private** record in the southbound
                      database. (OVN versions before v20.09.0 used the **nb_cfg**
                      column from the **Chassis** table. Although this column
                      still exists, it is deprecated in favor of the same
                      column in **Chassis_Private** and no longer updated.)
    
                  5.  **ovn-northd** monitors the **nb_cfg** column in all of the
                      **Chassis_Private** records in the southbound database. It
                      keeps track of the minimum value among all the records
                      and copies it into the **hv_cfg** column in the northbound
                      **NB_Global** table. (Thus, the CMS or another observer can
                      determine when all of the hypervisors have caught up to
                      the northbound configuration.)
    
       **Chassis Setup**
           Each chassis in an OVN deployment must be configured with an Open
           vSwitch bridge dedicated for OVN’s use, called the _integration_
           _bridge_. System startup scripts may create this bridge prior to
           starting **ovn-controller** if desired. If this bridge does not exist
           when ovn-controller starts, it will be created automatically with
           the default configuration suggested below. The ports on the
           integration bridge include:
    
                  •      On any chassis, tunnel ports that OVN uses to
                         maintain logical network connectivity.
                         **ovn-controller** adds, updates, and removes these
                         tunnel ports.
    
                  •      On a hypervisor, any VIFs that are to be attached to
                         logical networks. For instances connected through
                         software emulated ports such as TUN/TAP or VETH
                         pairs, the hypervisor itself will normally create
                         ports and plug them into the integration bridge. For
                         instances connected through representor ports,
                         typically used with hardware offload, the
                         **ovn-controller** may on CMS direction consult a VIF
                         plug provider for representor port lookup and plug
                         them into the integration bridge (please refer to
                         **Documentation/topics/vif-plug-providers/vif-plug-providers.rst**
                          for more information). In both cases the
                         conventions described in
                         **Documentation/topics/integration.rst** in the Open
                         vSwitch source tree is followed to ensure mapping
                         between OVN logical port and VIF. (This is pre-
                         existing integration work that has already been done
                         on hypervisors that support OVS.)
    
                  •      On a gateway, the physical port used for logical
                         network connectivity. System startup scripts add
                         this port to the bridge prior to starting
                         **ovn-controller**. This can be a patch port to another
                         bridge, instead of a physical port, in more
                         sophisticated setups.
    
           Other ports should not be attached to the integration bridge. In
           particular, physical ports attached to the underlay network (as
           opposed to gateway ports, which are physical ports attached to
           logical networks) must not be attached to the integration bridge.
           Underlay physical ports should instead be attached to a separate
           Open vSwitch bridge (they need not be attached to any bridge at
           all, in fact).
    
           The integration bridge should be configured as described below.
           The effect of each of these settings is documented in
           ovs-vswitchd.conf.db(5):
    
                  **fail-mode=secure**
                         Avoids switching packets between isolated logical
                         networks before **ovn-controller** starts up. See
                         **Controller Failure Settings** in ovs-vsctl(8) for more
                         information.
    
                  **other-config:disable-in-band=true**
                         Suppresses in-band control flows for the integration
                         bridge. It would be unusual for such flows to show
                         up anyway, because OVN uses a local controller (over
                         a Unix domain socket) instead of a remote
                         controller. It’s possible, however, for some other
                         bridge in the same system to have an in-band remote
                         controller, and in that case this suppresses the
                         flows that in-band control would ordinarily set up.
                         Refer to the documentation for more information.
    
           The customary name for the integration bridge is **br-int** , but
           another name may be used.
    
       **Logical Networks**
           Logical network concepts in OVN include _logical switches_ and
           _logical routers_ , the logical version of Ethernet switches and IP
           routers, respectively. Like their physical cousins, logical
           switches and routers can be connected into sophisticated
           topologies. Logical switches and routers are ordinarily purely
           logical entities, that is, they are not associated or bound to any
           physical location, and they are implemented in a distributed
           manner at each hypervisor that participates in OVN.
    
           _Logical switch ports_ (LSPs) are points of connectivity into and
           out of logical switches. There are many kinds of logical switch
           ports. The most ordinary kind represent VIFs, that is, attachment
           points for VMs or containers. A VIF logical port is associated
           with the physical location of its VM, which might change as the VM
           migrates. (A VIF logical port can be associated with a VM that is
           powered down or suspended. Such a logical port has no location and
           no connectivity.)
    
           _Logical router ports_ (LRPs) are points of connectivity into and
           out of logical routers. A LRP connects a logical router either to
           a logical switch or to another logical router. Logical routers
           only connect to VMs, containers, and other network nodes
           indirectly, through logical switches.
    
           Logical switches and logical routers have distinct kinds of
           logical ports, so properly speaking one should usually talk about
           logical switch ports or logical router ports. However, an
           unqualified ``logical port’’ usually refers to a logical switch
           port.
    
           When a VM sends a packet to a VIF logical switch port, the Open
           vSwitch flow tables simulate the packet’s journey through that
           logical switch and any other logical routers and logical switches
           that it might encounter. This happens without transmitting the
           packet across any physical medium: the flow tables implement all
           of the switching and routing decisions and behavior. If the flow
           tables ultimately decide to output the packet at a logical port
           attached to another hypervisor (or another kind of transport
           node), then that is the time at which the packet is encapsulated
           for physical network transmission and sent.
    
         _Logical Switch Port Types_
    
           OVN supports a number of kinds of logical switch ports. VIF ports
           that connect to VMs or containers, described above, are the most
           ordinary kind of LSP. In the OVN northbound database, VIF ports
           have an empty string for their **type**. This section describes some
           of the additional port types.
    
           A **router** logical switch port connects a logical switch to a
           logical router, designating a particular LRP as its peer.
    
           A **localnet** logical switch port bridges a logical switch to a
           physical VLAN. A logical switch may have one or more **localnet**
           ports. Such a logical switch is used in two scenarios:
    
                  •      With one or more **router** logical switch ports, to
                         attach L3 gateway routers and distributed gateways
                         to a physical network.
    
                  •      With one or more VIF logical switch ports, to attach
                         VMs or containers directly to a physical network. In
                         this case, the logical switch is not really logical,
                         since it is bridged to the physical network rather
                         than insulated from it, and therefore cannot have
                         independent but overlapping IP address namespaces,
                         etc. A deployment might nevertheless choose such a
                         configuration to take advantage of the OVN control
                         plane and features such as port security and ACLs.
    
           When a logical switch contains multiple **localnet** ports, the
           following is assumed.
    
                  •      Each chassis has a bridge mapping for one of the
                         **localnet** physical networks only.
    
                  •      To facilitate interconnectivity between VIF ports of
                         the switch that are located on different chassis
                         with different physical network connectivity, the
                         fabric implements L3 routing between these adjacent
                         physical network segments.
    
           Note: nothing said above implies that a chassis cannot be plugged
           to multiple physical networks as long as they belong to different
           switches.
    
           A **localport** logical switch port is a special kind of VIF logical
           switch port. These ports are present in every chassis, not bound
           to any particular one. Traffic to such a port will never be
           forwarded through a tunnel, and traffic from such a port is
           expected to be destined only to the same chassis, typically in
           response to a request it received. OpenStack Neutron uses a
           **localport** port to serve metadata to VMs. A metadata proxy process
           is attached to this port on every host and all VMs within the same
           network will reach it at the same IP/MAC address without any
           traffic being sent over a tunnel. For further details, see the
           OpenStack documentation for networking-ovn.
    
           LSP types **vtep** and **l2gateway** are used for gateways. See **Gateways** ,
           below, for more information.
    
         _Implementation Details_
    
           These concepts are details of how OVN is implemented internally.
           They might still be of interest to users and administrators.
    
           _Logical datapaths_ are an implementation detail of logical networks
           in the OVN southbound database. **ovn-northd** translates each logical
           switch or router in the northbound database into a logical
           datapath in the southbound database **Datapath_Binding** table.
    
           For the most part, **ovn-northd** also translates each logical switch
           port in the OVN northbound database into a record in the
           southbound database **Port_Binding** table. The latter table
           corresponds roughly to the northbound **Logical_Switch_Port** table.
           It has multiple types of logical port bindings, of which many
           types correspond directly to northbound LSP types. LSP types
           handled this way include VIF (empty string), **localnet** , **localport** ,
           **vtep** , and **l2gateway**.
    
           The **Port_Binding** table has some types of port binding that do not
           correspond directly to logical switch port types. The common is
           **patch** port bindings, known as _logical patch ports_. These port
           bindings always occur in pairs, and a packet that enters on either
           side comes out on the other. **ovn-northd** connects logical switches
           and logical routers together using logical patch ports.
    
           Port bindings with types **vtep** , **l2gateway** , **l3gateway** , and
           **chassisredirect** are used for gateways. These are explained in
           **Gateways** , below.
    
       **Gateways**
           Gateways provide limited connectivity between logical networks and
           physical ones. They can also provide connectivity between
           different OVN deployments. This section will focus on the former,
           and the latter will be described in details in section **OVN**
           **Deployments Interconnection**.
    
           OVN support multiple kinds of gateways.
    
         _VTEP Gateways_
    
           A ``VTEP gateway’’ connects an OVN logical network to a physical
           (or virtual) switch that implements the OVSDB VTEP schema that
           accompanies Open vSwitch. (The ``VTEP gateway’’ term is a
           misnomer, since a VTEP is just a VXLAN Tunnel Endpoint, but it is
           a well established name.) See **Life Cycle of a VTEP gateway** , below,
           for more information.
    
           The main intended use case for VTEP gateways is to attach physical
           servers to an OVN logical network using a physical top-of-rack
           switch that supports the OVSDB VTEP schema.
    
         _L2 Gateways_
    
           A L2 gateway simply attaches a designated physical L2 segment
           available on some chassis to a logical network. The physical
           network effectively becomes part of the logical network.
    
           To set up a L2 gateway, the CMS adds an **l2gateway** LSP to an
           appropriate logical switch, setting LSP options to name the
           chassis on which it should be bound. **ovn-northd** copies this
           configuration into a southbound **Port_Binding** record. On the
           designated chassis, **ovn-controller** forwards packets appropriately
           to and from the physical segment.
    
           L2 gateway ports have features in common with **localnet** ports.
           However, with a **localnet** port, the physical network becomes the
           transport between hypervisors. With an L2 gateway, packets are
           still transported between hypervisors over tunnels and the
           **l2gateway** port is only used for the packets that are on the
           physical network. The application for L2 gateways is similar to
           that for VTEP gateways, e.g. to add non-virtualized machines to a
           logical network, but L2 gateways do not require special support
           from top-of-rack hardware switches.
    
         _L3 Gateway Routers_
    
           As described above under **Logical Networks** , ordinary OVN logical
           routers are distributed: they are not implemented in a single
           place but rather in every hypervisor chassis. This is a problem
           for stateful services such as SNAT and DNAT, which need to be
           implemented in a centralized manner.
    
           To allow for this kind of functionality, OVN supports L3 gateway
           routers, which are OVN logical routers that are implemented in a
           designated chassis. Gateway routers are typically used between
           distributed logical routers and physical networks. The distributed
           logical router and the logical switches behind it, to which VMs
           and containers attach, effectively reside on each hypervisor. The
           distributed router and the gateway router are connected by another
           logical switch, sometimes referred to as a ``join’’ logical
           switch. (OVN logical routers may be connected to one another
           directly, without an intervening switch, but the OVN
           implementation only supports gateway logical routers that are
           connected to logical switches. Using a join logical switch also
           reduces the number of IP addresses needed on the distributed
           router.) On the other side, the gateway router connects to another
           logical switch that has a **localnet** port connecting to the physical
           network.
    
           The following diagram shows a typical situation. One or more
           logical switches LS1, ..., LSn connect to distributed logical
           router LR1, which in turn connects through LSjoin to gateway
           logical router GLR, which also connects to logical switch LSlocal,
           which includes a **localnet** port to attach to the physical network.
    
                                           LSlocal
                                              |
                                             GLR
                                              |
                                           LSjoin
                                              |
                                             LR1
                                              |
                                         +----+----+
                                         |    |    |
                                        LS1  ...  LSn
    
           To configure an L3 gateway router, the CMS sets **options:chassis** in
           the router’s northbound **Logical_Router** to the chassis’s name. In
           response, **ovn-northd** uses a special **l3gateway** port binding
           (instead of a **patch** binding) in the southbound database to connect
           the logical router to its neighbors. In turn, **ovn-controller**
           tunnels packets to this port binding to the designated L3 gateway
           chassis, instead of processing them locally.
    
           DNAT and SNAT rules may be associated with a gateway router, which
           provides a central location that can handle one-to-many SNAT (aka
           IP masquerading). Distributed gateway ports, described below, also
           support NAT.
    
         _Distributed Gateway Ports_
    
           A _distributed gateway port_ is a logical router port that is
           specially configured to designate one distinguished chassis,
           called the _gateway chassis_ , for centralized processing. A
           distributed gateway port should connect to a logical switch that
           has an LSP that connects externally, that is, either a **localnet**
           LSP or a connection to another OVN deployment (see **OVN Deployments**
           **Interconnection**). Packets that traverse the distributed gateway
           port are processed without involving the gateway chassis when they
           can be, but when needed they do take an extra hop through it.
    
           The following diagram illustrates the use of a distributed gateway
           port. A number of logical switches LS1, ..., LSn connect to
           distributed logical router LR1, which in turn connects through the
           distributed gateway port to logical switch LSlocal that includes a
           **localnet** port to attach to the physical network.
    
                                           LSlocal
                                              |
                                             LR1
                                              |
                                         +----+----+
                                         |    |    |
                                        LS1  ...  LSn
    
           **ovn-northd** creates two southbound **Port_Binding** records to
           represent a distributed gateway port, instead of the usual one.
           One of these is a **patch** port binding named for the LRP, which is
           used for as much traffic as it can. The other one is a port
           binding with type **chassisredirect** , named **cr-**_port_. The
           **chassisredirect** port binding has one specialized job: when a
           packet is output to it, the flow table causes it to be tunneled to
           the gateway chassis, at which point it is automatically output to
           the **patch** port binding. Thus, the flow table can output to this
           port binding in cases where a particular task has to happen on the
           gateway chassis. The **chassisredirect** port binding is not otherwise
           used (for example, it never receives packets).
    
           The CMS may configure distributed gateway ports three different
           ways. See **Distributed Gateway Ports** in the documentation for
           **Logical_Router_Port** in ovn-nb(5) for details.
    
           Distributed gateway ports support high availability. When more
           than one chassis is specified, OVN only uses one at a time as the
           gateway chassis. OVN uses BFD to monitor gateway connectivity,
           preferring the highest-priority gateway that is online.
    
           A logical router can have multiple distributed gateway ports, each
           connecting different external networks. Load balancing is not yet
           supported for logical routers with more than one distributed
           gateway port configured.
    
           _Physical VLAN MTU Issues_
    
           Consider the preceding diagram again:
    
                                           LSlocal
                                              |
                                             LR1
                                              |
                                         +----+----+
                                         |    |    |
                                        LS1  ...  LSn
    
           Suppose that each logical switch LS1, ..., LSn is bridged to a
           physical VLAN-tagged network attached to a **localnet** port on
           LSlocal, over a distributed gateway port on LR1. If a packet
           originating on LS _i_ is destined to the external network, OVN sends
           it to the gateway chassis over a tunnel. There, the packet
           traverses LR1’s logical router pipeline, possibly undergoes NAT,
           and eventually ends up at LSlocal’s **localnet** port. If all of the
           physical links in the network have the same MTU, then the packet’s
           transit across a tunnel causes an MTU problem: tunnel overhead
           prevents a packet that uses the full physical MTU from crossing
           the tunnel to the gateway chassis (without fragmentation).
    
           OVN offers two solutions to this problem, the
           **reside-on-redirect-chassis** and **redirect-type** options. Both
           solutions require each logical switch LS1, ..., LSn to include a
           **localnet** logical switch port LN1, ..., LNn respectively, that is
           present on each chassis. Both cause packets to be sent over the
           **localnet** ports instead of tunnels. They differ in which
           packets-some or all-are sent this way. The most prominent tradeoff
           between these options is that **reside-on-redirect-chassis** is easier
           to configure and that **redirect-type** performs better for east-west
           traffic.
    
           The first solution is the **reside-on-redirect-chassis** option for
           logical router ports. Setting this option on a LRP from (e.g.) LS1
           to LR1 disables forwarding from LS1 to LR1 except on the gateway
           chassis. On chassis other than the gateway chassis, this single
           change means that packets that would otherwise have been forwarded
           to LR1 are instead forwarded to LN1. The instance of LN1 on the
           gateway chassis then receives the packet and forwards it to LR1.
           The packet traverses the LR1 logical router pipeline, possibly
           undergoes NAT, and eventually ends up at LSlocal’s **localnet** port.
           The packet never traverses a tunnel, avoiding the MTU issue.
    
           This option has the further consequence of centralizing
           ``distributed’’ logical router LR1, since no packets are forwarded
           from LS1 to LR1 on any chassis other than the gateway chassis.
           Therefore, east-west traffic passes through the gateway chassis,
           not just north-south. (The naive ``fix’’ of allowing east-west
           traffic to flow directly between chassis over LN1 does not work
           because routing sets the Ethernet source address to LR1’s source
           address. Seeing this single Ethernet source address originate from
           all of the chassis will confuse the physical switch.)
    
           Do not set the **reside-on-redirect-chassis** option on a distributed
           gateway port. In the diagram above, it would be set on the LRPs
           connecting LS1, ..., LSn to LR1.
    
           The second solution is the **redirect-type** option for distributed
           gateway ports. Setting this option to **bridged** causes packets that
           are redirected to the gateway chassis to go over the **localnet**
           ports instead of being tunneled. This option does not change how
           OVN treats packets not redirected to the gateway chassis.
    
           The **redirect-type** option requires the administrator or the CMS to
           configure each participating chassis with a unique Ethernet
           address for the logical router by setting **ovn-chassis-mac-mappings**
           in the Open vSwitch database, for use by **ovn-controller**. This
           makes it more difficult to configure than
           **reside-on-redirect-chassis**.
    
           Set the **redirect-type** option on a distributed gateway port.
    
           _Using Distributed Gateway Ports For Scalability_
    
           Although the primary goal of distributed gateway ports is to
           provide connectivity to external networks, there is a special use
           case for scalability.
    
           In some deployments, such as the ones using **ovn-kubernetes** ,
           logical switches are bound to individual chassises, and are
           connected by a distributed logical router. In such deployments,
           the chassis level logical switches are centralized on the chassis
           instead of distributed, which means the **ovn-controller** on each
           chassis doesn’t need to process flows and ports of logical
           switches on other chassises. However, without any specific hint,
           **ovn-controller** would still process all the logical switches as if
           they are fully distributed. In this case, distributed gateway port
           can be very useful. The chassis level logical switches can be
           connected to the distributed router using distributed gateway
           ports, by setting the gateway chassis (or HA chassis groups with
           only a single chassis in it) to the chassis that each logical
           switch is bound to. **ovn-controller** would then skip processing the
           logical switches on all the other chassises, largely improving the
           scalability, especially when there are a big number of chassises.
    
       **Life Cycle of a VIF**
           Tables and their schemas presented in isolation are difficult to
           understand. Here’s an example.
    
           A VIF on a hypervisor is a virtual network interface attached
           either to a VM or a container running directly on that hypervisor
           (This is different from the interface of a container running
           inside a VM).
    
           The steps in this example refer often to details of the OVN and
           OVN Northbound database schemas. Please see ovn-sb(5) and
           ovn-nb(5), respectively, for the full story on these databases.
    
                  1.  A VIF’s life cycle begins when a CMS administrator
                      creates a new VIF using the CMS user interface or API
                      and adds it to a switch (one implemented by OVN as a
                      logical switch). The CMS updates its own configuration.
                      This includes associating unique, persistent identifier
                      _vif-id_ and Ethernet address _mac_ with the VIF.
    
                  2.  The CMS plugin updates the OVN Northbound database to
                      include the new VIF, by adding a row to the
                      **Logical_Switch_Port** table. In the new row, **name** is _vif-_
                      _id_ , **mac** is _mac_ , **switch** points to the OVN logical
                      switch’s Logical_Switch record, and other columns are
                      initialized appropriately.
    
                  3.  **ovn-northd** receives the OVN Northbound database update.
                      In turn, it makes the corresponding updates to the OVN
                      Southbound database, by adding rows to the OVN
                      Southbound database **Logical_Flow** table to reflect the
                      new port, e.g. add a flow to recognize that packets
                      destined to the new port’s MAC address should be
                      delivered to it, and update the flow that delivers
                      broadcast and multicast packets to include the new
                      port. It also creates a record in the **Binding** table and
                      populates all its columns except the column that
                      identifies the **chassis**.
    
                  4.  On every hypervisor, **ovn-controller** receives the
                      **Logical_Flow** table updates that **ovn-northd** made in the
                      previous step. As long as the VM that owns the VIF is
                      powered off, **ovn-controller** cannot do much; it cannot,
                      for example, arrange to send packets to or receive
                      packets from the VIF, because the VIF does not actually
                      exist anywhere.
    
                  5.  Eventually, a user powers on the VM that owns the VIF.
                      On the hypervisor where the VM is powered on, the
                      integration between the hypervisor and Open vSwitch
                      (described in **Documentation/topics/integration.rst** in
                      the Open vSwitch source tree) adds the VIF to the OVN
                      integration bridge and stores _vif-id_ in
                      **external_ids** :**iface-id** to indicate that the interface is
                      an instantiation of the new VIF. (None of this code is
                      new in OVN; this is pre-existing integration work that
                      has already been done on hypervisors that support OVS.)
    
                  6.  On the hypervisor where the VM is powered on,
                      **ovn-controller** notices **external_ids** :**iface-id** in the new
                      Interface. In response, in the OVN Southbound DB, it
                      updates the **Binding** table’s **chassis** column for the row
                      that links the logical port from **external_ids** : **iface-id**
                      to the hypervisor. Afterward, **ovn-controller** updates
                      the local hypervisor’s OpenFlow tables so that packets
                      to and from the VIF are properly handled.
    
                  7.  Some CMS systems, including OpenStack, fully start a VM
                      only when its networking is ready. To support this,
                      **ovn-northd** notices the **chassis** column updated for the
                      row in **Binding** table and pushes this upward by updating
                      the **up** column in the OVN Northbound database’s
                      **Logical_Switch_Port** table to indicate that the VIF is
                      now up. The CMS, if it uses this feature, can then
                      react by allowing the VM’s execution to proceed.
    
                  8.  On every hypervisor but the one where the VIF resides,
                      **ovn-controller** notices the completely populated row in
                      the **Binding** table. This provides **ovn-controller** the
                      physical location of the logical port, so each instance
                      updates the OpenFlow tables of its switch (based on
                      logical datapath flows in the OVN DB **Logical_Flow**
                      table) so that packets to and from the VIF can be
                      properly handled via tunnels.
    
                  9.  Eventually, a user powers off the VM that owns the VIF.
                      On the hypervisor where the VM was powered off, the VIF
                      is deleted from the OVN integration bridge.
    
                  10. On the hypervisor where the VM was powered off,
                      **ovn-controller** notices that the VIF was deleted. In
                      response, it removes the **Chassis** column content in the
                      **Binding** table for the logical port.
    
                  11. On every hypervisor, **ovn-controller** notices the empty
                      **Chassis** column in the **Binding** table’s row for the
                      logical port. This means that **ovn-controller** no longer
                      knows the physical location of the logical port, so
                      each instance updates its OpenFlow table to reflect
                      that.
    
                  12. Eventually, when the VIF (or its entire VM) is no
                      longer needed by anyone, an administrator deletes the
                      VIF using the CMS user interface or API. The CMS
                      updates its own configuration.
    
                  13. The CMS plugin removes the VIF from the OVN Northbound
                      database, by deleting its row in the
                      **Logical_Switch_Port** table.
    
                  14. **ovn-northd** receives the OVN Northbound update and in
                      turn updates the OVN Southbound database accordingly,
                      by removing or updating the rows from the OVN
                      Southbound database **Logical_Flow** table and **Binding**
                      table that were related to the now-destroyed VIF.
    
                  15. On every hypervisor, **ovn-controller** receives the
                      **Logical_Flow** table updates that **ovn-northd** made in the
                      previous step. **ovn-controller** updates OpenFlow tables
                      to reflect the update, although there may not be much
                      to do, since the VIF had already become unreachable
                      when it was removed from the **Binding** table in a
                      previous step.
    
       **Life Cycle of a Container Interface Inside a VM**
           OVN provides virtual network abstractions by converting
           information written in OVN_NB database to OpenFlow flows in each
           hypervisor. Secure virtual networking for multi-tenants can only
           be provided if OVN controller is the only entity that can modify
           flows in Open vSwitch. When the Open vSwitch integration bridge
           resides in the hypervisor, it is a fair assumption to make that
           tenant workloads running inside VMs cannot make any changes to
           Open vSwitch flows.
    
           If the infrastructure provider trusts the applications inside the
           containers not to break out and modify the Open vSwitch flows,
           then containers can be run in hypervisors. This is also the case
           when containers are run inside the VMs and Open vSwitch
           integration bridge with flows added by OVN controller resides in
           the same VM. For both the above cases, the workflow is the same as
           explained with an example in the previous section ("Life Cycle of
           a VIF").
    
           This section talks about the life cycle of a container interface
           (CIF) when containers are created in the VMs and the Open vSwitch
           integration bridge resides inside the hypervisor. In this case,
           even if a container application breaks out, other tenants are not
           affected because the containers running inside the VMs cannot
           modify the flows in the Open vSwitch integration bridge.
    
           When multiple containers are created inside a VM, there are
           multiple CIFs associated with them. The network traffic associated
           with these CIFs need to reach the Open vSwitch integration bridge
           running in the hypervisor for OVN to support virtual network
           abstractions. OVN should also be able to distinguish network
           traffic coming from different CIFs. There are two ways to
           distinguish network traffic of CIFs.
    
           One way is to provide one VIF for every CIF (1:1 model). This
           means that there could be a lot of network devices in the
           hypervisor. This would slow down OVS because of all the additional
           CPU cycles needed for the management of all the VIFs. It would
           also mean that the entity creating the containers in a VM should
           also be able to create the corresponding VIFs in the hypervisor.
    
           The second way is to provide a single VIF for all the CIFs (1:many
           model). OVN could then distinguish network traffic coming from
           different CIFs via a tag written in every packet. OVN uses this
           mechanism and uses VLAN as the tagging mechanism.
    
                  1.  A CIF’s life cycle begins when a container is spawned
                      inside a VM by the either the same CMS that created the
                      VM or a tenant that owns that VM or even a container
                      Orchestration System that is different than the CMS
                      that initially created the VM. Whoever the entity is,
                      it will need to know the _vif-id_ that is associated with
                      the network interface of the VM through which the
                      container interface’s network traffic is expected to go
                      through. The entity that creates the container
                      interface will also need to choose an unused VLAN
                      inside that VM.
    
                  2.  The container spawning entity (either directly or
                      through the CMS that manages the underlying
                      infrastructure) updates the OVN Northbound database to
                      include the new CIF, by adding a row to the
                      **Logical_Switch_Port** table. In the new row, **name** is any
                      unique identifier, **parent_name** is the _vif-id_ of the VM
                      through which the CIF’s network traffic is expected to
                      go through and the **tag** is the VLAN tag that identifies
                      the network traffic of that CIF.
    
                  3.  **ovn-northd** receives the OVN Northbound database update.
                      In turn, it makes the corresponding updates to the OVN
                      Southbound database, by adding rows to the OVN
                      Southbound database’s **Logical_Flow** table to reflect the
                      new port and also by creating a new row in the **Binding**
                      table and populating all its columns except the column
                      that identifies the **chassis**.
    
                  4.  On every hypervisor, **ovn-controller** subscribes to the
                      changes in the **Binding** table. When a new row is created
                      by **ovn-northd** that includes a value in **parent_port**
                      column of **Binding** table, the **ovn-controller** in the
                      hypervisor whose OVN integration bridge has that same
                      value in _vif-id_ in **external_ids** :**iface-id** updates the
                      local hypervisor’s OpenFlow tables so that packets to
                      and from the VIF with the particular VLAN **tag** are
                      properly handled. Afterward it updates the **chassis**
                      column of the **Binding** to reflect the physical location.
    
                  5.  One can only start the application inside the container
                      after the underlying network is ready. To support this,
                      **ovn-northd** notices the updated **chassis** column in
                      **Binding** table and updates the **up** column in the OVN
                      Northbound database’s **Logical_Switch_Port** table to
                      indicate that the CIF is now up. The entity responsible
                      to start the container application queries this value
                      and starts the application.
    
                  6.  Eventually the entity that created and started the
                      container, stops it. The entity, through the CMS (or
                      directly) deletes its row in the **Logical_Switch_Port**
                      table.
    
                  7.  **ovn-northd** receives the OVN Northbound update and in
                      turn updates the OVN Southbound database accordingly,
                      by removing or updating the rows from the OVN
                      Southbound database **Logical_Flow** table that were
                      related to the now-destroyed CIF. It also deletes the
                      row in the **Binding** table for that CIF.
    
                  8.  On every hypervisor, **ovn-controller** receives the
                      **Logical_Flow** table updates that **ovn-northd** made in the
                      previous step. **ovn-controller** updates OpenFlow tables
                      to reflect the update.
    
       **Architectural Physical Life Cycle of a Packet**
           This section describes how a packet travels from one virtual
           machine or container to another through OVN. This description
           focuses on the physical treatment of a packet; for a description
           of the logical life cycle of a packet, please refer to the
           **Logical_Flow** table in ovn-sb(5).
    
           This section mentions several data and metadata fields, for
           clarity summarized here:
    
                  tunnel key
                         When OVN encapsulates a packet in Geneve or another
                         tunnel, it attaches extra data to it to allow the
                         receiving OVN instance to process it correctly. This
                         takes different forms depending on the particular
                         encapsulation, but in each case we refer to it here
                         as the ``tunnel key.’’ See **Tunnel Encapsulations** ,
                         below, for details.
    
                  logical datapath field
                         A field that denotes the logical datapath through
                         which a packet is being processed. OVN uses the
                         field that OpenFlow 1.1+ simply (and confusingly)
                         calls ``metadata’’ to store the logical datapath.
                         (This field is passed across tunnels as part of the
                         tunnel key.)
    
                  logical input port field
                         A field that denotes the logical port from which the
                         packet entered the logical datapath. OVN stores this
                         in Open vSwitch extension register number 14.
    
                         Geneve tunnels pass this field as part of the tunnel
                         key. Ramp switch VXLAN tunnels do not explicitly
                         carry a logical input port, but since they are used
                         to communicate with gateways that from OVN’s
                         perspective consist of only a single logical port,
                         so that OVN can set the logical input port field to
                         this one on ingress to the OVN logical pipeline. As
                         for regular VXLAN tunnels, they don’t carry input
                         port field at all. This puts additional limitations
                         on cluster capabilities that are described in **Tunnel**
                         **Encapsulations** section.
    
                  logical output port field
                         A field that denotes the logical port from which the
                         packet will leave the logical datapath. This is
                         initialized to 0 at the beginning of the logical
                         ingress pipeline. OVN stores this in Open vSwitch
                         extension register number 15.
    
                         Geneve and regular VXLAN tunnels pass this field as
                         part of the tunnel key. Ramp switch VXLAN tunnels do
                         not transmit the logical output port field, and
                         since they do not carry a logical output port field
                         in the tunnel key, when a packet is received from
                         ramp switch VXLAN tunnel by an OVN hypervisor, the
                         packet is resubmitted to table 8 to determine the
                         output port(s); when the packet reaches table 42,
                         these packets are resubmitted to table 43 for local
                         delivery by checking a MLF_RCV_FROM_RAMP flag, which
                         is set when the packet arrives from a ramp tunnel.
    
                  conntrack zone field for logical ports
                         A field that denotes the connection tracking zone
                         for logical ports. The value only has local
                         significance and is not meaningful between chassis.
                         This is initialized to 0 at the beginning of the
                         logical ingress pipeline. OVN stores this in the
                         lower 16 bits of the Open vSwitch extension register
                         number 13.
    
                  conntrack zone fields for routers
                         Fields that denote the connection tracking zones for
                         routers. These values only have local significance
                         and are not meaningful between chassis. OVN stores
                         the zone information for north to south traffic (for
                         DNATting or ECMP symmetric replies) in Open vSwitch
                         extension register number 11 and zone information
                         for south to north traffic (for SNATing) in Open
                         vSwitch extension register number 12.
    
                  Encap ID for logical ports
                         A field that records an ID that indicates which
                         encapsulation IP should be used when sending packets
                         to a remote chassis, according to the original input
                         logical port. This is useful when there are multiple
                         IPs available for encapsulation. The value only has
                         local significance and is not meaningful between
                         chassis. This is initialized to 0 at the beginning
                         of the logical ingress pipeline. OVN stores this in
                         the higher 16 bits of the Open vSwitch extension
                         register number 13.
    
                  logical flow flags
                         The logical flags are intended to handle keeping
                         context between tables in order to decide which
                         rules in subsequent tables are matched. These values
                         only have local significance and are not meaningful
                         between chassis. OVN stores the logical flags in
                         Open vSwitch extension register number 10.
    
                  VLAN ID
                         The VLAN ID is used as an interface between OVN and
                         containers nested inside a VM (see **Life Cycle of a**
                         **container interface inside a VM** , above, for more
                         information).
    
           Initially, a VM or container on the ingress hypervisor sends a
           packet on a port attached to the OVN integration bridge. Then:
    
                  1.  OpenFlow table 0 performs physical-to-logical
                      translation. It matches the packet’s ingress port. Its
                      actions annotate the packet with logical metadata, by
                      setting the logical datapath field to identify the
                      logical datapath that the packet is traversing and the
                      logical input port field to identify the ingress port.
                      Then it resubmits to table 8 to enter the logical
                      ingress pipeline.
    
                      Packets that originate from a container nested within a
                      VM are treated in a slightly different way. The
                      originating container can be distinguished based on the
                      VIF-specific VLAN ID, so the physical-to-logical
                      translation flows additionally match on VLAN ID and the
                      actions strip the VLAN header. Following this step, OVN
                      treats packets from containers just like any other
                      packets.
    
                      Table 0 also processes packets that arrive from other
                      chassis. It distinguishes them from other packets by
                      ingress port, which is a tunnel. As with packets just
                      entering the OVN pipeline, the actions annotate these
                      packets with logical datapath metadata. For tunnel
                      types that support it, they are also annotated with
                      logical ingress port metadata. In addition, the actions
                      set the logical output port field, which is available
                      because in OVN tunneling occurs after the logical
                      output port is known. These pieces of information are
                      obtained from the tunnel encapsulation metadata (see
                      **Tunnel Encapsulations** for encoding details). Then the
                      actions resubmit to table 45 to enter the logical
                      egress pipeline.
    
                  2.  OpenFlow tables 8 through 39 execute the logical
                      ingress pipeline from the **Logical_Flow** table in the OVN
                      Southbound database. These tables are expressed
                      entirely in terms of logical concepts like logical
                      ports and logical datapaths. A big part of
                      **ovn-controller** ’s job is to translate them into
                      equivalent OpenFlow (in particular it translates the
                      table numbers: **Logical_Flow** tables 0 through 29 become
                      OpenFlow tables 8 through 39).
    
                      Each logical flow maps to one or more OpenFlow flows.
                      An actual packet ordinarily matches only one of these,
                      although in some cases it can match more than one of
                      these flows (which is not a problem because all of them
                      have the same actions). **ovn-controller** uses the first
                      32 bits of the logical flow’s UUID as the cookie for
                      its OpenFlow flow or flows. (This is not necessarily
                      unique, since the first 32 bits of a logical flow’s
                      UUID is not necessarily unique.)
    
                      Some logical flows can map to the Open vSwitch
                      ``conjunctive match’’ extension (see ovs-fields(7)).
                      Flows with a **conjunction** action use an OpenFlow cookie
                      of 0, because they can correspond to multiple logical
                      flows. The OpenFlow flow for a conjunctive match
                      includes a match on **conj_id**.
    
                      Some logical flows may not be represented in the
                      OpenFlow tables on a given hypervisor, if they could
                      not be used on that hypervisor. For example, if no VIF
                      in a logical switch resides on a given hypervisor, and
                      the logical switch is not otherwise reachable on that
                      hypervisor (e.g. over a series of hops through logical
                      switches and routers starting from a VIF on the
                      hypervisor), then the logical flow may not be
                      represented there.
    
                      Most OVN actions have fairly obvious implementations in
                      OpenFlow (with OVS extensions), e.g. **next;** is
                      implemented as **resubmit** , _field_ **=**_constant_**;** as
                      **set_field**. A few are worth describing in more detail:
    
                      **output:**
                             Implemented by resubmitting the packet to table
                             40. If the pipeline executes more than one
                             **output** action, then each one is separately
                             resubmitted to table 40. This can be used to
                             send multiple copies of the packet to multiple
                             ports. (If the packet was not modified between
                             the **output** actions, and some of the copies are
                             destined to the same hypervisor, then using a
                             logical multicast output port would save
                             bandwidth between hypervisors.)
    
                      **get_arp(**_P_**,**_A_**);**
                      **get_nd(**_P_**,**_A_**);**
                           Implemented by storing arguments into OpenFlow
                           fields, then resubmitting to table 66, which
                           **ovn-controller** populates with flows generated from
                           the **MAC_Binding** table in the OVN Southbound
                           database. If there is a match in table 66, then
                           its actions store the bound MAC in the Ethernet
                           destination address field.
    
                           (The OpenFlow actions save and restore the
                           OpenFlow fields used for the arguments, so that
                           the OVN actions do not have to be aware of this
                           temporary use.)
    
                      **put_arp(**_P_**,**_A_**,**_E_**);**
                      **put_nd(**_P_**,**_A_**,**_E_**);**
                           Implemented by storing the arguments into OpenFlow
                           fields, then outputting a packet to
                           **ovn-controller** , which updates the **MAC_Binding**
                           table.
    
                           (The OpenFlow actions save and restore the
                           OpenFlow fields used for the arguments, so that
                           the OVN actions do not have to be aware of this
                           temporary use.)
    
                      _R_ **= lookup_arp(**_P_**,**_A_**,**_M_**);**
                      _R_ **= lookup_nd(**_P_**,**_A_**,**_M_**);**
                           Implemented by storing arguments into OpenFlow
                           fields, then resubmitting to table 67, which
                           **ovn-controller** populates with flows generated from
                           the **MAC_Binding** table in the OVN Southbound
                           database. If there is a match in table 67, then
                           its actions set the logical flow flag
                           **MLF_LOOKUP_MAC**.
    
                           (The OpenFlow actions save and restore the
                           OpenFlow fields used for the arguments, so that
                           the OVN actions do not have to be aware of this
                           temporary use.)
    
                  3.  OpenFlow tables 40 through 44 implement the **output**
                      action in the logical ingress pipeline. Specifically,
                      table 40 serves as an entry point to egress pipeline.
                      Table 40 detects IP packets that are too big for a
                      corresponding interface. Table 41 produces ICMPv4
                      Fragmentation Needed (or ICMPv6 Too Big) errors and
                      deliver them back to the offending port. table 42
                      handles packets to remote hypervisors, table 43 handles
                      packets to the local hypervisor, and table 44 checks
                      whether packets whose logical ingress and egress port
                      are the same should be discarded.
    
                      Logical patch ports are a special case. Logical patch
                      ports do not have a physical location and effectively
                      reside on every hypervisor. Thus, flow table 43, for
                      output to ports on the local hypervisor, naturally
                      implements output to unicast logical patch ports too.
                      However, applying the same logic to a logical patch
                      port that is part of a logical multicast group yields
                      packet duplication, because each hypervisor that
                      contains a logical port in the multicast group will
                      also output the packet to the logical patch port. Thus,
                      multicast groups implement output to logical patch
                      ports in table 42.
    
                      Each flow in table 42 matches on a logical output port
                      for unicast or multicast logical ports that include a
                      logical port on a remote hypervisor. Each flow’s
                      actions implement sending a packet to the port it
                      matches. For unicast logical output ports on remote
                      hypervisors, the actions set the tunnel key to the
                      correct value, then send the packet on the tunnel port
                      to the correct hypervisor. (When the remote hypervisor
                      receives the packet, table 0 there will recognize it as
                      a tunneled packet and pass it along to table 43.) For
                      multicast logical output ports, the actions send one
                      copy of the packet to each remote hypervisor, in the
                      same way as for unicast destinations. If a multicast
                      group includes a logical port or ports on the local
                      hypervisor, then its actions also resubmit to table 43.
                      Table 42 also includes:
    
                      •      A higher-priority rule to match packets received
                             from ramp switch tunnels, based on flag
                             MLF_RCV_FROM_RAMP, and resubmit these packets to
                             table 43 for local delivery. Packets received
                             from ramp switch tunnels reach here because of a
                             lack of logical output port field in the tunnel
                             key and thus these packets needed to be
                             submitted to table 8 to determine the output
                             port.
    
                      •      A higher-priority rule to match packets received
                             from ports of type **localport** , based on the
                             logical input port, and resubmit these packets
                             to table 43 for local delivery. Ports of type
                             **localport** exist on every hypervisor and by
                             definition their traffic should never go out
                             through a tunnel.
    
                      •      A higher-priority rule to match packets that
                             have the MLF_LOCAL_ONLY logical flow flag set,
                             and whose destination is a multicast address.
                             This flag indicates that the packet should not
                             be delivered to remote hypervisors, even if the
                             multicast destination includes ports on remote
                             hypervisors. This flag is used when
                             **ovn-controller** is the originator of the
                             multicast packet. Since each **ovn-controller**
                             instance is originating these packets, the
                             packets only need to be delivered to local
                             ports.
    
                      •      A fallback flow that resubmits to table 43 if
                             there is no other match.
    
                      Flows in table 43 resemble those in table 42 but for
                      logical ports that reside locally rather than remotely.
                      For unicast logical output ports on the local
                      hypervisor, the actions just resubmit to table 44. For
                      multicast output ports that include one or more logical
                      ports on the local hypervisor, for each such logical
                      port _P_ , the actions change the logical output port to
                      _P_ , then resubmit to table 44.
    
                      A special case is that when a localnet port exists on
                      the datapath, remote port is connected by switching to
                      the localnet port. In this case, instead of adding a
                      flow in table 42 to reach the remote port, a flow is
                      added in table 43 to switch the logical outport to the
                      localnet port, and resubmit to table 43 as if it were
                      unicasted to a logical port on the local hypervisor.
    
                      Table 44 matches and drops packets for which the
                      logical input and output ports are the same and the
                      MLF_ALLOW_LOOPBACK flag is not set. It also drops
                      MLF_LOCAL_ONLY packets directed to a localnet port,
                      provided they aren’t RAs sent from a gateway or
                      distributed router which is checked via the presence of
                      the bitflag MLF_OVERRIDE_LOCAL_ONLY. It resubmits other
                      packets to table 46.
    
                  4.  OpenFlow tables 45 through 62 execute the logical
                      egress pipeline from the **Logical_Flow** table in the OVN
                      Southbound database. The egress pipeline can perform a
                      final stage of validation before packet delivery.
                      Eventually, it may execute an **output** action, which
                      **ovn-controller** implements by resubmitting to table 64.
                      A packet for which the pipeline never executes **output**
                      is effectively dropped (although it may have been
                      transmitted through a tunnel across a physical
                      network).
    
                      The egress pipeline cannot change the logical output
                      port or cause further tunneling.
    
                  5.  Table 64 bypasses OpenFlow loopback when
                      MLF_ALLOW_LOOPBACK is set. Logical loopback was handled
                      in table 44, but OpenFlow by default also prevents
                      loopback to the OpenFlow ingress port. Thus, when
                      MLF_ALLOW_LOOPBACK is set, OpenFlow table 64 saves the
                      OpenFlow ingress port, sets it to zero, resubmits to
                      table 65 for logical-to-physical transformation, and
                      then restores the OpenFlow ingress port, effectively
                      disabling OpenFlow loopback prevents. When
                      MLF_ALLOW_LOOPBACK is unset, table 64 flow simply
                      resubmits to table 65.
    
                  6.  OpenFlow table 65 performs logical-to-physical
                      translation, the opposite of table 0. It matches the
                      packet’s logical egress port. Its actions output the
                      packet to the port attached to the OVN integration
                      bridge that represents that logical port. If the
                      logical egress port is a container nested with a VM,
                      then before sending the packet the actions push on a
                      VLAN header with an appropriate VLAN ID.
    
       **Logical Routers and Logical Patch Ports**
           Typically logical routers and logical patch ports do not have a
           physical location and effectively reside on every hypervisor. This
           is the case for logical patch ports between logical routers and
           logical switches behind those logical routers, to which VMs (and
           VIFs) attach.
    
           Consider a packet sent from one virtual machine or container to
           another VM or container that resides on a different subnet. The
           packet will traverse tables 0 to 65 as described in the previous
           section **Architectural Physical Life Cycle of a Packet** , using the
           logical datapath representing the logical switch that the sender
           is attached to. At table 42, the packet will use the fallback flow
           that resubmits locally to table 43 on the same hypervisor. In this
           case, all of the processing from table 0 to table 65 occurs on the
           hypervisor where the sender resides.
    
           When the packet reaches table 65, the logical egress port is a
           logical patch port. **ovn-controller** implements output to the
           logical patch is packet by cloning and resubmitting directly to
           the first OpenFlow flow table in the ingress pipeline, setting the
           logical ingress port to the peer logical patch port, and using the
           peer logical patch port’s logical datapath (that represents the
           logical router).
    
           The packet re-enters the ingress pipeline in order to traverse
           tables 8 to 65 again, this time using the logical datapath
           representing the logical router. The processing continues as
           described in the previous section **Architectural Physical Life**
           **Cycle of a Packet**. When the packet reaches table 65, the logical
           egress port will once again be a logical patch port. In the same
           manner as described above, this logical patch port will cause the
           packet to be resubmitted to OpenFlow tables 8 to 65, this time
           using the logical datapath representing the logical switch that
           the destination VM or container is attached to.
    
           The packet traverses tables 8 to 65 a third and final time. If the
           destination VM or container resides on a remote hypervisor, then
           table 39 will send the packet on a tunnel port from the sender’s
           hypervisor to the remote hypervisor. Finally table 65 will output
           the packet directly to the destination VM or container.
    
           The following sections describe two exceptions, where logical
           routers and/or logical patch ports are associated with a physical
           location.
    
         _Gateway Routers_
    
           A _gateway router_ is a logical router that is bound to a physical
           location. This includes all of the logical patch ports of the
           logical router, as well as all of the peer logical patch ports on
           logical switches. In the OVN Southbound database, the **Port_Binding**
           entries for these logical patch ports use the type **l3gateway**
           rather than **patch** , in order to distinguish that these logical
           patch ports are bound to a chassis.
    
           When a hypervisor processes a packet on a logical datapath
           representing a logical switch, and the logical egress port is a
           **l3gateway** port representing connectivity to a gateway router, the
           packet will match a flow in table 42 that sends the packet on a
           tunnel port to the chassis where the gateway router resides. This
           processing in table 42 is done in the same manner as for VIFs.
    
         _Distributed Gateway Ports_
    
           This section provides additional details on distributed gateway
           ports, outlined earlier.
    
           The primary design goal of distributed gateway ports is to allow
           as much traffic as possible to be handled locally on the
           hypervisor where a VM or container resides. Whenever possible,
           packets from the VM or container to the outside world should be
           processed completely on that VM’s or container’s hypervisor,
           eventually traversing a localnet port instance or a tunnel to the
           physical network or a different OVN deployment. Whenever possible,
           packets from the outside world to a VM or container should be
           directed through the physical network directly to the VM’s or
           container’s hypervisor.
    
           In order to allow for the distributed processing of packets
           described in the paragraph above, distributed gateway ports need
           to be logical patch ports that effectively reside on every
           hypervisor, rather than **l3gateway** ports that are bound to a
           particular chassis. However, the flows associated with distributed
           gateway ports often need to be associated with physical locations,
           for the following reasons:
    
                  •      The physical network that the localnet port is
                         attached to typically uses L2 learning. Any Ethernet
                         address used over the distributed gateway port must
                         be restricted to a single physical location so that
                         upstream L2 learning is not confused. Traffic sent
                         out the distributed gateway port towards the
                         localnet port with a specific Ethernet address must
                         be sent out one specific instance of the distributed
                         gateway port on one specific chassis. Traffic
                         received from the localnet port (or from a VIF on
                         the same logical switch as the localnet port) with a
                         specific Ethernet address must be directed to the
                         logical switch’s patch port instance on that
                         specific chassis.
    
                         Due to the implications of L2 learning, the Ethernet
                         address and IP address of the distributed gateway
                         port need to be restricted to a single physical
                         location. For this reason, the user must specify one
                         chassis associated with the distributed gateway
                         port. Note that traffic traversing the distributed
                         gateway port using other Ethernet addresses and IP
                         addresses (e.g. one-to-one NAT) is not restricted to
                         this chassis.
    
                         Replies to ARP and ND requests must be restricted to
                         a single physical location, where the Ethernet
                         address in the reply resides. This includes ARP and
                         ND replies for the IP address of the distributed
                         gateway port, which are restricted to the chassis
                         that the user associated with the distributed
                         gateway port.
    
                  •      In order to support one-to-many SNAT (aka IP
                         masquerading), where multiple logical IP addresses
                         spread across multiple chassis are mapped to a
                         single external IP address, it will be necessary to
                         handle some of the logical router processing on a
                         specific chassis in a centralized manner. Since the
                         SNAT external IP address is typically the
                         distributed gateway port IP address, and for
                         simplicity, the same chassis associated with the
                         distributed gateway port is used.
    
           The details of flow restrictions to specific chassis are described
           in the **ovn-northd** documentation.
    
           While most of the physical location dependent aspects of
           distributed gateway ports can be handled by restricting some flows
           to specific chassis, one additional mechanism is required. When a
           packet leaves the ingress pipeline and the logical egress port is
           the distributed gateway port, one of two different sets of actions
           is required at table 42:
    
                  •      If the packet can be handled locally on the sender’s
                         hypervisor (e.g. one-to-one NAT traffic), then the
                         packet should just be resubmitted locally to table
                         43, in the normal manner for distributed logical
                         patch ports.
    
                  •      However, if the packet needs to be handled on the
                         chassis associated with the distributed gateway port
                         (e.g. one-to-many SNAT traffic or non-NAT traffic),
                         then table 42 must send the packet on a tunnel port
                         to that chassis.
    
           In order to trigger the second set of actions, the **chassisredirect**
           type of southbound **Port_Binding** has been added. Setting the
           logical egress port to the type **chassisredirect** logical port is
           simply a way to indicate that although the packet is destined for
           the distributed gateway port, it needs to be redirected to a
           different chassis. At table 42, packets with this logical egress
           port are sent to a specific chassis, in the same way that table 42
           directs packets whose logical egress port is a VIF or a type
           **l3gateway** port to different chassis. Once the packet arrives at
           that chassis, table 43 resets the logical egress port to the value
           representing the distributed gateway port. For each distributed
           gateway port, there is one type **chassisredirect** port, in addition
           to the distributed logical patch port representing the distributed
           gateway port.
    
         _High Availability for Distributed Gateway Ports_
    
           OVN allows you to specify a prioritized list of chassis for a
           distributed gateway port. This is done by associating multiple
           **Gateway_Chassis** rows with a **Logical_Router_Port** in the
           **OVN_Northbound** database.
    
           When multiple chassis have been specified for a gateway, all
           chassis that may send packets to that gateway will enable BFD on
           tunnels to all configured gateway chassis. The current active
           chassis for the gateway is the highest priority gateway chassis
           that is currently viewed as active based on BFD status.
    
           For more information on L3 gateway high availability, please refer
           to http://docs.ovn.org/en/latest/topics/high-availability.html.
    
         _Restrictions of Distributed Gateway Ports_
    
           Distributed gateway ports are used to connect to an external
           network, which can be a physical network modeled by a logical
           switch with a localnet port, and can also be a logical switch that
           interconnects different OVN deployments (see **OVN Deployments**
           **Interconnection**). Usually there can be many logical routers
           connected to the same external logical switch, as shown in below
           diagram.
    
                                         +--LS-EXT-+
                                         |    |    |
                                         |    |    |
                                        LR1  ...  LRn
    
           In this diagram, there are n logical routers connected to a
           logical switch LS-EXT, each with a distributed gateway port, so
           that traffic sent to external world is redirected to the gateway
           chassis that is assigned to the distributed gateway port of
           respective logical router.
    
           In the logical topology, nothing can prevent an user to add a
           route between the logical routers via the connected distributed
           gateway ports on LS-EXT. However, the route works only if the LS-
           EXT is a physical network (modeled by a logical switch with a
           localnet port). In that case the packet will be delivered between
           the gateway chassises through the localnet port via physical
           network. If the LS-EXT is a regular logical switch (backed by
           tunneling only, as in the use case of OVN interconnection), then
           the packet will be dropped on the source gateway chassis. The
           limitation is due the fact that distributed gateway ports are tied
           to physical location, and without physical network connection, we
           will end up with either dropping the packet or transferring it
           over the tunnels which could cause bigger problems such as
           broadcast packets being redirect repeatedly by different gateway
           chassises.
    
           With the limitation in mind, if a user do want the direct
           connectivity between the logical routers, it is better to create
           an internal logical switch connected to the logical routers with
           regular logical router ports, which are completely distributed and
           the packets don’t have to leave a chassis unless necessary, which
           is more optimal than routing via the distributed gateway ports.
    
         _ARP request and ND NS packet processing_
    
           Due to the fact that ARP requests and ND NA packets are usually
           broadcast packets, for performance reasons, OVN deals with
           requests that target OVN owned IP addresses (i.e., IP addresses
           configured on the router ports, VIPs, NAT IPs) in a specific way
           and only forwards them to the logical router that owns the target
           IP address. This behavior is different than that of traditional
           switches and implies that other routers/hosts connected to the
           logical switch will not learn the MAC/IP binding from the request
           packet.
    
           All other ARP and ND packets are flooded in the L2 broadcast
           domain and to all attached logical patch ports.
    
         _VIFs on the logical switch connected by a distributed gateway port_
    
           Typically the logical switch connected by a distributed gateway
           port is for external connectivity, usually to a physical network
           through a localnet port on the logical switch, or to a remote OVN
           deployment through OVN Interconnection. In these cases there is no
           VIF ports required on the logical switch.
    
           While not very common, it is still possible to create VIF ports on
           the logical switch connected by a distributed gateway port, but
           there is a limitation that the logical ports need to reside on the
           gateway chassis where the distributed gateway port resides to get
           connectivity to other logical switches through the distributed
           gateway port. There is no limitation for the VIFs to connect
           within the logical switch, or beyond the logical switch through
           other regular distributed logical router ports.
    
           A special case is when using distributed gateway ports for
           scalability purpose, as mentioned earlier in this document. The
           logical switches connected by distributed gateway ports are not
           for connectivity but just for regular VIFs. However, the above
           limitation usually does not matter because in this use case all
           the VIFs on the logical switch are located on the same chassis
           with the distributed gateway port that connects the logical
           switch.
    
       **Multiple localnet logical switches connected to a Logical Router**
           It is possible to have multiple logical switches each with a
           localnet port (representing physical networks) connected to a
           logical router, in which one localnet logical switch may provide
           the external connectivity via a distributed gateway port and rest
           of the localnet logical switches use VLAN tagging in the physical
           network. It is expected that **ovn-bridge-mappings** is configured
           appropriately on the chassis for all these localnet networks.
    
         _East West routing_
    
           East-West routing between these localnet VLAN tagged logical
           switches work almost the same way as normal logical switches. When
           the VM sends such a packet, then:
    
                  1.  It first enters the ingress pipeline, and then egress
                      pipeline of the source localnet logical switch
                      datapath. It then enters the ingress pipeline of the
                      logical router datapath via the logical router port in
                      the source chassis.
    
                  2.  Routing decision is taken.
    
                  3.  From the router datapath, packet enters the ingress
                      pipeline and then egress pipeline of the destination
                      localnet logical switch datapath and goes out of the
                      integration bridge to the provider bridge ( belonging
                      to the destination logical switch) via the localnet
                      port. While sending the packet to provider bridge, we
                      also replace router port MAC as source MAC with a
                      chassis unique MAC.
    
                      This chassis unique MAC is configured as global ovs
                      config on each chassis (eg. via "**ovs-vsctl set open .**
                      **external-ids:**
                      **ovn-chassis-mac-mappings="phys:aa:bb:cc:dd:ee:$i$i"** ").
                      For more details, see ovn-controller(8).
    
                      If the above is not configured, then source MAC would
                      be the router port MAC. This could create problem if we
                      have more than one chassis. This is because, since the
                      router port is distributed, the same (MAC,VLAN) tuple
                      will seen by physical network from other chassis as
                      well, which could cause these issues:
    
                      •      Continuous MAC moves in top-of-rack switch
                             (ToR).
    
                      •      ToR dropping the traffic, which is causing
                             continuous MAC moves.
    
                      •      ToR blocking the ports from which MAC moves are
                             happening.
    
                  4.  The destination chassis receives the packet via the
                      localnet port and sends it to the integration bridge.
                      Before entering the integration bridge the source mac
                      of the packet will be replaced with router port mac
                      again. The packet enters the ingress pipeline and then
                      egress pipeline of the destination localnet logical
                      switch and finally gets delivered to the destination VM
                      port.
    
         _External traffic_
    
           The following happens when a VM sends an external traffic (which
           requires NATting) and the chassis hosting the VM doesn’t have a
           distributed gateway port.
    
                  1.  The packet first enters the ingress pipeline, and then
                      egress pipeline of the source localnet logical switch
                      datapath. It then enters the ingress pipeline of the
                      logical router datapath via the logical router port in
                      the source chassis.
    
                  2.  Routing decision is taken. Since the gateway router or
                      the distributed gateway port doesn’t reside in the
                      source chassis, the traffic is redirected to the
                      gateway chassis via the tunnel port.
    
                  3.  The gateway chassis receives the packet via the tunnel
                      port and the packet enters the egress pipeline of the
                      logical router datapath. NAT rules are applied here.
                      The packet then enters the ingress pipeline and then
                      egress pipeline of the localnet logical switch datapath
                      which provides external connectivity and finally goes
                      out via the localnet port of the logical switch which
                      provides external connectivity.
    
           Although this works, the VM traffic is tunnelled when sent from
           the compute chassis to the gateway chassis. In order for it to
           work properly, the MTU of the localnet logical switches must be
           lowered to account for the tunnel encapsulation.
    
       **Centralized routing for localnet VLAN tagged logical switches**
           **connected to a Logical Router**
           To overcome the tunnel encapsulation problem described in the
           previous section, **OVN** supports the option of enabling centralized
           routing for localnet VLAN tagged logical switches. CMS can
           configure the option **options:reside-on-redirect-chassis** to **true**
           for each **Logical_Router_Port** which connects to the localnet VLAN
           tagged logical switches. This causes the gateway chassis (hosting
           the distributed gateway port) to handle all the routing for these
           networks, making it centralized. It will reply to the ARP requests
           for the logical router port IPs.
    
           If the logical router doesn’t have a distributed gateway port
           connecting to the localnet logical switch which provides external
           connectivity, or if it has more than one distributed gateway
           ports, then this option is ignored by **OVN**.
    
           The following happens when a VM sends an east-west traffic which
           needs to be routed:
    
                  1.  The packet first enters the ingress pipeline, and then
                      egress pipeline of the source localnet logical switch
                      datapath and is sent out via a localnet port of the
                      source localnet logical switch (instead of sending it
                      to router pipeline).
    
                  2.  The gateway chassis receives the packet via a localnet
                      port of the source localnet logical switch and sends it
                      to the integration bridge. The packet then enters the
                      ingress pipeline, and then egress pipeline of the
                      source localnet logical switch datapath and enters the
                      ingress pipeline of the logical router datapath.
    
                  3.  Routing decision is taken.
    
                  4.  From the router datapath, packet enters the ingress
                      pipeline and then egress pipeline of the destination
                      localnet logical switch datapath. It then goes out of
                      the integration bridge to the provider bridge (
                      belonging to the destination logical switch) via a
                      localnet port.
    
                  5.  The destination chassis receives the packet via a
                      localnet port and sends it to the integration bridge.
                      The packet enters the ingress pipeline and then egress
                      pipeline of the destination localnet logical switch and
                      finally delivered to the destination VM port.
    
           The following happens when a VM sends an external traffic which
           requires NATting:
    
                  1.  The packet first enters the ingress pipeline, and then
                      egress pipeline of the source localnet logical switch
                      datapath and is sent out via a localnet port of the
                      source localnet logical switch (instead of sending it
                      to router pipeline).
    
                  2.  The gateway chassis receives the packet via a localnet
                      port of the source localnet logical switch and sends it
                      to the integration bridge. The packet then enters the
                      ingress pipeline, and then egress pipeline of the
                      source localnet logical switch datapath and enters the
                      ingress pipeline of the logical router datapath.
    
                  3.  Routing decision is taken and NAT rules are applied.
    
                  4.  From the router datapath, packet enters the ingress
                      pipeline and then egress pipeline of the localnet
                      logical switch datapath which provides external
                      connectivity. It then goes out of the integration
                      bridge to the provider bridge (belonging to the logical
                      switch which provides external connectivity) via a
                      localnet port.
    
           The following happens for the reverse external traffic.
    
                  1.  The gateway chassis receives the packet from a localnet
                      port of the logical switch which provides external
                      connectivity. The packet then enters the ingress
                      pipeline and then egress pipeline of the localnet
                      logical switch (which provides external connectivity).
                      The packet then enters the ingress pipeline of the
                      logical router datapath.
    
                  2.  The ingress pipeline of the logical router datapath
                      applies the unNATting rules. The packet then enters the
                      ingress pipeline and then egress pipeline of the source
                      localnet logical switch. Since the source VM doesn’t
                      reside in the gateway chassis, the packet is sent out
                      via a localnet port of the source logical switch.
    
                  3.  The source chassis receives the packet via a localnet
                      port and sends it to the integration bridge. The packet
                      enters the ingress pipeline and then egress pipeline of
                      the source localnet logical switch and finally gets
                      delivered to the source VM port.
    
           As an alternative to **reside-on-redirect-chassis** , OVN supports
           VLAN-based redirection. Whereas **reside-on-redirect-chassis**
           centralizes all router functionality, VLAN-based redirection only
           changes how OVN redirects packets to the gateway chassis. By
           setting **options:redirect-type** to **bridged** on a distributed gateway
           port, OVN redirects packets to the gateway chassis using the
           **localnet** port of the router’s peer logical switch, instead of a
           tunnel.
    
           If the logical router doesn’t have a distributed gateway port
           connecting to the localnet logical switch which provides external
           connectivity, or if it has more than one distributed gateway
           ports, then this option is ignored by **OVN**.
    
           Following happens for bridged redirection:
    
                  1.  On compute chassis, packet passes though logical
                      router’s ingress pipeline.
    
                  2.  If logical outport is gateway chassis attached router
                      port then packet is "redirected" to gateway chassis
                      using peer logical switch’s localnet port.
    
                  3.  This redirected packet has destination mac as router
                      port mac (the one to which gateway chassis is
                      attached). Its VLAN id is that of localnet port (peer
                      logical switch of the logical router port).
    
                  4.  On the gateway chassis packet will enter the logical
                      router pipeline again and this time it will passthrough
                      egress pipeline as well.
    
                  5.  Reverse traffic packet flows stays the same.
    
           Some guidelines and expections with bridged redirection:
    
                  1.  Since router port mac is destination mac, hence it has
                      to be ensured that physical network learns it on ONLY
                      from the gateway chassis. Which means that
                      **ovn-chassis-mac-mappings** should be configure on all the
                      compute nodes, so that physical network never learn
                      router port mac from compute nodes.
    
                  2.  Since packet enters logical router ingress pipeline
                      twice (once on compute chassis and again on gateway
                      chassis), hence ttl will be decremented twice.
    
                  3.  Default redirection type continues to be **overlay**. User
                      can switch the redirect-type between **bridged** and
                      **overlay** by changing the value of **options:redirect-type**
    
       **Life Cycle of a VTEP gateway**
           A gateway is a chassis that forwards traffic between the OVN-
           managed part of a logical network and a physical VLAN, extending a
           tunnel-based logical network into a physical network.
    
           The steps below refer often to details of the OVN and VTEP
           database schemas. Please see ovn-sb(5), ovn-nb(5) and vtep(5),
           respectively, for the full story on these databases.
    
                  1.  A VTEP gateway’s life cycle begins with the
                      administrator registering the VTEP gateway as a
                      **Physical_Switch** table entry in the **VTEP** database. The
                      **ovn-controller-vtep** connected to this VTEP database,
                      will recognize the new VTEP gateway and create a new
                      **Chassis** table entry for it in the **OVN_Southbound**
                      database.
    
                  2.  The administrator can then create a new **Logical_Switch**
                      table entry, and bind a particular vlan on a VTEP
                      gateway’s port to any VTEP logical switch. Once a VTEP
                      logical switch is bound to a VTEP gateway, the
                      **ovn-controller-vtep** will detect it and add its name to
                      the _vtep_logical_switches_ column of the **Chassis** table
                      in the **OVN_Southbound** database. Note, the _tunnel_key_
                      column of VTEP logical switch is not filled at
                      creation. The **ovn-controller-vtep** will set the column
                      when the corresponding vtep logical switch is bound to
                      an OVN logical network.
    
                  3.  Now, the administrator can use the CMS to add a VTEP
                      logical switch to the OVN logical network. To do that,
                      the CMS must first create a new **Logical_Switch_Port**
                      table entry in the **OVN_Northbound** database. Then, the
                      _type_ column of this entry must be set to "vtep". Next,
                      the _vtep-logical-switch_ and _vtep-physical-switch_ keys
                      in the _options_ column must also be specified, since
                      multiple VTEP gateways can attach to the same VTEP
                      logical switch. Next, the _addresses_ column of this
                      logical port must be set to "unknown", it will add a
                      priority 0 entry in "ls_in_l2_lkup" stage of logical
                      switch ingress pipeline. So, traffic with unrecorded
                      mac by OVN would go through the **Logical_Switch_Port** to
                      physical network.
    
                  4.  The newly created logical port in the **OVN_Northbound**
                      database and its configuration will be passed down to
                      the **OVN_Southbound** database as a new **Port_Binding** table
                      entry. The **ovn-controller-vtep** will recognize the
                      change and bind the logical port to the corresponding
                      VTEP gateway chassis. Configuration of binding the same
                      VTEP logical switch to a different OVN logical networks
                      is not allowed and a warning will be generated in the
                      log.
    
                  5.  Beside binding to the VTEP gateway chassis, the
                      **ovn-controller-vtep** will update the _tunnel_key_ column
                      of the VTEP logical switch to the corresponding
                      **Datapath_Binding** table entry’s _tunnel_key_ for the bound
                      OVN logical network.
    
                  6.  Next, the **ovn-controller-vtep** will keep reacting to the
                      configuration change in the **Port_Binding** in the
                      **OVN_Northbound** database, and updating the
                      **Ucast_Macs_Remote** table in the **VTEP** database. This
                      allows the VTEP gateway to understand where to forward
                      the unicast traffic coming from the extended external
                      network.
    
                  7.  Eventually, the VTEP gateway’s life cycle ends when the
                      administrator unregisters the VTEP gateway from the
                      **VTEP** database. The **ovn-controller-vtep** will recognize
                      the event and remove all related configurations
                      (**Chassis** table entry and port bindings) in the
                      **OVN_Southbound** database.
    
                  8.  When the **ovn-controller-vtep** is terminated, all related
                      configurations in the **OVN_Southbound** database and the
                      **VTEP** database will be cleaned, including **Chassis** table
                      entries for all registered VTEP gateways and their port
                      bindings, and all **Ucast_Macs_Remote** table entries and
                      the **Logical_Switch** tunnel keys.
    
       **OVN Deployments Interconnection**
           It is not uncommon for an operator to deploy multiple OVN
           clusters, for two main reasons. Firstly, an operator may prefer to
           deploy one OVN cluster for each availability zone, e.g. in
           different physical regions, to avoid single point of failure.
           Secondly, there is always an upper limit for a single OVN control
           plane to scale.
    
           Although the control planes of the different availability zone
           (AZ)s are independent from each other, the workloads from
           different AZs may need to communicate across the zones. The OVN
           interconnection feature provides a native way to interconnect
           different AZs by L3 routing through transit overlay networks
           between logical routers of different AZs.
    
           A global OVN Interconnection Northbound database is introduced for
           the operator (probably through CMS systems) to configure transit
           logical switches that connect logical routers from different AZs.
           A transit switch is similar to a regular logical switch, but it is
           used for interconnection purpose only. Typically, each transit
           switch can be used to connect all logical routers that belong to
           same tenant across all AZs.
    
           A dedicated daemon process **ovn-ic** , OVN interconnection controller,
           in each AZ will consume this data and populate corresponding
           logical switches to their own northbound databases for each AZ, so
           that logical routers can be connected to the transit switch by
           creating patch port pairs in their northbound databases. Any
           router ports connected to the transit switches are considered
           interconnection ports, which will be exchanged between AZs.
    
           Physically, when workloads from different AZs communicate, packets
           need to go through multiple hops: source chassis, source gateway,
           destination gateway and destination chassis. All these hops are
           connected through tunnels so that the packets never leave overlay
           networks. A distributed gateway port is required to connect the
           logical router to a transit switch, with a gateway chassis
           specified, so that the traffic can be forwarded through the
           gateway chassis.
    
           A global OVN Interconnection Southbound database is introduced for
           exchanging control plane information between the AZs. The data in
           this database is populated and consumed by the **ovn-ic** , of each AZ.
           The main information in this database includes:
    
                  •      Datapath bindings for transit switches, which mainly
                         contains the tunnel keys generated for each transit
                         switch. Separate key ranges are reserved for transit
                         switches so that they will never conflict with any
                         tunnel keys locally assigned for datapaths within
                         each AZ.
    
                  •      Availability zones, which are registered by **ovn-ic**
                         from each AZ.
    
                  •      Gateways. Each AZ specifies chassises that are
                         supposed to work as interconnection gateways, and
                         the **ovn-ic** will populate this information to the
                         interconnection southbound DB. The **ovn-ic** from all
                         the other AZs will learn the gateways and populate
                         to their own southbound DB as a chassis.
    
                  •      Port bindings for logical switch ports created on
                         the transit switch. Each AZ maintains their logical
                         router to transit switch connections independently,
                         but **ovn-ic** automatically populates local port
                         bindings on transit switches to the global
                         interconnection southbound DB, and learns remote
                         port bindings from other AZs back to its own
                         northbound and southbound DBs, so that logical flows
                         can be produced and then translated to OVS flows
                         locally, which finally enables data plane
                         communication.
    
                  •      Routes that are advertised between different AZs. If
                         enabled, routes are automatically exchanged by
                         **ovn-ic**. Both static routes and directly connected
                         subnets are advertised. Options in **options** column of
                         the **NB_Global** table of **OVN_NB** database control the
                         behavior of route advertisement, such as
                         enable/disable the advertising/learning routes,
                         whether default routes are advertised/learned, and
                         blacklisted CIDRs. See ovn-nb(5) for more details.
    
           The tunnel keys for transit switch datapaths and related port
           bindings must be agreed across all AZs. This is ensured by
           generating and storing the keys in the global interconnection
           southbound database. Any **ovn-ic** from any AZ can allocate the key,
           but race conditions are solved by enforcing unique index for the
           column in the database.
    
           Once each AZ’s NB and SB databases are populated with
           interconnection switches and ports, and agreed upon the tunnel
           keys, data plane communication between the AZs are established.
    
           When VXLAN tunneling is enabled in an OVN cluster, due to the
           limited range available for VNIs, Interconnection feature is not
           supported.
    
         _A day in the life of a packet crossing AZs_
    
                  1.  An IP packet is sent out from a VIF on a hypervisor
                      (HV1) of AZ1, with destination IP belonging to a VIF in
                      AZ2.
    
                  2.  In HV1’s OVS flow tables, the packet goes through
                      logical switch and logical router pipelines, and in a
                      logical router pipeline, the routing stage finds out
                      the next hop for the destination IP, which belongs to a
                      remote logical router port in AZ2, and the output port,
                      which is a chassis-redirect port located on an
                      interconnection gateway (GW1 in AZ1), so HV1 sends the
                      packet to GW1 through tunnel.
    
                  3.  On GW1, it continues with the logical router pipe line
                      and switches to the transit switch’s pipeline through
                      the peer port of the chassis redirect port. In the
                      transit switch’s pipeline it outputs to the remote
                      logical port which is located on a gateway (GW2) in
                      AZ2, so the GW1 sends the packet to GW2 in tunnel.
    
                  4.  On GW2, it continues with the transit switch pipeline
                      and switches to the logical router pipeline through the
                      peer port, which is a chassis redirect port that is
                      located on GW2. The logical router pipeline then
                      forwards the packet to relevant logical pipelines
                      according to the destination IP address, and figures
                      out the MAC and location of the destination VIF port -
                      a hypervisor (HV2). The GW2 then sends the packet to
                      HV2 in tunnel.
    
                  5.  On HV2, the packet is delivered to the final
                      destination VIF port by the logical switch egress
                      pipeline, just the same way as for intra-AZ
                      communications.
    
       **Native OVN services for external logical ports**
           To support OVN native services (like DHCP/IPv6 RA/DNS lookup) to
           the cloud resources which are external, OVN supports **external**
           logical ports.
    
           Below are some of the use cases where **external** ports can be used.
    
                  •      VMs connected to SR-IOV nics - Traffic from these
                         VMs by passes the kernel stack and local
                         **ovn-controller** do not bind these ports and cannot
                         serve the native services.
    
                  •      When CMS supports provisioning baremetal servers.
    
           OVN will provide the native services if CMS has done the below
           configuration in the _OVN Northbound Database_.
    
                  •      A row is created in **Logical_Switch_Port** , configuring
                         the **addresses** column and setting the **type** to
                         **external**.
    
                  •      **ha_chassis_group** column is configured.
    
                  •      The HA chassis which belongs to the HA chassis group
                         has the **ovn-bridge-mappings** configured and has
                         proper L2 connectivity so that it can receive the
                         DHCP and other related request packets from these
                         external resources.
    
                  •      The Logical_Switch of this port has a **localnet** port.
    
                  •      Native OVN services are enabled by configuring the
                         DHCP and other options like the way it is done for
                         the normal logical ports.
    
           It is recommended to use the same HA chassis group for all the
           external ports of a logical switch. Otherwise, the physical switch
           might see MAC flap issue when different chassis provide the native
           services. For example when supporting native DHCPv4 service,
           DHCPv4 server mac (configured in **options:server_mac** column in
           table **DHCP_Options**) originating from different ports can cause MAC
           flap issue. The MAC of the logical router IP(s) can also flap if
           the same HA chassis group is not set for all the external ports of
           a logical switch.
    


## SECURITY



       **Role-Based Access Controls for the Southbound DB**
           In order to provide additional security against the possibility of
           an OVN chassis becoming compromised in such a way as to allow
           rogue software to make arbitrary modifications to the southbound
           database state and thus disrupt the OVN network, role-based access
           controls (see ovsdb-server(1) for additional details) are provided
           for the southbound database.
    
           The implementation of role-based access controls (RBAC) requires
           the addition of two tables to an OVSDB schema: the **RBAC_Role**
           table, which is indexed by role name and maps the the names of the
           various tables that may be modifiable for a given role to
           individual rows in a permissions table containing detailed
           permission information for that role, and the permission table
           itself which consists of rows containing the following
           information:
    
                  **Table Name**
                         The name of the associated table. This column exists
                         primarily as an aid for humans reading the contents
                         of this table.
    
                  **Auth Criteria**
                         A set of strings containing the names of columns (or
                         column:key pairs for columns containing
                         string:string maps). The contents of at least one of
                         the columns or column:key values in a row to be
                         modified, inserted, or deleted must be equal to the
                         ID of the client attempting to act on the row in
                         order for the authorization check to pass. If the
                         authorization criteria is empty, authorization
                         checking is disabled and all clients for the role
                         will be treated as authorized.
    
                  **Insert/Delete**
                         Row insertion/deletion permission; boolean value
                         indicating whether insertion and deletion of rows is
                         allowed for the associated table. If true, insertion
                         and deletion of rows is allowed for authorized
                         clients.
    
                  **Updatable Columns**
                         A set of strings containing the names of columns or
                         column:key pairs that may be updated or mutated by
                         authorized clients. Modifications to columns within
                         a row are only permitted when the authorization
                         check for the client passes and all columns to be
                         modified are included in this set of modifiable
                         columns.
    
           RBAC configuration for the OVN southbound database is maintained
           by ovn-northd. With RBAC enabled, modifications are only permitted
           for the **Chassis** , **Encap** , **Port_Binding** , and **MAC_Binding** tables, and
           are restricted as follows:
    
                  **Chassis**
                         **Authorization** : client ID must match the chassis
                         name.
    
                         **Insert/Delete** : authorized row insertion and deletion
                         are permitted.
    
                         **Update** : The columns **nb_cfg** , **external_ids** , **encaps** ,
                         and **vtep_logical_switches** may be modified when
                         authorized.
    
                  **Encap  Authorization** : client ID must match the chassis
                         name.
    
                         **Insert/Delete** : row insertion and row deletion are
                         permitted.
    
                         **Update** : The columns **type** , **options** , and **ip** can be
                         modified.
    
                  **Port_Binding**
                         **Authorization** : disabled (all clients are considered
                         authorized. A future enhancement may add columns (or
                         keys to **external_ids**) in order to control which
                         chassis are allowed to bind each port.
    
                         **Insert/Delete** : row insertion/deletion are not
                         permitted (ovn-northd maintains rows in this table.
    
                         **Update** : Only modifications to the **chassis** column are
                         permitted.
    
                  **MAC_Binding**
                         **Authorization** : disabled (all clients are considered
                         to be authorized).
    
                         **Insert/Delete** : row insertion/deletion are permitted.
    
                         **Update** : The columns **logical_port** , **ip** , **mac** , and
                         **datapath** may be modified by ovn-controller.
    
                  **IGMP_Group**
                         **Authorization** : disabled (all clients are considered
                         to be authorized).
    
                         **Insert/Delete** : row insertion/deletion are permitted.
    
                         **Update** : The columns **address** , **chassis** , **datapath** , and
                         **ports** may be modified by ovn-controller.
    
           Enabling RBAC for ovn-controller connections to the southbound
           database requires the following steps:
    
                  1.  Creating SSL/TLS certificates for each chassis with the
                      certificate CN field set to the chassis name (e.g. for
                      a chassis with **external-ids:system-id=chassis-1** , via
                      the command "**ovs-pki -u req+sign chassis-1 switch** ").
    
                  2.  Configuring each ovn-controller to use SSL/TLS when
                      connecting to the southbound database (e.g. via
                      "**ovs-vsctl set open .**
                      **external-ids:ovn-remote=ssl:x.x.x.x:6642** ").
    
                  3.  Configuring a southbound database SSL/TLS remote with
                      "ovn-controller" role (e.g. via "**ovn-sbctl**
                      **set-connection role=ovn-controller pssl:6642** ").
    
       **Encrypt Tunnel Traffic with IPsec**
           OVN tunnel traffic goes through physical routers and switches.
           These physical devices could be untrusted (devices in public
           network) or might be compromised. Enabling encryption to the
           tunnel traffic can prevent the traffic data from being monitored
           and manipulated.
    
           The tunnel traffic is encrypted with IPsec. The CMS sets the **ipsec**
           column in the northbound **NB_Global** table to enable or disable
           IPsec encrytion. If **ipsec** is true, all OVN tunnels will be
           encrypted. If **ipsec** is false, no OVN tunnels will be encrypted.
    
           When CMS updates the **ipsec** column in the northbound **NB_Global**
           table, **ovn-northd** copies the value to the **ipsec** column in the
           southbound **SB_Global** table. **ovn-controller** in each chassis
           monitors the southbound database and sets the options of the OVS
           tunnel interface accordingly. OVS tunnel interface options are
           monitored by the **ovs-monitor-ipsec** daemon which configures IKE
           daemon to set up IPsec connections.
    
           Chassis authenticates each other by using certificate. The
           authentication succeeds if the other end in tunnel presents a
           certificate signed by a trusted CA and the common name (CN)
           matches the expected chassis name. The SSL/TLS certificates used
           in role-based access controls (RBAC) can be used in IPsec. Or use
           **ovs-pki** to create different certificates. The certificate is
           required to be x.509 version 3, and with CN field and
           subjectAltName field being set to the chassis name.
    
           The CA certificate, chassis certificate and private key are
           required to be installed in each chassis before enabling IPsec.
           Please see ovs-vswitchd.conf.db(5) for setting up CA based IPsec
           authentication.
    


## DESIGN DECISIONS



       **Tunnel Encapsulations**
           In general, OVN annotates logical network packets that it sends
           from one hypervisor to another with the following three pieces of
           metadata, which are encoded in an encapsulation-specific fashion:
    
                  •      24-bit logical datapath identifier, from the
                         **tunnel_key** column in the OVN Southbound
                         **Datapath_Binding** table.
    
                  •      15-bit logical ingress port identifier. ID 0 is
                         reserved for internal use within OVN. IDs 1 through
                         32767, inclusive, may be assigned to logical ports
                         (see the **tunnel_key** column in the OVN Southbound
                         **Port_Binding** table).
    
                  •      16-bit logical egress port identifier. IDs 0 through
                         32767 have the same meaning as for logical ingress
                         ports. IDs 32768 through 65535, inclusive, may be
                         assigned to logical multicast groups (see the
                         **tunnel_key** column in the OVN Southbound
                         **Multicast_Group** table).
    
           When VXLAN is enabled on any hypervisor in a cluster, datapath and
           egress port identifier ranges are reduced to 12-bits. This is done
           because only Geneve provides the large space for metadata (over 32
           bits per packet). The mode with reduced ranges is called **VXLAN**
           **mode**. To accommodate for VXLAN, 24 bits available are split as
           follows:
    
                  •      12-bit logical datapath identifier, derived from the
                         **tunnel_key** column in the OVN Southbound
                         **Datapath_Binding** table.
    
                  •      12-bit logical egress port identifier. IDs 0 through
                         2047 are used for unicast output ports. IDs 2048
                         through 4095, inclusive, may be assigned to logical
                         multicast groups (see the **tunnel_key** column in the
                         OVN Southbound **Multicast_Group** table). For multicast
                         group tunnel keys, a special mapping scheme is used
                         to internally transform from internal OVN 16-bit
                         keys to 12-bit values before sending packets through
                         a VXLAN tunnel, and back from 12-bit tunnel keys to
                         16-bit values when receiving packets from a VXLAN
                         tunnel.
    
                  •      No logical ingress port identifier.
    
           The limited space available for metadata when VXLAN tunnels are
           enabled in a cluster put the following functional limitations onto
           features available to users:
    
                  •      The maximum number of networks is reduced to 4096.
    
                  •      The maximum number of ports per network is reduced
                         to 2048.
    
                  •      ACLs matching against logical ingress port
                         identifiers are not supported.
    
                  •      OVN interconnection feature is not supported.
    
           In addition to functional limitations described above, the
           following should be considered before enabling it in your cluster:
    
                  •      Geneve uses randomized UDP or TCP source ports that
                         allows efficient distribution among multiple paths
                         in environments that use ECMP in their underlay.
    
                  •      NICs are available to offload Geneve encapsulation
                         and decapsulation.
    
           Due to its flexibility, the preferred encapsulation between
           hypervisors is Geneve. For Geneve encapsulation, OVN transmits the
           logical datapath identifier in the Geneve VNI. OVN transmits the
           logical ingress and logical egress ports in a TLV with class
           0x0102, type 0x80, and a 32-bit value encoded as follows, from MSB
           to LSB:
    
             1       15          16
           +---+------------+-----------+
           |rsv|ingress port|egress port|
           +---+------------+-----------+
             0
    
           For connecting to gateways, in addition to Geneve, OVN supports
           VXLAN, because VXLAN-only support is common on top-of-rack (ToR)
           switches. Currently, gateways have a feature set that matches the
           capabilities as defined by the VTEP schema, so fewer bits of
           metadata are necessary. In the future, gateways that do not
           support encapsulations with large amounts of metadata may continue
           to have a reduced feature set.
    
           **VXLAN mode** is recommended to be disabled if VXLAN encap at
           hypervisors is needed only to support HW VTEP L2 Gateway
           functionality. See man ovn-nb(5) for table **NB_Global** column
           **options** key **vxlan_mode** for more details.
    


* * *