[SONiC Virtual Switch] EVPN-VXLANv4 / BGP unnumbered
Tags: sonic, bgp, vxlan, evpn
Published: 2022-08-15 20:25:00

# Summary
I tried BGP unnumbered and the combination with EVPN-VXLAN in the [SONiC on CML environment that we had previously built](https://dev.to/rhira777/how-to-use-sonic-on-cml-4j2m).

In this case, the virtual switch version of SONiC.202205 was configured with a minimum of communication checks in mind, and were able to achieve IPv4 forwarding using BGP unnumbered and L2 forwarding using EVPN-VXLAN.

The following is a record of that.

The main information that I was referring to is as follows:

[Edgecore SONiC] BGP Unnumbered
https://support.edge-core.com/hc/en-us/articles/900002377366--Edgecore-SONiC-BGP-Unnumbered

[Edgecore SONiC] EVPN L2 VxLAN
https://support.edge-core.com/hc/en-us/articles/900002720523--Edgecore-SONiC-EVPN-L2-VxLAN

[Edgecore SONiC] VLAN & Inter-VLAN Routing
https://support.edge-core.com/hc/en-us/articles/900000210426-VLAN-Inter-VLAN-Routing

SONiC USER MANUAL
https://github.com/sonic-net/SONiC/blob/master/doc/SONiC-User-Manual.md

SONiC Command Line Interface Guide
https://github.com/sonic-net/sonic-utilities/blob/master/doc/Command-Reference.md

Vxlan SONiC
https://github.com/sonic-net/SONiC/blob/master/doc/vxlan/Vxlan_hld.md


# Detail of what I did
### Network diagram(Topology)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cnhm8towao3skucbyq6d.JPG)

### Network information
#### <Transport-related>
| node | hostname | Loopback0 | ASN |
| ---- | ---- | ---- | ---- |
| sonicvsvm-0 | R001 | 10.0.0.1/32 | 65001 |
| sonicvsvm-1 | R002 | 10.0.0.2/32 | 65001 |
| sonicvsvm-2 | R011 | 10.0.0.11/32 | 65000 |
| sonicvsvm-3 | R012 | 10.0.0.12/32 | 65000 |
| sonicvsvm-4 | R013 | 10.0.0.13/32 | 65000 |
| sonicvsvm-5 | R014 | 10.0.0.14/32 | 65000 |
| sonicvsvm-6 | R101 | 10.0.0.101/32 | 65001 |
| sonicvsvm-7 | R102 | 10.0.0.102/32 | 65001 |

#### <Service-related>
| node | hostname | VLAN | VNI |
| ---- | ---- | ---- | ---- |
| sonicvsvm-6 | R101 | 2 | 5002 |
| sonicvsvm-7 | R102 | 2 | 5002 |

| node | interface | MAC address | VLAN | IP address | VRF |
| ---- | ---- | ---- | ---- | ---- | ---- |
| csr1000v-0 | Gi2 | 0000.0000.0101 | untag | 192.168.2.101 | 101-2 |
| | Gi3 | 0000.0000.0102 | untag | 192.168.2.102 | 102-2 |


### SONiC OS configuration (Excerpts)
#### <General>
- save and reload (execute as appropriate)

| node | command |
| ---- | ---- |
| sonicvsvm-0~7 | sudo config save -y |
| | sudo config reload -fy |

### <Transport-related>
- transit link: IPv6 link-local

| node | command |
| ---- | ---- |
| sonicvsvm-0~5 | sudo config ipv6 enable link-local |
| sonicvsvm-6/7 | sudo config interface ipv6 enable use-link-local-only Ethernet0 |
| | sudo config interface ipv6 enable use-link-local-only Ethernet4 |

- loopback: IPv4

| node | command |
| ---- | ---- |
| sonicvsvm-0 | sudo config interface ip add Loopback0 10.0.0.1/32 |
| sonicvsvm-1 | sudo config interface ip add Loopback0 10.0.0.2/32 |
| sonicvsvm-2 | sudo config interface ip add Loopback0 10.0.0.11/32 |
| sonicvsvm-3 | sudo config interface ip add Loopback0 10.0.0.12/32 |
| sonicvsvm-4 | sudo config interface ip add Loopback0 10.0.0.13/32 |
| sonicvsvm-5 | sudo config interface ip add Loopback0 10.0.0.14/32 |
| sonicvsvm-6 | sudo config interface ip add Loopback0 10.0.0.101/32 |
| | sudo config interface ip add Loopback0 10.255.0.101/32 secondary |
| sonicvsvm-7 | sudo config interface ip add Loopback0 10.0.0.102/32 |
| | sudo config interface ip add Loopback0 10.255.0.102/32 secondary |

#### <Service-related>
- VLAN

| node | command |
| ---- | ---- |
| sonicvsvm-6/7 | sudo config vlan add 2 |
| | sudo config vlan member add -u 2 Ethernet8 |

- VXLAN (EVPN compatible)

| node | command |
| ---- | ---- |
| sonicvsvm-6 | sudo config vxlan add VXLAN 10.255.0.101 |
| sonicvsvm-7 | sudo config vxlan add VXLAN 10.255.0.102 |
| sonicvsvm-6/7 | sudo config vxlan evpn_nvo add NVO VXLAN |
| | sudo config vxlan map add VXLAN 2 5002 |


### FRRouting configuration (Excerpts)
- sonicvsvm-0/1
```
no router bgp 65100
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor SERVICE peer-group
 neighbor SERVICE remote-as internal
 neighbor SERVICE update-source Loopback0
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 neighbor Ethernet8 interface peer-group TRANSPORT
 neighbor Ethernet12 interface peer-group TRANSPORT
 bgp listen range 10.0.0.0/8 peer-group SERVICE
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
 address-family l2vpn evpn
  neighbor SERVICE activate
  neighbor SERVICE route-reflector-client
 exit-address-family
exit

```

- sonicvsvm-2/3/4/5
```
no router bgp 65100
router bgp 65000
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 neighbor Ethernet8 interface peer-group TRANSPORT
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
exit

```

- sonicvsvm-6/7
```
no router bgp 65100
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor SERVICE peer-group
 neighbor SERVICE remote-as internal
 neighbor SERVICE update-source Loopback0
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor 10.0.0.1 peer-group SERVICE
 neighbor 10.0.0.2 peer-group SERVICE
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
 address-family l2vpn evpn
  neighbor SERVICE activate
  advertise-all-vni
 exit-address-family
exit

```

### Confirmation of operation
#### <Communication>
- csr1000v-0
```
HOST#ping vrf 101-2 192.168.2.102 source 192.168.2.101 repeat 3
Type escape sequence to abort.
Sending 3, 100-byte ICMP Echos to 192.168.2.102, timeout is 2 seconds:
Packet sent with a source address of 192.168.2.101
.!!
Success rate is 66 percent (2/3), round-trip min/avg/max = 5/5/6 ms
HOST#
HOST#!#ping:ok
```

#### <Captured packets>
- csr1000v-0 ~ sonicvsvm-7
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u07szbcyalc4nf3kxaqi.JPG)

- sonicvsvm-5 ~ sonicvsvm-7
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ma0u0sve7wlw66bei7i.JPG)


# Supplemental information and others
### Status of IPv4 FIB
In the preparation process, IPv4 routes exchanged by BGP were being imported, but IPv4 communication for those routes was failing. In this case, the entries displayed in "show ip route" were not displayed in "ip route show".
This problem was resolved by reloading once.

- before reloading
```
admin@R101:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>r 10.0.0.1/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.2/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.11/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:05:53
B>r 10.0.0.12/32 [20/0] via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:05:53
B>r 10.0.0.13/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.14/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
C>* 10.0.0.101/32 is directly connected, Loopback0, 10:02:07
B>r 10.0.0.102/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                      via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19

admin@R101:~$
admin@R101:~$ ip route show
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown
admin@R101:~$
admin@R101:~$ ping 10.0.0.102 -I 10.0.0.101 -c 3
PING 10.0.0.102 (10.0.0.102) from 10.0.0.101 : 56(84) bytes of data.

--- 10.0.0.102 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2027ms

admin@R101:~$
```
- after reloading
```
admin@R101:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 10.0.0.1/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
  *                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.2/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
  *                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.11/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
B>* 10.0.0.12/32 [20/0] via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.13/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:25
  *                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:25
B>* 10.0.0.14/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:17
  *                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:17
C>* 10.0.0.101/32 is directly connected, Loopback0, 00:06:34
B>* 10.0.0.102/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:25
  *                      via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:25

admin@R101:~$
admin@R101:~$ ip route show
10.0.0.1 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.2 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.11 nhid 227 via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 proto bgp src 10.0.0.101 metric 20
10.0.0.12 nhid 228 via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 proto bgp src 10.0.0.101 metric 20
10.0.0.13 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.14 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.102 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown
admin@R101:~$
admin@R101:~$ ping 10.0.0.102 -I 10.0.0.101 -c 3
PING 10.0.0.102 (10.0.0.102) from 10.0.0.101 : 56(84) bytes of data.
64 bytes from 10.0.0.102: icmp_seq=1 ttl=61 time=3.81 ms
64 bytes from 10.0.0.102: icmp_seq=2 ttl=61 time=3.29 ms
64 bytes from 10.0.0.102: icmp_seq=3 ttl=61 time=3.05 ms

--- 10.0.0.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.045/3.381/3.806/0.316 ms
admin@R101:~$
```

### Status of bridge table
After successful L2 communication, the EVPN Route Type 2 routes exchanged by BGP were imported, but their entries were not displayed in output such as "show mac".
In this environment, it is possible that this is a display problem or that unicast communication also success as BUM forwarding. It may be possible to proceed to isolate this point, for example, by adding NVEs.

- sonicvsvm-6
```
admin@R101:~$ #ping:ok
admin@R101:~$
admin@R101:~$ show mac
  No.    Vlan  MacAddress         Port       Type
-----  ------  -----------------  ---------  -------
    1       2  00:00:00:00:01:01  Ethernet8  Dynamic
Total number of entries 1
admin@R101:~$
admin@R101:~$ vtysh

Hello, this is FRRouting (version 8.2.2).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

R101#
R101# show bgp l2vpn evpn
BGP table version is 9, local router ID is 10.0.0.101
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.101:1
*> [2]:[0]:[48]:[00:00:00:00:01:01]
                    10.255.0.101                       32768 i
                    ET:8 RT:65001:5002
*> [3]:[0]:[32]:[10.255.0.101]
                    10.255.0.101                       32768 i
                    ET:8 RT:65001:5002
Route Distinguisher: 10.0.0.102:1
* i[2]:[0]:[48]:[00:00:00:00:01:02]
                    10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
*>i                 10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
* i[3]:[0]:[32]:[10.255.0.102]
                    10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
*>i                 10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8

Displayed 4 out of 6 total prefixes
R101#
```