# Virtual Private LAN Service – VPLS Basic Info – 2 #

L2 PROTOCOL FORWARDING OVER VPLS

In the first post about VPLS I tried to learn the very basic concept about Virtual Private LAN Services, now I want to move on some default behaviour that VPLS brings to us. Suppose that a new customer asks us to offer a L2VPN service that could connect at Layer2 their two remotely located buildings, for redundancy reasons both buildings will be connected to different PEs. Customer wants to extend transparently its vlans (1-10) from one site to another and preserve its L2 topology in terms of spanning-tree and other L2 protocols. The wanted behaviour is represented in the following figure:

Let’s check how CSR1Kv deals with L2 protocols, let’s start from CDP.

I manually assigned mac addresses to each switch interface to easy reading (look also at the topology above):

SW1-Site1#sh int e0/0 | i Hard
Hardware is AmdP2, address is aaaa.1111.1111 (bia aabb.cc00.0300)
SW1-Site1#sh int e3/3 | i Hard
Hardware is AmdP2, address is aaaa.1111.3333 (bia aabb.cc00.0333)

SW2-Site1#sh int e0/0 | i Hard
Hardware is AmdP2, address is aaaa.2222.1111 (bia aabb.cc00.0700)
SW2-Site1#sh int e3/3 | i Hard
Hardware is AmdP2, address is aaaa.2222.3333 (bia aabb.cc00.0733)

SW1-Site2#sh int e0/0 | i Hard
Hardware is AmdP2, address is cccc.1111.1111 (bia aabb.cc00.0800)
SW1-Site2#sh int e3/3 | i Hard
Hardware is AmdP2, address is cccc.1111.3333 (bia aabb.cc00.0833)

SW2-Site2#sh int e0/0 | i Hard
Hardware is AmdP2, address is cccc.2222.1111 (bia aabb.cc00.0600)
SW2-Site2#sh int e3/3 | i Hard
Hardware is AmdP2, address is cccc.2222.3333 (bia aabb.cc00.0633)

On switches CDP is enabled by default:

SW1-Site1#sh cdp
Global CDP information:
Sending CDP packets every 60 seconds
Sending a holdtime value of 180 seconds
Sending CDPv2 advertisements is enabled

All interfaces of the switches looking at PEs are configured as dot1q trunks carrying vlans in the range 1-10, for example e0/0 on SW1-Site1 is:

SW1-Site1#sh run int e0/0 | b interface
interface Ethernet0/0
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1-10
switchport mode trunk
mac-address aaaa.1111.1111

Let’s capture a CDP packet sent by the switch and let’s verify that it is an untagged packet sent over the default native vlan 1, this packet is captures on the link between SW1-Site1 and PE1:

I see that the packet is an untagged packet. How does CSR1Kv handles the CDP packet in the context of VPLS service? I know that to deploy a VPLS service I need:

– A service instance under the physical interface matching the packet (tagged or untagged) that comes in, since we verify that CDP is an untagged packet we need a service
instance that match untagged packet, let’s call this instance 999:

PE1(config)#int G2
PE1(config-if)#service instance 999 ethernet
PE1(config-if-srv)#encapsulation untagged

– A l2vpn Virtual Forwarding Instance (VFI) that connects other two PEs via Pseudowires, let its name be L2-CU1:

PE1#show run | s l2vpn
l2vpn vfi context L2-CU1
vpn id 999
member 10.3.3.3 13 encapsulation mpls
member 10.2.2.2 12 encapsulation mpls

– A Bridge-Domain linking the Service Instance with the VFI, let’s configure Bridge-Domain 999 (use the number you want)

PE1#sh run bridge-domain 999
Building configuration…
Current bridge-domain configuration:
999
member GigabitEthernet2 service-instance 999
member vfi L2-CU1
!

Similar VPLS configuration it’s present on the other two PEs. Is this enough to have Switches of the customer seeing each other as CDP neighbors via MPLS core? Let’s check:

VPLS service is working fine:

PE1#show mpls l2transport vc
Local intf     Local circuit              Dest address    VC ID      Status
————-  ————————– ————— ———- ———-
VFI L2-CU1     vfi                        10.2.2.2        12         UP
VFI L2-CU1     vfi                        10.3.3.3        13         UP

PE2#show mpls l2transport vc
Local intf     Local circuit              Dest address    VC ID      Status
————-  ————————– ————— ———- ———-
VFI L2-CU1     vfi                        10.1.1.1        12         UP
VFI L2-CU1     vfi                        10.3.3.3        23         UP

PE3#show mpls l2transport vc
Local intf     Local circuit              Dest address    VC ID      Status
————-  ————————– ————— ———- ———-
VFI L2-CU1     vfi                        10.1.1.1        13         UP
VFI L2-CU1     vfi                        10.2.2.2        23         UP

But CDP packets are not transported between switches, switches are neighbors only within their site:

SW1-Site1#sh cdp neighbors | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW2-Site1        Eth 3/3           146             R S I  Linux Uni Eth 3/3

SW2-Site1#sh cdp neighbors | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW1-Site1        Eth 3/3           147             R S I  Linux Uni Eth 3/3

This means that by default CSR1Kv doesn’t forward L2protocol packets inside the VPLS domain. Let’s check then how to tell CSR1Kv that we want to forward these packets, the first piece of VPLS, the packet hits entering PE1 interface G2, is the service instance 999 that matches it:

PE1#show ethernet service instance id 999 interface G2 detail
Service Instance ID: 999
Service Instance Type: Static
Associated Interface: GigabitEthernet2
Associated EVC:
L2PROTOCOL DROP
CE-Vlans:
Encapsulation: untagged
Interface Dot1q Tunnel Ethertype: 0x8100
State: Up
EFP Statistics:
Pkts In   Bytes In   Pkts Out  Bytes Out
2033     319745          0          0
EFP Microblocks:
****************
Microblock type: Bridge-domain
Bridge-domain: 999

Microblock type: L2Mcast
L2 Multicast GID: 1

Microblock type: dhcp_snoop
L2 Multicast GID: 1

We see that we are receiving packet on this service instance but no packet are flowing in the Out direction, I see also that the default action on L2protocol PDUs is “DROP”, then I need a command to instruct PE1 to forward these L2protocol PDUs inside the VPLS domain, the command is this one:

PE1(config)#int G2
PE1(config-if)#service instance 999 ethernet
PE1(config-if-srv)#l2protocol forward cdp

PE1#show ethernet service instance id 999 interface G2 detail | i L2proto
L2protocol forward cdp ==> now we are forwarding cdp packets, but where PE1 forwards this packets?

Our target is to have switches as cdp neighbors, for the moment I leave powered off PE3 so the end result should be that SW1-Site1 and SW1-Site2 become cdp neighbors via VPLS service, let’s configure PE2 too to forward cdp packets under its service instance:

PE2#sh run int G2 | b interface
interface GigabitEthernet2
no ip address
negotiation auto
service instance 999 ethernet
encapsulation untagged
l2protocol forward cdp

PE2#show ethernet service instance id 999 int G2 detail | i L2proto
L2protocol forward cdp

Is this enough to have SW1-Site1 and SW1-Site2 cdp neighbors? Let’s see:

SW1-Site1#sh cdp neighbors | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW2-Site1        Eth 3/3           179             R S I  Linux Uni Eth 3/3

SW1-Site2#sh cdp neighbors | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW2-Site2        Eth 3/3           137             R S I  Linux Uni Eth 3/3

Answer is no, though I have enabled l2protocol forward for cdp packets, cdp packets don’t reach the switch in the other site and switches see as cdp neighbors only their local peer over the e3/3 trunk.

Then which is the missing point? I have a VPLS service up, I enabled l2protocol forward for cdp but cdp doesn’t work over VPLS. To solve the puzzle it’s worth thinking to a PE router in terms of VPLS service- look here to have an idea of this PE logical function – when the cdp packet is received on the attachment circuit (service instance 999) it is forwarded to the bridge module inside the virtual switch, then it needs to cross the VFI software structure to reach the pseudowire that connect bridge-domain 999 on PE1 to bridge-domain 999 on PE2, what we need to do is then to enable the forwarding of cdp packets under the VFI too:

PE1(config)#l2vpn vfi context L2-CU1
PE1(config-vfi)#l2protocol forward cdp ==> when you enable this, VFI status flaps to DOWN and then COMES BACK to UP

*May 24 2017 11:07:52.007 ITALY: %VFI-6-VFI_STATUS_CHANGED: Status of VFI L2-CU1 changed from UP to DOWN
*May 24 2017 11:07:52.008 ITALY: %VFI-6-VFI_STATUS_CHANGED: Status of VFI L2-CU1 changed from DOWN to UP

Let’s do the same with PE2:

PE2(config)#l2vpn vfi context L2-CU1
PE2(config-vfi)#l2protocol forward cdp

*May 24 2017 11:10:45.995 ITALY: %VFI-6-VFI_STATUS_CHANGED: Status of VFI L2-CU1 changed from UP to DOWN
*May 24 2017 11:10:45.995 ITALY: %VFI-6-VFI_STATUS_CHANGED: Status of VFI L2-CU1 changed from DOWN to UP

What our switches do after that?

SW1-Site1#sh cdp neighbors | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW1-Site2        Eth 0/0           148             R S I  Linux Uni Eth 0/0
SW2-Site1        Eth 3/3           154             R S I  Linux Uni Eth 3/3

SW1-Site2#sh cdp neighbors | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW1-Site1        Eth 0/0           122             R S I  Linux Uni Eth 0/0
SW2-Site2        Eth 3/3           131             R S I  Linux Uni Eth 3/3

Now switches see each other through interface e0/0 that looks toward their PEs.

Let’s activate also PE3 and let’s see the final results:

SW1-Site1#sh cdp neighbors | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW1-Site2        Eth 0/0           149             R S I  Linux Uni Eth 0/0
SW2-Site1        Eth 0/0           92              R S I  Linux Uni Eth 0/0
SW2-Site1        Eth 3/3           150             R S I  Linux Uni Eth 3/3
SW2-Site2        Eth 0/0           95              R S I  Linux Uni Eth 0/0

SW1-Site2#sh cdp neighbors | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW2-Site1        Eth 0/0           56              R S I  Linux Uni Eth 0/0
SW1-Site1        Eth 0/0           172             R S I  Linux Uni Eth 0/0
SW2-Site2        Eth 0/0           59              R S I  Linux Uni Eth 0/0
SW2-Site2        Eth 3/3           177             R S I  Linux Uni Eth 3/3

SW2-Site1#show cdp neighbors | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW1-Site2        Eth 0/0           116             R S I  Linux Uni Eth 0/0
SW1-Site1        Eth 0/0           114             R S I  Linux Uni Eth 0/0
SW2-Site2        Eth 0/0           60              R S I  Linux Uni Eth 0/0
SW1-Site1        Eth 3/3           172             R S I  Linux Uni Eth 3/3

SW2-Site2#sh cdp neighbors | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW1-Site2        Eth 0/0           119             R S I  Linux Uni Eth 0/0
SW2-Site1        Eth 0/0           120             R S I  Linux Uni Eth 0/0
SW1-Site2        Eth 3/3           177             R S I  Linux Uni Eth 3/3
SW1-Site1        Eth 0/0           177             R S I  Linux Uni Eth 0/0

From this print we see that Switches see each other via CDP, each switch sees three other switches via VPLS service and of course via direct link e3/3. The following picture shows the virtual topology of the switch connected via VPLS:

Now take the same virtual topology and think in terms of STP instead of CDP, the same concepts, about forwarding L2protocol PDUs, apply to STP, we need to forward STP PDUs inside the VPLS domain to have the wanted STP topology. Using VPLS made of 3 fully meshed PWs we see that it’s like to have 6 links between the 4 switches, STP needs to block some of the links to have a L2 free topology.

I configured 2 MST instances to map the 10 Vlans (1-10) used by the customer, instance 1 will map to odd vlans, and instance 2 will map to even vlans. Also I configured MST to have this bridge-root placement:

SW1-Site1#show run | s mst
spanning-tree mode mst
spanning-tree mst configuration
name VPLS-REGION
revision 1
instance 1 vlan 1, 3, 5, 7, 9
instance 2 vlan 2, 4, 6, 8, 10
spanning-tree mst 0,2 priority 4096
spanning-tree mst 1 priority 0

SW2-Site1#show run | s mst
spanning-tree mode mst
spanning-tree mst configuration
name VPLS-REGION
revision 1
instance 1 vlan 1, 3, 5, 7, 9
instance 2 vlan 2, 4, 6, 8, 10
spanning-tree mst 0,2 priority 0
spanning-tree mst 1 priority 4096

SW1-Site2#sh run | s mst
spanning-tree mode mst
spanning-tree mst configuration
name VPLS-REGION
revision 1
instance 1 vlan 1, 3, 5, 7, 9
instance 2 vlan 2, 4, 6, 8, 10
spanning-tree mst 0,2 priority 16384
spanning-tree mst 1 priority 8192

SW2-Site2#show run | s mst
spanning-tree mode mst
spanning-tree mst configuration
name VPLS-REGION
revision 1
instance 1 vlan 1, 3, 5, 7, 9
instance 2 vlan 2, 4, 6, 8, 10
spanning-tree mst 0,2 priority 8192
spanning-tree mst 1 priority 16384

With this configuration SW1-Site1 will be the Root Bridge (RB) for MST1 and Backup Root Bridge (BRB) for MST2, SW2-Site1 will be RB for MST2 and BRB for MST1, Switches in Site2  will assume the roles of RB and BRB if Switches in Site1 fail. To have switches speaking MST we need to forward MST PDUs over VPLS, the command to use is:

PE1#show run int G2 | b interface
interface GigabitEthernet2
no ip address
service instance 999 ethernet
encapsulation untagged
L2PROTOCOL FORWARD STP
!
PE1#show run | s l2vpn
l2vpn vfi context L2-CU1
vpn id 999
L2PROTOCOL FORWARD STP
member 10.2.2.2 12 encapsulation mpls
member 10.3.3.3 13 encapsulation mpls

Similar configurations must be applied to PE2 and PE3. Let’s check the final STP topology

SW1-Site1#show spanning-tree mst 1
##### MST1    vlans mapped:   1,3,5,7,9
Bridge        address aabb.cc00.0300  priority      1     (0 sysid 1)
Root          this switch for MST1
Interface        Role Sts Cost      Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Et0/0            Desg FWD 2000000   128.1    Shr
Et3/3            Desg FWD 2000000   128.16   Shr

SW2-Site1#show spanning-tree mst 1
##### MST1    vlans mapped:   1,3,5,7,9
Bridge        address aabb.cc00.0700  priority      4097  (4096 sysid 1)
Root          address aabb.cc00.0300  priority      1     (0 sysid 1)
port    Et0/0           cost          2000000   rem hops 19
Interface        Role Sts Cost      Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Et0/0            Root FWD 2000000   128.1    Shr
Et3/3            Altn BLK 2000000   128.16   Shr

SW1-Site2#show spanning-tree mst 1
##### MST1    vlans mapped:   1,3,5,7,9
Bridge        address aabb.cc00.0800  priority      8193  (8192 sysid 1)
Root          address aabb.cc00.0300  priority      1     (0 sysid 1)
port    Et0/0           cost          2000000   rem hops 19
Interface        Role Sts Cost      Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Et0/0            Root FWD 2000000   128.1    Shr
Et3/3            Desg FWD 2000000   128.16   Shr

SW2-Site2#show spanning-tree mst 1
##### MST1    vlans mapped:   1,3,5,7,9
Bridge        address aabb.cc00.0600  priority      16385 (16384 sysid 1)
Root          address aabb.cc00.0300  priority      1     (0 sysid 1)
port    Et0/0           cost          2000000   rem hops 19
Interface        Role Sts Cost      Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Et0/0            Root FWD 2000000   128.1    Shr
Et3/3            Altn BLK 2000000   128.16   Shr

The following figure shows the STP topology built over VPLS domain:

It’s possible to see how switches on Site2 see SW1-Site1 as root bridge for MST1, a similar topology exists for MST2 where SW2-Site1 is the root bridge; to avoid suboptimal Layer2 connectivity within Site1 it’s worth to modify port priority in such a way that the direct link on interfaces e3/3 is used as active link, leaving VPLS virtual link between SW1-Site1 and SW2-Site1 as backup link:

SW1-Site1(config)#int e3/3
SW1-Site1(config-if)#spanning-tree mst 1 port-priority 0

SW2-Site1(config)#int e3/3
SW2-Site1(config-if)#spanning-tree mst 2 port-priority 0

Here the results:

SW1-Site1#show spanning-tree mst 1
##### MST1    vlans mapped:   1,3,5,7,9
Bridge        address aabb.cc00.0300  priority      1     (0 sysid 1)
Root          this switch for MST1
Interface        Role Sts Cost      Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Et0/0            Desg FWD 2000000   128.1    Shr
Et3/3            Desg FWD 2000000     0.16   Shr

SW2-Site1#show spanning-tree mst 1
##### MST1    vlans mapped:   1,3,5,7,9
Bridge        address aabb.cc00.0700  priority      4097  (4096 sysid 1)
Root          address aabb.cc00.0300  priority      1     (0 sysid 1)
port    Et3/3           cost          2000000   rem hops 19
Interface        Role Sts Cost      Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Et0/0            Altn BLK 2000000   128.1    Shr
Et3/3            Root FWD 2000000   128.16   Shr

SW1-Site1#show spanning-tree mst 2
##### MST2    vlans mapped:   2,4,6,8,10
Bridge        address aabb.cc00.0300  priority      4098  (4096 sysid 2)
Root          address aabb.cc00.0700  priority      2     (0 sysid 2)
port    Et3/3           cost          2000000   rem hops 19
Interface        Role Sts Cost      Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Et0/0            Altn BLK 2000000   128.1    Shr
Et3/3            Root FWD 2000000   128.16   Shr

SW2-Site1#show spanning-tree mst 2
##### MST2    vlans mapped:   2,4,6,8,10
Bridge        address aabb.cc00.0700  priority      2     (0 sysid 2)
Root          this switch for MST2
Interface        Role Sts Cost      Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Et0/0            Desg FWD 2000000   128.1    Shr
Et3/3            Desg FWD 2000000     0.16   Shr

This concludes this basic example about VPLS and forwarding of L2protocol PDUs, I learned that we need to enable the forwarding of L2protocol PDUs (in this example cdp and stp) both under Service Instance and Virtual Forwarding Instance, once you enabled this forwarding you can play with VPLS virtual link as if they were true physcal links. Loop free topology is fostered within the VPLS domain by the default L2 split-horizon rules between the fully-meshed PEs.

Here you can find the final configuration I used for all the devices ==> Config – note that when trying to forward STP I disabled CDP on all devices to prevent excessive L2 traffic on this virtual setup.

A last note about the virtual setup I used: My laptop is an Intel Core I5 with 16Gb RAM running Windows 10, I created an Ubuntu Virtual Machine with VmWare Workstation giving 12GB RAM to this VM , inside this Ubuntu VM I installed GNS3 and used qemu to run the CSR1Kv that works as PEs, one qemu image is used as linked base for the other two CSR1Kv, each CSR1Kv is configured to use 2,5GB RAM, sometimes especially when running l2protocol forwarding for STP I saw some spike on CPU and I needed to increase memory for other P routers (from default 256Mb to 512Mb) to stop LDP flapping. When the network reaches a stable state [if you do the right thing with STP 😉 ] CPU usage was never a problem and RAM on the Ubuntu VM reach maximum 88/89%, hope this helps you in reproducing the scenario if you want.