Basic L3VPN (BGP/MPLS VPN or VPRN) configuration. Nokia (Alcatel-Lucent) & Juniper

00_intro_final

This time I will cover in details basic Layer 3 VPN (L3VPN or VPRN) configuration, and I decided to kill two birds with one stone by inviting Juniper vMX (read here on how to add vMX to Unetlab) to our cozy Alcatel-Lucent environment.

In this post we will configure BGP/MPLS VPN [RFC 4364] including:

  • PE-PE relationship configuration with VPN IPv4 family introduction
  • PE-CE routing configuration with both BGP and OSPF as routing protocols
  • Export policy configuration for advertising VPN routes on PE routers
  • AS override configuration
  • and many more

As a bonus track I will share with you Control Plane and Data Plane evaluation diagrams which help a lot for understanding the whole BGP VPN picture. Take your seats, there is no flying from VPNs!

I will emulate the following topology (via UNetLab, as always) where we have two customers (Alcatel and Juniper) which have two remote sites and want to get connectivity between them by means of L3VPN service:

We start off with ISIS running smoothly in our provider core so every router can reach every other’s loopback address. Another prerequisite is a MPLS enabled core, since L3VPN uses MPLS encapsulation for dataplane communication. I configured RSVP-TE tunnels between PE routers for this tutorial in the way that PE1_ALU can resolve PE2_JUN (and vice versa) loopback address via RSVP-TE tunnel.

 

BGP L3VPN terminology

Before we dive deep into BGP L3VPN configuration it is necessary to refresh some basic theory. To get a deeper and broader knowledge on the following topic please read Juniper’s JUNOS MPLS and VPNs student guide and Alcatel-Lucent’s Service Routing Architect guide.

VRFs

In order to maintain different customer’s routes independently PE routers use separate logical routing tables called Virtual Routing and Forwarding (VRF).

RFC 4364. VRFs: Multiple Forwarding Tables in PEs

Each PE router maintains a number of separate forwarding tables. One of the forwarding tables is the “default forwarding table”. The others are “VPN Routing and Forwarding tables”, or “VRFs”.

3.1. VRFs and Attachment Circuits
Every PE/CE attachment circuit is associated, by configuration, with one or more VRFs. An attachment circuit that is associated with a VRF is known as a “VRF attachment circuit”. In the simplest case and most typical case, a PE/CE attachment circuit is associated with exactly one VRF. When an IP packet is received over a particular attachment circuit, its destination IP address is looked up in the associated VRF. The result of that lookup determines how to route the packet.

Provider Edge routers must have a VRF configured for each connected site. VRFs are totally separated in routers control plane by default, so we can depict VRFs as routers nested in a single hardware:

VRFs are also local to PE routers and theirs number representation or names are never propagated to other PEs. In our example we have four VRFs:

  1. VRF Alcatel and VRF Juniper on site A
  2. VRF Alcatel and VRF Juniper on site B

Route Distinguisher

Since one router can be represented as many separate routers by means of VRF, it is necessary to help it to distinct between different routes in different VRFs. It is highly possible that different customers connected to a single PE will have overlapping IP addresses. I emulated this situation as well, see, Juniper’s loopback address for CE1/CE2 is overlapping with Alcatel’s customer loopback addresses. How will router PE1_ALU distinct between to routes?

Route Distinguisher (RD) comes to the rescue.

RFC 4364. Route Distinguisher definition

An RD is simply a number, and it does not contain any inherent information; it does not identify the origin of the route or the set of VPNs to which the route is to be distributed. The purpose of the RD is solely to allow one to create distinct routes to a common IPv4 address prefix.

RD can be written in several forms, but it is handy to use IP address as Administrator subfield and VPN number as Assigned number subfield:

VPN-IPv4 routes

Combination of Route Distinguisher and an IPv4 route effectively makes a VPN-IPv4 route. VPN-IPv4 routes are 12 byte length (8byte RD and 4byte IPv4) addresses exchanged by MP-BGP speakers. PE routers compose VPN-IPv4 and allocate MPLS labels for routes before sending them to MP-BGP neighbors. See an example of VPN-IPv4 from a picture dedicated to RD.

Route Targets

Yeah, Route Distinguishers makes every VPN-IPv4 route unique for Service Provider but we still need a mechanism to tell a PE router what VRFs a VPN-IPv4 routes belong to? As in classic BGP, community is a good method to solve this problem. For L3VPN specific extended community was defined [RFC 4364 Section 4.3.1] called Route Target.

RFC 4364. Route Target definition

Every VRF is associated with one or more Route Target (RT) attributes. When a VPN-IPv4 route is created (from an IPv4 route that the PE has learned from a CE) by a PE router, it is associated with one or more

Route Target attributes. These are carried in BGP as attributes of the route. Any route associated with Route Target T must be distributed to every PE router that has a VRF associated with Route Target T. When such route is received by a PE router, it is eligible to be installed in those of the PE’s VRFs that are associated with Route Target T. (Whether it actually gets installed depends upon the outcome of the BGP decision process, and upon the outcome of the decision process of the IGP (i.e., the intra-domain routing protocol) running on the PE/CE interface.) A Route Target attribute can be thought of as identifying a set of sites. (Though it would be more precise to think of it as identifying a set of VRFs.) Associating a particular Route Target attribute with a route allows that route to be placed in the VRFs that are used for routing traffic that is received from the corresponding sites.

There is a set of Route Targets that a PE router attaches to a route received from site S; these may be called the “Export Targets”. And there is a set of Route Targets that a PE router uses to determine whether a route received from another PE router could be placed in the VRF associated with site S; these may be called the “Import Targets”. The two sets are distinct, and need not be the same. Note that a particular VPN-IPv4 route is only eligible for installation in a particular VRF if there is some Route Target that is both one of the route’s Route Targets and one of the VRF’s Import Targets.

Most of the time RTs are represented as <AS Number of a client network>:<VRF ID>

PE<->PE MP-BGP configuration

First thing to accomplish in L3VPN configuration is BGP peering inside provider’s core network. We have two Provider Edge routers (PE) and one core provider (P) router in our simple network, and we plan to provide L3VPN service to our beloved customers (JUN and ALU). To do so we need to configure BGP peering between all PE routers involved in L3VPN service providing, these two routers are PE1_ALU and PE2_JUN.

Their BGP configuration is simple (read BGP configuration tutorial to grab the basics), the only part which is different from a classic iBGP peering is a necessity of a new BGP address family. We need to enable this address family to deal with VPN routes, which are different from IPv4 routes.

In Juniper this family is called inet-vpn, in ALU it is vpn-ipv4, but nonetheless it is just an address family which enables communication of VPN routes between BGP peers. We will see later how this family differs from a classic IPv4, but for now just look at the BGP configuration part for both PE routers

As you see, the only part which is related to L3VPN – is this new VPN address family (I highlighted strings where I configured this VPN family). Support for an additional family transforms classic BGP to Multi-Protocol BGP [RFC 4760]. Lets see how this family is communicated in BGP messages (wireshark dump):

Both routers announces in BGP OPEN messages their capability to exchange VPN Unicast IPv4 routes and at this point thats all. After the BGP OPEN messages go KEEPALIVEs and nothing else. This simple configuration enables our PE routers to speak with VPN IPv4 routes.

Configuring VRFs on PE1_ALU (Alcatel-Lucent)

We configured PE-PE relationship which is a base for a L3VPN service. Our next step is VRFs configuration which serve as a customers facing dedicated routers. We start with PE1_ALU and will configure VRFs 20 and 30.

1. Ports configuration

At first we should ensure that customer facing ports operate in access mode.

2. Customers configuration

Once we have our ports ready we can move on to L3VPN configuration. SR OS uses concept of customers which is similar to tenants in virtualization world. I will define two customers (Customer 1 is a default one):

3. VRF configuration

After that I will create VPRN service (which is a fancy Alcatel-Lucent’s name for BGP VPN) for each customer:

VRF 30 configuration repeats the same steps:

4. Configuring PE -> CE routing protocols

Ok, our VRFs 20 and 30 are configured on PE1_ALU router and have customers interfaces attached. What we need to do next is to configure a routing protocol which will propagate customers routes to PE router. On PE1_ALU router both customers decided to use BGP as a routing protocol. Lets configure BGP instances for VRFs 20 and 30:

AS override

I used as-override command here to resolve an issue with AS-PATH loop prevention mechanism. When a route goes from CE1_JUN over eBGP to PE1_ALU it has AS-PATH value of 200. Then this route update message leaves Service Provider’s AS and goes over eBGP session to CE2_JUN with AS-PATH value of “100 200”. But CE2_JUN is a part of AS 200 itself, so it will silently discard a route update with AS-PATH value containing its AS number.

as-override command placed under BGP context of the receiving VRF replaces customers AS number  with its own AS, so AS-PATH string of “100 200” will become “100 100” and will be accepted by the CE router residing in AS 200 since no loop will be detected.

Export policies

Note, that it is mandatory for Alcatel-Lucent SR OS to create an export policy for incoming routes to leave VRF over PE -> CE routing protocol to the CE router:

Now add this policy under bgp context in the VRF:

Super, now repeat the steps for VRF 30 (Alcatel) and observe the whole services configuration part on PE1_ALU:

Configuring VRFs on PE2_JUN (Juniper)

Juniper JUNOS does not use concept of network/access ports, thats why you deal with CE-facing interfaces just like with normal ones:

Then go and configure a VRF. VRF is called routing-instance in JUNOS.

VRF configuration on Juniper looks almost identical to ALU’s. The major difference here is that you dont have to tell JUNOS to resolve VRF’s next-hop address via MPLS tunnel. And you dont have to configure export policy in case you are using eBGP as PE-CE protocol. Juniper defaults to that behavior.

Now configure PE -> CE routing protocols.

For VRF 20 (note that with Juniper we may omit explicit configuration of autonomous system number under nested BGP speaker configuration. Globally configured for PE AS number will be used:

For VRF 30 we have OSPF adjacency with CE router. This adds some configuration since we have to configure export policy to let vpn-ipv4 routes imported into VRF 30 to be exported to CE2_ALU over OSPF.

Configure an export policy first:

Then configure PE-CE OSPF protocol with export policy applied:

Complete VRF configuration for PE2_JUN goes like this:

CE -> PE routing protocol configuration

Now it is time to connect our customers to the service provider’s network via VRFs created earlier and finally add some VPN routes to exchange. CE routers completely unaware of a complex L3VPN configuration on PE routers, what they need to do is just setup routing protocol over which customers routes could be delivered to the Service Provider. Let’s cover them all starting with Juniper. Both CE1_JUN and CE2_JUN run eBGP with PE routers, so they configuration is almost identical

Now its Alcatel-Lucent’s CE routers time. Pay attention to CE2_ALU router, since we are using OSPF on CE2-PE2 link configuration is a little bit different from other CE’s.

Control plane walkthrough

We are done with configuration, all in all it was not a complex task, what is more important is an understanding what’s going on with control and data planes? I believe you will like this step-by-step walk through every node in the network.

I will start with dissection of a control plane operation from the point where MP-BGP session between PE routers has been already established and we are enabling VRFs on customers routers. Refer to this overview chart and see how an information about CE1_JUN’s loopback interface propagates through the entire network to CE2_JUN counterpart:

All pictures are clickable, to see the full sized pics choose “open an image in a separate tab” option in your browser.

Steps 1-3

Step 1
CE1_JUN router has an export policy export_loopback configured which is used by BGP to construct the BGP UPDATE message with lo0 prefix as NLRI.

Step 2
CE1_JUN sends a regular BGP UPDATE message to its eBGP peer PE1_ALU.

Step 3
PE1_ALU router receives this update via its interface toCE1 configured in  vprn 20  context. PE1_ALU populates its VRF 20 with a route to 1.1.1.1/32 via 10.20.99.1.

Step 4

PE1_ALU router has an established MP-iBGP session with PE2_JUN so it takes a BGP route from VRF 20 and automatically sends a MP-BGP UPDATE message to its peer. Note, that ALU routers will send MP-BGP update automatically only for the connected to VRF routes and the routes received via BGP. If we had OSPF between CE1 and PE1, we would configure an export policy to propagate this update over MP-BGP session.

Since PE1_ALU router wants to send an update for a VRF’s route it should construct a MP-BGP Update message which has a specific Path attribute – MP_REACH_NLRI – to communicate routing information. And PE1_ALU will transform the 1.1.1.1/32 IPv4 prefix to a VPN-IPv4 one.

Take a close look at this BGP message. See how PE1_ALU router added some valuable additional information to correctly pass CE1_ALU’s loopback address via MP-BGP. First of all examine how NLRI has been transformed in MP-BGP: it now has a route distinguisher which we configured for VRF 20 earlier, it has the IPv4 prefix itself and it has the MPLS label (131068 value)!

PE1_ALU router allocated a VPN label which it associated with VRF 20. This label tells PE1_ALU router that if it ever receives a data packet with this label it should consult encapsulated with this label data with VRF 20! This way ingress PE routers tell others what label should be used as a VPN label for a routes residing in particular VRF.

There are two methods of allocating VPN labels:

  1. per VRF. all routes originated from a single VRF will have the same VPN label. Alcatel-Lucent routers default to this.
  2. per VRF per next-hop. If a VRF has >1 CE interfaces, PE router will allocate different labels for different CE interfaces inside one VRF. Juniper routers default to this.

And of course we see the Route Target 200:20 as a part of Extended Community attribute.

Important things happened to the Next-Hop, not only Next-Hop looks now like VPN-IPv4 route with Route Distinguisher value of 0:0 and without MPLS label, but Next-Hop IPv4 address has been changed to PE1_ALU system (loopback) interface 10.10.10.1. This is how PE1 router tells PE2 that PE2 can reach VRF 20 routes via PE1.

All in all, PE1_ALU’s update reaches PE2_JUN since it has the IP destination address of 10.10.10.3.

Notice, BGP updates traverse Service Provider’s network as simple IP packets, MPLS is out of play at the moment. P1_ALU router simply routes IP packets and has no knowledge about BGP at all.

Steps 5-7

Step 5
PE2_JUN receives the BGP UPDATE with VPN-IPv4 route. Once this route passes validation checks (Nexhop resolvable, no AS Path loop) it places this route to a specific table called bgp.l3vpn.0 . This table stores all BGP VPN routes, refer to this figure to examine some of its content:

PE2_JUN extracts routing information from this update an based on Route Target value and installs an IPv4 route 1.1.1.1/32 into VRF 20 – 20.inet.0 . PE2_JUN resolved next-hop address of the fellow PE1_ALU (10.10.10.1) via MPLS Label Switched Path (LSP) and placed this information in the  20.inet.0 table:

Remember that it is mandatory to have an active LSP to the remote PE, since we have to have MPLS transport to remote end.

Step 6
Since we installed route for the 1.1.1.1/32 IPv4 prefix into VRF 20 and we have an active eBGP peer in VRF 20, we should send an update for this IPv4 prefix to CE2_JUN router. This update goes as an ordinary eBGP update

Step 7
CE2_JUN receives BGP UPDATE and installs a route into the only table it has for IPv4 routes – inet.0 .

This completes Control Plane operation regarding the prefix  1.1.1.1/32 , same process goes for other loopbacks and connected to VRFs link addresses for both Alcatel and Juniper customers.

Data plane walkthrough

To complete this post we should see how data plane uses all the hard work control plane did. We will see how data packets destined to  1.1.1.1 propagate through our network.

Step 1
CE2_JUN wants to send a data packet to CE1_JUN via L3VPN service provided by our network. CE2 has an active route in its route table inet.0  pointing to connected address 10.20.99.2 via ge-0/0/0(fig. 1.1). CE2 has data layer MAC address (fig. 1.2) so it constructs the whole packet and puts it on the wire.

Step 2
PE2_JUN receives Ethernet frame on the interface ge-0/0/1 which belongs to VRF 20, that is how PE2 decides to associate this packet to VRF 20. PE2 consults with VRF 20 routing table and sees that it has to use LSP toPE1 (fig. 2.1) to propagate incoming data packet further. Then PE2 gets MPLS label which it received earlier from RSVP neighbor P1_ALU (fig. 2.2) during LSP signalization process.

But this was just a transport MPLS label, it helps PE2_JUN to reach PE1_ALU, but PE2 needs one more – VPN Label – to tell PE1_ALU to which VRF it should put encapsulated data. This label was signalled earlier (see Control Plane operation section) via MP-BGP (fig. 2.3).

Now PE2 has everything it needs:

  1. MPLS VPN label to encapsulate data packets from its VRF 20 destined to VRF 20 on PE1_ALU
  2. Transport MPLS Label to get to PE1_ALU via MPLS core

and thus it constructs a packet (fig. 2.4) with two labels stacked and fires it off.

Step 3
P1_ALU is totally unaware of the whole services and customers mess, it just switches MPLS packets (fig. 3.1, 3.2) by replacing incoming transport label with outcoming one.

Step 4
PE1_ALU receives an MPLS packet from P1_ALU. It pops out the transport label (fig. 4.1) and examines enclosed MPLS label. This label value 131068 was signalled by PE1_ALU via MP-BGP during Control Plane operation. So PE1 knows that it has to pop this label and associate enclosed packet with VPRN 20 (VRF 20) (fig. 4.2)

VRF’s 20 routing table says that packets destined to 1.1.1.1 should be forwarded to 10.20.99.3 address (fig. 4.3), which is a connected network leading to CE1_JUN (fig. 4.4). PE1_ALU constructs the packet and moves it via Ethernet out of the 1/1/2 port (fig. 4.5).

Step 5
CE2_JUN receives an ordinary packet with a destination address matching its interface. It decapsulates ICMP echo request and send back echo reply.

Roman Dodin

Network engineer at Nokia
Eagerness to learn multiplied by passion to share.
You can reach me at LinkedIn

You Might Also Like

  • Sk1f3r

    Great article, Roman!

  • http://google.com scorpio

    awesome…

  • Timothy Manito

    Hi Roman, may I ask if you can create a video tutorial for designing such beautiful diagrams? and where did you find the Juniper and Alcatel icons? Thanks!

    • http://hellt.ru/ Roman Dodin

      Hi! Sorry, dont have time nor skills for video tutors. Its plain Visio, and stencils are from http://www.visiocafe.com/vsdfx.htm

      • Liam

        You’re diagrams are fantastic! I can find the Juniper icons, but not the Nokia/Alcatel ones. Are you able to point me in the right direction?

        • http://hellt.ru/ Roman Dodin

          Thanks Liam!
          Unfortunately I don’t know if these ALU icons are proprietary or not. I will try to find out.

  • m237b

    Hi Roman, great article. Couldn’t make it work on CE-PE (ALU) link with eBGP, session simply wouldn’t come up. PE-PE MP-BGP looks fine.
    As well, “as-override” option doesn’t look like there, I’m running it on vSR OS. Thanks

    • http://hellt.ru/ Roman Dodin

      Hi! Can’t say that I had some issues. This setup was tested against vSROS as well (release 12.0.R12) and everything was fine

      • m237b

        mine is 12.0.R6. Is there some special debugging I can enable to see the reason of BGP open failing ? I’ve tried to enable all debugs under BGP plus router ip packet debug, no specific reason is given why no connection is established

        • http://hellt.ru/ Roman Dodin

          debugging under BGP section should be enough… can you post somewhere you configuration on PE and CE?

          • m237b

            Hi Roman, please try to access this URL with configs and debug output: https://drive.google.com/open?id=0B14N4DgPfja4YzNVeV9FREtWNVk

          • http://hellt.ru/ Roman Dodin

            I cant see interface 10.12.12.2 on PE1 which you are using on a point-to-point subnet between CE1-PE1

          • m237b

            My understanding this interface is defined under vprn 20 now, as per your example:
            interface “PE1-CE1” create
            address 10.12.12.2/24
            sap 1/1/1 create

            I do have reachability if I ping from PE to CE or vice versa:

          • http://hellt.ru/ Roman Dodin

            oh, sorry, took a closer look – your problem is that you configuring PE-CE BGP protocol in the wrong section
            You should do it in VRF 20 (not in global BGP conf section)
            So you can remove your BGP group “eBGP-to-CE1” from “configure router bgp” section and put it in “configure service vprn 20 bgp”

          • m237b

            Thank you Roman, CE-PE BGP session is up now. I was following config at section #4 it didn’t mention the reference to vrf in there. Will go and try the rest of the config now

      • m237b

        Hi Roman, one more question on this Lab: is there a need to “redistribute” (or export) on PE prefixes coming from CE towards BGP-VPN ? or it should be done automatically. I can see you do it in opposite direction PE to CE but not vice versa.

        • http://hellt.ru/ Roman Dodin

          Hi! No, routes in VRF’s route-tables will go straight to BGP-VPN protocol. No need to export them manually

          • m237b

            ok, thank you, yes I see the CE prefix at the PE2 on opposite side. I’m trying to use OSPF on PE2 (which is ALU) towards CE. Is there some special consideration should be taken ? As for now OSPF adjacency between PE2 and CE2 is not coming up and stuck in Exch state. Configured OSPF on PE2 under VPRN. On CE2 end I can see OSPF keep alive packets are coming in

          • m237b

            looks like found it…issue was with MTU …had to set on both ends to allow adjacency to come up

          • m237b

            Hi Roman, if you could look at the configs when you have time. Looks like the only thing which is not working yet is exporting from MP-BGP into CE routers on both ends. I followed your example on the export – no luck so far.
            I can see that CE prefixes reaching local PE and further to the second PE on other end, but no redistribution from MP-BGP into CE protocol is happening
            Here is the URL: https://drive.google.com/open?id=0B14N4DgPfja4QVNfX003NFVtX2c

          • http://hellt.ru/ Roman Dodin

            It is because your remote routes on PEs do not have flag “used”.
            The reason for this is because SR2 cant resolve next-hop 10.23.23.3 via MPLS tunnel.
            What you need to do is replace “local-address” on both PEs from link addresses to system addresses.
            For example on SR2 it should be:
            local-address 2.2.2.2 (instead of 10.23.23.2)

  • http://networkop.github.io Michael

    Awesome mate! Exactly what I was looking for. Can I ask you a few questions about the setup?
    1) What’s the difference between a port and a sap?
    2) Why are you setting up an RSVP-TE tunnel between the PE routers? Won’t you have an LSP between the two PE routers simply after enable LDP on all interfaces?
    3) And in general, what are the rules for MPLS label advertisement in SR? i.e. Cisco will advertise for every IGP-known prefix, Junos will advertise only for loopbacks and what is the default behaviour for SR?

    • http://hellt.ru/ Roman Dodin

      Hi!
      1) a sap is a port enabled for serice access. Literally it is a customer facing port.
      2) RSVP-TE could be easily substitued by LDP. I chose RSVP-TE with no particular intention.
      3) SR routers behave like Juniper. When LDP is enabled, only labels for system interface (loopback) will be allocated.