Azure ExpressRoute

Owned by Roger Cruz, created
Last updated: Feb 28, 2022
12 min read

 

Intro

ExpressRoute lets you extend your on-premises networks into the Microsoft cloud over a private connection with the help of a connectivity provider

Documentation

 

Tips and Tidbits

 

  • ExpressRoute is a private and resilient way to connect your on-premises networks to the Microsoft Cloud.

  • You can access many Microsoft cloud services such as Azure and Microsoft 365 from your private data center or your corporate network

  • Connectivity can be from an any-to-any (IP VPN) network, a point-to-point Ethernet network, or a virtual cross-connection through a connectivity provider at a colocation facility.

  • ExpressRoute connections don't go over the public Internet.

    • This allows ExpressRoute connections to offer more reliability, faster speeds, consistent latencies, and higher security than typical connections over the Internet

 

  • Azure private peering

  • Azure compute services, namely virtual machines (IaaS) and cloud services (PaaS), that are deployed within a virtual network can be connected through the private peering domain.

  • The private peering domain is considered to be a trusted extension of your core network into Microsoft Azure.

  • You can set up bi-directional connectivity between your core network and Azure virtual networks (VNets).

  • This peering lets you connect to virtual machines and cloud services directly on their private IP addresses.

 

  • With ExpressRoute Global Reach, you can link ExpressRoute circuits together to make a private network between your on-premises networks.

  • With the addition of ExpressRoute Global Reach, your San Francisco office (10.0.1.0/24) can directly exchange data with your London office (10.0.2.0/24) through the existing ExpressRoute circuits and via Microsoft's global network.

  • To enable ExpressRoute Global Reach between different geopolitical regions, your circuits must be Premium SKU.

 

 

 

  • Forced tunneling in Azure is configured using virtual network custom user-defined routes. Redirecting traffic to an on-premises site is expressed as a Default Route to the Azure VPN gateway.

  • ExpressRoute forced tunneling is not configured via this mechanism, but instead, is enabled by advertising a default route via the ExpressRoute BGP peering sessions.

  • To connect your Azure virtual network and your on-premises network via ExpressRoute, you must create a virtual network gateway first.

    • A virtual network gateway serves two purposes: to exchange IP routes between the networks and to route network traffic.

    • Each virtual network can have only one virtual network gateway per gateway type.

      • For example, you can have one virtual network gateway that uses -GatewayType VPN, and one that uses -GatewayType ExpressRoute.

 

Design and implement Azure ExpressRoute

 

  • Design and implement Azure ExpressRoute

  • ExpressRoute lets you extend your on-premises networks into the Microsoft cloud over a private connection with the help of a connectivity provider.

  • Layer 3 connectivity between your on-premises network and the Microsoft Cloud through a connectivity provider

  • Connectivity can be from an any-to-any (IPVPN) network, a point-to-point Ethernet connection, or through a virtual cross-connection via an Ethernet exchange

  • ExpressRoute gives you a fast and reliable connection to Azure with bandwidths up to 100 Gbps, which makes it excellent for scenarios such as periodic data migration, replication for business continuity, disaster recovery and other high-availability strategies.

 

  • Direct from ExpressRoute sites You can connect directly into the Microsoft's global network at a peering location strategically distributed across the world.

    • ExpressRoute Direct provides dual 100 Gbps or 10-Gbps connectivity, which supports Active/Active connectivity at scale.

    • the ability to connect directly into Microsoft’s global network at peering locations

    • Massive Data Ingestion into services like Storage and Cosmos DB

    • Physical isolation for industries that are regulated and require dedicated and isolated connectivity like: Banking, Government, and Retail

    • Requires 100 Gbps/10 Gbps infrastructure and full management of all layers

 

Route advertisement

  • Route advertisement

  • When Microsoft peering gets configured on your ExpressRoute circuit, the Microsoft Edge routers establish a pair of Border Gateway Protocol (BGP) sessions with your edge routers through your connectivity provider.

    • No routes are advertised to your network.

    • To enable route advertisements to your network, you must associate a route filter.

    • list of BGP community values you want to use in the route filter.

  • BGP community values associated with services accessible through Microsoft peering is available in the ExpressRoute routing requirements page.

 

Bidirectional Forwarding Detection

 

  • You can enable ExpressRoute circuit either by Layer 2 connections or managed Layer 3 connections.

    • In both cases, if there are more than one Layer-2 devices in the ExpressRoute connection path, the responsibility of detecting any link failures in the path lies with the overlying BGP session.

  • BGP keep-alive and hold-time are typically configured as 60 and 180 seconds, respectively.

    • For that reason, when a link failure happens it can take up to three minutes to detect any link failure and switch traffic to alternate connection.

  • BFD provides low-overhead link failure detection in a sub second time interval.

  •  

  • BFD is configured by default under all the newly created ExpressRoute private peering interfaces on the MSEEs.

    • As such, to enable BFD, you only need to configure BFD on both your primary and secondary devices.

    • Configuring BFD is two-step process.

      • You configure the BFD on the interface and then link it to the BGP session.

Configure encryption over ExpressRoute

  • use Azure Virtual WAN to establish an IPsec/IKE VPN connection from your on-premises network to Azure over the private peering of an Azure ExpressRoute circuit.

  • This technique can provide an encrypted transit between the on-premises networks and Azure virtual networks over ExpressRoute

  •  

  • An important aspect of this configuration is routing between the on-premises networks and Azure over both the ExpressRoute and VPN paths.

  • For traffic from on-premises networks to Azure, the Azure prefixes (including the virtual hub and all the spoke virtual networks connected to the hub) are advertised via both the ExpressRoute private peering BGP and the VPN BGP.

    • This results in two network routes (paths) toward Azure from the on-premises networks:

      • One over the IPsec-protected path

      • One directly over ExpressRoute without IPsec protection

    • you must make sure that the Azure routes via on-premises VPN gateway are preferred over the direct ExpressRoute path

  • Traffic from Azure to on-premises networks

    • ensure that the IPsec path is preferred over the direct ExpressRoute path (without IPsec)

      • Advertise more specific prefixes on the VPN BGP session for the VPN-connected network.

        • You can advertise a larger range that encompasses the VPN-connected network over ExpressRoute private peering, then more specific ranges in the VPN BGP session.

      • Advertise disjoint prefixes for VPN and ExpressRoute.

        • If the VPN-connected network ranges are disjoint from other ExpressRoute connected networks, you can advertise the prefixes in the VPN and ExpressRoute BGP sessions, respectively.

  • If you advertise the same prefixes over both ExpressRoute and VPN connections, Azure will use the ExpressRoute path directly without VPN protection.

 

Design redundancy for an ExpressRoute deployment

  • Design redundancy for an ExpressRoute deployment

  • There are 2 ways in which redundancy can be planned for an ExpressRoute deployment.

    • Configure ExpressRoute and site to site coexisting connections

    • Create a zone redundant VNET gateway in Azure Availability zones

  • Only route-based VPN gateway is supported.

  • The ASN of Azure VPN Gateway must be set to 65515.

    • Azure VPN Gateway supports the BGP routing protocol.

    • For ExpressRoute and Azure VPN to work together, you must keep the Autonomous System Number of your Azure VPN gateway at its default value, 65515.

  • The gateway subnet must be /27 or a shorter prefix, (such as /26, /25),

 

  • Create a zone redundant VNet gateway in Azure availability zones

  • You can deploy VPN and ExpressRoute gateways in Azure Availability Zones.

    • This brings resiliency, scalability, and higher availability to virtual network gateways.

    • Deploying gateways in Azure Availability Zones physically and logically separates gateways within a region, while protecting your on-premises network connectivity to Azure from zone-level failures.

  • Zone-redundant and zonal gateways are available as gateway SKUs.

    • You can identify these SKUs by the "AZ" in the SKU name.

    • Zone-redundant gateways and zonal gateways both rely on the Azure public IP resource Standard SKU.

 

Zone-redundant gateways

  • To automatically deploy your virtual network gateways across availability zones, you can use zone-redundant virtual network gateways.

 

 

Zonal gateways

  • To deploy gateways in a specific zone, you can use zonal gateways.

  • When you deploy a zonal gateway, all instances of the gateway are deployed in the same Availability Zone.

  •  




ExpressRoute circuit SKUs

 

  • ExpressRoute circuit SKUs

  • Azure ExpressRoute has three different circuit SKUs: Local, Standard, and Premium.

    • Local SKU - With Local SKU, you are automatically charged with an Unlimited data plan.

      • Billing is based on a monthly fee; all inbound and outbound data transfer is included free of charge

      • Rates: 1 Gbps, 2Gbps, 5Gbps, 10Gbps

      • Provides the cheapest ExpressRoute solution!

    • Standard and Premium SKU - You can select between a Metered or an Unlimited data plan.

      • Rates: 50 Mbps, 100 Mbps, 200 Mbps, 500 Mbps, 1 Gbps, 2 Gbps, 5 Gbps, 10 Gbps

    • All ingress data are free of charge except when using the Global Reach add-on.

 

  • What is ExpressRoute Local?

  • A key feature of Local is that a Local circuit at an ExpressRoute peering location gives you access only to one or two Azure regions in or near the same metro.

    • In contrast, a Standard circuit gives you access to all Azure regions in a geopolitical area and a Premium circuit to all Azure regions globally

  • While you need to pay egress data transfer for your Standard or Premium ExpressRoute circuit, you don't pay egress data transfer separately for your ExpressRoute Local circuit.

    • In other words, the price of ExpressRoute Local includes data transfer fees.

  • ExpressRoute Global Reach is not available on Local

  • ExpressRoute Local is available at the peering locations where one or two Azure regions are close-by.

    • It is not available at a peering location where there is no Azure region in that state or province or country/region.

 

ExpressRoute partners and peering locations

 

  • ExpressRoute partners and peering locations

  • ExpressRoute locations (sometimes referred to as peering locations or meet-me-locations) are co-location facilities where Microsoft Enterprise edge (MSEE) devices are located.

  • ExpressRoute locations are the entry point to Microsoft's network – and are globally distributed, providing customers the opportunity to connect to Microsoft's network around the world.

  • These locations are where ExpressRoute partners and ExpressRoute Direct customers issue cross connections to Microsoft's network.

  • In general, the ExpressRoute location does not need to match the Azure region.

    • For example, a customer can create an ExpressRoute circuit with the resource location East US, in the Seattle Peering location.

 

Configure peering for an ExpressRoute deployment

  • Configure peering for an ExpressRoute deployment

  • An ExpressRoute circuit two peering options associated with it: Azure private, and Microsoft.

    • Each peering is configured identically on a pair of routers (in active-active or load sharing configuration) for high availability.

    • you must make sure that you complete the configuration of each peering one at a time.

    • To configure peering(s), the ExpressRoute circuit must be in a provisioned and enabled state.

  • Each peering requires separate BGP sessions (one pair for each peering type). The BGP session pairs provide a highly available link

 

 

private peering

  • Azure compute services, namely virtual machines (IaaS) and cloud services (PaaS), that are deployed within a virtual network can be connected through the private peering domain.

  • The private peering domain is a trusted extension of your core network into Microsoft Azure.

  • You can set up bi-directional connectivity between your core network and Azure virtual networks (VNets).

  • This peering lets you connect to virtual machines and cloud services directly on their private IP addresses.

  • You can connect more than one virtual network to the private peering domain

 

Configure Microsoft peering

  • Microsoft 365 was created to be accessed securely and reliably via the Internet

  • Connectivity to Microsoft online services (Microsoft 365 and Azure PaaS services) occurs through Microsoft peering.

  • You must connect to Microsoft cloud services only over public IP addresses that are owned by you or your connectivity provider.

 

Create a route filter and a filter rule

 

Connect an ExpressRoute circuit to a virtual network

 

  • Connect an ExpressRoute circuit to a virtual network

  • An ExpressRoute circuit represents a logical connection between your on-premises infrastructure and Microsoft cloud services through a connectivity provider.

    • You can order multiple ExpressRoute circuits.

    • Each circuit can be in the same or different regions and can be connected to your premises through different connectivity providers.

    • ExpressRoute circuits do not map to any physical entities.

    • A circuit is uniquely identified by a standard GUID called as a service key (s-key).

  • Connect a virtual network to an ExpressRoute circuit

    • You must have an active ExpressRoute circuit.

    • Ensure that you have Azure private peering configured for your circuit.

    • Ensure that Azure private peering gets configured and establishes BGP peering between your network and Microsoft for end-to-end connectivity.

    • Ensure that you have a virtual network and a virtual network gateway created and fully provisioned.

      • A virtual network gateway for ExpressRoute uses the GatewayType 'ExpressRoute', not VPN.

    • You can link up to 10 virtual networks to a standard ExpressRoute circuit.

      • All virtual networks must be in the same geopolitical region when using a standard ExpressRoute circuit.

    • A single VNet can be linked to up to 16 ExpressRoute circuits.

    • If you enable the ExpressRoute premium add-on, you can link virtual networks outside of the geopolitical region of the ExpressRoute circuit.

      • The premium add-on will also allow you to connect more than 10 virtual networks to your ExpressRoute circuit depending on the bandwidth chosen.

    • number of address spaces advertised from the local or peered virtual networks needs to be equal to or less than 200.

      • Once the connection has been successfully created, you can add additional address spaces, up to 1,000, to the local or peered virtual networks.

Add a VPN to an ExpressRoute deployment

  • Configuring a secure tunnel over ExpressRoute allows for data exchange with confidentiality, anti-replay, authenticity, and integrity.

  • VPN tunnels over Microsoft peering can be terminated either using VPN gateway or using an appropriate Network Virtual Appliance (NVA).

  • You can exchange routes statically or dynamically over the encrypted tunnels without exposing the route exchange to the underlying Microsoft peering.

    • BGP (different from the BGP session used to create the Microsoft peering) is used to dynamically exchange prefixes over the encrypted tunnels.


Connect geographically dispersed networks with ExpressRoute global reach

  • Connect geographically dispersed networks with ExpressRoute global reach

  • Both branch offices have high-speed connectivity to Azure resources in US West and UK South.

    • However, the branch offices cannot connect and send data directly with one another.

    • In other words, 10.0.1.0/24 can send data to 10.0.3.0/24 and 10.0.4.0/24 network, but NOT to 10.0.2.0/24 network.

 

  • ExpressRoute Global Reach is designed to complement your service provider’s WAN implementation and connect your branch offices across the world.

  • You can enable ExpressRoute Global Reach between the private peering of any two ExpressRoute circuits, if they are in the supported countries/regions.

    • The circuits are required to be created at different peering locations.

  • If the two circuits are in different Azure subscriptions, you need authorization from one Azure subscription.

    • Then you pass in the authorization key when you run the configuration command in the other Azure subscription.

  • Select the ExpressRoute circuit you want to connect this circuit to and enter in a /29 IPv4 for the Global Reach subnet.

  • Azure uses IP addresses in this subnet to establish connectivity between the two ExpressRoute circuits.

  • Do not use the addresses in this subnet in your Azure virtual networks, or in your on-premises network.

 

Improve data path performance between networks with ExpressRoute FastPath

 

  • Improve data path performance between networks with ExpressRoute FastPath

  • ExpressRoute virtual network gateway is designed to exchange network routes and route network traffic.

  • FastPath is designed to improve the data path performance between your on-premises network and your virtual network.

  • When enabled, FastPath sends network traffic directly to virtual machines in the virtual network, bypassing the gateway.

    • FastPath still requires a virtual network gateway to be created to exchange routes between virtual network and on-premises network.

  • To configure FastPath, the virtual network gateway must be either:

    • Ultra-Performance

    • ErGw3AZ

  • VNet Peering: If you have other virtual networks peered with the one that is connected to ExpressRoute, the network traffic from your on-premises network to the other virtual networks (i.e., the so-called "Spoke" VNets) will continue to be sent to the virtual network gateway.

    • The workaround is to connect all the virtual networks to the ExpressRoute circuit directly.

    • To avoid traffic being routed through the VNet gateways, connect all the VNets to ExpressRoute FastPath circuit directly.

  • Basic Load Balancer: If you deploy a Basic internal load balancer in your virtual network or the Azure PaaS service you deploy in your virtual network uses a Basic internal load balancer, the network traffic from your on-premises network to the virtual IPs hosted on the Basic load balancer will be sent to the virtual network gateway.

    • The solution is to upgrade the Basic load balancer to a Standard load balancer.

  • Private Link: If you connect to a private endpoint in your virtual network from your on-premises network, the connection will go through the virtual network gateway.

 

Border gateway protocol

  • Border gateway protocol

  • An on-premises network gateway can exchange routes with an Azure virtual network gateway using the border gateway protocol (BGP).

  • Using BGP with an Azure virtual network gateway is dependent on the type you selected when you created the gateway.

  • If the type you selected were:

    • ExpressRoute: You must use BGP to advertise on-premises routes to the Microsoft Edge router.

      • You cannot create user-defined routes to force traffic to the ExpressRoute virtual network gateway if you deploy a virtual network gateway deployed as type: ExpressRoute.

      • You can use user-defined routes for forcing traffic from the Express Route to, for example, a Network Virtual Appliance.