Chapter 14 — Remote Peering

As of June 2013, according to Job Witteman (AMS-IX) 130 of the 630 AMS-IX customers are connecting to the peering fabric via Remote Peering. During this same year (2013), Andreas Sturm (DE-CIX) said that 50% of all new attachments to the DE-CIX peering fabric were made via Remote Peering. Remote peering is no longer a fringe interconnection technique — it is an interconnection method that brings peering within the reach of smaller and new entrant network operators. Remote Peering is breathing new life into the Internet Peering Ecosystem.

In this chapter we will describe the Remote Peering service model, how it is implemented by the Remote Peering Provider, and how and why it is being adopted across the Internet Peering Ecosystem. Since Remote Peering is a hotly debated topic today, we finish up by highlighting the strongest arguments for and against using Remote Peering.

[ Full disclosure: At the time of this publication, the author is materially employed by the International Internet Exchange (IIX), an IXP operator that offers among other things Remote Peering services. ]

The Traditional Internet Peering Model

Peering usually involves selecting Internet Exchange Points and extending one's network into associated colocation facilities using point-to-point circuits. The network operator deploys a router into each colocation facility that hosts a popular Internet Exchange Point (IXP). The network topology and associated deployment characteristics are shown in the Figure 14-1.

Here we see ISP A extending a router presence into three colocation facilities to peer at three IXPs with a variety of networks labeled “B” to “L.”

Deployment in this traditional peering model requires overcoming a few hurdles.

1) Peering deployment is time and labor intensive. The provision of transport and the deployment of equipment into dispersed colocation facilities can take months. An ISP shared that their 2013 IXP network deployment took over 6 months to implement!

The administrative process is onerous. Deploying peering to three sites for example requires contracts with up to three colocation vendors and three transport providers, requires the purchase and deployment of three customized network kits along with the associated support contracts. These activities require substantial time and engineering resources to complete.

2) Peering deployments require capital expenses. Each deployment in the traditional peering model requires a potentially large one-time expense of a networking kit — typically a router and perhaps associated equipment such as load balancers, peering switches, out-of-band access, etc.

3) Peering deployments require ongoing operating expenses. Each deployment traditionally requires the recurring monthly transport expense, colocation expense, equipment maintenance agreement, and associated internal operating overhead.

These hurdles have made it difficult to make the business case for peering. Rather than backing away from peering, some segments of the peering population have embraced the “Remote Peering Model.”

The Remote Peering Model

Remote Peering allows network operators to peer without any additional network equipment. They simply “Remotely Peer” from their existing locations using the services of a “Remote Peering Provider.”

Definition: Remote Peering — (aka ‘tethering’) is peering without one’s own physical presence at the peering point.

Definition: Remote Peering Provider — an entity that sells access to exchange points across their transport infrastructure.

Let’s consider a Remote Peering example that mirrors the traditional peering model we just described. Figure 14-2 shows ISP A purchasing Remote Peering from a Remote Peering Provider that already has established a presence at the desired colocation facilities. Through this provider, ISP A can “virtually” extend its network and peer with networks at each of the IXPs, without deploying any incremental network equipment.

The Remote Peering Provider charges ISP A a fixed monthly price for each Virtual Local Area Network (VLAN) extension from the Internet Exchange Point.

Where allowed, the Remote Peering Provider bundles the IXP peering ports and cross connects to the peering fabric, further reducing the workload and number of contracts ISP A has to sign. Most European IXPs allow port resale today, and the U.S. IXPs are increasingly enabling port resale. If the IXP does not allow peering port resale, the network operator signs a contract with the IXP for the peering port and orders a cross connect from the Remote Peering Provider equipment to the peering fabric.

Note that we have described a one-to-many Remote Peering deployment as shown in the close-up in Figure 14-3. In this form, a single 10 Gbps interface can deliver potentially many 1 Gbps VLANs extended from potentially many IXPs.

Remote peering also takes the form of a one-to-one Remote Peering deployment where a single 10 Gbps peering fabric is extended to a single 10 Gbps customer router port.

The Remote Peering Provider

The Remote Peering Provider deploys networking equipment to each colocation center it intends to service. There are many different technologies that may be deployed that enable the Ethernet peering fabric to be extended across the wide area network but the most common is Multi-Protocol Label Switching (MPLS).

Beyond the networking technology, the Remote Peering Provider needs to manage the variability across colocation centers, transport providers and Internet Exchange Points. This includes handling the paperwork required to connect to or resell connections to each Internet Exchange Point, provisioning and managing a mix of transport vendor services, and facilitating the relationship between the customers and the IXP and potentially even assist with peering strategies.

The Remote Peering Provider is best positioned as a partner that handles everything except the bilateral peering negotiations. The Remote Peering Provider must be neutral in the sense that it doesn’t sell Internet Transit or compete against the customer in a material way; companies generally avoid paying their competitors or reinforcing their competitors’ position in the market.

Customer Benefits of the Remote Peering Model

Simplicity.

There is no additional equipment to purchase and deploy, and there are no contracts with colocation companies to house networking equipment. Instead, a single contract with the Remote Peering Provider replaces potentially many contracts for dedicated point-to-point peering circuits and potentially many colocation contracts. In some cases the Remote Peering Provider can also handle the paperwork for the remote IXP as well.

Speed.

Since the process of peering is simplified, the peering deployment process takes weeks instead of months. Remote Peering allows a network operator to start peering remotely prior to a full colocation-based POP deployment. Once the network operator determines that there is enough traffic and performance improvement to justify a physical presence, he can deploy networking gear into the colocation center and transition to a physical server presence.

These benefits are in addition to the benefits of peering in general:

Performance.

Content companies go to great lengths to improve performance. Amazon is on record as tying 100 ms of latency to as much as 1% of revenue loss. PayPal believes that up to 70% of payments are at risk of being aborted if they take more than 3 seconds. Google’s real-time bidding for ads model means that content companies risk losing their ad revenue if their latency is too high (50–60 for the end-to-end interaction).

If traffic is not peered, it is handled by the commodity Internet Transit service, a service model that intermingles all Internet traffic together into the same shared network. The commodity Internet Transit service works well until times of congestion as shown in Figure 14-4.

Security.

During times of network congestion, whether from spot events, denial of service attacks, or any other reason, the commodity Internet traffic is vulnerable to packet loss and latency along impacted paths. But traffic that is peered directly between content and eyeballs is effectively segregated from commodity Internet Transit traffic, and therefore is unaffected by the congestion side effects of the commodity Internet networks.

As Andreas Sturm (DE-CIX) put it:

Important Traffic is Peered.

— Andreas Sturm (DE-CIX)

This leads to the observation that Internet Peering in general provides a degree of control and enhanced security as well.

Traditional Peering vs. Remote Peering

The most common question about Remote Peering is:

When does it make sense to establish a network (router) presence at an IXP versus Remote Peer at an IXP?

The answer to this question depends on many factors based on the business and network application.

If the network application requires the lowest latency possible and capital costs are not an issue, then deploying both the network and servers topologically close to the eyeballs provides the lowest latency solution. If control over and visibility into the transport layers is important to the business, or if operating all layers of the networks is strategic, then traditional peering makes sense. If there is a requirement to also buy transit on-site or privately peer or with more than a few peers, then a router presence is generally preferred.

If the goal is to peer as widely as possible, then Remote Peering is the most cost effective way to do so. If it is not easy to obtain capital, if the network engineering resources are limited, or if there is no need to have the content physically deployed and served in the region, then Remote Peering cost effectively meets the application requirements. If a regional network presence is required quickly to improve the performance for a customer base, then Remote Peering is the fastest way to improve the end-user experience in a region.

A Case Study for Remote Peering: LinkedIn

A leading social networking site, LinkedIn, compared traditional peering against Remote Peering for a European peering deployment. They already had established a presence in Dublin and so they were comparing traditional and Remote Peering models to get into four well-populated peering points in Frankfurt, London, Amsterdam, and Paris.

With the traditional peering model, they purchase, assemble and deploy their network equipment at a cost of $275K per deployment, for a total of $1.1M for all four sites. They lease colocation space and power in each city at a total cost of $16K per month so they can attach to the exchange point peering fabrics contained within. The transport cost to extend their Dublin presence into all four locations worked out to be $6K per month. These one-time and recurring costs are summarized in Table 14-1.

With Remote Peering, LinkedIn was able to remove the cost of equipment, the cost of colocation and the cost of the point-to-point circuits from the equation. The Remote Peering Provider was able to extend VLANs from each of these exchange points to Dublin for a total cost of $6K/month.

Another large content provider, GoDaddy, followed the same peering analysis and verified that their conclusion was similar. GoDaddy removed over $1M from their network costs by remote peering, and found that the peering break-even point for Remote Peering was 3 Gbps as opposed to 8 Gbps with traditional peering. Remote Peering enabled GoDaddy to improve performance and substantially reduce their costs for Internet traffic exchange.

One will observe however, that in our LinkedIn example, the unit cost for peering works out to be $6/Mbps ($6K/month divided by 1000 Mbps), a price higher than the market price for transit across Europe today ($1–$4/Mbps). Zaid Ali Kahn (Head of Network Architecture and Engineering, LinkedIn) confirmed that their blend of inexpensive commodity transit and remotely peering away strategic traffic has proven to be a sound performance and revenue-enhancement strategy.

These cases illustrate the general peering perspective of content providers:

The #1 priority is to improve the end-user experience.

Other content providers shared that they view their Internet traffic packets as “gold-plated” in the sense that packet drops result in lost revenue far beyond the cost of traffic exchange. This sentiment is reflective of a common thread of comments from the medium to large-scale content providers.

Remote Peering Use Cases

Let’s continue to apply the Remote Peering model by demonstrating how Remote Peering is used in practice across a variety of different Internet companies.

Next Generation ISPs.

Most ISPs and content companies connect to the Internet by simply purchasing Internet Transit. If network connectivity is a strategic part of the business, these networks will blend Internet Transit from different regional providers into a multi-homed interconnection solution as shown in Figure 14-5.

When important regional customer bases are identified, extending the regional network presence with Remote Peering enables regional network optimization. To these companies, control over routing is strategic, so they may distribute nodes into Chicago, Hong Kong and London and utilize Remote Peering to extend their directly reachable networks strategically in each region.

Remote Peering is a way to strategically offload regional traffic without the cost or time investment required for a traditional peering deployment.

Content Providers.

Content Providers leverage interconnection blends to improve the end-user experience and to improve their website conversion rates. They often blend Internet Transit with Content Delivery Network (CDN) services to get their static content deployed as close to the consumers as possible. They may also blend in Internet Peering and Remote Peering to distribute their real-time dynamically-generated content directly to the access network customers.

The choice between Remote Peering and traditional peering depends on the nature of the backend content generation. These content providers focus their attention on optimizing the traffic delivery across their transit, peering and CDN interconnections. Viral videos for example might best live on the CDN while long-tailed less popular content may be delivered via the remote peering and transit paths.

Rapid Turn-up with Transition to Full-POP Deployment.

The lowest latency solution for Content-Heavy network operators is a full Point-of-Presence (POP) deployment, where the network and servers are extended into a region and the content is delivered from that regional POP. Many content providers strategically deploy a full POP system into Internet Regions to service a strong and growing customer base. Full-POP deployments however can be time-intensive and cost-prohibitive. As a result, some content-heavy network operators employ Remote Peering to service these markets until they transition to a full-POP deployment as shown in Figure 14-7.

To illustrate and example of this is, consider a content provider with a growing user base in India as well as across Asia. One solution to this challenge is to Remote Peer this content to India from a Hong Kong, Singapore or Tokyo POP until a local India POP deployment can be justified. This justification is usually based on the loss and latency performance characteristics of the current delivery system (delivering Indian traffic from Hong Kong) against the cost of bringing the content into India. When it makes sense, the Remote Peering Provider can migrate this connectivity to a POP deployment in India.

International ISPs.

Some international ISPs extend a Remote Peering presence to peer traffic away for free in a foreign market, and perhaps purchase inexpensive transit (see Figure 14-8). Across Africa for example, Internet Transit in 2013 is priced around $300/Mbps, and 98% of their traffic is coming from the U.S. and Europe.

An African ISP may therefore decide to deploy Remote Peering into London to pick up cheap transit ($2/Mbps) and peer away a potentially large proportion of its traffic. Given the right traffic volume and transport price points, this may be a clever financial maneuver.

Cable Companies.

A common blend for cable companies in North America includes extending Remote Peering into Los Angeles to pick up Asian Internet routes and into Miami to pick up South American routes and traffic as shown in Figure 14-9.

There may not be enough traffic to justify a full peering deployment but enough for a Remote Peering extension into IXPs with strong regional routes that match their subscriber base.

New Entrant Internet Service Providers.

For new ISPs, Remote Peering provides an easy way to build modern national or international networks. They may employ peering primarily to save money, but their motivation is also to differentiate through customer service, massive scope and scale, high availability, and performance improvements. Here remote peering plays an invaluable role. Since the cost of extending into peering points via Remote Peering is less costly, an ISP may deploy Remote Peering into smaller regional IXPs as shown in Figure 14-10.

We have examined a number of use cases in this section. The optimal blend varies based on the network flows, the traffic volumes, and the network requirements of the applications. The job of the network architect and engineer is to determine how to best blend the interconnection methods to the needs of the network operator.

Protected or Unprotected Transport Options

One implementation decision that Remote Peering customers face is whether they require protected transport or whether they prefer unprotected transport services. There tend to be two camps on this issue.

Protected transport is implemented by deploying across two diverse paths, typically with equipment to automate recovery in the event of an outage. Those that insist on protected transport want their peering infrastructure to be as reliable and robust as their Internet Transit service, and they are willing to pay the premium for it (about double the price of unprotected transport).

Others prefer the lower cost unprotected transport option for their Remote Peering. They view their peering as a local optimization, and if the underlying transport breaks, routing will automatically failover to their Internet Transit service. Many of the network operators in this camp prefer as few active electronics in between them and their peering point. They prefer to know when the underlying transport fails and deal with these temporary failures themselves.

Implications of Remote Peering to the Internet Peering Ecosystem

How does Remote Peering impact the Internet Peering Ecosystem?

1) Remote Peering extends the life of Internet Peering. Many network operators have not attempted Internet peering because the cost, complexity, and time to implement are formidable hurdles. Remote Peering enables these players to ease into the Internet Peering Ecosystem with fewer logistical impediments and a lower Peering Break-Even Point as shown in Figure 14-11.

2) Remote Peering increases the value of the Internet exchange point. As more remote peering participants connect to exchange points, those that are already peering there benefit from the additional traffic and routes available. Without Remote Peering, the IXP reaches its maximum value when there is no more space and power in the designated colocation facilities as shown in Figure 14-12.

3) Remote Peering reduces the complexity of Internet Peering. Remote Peering allows network operators to deal directly with one provider to establish a peering presence at potentially many exchange points (see Figure 14-13). Fewer contracts to sign, fewer circuits to purchase, provision and debug, and fewer colocation agreements all simplify network interconnections.

4) Remote Peering speeds up deployment of peering and enables trials without deploying equipment. If the customer is present at a colocation with Remote Peering Providers, the turn up could be done in a matter of days.

5) Remote Peering provides a single neck to wring but also introduces a single point of failure. If the Remote Peering fails, there is a single point of contact to fix things: the Remote Peering Provider. These eases operations for the customer but it also means that a single Remote Peering Provider failure could take down all associated peering.

6) Remote Peering enables access into small IXPs where the cost is otherwise not justified. Remote Peering lowers the cost of peering and therefore the risk of not recouping the investment by peering away enough traffic for free. Thus, network operators may expand into IXPs that otherwise were not viable.

Collectively these implications demonstrate that Remote Peering represents not only an emerging trend, but also a major shift in the Internet Peering Ecosystem.

Breathing New Life into the Internet Peering Ecosystem

In 2013, commodity Internet Transit prices fluctuate around the $1–$4/Mbps range in well-connected Internet Regions. As always, network operators ask the question:

Does peering make sense anymore?

To answer this question we typically look at the Peering versus Transit graph (see Figure 14-14). The monthly costs of the traditional peering model (colocation, equipment deployment, transport) are collectively not dropping as fast as the price of Internet Transit. This means that there is a narrow range, or “runway” where peering is financially justifiable.

Remote Peering eliminates many of the peering costs, and therefore makes blending peering with transit a more financially attractive interconnection strategy for a larger market of potential peers. As a result, more networks attach to peering points, more traffic is peered away for free, and the value of IXPs and peering in general continues to grow. Without Remote Peering, peering points would continue to diminish in their appeal for all but the largest networks.

The Great Remote Peering Debate

It couldn’t be called a “Great Debate” without ferreting out some dissenting opinions about remote peering, and as usual, it turns out there are strongly held, almost religiously held, beliefs in the peering community about peering methodologies. Over a dinners at recent Internet operations meetings I teased out a few of these contrary views of remote peering.

Here are some of the criticisms expressed about remote peering:

1. Remote Peering is outsourcing (and hiding) some important complexity.

a. From the customer perspective, the MPLS cloud that brings the VLANs to the IXPs is not as easy to monitor and debug as a point-to-point circuit. When things break, there is a lot of active equipment and potentially many parties between the customer and its peers.

Counter-argument: Zaid Ali Kahn (LinkedIn) shared these initial concerns but after deploying remote peering, he is a big Remote Peering proponent:

Our engineering team initially expressed concerns about the reliability of a Layer 2 Virtual Private Network (L2VPN) service, but we have found that Remote Peering solutions using our vendor’s MPLS network are very stable. Remote Peering allows us to be in control of our own destiny and get closer to our eyeballs; with transit we don’t have control of the routes.

— Zaid Ali Kahn, LinkedIn

b. Some network operators do not want to peer with those doing Remote Peering. The added latency and complexity of the remote peering infrastructure tips the scale to make the peer not want to peer with remote peers.

2. Remote Peering can lead to routing inefficiencies. We used to say “peering keeps local traffic local,” but remote peering changes this notion. You could pick up routes in Amsterdam or Frankfurt and find out that some of this peered traffic is traversing oceans twice over layer 2 where that traffic could be peered at a colocation center in common in Ashburn, Virginia!

As a result of Remote Peering:

• Some implicit latency benefits of peering at a local IXP may get worse.
• If everyone remote peers, then there could be quite a tangled mesh of routing anomalies that spans layer 2 and layer 3.

The counter arguments include that there has long been a diversity of local and international peers at the popular IXPs. For example, roughly 75% of all AMS-IX traffic is from outside of Holland, 45% of all LINX traffic is from outside of the U.K. and 55% of all DE-CIX traffic is from outside of Germany. It is up to engineers to mange their own routing efficiently whether at layer 2 or layer 3.

3. BGP Timers will keep Remote Peering sessions alive long after the remote transport has disappeared. Further, when the Remote Peering transport provider network goes down, it takes ALL REMOTE PEERING WITH IT! Individual circuits don’t often all go down all at the same time, and one can engineer across different circuit providers to mitigate individual problems. With traditional peering, the router is informed immediately when the underlying transport disappears. With this more complicated layer 2 VLAN system, all peering traffic could be black holed for minutes before the BGP timers shut down the BGP session.

The counter argument is that engineers can adjust the BGP Keep Alive thresholds that turn down a BGP session. In the event of a L2VPN failure, peers can turn down the session within 30–60 seconds if they so configure.

Another counter argument is that traditional peering across an Ethernet switch also suffers from this same systemic failure — customer routers are not informed when their peer’s router gets disconnected from the fabric. The customer finds out when the BGP timers expire as well.

4. Remote Peering signals that the peer is not committed to a real physical presence. If they are not committed to a physical peering in the region, then there is some doubt whether peering with them leads to a long-term peering benefit.

A counter argument, a pivot actually, is that peering is considered a local optimization. If the remote peer pulls out, then that just means the optimization goes away. One is no worse off than before the peering was established.

5. Remote Peering with route servers may deliver suboptimal routes. Remote peering may make sense, or it may not, but history has taught us that not everyone does the careful analysis of the routing impact before turning up a peering session. Some peers, for example, will peer with Route Servers (a multi-lateral peering technology) and indiscriminately pick up routes. Remote peers that accept all routes from the Route Servers may accept and install these suboptimal paths by default.

The counter argument is that poor routing can be obtained equally at layer 2 or layer 3. It is a network engineering decision as to how to implement and blend interconnection strategies.

Summary

Remote Peering can be thought of as making peering frictionless from afar. With no incremental equipment to deploy and therefore no deployment or colocation expense, the Remote Peering Provider service model makes it relatively easy to justify and to connect to peering fabrics.

Of the 630 AMS-IX attachments, 130 are embracing remote peering, and half of the new DE-CIX attachments are embracing this remote peering trend as well. Whether you blend remote peering into your interconnection strategy or not, it is important to understand what it is and how it works.