DPDrPeering.net
Buy the Book
Chapter 4 Table of Contents

Chapter 4 — Internet Peering

Three photos of Internet peering fabrics: Equinix, CoreSite Any2, and the Toronto Internet Exchange
Examples of peering fabrics from Equinix (above), CoreSite’s Any2 exchange (right) and the Toronto Internet Exchange (below). Photos courtesy of Equinix and Matt Peterson.

In this chapter we introduce the other dominant form of interconnection, called Internet Peering. Internet Peering is a local routing optimization, a way to exchange some of your traffic with another party, with neither party incurring Internet Transit fees. We describe the process of peering, introduce (a lot of) terminology, and enumerate the top five motivations ISPs give for peering. The chapter closes with a brief discussion of Paid Peering.

Introduction

Some people in the peering community see peering as like dating, finding a suitable partner where each party derives about equal value.

Internet Peering

One of the most common discussions within the peering coordinator community is

“Internet Transit is so inexpensive, why do we need anything else?”

The answer is related to transit traffic volume.

As stated earlier, as the price of Internet Transit has dropped an average of 30% per year, the average volume of Internet traffic has historically grown by 40–50% per year. This growth leads to the growth of the transit fees paid by and paid to the upstream ISPs.

For example, while the price for Internet Transit drops from $3/Mbps down to $2/Mbps, the volume of traffic during the same period might grow from 6Gbps to 10Gbps. The net result of these simultaneous forces is a 25% increase in monthly transit fees, from $16,000 per month to $20,000 per month. The percentages vary, but every year the Internet packet transmission market could grow as shown in Figure 4-1.

Transit prices drop while volume increases
Figure 4-1. The unit cost of Internet bandwidth drops, but the volume increases. The result is a monthly Internet Transit bill that continues to rise.

So even though the unit cost of transit drops, the monthly transit bill increases. Decreasing the absolute cost of Internet Transit is one of the principal drivers for pursuing an Internet Peering strategy.

Definition: Internet Peering is the business relationship whereby two companies reciprocally provide access to each other’s customers.

Internet Peering is typically settlement-free, meaning that neither party pays the other for access to each other’s customers, reflective of the underlying notion that peering is a relationship of approximately equal value to each party. Since both parties benefit about the same from the relationship, there is no need to bother with the overhead of measurement and settlement.

There is also no standard way to calculate and monitor the absolute value derived from a peering relationship. Is the value of the peering relationship proportional to the volume of traffic freely peered bi-directionally? Or is it proportional to the desirability or uniqueness of the routes? Or is the value the number of people reached?

For these and other reasons, the dominant form of peering is settlement-free. When you see the term “peering” from this point on, it means settlement-free peering.

To illustrate Internet Peering, consider the mini-Internet Peering Ecosystem shown in Figure 4-2 with only three ISPs: WestNet, MidNet, and EastNet.

Internet Peering diagram with WestNet, MidNet, EastNet
Figure 4-2. Internet Peering provides reciprocal access to each other’s customers.

Using graphical peering notation, we see that:

After these two peering sessions are established, the routing tables are in place (as graphically shown as colored circles in the “routing table” beneath the ISP clouds). This diagram shows that MidNet peers with both EastNet and WestNet, and therefore MidNet customers can reach both EastNet and WestNet customers.

It is important to observe that the routing announcements go in the opposite direction as the traffic to that destination. The Internet routing scheme separates the control plane from the data plane.

For convenience, we will use the simplified and equivalent notation shown in Figure 4-3 to indicate a settlement-free peering relationship between two parties.

Equivalent peering notations
Figure 4-3. Equivalent graphical notations for an Internet Peering relationship between ISP A and ISP B.

These notations concisely convey that the two parties (ISP A and ISP B) are in a peering relationship.

Three Key Points about Internet Peering

Three key points often get lost when one is first introduced to Internet Peering. It is worth reading the following points a few times:

The Top Five Motivations to Peer

Discussions with the peering coordinators highlighted several dominant motivations for Internet Peering:

  1. Transit costs are reduced. Internet Transit is often a large component of the cost of operating an Internet service. Peering provides a more direct traffic path between the parties while simultaneously reducing the load on these expensive transit services (as shown in Figure 4-4). If the cost of exchanging traffic in peering relationships is less than the cost of sending that traffic through a transit service, then peering can be proven to be a financially rational decision.
    Internet Peering bypasses metered Internet Transit
    Figure 4-4. Internet Peering bypasses metered Internet Transit.

    We will work through a detailed mathematical calculation of this reduction of transit costs in “The Business Case for Peering” chapter.

  2. End-user experience is better. By interconnecting directly with peers, the ISP’s customers experience lower latency to the other entity’s customers. Transit usually provides a more circuitous path than peering — a path through potentially many networks. (It is not uncommon to find more than 30 router hops from eyeballs to content.)
  3. Control over routing is strategic. The performance of applications like gaming and video require special attention because of the adverse affect poor performance has on the application. For example, the Hollywood studios that I spoke with said that the “suspension of disbelief” is destroyed with just one or two “artifacts” such as pixilation, audio garbling, or freeze frames. For some of these companies, monitoring and managing the network traffic is too important to casually select the cheapest way to deliver video to the Internet.
  4. Traffic billing is usage-based. Most ISPs charge customers based upon how much traffic they send or receive. Since packet loss and latency severely restrict traffic consumption, these ISPs strive for the lowest-latency, lowest-packet-loss Internet Transit service for their customers. It is in their best interest to ensure that customers use as much bandwidth as possible. Minimizing loss and latency through effective traffic engineering helps them make more money.
  5. ISPs enjoy marketing benefits. ISPs market their extensive peering capabilities in their marketing literature. They market their well-peered backbone, their shorter paths, lower latency, etc. Wouldn’t you buy from an ISP that cares enough about performance to peer its traffic on a large scale?

Now let’s talk about the process of peering.

The Internet Peering Process

Internet Peering has essentially three phases:

We will examine each peering negotiation phase in turn. This section speaks from the perspective of an ISP exploring whether peering makes sense. Most ISPs go through these processes, and later on we will see that they apply to content providers as well.

Phase 1 — Identification of Peering Target

The first process is identifying the set of potential peering targets. ISPs determine where their traffic is going to and coming from. They are looking for peering partners that should be similarly incentivized to directly interconnect their network, to save money, to improve performance, etc., as shown in Figure 4-5.

Traffic analysis to identify potential peers
Figure 4-5. Traffic analysis helps identify potential peers.

For example, in the figure, ISP A would look at its traffic and determine that a large volume of traffic goes to or comes from ISP B. Since both ISP A and ISP B are paying transit fees for that traffic exchange, it might be in both parties’ interest to “peer” with each other.

This traffic analysis is done with NetFlow or other traffic management software such as PeakFlow provided by Arbor Networks. The result of this analysis is a top 50 list of networks sorted by traffic volume. When sorted, a typical transit traffic histogram looks something like the graph shown in Figure 4-6.

Traffic distribution follows a power law
Figure 4-6. Traffic distribution follows a power law.

The bad news is that the top traffic sources and sinks are typically the larger ISPs that do not peer openly. These top destinations represent a large chunk of traffic that is ultimately not peerable; if you are a typical small or medium-sized ISP, these ISPs simply will not peer with you (we will discuss why later). The remainder of the traffic, however, is potentially “peerable.”

Once the top traffic destinations are identified and associated with specific ISPs, these ISPs are targeted for potential peering relationship discussions. Figure 4-7 shows a sample “Peering Top 50 list” template — peering coordinators use some form of this template to help them systematically track their interactions and progress with new potential peers.

Worksheet for prioritizing peering discussions
Figure 4-7. Worksheet for prioritizing and tracking peering discussions.

These target peers are ordered by traffic volume and sometimes color-coded to reflect the likelihood of obtaining peering.

Phase 2 — Contact Qualification and Peering Negotiation

ISPs typically have a person or group specifically tasked with peering and traffic engineering issues. For example, Verizon has a “Peering Steering Committee” to evaluate peering requests, a structure now common in many of the larger ISPs.

Interviews I have had with ISPs have highlighted a key challenge: Finding the right person to speak with at the target ISP is more difficult than you might expect; in fact, it is a time-intensive process.

Peering coordinators change jobs, and there is no single standard way to find out who handles this task. Even if the name is known, peering coordinators are often traveling, way behind in e-mail, and prioritizing e-mail based on the subject or the sender. In this area, “people networking” helps a great deal, and hiring experts for their contacts speeds this initial contact process up significantly.

Next we will enumerate the most effective ways to put target ISPs in context.

The top 10 ways seasoned peering coordinators contact target ISPs follow:

  1. Face-to-face at informal meetings in an “open” Internet Operations forums like NANOG, IETF, RIPE, APNIC, AFNOG, etc. (Here “open” means anyone can attend.)
  2. Face-to-face at “selective” commercially sponsored peering forums like the Global Peering Forum (Here “selective” means there are some prerequisites to attend, like being a customer of a GPF IXP sponsor.)
  3. Face-to-face at “selective” IXP Member Meetings like DE-CIX, LINX, or AMS-IX member meetings
  4. Via introductions made through an IXP Chief Technical Liaison or a peer who knows the right contact
  5. Via electronic mail, using the pseudo standard peering@ispdomain.net or a personal contact
  6. From contacts listed on an exchange point participant list, or peeringdb registrations
  7. With tech-c or admin-c from DNS or ASN registries
  8. Through a search for “peering contact $ASN peering”
  9. From the target ISP sales force, at a trade show or as part of the sales process
  10. From the target ISP NOC

A short aside about e-mailing peering requests:

Peering request e-mails are among the easiest e-mails to ignore. Many ISPs pass peering e-mail duties around on a weekly basis, with many engineers viewing the peering activity as “not real engineering.” Some sneeringly view peering work as secretarial in nature. Therefore, e-mail messages that are poorly written or missing information are much less likely to receive a reply. It is worth your time to write the peering request carefully and completely.

Peering requests need to include the relevant information. Peering request e-mail messages that fail to include the necessary information will delay or prevent peering from occurring. Including too much information is also a problem; the request may be perceived as too burdensome. Figure 4-8 illustrates the point with a real-world example of a poorly worded peering request.

Poorly worded peering request example
Figure 4-8. Example of a poorly worded peering request.

A Better Peering Request

The template shown in Figure 4-9 is based on a sample peering template from Serge Radovcic (from Euro-IX at the time):

“I put this together a few years ago. I did this by first looking through several peering coordinators’ e-mails that they had passed on to me, and after putting together a first draft, I sent it back to three or four coordinators and I ended up with the draft I sent you.”
Template for a better peering request
Figure 4-9. A template for a better peering request.

Serge goes on to say, “My original idea was based on a conversation that we had a few years ago where we would ask some peering coordinators from different countries to translate this letter and we would then automate the languages to what the Requestor and Requestee needed.”

The full template is available at: http://scripts.drpeering.net/pr3.html

Peering and Interconnection Contracts and Negotiations

Next, if a discussion is scheduled, mutual nondisclosure agreements (NDAs) may be negotiated and signed, and a discussion of Peering Policy and prerequisites then follows.

Note that NDAs are an optional step, and many ISPs do not require signed NDAs prior to discussions.

Traffic engineering discussions and data disclosure may be needed to justify the peering relationship. Each ISP typically has a set of requirements for peering. During these discussions both sides explore the peering prerequisites such as peering at some number of geographically distributed locations or peering at specific public exchange points.

Traffic volume is often a key determinant. As a rule, the decision hinges upon whether or not there is sufficient value from peering to justify spending time and money. A Bilateral Peering Agreement (BLPA) is the legal form that details each party’s understanding of acceptable behavior, and it legally defines the arms-length (loosely coupled) relationship that each side agrees to.

In some cases the effect on the business is also examined. Might this “peer” be a customer? How will peering impact the network?

After this initial discussion, either party may decide to walk away from the peering discussions until certain criteria are met. If both parties agree that their requirements are sufficiently met, they discuss interconnect methods as described next.

Phase 3 — Implementation of Peering Session

Regional interconnection has two dominant forms: The Direct-Circuit Peering model and the Internet Exchange Point peering model.

The Direct-Circuit Peering Model

Definition: Direct-Circuit Peering is peering using a point-to-point circuit.

This peering model doesn’t require additional Points of Presence (POPs), nor does it require the deployment of additional hardware; the two peers simply purchase a circuit between their existing POPs. If there is a failure in the interconnection, only three parties are involved: the two ISPs and the circuit provider.

The Direct-Circuit Peering model has two dominant uses:

First, Metro-area Direct-Circuit Peering is used between two parties that seek interconnection with only a few other parties within the metro area. If more than one interconnect region is required, the two parties often split the interconnect costs by alternating who pays for each additional regional interconnection (Figure 4-10).

Metro-area Direct-Circuit Peering
Figure 4-10. Metro-area Direct-Circuit Peering.

The second place you will see the Direct-Circuit Peering model is with “transoceanic half-circuits,” interconnections between the large incumbent ISPs that seek interconnection without the foreign peer having a local presence in their home markets. This interconnection model is sometimes called “half-circuits” because each party pays for half of the cost.

The Direct-Circuit Peering cost model scales linearly; the more interconnections one has, the more it costs to interconnect. Every additional peer incurs the cost of an additional point-to-point metro circuit.

The Internet Exchange Point Peering Model

The alternative to the Direct-Circuit Peering model is to peer at an Internet Exchange Point.

Definition: An Internet Exchange Point (IXP) is a place where multiple ISPs interconnect their networks together (Figure 4-11).

The IXP model
Figure 4-11. The IXP model.

Potentially many peering sessions can be established across a single well-populated IXP peering fabric.

The IXP Peering Cost Model

The cost of peering at an IXP usually involves the following cost components (Figure 4-12):

The costs of the IXP model
Figure 4-12. The costs of the IXP model.

Definition: Transport Fees refer to the monthly recurring expenses associated with a physical/data link layer media interconnection into a peering location.

Unlike transit service, transport is not metered; it is sold as a fixed capacity circuit that costs the same regardless of the amount of traffic exchanged over it.

Definition: A Colocation Facility refers to a specialized data center that houses telecommunications equipment for multiple network operators.

Definition: Colocation Fees are the expenses paid to the colocation facility operator for housing the telecommunications equipment and facilitating interconnections.

Not only do colocation facilities provide the operations environment necessary for the equipment, but the better ones also make it easy and cost-effective for their population to interconnect with each other. They understand their customers’ businesses and seek to establish and grow a community of participants. A handful of these colocation centers provide much more than space, power, and cross-connects. They facilitate peering.

Definition: Equipment Fees are the amortized costs of the networking equipment used for Internet Peering.

(In modeling the Internet exchange later on, we will calculate the total cost of peering at an exchange, including equipment, and make assumptions about pricing, depreciating the deployed router gear.)

Definition: Peering Port Fees are the monthly recurring costs associated with peering across a shared peering fabric.

Together, these fees are the monthly cost of peering. These fees are typically the same monthly recurring cost regardless of the amount of traffic that is exchanged over the infrastructure.

For the cost of the interconnection, both parties can then send and receive as much traffic as can fit across the transport circuit and peering fabric.

Public and Private Peering

The interconnections at the IXP take one of two forms: Private Peering or Public Peering.

Definition: Private Peering is peering across a dedicated layer 2 circuit between exactly two parties, typically using a fiber cross-connect or a VLAN between two parties at an IXP.

Private Peering is the same as the Direct-Circuit Peering model but within a building or set of interconnected buildings. There is typically a nominal cost to Private Peering (a few hundred dollars per month for a fiber cross-connect, for example), whereas circuits are typically more expensive (a few thousand dollars per month for a 10G circuit, for example).

Definition: Public Peering is peering across a shared fabric such as an Ethernet switch.

Public Peering is the dominant method of peering in the peering ecosystems we studied, although many support both Public and Private Peering.

A basic and important question emerges:

Question: When does it make sense to peer?

We will make the business case for peering in the next chapter, but to make this chapter independently complete we will answer this question in brief.

Answer: From a strictly financial view…

Peering makes sense when it is cheaper to send traffic to peers than through a transit provider.

A Paid Peering Aside

Some companies offer a derivative service called Paid Peering.

Definition: A Paid Peering relationship is a peering relationship with an exchange of compensation from one party to the other (Figure 4-13).

Paid Peering notation
Figure 4-13. The graphical notation for Paid Peering.

The compensation may take the metered form of $/Mbps. In other cases, one side might cover more of the peering costs than the other.

When does peering become Paid Peering?

My litmus test: If the peering is not a settlement-free and no-strings-attached peering relationship, then it is Paid Peering.

Peering Workshop Practice Questions

Here are a few practice questions from the Internet Peering Workshop:

  1. “I am ISP A. If I peer with B and C, won’t B and C start sending their traffic to each other through my network?” [answer]
  2. After delivering a speech, “Why Telstra won’t peer,” I was approached by an Australian policy maker who suggested that peering sounded a lot like bartering; that things of similar value (routes) are exchanged without the exchange of money. He pointed out that “Barter is something we tax here in Australia.” How is Paid Peering like bartering? How is Paid Peering unlike bartering? [answer]
  3. Draw the network diagram using the peering and transit notation for the following scenario: ISP A and ISP B purchase transit from ISP C, which peers with ISP D and ISP E. ISP D sells transit to ISP X and ISP E sells transit to ISP Y. [answer]
  4. In #3, would ISP C likely be interested in peering with ISP A? [answer]
  5. Should Paid Peering be priced the same as Internet Transit? What is the case for its being priced cheaper than transit, and what is the case for pricing it higher than the price of transit? [answer]

View answer key →

← Section II — Connecting to the Core Chapter 5 — The Business Case for Peering →