Chapter 10 – The 21st Century Internet Peering Ecosystem
In this chapter we will show how the basic U.S. Internet Peering Ecosystem has morphed into the 21st century Internet Peering Ecosystem of today.
The following six evolutions demonstrate how the players react to the changes in the environment. We will first highlight the evolutions that occurred during the economic downturn in the year 2001.
This chapter walks through six Evolutions in sequence:
Evolution #1 – The Cable Companies Peer with Each Other
Up until 2001, a company called @Home provided essentially all Internet access services to the cable companies. But in 2001, @Home unexpectedly went bankrupt. Suddenly, the cable companies were forced to in-source broadband Internet services. The cable companies were given only 30 days’ notice to establish high-capacity transit relationships and launch their own Internet services. They entered the ecosystem as independent broadband access network providers as shown graphically in Figure 10-1.
Figure 10-1. The Cable Company model.
The cable company player model is similar to that for Tier 2 ISPs but is a little different in that the cable companies are regional and don’t tend to compete in overlapping areas, and they tend to pull enormous volumes of traffic.
When the cable companies attached to their new upstream ISPs, each cable company experienced the same thing: Internet Transit circuits were congested almost immediately. Each of the cable companies ordered upgrades to these circuits, and almost immediately the Internet Transit connections were again congested!
What was going on here?
After some analysis, the cable companies found that about 40% of all cable company traffic was peer-to-peer (file sharing) traffic. Importantly, this was traffic that was ultimately destined for other cable company networks. Peer-to-peer traffic is different than other kinds of traffic, in that it will expand to consume all available bandwidth. At that time, I had been working with the cable companies on the benefits of peering, and as a result, many of them built into multiple regional IXPs and started peering openly with each other and with the other Tier 2 ISPs in the ecosystem.
These new players and their peering relationships are shown graphically in Figure 10-2.
Figure 10-2. The cable companies emerge as independent players.
One might ask, why is there a separate species of player; isn’t the cable company model identical to the Tier 2 ISP model? Why is this evolution significant?
This evolution represented a significant change in the U.S. Internet Peering Ecosystem because:
The volume of traffic being exchanged directly in a peering relationship, among the cable companies and between the cable companies and the Tier 2 ISPs was huge – second only to the Tier 1 ISPs’ traffic exchange volume estimates at the time. Therefore, the amount of traffic that bypassed the Tier 1 ISPs, and therefore the amount of money that peering saved the cable companies, could be measured in millions of dollars.
In 2003, the cable companies had an Open Peering Policy. The cable companies were not generally interested in selling transit to ISPs or Content Providers, and they generally worked in non-overlapping regions, so they didn’t compete against each other. As a result, there was no reason not to peer openly.
Peer-to-peer traffic consumed all available bandwidth. The typical ISP sinusoidal demand curve was replaced with a flatter demand curve as peer-to-peer file-sharing software now preferred the higher-bandwidth, lower-latency paths to peered cable company customers.
It was also noted that the peer-to-peer traffic volume grew immediately after the cable companies peered with each other. Peering caused the file-sharing service Kazaa to prefer to fetch files across the now peered network path. The Kazaa selection protocol at the time used latency to determine which Kazaa file sharer was “more local,” and automatically selected that file sharer. The result was that when cable companies peered with each other, there was an immediate 20% growth in Kazaa-originated peering traffic volume.
Evolution #2 – Large Scale Network Savvy Content Providers Peer
Around the same time, a small group of Large-Scale Network-Savvy Content Providers emerged as large volume peers.
Definition: A Large-Scale Network-Savvy Content Provider (LSNSCP) is a content provider that sees networking as strategic enough to build a backbone and peer.
Most of the LSNSCPs peer at least bi-coastally, and some peer more broadly across the country. The dominant three motivations to peer are similar to the Tier 2 ISP motivations to peer:
To improve the end-user experience. Yahoo! for example sees the end-user experience as mission-critical. The company applies considerable resources monitoring (in real-time!) its application network performance characteristics. It identifies congestion, notes the network path, and alters its interconnections so that traffic traverses an alternative network path to improve the end-user experience. Around the world, this motivation is the #1 motivation that leads Content Providers to peer.
To reduce transit costs. Any traffic that can be sent directly to the access-heavy (also called eyeball) network is traffic that doesn’t have to go over the metered transit connection. In a related way, e-mail service providers such as MSN HotMail and Yahoo! can exchange e-mail without incurring any transit charges.
They needed to move to a new data center anyway. Many colocation and data centers were going out of business. Given the opportunity, some content players chose a data center where they could peer as well as buy transit in an open market.
Large-scale content players have always purchased transit to deliver their content to the end users on the Internet. Therefore, their role and behavior in the Peering Ecosystem is similar to the Tier 2 ISPs with the exceptions that Content Players:
do not sell transit
focus on content creation, and if they do operate a network it is for exactly one customer: themselves
have visibility into the end-to-end performance characteristics (Tier 2 ISPs see packets, while LSNSCPs can see packets and flows)
The model for the Network-Savvy Large-Scale Content Provider is shown in Figure 10-3.
Figure 10-3. The Large-Scale Network-Savvy Content Provider model.
These players still purchase transit (generally from the Tier 1 ISPs), but they supplement it by peering openly with anyone and everyone.
Examples of these Large-Scale Content Players are Yahoo!, Google, Microsoft, Amazon, Walmart.com, Apple, Electronic Arts, and Sony Online.
This evolution represents a significant change to the Peering Ecosystem because:
The volume of traffic being diverted to peering was huge, and it represented traffic that previously was sent through and available only from the Tier 1 ISPs.
The LSNSCPs had an Open Peering Policy, and even actively promoted peering with them so this large amount of traffic would be freely available to peers.
These leading players paved the way for other Large Scale Content Companies and Enterprise companies to examine Peering as a cost reducing / performance improving strategy.
We can see a graphic depiction of this evolution below in Figure 10-4.
Figure 10-4. The Large-Scale Network-Savvy Content Providers peer openly.
Evolution #3 - Large Scale Network Savvy Content Providers Peer with Cable Companies
The peering of the Network Savvy Large Scale Content Companies with the Cable Companies put the most popular content on the Internet directly onto the same network as the broadband eyeballs. The end result was a major disruption in the Internet Ecosystem, resulting in network performance improvements and significant cost savings from peering!
And the middle of the Internet Peering Ecosystem got a little bit fatter as shown in Figure 10-5.
Figure 10-5 Content peers with eyeballs. The Tier 2 ISPs and the cable companies peer with the content companies that entered the peering ecosystem.
This represents a significant dynamic shift in the peering ecosystem since the Tier 1 ISPs are being cut out of a high volume traffic exchange loop. The hierarchy of the Basic Internet Peering Ecosystem has evolved into a flatter mesh, with the Tier 1 transit providers required only as the route of last resort. Traffic destined to locations too far away or too expensive to reach will ultimately need the services of an international transit provider. At this stage, the Tier 1 transit providers have lost their grip on the U.S. Internet Peering Ecosystem.
Evolution #4 – CDNs Dominate Traffic Volume
We have learned that Internet traffic volume is a key determinant as to whether peering makes sense financially. Video represents a proportionately larger amount of traffic when compared against e-mail, web browsing, and other traditional request-response messages. Amplifying the importance of video traffic is the fact that not only is it a larger amount of traffic, but it is also an increasingly popular type of traffic across the Internet. As it turns out, Content Delivery Networks (CDNs) are really good at distributing video to the edge for offloading to the last mile consumers.
Definition: A Content Delivery Network (CDN) is an entity that distributes web objects and network services as close as possible to the access network customers.
The CDN player (Figure 10-6) is similar to the Tier 2 ISP but instead of handling a steady stream of content, they distribute any “web objects” to caches at the edge. Today the service is priced the same as Internet Transit, so for modeling purposes we will make the simplifying assumption that the CDN is essentially providing transit - just a much better performing one due to the caching at their edge.
Figure 10-6. The Content Delivery Network model.
Evolution #5 – The Video Internet Is Activated
It may have been YouTube that activated the video revolution around 2006, but there was an entire underlying Video Internet Ecosystem that conspired to enable it starting back as far back as 2002.
Consider one slice of Internet Video category is User-Generated Content. To enable content creation side, small easy-to-use video cameras came onto the market. Home video editing software (like iMovie) was inexpensive or bundled with the sale of new computers. Next in the supply chain, video distribution became free and easy thanks to YouTube, a service that leveraged the inexpensive transit and CDN services. These services leveraged volume pricing with ISPs that interconnected with last mile providers that upgraded the last mile infrastructure (to DOCSIS-3 for example). All of this would not have mattered if it wasn’t for the high-speed wireless plug-and-play networks deployed in every home today. Innovations along each link of the supply chain were required for end-to-end video distribution to work well. Once these services matured, Internet backbones quickly became dominated by video.
Cisco estimates that by 2013, about 80% of all Internet traffic will be video, leading me to coin the phrase, the “Video Internet.” Friends in Japan tell me the Internet in Japan is already 80% video, while others in the U.S. say conservatively 40%-50% of all U.S. traffic was video in 2011. While the term “Video Internet” is still gaining acceptance, I believe it more accurately reflects the dominant traffic type across the Internet today.
It was the innovations across the entire supply chain that activated the Video Internet, a system that is now positioned to service the video distribution needs of the planet.
