Jupiter now scales to 13 Petabits per second

Guiding principles

Our nettoil evolution has been directd by a restricted key principles:

• Anyleang, anywhere: Our data caccess nettoils aid efficiency and srecommendedy by allothriveg big-scale jobs to be placed anywhere among 100k+ servers wilean the same nettoil fabric, with high-speed access to insisted storage and aid services. This scale betters application carry outance for inner and outer toilloads and take aways inner fragmentation. 

• Predictable, low tardyncy: We rank reliable carry outance and minimizing tail tardyncy by provisioning prohibitdwidth headroom, upretaining 99.999% nettoil useability, and provivaciously managing congestion thraw finish-structure and fabric cooperation.

• Software-expoundd and systems-centric: Leveraging gentleware-expoundd nettoiling (SDN) for flexibility and agility, we qualify and globassociate liberate dozens of recent features every two weeks atraverse our global nettoil.

• Incremental evolution and vibrant topology: Incremental evolution helps us to rerecent the nettoil granularly (rather than transporting it down wholesale), while vibrant topology helps us to continuously alter to changing toilload insists. The combination of chooseical circuit switching and SDN aids in-place physical fortifys and an ever-evolving, heterogeneous nettoil that aids multiple difficultware generations in a one fabric.

• Traffic engineering and application-centric QoS: Optimizing traffic flows and ensuring Quality of Service helps us tailor the nettoil to each application’s insists.

Integrating atraverse the above principles is the set upation for our toil. The nettoil is the set upation of reliability for all other compute services, from storage to AI. As such, the nettoil must flunk last and flunk least. To aid this set upational responsibility, we rigorously expound and watch every terrible minute¹ atraverse hundreds of clusters and millions of ports atraverse our global nettoil. Our proceed on reliability is such that our in-hoemploy, gentleware-expoundd Jupiter nettoils transfer a factor of 50x more reliability than prior versions of our data caccess nettoils. 

2015 – Jupiter, the first Petabit nettoil 

In a seminal paper, we showed that Jupiter data caccess nettoils scaled to 1.3 Pb/s of aggregate prohibitdwidth by leveraging merchant switch silicon, Clos topologies and Software Defined Nettoiling (SDN). This generation of Jupiter was the culmination of five generations of data caccess nettoils broadened in hoemploy by the Google nettoiling team. At that time, this data rate — in one Google data caccess — was more than the approximated aggregate IP traffic data rate for the global internet. 

2022 – Enabling 6 Petabit per second

In 2022 we proclaimd that our Jupiter nettoils scaled to over 6 Pb/s, with proset up integration of chooseical circuit switching (OCS), wave division multiplexing (WDM), and a highly scalable Orion SDN administerler. These technologies unlocked a range of proceedments, including incremental nettoil originates, betterd carry outance, lessend costs, drop power consumption, vibrant traffic administerment, and seamless fortifys.

2023 – 13 Petabit per second nettoil

We have further betterd Jupiter to aid native 400 Gb/s join speeds in the nettoil core. The fundamental originateing block of Jupiter nettoils (called the aggregation block) now consists of 512 ports of 400 Gb/s of joinivity both to finish structures and to the rest of the data caccess, for an aggregate of 204.8 Tb/s of bihonestional non-blocking prohibitdwidth per block. We aid 64 such blocks for a total bisection prohibitdwidth of 64*204.8 Tb/s = 13.1 Pb/s. This technology has been powering Google’s production data caccesss for over a year, fueling the rapid proceedment of man-made ininestablishigence, machine lgeting, web search, and other data-intensive applications.

2024 and beyond – Extreme nettoiling in the age of AI

While celebrating over two decades of innovation in data caccess nettoiling, we’re already charting the course for the next generation of nettoil infrastructure to aid the age of AI. For example, our teams are busy toiling on nettoiling infrastructure insists for our upcoming A3 Ultra VMs, that feature NVIDIA ConnectX-7 nettoiling,  aids non-blocking 3.2 Tbps per server of GPU-to-GPU traffic over RoCE (RDMA over combined ethernet) and our future recommendings based on NVIDIA GB200 NVL72.

Over the next restricted years, we will transfer meaningful proceeds in nettoil scale and prohibitdwidth, both per-port and nettoil-expansive. We will persist to push the boundaries of finish-structure integration, including the carry and congestion administer stack, and streamline nettoil stages to accomplish even drop tardyncy with firmer tails. Real-time topology engineering, proset uper integration with the compute and storage stacks, and persistd enhancements to structure-based load balancing techniques will further better nettoil reliability and tardyncy. With these innovations, our nettoil will remain a cornerstone for the alterative applications and services that better the lives of our employrs thrawout the world while simultaneously aiding the groundshattering AI capabilities that power both our inner services and Google Cdeafening products.

We are excited to apshow on these disputes and opportunities to see what the next 25 years hgreater for Google nettoiling!

Further resources

• Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacaccess Nettoil, SIGCOMM ‘15 [paper]

• Journey of the first Jupiter datacaccess nettoil leveraging merchant switch silicon, Clos topologies and Software Defined Nettoiling (SDN).

• First deployed in production in 2012.

• Mission Apollo: Landing Optical Circuit Switching at Datacaccess Scale, arxiv.org, 2022 [paper]

• Orion: Google’s Software-Defined Nettoiling Control Plane. NSDI ‘21 [paper]

• Google’s high-carry outance, scalable, intent-based allotd SDN platestablish employd in both datacaccess and expansive area nettoils.

• First deployed in production in 2016.

• Jupiter Evolving: Transestablishing Google’s Datacaccess Nettoil via Optical Circuit Switches and Software-Defined Nettoiling, SIGCOMM ’22 [paper]

• Enabling technologies: OCS (2013), Orion SDN (2016), 200Gbps nettoiling (2020), honest-join topology (2017), vibrant traffic engineering (2018), vibrant topology engineering (2021).

• Swift: Delay is Simple and Effective for Congestion Control in the Datacaccess, SIGCOMM ‘20 [paper]

• Swift, a congestion administer protocol using difficultware timestamps and AIMD administer with a procrastinate aim, transfers excellent carry outance in Google datacaccesss with low flow completion times for stupidinutive RPCs and high thrawput for extfinished RPCs.

• First deployed in production in 2017

• PLB: Congestion Signals are Simple and Effective for Nettoil Load Balancing, SIGCOMM ‘22 [paper]

• Protective Load Balancing (PLB) is a modest, effective structure-based load balancing summarize that lessens nettoil congestion and betters carry outance by randomly changing paths for congested joinions, pickring to repath after idle periods to lessen packet reordering.

• First deployed in production in 2020

^{1. Any minute where a statisticassociate meaningful number of nettoil flows in the data caccess nettoil experience a total or fragmentary outage above a expoundd threshgreater.}