At cluster scale, AI performance is set by communication semantics, not raw link speed.
01 THE PROBLEM
AI networking bottleneck is the failure mode where accelerators spend more time waiting on each other than computing.
That is the core shift most software leaders underestimate. In modern AI systems, you do not lose efficiency because a NIC is “slow” in the abstract. You lose efficiency because the communication pattern of distributed training and large-scale inference does not fit the assumptions baked into mainstream network protocols.
Once you move from a single server to a training pod of tens, hundreds, or thousands of GPUs, network behavior stops being an infrastructure detail and becomes part of the execution model. A stalled all-reduce, an incast burst on parameter sync, or head-of-line blocking during mixture-of-experts routing is no longer a minor performance regression. It can drop cluster-wide utilization fast enough to erase millions in accelerator spend over a quarter.
The failure shows up early.
A team can get away with ordinary Ethernet and conventional transport settings at 8 GPUs. At 64 GPUs, cracks appear as intermittent training jitter. At 256 GPUs, the same design starts producing unstable step times, poor scaling efficiency, and hard-to-diagnose timeout behavior. At 1,000+ accelerators, protocol semantics become a first-order architecture decision.
This is why “more bandwidth” is not the answer.
A 400G or 800G fabric does not solve synchronized fan-in, microburst loss, congestion spreading, or strict tail-latency sensitivity during collective communication. AI workloads are dominated by repeated, coordinated communication phases where many peers must make progress together. If one flow slows, the whole step slows.
NVIDIA has made this visible in practice by treating networking as part of the accelerated compute system rather than a separate layer of plumbing. The reason is straightforward: large language model training depends heavily on collectives such as all-reduce, all-gather, and reduce-scatter, and those primitives punish protocols designed for independent, bursty client-server traffic.
This is the exact gap custom AI network protocols are trying to close.
They are not “custom” for branding purposes. They are custom because AI workloads require guarantees and behaviors that conventional TCP/IP stacks and generic Ethernet deployments were not designed to provide: lower transport overhead, congestion control tuned for synchronized collectives, packet spraying or adaptive routing that preserves throughput under hot spots, NIC-level offloads, and memory access semantics closer to a fabric than a classic enterprise network.
The real-world consequence is simple. If your model roadmap assumes linear scaling from 128 GPUs to 1,024 GPUs and your network stack delivers 55–65% effective scaling instead of 80%+, your training calendar slips by weeks, your experiment throughput collapses, and your infrastructure budget is now being spent on waiting.
That is a CTO problem this quarter, not a networking problem next year.
02 WHY IT HAPPENS
The structural reason is that AI traffic is coordinated, loss-sensitive, and synchronization-heavy, while general-purpose protocols were optimized for fairness, survivability, and heterogeneous traffic.
That tradeoff made sense for decades. TCP/IP won because it is robust, routable, interoperable, and good enough for almost every business workload. Web traffic, database replication, service-to-service calls, and batch jobs benefit from a transport stack that handles arbitrary paths, mixed flows, retransmissions, and imperfect conditions without requiring every host to understand the full topology.
Distributed AI flips those assumptions.
Training traffic is not mostly independent request-response. It is many accelerators repeatedly exchanging large tensors under tight synchronization barriers. Throughput matters, but consistency of progress matters more. A single straggler in a ring all-reduce or tree reduction can hold back the entire operation. This makes tail latency and congestion collapse much more expensive than in conventional microservice systems.
The protocol stack matters because overhead accumulates at every layer.
Kernel networking paths add variance. TCP congestion control is conservative for a reason, but its fairness goals are often misaligned with synchronized collectives. Packet loss is survivable in general networking; in AI clusters, loss can trigger retransmissions and latency spikes that amplify across a job. Routing optimized for average-case utilization may still create hotspots that break collective efficiency.
That is why the industry keeps converging on RDMA, transport offloads, fabric-aware routing, and custom collectives libraries.
Meta has written extensively about network-aware infrastructure at scale, including how workload patterns drive fabric design and transport choices across data center systems. The exact implementation differs across products, but the principle is stable: once east-west traffic dominates and synchronization cost grows, the network has to be co-designed with the compute workload.
NVIDIA’s InfiniBand stack is the clearest example of protocol specialization winning on AI training because it couples RDMA, congestion control, adaptive routing, and in-network computing features such as SHARP for collective acceleration. Whether a team buys NVIDIA’s worldview or not, the lesson is hard to ignore: the closer your transport semantics are to the communication pattern of distributed training, the better your scaling efficiency tends to be.
Ethernet vendors are responding by making Ethernet less “ordinary.”
The Ultra Ethernet Consortium exists because hyperscale AI exposed a gap between commodity Ethernet behavior and what large AI clusters need. The push is not just for faster ports. It is for transport semantics and congestion behavior that can compete with purpose-built HPC and AI fabrics.
There is also an economic root cause.
Accelerators got expensive faster than networks got smarter.
A cluster of 1,024 H100-class GPUs can represent capital or reserved spend large enough that a 10–20% utilization swing dwarfs the cost of more sophisticated switching, NICs, or fabric engineering. That changes the optimization function. It is rational to spend heavily on custom network protocols and fabric controls if they recover GPU idle time.
The incentive misalignment inside many companies is that the GPU budget is owned by one leader, the network by another, and the model team by a third. The result is predictable: network choices get treated as procurement decisions, while scaling problems show up later as “training instability” or “framework issues.”
By the time engineers discover the real cause, they are already debugging symptoms one abstraction layer too high.
03 WHAT MOST GET WRONG
The most common mistake is to treat AI networking as a bandwidth procurement problem.
Teams ask the wrong question: “Should we buy 400G or 800G?” The right question is: “What communication pattern will dominate this cluster, and what transport behavior preserves synchronized progress under load?”
That distinction matters because protocol semantics often beat nominal link speed.
A poorly tuned 400G fabric with conventional congestion handling, weak ECMP behavior for elephant flows, and no effective transport offload can underperform a slower but better-matched stack on real distributed training jobs. In practice, teams then misdiagnose the outcome as “PyTorch doesn’t scale,” “NCCL is flaky,” or “our model is too communication-heavy,” when the fabric is the hidden variable.
The second mistake is assuming that RoCE or InfiniBand is a binary ideological choice.
It is not. It is an operational choice with concrete tradeoffs.
InfiniBand typically offers a more integrated path to low-latency, high-performance collectives, with mature tooling and strong results in tightly coupled GPU clusters. RoCE lets teams stay closer to Ethernet ecosystems and multi-vendor strategies, but it demands much more discipline around losslessness, priority flow control, congestion management, and switch behavior. The wrong RoCE deployment is worse than an honest, simpler design because it creates non-deterministic failure modes that only appear under synchronized load.
This pattern is well known in high-performance networking circles: “RDMA over Ethernet” only performs like a specialized fabric if the entire network is engineered to preserve those assumptions.
The third mistake is believing software retries and orchestration can paper over transport flaws.
They cannot.
Google’s SRE book is clear on a general principle that applies here: retries can multiply load and turn partial failures into systemic failures. In AI clusters, retransmissions and framework-level retries are even more dangerous because they interact with synchronized collectives. One delayed participant can stall many healthy ones. “The job eventually recovers” is not a success condition if every recovery event destroys throughput.
The fourth mistake is importing cloud-native intuitions directly into AI cluster design.
Stripe, GitHub, Shopify, and Netflix all operate massive networked systems, but their service architectures are built around graceful degradation, queueing, caching, and isolation between requests. AI training is different. It behaves much closer to HPC than to a horizontally scaled stateless API. If one shard of a web service slows, some users may get a slower response. If one rank in a collective slows, the whole step can wait.
This is exactly why operators who are excellent at Kubernetes, service meshes, and L7 observability often find AI clusters unexpectedly unforgiving. The network is not carrying independent transactions. It is carrying coordinated progress.
A useful analogue comes from Cloudflare’s engineering culture around tail latency and network path performance. Cloudflare has repeatedly shown in its blog that average performance hides the operational truth; tail behavior is what users feel. In AI clusters, that principle is harsher: tail behavior is not just user-visible, it is throughput-destructive. You are only as fast as the slowest participant in the collective.
The fifth mistake is overfitting to benchmark demos.
Vendors love showing near-linear scaling on a narrow set of models, topologies, and collectives. Buyers then discover that their actual workload includes checkpointing traffic, dataset streaming, noisy neighbor jobs, MoE routing, storage contention, and mixed training/inference windows. The benchmark was measuring a fabric in ideal conditions. Production is not ideal.
This is where teams burn quarters.
They buy premium accelerators, stand up an impressive cluster, and then spend 8–12 weeks debugging unexplained utilization loss because no one modeled communication semantics, oversubscription, transport tuning, and failure handling together.
That is not a rare edge case. That is the default outcome when the network is chosen after the GPU.
04 THE FRAMEWORK
The approach that works is to design the network as part of the distributed systems contract, not as a downstream infrastructure layer.
There are six parts to doing this well.
1. Start with the communication pattern, not the hardware SKU
Map the workload before you touch a vendor deck.
You need four inputs:
- Collective intensity: how much of each step is spent in all-reduce, all-gather, reduce-scatter, or all-to-all.
- Parallelism strategy: data parallel, tensor parallel, pipeline parallel, expert parallel, or a combination.
- Target cluster size: 64, 256, 1,024, or more accelerators.
- Failure budget for jitter: how much step-time variance the training system can absorb before throughput collapses.
If your roadmap includes tensor or expert parallelism across racks, your tolerance for network variance is far lower than teams doing mostly data parallel training within a node or single rack.
This is where protocol specialization starts to make economic sense.
A practical operator rule: once training efficiency becomes highly sensitive to cross-host collectives, assume the network is part of the model architecture. Treat it like a dependency, not a utility.
2. Distinguish scale-up from scale-out early
Most teams mash these together. That is a mistake.
Scale-up is communication within a server or tightly coupled node complex: GPU-to-GPU, GPU-to-CPU, or accelerator memory sharing within a box or near-box topology. This is where NVLink, NVSwitch, PCIe Gen5/Gen6, CXL, and UALink-style designs matter. Scale-out is communication across servers, racks, and pods. This is where InfiniBand, RoCE, Ultra Ethernet, adaptive routing, congestion control, and topology choices dominate.The protocol requirements are different.
Scale-up favors memory semantics, coherent access patterns, and extremely low latency over short distances. Scale-out favors predictable throughput, congestion resilience, path diversity, and collective-friendly transport behavior.
Marvell, Astera Labs, and other infrastructure vendors are focusing heavily on this distinction because the next bottleneck is often the boundary between these two domains. If your intra-node fabric is excellent and your inter-node fabric is generic, the whole system still scales poorly.
For CTOs, this means procurement should not separate server architecture from cluster fabric architecture. They are the same performance budget.
3. Optimize for tail step time, not average bandwidth
This is the metric shift that matters most.
Do not ask whether links are saturated on average. Ask whether distributed training step time remains stable at P95 and P99 under realistic load.
A useful benchmark is not “Did we achieve line-rate in a synthetic test?” It is “What scaling efficiency do we achieve when moving from N to 4N accelerators on the actual model family we expect to train?”
NVIDIA and hyperscaler public numbers often frame cluster performance in terms of model throughput and scaling efficiency because that is what operators actually buy. Internal teams should do the same.
For software leaders, one simple threshold is enough to force the right conversations:
- If moving from 128 GPUs to 512 GPUs produces less than roughly 75–80% scaling efficiency on your real training stack, treat the network as the primary suspect until proven otherwise.
That number is a practitioner threshold, not a universal law, but it is directionally useful. Below it, communication and synchronization are usually consuming too much of the added hardware.
Also watch:
- Step time variance at P95/P99
- Retransmission or packet loss rates
- Collective operation latency distribution
- GPU idle or wait time attributable to communication
- Queue buildup on top-of-rack and spine switches during synchronized bursts
Average throughput will hide all of these.
Cloudflare’s broader performance discipline is relevant here: you do not run critical systems on median metrics. AI clusters are one of the clearest examples of why.
4. Choose protocol semantics based on operational reality
There is no free lunch here. There is a menu of tradeoffs.
InfiniBand
- Best fit for tightly coupled large-scale training where performance per GPU-hour matters more than broad ecosystem compatibility.
- Strong RDMA model, mature collective acceleration, and operational patterns proven in HPC and frontier AI training.
- Tradeoff: tighter vendor coupling, narrower talent pool, and less reuse of existing Ethernet operations playbooks.
RoCEv2 on Ethernet
- Best fit for teams that want Ethernet economics, multi-vendor options, and possible convergence with broader data center strategy.
- Can perform very well, especially with careful lossless configuration, QoS classes, ECN tuning, and switch/NIC selection.
- Tradeoff: more fragile if your network engineering discipline is mediocre. Priority flow control misconfigurations and congestion pathologies are expensive.
Emerging Ultra Ethernet approaches
- Best fit for organizations planning multi-year AI infrastructure strategy and willing to bet on standards evolution.
- Attractive because Ethernet familiarity matters to enterprise operators.
- Tradeoff: ecosystem maturity and interoperability will matter more than slideware.
For startups between Series A and C, the practical decision is often simpler than the market makes it sound.
If you are renting large clusters from a cloud or managed GPU provider, your real choice is less about protocol standard and more about what the provider has already optimized. In that case, your job is to benchmark job-level scaling and failure behavior, not to argue theology about fabrics.
If you are building your own cluster above a few hundred accelerators, protocol choice becomes strategic. This is where a CTO should insist on owning the decision directly, because the implications hit roadmap velocity, capex, and vendor lock-in at the same time.
5. Instrument the fabric like an application
This is where most teams are underbuilt.
They monitor link up/down states, interface counters, and maybe some switch telemetry. That is not enough for AI.
You need correlation between training events and network events.
At minimum, collect and join:
- NCCL or equivalent collective timing
- Per-rank communication wait time
- NIC queue depth and retransmits
- ECN marking rates
- PFC pause events if using RoCE
- Switch buffer occupancy during bursts
- Job placement and topology awareness
- Storage traffic overlap with training windows
Without this, every postmortem becomes superstition.
Netflix’s engineering culture is useful here, even though the workload class is different. Netflix repeatedly emphasizes rich, context-linked telemetry because complex distributed failures are otherwise impossible to localize quickly. AI clusters need the same discipline. “GPU utilization dropped” is not an actionable diagnosis.
A stronger pattern is to assign a single owner for end-to-end training path observability: framework, scheduler, and network. If ownership is split across separate silos, fabric issues get rediscovered every quarter in different forms.
This is also where AI infrastructure observability would fit naturally.
6. Design for degraded modes, not perfect runs
Custom protocol stacks improve performance, but they do not eliminate failures. They often make failures more specialized.
The right question is not whether a fabric can deliver ideal scaling in the happy path. The right question is what happens when:
- one switch drains slowly,
- one NIC firmware version behaves differently,
- one rack sees elevated pause frames,
- one collective library release changes traffic patterns,
- one storage sync overlaps with checkpointing,
- one provider silently shifts placement or topology.
Your implementation needs explicit degraded-mode behaviors:
- job admission control based on topology quality,
- isolation for checkpoint and dataset traffic,
- topology-aware placement in the scheduler,
- burn-in tests before jobs enter production pools,
- canary jobs after firmware and driver changes,
- rollback playbooks for transport-level tuning changes.
This is the same reason strong software teams use progressive delivery.
GitHub, Stripe, and Shopify all normalize staged rollouts for software because broad blast radius is expensive. Fabric and transport changes deserve the same caution. A cluster-wide ECN tweak or NIC firmware rollout can do more damage to throughput than a bad application deploy.
A concrete architecture decision pattern
If I were advising a 100-person AI startup training models beyond a single node, I would push them toward a phased protocol strategy:
Phase 1: Under 64 GPUs
- Buy or rent a topology with excellent intra-node interconnect first.
- Avoid overengineering the inter-node network.
- Measure communication overhead from day one.
Phase 2: 64–256 GPUs
- Benchmark real scaling on target models.
- If collective-heavy jobs dominate, move to RDMA-capable networking and instrument every collective path.
- Add topology-aware scheduling and separate storage traffic where possible.
Phase 3: 256–1,024+ GPUs
- Treat the network as product infrastructure.
- Choose fabric based on training economics, not familiarity.
- Invest in dedicated network performance ownership.
- Make transport tuning, firmware management, and collective observability part of release engineering.
This is where org design matters. You likely need one staff-level engineer who can cross the boundary between distributed training systems and network behavior. Most teams do not have this person until after they need them.
That is expensive.
05 STRATEGIC TAKEAWAY
Custom network protocols for AI are not an optimization layer. They are the control plane for GPU economics. If your team applies this lens, you will make different decisions about cluster size, vendor selection, observability, and hiring within one planning cycle. If you do not, the cost appears as slower training calendars, lower experiment throughput, and underutilized accelerators that still hit the invoice at full price. The CTO decision is not whether custom protocols are “interesting.” It is whether the next 6–12 months of model roadmap justify treating transport behavior as part of the system you are building.
06 IMPLEMENTATION ANGLE
Start with one brutal audit: for your top three AI workloads, quantify step time, scaling efficiency, collective latency, and GPU idle time as cluster size grows. If you cannot produce that view within two weeks, your observability is not ready for serious AI scale.
Then force a single architecture review that includes model engineers, platform, and whoever owns the network or cloud relationship. The review should answer three questions only: what communication pattern dominates, where synchronized progress is breaking, and whether current transport semantics match that workload. Everything else is secondary.
If you are still early, buying proven infrastructure beats building protocol expertise from scratch. But once AI infrastructure becomes a differentiator, the team pattern changes. You need one owner who can bridge scheduler behavior, framework collectives, and fabric telemetry. That is often the missing role when scaling stalls. If you are building that capability internally, Amplify can help engineering teams scale the org side of that transition by identifying where specialized systems ownership is missing before the roadmap slips.
