Multicast at the Edge, Unicast at the Core

We’ve argued before that a Dynamic Media Facility runs 2110 at the edge and MXL at the core. That piece was about where each protocol belongs. This one is about the wire underneath the core, because the moment media leaves multicast and moves onto a point-to-point data plane, the network problem changes shape, and a good deal of what a broadcast engineer knows about provisioning a 2110 plant stops applying.

Start with the difference, because it drives everything that follows. 2110 is multicast. A source publishes essence to a group, receivers join it, and the switch fabric lays one copy on each link that asked for one. You size the plant for the groups, keep the fabric non-blocking, and lean on IGMP and PTP to keep it orderly. The failure you plan for is a subscription gone wrong or a spine that blocks under load.

MXL in the core is the other thing entirely. On a single node there is no network at all: grains are ring buffers in shared memory, and feeding a compositor, an encoder, and an AI reader at once is three readers on one buffer, for free. Across nodes, MXL Fabrics opens a point-to-point bridge, one unicast flow from this node’s domain into the next, over RDMA where the hardware allows and plain TCP where it does not, the same bridge binary either way. Nothing joins a group and nothing gets replicated in the switch. Every cross-node consumer is its own flow.

That trade mostly favors the core. The copies a multicast fabric would replicate you never make on a node, and the ones you do make across nodes land straight in the next node’s memory. What it changes is the hard question. It is no longer whether the multicast fabric is non-blocking. It is which unicast transport you run, and how that transport behaves when the network is not perfect. So we spent time measuring it.

What we measured

#

Two NVIDIA DGX Spark nodes, a single top-of-rack switch between them, ConnectX-7 100 GbE on each side. One 1080p29.97 stream in 10-bit v210, about 1.4 Gb/s on the wire, plus a 48 kHz stereo audio flow. The same MXL Fabrics bridge binary carried both runs. The only thing that changed between them was the libfabric provider: tcp for the first, verbs for the second. Both nodes were slaved to one PTP grandmaster, with each host’s clock disciplined from the NIC’s own hardware timestamp to within about 13 nanoseconds, four orders of magnitude tighter than the smallest latency we report.

One detail about the method matters: we checked every frame, not a sample of them. Each MXL grain’s index is also its nominal timestamp, so we could ask the receiver for every frame in turn and record when it actually arrived. A dropped frame is then a specific missing frame, not a loss rate inferred from a counter.

On a clean network, TCP is fine

#

Give both transports a quiet, properly sized fabric and neither loses a frame. Across 5,324 frames paired by index, three repeats each, there were zero source misses, zero target misses, and zero gaps. Every frame committed on one node arrived on the other. The difference is entirely in latency and CPU.

Both transports are bounded. TCP’s worst frame out of 2,664 was 13.6 ms, verbs’ worst was 2.9, and neither grew a latency tail. The repeats agree to a third of a millisecond for TCP and a twentieth for verbs, so these are stable figures, not lucky single runs. The retransmit line is the tell: even on an idle, lossless fabric, TCP still spends a couple thousand retransmits every five minutes keeping its window honest, where verbs spends thirty-three. That work is the price of the extra latency and the extra CPU, and on one flow per node it buys nothing you can see. Where it bites is flow density, because CPU per flow is what sets the ceiling on how many flows a node can carry, and there verbs at a fifth of the cost wins outright.

So on a network you own and have sized correctly, plain TCP carries uncompressed 1080p with margin. The cost is CPU and latency, not reliability. That is the first and simplest case where TCP stands in for verbs, and for a lot of plants it is the only case that matters.

It depends on how the network is tight

#

Then we stressed the network in three different ways, and no transport wins all three. Which one holds up depends on how the network is tight, not just that it is.

A narrow link favors TCP. Shape the receiver’s egress port down toward the media rate and you find a cliff, not a curve, because v210 carries no error concealment and a frame missing any slice is not a frame. TCP delivered every frame with under two percent of headroom above the media rate. Verbs needed closer to six, three times as much, and below that it delivered nothing at all. The reason is the same mechanism that makes verbs a good citizen: it applies backpressure and throttles its own source rather than overrun the link, and a throttled source never finishes a grain. TCP just overshoots, lets the switch drop a couple hundred packets, retransmits them, and lands every frame. If your constraint is a genuinely narrow or variable link, a constrained uplink or a contribution circuit, TCP’s willingness to be rude is an asset.

A contended link favors verbs. Shaping a port is not the same as competing for one, so next we shaped the port to a realistic 10 Gb/s, put the media at thirteen percent of it, and had three other nodes flood the same egress queue. Everything was perfect until the port went oversubscribed, and the break landed exactly there. At four percent past line rate, TCP delivered zero video while verbs still delivered a fifth of it, the mirror image of the narrow-link result, produced by the same backpressure. Here TCP is the antisocial one: its refusal to back off is what makes the switch drop the packets its neighbours would otherwise have carried. Push the neighbours hard enough to oversubscribe the port on their own, though, and the distinction collapses. Once the wire is full without you, the transport’s manners stop mattering.

A packet flood on the same host is decided, and it is not close. The third case is the one a shared or cloud environment produces most easily, and it is the only one where the choice is pass or fail rather than a tradeoff. We removed the shaper and simply flooded the receiving host with UDP on an unshaped 100 GbE port. The port never passed a quarter of its capacity. The switch dropped nothing, the inter-switch link dropped nothing, the network did not lose a single packet. Under TCP the media died anyway: fifty-three million packets hit the NIC and found nowhere to go, rx_out_of_buffer, and not one frame got through. Under verbs, against the identical flood, every frame arrived. Three thousand out-of-buffer events against fifty-three million, a factor of seventeen thousand on the same wire, with the link three-quarters empty the whole time. What starves the receiver is a neighbour’s packet rate, not their bandwidth, and verbs wins because it places media into registered memory by RDMA and never touches the kernel receive path the flood is drowning. This is the one failure more network cannot fix, because there was never a network shortage to begin with.

Which transport, and when

#

Put together, the practical reading is short: reach for verbs first, and treat TCP as the exception. Scale and reliability both point that way. Verbs moves the same media for about a fifth of the CPU, and since CPU per flow is what caps how many flows a node can carry, that gap is really a scale ceiling: verbs fits more media on a box, which is the number that decides how large a facility the box can run. It is also the more robust choice when the network is not perfect. It survives a contended switch, and it is the only transport that survives a shared host, where a neighbour’s packet flood drowns the kernel receive path TCP rides on and verbs bypasses by construction. Anything built to grow, or meant to run somewhere you do not own end to end, should start there.

TCP earns its place at the small end. On a handful of flows over a network and hosts that are entirely yours, it carries uncompressed 1080p without dropping a frame, and on a genuinely narrow link it is the better tool, since it overshoots and retransmits where verbs throttles itself to nothing. The cost, more CPU and about ten milliseconds instead of one and a half, does not bite when the flow count is low and the whole path is yours. So the split is scale and ownership, not speed: verbs first, for reach and for anything shared, and TCP when the deployment is small and you own all of it.

What this does not settle

#

In the interest of not overselling a bench: this is one NIC family on one platform, one flow per node, a single switch hop, and a synthetic source rather than genlocked cameras. The longest continuous run was forty-five minutes, enough to rule out fast drift and nothing like enough to speak to 24/7. We congested ports and hosts, which is where RFC 8239 says plants actually fail, but we never saturated the switch fabric itself. The mechanism, kernel receive starvation versus an RDMA bypass, should carry to other hardware. The exact thresholds will not. Treat the numbers as the shape of the thing, not as a spec sheet, and size your own fabric before you trust either transport on it.

Measured on two NVIDIA DGX Spark nodes over a single switch hop, ConnectX-7 100 GbE, one 1080p29.97 v210 flow plus 48 kHz stereo, the same MXL Fabrics bridge binary on both runs with only the libfabric provider changed. Delivery is counted frame by frame from the received grain indices, not from the bridge’s counter. Latency is one-way transport delay, producer commit lateness removed, both hosts disciplined to one PTP grandmaster within tens of nanoseconds.