Data Center Interconnect: What It Gets You And What To Test

Data Center Interconnect, or DCI, gets sold on spec sheets full of gigabits per wavelength and latency guarantees. Whether it holds up gets decided by things that you have to oversee like route-miles of glass, shared rights-of-way, and which layers of the link you actually control. 

Fast Facts: What You Need To Know About Data Center Interconnect (DCI)

• A data center interconnect is a stack of fiber, optics, line system, and service. Most operators own only one or two of those layers.
• Light traveling in standard fiber takes about 5 microseconds per kilometer each way. 
• Synchronous replication generally needs round-trip latency in the 5-10 ms range.
• Two separate carriers don’t guarantee two separate paths. You only get real diversity if the fiber runs along different routes and enters the building at different points, and the only way to know is to check the carriers’ route maps.
• Dark fiber pays off at high bandwidth over long commitments.
  

A Data Center Interconnect Is A Stack, And You Probably Own Only Part Of It

Single-mode fiber handles the long runs between sites, while multimode is reserved for short connections. The optics live on top of it. Coherent transceivers now carry up to 1.6 Tb/s on a single wavelength over metro distances, and 800 Gb/s over long-haul routes. L-band roughly doubles that capacity. 

The service layer is what most enterprises actually buy. Inside a colocation facility, cross-connects and direct cloud connections feed the interconnect, MPLS, VPLS, and SDN overlays build logical networks on top of the physical links.

Every 100 Kilometers Of Glass Costs You A Millisecond, Round Trip

Light moves through standard fiber at roughly two-thirds of its speed in a vacuum, about 5 microseconds per kilometer in each direction. Double that for the round trip, and every 100 km of cable route costs about 1 ms before a switch, optic, or storage controller adds its own delay. 

Synchronous replication holds every write until the remote array confirms it, so storage vendors typically cap it at around 50-100 miles or 5-10 ms round trip. Asynchronous replication tolerates 50-100 ms, at the price of losing whatever writes were still in flight when the primary site went down.

A 100-mile route is about 161 km of fiber, which costs roughly 1.6 ms of round-trip time from the glass alone. The rest of a 5 ms budget goes to optics, switching, array processing, and how much write delay your application will tolerate before users notice. Be skeptical of any provider pitch that puts a 12-15 ms route and the words “synchronous replication” in the same paragraph. At those numbers you’re designing an asynchronous recovery plan, and your recovery point objective should say so.

Why Two Carriers Can Still Share One Trench

Carrier diversity and path diversity are separate guarantees. Two circuits running through the same conduit give you no redundancy even when different providers sell them. One cut can take both down.

Ask each carrier for route documentation covering the full path, and confirm the circuits never share a right-of-way. Make sure they reach your building through separate entrances.

Test the link as if you don’t trust the SLA. Active tools such as iPerf3 or Spirent generate traffic to measure what the circuit really delivers, while passive monitors like PRTG or MRTG watch how it behaves under production load.

Private circuits are harder to intercept than internet paths, and current packet-optical gear can encrypt traffic at wire speed while it crosses the optical link. Turn it on for any DCI carrying regulated data.

Dark Fiber Or Lit Service Depends On What You’ll Need In Year Three

Dark fiber gets you bare strands and makes you responsible for everything that lights them, from transceivers to multiplexers. Because the monthly cost stays roughly flat regardless of capacity, moving from 10G-100G means changing the endpoint equipment instead of ordering a new circuit.

If you light your own fiber, the optics choice splits in two. Performance-optimized coherent systems carry the most capacity over the longest distances. Footprint-optimized pluggables fit the coherent optics into a standard router port, which suits metro and edge links where density and power matter more than spectral efficiency.

Routing between sites at Layer 3 keeps a fault at one location from spreading and lets you balance traffic across several circuits. That’s why most enterprise designs default to it.

Stretching Layer 2 across sites makes sense when you need live VM migration or servers at both locations on one subnet. The trade is that a broadcast or spanning tree problem at one site can now hit every connected site.

A capacity upgrade on a data center interconnect can often retire hardware: the 100G coherent optics, and edge routers from the last design cycle. If that gear ends up stacked in a corner of the cage after your next jump in bandwidth,  find out what your used networking equipment is worth.