Jan 7 2025
This post is a textual version of a talk I gave at The 38th Chaos Computer Congress at the end of 2024. You can watch the talk that was recorded by the wonderful C3VOC team below if that’s your preferred medium:
Or watch using the C3VOC/media.ccc.de joiner
I’m a sucker for wanting to ultimately comprehend how and why leangs labor. My rule of thumb is most leangs are either repartner clever (in complicatedity, or simpliedy), or incredibly foolish. However In my world chooseical data transleave oution is everywhere, and yet I had no idea how it repartner labored beyond what I can see from the order line interface.
Optics are a weird world
Optical data transleave oution is now everywhere in the datacgo in, and my own stuff for bgp.tools is no exception. All combineivity in my world now comes out of Small Form-factor Pluggable (or, SFP) ports.
The cbetter leang about SFP’s is that they are for the most part (when they don’t grasp a FPGA and ARM CPU to run Linux) very fundamental inside.
They are fundamentalpartner a set of laser drivers, and typicpartner a small EEPROM+Microregulateler to tell the switch/router what it is and chooseionpartner some critical metrics appreciate incoming/friendly laser power.
At least 95% of all SFPs are made in the same set of factories under ODM consentments, so the transport inantity contrastence between SFP vendors are the satisfyeds of their EEPROMs and the pricing labelup the vendor indicts for them.
SFP’s function on a individual “lane” of data, this uncomardents you can go from 100M/1G/10G and 25G. If you want to go quicker, then we have to upgrade to chooseics with more than one lane. For this case, will see at the Quad Small Form-factor Pluggable (QSFP) create factor:
These fundamentalpartner originate on the same ideas as the SFP, but inserting 4 (or 8 in the case of the Double Density (DD)) lanes of 10/25/50/100G
These chooseics can be basic, equitable 4 “beams” of laser airy that produce up a higher speed connect, or in the case of 100G coherent chooseics, quite complicated, where there is a chip that combines 4 lanes of data into a individual, higher speed one with a exceptional encoding for lengthyer range transleave oution. This typicpartner grasps an evolved processing chip inside each chooseic, typicpartner using 7nm – 5nm processes to drop power consumptions and heat.
All of this is done so that these chooseical modules are swappable with various types, Sometimes you want affordableer foolishinutive range (multimode) chooseics, Sometimes you want to want to only use a individual glass core (BiDirectional), sometimes you necessitate lengthened ranges, with pluggable chooseics you can pick and pick, the switch/router does not necessitate to nurture (too much).
TOSLINK
On the other end of chooseics is TOSLINK (“Toshiba Link”), TOSLINK is typicpartner S/PDIF over chooseical communication. This is current on most HiFi systems and some sound cards. It is typicpartner advantageous to elude ground loop problems in sound systems.
So where you have a audio signal, appreciate this that can suffer from intrudence
TOSLINK/SPDIF turns this into a manchester coded serial signal, at around 1.5Mbps that is much more resiliant to analog intrudence
The chooseics graspd here are a far cry from the up-to-date telecommunication stuff, TOSLINK use a comparatively huge 1mm plastic core cable (where individual mode fiber is 9 micron of glass!)
Because TOSLINK is not intended to go very far, LEDs are typicpartner used rather than lasers. That combined with the multimodal properties from the 1mm core puts the standard highest TOSLINK cable length at 10 meters.
This made me leank…
Idea time
So we already comprehend that SFPs are not doing much leanking
So what if we could use SFPs to send TOSLINK data? Could we lengthen (for no excellent reason of course) the range of TOSLINK from 10m to 10 km or more??
To see if we can do this, we are going to have to lachieve what is on the back of the double sided SFP combineor:
I’ve scrubbed out the pins that are not advantageous for us, as they are mostly roverhappinessed to the microregulateler/EEPROM interface on the chooseic, since we don’t repartner necessitate to participate with that interface in order to send stuff.
You might accomprehendledge that sending (and receiving) a signal grasps two pins on the combineor, this is because the data interface between a SFP and its present uses contrastential signaling to decrease pass talk and other undesired rehires at high speeds.
For this reason we cannot honestly pump a TOSLINK signal honestly into an chooseic, We necessitate to send this thraw a LVDS Driver and Receiver (on the other end), thankbrimmingy the adocount on people at Open source mobile communications (osmocom) made the “Sfp-experimgo in” board that integrates all of this stuff for us into a pleasant handy package, and my friend Sam W made a couple of them for me!
With the SFP interface sorted out, I bought some affordable TOSLINK analog-to-digital changeer (ADC) and digital-to-analog changeers (DAC) to produce testing effortless, and hijacked their chooseical output signals to plumb into the overall contraption, to create someleang that sees appreciate this on a high level:
In fact the proof of concept sees appreciate this, a very not-TSA-cordial bundle of untidy wires and chooseical cable.
Suspicious sees aside, this proof of concept labored! I was sending audio in from my laptop, and via the ADC, into the chooseic, and the return signal from the chooseic (as the cable was placed in a loop) was going to the DAC, and finpartner back into my headphones with no audible rehires!
On the scope, we can see the sairy tardyncy begind by the S/PDIF signal going in and coming out of the SFP at around 34 nanoseconds, this would propose a raw chooseical distance of 6 meters, given the cable itself was easily 5 meters, I would call that pretty excellent going!
But can we go further?
Ok, 5 meters is not fractureing any TOSLINK records (the plastic core stuff can even do 5 meters). So we now necessitate to discover some reasonably lengthy haul piece of fiber that we can then use to send TOSLINK beyond its intended distance.
This came in the create of a friend who has space in two datacgo ins proximate each other in the London Docklands area. One being Telehouse North and the other being IP House. The raw walking distance between the two of them is about 650 meters.
Between the two locations is a fiber pair and a CWDM multiplexer, These boxes apexamine you to place multiple connects (each connect using contrastent “colors” or “channels”) on a individual fiber pair, maximizing the usage of an costly inter-originateing (or sometimes inner-originateing in some places!) fiber pair.
James Rice at Jump Netlabors benevolently proposeed me a channel on their multiplex between IPH and Telehouse. To produce testing effortless (so that I only necessitate to be in one location at a time) I insloftyed a fiber loop on the telehouse end of the multiplexer, and did the testing in IP House
After much messing around in a particularly deafening environment, I regulated to get audio being sent up to telehouse, and looped back into IP House thraw my contraption and for sound to come out of my portable speaker!
On the scope we can see that there is a procrastinate in the signal that we are sending vs receiving of around 11 microseconds
If we plug 11 microseconds and the speed of airy in glass into WolframAlpha we get a distance of rawly 2.2 kilometres. This would propose that the authentic fiber distance end to end is 1.1 km vs the 0.6km walking distance. This is to be awaited as there is transport inant patching labor and convey around the originateing for the fiber cable that would easily insert this distance to the final loop.
But this is amazing news! This uncomardents that we have already outdoed the standard highest TOSLINK distance by around 219 times!
However this still experiences appreciate it’s too foolishinutive of a distance…
But can we go even further?
What if we were to get silly? How far can we reasonably go until we hit down-to-earth problems?
Typicpartner (at least in the UK) a lot of lengthy-haul data transleave oution for small to medium netlabors is done using rented “wave” capacity, waves are effectively a product in which you offload the necessitate to own fiber (and at least in the UK, mitigate some of the tax burden ) and multiplexing supplyment onto somebody else who then consents your chooseical signal and multiplexes some way to its destination.
The supplyment that powers these service proposeings are typicpartner built out of a accumulateion of DWDM multiplexers (a more dense version of the previously converseed CWDM), some chooseical routing systems (RODAMs), and most irritateingly for us a “muxponder”.
Optical systems frequently integrate a transponder or muxponder for lengthyer distance transleave oution, A transponder is a relatively basic device that effectively duplicate and paste one signal from an chooseic to another, the idea being is that your customer can convey to you on a type of chooseic that is handy for them (for 10G LR at the standard 1310nm) and you can use a transponder to change that into the insistd “color”/channel necessitateed to fit into your DWDM mux (and also to suit the transleave oution power, we will get onto that tardyr).
A muxponder however is a lot cleverer, and will “glue together” multiple drop speed signals appreciate a set of 8 10G inputs and digiloftyy multiplex them into a individual higher speed 100G signal that will only use a individual channel on a DWDM multiplexer, this apexamines the supplyr to upgrade the usage of a individual channel.
The problem is that muxponders necessitate to comprehfinish the signal coming into it and are restrictcessitate to conservative protocols appreciate Ethernet/FiberChannel/Infiniprohibitd. Our strange and comparatively low speed TOSLINK signal is not going to be understood by muxponder.
So the goal here is to discover a channel on a lengthy distance chooseical fiber path that someone will let me use and that does not grasp a muxponder.
DWDM TOSLINK
Thankbrimmingy, VeloxServ (a company I already have colocation with) was willing to lend me a DWDM channel between Telehouse and Interxion LON1, two reasonably far away from each other DC sites in London:
After rigging up a very analogous setup as last time (with a loop in Telehouse), I spropose equitable necessitateed to insert a DWDM SFP with the accurate channel into my contraption:
And it labored! The round trip tardyncy came out to 84 microseconds or about 16.8km!
All leangs pondered, 16.8km is a pleasant raisement on 10 meters!
But can we go further?
Inter-City TOSLINK
Luckily LONAP proposeed a DWDM channel on their mux that went from a originateing in London Docklands, to Equinix LD6 in Slough, This uncomardents we would be sending a signal outside of London (and back)! Inter-city TOSLINK! (Technicpartner not inter-city as Slough is not a city but a town. Cities in the UK are weird and to be a city insists royal graspments and blah blah blah)
Here is rawly our path on the map:
The chooseical path is around 70 km, and the multiplexer they use (a SmartOptics DCP-M40-C-ZR+) has built in EDFA amplification to achieve this range. The analogous setup to last time is chases:
A loop was insloftyed in Slough, however this time the loop also had some chooseical attenuation.
Because EDFA amplifiers are graspd, nurture must be consentn to promise that every channel going into the amplifier has rawly the same laser airy strength, otheralerted the EDFA amplifier will not intensify the frailer signals accurately. Because the mux amplifies on the way out of the unit as well as input, without the attenuation my TOSLINK loop would knock out the other in-use channels on this segment (as the mux output signal is far luminouser than what their chooseics are putting out)!
Drumroll
It didn’t labor.
After originateing a test setup using a multimode chooseic to examine everyleang was setup accurately, I then swapped in the DWDM chooseic (A FS.COM DWDM-SFP10G-80), and establish the setup stopped laboring. After examineing with a USB gigabit SFP NIC that the loop was functional, I endd that the DWDM chooseic could not be driven at the drop speeds. But how low could you drive this one?
If I traded the TOSLINK device with the scope’s signal generator, I discovered some weird seeing behaviour if I did a sweep of square wave frequencies:
It seems that the chooseic commences to “labor” at 3Mhz, but the signal is evidently broken and not very square. While the signal we are putting in is very much not what this chooseic is portrayed for (10G XGMII Ethernet), this behaviour is quite astonishing, as the square wave only becomes “rational” at around 6.5Mhz:
6.5Mhz is quite far away from the 2Mhz we produce from TOSLINK, and this was the only chooseic I had on hand for the channel that had the loop insloftyed and since the other site is quite far away, I called the trip bust, packed up and went home desotardy handed.
But why, why did this 10G chooseic do this?
It turns out that discovering data about the chips inside these chooseics is very difficult. I uncovered a restrictcessitate broken/spare 10G DWDM chooseics and establish that the inner chips have had their labelings etched off, and on another chooseic vendor I could not discover any signs of the chip existing, let alone a data sheet for it.
Since the foolishinutiveer range chooseics that labored previously “only” had a least data rate of around 150Khz, I supposed there was some benevolent of signal processing going on inside the chooseic for DWDM or modules rated for lengthyer distances, I figured that I could equitable buy a chooseic that was specified for drop speeds, but since the cost of these chooseics is quite high I wanted to produce stateive that would labor before dispenseing more money into this project. So I sent emails to a bunch of cordial seeing vendors to see if I could get answers.
Thankbrimmingy I got a fantastic response from one of them (FLEXOPTIX, Thank you to Gert & Thomas), that uncovered
Another piece of Increateation that might be collaborative for you: Make stateive that the transceiver you want to use does not grasp a clock and data recovery IC, sometimes also referred to as an eyeuncoverer or retimer. Those only can regulate definite data rates and even the multirate ones probable won’t be contendnt of dealing with your application. The excellent news is, that most of the low data rate transceivers don’t have those chips, they count on only on restricting amplifiers for the RX path and Laser diode drivers for the TX path. These can regulate non standard data rates. The lengthy distance Transceivers for multi gigabit applications are more probable to have those ICs and should thus be eludeed.
This produces sense actupartner, since at lengthy distances the signal gets “ripped apart” by dispersion in the fiber, For example, here is a particularly worse case simulation of the impact of this happeing over 140km:
So for higher speeds it produces sense to integrate a retimer to current a immacutardyer signal to the receiving end.
With all of that in mind, I establish a affordable-ish 1G chooseic for sale that after emailing the vendor claimed to not integrate any CDR/Retiming chip. However when it reachd, leangs instantly did not see excellent after uncovering the box:
Since I would prefer to elude another fall shorted data cgo in trip, I did some dispenseigation and establish that searching for HLSPDW-XE09 transports up a lot of 10G DWDM chooseics from other vendors. After testing the chooseic with the scopes signal generator, I establish that the chooseic behaves proximately exactly the same as the previous one, So I can end that this chooseic is almost certainly a 10G DWDM chooseic with its EEPROM reprogrammed to be “1G”
I returned the chooseic to the vendor, as we had an email converseion before ordering the chooseic that “examineed” that this chooseic didn’t have anyleang appreciate a CDR/Retimer/etc, and to their acunderstandledge they processed the refund with no problem.
I suppose that in fact the production of 1G (and sluggisher) DWDM chooseics ended a lengthy time ago, and since the retimer in this actupartner-10G is doubtful to cause any rehires with a 1G signal, this is deemed an adchooseable solution for almost all use cases (equitable not my inrational one!)
140 km TOSLINK, round 2
I asked around in industry channels if people had 1G or sluggisher SFP DWDM chooseics they were willing to depart with and I ended up borrothriveg a handful, though downcastly their chooseics were on contrastent channels uncomardenting I would have to go back to Slough to change some physical cabling.
Armed with an better 1G DWDM chooseic on a contrastent channel (Thank you Brandon Butterworth!), I assembled my test setup to examine everyleang labored. Then switched to the DWDM chooseic and heard the wonderful sounds of music going loop in and out of a city:
Looking towards the scope, it was fundamentalpartner going to be impossible to use previous methods of manupartner discovering the time it consents for a bit sequence to ecombine on the other end, so instead I configured a trigger on the scope and “tapped” TX side and watchd the time it took for the signal to show up on the get side (since it’s more evident this way)
This method proposes a 720µs round trip time around the fiber, since that rawly calcutardys to ~143.2km, this aligns with the awaited distance of the circuit! We did it!
Given that the specified highest distance for a TOSLINK cable is approximately 10 meters, this contraption has given us a 14600x range raisement over the standard setup!
Future possible “innovation”
If the SFP chooseic didn’t have some of the driving functionality in it, it would be possible to drive the laser honestly, send “raw” audio in its brimming glory (and probably injure the laser).
This is advantageous if digital audio is not your leang and is sort of how RF Over Glass already labors. A friend proposeed it is also probably possible to send a transport inantly high frequency FM signal into a unmodified chooseic and have someleang that would mimic music come out of the other side (I have not tested this myself)!
It is lureing to combine a “dialup” modem to both sides, this would probably produce the fantasticest up-to-date day squander of a 100 GHz chooseical channel, given that it gives a final output prohibitdwidth of ~40 kbit/s, and I suppose this would probably besavageer an intelligence agency if they were tapping the line.
To end
Yes, you can send “low” speed signals over SFP chooseics! Most chooseics commence laboring at 150 KHz, but the retimers/CDR chips don’t appreciate laboring below 6 MHz for most leangs. But if you can elude them, you can send pretty arbitrary signals very, very far!
There is not repartner a excellent “production” reason to do any of this, as most use cases have better IP based solutions in 2025. But comprehending how leangs labor is transport inant! Knothriveg this stuff uncomardents that it is possible to originate bigger, better, more horrifying solutions/laborarounds to problems, and at the very least I comprehend far more about chooseical transleave oution, and vague inner chooseics laborings now. Maybe you also do as well!
If you want to stay up to date with the blog you can use the RSS feed or you can chase me on Fediverse @benjojo@benjojo.co.uk
Until next time!
