During yesterday's LibreQoSAPNIC webinar, I posed the question: won't more bandwidth solve these problems? As Herbert Wolverson said, yes, more bandwidth is good, but, still these problems remain if queueing is done incorrectly. Here's my take on why bandwidth alone is not as good as bandwidth + good queueing policies:
Generally it was believed that 'data is important, so don't throw it away; hang on to it and send it later'. In practice, this has proven to be suboptimal as two issues may emerge: 1) latency increases for some flows due to heavy demand from other flows using the same bottleneck link, 2) even a single flow can have excessive latency due to aspects of typical TCP behavior (referred to bufferbloat) when the buffers grew large enough that the data being buffered was retransmitted anyway.
It turns out that not all data is equally important. Active Queue Management is the art of deciding priorities, both in deciding what data to throw away, but also in allowing some later arriving data to be transmitted ahead of data in another connection that arrived before it.
These problems have been studied, and good solutions have been found by using certain queueing policies in routers and switches, referred to as “fq_codel’ and “cake”. These track the different flows not by classifying the data, but by watching the behavior. Flows that send relatively little data (DNS lookups), or at a measured pace (video chat) have priority over flows that send a lot of data as quickly as possible (App and System updates, Video and ISO downloads).
Curious to your thoughts about using packet drops as a transport layer control loop vs ECN CE bits. Wi-Fi can have PER unrelated to queue drops but the transport won't know the difference. It seems like ECN could be very helpful too.
Very interesting presentation! A middle box that tracks all the flows, including TCP timestamps, is quite useful to see what’s happening end to end, from the user devices and user WiFi through your network and to the other side.
Great APNIC webinar on LibreQoS by Herbert Wolverson just completed. Watch for the recording, which they say will be out in a couple of weeks.
Or don't wait, dive in and start using it!
LibreQoS is a Quality of Experience (QoE) platform that improves ISP Customers’ experiences by reducing the latency for interactive applications. By using state of the art Flow Queueing (FQ) and Active Queue Management (AQM) algorithms in a middle box upstream of an existing network, no changes to existing Routers, Switches and Access points are required.
LibreQos also helps network operations and planning by measuring end-to-end TCP round trip time for each Subscriber, Access Point, and Site on a network and saving the relevant data for later analysis.
LibreQoS runs on a server acting as a managed bridge between an ISP’s edge router and the core of the network. Servers running LibreQoS can shape traffic for many thousands of customers. LibreQoS code is Open Source, with paid support packages and subscriptions to additional analysis service features available.
Internet access bandwidth has increased continually, somewhat like Moore’s Law, and this will likely continue. Bandwidth demand also increases to fill the supply. Internet data flows are not smooth, steady progression of packets; many have bursts of activity followed by periods of inactivity. Consequently, a bottleneck may exist in the packet’s path across the Internet.
Generally it was believed that 'data is important, so don't throw it away; hang on to it and send it later'. In practice, this has proven to be suboptimal as two issues may emerge: 1) latency increases for some flows due to heavy demand from other flows using the same bottleneck link, 2) even a single flow can have excessive latency due to aspects of typical TCP behavior (referred to bufferbloat) when the buffers grew large enough that the data being buffered was retransmitted anyway.
It turns out that not all data is equally important. Active Queue Management is the art of deciding priorities, both in deciding what data to throw away, but also in allowing some later arriving data to be transmitted ahead of data in another connection that arrived before it.
These problems have been studied, and good solutions have been found by using certain queueing policies in routers and switches, referred to as “fq_codel’ and “cake”. These track the different flows not by classifying the data, but by watching the behavior. Flows that send relatively little data (DNS lookups), or at a measured pace (video chat) have priority over flows that send a lot of data as quickly as possible (App and System updates, Video and ISO downloads).
These algorithms have been integrated into Linux and deployed in some systems, but still, many routers, switches, RF access points, and base stations do not yet have these improved queueing policies. By deploying a LibreQoS system ISPs can improve their network’s performance at very little cost.
@dtaht:matrix.org - Truly speeding up the Net, one smart ISP at a time
LibreQos v1.4-rc9 is out. ( https://lnkd.in/e_Q5kSJz ) We can look at this two ways. Either v1.4 is months and months late (and it is!), or several of the features we planned for v1.5 are months and months early (and they are, including some new analytical methods nobody has ever seen before)! We have dozens of happy ISPs using it now, and over 160k known shaped devices using the CAKE algorithms (via our opt-in stats). We believe rc9 will be the last step towards formally releasing v1.4. Please give it a shot. Please tell your bufferbloated ISP about it, also. The network you save may be your own. The best way to get on it quickly is to join our chatroom, but we do hope that the documentation has improved to the point where someone can get the basics running over a weekend. Feedback wanted!
It would be nice if the industries we are making more performant, like gaming, and videoconferencing, and also some bigger ISPs and data center providers chipped in with a few donations, or more enlightened orgs if any existed, that could grasp what consistently lower latency meant for the internet aws a whole. A huge thanks to all our existing donors! and we are deeply grateful to Equinix for their donation of hardware for our testbed, but we could do so much more (and faster) if this was not a spare time activity for those involved.
We could also use some more volunteers with a QA background, others with UI and of course, anyone with python or rust or C experience is welcomed into the main project, but I have to warn: Fixing the internet with LibreQos has been an inspiring fight for so many, but so far, comes with mainly karmic rewards (and improved personal networks - the chat room is a wonderful shared pool of industry experience that I love being in!) Honestly at this point though I think our biggest need is to find more people with the skills and connections to move our efforts to the next level and help us get the world to 10s of millions more well-shaped devices over the next years.
A great opportunity for a smart ISP to get even smarter! LibreQoS will very likely improve your customer's experience by reducing the latency for the more interactive traffic, and it will improve your experience by helping you understand what's happening on your network.
@dtaht:matrix.org - Truly speeding up the Net, one smart ISP at a time
Is there anyone out there in Asia that would like to experiment with LibreQos at scale at their ISP or IXP, but is lacking on budget? We (LibreQos) are very interested in learning more about how wireless (wimax?), cable, and fiber networks actually are configured and function around the world, and how the workloads differ.
https://lnkd.in/ds33cmpE
Congratulations Rhett Jones, Tom O., Gio Bacareza, Ben Matranga!
Rhett -- you've made great progress toward your vision for the Internet in The Phillipines. Keep it up! GetaFIX has made a remarkable effect on the wholesale Internet.
PS: for extra credit: who knows where the name Getafix came from?
Heads up - we’re looking for a badass CFO to help build a company at the intersection of electric mobility and emerging markets. You’ll help us build our financial strategy and raise capital from the ground up.
Can’t say too much about what we’re building yet, but it involves batteries, vehicles, and the world’s toughest road commuter environments. We’re led by engineers and executives that have built and scaled the world’s best selling electric vehicles, and are backed by the some of the world’s largest and most ambitious climate tech funds.
We’re looking for someone who speaks manufacturing and supply chain finance in their sleep, designs new financial products for climate infra deployment in their spare time, and can raise enough debt to fund a small government with a few phone calls. Bonus points if your friends are tired of hearing you talk about clean energy in emerging markets.
Sounds like you? Hit me up and we'd love to share more.
ISP Strategic Leadership, Engineering, and Operations
As an organization scales, there is often a need to separate strategic technology leadership under the CTO from internal execution under engineering and operations management. This allows the CTO to focus on outward-facing duties while enabling streamlined operational efficiency.
However, it's crucial to recognize the differing motivations and needs of these distinct technical roles. On one hand, engineers thrive on tackling interesting, challenging projects that push their abilities and knowledge. They value working alongside capable peers, genuine recognition of technical accomplishments, and contributing to products/services that reach the market.
In contrast, operations teams should strive for repeatable processes, continual improvement, and driving efficiency at scale. Their priorities revolve around standardizing workflows, minimizing disruptions, and incrementally optimizing existing systems and procedures to maintain consistent service delivery as the company grows.
An effective organizational structure acknowledges this dichotomy, fostering an environment that fuels engineering innovation while providing operations with the stability and process-orientation they require. Striking this balance enables the company to stay agile and innovative through its engineering efforts, while simultaneously scaling operational execution in a reliable and repeatable manner.
Competitive compensation remains an important motivator across both domains. By properly aligning structural elements, motivational factors, and leadership priorities, a rapidly growing ISP can position itself to attract and retain top talent across engineering and operations, driving sustainable growth and success in the dynamic technology landscape.
--
2wCurious to your thoughts about using packet drops as a transport layer control loop vs ECN CE bits. Wi-Fi can have PER unrelated to queue drops but the transport won't know the difference. It seems like ECN could be very helpful too.