About 12 years ago I went to my first of (admittedly few) Linux User Groups Meetings in Silicon Valley.

I was new to the Valley, and a bit bright-eyed (that would fade FAST). For a new transplant, I was looking forward to visiting the holy Cisco campus and hanging out with other developers of the first generation of internet superstartups to discuss Linux development with its inventor, Linus Torvalds.

He might as well have been God.

But I learned quickly that Linus is very open, honest, and frank about Linux’s shortcomings – much more than most of his followers.

He went on to talk at length, in fact, about the superiority of the Windows CPU scheduler, and how Linux had not yet been able to touch its performance. It was very important to him that it improve, but confessed that it was one of the most difficult problems to solve technologically.

[The CPU scheduler is the part of the operating system kernel whose purpose is to efficiently assign slices of time to all the running processes to the CPU(s)]

This was kind of disconcerting to some of the attendees, as there has always been this aura of technical superiority of the Linux kernel to Windows. But the reality is, and remains to this day, that all of the major operating systems have their advantages and disadvantages, and to religiously ignore that fact is to just ignore reality.

Recently, there was another big problem uncovered in the Linux kernel scheduler that remains unfixed in all major distribution releases today. Of course, as always there is the option for the user to download or compile newer forks of the kernel which fix the problem, but that is not an expectation of everyday users, nor is it something desirable to IT departments for whom stability and uniformity of production OS software is essential.

Users had started to notice that the popular x264 video encoding software, which is one of the most popular, if not the most popular applications which can truly exploit efficient CPU scheduling, just ran a lot faster on Windows then Linux on the same computer.

Now, we all like software that runs faster, but video encoding is a special case of where performance is far more important than most because of the time required. You might not even notice if your browser rendered a web page 50% faster between one page load and another these days, but that same level of improvement is a godsend for video content creators.

As it turns out, changes in the Linux scheduler over time had introduced performance problems for some multithreaded applications. For most of these apps – like a webserver for instance, the issue impact is negligible, so the problem was unknown for some time. But for x264, the problem was enormous.

How enormous?

Well, the version of Linux kernel most users are running today are likely experiencing a performance penalty with x264 of up to 70%. For a rendering farm this is a massive cost penalty, as they would logically need to spend 70% more on hardware to meet the exact same workload requirements.

The problem was so severe and obvious that once notified, Linux kernel developers checked in the fix overnight, and a patch was issued. Since there attention was already on it, kernel developers were to able to later add even more performance to the application through additional kernel changes.

However, Linux distributions like Ubuntu do not run the version of the Linux kernel which was built last night, they run a proven and stable kernel which has been around for a while, with good reason. Enterprise Linux versions from Red Hat and Suse tend to use these stable kernel versions even longer.

So, for one client, who does about 18 CPU-hours of x264 encoding on Linux DAILY, this patch was really important, as it returned large gains of efficiency instantly with no new hardware costs. But lacking a vigilance of the monitoring of kernel patch developments, the problem and solution had existed outside of our awareness.