Pushing Kernel Changes to Millions and Millions of Machines

Twice a month a button gets pushed. It's a rather standard looking button with no blinking text or vivid color treatment and you'd be forgiven to not think much of it. Once pushed though, tens of megabytes of finely crafted compiled kernel extensions are launched onto millions and millions of customer machines. Considering the industry average is one bug per one thousand lines of code, there's likely bugs in there. If you're lucky th bug you made will just kernel panic their machine. If you're unlucky, the machine won't boot or enter an endless reboot cycle. If the gods of silicon hate you, you start destroying their data.

Yet twice a month this happens rather unceremoniously at my company. This talk covers how and why that happens from feature development through testing and deployment to customer support. Real world stories of mistakes and near misses will be used to support the counterintuitive solution to this problem. It'll be a fast paced talk so make sure to get some coffee.

• Development