There is no question of the importance of keeping your servers and computers fully patched. Every second of every day hackers are scouring the Internet looking for an opportunity to take advantage of an unpatched server or computer. But what happens when a patch is faulty?
On April 19, 2016, Microsoft released patch KB3148812 a key fix for Windows 10 that would allow native decryption for updates and let them release updates for all regions simultaneously. In their words ,“Until now, we have been manually decrypting these packages prior to releasing to the WSUS channel, the process of which is both time consuming and error prone.” Oh, what a relief! The only problem? The update to fix the update caused the Windows Server Update Services (WSUS) admin console to be inaccessible on Windows Server 2012 and Windows Server 2012 R2. On April 20, a notification was sent out advising admins that there were some manual steps missing. On April 22, the manual instructions were released to the field with a cryptic instruction to un-install the earlier “fix” or if already installed and fixes with their earlier manual steps, to call support. A new update was finally released on May 5.
In all this confusion, and even if you are a pro at walking slowly and carefully out of a bad update, this whole “failed fix” would have eaten up gobs of time. How much time of yours did it waste?
Taken as a learning experience, this debacle provides a good opportunity to define or redefine ones’ strategy approaching a patch update. Here are 10 ways to save yourself some time and lower your risk.
1) Read the documentation. Sounds like the logical thing to do but sometimes in a rush, you may just skim the document and may miss something important or take it out of context. As an extra measure, have a peer read the document as well to minimize the risk of missing important information. Every organization is unique. By reviewing and double-checking the document, you may catch potential issues before the patch solution becomes a new problem.
2) Test the patch before deploying. Testing the patch in an environment that mimics production allows you to validate its quality before the implementation. Will it break anything? Will it cause downtime? You’ll know before your users do.
3) Use version control. This is especially important if you need to rollback to an earlier state. If the patch is faulty, you will want to recall the previous version that was in use.
4) Have a rollback plan. If the patch is pushed to production and there is an issue, a plan needs to be created and documented so everyone knows what actions need to be taken to restore the environment to its original state.
5) Stagger the rollouts. If the patch is for users’ computers, don’t push it to all of them at the same time. Stagger the release. If there is an issue, you will have a smaller number of computers to fix instead of the entire user base. If it is a server patch, deploy it first to a few non-mission critical servers, then progress to production systems in line with business applications, and then finally, any mission-critical servers in your network. While staggering the rollout, you will still need to assess the vulnerability of the remaining unpatched servers.
6) Schedule server updates during off hours. It’s always a good idea to effect updates when there is less traffic on the servers and fewer users that can be potentially impacted.
7) Inform technical support of the patch deployment. By informing technical support ahead of the release, they get an opportunity to prepare for the patch deployment and to watch for irregularities that may relate to the update.
8) Ensure the appropriate service pack is installed before deploying a hotfix. If the service pack is newer than the hotfix, the hotfix will not install. Don’t fall too far behind in the installation of service packs. It will create issues and additional work to install future patches.
9) Weigh the risk of not installing the patch. Does it fix a security issue or does it contain new features? If it is a security update, does the firewall already block the service? Do the new features benefit the organization? Once you review all the information, you can determine if the benefits of installing the patch outweigh the risks.
10) Define a Patch Policy. While defining some of the suggestions above within your policy, it should also include: a comprehensive list of all computers and servers on the network, a patch schedule, a list of which patches are considered low risk and which are more prone to failure. Creating an effective patch policy informs the team what is expected of them and what they need to do.
With these recommendations in mind, the KB3148812 patch issue also suggests that it’s not always a good idea to rush into a patch just because Microsoft is pushing it and bullish about its efficacy. Those system administrators who waited two weeks certainly saved themselves a great deal of time and trouble. Remember that deploying a patch should be seamless and not disrupt your business. When you incorporate best practices into your patching policy, you’ll have a clearer idea of the relative risks and benefits of a given patch. Taking steps to prevent or minimize the issues caused by a faulty patch will help keep your business operations running smoothly.
Leave a Comment