Updates, upgrades, and patches must be installed in a timely manner, following pre-defined patch management procedures with thorough monitoring to confirm proper installation and functionality.
Control Type: Technical
Control Function: Preventive
One of the best precautions for keeping IT systems protected from adversaries is ensuring they remain continuously updated, upgraded, and patched to the latest versions. New vulnerabilities are discovered every day, and vendors race against the clock to release fixes before adversaries can develop exploits. However, these fixes do no good if the end users do not apply them.
Every IT vendor has their own schedule and process for releasing updates and patches. For example, Microsoft uses the following, well-known schedule:
- Patch Tuesday (Second Tuesday of each month at 10AM Pacific Time) = cumulative updates including security and non-security fixes.
- Optional Non-Security Preview Updates (Fourth week of each month) = preview updates allowing early testing and validation prior to the next Patch Tuesday.
- Out of Band (OOB) Updates = as needed updates to address critical, high risk security vulnerabilities or stability issues
- Feature Updates = released once a year in the second half of the year (around October) to implement new features and major version changes
It is the responsibility of each organization to understand release schedules from each vendor, and to have a formal process and procedure for deploying them.
To start, it is necessary to understand the differences between updates, upgrades, and patches. Updates are regular maintenance releases, usually in the form of application packages for individual programs, and firmware for hardware assets. Upgrades are the replacement of old systems/platforms with new ones. For example, when fifty company computers are moved from Windows 10 to Windows 11, this is an upgrade. Finally, patches are any software maintenance release that covers both immediate fixes (patching zero-day vulnerabilities) and regular, planned maintenance. Oftentimes, updates and patches are used interchangeably.
All updates, upgrades, and patches should be handled according to the organization’s Configuration and Change Management (CM) procedures. The organization should use their asset inventories created in Stage 1 to guide the updating process. By understanding what applications and software platforms are running in their environment, the organization can track new updates as soon as they are released.
Next, the organization needs to define the methods they will use to implement the updates. This can vary widely depending on the size of the organization. There are two implementation approaches:
- Automated: the software updates itself on its own schedule.
- Manual: the organization deploys and manages updates from a centralized location on its own schedule or installs them one by one if no server exists.
For smaller organizations that have limited numbers of systems and applications in use, automatic updates are generally recommended. This allows the organization to keep a hands-off approach with the installation, ensuring the updates are installed when the vendor releases them. However, the organization needs to ensure that all platforms and applications have the automatic updates feature turned on, as some vendors have it off by default. The organization should also block the ability of end users to turn off the feature using Group Policy restrictions or an equivalent control. The downside of automatic updates for many users is that they can be installed at inconvenient times, potentially interfering with workflows. On many platforms, this can be mitigated by setting Active Hours, which tells the application when the end users are expected to be working. The application will adjust accordingly and only be installed outside of the defined window.
In many cases however, automatic updates are too much of a burden and inconvenience. This is especially true for larger organizations with hundreds of endpoints, and complex systems that are more sensitive to change. In such cases, manual updates are preferred. Various platforms exist for downloading and deploying mass updates from a centralized location. For Microsoft Windows based environments, options include Autopatch with Intune, Windows Server Update Services (WSUS), and Configuration Manager.
For manual updates, the organization needs to have a process defining when the updates will be deployed to the environment. There are two modes an organization’s systems can exist in: normal and maintenance. Normal mode consists of general day-to-day baseline activities, while maintenance mode is any period where the systems are removed from production for maintenance. Maintenance mode is where manual updates should be applied. Since systems are unavailable to end users in maintenance mode, the organization needs to clearly schedule maintenance windows and inform end users ahead of time. This may take the form of a mass email sent to internal users informing them of an upcoming maintenance window, or a temporary webpage informing Internet users that systems/resources are currently down for maintenance and will be back shortly. This prevents confusion and frustration from occurring when systems go down.
The organization also needs to decide whether updates will be deployed in one shot as soon as they are available, or whether they will be pushed on a phased rollout schedule, a few devices or departments at a time. The latter option is often preferred for organizations with complex operating environments and sensitive systems that may encounter issues with the new update features. If two hundred devices are given a faulty update all at once, havoc will quickly ensue when every employee in the organization starts having the same issue. A phased rollout approach can alleviate this. Updates are pushed out to one department first, and the organization monitors the systems for a defined period to ensure no issues arise. If the updates do not cause any issues, they can then be pushed out to the rest of the departments.
There are several deeper issues arising from the update/patch management process that organizations need to be wary of. Sometimes, target systems are not powered on or connected to the network when the updates are pushed out. When configuring the deployment, administrators can often configure update servers to force the update the next time the target system calls home, whether that be the day the updates are pushed out, or a week later. However, some update server configurations may only have target systems check for updates on the same day, meaning powered off systems won’t get them that day, or any day, since they will always be looking for updates from the current day and not the past. Oftentimes, the organization’s administrators will not see the missed systems until they look at the asset inventory and see that certain systems are behind. In that time, an adversary could have taken advantage of the unpatched system. It is important that organizations properly configure their update services to prevent this.
Major technology vendors are not perfect, and there are many cases of official updates and patches causing operational difficulties or opening new vulnerabilities. Organizations should not assume they can just deploy an update and call it good. In the time immediately after the confirmed deployment, the target systems should be monitored to ensure they are functioning at a baseline level. Vulnerability bulletins and major tech news sites should be consulted to see if any major vulnerabilities are discovered in the new updates. Vulnerabilities are in fact found in new software updates more often than one would expect. This is why it is imperative for organizations to include a rollback process in their overall update/patch management capabilities. Some new vulnerabilities may be so severe that the organization decides the level of risk from using the previous unpatched version is lower than the level of risk posed by the new vulnerability. In such cases, administrators should immediately enact rollback features on their update services to return the systems to the previous baseline. Major update management platforms usually include such capabilities, and they can also be done manually on individual applications and operating systems, although this takes quite a bit of time.
Throughout the entire update process, it is important for administrators to keep proper documentation. This helps the organization track successful updates, errors during deployment, resulting issues, and other data that can help guide future updates.