SMS and SCCM Patch management –> An automated security update rollback process

Mar 01 2009 Published by admin under Tech Tips

Introduction

SMS and SCCM give us the ability to build a process for full automated patch deployment. A healthy site with healthy clients generally leads to a “smooth” automated patch deployment process.

One of the first challenges I faced as an SMS/SCCM administrator, was answering the change management question “what is the rollback process for patch deployment?”  The only answer available is manual rollback (all hands on deck). This presents a major challenge if you do not have resources readily available during an emergency rollback scenario. Why not use your automated patch deployment tool to address this challenge.

In this article, I provide a method for rolling back security patches in line with Microsoft best practices. This process only applies to patches deployed to windows server 2003, Windows XP and below operating systems. I am working on updating the process for Vista and Windows Server 2008.

Background to process

This link provides the background to this rollback process Removing Windows software updates in the wrong order may cause the operating system to stop functioning.

The recommend method for rolling back patches is to remove patches in the reverse order of installation. This recommendation is based on the fact that most patches update the same DLLs etc. So in a scenario where 3 patches update the same DLL,

  • Install Patch 1 (DLL updated to V1 backup original DLL for rollback)
  • Install Patch 2 (DLL updated to V2 backup V1 DLL for rollback)
  • Install Patch 3 (DLL updated to V3 backup V2 DLL for rollback)

Removing patch 2 will return the DLL to V1 and lose the update made by patch 3. So how do we keep the system consistent and not lose other updates? . The answer is to rollback all patches and redeploy without the unwanted patch(es). Another challenge is, can this be automated?

In order to achieve the above, we first need to establish the original order of deployment and create an automated rollback deployment using SMS/SCCM software distribution.

Summary of process

  1. Query the client for all patches deployed and list by installation date time order.
  2. All patches for the latest date listed to be removed (in general deployments would be for same day and not across multiple days)
  3. Run spuninst.exe for the patch(es) to remove in the reverse order from the %SystemRoot%\$NtUninstall[KBArticleNumber]$\
  4. Steps 1 to 3 achieved with a VB script delivered as a standard software distribution package advertisement
  5. Initiate rollback by advertising to SMS/SCCM clients in scope using a collection

Script and Sample Screenshots

The script supplied is set to log only mode (need to change the test mode parameter to 1 for it to be in live mode). Both modes would create a hotfixundo.log file on the C:\ drive. Script kindly written by Gavin Woodall.

Copy the script to notepad and save as hotfix_undo_Live.vbs (or to any preferred name). In my case I have a package called Patch Rollback – Live. The Data Source is a package directory called Patch_Rollback (store the vbs script here and reference during package creation)

image image

Create a program for the package using the following command line : cscript %scriptname% (in my case %scriptname% = Hotfix_undo_live.vbs). Ensure that the program is set to run whether or not a user is logged on for non interactive deployments/advertisements.

image image

Create an advertisement for the package. Do not leave on a recurring schedule!!! – This would remove all patches from the targeted clients.

image

SMS 2003 Process: After each rollback create a new program (by default you will not be able to use the same program again if it has successfully run on a client). I create a new program every month just to be sure.

SCCM Process note: SCCM overcomes the SMS 2003 limitation because programs can be rerun even when successful

image

Copy Below to notepad and save as hotfix_undo_live.vbs (change testmode to 0 to make live)

‘ Script to enumerate last applied hotfixes, and rollback

on error resume next
const forappending = 8
const forwriting=2
const forreading=1
Const dictKey  = 1
Const dictItem = 2

‘ **********set to 0 to get out of testmode**********
testmode=1
‘ ***************************************************

Logpath=”C:\hotfixundo.log”

Set fso = createObject(“Scripting.FileSystemObject”)
set windir=fso.GetSpecialFolder(0)
call stamplog(“*************************************************************”)
call stamplog(“Starting process, windows directory is “&windir.path)

lastdate=”"
‘ enumerate subfolders, check date.
For Each Subfolder in windir.SubFolders
if instr(lcase(subfolder.name),”$ntuninstall”)<>0 then
if lastdate=”" then
lastdate=subfolder.datecreated
end if
if datediff(“d”,lastdate,subfolder.datecreated)>1 then
lastdate=subfolder.datecreated
end if
end if
Next

call stamplog(“Latest date found for uninstall folder is “&lastdate)
‘ loop again, creating a list of directories to be targeted.

set list = CreateObject(“Scripting.Dictionary”)

call stamplog (“Processing the following directories:”)
For Each Subfolder in windir.SubFolders
if instr(lcase(subfolder.name),”$ntuninstall”)<>0 then
if datediff(“d”,lastdate,subfolder.datecreated)<1 and datediff(“d”,lastdate,subfolder.datecreated)>=0 then
list.add subfolder.datecreated,subfolder.path
call stamplog(subfolder.path)
end if
end if
Next

‘ sort dictionary
sortdictionary list,dictkey

‘ loop through list, shell out to run spuninst for each directory, last first
for each location in list
call stamplog(“Launching “& list.item(location)&”\spuninst\spuninst.exe”)
err.clear
if testmode=0 then
run list.item(location)&”\spuninst\spuninst.exe /quiet /passive /norestart”
else
call stamplog(“***TESTMODE – Uninstall NOT run***”)
end if
next
call stamplog(“Finished at “&date&” “&time)

‘ Stamp line of text to specified logfile
sub stamplog(text)
Set objFSO = CreateObject(“Scripting.FileSystemObject”)
Set objTextFile = objFSO.OpenTextFile _
(logpath, Forappending, True)
objtextfile.writeline(text)
wscript.echo (text)
objTextFile.Close
end sub

‘ Run function
Function Run (ByVal cmd)
Dim sh: Set sh = CreateObject(“WScript.Shell”)
Dim wsx: Set wsx = Sh.Exec(cmd)
If wsx.ProcessID = 0 And wsx.Status = 1 Then
‘ (The Win98 version of VBScript does not detect WshShell.Exec errors)
Err.Raise vbObjectError,,”WshShell.Exec failed.”
End If
Do
Dim Status: Status = wsx.Status
WScript.StdOut.Write wsx.StdOut.ReadAll()
WScript.StdErr.Write wsx.StdErr.ReadAll()
If Status <> 0 Then Exit Do
WScript.Sleep 10
Loop
Run = wsx.ExitCode
End Function

‘ Runs an internal command interpreter command.
Function RunCmd (ByVal cmd)
RunCmd = Run(“%ComSpec% /c ” & cmd)
End Function

‘ Sort function
Function SortDictionary(objDict,intSort)
‘ declare our variables
Dim strDict()
Dim objKey
Dim strKey,strItem
Dim X,Y,Z

‘ get the dictionary count
Z = objDict.Count

‘ we need more than one item to warrant sorting
If Z > 1 Then
‘ create an array to store dictionary information
ReDim strDict(Z,2)
X = 0
‘ populate the string array
For Each objKey In objDict
strDict(X,dictKey)  = CStr(objKey)
strDict(X,dictItem) = CStr(objDict(objKey))
X = X + 1
Next

‘ perform a a shell sort of the string array
For X = 0 to (Z – 2)
For Y = X to (Z – 1)
If StrComp(strDict(X,intSort),strDict(Y,intSort),vbTextCompare) > 0 Then
strKey  = strDict(X,dictKey)
strItem = strDict(X,dictItem)
strDict(X,dictKey)  = strDict(Y,dictKey)
strDict(X,dictItem) = strDict(Y,dictItem)
strDict(Y,dictKey)  = strKey
strDict(Y,dictItem) = strItem
End If
Next
Next

‘ erase the contents of the dictionary object
objDict.RemoveAll

‘ repopulate the dictionary with the sorted information
For x=(z-1) to 0 step -1
‘    For X = 0 to (Z – 1)
objDict.Add strDict(X,dictKey), strDict(X,dictItem)
Next

End If

End Function

Additional Notes:

Every security update has a Removal information section listed under Security Update Deployment. So for MS09-001 you would find below for the XP operating systemhttp://www.microsoft.com/technet/security/Bulletin/MS09-001.mspx

Removal Information

Use Add or Remove Programs tool in Control Panel or the Spuninst.exe utility located in the %Windir%\$NTUninstallKB958687$\Spuninst folder

No responses yet

ITMU to SCCM Patch management –> A process approach

Feb 25 2009 Published by admin under Tech Tips

Introduction

Having worked extensively with ITMU in SMS 2003 for datacenter patch management of servers, I welcomed the new architecture promised for SCCM.

If you are using ITMU now and are new to SCCM here is a high level summary of the patch management components:

  • SCCM Clients are scanned using the clients Windows Update Agent (WUA)
  • WSUS used as the scan catalog known as a SUP (basically WSUS dedicated to SCCM and no more 5MB local catalog downloads to all clients)
  • Download and execute option now does a scan before and, only downloads required updates
  • Security updates are categorized as in native WSUS and now have the ability to deploy non security updates including service packs.
  • Status of patch deployment is provided near real-time (well every 15 minutes by default) by state messages; no longer uses advertisement reports and hardware inventory . I have an earlier blog that shows you how you can get basic information collected using hardware inventory.

Below is a link to a very good whitepaper providing extensive details.

Configuration Manager Software Updates Management Guidance – Migration from ITMU.doc

My aim in this article (blog) is to give you a field view of what it means to translate these changes into existing processes. In a nutshell going from reading about it to using it.

SCCM SUM Reduces Wizard Screens?

A statement I have read many times about SCCM is, it reduces the old ITMU wizard screens from 18 to about 7. I disagree and will quantify it with, only if you use the power and flexibility of the templates under deployment management.

If you are like me, the first thing you do with a new version of a product is to make it work like the old one (how many people turned the WK3 interface into W2K?).

My attempt at an ITMU to SCCM translator below should hopefully ease some of your pain.

ITMU to SCCM translator

SMS 2003 ITMU SCCM Software Updates Management (SUM)
ITMU Scan Tool Software updates scan agent
Recurring ITMU Scan Tool Advertisement Software updates scan agent schedule – WUA scan using SUP (SCCM dedicated WSUS)
Advertisements Deployments
Packages (one to one relationship with selected patches) Deployment Packages (selected patches not linked to one package; will search all packages on the DP and download from any package)
Advertisement Start time Deployment Deadline
Expiring Advertisements Use maintenance window on targeted collection with Recurrence set to None. TIP:
Advertisement Start Time = Deployment Deadline = Maintenance Windows start.
Expiry time = Maintenance Window End

Useful and New to SCCM

Now lets take a closer look at the SUM components and sample patch management process.

Summary of the steps for a sample process:

  1. Create a search folder to group security updates
  2. Create an empty collection with no members (to be used for the deployment templates)
  3. Create a deployment template (I create two; 1 for Patch Only and 1 for Patch with Reboot)
  4. Create a folder for storing the source files for packages
  5. Create an update list (e.g. Select required patches for your deployment), specify download updates to create the package.
  6. Drag the update list onto the deployment template to create the deployment (Deployment type will be determined by the template in this case)
  7. Create a maintenance window for the collection to be targeted
  8. Modify the Deployment by changing the collection specified (inherited from the template) and also the deadline date and time.

Detailed steps:

Update Repository: this is where the software updates are displayed and categorized. Shows all software updates depending on what you have selected under the SUP (WSUS) configuration.

image

  • Search Folders: allow you to group software updates logically for ease of selection when creating deployment packages. In my example I have a master search folder for all Security patches and one folder for every year from 2003 – 2009. Use a search criteria on Bulletin ID using % so for 2009 would be MS09%

imageimage

  • Create an empty collection: I am a great fun of place holder collections. I use them as a safety check before targeting the real collections. In this case I created a collection called   image with no members (safe to ignore the warning).
  • Create deployment templates: Now this is where the wizard pages reduction takes place. Right click the deployment templates node and select new deployment template. Once created, using the template significantly reduces the number of wizard screens. I created two, one with suppressed reboots and, the other without.

imageimage

image image

image image

  • Create a package source folder: I typically create top level folder for all packages and then sub-folders for categories of packages. In this example process we will use a subfolder called “Security_Updates”

image

  • Create an update list: Using the “All Security updates” search folder as an example select the security updates required for the SUM package. Selection is now much better as you can use the shift key, and the control key, to block select security updates.

image image

Select download updates during the creation of the update list. You can create a new package or select an existing page. NB be sure to specify a new subdirectory as part of the UNC to the package directory. If you do not specify a subdirectory all updates are placed in the root folder (near impossible to tidy up when you delete a package)

image

image image

  • Create a deployment (replaces advertisements in the ITMU deployment process):Drag and drop the update list onto a deployment template. In this example we use the patch only template. Notice that the collection used is the place holder we created and selected for our template. In addition the suppress restart and any other general properties are inherited from the template. This is the magic of the wizard reduction I mentioned. Modify the settings to required deployment deadline and target collection

image image

Monitor the deployment using the new Software Updates category reports.

No responses yet