SCOM management packs in Zabbix – a year later

I discussed this about a year ago but in the end I didn’t publish anything. I actually did get “Windows  Server Operating System” MP to be pretty much feature-complete (no to little OS metadata – health checks only) and it pretty much blows away any Zabbix built-in template and any other I’ve seen. There’s a few addition bits that I found useful. Works fine on Windows 2012+ and… more-less fine on 2008 and 2008R2. Some items are missing due to different performance monitors but I really haven’t bothered to edit it (physical disk and networking if I remember correctly). All items and triggers use macros so it’s easy to override checks.

The main issue remains 256 char item limit. I did make some progress in packing extra PowerShell in this small limit so previous posts may not be up to date, so templates still don’t require any changes to agent or any local scripts. Another issue is that I can’t reference items from other (linked) templates in triggers. And as you can’t add the same item in another template, it makes some templates REALLY annoying. 30 second command timeout remains an issue so you can’t actively defrag/chkdsk/unmap/trim or do very expensive checks. Command timeout with proxy seems to cause proxy to reissue commands every few minutes, causing performance issues as commands never complete and just repeat indefinitely. I did leave the checks in but disabled them. File system health is checked from just dirty flag and fragmentation information is checked from registry last run data. It seems to trigger false positives occasionally from VMware snapshots but works reasonably well. I did figure out how to change disk optimization from weekly to daily in PowerShell but it’s waaaaay too big to fit in item for all OS. I did consider building item command from multiple macros but this change would have little value. For reference (2012+ only):

$v=[environment]::OSVersion.Version;If($v.major -gt 6 -or ($v.major -eq 6 -and $v.minor -ge 2)){$s='ScheduledDefrag';$t=Get-ScheduledTask $s|export-scheduledtask;$t.Task.Settings.MaintenanceSettings.Period='P1D';register-scheduledtask -TaskN $s -TaskP '\Microsoft\Windows\Defrag' -X $t.outerxml -F}

I did some work on ADDS and File server MPs but it’s really time-consuming and they remain incomplete (they have helped to catch a few incidents though). I did mostly complete Exchange template but it’s mostly telemetry (as in original MP) and alerting mostly works by querying health monitor – but again, it has helped to diagnose issues and catch incidents early.

I’ll try to clean them up and release somehow… someday.

PS! I still think that Zabbix sucks but it’s one of the best among free stuff. 🙂

Workaround for NTFS deduplication error 0x8007000E Not enough storage is available to complete this operation

This can pop up when starting an optimization job, even when you have plenty of RAM, even if you give tons of memory to job. Error message is misleading, storage here means memory.

Workaround is to just increase page file. I came across this issue on a Server Core 2016 that had 24GB of RAM for a 16TB volume. Analysis job caused commit to grow to almost 90% (without releasing it in time) so optimization could not allocate any memory. I didn’t go in depth (RAMMap etc) though. After increasing page file from automatic ~2GB to 16GB, jobs work just fine.

Keep in mind that commit does not mean that memory or page file is actually used. It just means that application has been promised that this memory will be available when it will be actually used. Unused commit is taken from pagefile first so it’s basically free performance-wise, except for increased disk space use.

Online P2V of domain controllers

Don’t do it or do it in DSRM. Until for various reasons you just… can’t. Unacceptable downtime, Exchange/SBS, Windows 2003 (can’t stop AD services), etc. Doesn’t matter, you just have to do the P2V online.

It’s not supported (probably) or recommended but if you really need to then (skipping obvious steps):

  1. Stop replication some time before finalizing conversion
    repadmin /options %COMPUTERNAME% +DISABLE_OUTBOUND_REPL
    repadmin /options %COMPUTERNAME% +DISABLE_INBOUND_REPL
  2. Disconnect target VM network and boot to DSRM.
  3. Set “database restored from backup” flag in registry – just in case!
    https://technet.microsoft.com/nl-nl/library/dd363545(v=ws.10).aspx
  4. Boot normally
  5. Enable replication
    repadmin /options %COMPUTERNAME% -DISABLE_OUTBOUND_REPL
    repadmin /options %COMPUTERNAME% -DISABLE_INBOUND_REPL

     

Again, not supported nor recommended but it has worked for me.

Windows 7 refuses to connect to 802.1X network if server certificate’s subject is empty

If the following are true…

  • Windows 7 connects to 802.1X enabled network
  • EAP method has something to do with TLS (PEAP, EAP-TLS…)
  • Server certificate’s subject field is empty

…then Windows 7 will refuse to connect with useless error messages. You’ll just have to know that Windows 7 doesn’t accept server certificate with empty subject. Some Certificate Services templates (Kerberos Authentication) keep subject empty by default so watch out if you have NPS on DC for example. Windows 8.1+ will work fine.

There’s little information about it online and the issue is quite hard to track down.

vSphere 6.5 guest UNMAP may cause VM I/O latency spikes – fixed in update 1

I converted some VMs to thin and upgraded VM hardware version to 13 to test out savings. Initial retrim caused transient I/O slowdown in VM but the issue kept reappearing randomly. I/O latency just spikes to 400ms for minutes for no apparent reason. It also seems to affect other surrounding VMs, just not as badly. After several days, I converted VMs back to thick and issues disappeared.

I’m not sure where the problem is and I can’t look into it anymore. Might be a bug in vSphere. Might be the IBM v7000 G2 SAN that goes crazy. As I said, I cannot investigate it any further but I’ll update the post if I ever hear anything.

PS! Savings were great, on some systems nearly 100% from VMFS perspective. On some larger VMs with possible alignment issues, reclamation takes several days though. For example, a 9TB thick file server took 3 days to shrink to 5TB.

Update 2017.o6.29:

Veeam’s (or Anton Gostev’s) newsletter mentioned a similar issue just as I came across this issue again in a new vSphere cluster. In the end VMware support confirmed the issue with expected release of 6.5 Update 1 at the end of July.

Update much later in november

I’ve been running Update 1 since pretty much release date and UNMAP works great! No particular performance hit. Sure, it might be a bit slower during UNMAP run but it’s basically invisible for most workloads.

I’ve noticed that for some VM’s, you don’t space back immediately. On some more internally fragmented huge (multi-TB) VMs, particularly those with 4K clusters, space usage seems to reduce slowly over days or weeks. I’m not sure what’s going on but perhaps ESXi is doing some kind of defrag operation in VMDK…? And yeah, doing a defrag (you can do it manually form command line in Windows 2012+) and then UNMAP helps too.

vSphere 6.5 virtual NVMe does not support TRIM/UNMAP/Deallocate

I was playing with guest TRIM/UNMAP the other day and looked at new shiny virtual NVMe controller. While it would not help much in my workloads, cutting overhead never hurts. So I tried to do “defrag /L” in VM and it return that device doesn’t support it.

So I looked up release notes. Virtual NVMe device: “Supports NVMe Specification v1.0e mandatory admin and I/O commands”.

The thing is that NVMe part that deals with Deallocate (ATA TRIM/SCSI UNMAP in NVMe-speak) is optional. So back to pvscsi for space savings…

An unpopular opinion about Vista

I have said it again and again. I think Vista was not a bad OS at all. Not the greatest but somewhere between good and great.

While I missed very early teething issues, I did catch a few. I didn’t get to use Vista until I completed my military service, in summer of 2007. This was the first and last OS that caused me to say “wow” on first boot. It just looked so great! Sure, Linux had all the bells and whistles and XP had WindowBlinds but they never looked as clean and classy. But to get that far, I had to remove some RAM as setup hung when you had more than 2GB… And then I got a BSOD due to Bluetooth stack. 🙂
I did keep on using Vista personally until a few months after 7 came out.

I did plenty of Vista rollouts in 2008 and 2009 and… it worked great. By that time SP1 was out and drivers had stabilized. On most of hardware it ran just fine. Maybe not as fast but XP the difference was not noticeable and people actually liked Vista. For most of enterprises, I think it was a mistake to skip Vista. As tooling and many OS concepts had changed considerably, I saw many people complaining after Windows 7 release. They hadn’t even touched Vista and were surprised how similar Vista and 7 were.

Security was better. UAC was actually great (it had some nice side-effects). Quite a few features actually became usable compared to XP. It had some nice features for sysadmins that went relatively unnoticed. On the other hand, early tools sucked big time. Later WAIKs were much better and by SP2 it pretty much looked as it does today.

I switched jobs in 2010 and didn’t get to professionally touch Vista since. Kind of sad actually. Technology was solid but teething issues caused an unrecoverable PR nightmare.

Clearing Offline Files temporary files from script

There’s a nice button “Delete temporary files” in GUI to clear automatically cached data but no public information how to invoke it from script/API.
I found some nice WMI documentation and after some experimentation I came up with this.
It only runs from admin context. If you want to run it from regular user context, modify flags according to documentation (use only 0x00000002 flag).
It might be a little faster if you filter item list to only include servers (add -Filter ‘itemtype=3’) as default list includes whole UNC trees but I didn’t test it out.

$CSCItemList=(gwmi win32_offlinefilesitem).ItemPath
$CSCWMI = [wmiclass]'\\.\root\cimv2:win32_offlinefilescache'
#0x00000002+0x80000000 to Base10 eq 2147483650
$CSCWMI.DeleteItems($CSCItemList,2147483650)

Workaround script to clean up SCCM 1610 orphaned cache

SCCM 1610 at launch had a bug that caused agent upgrades to forget about cached content. Cached data stays behind until you clean it up manually, not cool for small SSDs. More here https://support.microsoft.com/en-us/kb/3214042

So I wrote a small script to roll out with compliance and remove stale data.

Seems to work but test before use. See comments for PowerShell 2.0 fix.

$CCMCache = (New-Object -ComObject "UIResource.UIResourceMgr").GetCacheInfo().Location
#For some reason it doesn't properly directly select required attribute for returned multi-instance object so I have to loop it. Some strange COM-DotNet interop problem?
$ValidCachedFolders = (New-Object -ComObject "UIResource.UIResourceMgr").GetCacheInfo().GetCacheElements() | ForEach-Object {$_.Location}
$AllCachedFolders = (Get-ChildItem -Path $CCMCache -Directory).FullName

ForEach ($CachedFolder in $AllCachedFolders) {
    If ($ValidCachedFolders -notcontains $CachedFolder) {
        Remove-Item -Path $CachedFolder -Force -Recurse
    }
}

Script to modify SCCM client cache ACL for Peer Cache

SCCM 1610 now supports inter-node content sharing without BranchCache or 3rd party tools. Annoying part is that you have to modify client cache ACL. I threw together some quick-n-dirty bits in a few minutes and it didn’t blow in my face just yet. I rolled it out with a compliance baseline to some pilot systems and it seems to work.
Caution is advised as I didn’t test it fully yet (or if Peer Cache actually works properly). It just adds required ACE for your SCCM network access account.

#SCCM Network Access account. I think it's not possible to query it from client
$NetworkUserAccount = New-Object System.Security.Principal.NTAccount("DOMAIN\User")
#SCCM Cache path from WMI. It's pretty much the same always but just in case...
$CCMCache = (New-Object -ComObject "UIResource.UIResourceMgr").GetCacheInfo().Location

#Enums for NTFS ACLs, static stuff. Could do better but stringbased cast works fine
$ACLFileSystemRights = [System.Security.AccessControl.FileSystemRights]::FullControl
$ACLAccessControlType = [System.Security.AccessControl.AccessControlType]::Allow 
$ACLInheritanceFlags = [System.Security.AccessControl.InheritanceFlags]"ContainerInherit, ObjectInherit"
$ACLPropagationFlags = [System.Security.AccessControl.PropagationFlags]::InheritOnly

#If cache folder doesn't exist, quit with error
If (!(Get-Item -Path $CCMCache)) {
    Exit 1
}

#Current ACL
$ACL = Get-Acl -Path $CCMCache

#Check if ACL already has required entry. If it has, quit cleanly
If ($ACL.Access | Where-Object -FilterScript {
    #Specific checks
    $_.FileSystemRights -eq $ACLFileSystemRights -and 
    $_.AccessControlType -eq $ACLAccessControlType -and
    $_.IdentityReference -eq $NetworkUserAccount -and
    $_.InheritanceFlags -eq $ACLInheritanceFlags -and
    $_.PropagationFlags -eq $ACLPropagationFlags
    }
) {
    #ACL entry exists
    Exit 0
} Else {
    #Modify ACL
    $ACE = New-Object System.Security.AccessControl.FileSystemAccessRule ($NetworkUserAccount, $ACLFileSystemRights, $ACLInheritanceFlags, $ACLPropagationFlags, $ACLAccessControlType) 
    $ACL.AddAccessRule($ACE)
    Set-Acl -Path $CCMCache -AclObject $ACL
}