Cross-forest Kerberos custom SPN routing

One day I was working on a problem of setting up a kerberized service with custom SPN across multiple trusted AD forests. Cross-forest Kerberos is not unusual but I had never worked with custom/disjoint SPNs/domains.

User forest containing domainA.com would need to authenticate to SPN myservice.com located at resource forest containing domain forestB.com.

By default you get suffix routing for default UPN/domain name/Kerberos realm name. As client nor DC will search GC in other forests for SPNs, i scratched my head a bit and went back to documentation that I hadn’d probably read for over a decade.

Thank god Microsoft hasn’t yet deleted all Windows 2003 era stuff. While initially reading about name suffix routing i was a bit puzzled as it was talking about SPNs and UPNs together yet I could not remember nor find anything specific to SPNs. Cross-forest access pointed me to Trust Domain Objects (TDO). This is a bit of dead end as the important msDS-TrustForestTrustInfo attribute is a blob and I didn’t really want to start parsing it.

However as again UPN and SPN suffixed were being discussed together, I added custom UPN suffix myservice.com to forestB.com. After enabling routing – lo and behold – it worked for SPNs as well. This clicked some memories where I had browsed some old AD environments and seen some UPN suffixes that had not made any sense. Probably the same case.

This didn’t leave me satisfied as there definitely there had to be a more elegant solution. UPNs show up in user management and this may not be something that you want.

After digging through schema and some other searches I came upon ms-DS-SPN-Suffixes attribute that seemed interesting and actually seemed to do the job when manually edited. Yet more searches led me (back) to Set-ADForest CmdLet that has parameter to modify SPN suffixes. Huh, never noticed that.

Now looking back and knowing what to look for, it has been well described in great detail for 10 years. The linked blog post has some other great details or options.

TL;DR: For cross-forest Kerberos with custom SPN suffixes you need to use Set-ADForest to set routing hints for trusts.

ADMT Password Export Service and fixing RPC Server Unavailable

I was working on a forest migration the other day and stumbled on this infamous error a few days ago. 1722 RPC Server Unavailable.

Huh… tried a few simple Googled things

  • Check firewall for ports
  • Reinstall
  • Check permissions
  • Check registry flags
  • Check if service is running
  • Change service account
  • Probably something else that I don’t remember

Debugging

When none helped, I started to dig deeper. Password Export Server (PES) is quite a niche tool so there’s very little information on how it actually works. Portqry showed that RPC was listening on a named pipe. Therefor it talks on TCP445 and high ephemeral ports access through firewall is irrelevant, it only talks over SMB.

Tried rpcping and rpcdump from old Windows 2003 Resource Kit, RPC part seemed to work just fine.

Well Wireshark it is. After some packet capture it was clear that initial pipe setup was successful but error was coming from a FSCTL after opening the pipe, FSCTL_PIPE_TRANSCEIVE. This has exactly one useful result, about Microsoft Exchange. That rabbit hole led nowhere, let me save you the trouble, ACL problem described actually is irrelevant.

At this point it was getting quite interesting. I had exhausted simple debugging options so I started to investigate, how this actually works. First I opened installation MSI in ORCA. What I saw is that it installs 3 files including a library (DLL) that does not seem to be used anywhere. I analyzed some dependencies and it seemed independent. Interesting…

Further analysis of MSI let me to an interesting CustomAction function call “AddToLsaNotificationPkgValue”. CustomAction binary itself is a DLL. I don’t have the skills decompile this but I saw some function calls that caught my interest.

LSA? Notfication Packages? PasswordChangeNotify? Sounds like Active Directory Password Filters! Quick check with Nirsoft DLL Export Viewer confirmed that this PwMig.dll is indeed a password filter, with only 3 compulsory functions exported. At this point it clicked to me how PES actually works – back to this later, let’s continue with debugging.

To register a password filter, you have to place the relevant DLL file in System32 folder and add a registry value. Password filters are quite rare in the real world, they have 2 real-world uses, neither that is very widespread

  • Password synchronization (from Active Directory)
  • Custom password complexity requirement enforcement

However this environment had a password synchronization solution in use so the value existed. Also the required registry value was of the wrong type. Documentation requires REG_MULTI_SZ however REG_SZ was actually used.

Fix

The real fix was adding recreating Notification Packages value with correct type and adding existing password sync DLL name with PES DLL name. I’m not sure if the real problem was wrong value type or the value simply existing. Either way the setup didn’t handle the error and didn’t give any feedback that the registry field update had failed.

You must restart the DC as the DLL only gets loaded at LSASS start.

How it works

By design you cannot read object’s password hash in Active Directory. At least this is what the documentation tells you. However it is very useful for migration scenarios to be able to migrate password (as a hash).

LSASS itself can of course read the hash, for password validation. There is just no API for user to access it. Getting code into LSASS also tends to be quite hard without injection.

My suspicion is that Microsoft used password filter to… let’s say legally/officially… get code into LSASS. Password filter part is probably a red herring. I don’t have the skills to investigate but these 3 functions likely do nothing. On the other hand, DLL probably enables a new interface (or backdoor?) to read hashes. I had noticed earlier that PES holds a handle to LSASS, so probably PES is just a proxy to LSASS. RPC error is probably returned by LSASS that doesn’t have the DLL loaded and therefor no interface for PES to access. PES merrily proxies the error and you are left scratching your head. Now you know.

Loading certificates from SK LDAP for Estonian ID-Kaart SmartCard authentication to Active Directory – the old way

Phew, that’s a long title. But to the point. Many years ago I promised to release that script. In the meanwhile ID-Kaart PKI topology has changed but I think that the script remains quite relevant as it should be quite easy to fix up.

About LDAP interface. I think you need to query both as not all cards from old root have expired.

The official doc for configuring ID-Kaart login:

Unfortunately it lacks mass-loading. Using ADUC per-certificate is just… not scalable at all.

Remarks:

  • It was originally written… I guess about 7 or 8 years ago for exactly that reason – manual loading of certificates is just impossible but in the smallest of environments. First attempt used commercial cmdlets as native LDAP in PowerShell used to require (still does?) some native .Net binding and it was easier that way.
  • There were a few commercial products for mass-loading but I guess I just closed their businesses if they even still exist (didn’t check)
  • In the olden days you required a contract with SK as LDAP was (is?) throttled for those without whitelisted IPs. Too many queries got you blocked for some time. Maybe a few sleeps here and there helps…
  • As usual, some logging and crust have been removed.
  • I’m not going to discuss all the requirements for SmartCard login, SK’s document has a pretty good overview.
  • But you CAN use one certificate with several accounts, unlike stated in SK’s document. Maybe more on this later.
  • I don’t remember exactly where I got the LDAP code from but I think it was some SDK example for C# or something. Who knows, MS keeps dropping useful doc all the time so it’s probably gone anyways.
  • Maybe oneday I’ll fix it up for new topology, perhaps one query per person or more optimizations…
  • Not supported, not tested (after a few changes just now),  a bit of code rot (not used by me for years) – understand what you are doing

 

Function Get-AuthenticationCertificate {
    param(
        [long]$IDCode,
        [string]$Type
    )
    $Filter = "serialnumber=$IDCode"
    $BaseDN = "ou=Authentication,o=$Type,c=EE"
    $Attribute = "usercertificate;binary"
    $Scope = [System.DirectoryServices.Protocols.SearchScope]::subtree
    $Request = New-Object System.DirectoryServices.Protocols.SearchRequest -ArgumentList $BaseDN, $Filter, $Scope, $Attribute
    $Response = $LdapConnection.SendRequest($Request, (New-Object System.Timespan(0,0,120))) -as [System.DirectoryServices.Protocols.SearchResponse]
    If ($Response.Entries.Attributes.$Attribute) {
        $Certificate = [System.Security.Cryptography.X509Certificates.X509Certificate2] [byte[]]$Response.Entries.Attributes.$Attribute[0] #Cast byte array to certificate object
        Return ("X509:<I>" + $Certificate.GetIssuerName().Replace(", ",",") + "<S>" + $Certificate.GetName().Replace(", ",",")) #Probably string replacement is not needed, just following empirical behavior from ADUC.
    }
}

#Contains all useful SK LDAP Certificate branches
$SKCertificateBranches = @("ESTEID","ESTEID (DIGI-ID)")
[Reflection.Assembly]::LoadWithPartialName("System.DirectoryServices.Protocols") 
$LdapConnection = New-Object System.DirectoryServices.Protocols.LdapConnection "ldap.sk.ee" 
$LdapConnection.AuthType = [System.DirectoryServices.Protocols.AuthType]::Anonymous
$LdapConnection.SessionOptions.SecureSocketLayer = $false #New one uses TLS
$LdapConnection.Bind()
#Loads AD Users. For example you store ID code in extensionAttribute1.
#There is no validation or filter IF actually user has ID-code stored. That's a task left to you as it's quite environment dependent. For example refer to my article about ID-code validation
$ADUsers = Get-ADUser -Filter *-SearchBase "DC=my,DC=domain,DC=com" -Properties altSecurityIdentities,extensionAttribute1
ForEach ($ADUser in $ADUsers) {
    $UserSKCerts = @()
    ForEach ($SKCertificateBranch in $SKCertificateBranches) {
        $UserSKCert = Get-AuthenticationCertificate $ADUser.extensionAttribute1 $SKCertificateBranch #positiional attributes
        If ($UserSKCert) {
            $UserSKCerts += $UserSKCert #Slow but whatever, it's a small array
        }
    }
    #Arrays must be sorted before compare because they are retrieved in undetermined order
    If (Compare-Object -ReferenceObject $UserSKCerts -DifferenceObject $ADUser.altSecurityIdentities) {
        Set-ADUser $ADUser -Replace @{"altSecurityIdentities"=$UserSKCerts}
    }
}
$LdapConnection.Dispose()

Online P2V of domain controllers

Don’t do it or do it in DSRM. Until for various reasons you just… can’t. Unacceptable downtime, Exchange/SBS, Windows 2003 (can’t stop AD services), etc. Doesn’t matter, you just have to do the P2V online.

It’s not supported (probably) or recommended but if you really need to then (skipping obvious steps):

  1. Stop replication some time before finalizing conversion
    repadmin /options %COMPUTERNAME% +DISABLE_OUTBOUND_REPL
    repadmin /options %COMPUTERNAME% +DISABLE_INBOUND_REPL
  2. Disconnect target VM network and boot to DSRM.
  3. Set “database restored from backup” flag in registry – just in case!
    https://technet.microsoft.com/nl-nl/library/dd363545(v=ws.10).aspx
  4. Boot normally
  5. Enable replication
    repadmin /options %COMPUTERNAME% -DISABLE_OUTBOUND_REPL
    repadmin /options %COMPUTERNAME% -DISABLE_INBOUND_REPL

     

Again, not supported nor recommended but it has worked for me.