ADMT Password Export Service and fixing RPC Server Unavailable

I was working on a forest migration the other day and stumbled on this infamous error a few days ago. 1722 RPC Server Unavailable.

Huh… tried a few simple Googled things

  • Check firewall for ports
  • Reinstall
  • Check permissions
  • Check registry flags
  • Check if service is running
  • Change service account
  • Probably something else that I don’t remember

Debugging

When none helped, I started to dig deeper. Password Export Server (PES) is quite a niche tool so there’s very little information on how it actually works. Portqry showed that RPC was listening on a named pipe. Therefor it talks on TCP445 and high ephemeral ports access through firewall is irrelevant, it only talks over SMB.

Tried rpcping and rpcdump from old Windows 2003 Resource Kit, RPC part seemed to work just fine.

Well Wireshark it is. After some packet capture it was clear that initial pipe setup was successful but error was coming from a FSCTL after opening the pipe, FSCTL_PIPE_TRANSCEIVE. This has exactly one useful result, about Microsoft Exchange. That rabbit hole led nowhere, let me save you the trouble, ACL problem described actually is irrelevant.

At this point it was getting quite interesting. I had exhausted simple debugging options so I started to investigate, how this actually works. First I opened installation MSI in ORCA. What I saw is that it installs 3 files including a library (DLL) that does not seem to be used anywhere. I analyzed some dependencies and it seemed independent. Interesting…

Further analysis of MSI let me to an interesting CustomAction function call “AddToLsaNotificationPkgValue”. CustomAction binary itself is a DLL. I don’t have the skills decompile this but I saw some function calls that caught my interest.

LSA? Notfication Packages? PasswordChangeNotify? Sounds like Active Directory Password Filters! Quick check with Nirsoft DLL Export Viewer confirmed that this PwMig.dll is indeed a password filter, with only 3 compulsory functions exported. At this point it clicked to me how PES actually works – back to this later, let’s continue with debugging.

To register a password filter, you have to place the relevant DLL file in System32 folder and add a registry value. Password filters are quite rare in the real world, they have 2 real-world uses, neither that is very widespread

  • Password synchronization (from Active Directory)
  • Custom password complexity requirement enforcement

However this environment had a password synchronization solution in use so the value existed. Also the required registry value was of the wrong type. Documentation requires REG_MULTI_SZ however REG_SZ was actually used.

Fix

The real fix was adding recreating Notification Packages value with correct type and adding existing password sync DLL name with PES DLL name. I’m not sure if the real problem was wrong value type or the value simply existing. Either way the setup didn’t handle the error and didn’t give any feedback that the registry field update had failed.

You must restart the DC as the DLL only gets loaded at LSASS start.

How it works

By design you cannot read object’s password hash in Active Directory. At least this is what the documentation tells you. However it is very useful for migration scenarios to be able to migrate password (as a hash).

LSASS itself can of course read the hash, for password validation. There is just no API for user to access it. Getting code into LSASS also tends to be quite hard without injection.

My suspicion is that Microsoft used password filter to… let’s say legally/officially… get code into LSASS. Password filter part is probably a red herring. I don’t have the skills to investigate but these 3 functions likely do nothing. On the other hand, DLL probably enables a new interface (or backdoor?) to read hashes. I had noticed earlier that PES holds a handle to LSASS, so probably PES is just a proxy to LSASS. RPC error is probably returned by LSASS that doesn’t have the DLL loaded and therefor no interface for PES to access. PES merrily proxies the error and you are left scratching your head. Now you know.