My husband changed his Active Directory password. Routine enough – we’ve got 15k accounts at the office and require a password change every 90 days. That’s 150-200 people changing their password every day. They get themselves locked out a lot (mobile devices, cached workstation credentials, and a host of other unique places people manage to store their creds), but it’s trivial to unlock an individual user.

*Except* — after the account was unlocked, his Windows 10 mail client updated properly and was interacting with the Exchange server. Android, however, still wouldn’t accept his new password. If he typed the wrong thing, it would say invalid password. But whenever he typed the right thing, he got an error indicating the phone and tablet were unable to communicate with the server. Which was bogus — I could see the communication coming across the reverse proxy server. With 200 codes — although you can have a very successful HTTP call deliver an application error message. But it wasn’t like he couldn’t COMMUNICATE with the server. He turned sync off on the phones to avoid getting locked out again, and in the process of troubleshooting ended up deleting all of his accounts hosted on our Exchange 2013 server.

I looked through all of the event logs, Exchange logs … nothing interesting. In desperation, I enabled the individual user ActiveSync logging:

Set-CASMailbox mailNickName -ActiveSyncDebugLogging:$true

Had him attempt to add the mailbox profile again, and dropped the log myself:

Get-ActiveSyncDeviceStatistics -Mailbox mailNickName -GetMailboxLog:$true -NotificationEmailAddress mysmtp@mydomain.ccTLD

Bingo! An exception in the provisioning (Microsoft-Server-ActiveSync?Cmd=Provision) call — I see the phone information come across, the mobile device gets partially added to his account (no OS, phone number, carrier type information … but if you go into OWA and remove the mobile device, an Android device gets added). Error:

Command_WorkerThread_Exception :
— Exception start —
Exception type: System.IO.FileLoadException
Exception message: Could not load file or assembly ‘Microsoft.Exchange.Configuration.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Exception level: 0
Exception stack trace: at Microsoft.Exchange.AirSync.DeviceInformationSetting.ProcessSet(XmlNode setNode)
at Microsoft.Exchange.AirSync.DeviceInformationSetting.Execute()
at Microsoft.Exchange.AirSync.ProvisionCommand.Microsoft.Exchange.AirSync.IProvisionCommandHost.ProcessDeviceInformationSettings(XmlNode inboundDeviceInformationNode, XmlNode provisionResponseNode)
at Microsoft.Exchange.AirSync.ProvisionCommandPhaseOne.Process(XmlNode provisionResponseNode)
at Microsoft.Exchange.AirSync.ProvisionCommand.ExecuteCommand()
at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows…
Exception type: System.IO.FileLoadException
Exception message: The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Exception level: 1
Exception stack trace:
— Exception end —

Now that is an error I’ve never seen before. As a programmer, I know what it means … you’ve basically got some components that don’t match another. But … huh? He changed his password. Connected to the Exchange server directly (instead of remotely viewing logs & files) and saw Windows Update had dropped files and a reboot was pending. Which … some files replaced, others staged for replacement pending a reboot. *That* is some components not matching others. Rebooted our box, and voila … registration goes through, mailbox sync started.

I don’t know how many people allow auto-updates with a manual reboot on a production enterprise server (we manually patch and reboot during a scheduled maintenance window) where this could happen … but evidently Windows Update can get your Exchange server into a state where already configured clients are able to send and receive mail. But clients are unable to update passwords, and new clients cannot be configured.