Tag: Exchange

Exchange Disaster Recovery

I guess with everyone moving to magic cloudy pay-per-month Exchange, this isn’t such a concern anymore … but for those still running on-premise Exchange:

(1) Before you can restore your AD system state, you’ve got to build a server & bring up a temporary domain. There’s a “System Configuration” program that lets you select to restart in safe mode / directory services restore mode without having to time the F8 key or anything.

(2) The system state backup of a domain controller backs up a lot of stuff — including the registry which tells the server what software is installed and services. This means it is not possible to just run the Exchange setup.exe with the disaster recovery option. Fortunately, I was able to copy the Exchange folder from program files off of a backup. Unfortunately, the Exchange services wouldn’t start because DLLs couldn’t register. Did a diff between old server backup & new one — copied any missing stuff from c:\windows\system32 and c:\windows\syswow64 and, voila, Exchange is starting. Couldn’t mount the ebd file, though …

(3) Which brings me to eseutil an attempt to replay the transaction logs (eseutil /r) and then repair the database as much as possible (eseutil /p) got me an EDB file that the Exchange server could mount.

Why doesn’t everyone do this — non-working hours clarification

I like that Microsoft has added “they are x hours behind you” to individual profiles, but that assumes people all work 8-5 in their local time. Which isn’t the case, so I’ve been introducing myself to new people that I need to engage in meetings including something like “I work in the Eastern time zone but am generally available until about 6PM Eastern if that’s better for you” & asking for a similar response from them. I know some people who live in the Central, Mountain, or Pacific time zones but work 8-5 Eastern. I know others who live in the same area work 9-6 or 11-8 Eastern. We have overseas contractors who work from 3:30 AM to 12:30 PM Eastern, and others who who start working around 10 AM.

Seems like it would make collaborating with others easier if we all had recurring appointments to clarify our non-working hours. A recurring each-weekday appointment like below — away so it doesn’t look like I’m just booked solid at dark-o-clock, recurring, and no reminder (because that would get super annoying). And maybe a recurring weekly one from whatever PM on Friday through whatever AM on Monday if there are a statistically significant of people who’d be working T-Sat or Sun-Thur.

Doesn’t really provide much value implemented in a small group – you generally get a good idea of when your immediate coworkers are working. But it would help a lot reaching out to other groups!

 

Android Outlook Message Sending Failure

I’ve been getting a strange error when trying to send pictures within e-mail messages from my Android phone. I say a strange error because there’s literally one entry that comes back when you search for MessageDeliveryFailedException f5f0 — and no pointer at all as to what might have gone wrong. Just a non-delivery report popping into the Inbox on my phone:

     Technical details
     MessageDeliveryFailedException: Could not deliver the message [len=70, data=50005…C090005] sent at 8/25/2020 10:06:28 PM.Failure code: f5f0

I’ve got a reverse proxy with an application firewall and suspected that was the source of my problems. Mostly because errors caused within the Microsoft Exchange system are generally easy to find online. An oddball error is going to come from an oddball source. And I was right — my application proxy log shows an error each time I attempt to send one of the failed messages.

Edited /etc/httpd/conf.d/mod_security.conf and upped the SecRequestBodyNoFilesLimit. Once Apache HTTPD was restarted, I was able to send my messages without problem.

Create a New Exchange Account with Powershell

We use unique e-mail addresses every time we give out our address. Because that occasionally means I need to reply to someone using that address, I set up a quick web form to send e-mail from any arbitrary address in my domain. Every now and again, though … we need to actually reply to a message (attach pictures, get some really specific formatting without wasting a lot of time coming up with the appropriate HTML, etc).

If the Exchange powershell snap-in is not loaded (the new-mailbox command is not found), load the snap-in:

Add-PSSnapIn -Name Microsoft.Exchange.Management.PowerShell.SnapIn

Then create a new Exchange account and granting full access to the person who wants to send mail from that address.

New-Mailbox -Name "NewAccount" -Alias NewAccount -LastName "NewAccount" -UserPrincipalName NewAccount@example.com -Password (ConvertTo-SecureString -String 'haW29oihOI#192QHe983QHR9' -AsPlainText -Force)
Add-MailboxPermission -Identity "NewAccount@example.com" -User "me@example.com" -AccessRights FullAccess -InheritanceType All

Or adding a new primary SMTP address to an existing account:

set-mailbox -Identity NewAccount@example.com -PrimarySmtpAddress "ANewAddress@example.com" -EmailAddressPolicyEnabled $false

When adding a new address, the existing primary SMTP address becomes a secondary proxy address.

Did you know … you can prevent meetings from being forwarded?

Have you ever had an attendee forward a meeting that was supposed to be confidential? Microsoft Exchange will notify you when a meeting attendee has forwarded your meeting; unless you are really close on that time machine project, what’s done is done. Unless … did you know that you can prevent the meeting from being forwarded? 

* The forwarding restriction is enforced on the mail client, so attendees outside the company may still be able to forward the meeting request. Additionally, there are ways to circumvent this forwarding restriction – e.g. meeting content can still be copied and pasted into a new appointment item. While restricting forwarding is a way to convey the confidentiality of the meeting and deter casual forwarding, this doesn’t guarantee eyes-only security.

How do I do it?

Right now, you can only restrict meeting forwarding when using the Outlook client on Windows or the Web – Mac, iOS, and Android client users will need to use the Web client.  

Outlook for Windows

This feature has not been deployed to all of the Office 365 channels as of this writing. The screen-shots below were created using an Office 365 installation with the monthly update channel. The semi-annual channel is slated to be updated in March 2019, so use Outlook Web until then!

Create a new meeting:

On the ribbon bar, select “Meeting”. You can restrict forwarding under the “Response Options” button.

Outlook Web

Create a new meeting:

Once you have added an attendee, a gear icon will be displayed above the attendee list.

Click the gear icon – by default, meetings can be forwarded. You can click “Allow forwarding” to prevent the meeting from being forwarded to others.

What does the recipient see?

Exchange Online recipients using Outlook Web will see a banner indicating that forwarding is disabled. The forward option will be grayed out.

Exchange online recipients using Outlook with the Monthly update channel will see the banner as well. Those will the semi-annual update channel will not see any indication that they cannot forward the invitation … in fact, their client will seemingly let them forward the meeting. But Exchange Online will refuse the message and they will get a non-delivery report indicating that the meeting could not be forwarded.

Recipients outside of Exchange online not notice any change — Gmail, for example, happily allows me to forward the meeting request.

Reinitializing The Exchange Content Index Database

When you search your inbox by copying a word from a message subject and searching for it by subject … but get nothing back, that’s a good indication that the content index database has gone corrupt. With Exchange 2013, you can manually reinitialize that database as follows:

Stop-Service MSExchangeFastSearch
Stop-Service HostControllerService

rename “C:\program files\microsoft\Exchange Server\V15\Mailbox\Mailbox Database 1440585757\1CDB1E55-A129-46BC-84EF-2DDAE27B808C12.1.Single” “c:\program files\microsoft\Exchange Server\V15\Mailbox\Mailbox Database 1440585757\1CDB1E55-A129-46BC-84EF-2DDAE27B808C12.1.Single.bad”

Start-Service MSExchangeFastSearch
Start-Service HostControllerService

# Wait a bit for the content indexing process to start
Get-MailboxDatabaseCopyStatus | FL Name,*Index*

ContentIndexState of “Crawling” means it’s still working on it. Healthy means it’s done.

Unable to ‘send as’ from Outlook With Exchange

I’ve had a confounding problem — we have sendmail magicing up millions of e-mail addresses for us, but occasionally we need to be able to send from one of these addresses too. I’ve got a web form that allows text-based messages (html or plain text), but I don’t want to figure out how to upload and attach images via a web form. Until I get around to updating the web form, I just set up a new Exchange mailbox and grant myself full access (which includes send as permission)

Except ever since we got always updating Office 2016, I’ve gotten nondelivery reports when I subsequently try to send from this new mailbox that claim I don’t have permission to send as the user in question. And I’ve verified my access three times. Even added explicit send-as in addition to full mailbox access.

I’ve finally discovered why I get this false error. The ‘from’ in Outlook allows free-form text which then may or may not resolve against an Exchange mailbox. And based on the permissions of the mailbox (or the lack of permissions of the non-resolved mailbox), it may or may not work. So I don’t have permission to send from newmailbox@ourdomain.ccTLD, I do have permission to send from the Exchange mailbox that happens to have that as its primary SMTP address. Sigh!.

When you use offline mode / cached Exchange mode, and an offline address list, the SMTP address doesn’t resolve out to that mailbox. And Exchange quite properly reports an error. To get the whole thing to work (assuming “wait until tomorrow” isn’t a good answer):

First, the offline address needs to be updated (either wait or hit the powershell management console on the server)

Update-OfflineAddressBook -Identity "Default Offline Address Book"

Secondly, Outlook needs to retrieve the updated address book. Within the Outlook client, use send/receive to update the address lists. Then you can send as the mailbox to which you have perfectly configured access.

Exchange 2013 Calendar Events In OpenHAB (CalDAV)

We’ve wanted to get our Exchange calendar events into OpenHAB — instead of trying to create a rule to determine preschool is in session, the repeating calendar event will dictate if it is a break or school day. Move the gymnastics session to a new day, and the audio reminder moves itself. Problem is, Microsoft stopped supporting CalDAV.

Scott found DAVMail — essentially a proxy that can translate between CalDAV clients and the EWS WSDL. Installation was straight-forward (click ‘next’ a few times). Configuration — for Exchange 2013, you need to select the “EWS” Exchange protocol and use your server’s EWS WSDL URL. https://yourhost.domain.cTLD/ews/exchange.asmx … then enable a local CalDAV port.

On the ‘network’ tab, check the box to allow remote connections. You *can* put the thumbprint of the IIS web site server certificate for your Exchange server into the “server certificate hash” field or you can leave it blank. On the first connection through DAVMail, there will be a pop-up asking you to verify and accept the certificate.

On the ‘encryption’ tab, you can configure a private keystore to allow the client to communicate over SSL. I used a PKCS12 store (Windows type), but a java keystore should work too (you may need to add the key signing key {a.k.a. CA public key} to the ca truststore for your java instance).

On the advanced tab, I did not enable Kerberos because the OpenHAB CalDAV binding passes credentials. I did enable KeepAlive – not sure if it is used, the CalDAV binding seems to poll. Save changes and open up the DAVMail log viewer to verify traffic is coming through.

Then comes Scott’s part — enable the bindings in OpenHAB (there are two of them – a CalDAVIO and CalDAVCmd). In the caldavio.cfg, the config lines need to be prefixed with ‘caldavio’ even though that’s not how it works in OpenHAB2.

caldavio:CalendarIdentifier:url=https://yourhost.yourdomain.gTLD:1080/users/mailbox@yourdomain.gTLD/calendar
caldavio:CalendarIdentifier:username=mailbox@yourdomain.gTLD
caldavio:CalendarIdentifier:password=PasswordForThatMailbox
caldavio:CalendarIdentifier:reloadInterval=5
caldavio:CalendarIdentifier:disableCertificateVerification=true

Then in the caldavCommand.cfg file, you just need to tell it to load that calendar identifier:

caldavCommand:readCalendars=CalendarIdentifier

We have needed stop openhab, delete the config file from ./config/org/openhab/ related to this calendar and binding before config changes are ingested.

Last step is making a calendar item that can do stuff. In the big text box that’s where a message body is located (no idea what that’s called on a calendar entry):

BEGIN:Item_Name:STATE
END:Item_Name:STATE

The subject can be whatever you want. The start time and end time are the times for the begin and end events. Voila!

Really Wacky Exchange (ActiveSync) Error

My husband changed his Active Directory password. Routine enough – we’ve got 15k accounts at the office and require a password change every 90 days. That’s 150-200 people changing their password every day. They get themselves locked out a lot (mobile devices, cached workstation credentials, and a host of other unique places people manage to store their creds), but it’s trivial to unlock an individual user.

*Except* — after the account was unlocked, his Windows 10 mail client updated properly and was interacting with the Exchange server. Android, however, still wouldn’t accept his new password. If he typed the wrong thing, it would say invalid password. But whenever he typed the right thing, he got an error indicating the phone and tablet were unable to communicate with the server. Which was bogus — I could see the communication coming across the reverse proxy server. With 200 codes — although you can have a very successful HTTP call deliver an application error message. But it wasn’t like he couldn’t COMMUNICATE with the server. He turned sync off on the phones to avoid getting locked out again, and in the process of troubleshooting ended up deleting all of his accounts hosted on our Exchange 2013 server.

I looked through all of the event logs, Exchange logs … nothing interesting. In desperation, I enabled the individual user ActiveSync logging:

Set-CASMailbox mailNickName -ActiveSyncDebugLogging:$true

Had him attempt to add the mailbox profile again, and dropped the log myself:

Get-ActiveSyncDeviceStatistics -Mailbox mailNickName -GetMailboxLog:$true -NotificationEmailAddress mysmtp@mydomain.ccTLD

Bingo! An exception in the provisioning (Microsoft-Server-ActiveSync?Cmd=Provision) call — I see the phone information come across, the mobile device gets partially added to his account (no OS, phone number, carrier type information … but if you go into OWA and remove the mobile device, an Android device gets added). Error:

Command_WorkerThread_Exception :
— Exception start —
Exception type: System.IO.FileLoadException
Exception message: Could not load file or assembly ‘Microsoft.Exchange.Configuration.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Exception level: 0
Exception stack trace: at Microsoft.Exchange.AirSync.DeviceInformationSetting.ProcessSet(XmlNode setNode)
at Microsoft.Exchange.AirSync.DeviceInformationSetting.Execute()
at Microsoft.Exchange.AirSync.ProvisionCommand.Microsoft.Exchange.AirSync.IProvisionCommandHost.ProcessDeviceInformationSettings(XmlNode inboundDeviceInformationNode, XmlNode provisionResponseNode)
at Microsoft.Exchange.AirSync.ProvisionCommandPhaseOne.Process(XmlNode provisionResponseNode)
at Microsoft.Exchange.AirSync.ProvisionCommand.ExecuteCommand()
at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows…
Exception type: System.IO.FileLoadException
Exception message: The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Exception level: 1
Exception stack trace:
— Exception end —

Now that is an error I’ve never seen before. As a programmer, I know what it means … you’ve basically got some components that don’t match another. But … huh? He changed his password. Connected to the Exchange server directly (instead of remotely viewing logs & files) and saw Windows Update had dropped files and a reboot was pending. Which … some files replaced, others staged for replacement pending a reboot. *That* is some components not matching others. Rebooted our box, and voila … registration goes through, mailbox sync started.

I don’t know how many people allow auto-updates with a manual reboot on a production enterprise server (we manually patch and reboot during a scheduled maintenance window) where this could happen … but evidently Windows Update can get your Exchange server into a state where already configured clients are able to send and receive mail. But clients are unable to update passwords, and new clients cannot be configured.