Rebuilding an Exchange DAG

Recently I’ve been working with a customer where their DAG has been misbehaving.

The DAG was stretched between 2 AD sites. There were multiple issues with cluster resources in addition to the PAM (Primary Active Manager) not failing over properly

After looking at it for a few hours I decided that the best course of action would be to rebuild the DAG

The following are the steps required in order to remove the DAG and rebuild it

• Remove All Database Copies from the DAG

Remove-MailboxDatabaseCopy “DBName\ServerToRemove” –Confirm:$False

• Turn off DataCenter Activation Mode

Set-DatabaseAvailabilityGroup -Identity DAG1 -DatacenterActivationMode Off

• Remove the MailboxServers from the DAG

Remove-MailboxDatabaseAvailabilityServer DAG1 –MailboxServer ServerToRemove

Be sure to allow time for replication to occur between executing commands

• Recreate the DAG

New-DatabaseAvailabilityGroup -Name DAG1 –WitnessServer “WitnessServer” -WitnessDirectory D:\DAG1  -DatabaseAvailabilityGroupIpAddresses 192.168.100.100,192.168.101.100

• Add the servers back into the DAG

Add-DatabaseAvailabilityGroupServer -Identity DAG1 –MailboxServer “SourceMBXServer”
repeat this command for all servers that you wish to add

• Enable Database Activation Mode

Set-DatabaseAvailabilityGroup -Identity DAG1 -DatacenterActivationMode DAGOnly

• Finally Add the mailboxdatabase copy server

Add-MailboxDatabaseCopy –Identity DBName -MailboxServer MBXServerToAdd -ActivationPreference 2

• Finally we can test the Primary Active Manager Failover

Cluster.exe DAG1.DOMAIN.LOCAL Group “Cluster Group” /MoveTo:PassiveServerNode

Error 0x800706D9 deploying Exchange 2010 Client Access Servers

As part of a large scale migration from Exchange 2007 to Exchange 2010 I’ve been installing  a number of Client Access Server using the SP2 binaries of Exchange 2010

The servers displayed the error below when installing the Client Access Server role complaining about End Point mappings

 

After digging around the ExchangeSetup.log file located in the C:\ExchangeSetupLogs folder I noticed that setup was trying to configure some firewall rules

I checked the status of the Firewall and sure enough it was disabled.

All servers were deployed from a template which has the firewall disabled.

I uninstalled Exchange from the server, set the firewall to Automatic and rebooted.

After the reboot the Exchange Client Access role installed on all servers without issue.

Reseed failed DAG database copies

I have been working with a customer that has a large number of Exchange databases and wishing to to protect these with a DAG solution.

A multi node DAG had been built for the customer and one of the nodes was being problematic when it came to the initial seeding of the database.
The databases would initially seed fine, however as soon as they finished the DB’s momentarily reported “Healthy” before changing status to “FailedandSuspended”

I tried a number of fixes for this including manually copying log and catalog files, however nothing seemed to work. Then in true Microsoft fashion a reboot of the server resolved the issues and the databases seeded properly. After running an update against a DB it’s status remained “Healthy”

As previously stated the customer had a large number of DB’s that were being replicated so I resorted to the Shell to fix the remainder of the DB’s that were “FailedandSuspended”

The following command was used to bring the remainder of the DB’s back to a healthy state

Get-MailboxDatabaseCopyStatus –Server (MBXServerName) | Where-Object {$_.Status –eq “FailedandSuspended”} | Update-MailboxDatabaseCopy –DeleteExistingFiles –SourceServer (SourceMBXServerName)

Job done

Configuring Exchange 2010 Archive Mailboxes

I’ve been doing a large migration lately which has involved configuring all users with an Archive mailbox, setting the retention policy and moving items past retention into the Archive databases.

The users primary mailbox would reside on SAS disk while the archive databases were being stored on cheaper SATA disk.

The “Default Archive Policy”  of moving items older than 2 years into the Online Archive would be used

The high level steps for this were as follows.

1. Enable the users for archive
2. Setting the archive database and enabling the retention policy
3. Testing move of mail items past retention into the Archive database

As there were several thousand users on the system the only sensible way to achieve this was through the shell.

The users were already broken in 14 databases so naturally I approached this on a per database.

The following shell commands were used

Get-MailboxDatabase (DBName) | Get-Mailbox | Enable-Mailbox –Archive

Get-MailboxDatabase (DBName) | Get-Mailbox | Set-Mailbox –ArchiveDatabase (ArchiveDBName) –RetentionPolicy “Default Archive Policy”

Get-MailboxDatabase (DBName) | Get-Mailbox | Start-ManagedFolderAssistant

The Managed Folder Assistant runs nightly, however I used the above command to kick the process off immediately and verify that data was indeed moving to the new Archive DB

How to view Whitespace in Exchange 2010 Databases

With older versions of Exchange admins would use the 1221 EventID to see how much whitespace was available in each database.

With Exchange 2010 EventID 1221 is no longer used.

Admins now can query the database directly with the following PowerShell command to see the whitespace in each Database

Get-MailboxDatabase -Status | FT Name, AvailableNewMailboxSpace

Unable to expand Public Folders in Outlook on Exchange 2010

Outlook clients were unable to expand public folders with the following error being displayd

Public folders access using  OWA was fine.

After looking at the Mailbox server the RPC Client Access service was stopped

The service wouldn’t come online complaining that the port was already in use.

A reboot of the server resolved the issue

Exchange 2010 NLB and remote Subnets

While configuring an Exchange 2010 NLB I had no choice but to publish direct access to the Exchange servers from the Internet instead of using my preferred method of directing all traffic through a TMG\ISA server.

Consequently access to the published NLB wouldn’t resolve properly from external locations. All traffic on the local LAN was fine although traffic originating from outside the local subnet was dropped and no response was received by clients.

The servers were configured with dual NIC’s with a dedicated NIC on each host being assigned for NLB traffic. As the NLB NIC only has an IP and Subnet entered I suspected that the lack of default gateway to be the issue.

I changed the Firewall rules to point to an individual servers Public NIC then everything was fine although this bypassed the NLB and as such wasnt really of much use to me.

As of Windows 2008 R2 all networking uses “Strong Host Model” whereby traffic can only exit from the interface that it entered on.

A resolution for this was to allow forwarding of traffic from the NLB NIC to the public NIC via the following command.

netsh int ipv4 set int “[Name of NLB NIC]” forwarding=enabled

Recreate arbitration accounts in Exchange 2010

Following a DR situation with a customer where I had to disable the arbitration accounts in order to remove a database, I was able to sucessfully restore these accounts to their prior state once the DB was restored with the help of Martin’s blog post

Public Folders Fail to Replicate between 2007 and 2010

I have just migrated a company from a mixed 2003 & 2007 environment to 2010.

2003 server removed successfully without issue, however when trying to add the 2010 server as a public folder replica the issues began.

Lots of 1020 MSExchangeStoreDriver and 3091 Replication Errors in the event logs

Finally tracked the issue down to the server object in First Administrative Group in ADSIEDIT.

Open adsiedit and check if there any objects in

CN=Servers,CN=Fist administrative group,CN=Administrative Groups,CN=Erste Organisation,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=local. Is the container “CN=Servers”

If servers is empty, then delete it.

Full blog post available at

http://get-exchange.blogspot.com/2009/11/public-folder-mayhem-exchange-2010.html?showComment=1260569369909#c8199449617092465069

More Exchange 2010 SP1 Failures

Another issue has arisen around upgrading a mailbox server to SP1.

Error is

“$error.Clear();$vdirName = “PowerShell”;$InternalPowerShellUrl=”http://” + $RoleFqdnOrName + “/powershell”;new-PowerShellVirtualDirectory $vdirName -InternalUrl $InternalPowerShellUrl -DomainController $RoleDomainController -BasicAuthentication:$false -WindowsAuthentication:$false -RequireSSL:$false;” was run: “The virtual directory ‘PowerShell’ already exists

To resolve this issue

1. Open ADSIEdit
2. Navigate to
   Configuration – Services – Microsoft Exchange – “Domain Name” – Administrative Groups – CN=Exchange Administrative Group (FYDIBOHF23SPDLT) – Servers – “Server Name” – Protocols -HTTP
3. Delete the PowerShell vDir
4. Also delete the Powershell vDir from IIS
5. Restart server

Hopefully this will save someone else a day of wasted time.