The new SBS 2011 was deployed using the free edition of VMware vSphere 5 Hypervisor (ESXi) so I was keen to try out the latest version of Veeam Backup and Recovery v6 with this.
I download Veeam Backup & Recovery v6 and set it up using my trial Key.

The product itself is a breeze to get installed and setup but I immediately ran into issues when trying to run my first backup.
The backup job would hang for around 15 – 20mins before throwing the following error

Failed to prepare guest for hot backup. Error: VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec
Error: VSSControl: Failed to prepare guest for freeze, wait timeout 900 se

After a bit of digging I stumbled across an SBS post relating to SharePoint Foundation 2010 SP1 and the need to manually complete the SP1 process.
Failure to complete the process can prevent all backups from working which is what happened in my case.

See the following article for full details on how to resolve the issue.

http://blogs.technet.com/b/sbs/archive/2011/07/06/potential-issues-after-installing-sharepoint-foundation-2010-sp1.aspx

As part of a project that I have been working on, I had a need to replace the default certificates that get installed during the installation of vCenter and vSphere 4.1 with OpenSSL certificates.

This guide is based on vSphere & vCenter 4.1 although it should also work for vSphere 5

The process will be broken into a number of steps

1. Installation of OpenSSL & creating the working directory structure
2. Creating a Root CA
3. Creating the CSRs (Certificate Signing Requests) for vSphere and vCenter
4. Signing the CSRs using the Root CA
5. Assign certificates to vCenter & vSphere
6. Deploying the OpenSSL RootCA via GPO

I want to make this process as easy as possible and as such I’ll be using the default values that are already predefined in the openssl.cfg file

So lets get started with replacing vSphere certificates.

Installing OpenSSL & creating the directory structure

Download OpenSSL for Windows x64 from
http://www.slproweb.com/products/Win32OpenSSL.html

Accept the license agreement and use the defaults when installing.

OpenSSL has now been installed to C:\OpenSSL

The next thing that needs to be done is to create the working directory structure. The directory structure will be used when signing the certificates using the RootCA.

Use the following commands

mkdir C:\openssl\bin\demoCA\newcerts
mkdir C:\OpenSSL\bin\demoCA\private
Copy C:\OpenSSL\bin\PEM\demoCA\serial C:\OpenSSL\bin\demoCA\
copy con C:\OpenSSL\bin\index.txt
(after issuing the above command use CTRL-Z then enter to finish the command. This will create a blank document called index.txt)

Creating the Root CA

We are now ready to create the RootCA.
Open a command prompt at C:\OpenSSL\bin and issue the below command

openssl req -new -x509 -extensions v3_ca -keyout rootca.key -out rootca.crt -days 3650 -config openssl.cfg

The first thing that you will be asked for is a “Enter PEM pass phrase:”

This is a pass phrase of your own choosing (Should be a strong pass phrase) and will be used again when signing the certificates for vSphere & vCenter

Complete the remainder of the fields that you are asked information about.
The relevant fields have been highlighted in the below graphic.

Once the command has complete you will find 2 new files named rootca.crt and rootca.key within C:\OpenSSL\bin

Creating the CSR’s

Next we need to create the CSRs for the vsphere and vcenter certificates

Again open a command prompt from C:\OpenSSL\bin and issue the following commands

openssl req -new -nodes -out vsphere.csr -keyout vsphere.key -config openssl.cfg

openssl req -new -nodes -out vcenter.csr -keyout vcenter.key -config openssl.cfg

The above commands will create the CSRs, each CSR will consist of 2 files, the csr file and the key file.

At the end of the request you will be prompted to enter ‘extra attributes’. Leave these 2 options blank.

The yellow highlight below are the options that you need to fill in, the green are the optional setting that are to be left blank.

We now have the private key and the CSR for both vsphere and vcenter within C:\OpenSSL\bin.

Signing CSR

The next step is issue certificates based on the the CSRs using the RootCA.

Remember back to that start when we created the directory structure? Well this is where it comes into use.

Issue the below commands at a command prompt from C:\OpenSSL\bin.
The first thing that you will be asked for is the pass phrase from the rootca we entered when creating the initial rootca files

openssl ca -cert rootca.crt -keyfile rootca.key -out vsphere.crt -config openssl.cfg -infiles vsphere.csr

openssl ca -cert rootca.crt -keyfile rootca.key -out vcenter.crt -config openssl.cfg -infiles vcenter.csr

Going back to C:\OpenSSL\bin we see 2 new files, vsphere.crt & vcenter.crt.
Double clicking these files will show that they have been signed by the Root CA.

Creating PFX

A PFX file is required for vcenter, a PFX file is the amalgamation of the certificate and its associated private key

The following command will create the PFX for vcenter

openssl pkcs12 –export -in vcenter.crt -inkey vcenter.key -name vcenter  -passout pass:testpassword -out vcenter.pfx

Again looking in C:\OpenSSL\bin you will see the newly created PFX file.

Replacing Certificates on vCenter

We now need to replace the default certificate that are installed on vcenter with are newly created certs.
The SSL certificates for vCenter are located at

C:\Users\All Users\VMware\VMware VirtualCenter\SSL

Opening this folder you will find 3 files

• rui.crt
• rui.key
• rui.pfx

Copy these certificates to a safe location as you will need to replace these if anything has gone wrong with the newly created certificates.

Copy across the vcenter.crt, vcenter,key and vcenter.pfx files created earlier and rename these to reflect the original rui files

On the vCenter Server, restart the service VMware VirtualCenter Management Webservices

Installing Root CA on Windows

Now that we have out certificate signed by our Root CA we need to ensure that the Root CA is trusted by servers and clients that will be connecting to both vsphere & vcenter.

The easiest way to do is this is via a GPO

Create a new GPO and give it a meaning full name

Navigate to
Computer Configuration – Policies – Windows Settings – Security Settings – Public Key Policies – Trusted Root Certificate Authority
and choose Import

Import the rootCA.crt certificate

You can now close Group Policy.

Uploading Certificates to vSphere

The part went really smoothly for me due to a PowerShell script by Martijn Baecke that’s available on the VMware communities website.

See the post at http://communities.vmware.com/docs/DOC-14655 for full instructions on uploading the certs to vSphere.

Testing certificates from Clients

Finally we have everything in place and we are now ready to test the certificates for trust from out clients.

First thing to do is to issue a group policy update from the client. This will install the Root CA into the clients Trusted Root Certificate Authorities container

Once the update has been issues we can check that the certificate has been installed.

1. Click Start, click Run, type mmc, and then click OK.
2. In the File menu, click Add/Remove Snap-in.
3. In the Add/Remove Snap-in box, click Add.
4. In the Available Standalone Snap-ins list, click Certificates, and then click Add.
5. Click Computer Account, and then click Next.
6. Click the Local computer (the computer this console is running on) option, and then click Finish.
7. Click Close, and then click OK.

Expand Certificates – Trusted Root Certificate Authorities – Certificates and you should now see the RootCA installed

Finally open Internet Explorer and point the browser to your vCenter URL. If all has gone correctly the page will display without the certifcate security warning.

My next post will cover using a Windows CA to sign the CSR

Everytime I try to add a VIP via the web interface I get the following error

“The cmdb add entry failed”

The firewall still allows the rules to be added via the CLI but the web interface is a no go.

The solution for me was to reboot the FortiGate

One of the first things I wanted to do was fix the certificate error by replacing the default certificate with a one from my internal PKI.

The only way I can see to do this is via the shell. Shell access is disabled by default so the first thing to do was enable Shell access.

To enable shell access I went to the Configuration tab on the host and drilled down to Software – Security Profiles and enabled the ESXi Shell to Start and stop with host

I now have shell access enabled but I now also have a warning message on the host complaining about configuration issues and that ESXi Shell access has been enabled.

To disable the warning its simply a matter of going back to the Configuration tab on the host then – Software – Advanced Settings – uservars and changing the UserVars.SupressShellWarning to 1

A multi node DAG had been built for the customer and one of the nodes was being problematic when it came to the initial seeding of the database.
The databases would initially seed fine, however as soon as they finished the DB’s momentarily reported “Healthy” before changing status to “FailedandSuspended”

I tried a number of fixes for this including manually copying log and catalog files, however nothing seemed to work. Then in true Microsoft fashion a reboot of the server resolved the issues and the databases seeded properly. After running an update against a DB it’s status remained “Healthy”

As previously stated the customer had a large number of DB’s that were being replicated so I resorted to the Shell to fix the remainder of the DB’s that were “FailedandSuspended”

The following command was used to bring the remainder of the DB’s back to a healthy state

Get-MailboxDatabaseCopyStatus –Server (MBXServerName) | Where-Object {$_.Status –eq “FailedandSuspended”} | Update-MailboxDatabaseCopy –DeleteExistingFiles –SourceServer (SourceMBXServerName)

Job done

The users primary mailbox would reside on SAS disk while the archive databases were being stored on cheaper SATA disk.

The “Default Archive Policy”  of moving items older than 2 years into the Online Archive would be used

The high level steps for this were as follows.

1. Enable the users for archive
2. Setting the archive database and enabling the retention policy
3. Testing move of mail items past retention into the Archive database

As there were several thousand users on the system the only sensible way to achieve this was through the shell.

The users were already broken in 14 databases so naturally I approached this on a per database.

The following shell commands were used

Get-MailboxDatabase (DBName) | Get-Mailbox | Enable-Mailbox –Archive

Get-MailboxDatabase (DBName) | Get-Mailbox | Set-Mailbox –ArchiveDatabase (ArchiveDBName) –RetentionPolicy “Default Archive Policy”

Get-MailboxDatabase (DBName) | Get-Mailbox | Start-ManagedFolderAssistant

The Managed Folder Assistant runs nightly, however I used the above command to kick the process off immediately and verify that data was indeed moving to the new Archive DB

With Exchange 2010 EventID 1221 is no longer used.

Admins now can query the database directly with the following PowerShell command to see the whitespace in each Database

Get-MailboxDatabase -Status | FT Name, AvailableNewMailboxSpace

The customer was running a virtual infrastructure and all their VMs were hosted on the SAN, naturally this wasn’t a good situation to be in.

After looking at the SAN it was apparent that the primary datamover had failed over resulting in loss of access to all the vSphere hosts.

Luckily this was a fairly straight forward situation to resolve.

The resolution was thus.

1. SSH to control station
2. “nas_server -list” to check whether there is a datamover failover.
3. “/nas/sbin/getreason” to check the status of the datamovers. The 2 datamovers should be both in “contacted” status.
4. “server_standby server_2 –restore mover” to failback the datamover.
5. Run “/nas/sbin/getreason” several minutes later to make sure both datamovers are in contacted status.

I ran Tree Size and noticed that the MSOCache for Office 2010 was 1.2GB.
This is a hidden folder located in C:\MSOCache

I wanted to move this to my D: drive to free up the valuable space from my SSD but after some reading I found it is not possible to move the location of this using office install.

This is where Junction came to my resuce

Junction is a free utility from sysinternals and is availble from http://technet.microsoft.com/en-us/sysinternals/bb896768

Firstly I move the MSOCache using RoboCopy to my D: drive using the follwing command

Robocopy C:\MSOCache D:\MSOCache /MIR

Be sure to run the above as an elevated command prompt.

Using the /MIR switch mirrors the folder contents, security & attributes so the folder remains hidden on the D: drive

After Robocopy ran I deleted the MSOCache folder from the C drive.

The last piece was to create a junction point that links C:\MSOCache to D:\MSOCache and gives the 1.2GB back to my SSD

I downloaded junction and copied the contents to %windir%\system32 and then ran the following

Junction C:\MSOCache D:\MSOCache

Job done

Public folders access using  OWA was fine.

After looking at the Mailbox server the RPC Client Access service was stopped

The service wouldn’t come online complaining that the port was already in use.

A reboot of the server resolved the issue