Commvault Mailbox backup using the new pseudoclient architecture

* This process has changed significantly since the release of v11 SP12. I’ll be writing an updated blog post once I’ve had a chance to play with the revised architecture. 

In a previous post I detailed the process for implementing O365 mailbox backups using the “Classic” mailbox agent. Unfortunately support for RPC over HTTP  will be depreciated on the 31st October 2017; this makes it necessary to migrate any O365 mailbox backups to the new mailbox agent. Commvaults support site shows the following warning when viewing the documentation relating to the classic agent and O365:

“On October 31, 2017, Microsoft deprecates RPC over HTTP for Office 365. For more information, see the Microsoft support article: “RPC over HTTP deprecated in Office 365 on October 31, 2017″, https://support.microsoft.com/en-us/help/3201590/rpc-over-http-deprecated-in-office-365-on-october-31–2017.
You can no longer use Office 365 with Exchange for the Exchange Mailbox (Classic) Agent or OnePass for Exchange Mailbox (Classic).
We recommend that you transition to the Exchange Mailbox Agent. The Exchange Mailbox Agent uses EWS for archiving instead of MAPI. With EWS, the archiving throughput is increased.
For information, see Transitioning to the Exchange Mailbox Agent.”

The new agent does promise better throughput than the legacy method; however, it is worth noting that additional licensing may be required. License is consumed per mailbox as opposed to the previous capacity (or in some cases per proxy) based model. If you do not have this currently included you will get 30 days worth of evaluation licensing.

Prerequisites

The new architecture has some important differences when compared to the legacy “Classic” agent. The following prerequisites are necessary:

  • Sp9
    • This document requires features only introduced in SP9
  • Index Server
    • This can be either interactively installed or pushed from the CommCell Console. The package is Index Store.
    • This should be visible to each of the mailbox proxies allowing index data to be shared.
    • You can use only one index server per Exchange Mailbox client. You must create an index server for each Exchange Mailbox client in your CommCell environment.
      • If you are running the solution as an MSP it is possible to use 1 mailbox index for multiple clients, however this is based on the assumption that mailboxes cannot migrate between clients. More details on the additional setting required here.
    • This should ideally be a different server to your MediaAgent, however for smaller clients (500 mailboes or less) it is possible to combine the two.
    • The CommVault  recommended sizing for the Index server is significant. Based on up to 500 million messages (of approx 150KB each) the following servers specifications are advised for index server without content indexing:
      • 16 cores
      • Index storage space of 3 to 5 percent of application size
      • 48GB RAM
      • Minimum 800IOPS, recommended 1600IOPS
    • As with previous index node hardware prereqs; I would advise starting small and scaling up as required.
  • Job Results directory – This must be a network share visible to all mailbox proxies. If you have multiple proxies they are used in a round-robin order and each will require access to the Job Results directory.
  • Archiving, cleanup, and retention policies
    • Policies around mailbox retention have changed. Even if you are not using the archiving & cleanup features of CommVault, you will want to pay close attention here.
    • Retention (Primary Copy) has been moved away from the storage policy and into the retention policy. The Exchange Mailbox Agent uses the retention rules that are defined in the Exchange retention policy. The agent does not use the retention rules that are defined in the storage policy.

Configuring the Index Server

Index server setup is relatively easy. The following steps will have it ready:

  • Install the Index Store binaries using either push or interactive install.
  • From the CommCell Console, Right click Index Servers and select New Index Server.
    newindexserver
  • Give the Index Server a meaningful name (in the Cloud Name field), assign it to a storage policy (optional) and specify the index directory. Ensure the Index directory is a properly sized dedicated drive.
  • On the Roles tab, add the Exchange Index role.
  • On the Nodes tab, be sure to include your new installed index node.

Click OK and the index directories will be populated, Index server creation is complete.

Configuring the Proxy (or proxies)

This is similar to my previous post Backing up Office365 Mailboxes with CommVault however, as there are some important differences I have included the steps below.

The proxy client is used to connect to O365 and use its installed Outlook client to stream messages back to the ContentStore. Ensure the following prerequisites are available/in place before proceeding.

  • Windows service account/Office365 Account
    • This account should be synced between o365 & the local Active Directory
    • It should be a local admin account on the proxy VM
    • It should be a global administrator on o365
  • Office 2013 x64 SP1 or above + Updates
  • If you are using a hybrid (On-prem & Online) you must use a separate proxy for both. This guide is aimed at an online-only deployment.
  • dotNET framework 3.5
  • Disable UAC

Connect to Office365 and apply permissions to mailboxes. Log onto the proxy VM using the Windows service account/Office365 Account mentioned above. Using an elevated powershell prompt run the following commands:

Get-ExecutionPolicy Remotesigned
$UserCredential = Get-Credential
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection
Import-PSSession $Session

This will allow you to run powershell commands on your exchange online environment.

Apply Full access rights to all mailboxes for the service account you will be using for mailbox backup. If you are using more than one proxy, you will need more than one service account.

Get-Mailbox -ResultSize unlimited | Add-MailboxPermission -User "<service_account>" -AccessRights FullAccess -InheritanceType all -AutoMapping: $false
  • Note: These permissions are applied directly and will not affect any mailboxes added after this command is run. I would recommend a scheduled task to run the permissions application on a regular basis to ensure no mailboxes are missed.  To do this you will need a way to securely store the credential for use in the script, details here.

The following command needs to be run to allow impersonation permissions to the service accounts.

New-ManagementRoleAssignment -name:NameOfAssignment -Role:ApplicationImpersonation -User:service_account
  • NameOfAssignment = Unique name for assignment
  • ServiceAccount = Exchange Online Administrator

It is not necessary for the powershell session to remain open during the backup process. Once finished setting permissions you need to close the powershell session using the following command. This will avoid using up the sessions available to you.

Remove-PSSession $Session

You can now configure the Outlook profile:

  • Create the Outlook profile using Control Panel – Mail. Name the profile as you see fit (try to keep it simple i.e. Outlook – you will need to know this later)
    Capture
  • Ensure the profile doesn’t use cached mode. Once you have created the profile, start Outlook and when prompted for credentials, ensure the “Remember Credentials” option is selected.
  • You now need to ensure outlook uses RPC/HTTP as opposed to MAPI/HTTP to connect. Using regedit navigate to HKEY_CURRENT_USER –> Software –>  Microsoft –> Exchange. Right click Exchange and choose new DWORD.
  • Name the new DWORD MapiHttpDisabled and give it a value of 1Registry
  • Open Outlook and use Ctrl+Right click to select the Outlook icon in the taskbar. Click Connection Status.
  • OutlookTaskbarThe protocol Column in the Outlook Connection Status windows should read “RPC/HTTP”. If it doesn’t, double check the previous registry change and restart Outlook.

The Commvault Client Manager & Commvault Communications Service services need to be running under the same service account you are currently logged on with (The account with local admin and o365 privileges). Open Services.msc, adjust the Log On parameters as shown below for each service and then restart both services.

service

Setting up the new Client

From the CommCell Console, Right click the CommCell name and create a new mailbox client as shown below.

NewMailboxClient

This starts the wizard for creating a mailbox pseudoclient. Just as with VMware pseudoclients; this creates a logical client which is backed by one or more physcial proxies allowing for load balancing and scaleability.

General Tab

NewMailboxClient-pseudo

The fields required are as follows:

  • Client Name – Whatever you want
  • Storage Policy -Where will the protected data be stored
  • Index Server – As configured earlier
  • Recall Service – Used for recall of mailbox items. Web server must be accessible from the proxies.
  • Job Results – A CIFS share accessible to all proxies.

Proxies

Add one or more configured proxies here. These must have Outlook & the mailbox client installed in addition to all other customisation covered earlier.

Exchange Server/Azure AD Settings

Details of any on-prem Exchange or AzureAD subscriptions should be entered here. If you dont have On-Prem or Azure AD details it isn’t mandatory.

Service Account Settings

ServiceAccounto365

The fields required are as follows:

  • Email Address – Global Administrator Account for o365
    • For this example the account is synced with the local AD. You can view the accounts at portal.office.com in the Admin portal.
  • Username – The same account as above but in the DOMAIN\Username format
  • Password – This is the password for the above accounts.
  • Service Type – Exchange Online
  • Use Static Profile = true
  • Profile Name = profile created earlier

Configuring the Policies

The four new policy types are as follows:

  • Archiving – Archive is the new Backup. This policy dictates what messages will be protected, it has no effect on stubbing.
  • Cleanup – If you are archiving, this is where it is configured.
  • Retention – Primary Copy retention is configured here and will override any retention settings configured in the storage policy. Secondary copies will still adhere to the Storage Policy values.
  • Journal – The new compliance archive. Use this for journal mailboxes.

Policies are configured under Policies, Configuration Policies, Exchange policies as shown below:

Policies

Only configure the policies you need, for a standard mailbox backup (no archive) setup, your policies listing may look like this:

PolicyExample

Subclient Configuration

As with the previous steps, this area of the configuration has had a complete overhaul. I will be associating the mailboxes based on AD Groups, so ensuring the AD username and password is configured correctly. NOTE: If you are transitioning from an existing classic mailbox agent, you must follow the transition steps here.

From the Exchange mailbox pseudoclient; navigate to Exchange Mailbox then User Mailbox.

newmailbox

At the bottom of the right hand side of the screen, select Auto Discover Associations.

autodiscover

Right click in the white space above and choose New Association then AD Groups.

NewAss

Click Configure then Discover, accept the warning (Yes). Select the Group(s) you want included in this Association and click OK. These will be configured with the same policies (Archive, Cleanup etc). Select the Policies tab; Select a policy for each, cleanup is not necessary if not stubbing.  Remember: Retention is only concerning the primary copy, secondary copies are retained according to the storage policy config.

Once you’ve selected your policies click OK. Click Mailboxes at the bottom of the screen to see what mailboxes have been discovered.

You are now in a position to test the mailbox backups. Select a mailbox from the Mailbox tab of the the User Mailbox backup set. Right click and choose Archive. Keep in mind that all archive (backup) operations are incremental. If all goes well; feel free to schedule as you see fit.

Backing up Office365 Mailboxes with CommVault

If litigation hold isn’t enough of a retention method for your Office365 mailboxes, Commvault v11 gives you the option  of mailbox  backups using a traditional mailbox agent. At the time of writing this process has much room for improvement; it’s slow and fairly cumbersome to configure; however it does work.

SP8 also introduces the new pseudoclient style agent for mailbox backups, where multiple proxies can be placed under a single client (similar to the virtual server agent) however; as this is still early release this post focuses on the traditional mailbox agent method.

Prerequisites

Proxy Client

The proxy client is used to connect to O365 and use its installed Outlook client to stream messages back to the ContentStore.

  • Windows service account. Needs to be local admin on the proxy.
  • Office 2013 x64 SP1 or above + Updates
  • If you are using a hybrid (On-prem & Online) you must use a separate proxy for both. This guide is aimed at an online-only deployment.
  • dotNET framework 3.5
  • Disable UAC

Commvault Books Online provides the following examples when sizing how many proxies to use:

Example 1
If your Office 365 with Exchange environment includes 300 mailboxes:

  • Create 5 subclients.
  • Assign 60 mailboxes to each subclient.
  • Use one proxy server.
  • Use one Online Service account.

Example 2
If your Office 365 with Exchange environment includes 3000 mailboxes:

  • Create 30 subclients.
  • Assign 100 mailboxes to each subclient.
  • Use two proxy servers (15 subclients per proxy server).
  • Use two Online Service accounts (one per proxy server).

Office365

Office365 requires a few changes to allow mailboxes to be accessed and protected.

  • Global Administrator account with access to all mailboxes
    • Update: Make sure you limit the number of “special” characters in the password. Stick with @ and # if possible, as of SP8 some of the others cause issues.
  • Connection initiated from proxy via powershell – details here

Configuration

Office365

Proxy Client

  • Log on using the windows service account.
  • Install the OS & Outlook x64. In this example i’m using Server 2012R2 with Outlook 2013 64bit SP1. Ensure all updates are installed before continuing. For supported OS’s and Outlook editions click here, note: at present “click to run” editions of office are not supported.
  • Install .NET framework 3.5
  • Connect to Office365 and apply permissions to mailboxes. Using an elevated powershell prompt run the following commands:
Set-ExecutionPolicy Remotesigned
$UserCredential = Get-Credential
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection
Import-PSSession $Session

This will allow you to run powershell commands on your exchange online environment.

  • Apply Full access rights to all mailboxes for the service account you will be using for mailbox backup. If you are using more than one proxy, you will need more than one service account.
Get-Mailbox -ResultSize unlimited | Add-MailboxPermission -User "<service_account>" -AccessRights FullAccess -InheritanceType all -AutoMapping: $false
  • If you want to apply permissions to a specific mailbox you can use:
Add-MailboxPermission -Identity "<mailbox_name>" -User "<service_account>" -AccessRights FullAccess -InheritanceType all -AutoMapping:$false

It is not necessary for the powershell session to remain open during the backup process.  Once finished setting permissions you need to close the powershell session using the following command. This will avoid using up the sessions available to you.

Remove-PSSession $Session
  • Create the outlook profile using Control Panel – Mail. Name the profile the same as the O365 account alias (before the @). Profile
  • Ensure the profile doesn’t use cached mode. Once you have created the profile, start Outlook and when prompted for credentials, ensure the “Remember Credentials” option is selected.
  • You now need to ensure outlook uses RPC/HTTP as opposed to MAPI/HTTP to connect. Using regedit navigate to HKEY_CURRENT_USER –> Software –>  Microsoft –> Exchange. Right click Exchange and choose new DWORD.
  • Name the new DWORD MapiHttpDisabled and give it a value of 1Registry
  • Open outlook and use Ctrl+Right click to select the Outlook icon in the taskbar. Click Connection Status.

OutlookTaskbar

  • The protocol Column in the Outlook Connection Status windows should read “RPC/HTTP”. If it doesn’t, double check the previous registry change and restart Outlook.

CommVault Mailbox Agent

The CommVault mailbox agent (traditional not archiver) can be pushed from the central CommServe. Before doing so, you’ll need to add the “bEnableExchangeOnline” additional setting to the ComCell (top level) as shown below:

  • Right click on the CommCell name in the ComCell browser windows and choose properties.
  • On the additional settings tab; add the bEnableExchangeOnline key with a value of true

bEnableExchangeOnline

Once that’s done you’re ready to deploy the mailbox agent to your proxy VM (Same place that you installed outlook. To deploy follow the instructions below:

  • Choose Tools from the CommCell ribbon and Select Add/Remove software.
  • Use the wizard to deploy the Exchange Mailbox software to the proxy, selecting defaults along the way. When prompted for exchange details, ensure the entries are blank & simply click Next.
  • You can monitor the installation progress from the job monitor.

Once the software has been installed; you can configure the proxy client as follows. Do this from the Commvault console, you may have to press F5 to refresh the clients view.

  •  Right Click the Exchange mailbox (Classic) agent under the proxy client and choose properties

MbxProperties

  • Enter the Exchange Administrator account using the “Change Account” button higher up the page. This is the service account you used to log onto the proxy and configure the Outlook profile.
  • Select “Exchange Online”, click OK on the warning.
  • Enter the profile name used on the proxy.
  • Click the lower “Change Account” button. Enter the full SMTP address and password for the exchange online service account.
  • Click OK to return to the main CommServe screen. Right click on the proxy client and choose properties.
  • Click “Advanced” and select the “Additional Settings” tab. Add the following Keys:
    • nMailboxesperSession = 1
    • nRestartMAPIOnNetworkError = 1
    • nSkipUserImpersonation = 1
    • nExchangeOnlineOnly = 1

ProxyRegKeys

Disable User Impersonation

Now that the mailbox client & agent are configured, the logon user for the CommVault services on the proxy need to be adjusted.

  • Log onto the proxy VM. Open services.msc from the Run command.
  • Adjust the Log On As properties for the following services to reflect the windows service account.
    • Commvault Communications Service
    • Commvault Client Event Manager
    • Commvault Communications Service
  • If prompted to allow Log On As rights, click OK. Restart the services once complete.

Services

Add Content

Content is added via subclients in the same way you would many other agents. As descrived in the prerequisites section, in order to get the best throughput from your mauilbox backups, it would be wise to split the mailboxes into multiple subclients. In this example I have split the content into 4 subclients, in a real life scenario I would recommend considerably more.

Subclients

The subclients are split by first letter resulting in any mailboxes with the alias starting with A-F being picked up by the one subclient, any with their alias starting with H-K selected by another etc. This is achieved by enabling “Auto Discover” on the defaultBackupSet properties as shown below.

Auto Discover

Once thats done you can use wildcards (more detail here) to specify the content. Wildcards used for my H-K subclient are shown below:

Subclient Wildcard Content

Run Backup

Now the content is configured you can run the first backup & schedule accordingly. As stated previously if you have a lot of mailboxes this will take a while. Make sure you have sufficient subclients & proxies to suit the size of your environment. If you have multiple proxies, that means multiple accounts need to be created in O365. SP8 brings new functionality to mailbox backup which will be discussed in a later post, some simplification of the process would be a very good thing!