Microsoft Exchange Server 2013 : How the Managed Folder Assistant implements retention policies

After you apply a retention policy to a mailbox, you can either wait for the next scheduled run of the MFA or start it manually so that the new policy is applied immediately. When the MFA runs, it performs the following tasks:

  • It applies the tags specified in retention policies to the mailboxes covered by these policies and stamps the items in the various folders covered by the policies with the appropriate tag name and expiration date.
  • If a policy defines a retention or expiry period for items, it stamps a Messaging Application Programming Interface (MAPI) property (ElcMoveDate) on the items, indicating the date and time from which the retention period will start. A future run of the MFA can then use this date and time to calculate when to delete an item or mark it as expired.
  • It locates items in folders that are past their expiration date and takes whatever action is defined in the policy (delete, age out, move to another folder).

The MFA works on a work cycle basis: it assesses the expected workload in terms of the number of mailboxes it has to process and then spreads out its processing across the complete window. For example, if 600 mailboxes are to be processed over three hours, the MFA creates its own internal schedule to process 200 mailboxes per hour or roughly three mailboxes per minute. In addition, a checkpoint is defined for the work cycle, at which time the MFA looks for new mailboxes that should be added to its list for processing. The default values for the work cycle and checkpoint are both 1 day, so the MFA attempts to process every mailbox in its list daily and checks for new mailboxes daily. The MFA logs event 9017 when it begins to process mailboxes, but the more important event to look for is 9025, logged when the MFA has had to skip a mailbox for some reason, such as when the mailbox is being moved to another database.

Overall, the work cycle mechanism makes more effective use of server resources in an easy and relaxed manner throughout the day and doesn’t create potential spikes in demand.

You might want to run the MFA immediately, perhaps to apply a policy to a group of users for the first time. Forcing immediate execution for a selected mailbox is useful when you start to apply policies to mailboxes and want to gauge the effect of the policy by examining the contents of a known mailbox. This might be easier than asking users what happened to items in their mailboxes (especially if you’ve made a mistake with the policy and just removed half the items from the mailbox). To force processing for a selected mailbox, specify its name with the –Identity parameter:

Start-ManagedFolderAssistant –Identity 'Akers, Kim'

To process a group of mailboxes, either provide a set of mailbox identifiers as input or use the Get-Mailbox cmdlet with a filter to retrieve a set of mailboxes and pipe it as input to Start-ManagedFolderAssistant. In the first example, two mailbox identifiers are provided as input. In the second, you process all the mailboxes in a database. In the third, you use a filter to find all the mailboxes from a particular office.

"Redmond, Tony", "Akers, Kim" | Start-ManagedFolderAssistant
Get-Mailbox –Database 'VIP Data' | Start-ManagedFolderAssistant
Get-Mailbox –Filter {Office –eq 'Dublin'} | Start-ManagedFolderAssistant

The time required for the MFA to complete its run depends on the number of mailboxes and the number of items to which it has to apply retention policies. A run on a small server that hosts a few hundred mailboxes will complete in a couple of minutes unless the mailboxes hold thousands of items. However, processing 7,000 mailboxes, each of which holds an average of 20,000 items, could take several hours, especially if the server is loaded with other tasks or the policies cause a heavy I/O load because many items are permanently removed or moved from primary to archive mailboxes. You should monitor the first runs of the MFA on a server to gauge the scope of the activity and how long a normal run takes to complete. Equipped with this information, you can quickly assess whether future runs are progressing as expected.

After the MFA has applied a new policy to a mailbox, the next time the user connects to the mailbox with a client that supports retention policies, she will see that retention tags are shown on items and the retention policy options are visible. Another important point that you should understand is that if you apply a retention policy that contains a default policy tag, the MFA stamps the default tag on every item in the mailbox. This action forces Outlook to download the complete contents of the mailbox the next time the client connects and synchronizes with Exchange. Such a massive synchronization has the potential to flood a network and keep clients fully occupied for a long time. Including a default archive tag in a policy does not have the same effect because the MFA does not stamp every item with this tag.

Behind the scenes with the MFA

When the MFA first processes a mailbox, it creates a hidden FAI item of type IPM.Configuration.MRM in the Associated Contents table of the Inbox folder. The item stores MRM configuration in XML format in a PR_ROAMING_XMLSTREAM property. (PR = property; Roaming means that the information is available wherever the client connects, and XMLSTREAM indicates the type.) The MFA then updates the item any time a change occurs in the retention policy assigned to the mailbox, such as when a new personal tag is added to the policy. If a new retention policy is assigned, the MFA updates the item with details of that policy.

Exchange uses FAIs as a means to hold configuration and other data it needs to store in a mailbox but does not want to reveal to users when they run clients such as Outlook. The item the MFA creates holds details of the retention policy that has been assigned to the mailbox, including details of the retention tags the client needs to display in its user interface. If the item does not exist in the mailbox, a client remains unaware that a retention policy is in force. This is why the MFA has to process a mailbox before the client user interface is populated with details of the policy. In addition to the tags provisioned through the retention policy, the item also holds information about any personal tags the user has selected for his own use through Outlook Web App Options . As noted earlier, you cannot see this information with a normal client, but you can with the MFCMAPI utility by opening a mailbox that has been assigned a retention policy, opening the Inbox folder, and then opening the associated contents table before finally identifying the MRM item in the set held in the table. Figure 1 shows the details of the MRM configuration item as exposed by MFCMAPI, and you can clearly see details of the retention tags specified in the policy in the text box.

Figure 1. Viewing the MRM configuration item with MFCMAPI

The MFA uses a number of MAPI properties for mailbox folders and items during its processing. These properties are:

  • PR_START_DATE_ETC. For a folder, this property holds the GUID of the retention tag that governs a folder. Every tag is assigned a GUID (stored logically in the GUID property and visible with EMS). This GUID is stamped into this property so the MFA knows which retention tag, action, and period applies to the folder. For an item, the property holds a composite value containing the default retention period plus the start time for the retention period. By adding the retention period to the start time, the MFA determines the expiry date.

  • PR_RETENTION_PERIOD. This is the number of days items should be retained in a folder. When a personal tag is applied to an item, this property is set. However, if the property does not exist, the item (or subfolder) inherits the retention period from its parent folder.

  • PR_RETENTION_DATE. This is the calculated date when an item’s retention period expires. Clients display this information to users. When clients work in online mode (such as in Outlook Web App), Exchange takes care of calculating this value, otherwise clients that work in offline mode perform the calculation. otherwise.

  • PR_RETENTION_FLAGS. For a folder, this flag indicates whether the retention tag is inherited from the parent folder. If the user sets an explicit tag on a folder, the value is nonzero.

  • PR_POLICY_TAG. This exists only for items and contains a binary encoded value pointing to the policy tag that governs the item.

  • PR_ARCHIVE_TAG. This exists only for items and points to the archive tag that governs an item.

  • PR_ARCHIVE_PERIOD. This exists only when an item has been stamped with an explicit archive tag to contain the number of days in the archive retention period.

  • PR_ARCHIVE_DATE. This contains the date when an item will be archived.

You can view these properties through MFCMAPI. Figure 2 shows the value of the PR_RETENTION_DATE property for an item.

Figure 2. Viewing the PR_RETENTION_DATE property for an item

Retention date calculation

Date calculation is a very important part of the work the MFA does when it processes items. The MFA has to understand the date that should be used to calculate the age of the item (when the item first appears in the mailbox) and either the date when the item will expire or the date when the MFA has to take the action the retention policy requires (deleting the item or moving it into the archive).

Given the nature of email, most items enter a mailbox when they are delivered in the Inbox or are sent and stored in Sent Items. Items often stay in these folders for much of their lifetime and, if they do stay, will probably come under the control of either a folder tag (if defined for the folder) or a default tag (that applies to untagged items in the mailbox). The exception, of course, occurs when a user explicitly applies a personal tag to an item in the Inbox or Sent Items.

When the MFA runs, it examines items in the mailbox and determines what processing is required. Assume that an Inbox item exists that was delivered on 1 April 2013 and that a folder tag that requires items to be deleted and allow recovery after 30 days is applied to the Inbox. MFA stamps this item with a start date of 1 April 2013 by updating the PR_START_DATE_ETC property. It then calculates the expiration date by adding 30 days to the start date; 1 May 2013 is the result. This date is then stamped on the item by updating the PR_RETENTION_DATE property. On or after 1 May 2013, the MFA returns to the item and discovers that its retention period has expired. (If you view the item by using Outlook before the MFA returns, you’ll see the item marked as Expired.) When the MFA processes an expired item, it takes the retention action defined in the tag. In this case, it moves the item into the Recoverable Items folder, where it will remain until the retention period for the mailbox expires, at which time the MFA will remove the item permanently from the database.

This is the simplest kind of processing the MFA is called on to perform, but it happens for a large percentage of items because many users leave messages in the Inbox and Sent Items folders. People who let items accumulate in these folders are often called pilers because they create piles of messages and then rely on client search facilities to locate specific items when required.

Broadly speaking, the other kind of user behavior is represented by the filers, or people who move items from the Inbox and Sent Items into more appropriate folders in which the items form collections that represent important categories of work (or play) as viewed by the user. Or indeed the user attempts to keep the Inbox and other folders under some form of control by deleting unwanted items on a regular basis.

Assume that your user receives a message, reads it, and then comes back to the message a week or so later when its content is no longer required, perhaps because he has acted on a request that was contained in the message. Because the item has been in the mailbox for more than a day, the MFA should have already processed it and stamped its properties with a start and an end date for retention purposes. The user now deletes the message as part of his cleanup effort, and the client moves the item into the Deleted Items folder.

You now have a different retention context because the item has moved out of the sphere of control exerted by the folder tag placed on the Inbox. Retention policies usually include a folder tag to control the Deleted Items folder and, if so, the MFA now must apply the instructions contained in that tag. Otherwise, the MFA will apply the instructions contained in the default tag.

Assume that the folder tag assigned to Deleted Items requires items to be removed from this folder (using Delete And Allow Recovery) after seven days. The MFA examines the item and discovers that its start date is 1 April 2013 and that it should now be removed on 8 April 2013. If the user moved the item into Deleted Items before 8 April 2013, the item is kept until 8 April 2013 and then removed. However, if the MFA processes the items on 8 April 2013 or after, it knows that the expiration date for the item is already past and therefore immediately executes the retention action and moves the item into the Recoverable Items folder, where it will remain until the deleted items retention period specified for the database elapses. The default deleted items retention period for newly created databases on an Exchange 2013 server is 60 days. However, this period is calculated based on the item’s deletion date rather than its creation date, so it will be held for another 60 days to allow the user to recover the item during this period. (For example, if an item is deleted on 2 April 2013, the MFA will purge it on 1 June 2013.) Some companies use an extended deleted items retention period of up to a year to ensure that users can recover deleted items themselves without involving administrators, a useful practice that costs a little more disk space for databases while releasing administrators from the tedium of having to restore databases to recover a now-important item deleted some months in the past.

Calendar and task items pose some particular difficulties for the MFA when it comes to calculating their expiration date, which is why Microsoft took some time to find the best way for the MFA to proceed and implemented only the processing from Exchange 2010 SP2 RU4. Calendar items are either nonrecurring or recurring and will be either in the Calendar folder or in Deleted Items. It is possible that calendar items might turn up in other folders, but this happens very rarely in practice; if it does, the MFA deals with these items as if they are in the Calendar folder. The following rules are observed:

  • Nonrecurring items in the Calendar folder expire according to the end date of the appointment or meeting. For example, if you create a calendar entry for a trip that lasts from 1 June 2013 to 10 June 2013 and the folder tag for the Calendar folder specifies a retention period of two years, the item will expire on 10 June 2015.
  • Recurring items in the Calendar folder expire according to the end date of their last occurrence. For example, if a series of meetings is scheduled from 1 May 2013 to 1 September 2013, the item expires on 1 September 2015. If a recurring event has no end date, it will never expire.
  • Calendar items in the Deleted Items folder expire based on their message-received date. If this property is not populated (for instance, for an item the user created in his own calendar and never sent to anyone else), the MFA bases its decision on the message-creation date. If neither of these dates is populated, the MFA ignores the item.

Task items receive similar care when the MFA calculates their expiration. The following rules are used:

  • Nonrecurring (simple one-off) tasks stored in any folder other than Deleted Items expire based on the message-received date. If this property does not exist, the MFA uses the message-creation date.
  • Recurring tasks expire based on the date of their last occurrence. A recurring task with no end date will never expire.
  • A regenerating task (a special form of recurring task that generates a specified time after the preceding instance of the task is completed; see Figure 3) never expires. Only the most dedicated of users who understand how to make maximum use of tasks create these items!

    Figure 3. Specifying a regenerating task

  • Tasks that have been placed in the Deleted Items folder expire based on their message-received date. If this property does not exist, the MFA uses the message-creation date. If neither date is available, the task never expires.

As explained in the previous section, you can use MFCMAPI to validate the dates the MFA uses to calculate item expiration.