Saturday, January 26, 2013

New Capabilities eDiscovery with Office 365


I can happily say I am very excited about the new eDiscovery capability that is part of SharePoint 2013. I am even more excited for this capability being delivered with Office 365. Up to this point in Office 365 you have had the ability to do eDiscovery in the cloud but you have had to execute multiple searches. Now with the new features that will be released, there is a unified eDiscovery for Exchange, SharePoint and Lync data in Office 365. Additionally there are some new features for Exchange 2013, which are available in Office 365, which allows more granular way of eDiscovery and then placing a legal hold on that item.

The following are my notes and some additional resources which you will valuable as you start exploring this technology more.

  • The new eDiscovery solution is part of SharePoint Online Plan 2 and is part of the SharePoint 2013 Enterprise CAL on-premise.
  • This solutions allows you to search, hold, and export content from Exchange and SharePoint. Lync instant messages can be captured when they in Exchange’s conversation history folder.
  • The high level process is you create a case, identify locations of where to search, and any filter to find it. You can manage sources, eDiscovery Sets (sources/filter combos), queries and all exports done. All of these operations can be done from a site in SharePoint Online.
  • The major steps are to Create a Case >> Create a eDiscovery set to find and preserve content (optional in-place hold) >> Create a Query to find and export content (previewing and filtering) >> Release Hold

General Legal Hold Improvements for SharePoint and Exchange

With SharePoint 2013 delivered in Office 365, there will be some additional new features available.

  • The state of the content is recorded, thus allowing users to continue to work with the data. With SharePoint 2010 technologies this was not possible as once the item placed on Hold, it is locked down until the Hold is released. Even though users have the ability to edit or delete, SharePoint will ensure there is no loss of an item that was placed on hold. Discovery managers will continue to have access to all the data that was put on hold. There is a new special SharePoint library created at the site level to handle edit and delete scenarios. Basically when anyone of those transactions occur, the file or item on hold will be stored there.
  • Preservation can be done at the site level now. Users can continue to use preserved content.

With Exchange 2013, there is again a several new features that are available.

  • With Exchange Server 2010, the notion of legal hold was to hold all mailbox data for a user indefinitely or until when hold is removed. The legal hold was placed at the mailbox level. With Exchange 2013 in Office 365 you can now determine what to hold and for hold long.
  • Indefinite Holds – This is the way it was done with Exchange 2010 and it is still available. The entire mailbox is put on indefinite hold; nothing can be deleted and edits are managed until the mailbox is released.
  • Query Holds – This is new. If you need the ability to query for items to be placed on hold, this is now supported. When items are found in the query, just the items are placed on hold. Additionally this does support not just finding existing items, but to future email items that have not arrived yet.
  • Time-based Holds – is the idea where legal hold and retention policies are used in conjunction with each other. Specifically this allows you to place a hold on items for a specific period of time which is calculated from the date the item is received or when the hold is created. What this allows you to easily do is create a rule to ensure that all items are retained for X days / years. For instance a policy that says all emails must be kept for 90 days. When an item is deleted out of the user’s mailbox, it will be retained for the remaining amount of time required by the policy.
  • Multiple Holds – this is new where by a user can be placed on multiple holds. When a user is on multiple holds, all of the holds are applied together using an OR operator.
  • In summary, with the addition of these new in Exchange Online you will be able to place an entire mailbox or specific items on hold; Email will be preserved whether a user or process edits or deletes an email; Users can be placed on multiple holds; Items can be held indefinitely; Legal hold can be made transparent to the user; eDiscovery searches can be done on items that have been placed on hold.
  • One Small Note – the In-Place hold utilizes the Recoverable Items folder which is a replacement to the “dumpster”. The Recoverable Items folder is used in support of Legal Hold. There are four sub folders which are used to manage items. First there is the Deletions Sub Folder which is used when a user were to shift delete from inbox or when an item is deleted from the deleted items folder in their mailbox. Users have the ability to recover items from that folder using the item recovery feature from Outlook or OWA. Second there is the DiscoveryHold Sub Folder which manages items that were on legal hold but deleted by the user. Third is the Versions Sub Folder which manage email items that edited and all versions of the edits. For both DiscoveryHold and Versions Sub Folders items will be removed once the hold is released. None of the data stored in these three folders count against the end-user’s mailbox size limit either. Finally there is the Purgres Sub Folder which is responsible for deleting items once all rules have passed.

If you are deep in both SharePoint and Exchange, what you may not is some convergence. SharePoint has always had query based holds which has been brought forward into Exchange. As well Exchange has always allowed user to still work with their mailbox even though they were on hold which SharePoint did not allow. Additionally SharePoint only support legal hold at the item level and not the site level (which is similar conceptually to a mailbox). This is great and is needed to support the next part of this discusion!!!

eDiscovery Sites

Now let’s talk a little bit about this new eDiscovery site that will be available in SharePoint Online to do eDiscovery on Exchange, SharePoint and Lync.

First you create a case using a site template; pretty straight forward process. Once the site is created you will have the some major buckets of features. There are Discovery Sets which facilitates define the sources the search and creating holds. Second there are Queries which search those sources, allow for previewing of results and ultimately exporting the data.


The below screen a new discovery set. As you can see you have the ability to identify the sources, filters, data ranges, etc. for information you want to hold. You have the ability to define what type of hold you want, and in most cases an In-Place hold will be utilized.


Then once items have be placed on hold, you can use the Search and Export feature to run queries to narrow down and find items. Once you find those items, you can export them.

Below is a screen capture of shows has a query can be created across your held data sources. You have the ability to run the query, get information about that query and even preview items before you do an export. You can see that when the Exchange tab is selected you can see all the email, contacts, etc. objects that were round.


The following is from the same query above and shows you the SharePoint data that was returned.


Finally you can export a query once you have it fined tuned. The big question everyone always asks is how the data will be exported. Data will be exported conforming to Electronic Discovery Reference Model standard:

  • SharePoint Documents – in their format
  • Lists - .csv file will be created.
  • Pages – will be exported in MIME HTML (.mht) files
  • Exchange Objects – email, tasks, contacts, calendar, attachments exported in .pst.
  • Additional XML manifest that complies with EDRM is provided that captures all the information exported.



Friday, January 25, 2013

New Exchange Online DLP

There is a great new feature of Exchange 2013 that will be part of Exchange Online. I am really happy to now have native Data Loss Prevention (DLP) features to share with customers. Up to this point, you have been able to utilize Transport Rules to implement light DLP however if you wanted to implement real DLP organizations were required to manage an appliance on premise to support this. Now organizations have the ability to now remove that dependency and utilize DLP delivered through the Office 365 cloud.
Below are some notes and resources that you should know about. The new DLP capability:
  • The goal is to help identify, monitor, and protect sensitive information from leaving the organization.
  • DLP can be configured through the Exchange Administration Center.
  • You have the ability to start with a pre-configured DLP templates to detect information such as PII. You have the ability to create custom templates with sensitive information types. This will save you a lot time.
  • Types – Detect sensitive information in attachments, body text, subject lines and adjust sensitive level to take action rules (transport rules).
  • DLP Policies are tied directly into Transport Rules. They are no more that packages of conditions, transport rules, action and exceptions.
  • Transport Rules – You have the ability to coordinate DLP rules with Transport Rules and create actions to capture information. Transport Rules look for specific conditions on a message and then takes action on them. Transport Rules let you apply messaging policies, secure messages, protect message and prevent leakage. You can prevent information from leaving, filter confidential information, track / copy messages sent / received by individuals, redirect email for inspection, apply disclaimers, etc. You have the ability to incorporate classifying sensitive information. Additionally you can perform content analysis through keyword matches, dictionary matches, regular expression, etc.
  • Testing – There is the ability to test rules before actually enforcing them. This is possible by creating rules but not activating. Email-flow is not affected until they are finalized.
  • Policy Tips – This is truly a great feature in that preventive actions can be taken with an end user before they actually send an email that could violate a DLP policy. Policy Tips to show users warnings in Outlook in the same manner as Mail Tips. This does require Outlook 2013 client.
  • Reporting – DLP Reports are available and can create own specific reports to monitor issues.
Some additional deeper notes:
  • There are three ways to create a template – 1) Create template using an OOB one. 2) Import ones 3) Start from scratch. There are OOB templates like PCI Data Security Standard, US Financial data, U.S. Gramm-Leach-Bliley Act (GLBA), HIPPA, Patriot Act, PII, etc. Types of common modifications could be to make certain types of users exempt from specific policies for specific situations. Or maybe even invoke RMS in certain situations when a DLP policy may be broken. This native integration into Exchange Online itself is really exciting.
  • There are Sensitive Information Types like a US SSN, Drivers License Number, etc. It is the common rule to find that type. You have ability to create XML files that can be imported through PowerShell to do customer ones. You have the ability to create Entity Rules which define identifiers like SSN. Then there are Affinity Rules which are targeted towards documents. Built of a multiple evidence rules when aggregated together and matches happening in proximity to each other can constitute a DLP policy being triggered. So depending on how many times a rule it tripped in a single item can create a DLP policy to be tripped.
  • Sensitive Information Rules can be used to with transport rules to create hard and soft rules. There is a new “If this message contains…Sensitive Information” transport rule. This can be used with existing transport rules and Boolean logic. For example: Limit interaction between recipient and senders – between internal groups and external groups, Applying separate policies for communications internal and external, Prevent inappropriate information from entering or leaving, Filter confidential information, Tracking or archiving messages sent / received by specific individuals, Redirect inbound / outbound message for inspection before delivery, and Disclaimers.
  • DLP Supported File Types – All the core file types are supported (including zips and cabs). However if there is an unknown file attached that must go through DLP evaluation an exception will be raised to allow you to take action. For Exchange Online you cannot extend this like you can on-premise because you need to create your own IFilter packages which is not supported in the cloud.

Windows Azure AD Whitepaper

Here is a new Whitepaper on entitled “Active Directory from on-premises to the cloud” -

This is a great whitepaper if you want to learn more about Windows Azure AD. Additionally you will find out that Windows Azure AD is the AD solution that is used for Office 365. This whitepaper gives insightful information on:

  • How AD Authentication actually works inside of Office 365.
  • There is some great information in here that explains how once you get it set up, it can be utilized across Office 365 and Azure services.
  • There is information in here that will shed light on how Office 365 is support non-AD directories. This is because you can sync your corporate LDPA directors up to Windows Azure AD and Office 365 will utilize it.

Wednesday, January 9, 2013

SharePoint 2013 and SharePoint Online Upgrade Notes


For SharePoint 2013, there is a lot of great stuff coming for performing upgrades. I may have mentioned this recently on my blog but a lot of this optimization in upgrading SharePoint. Much of this is driven by experience from the five major releases of the SharePoint however SharePoint Online is proving to be a strong catalyst for improvement. When SharePoint 2010 was released we saw how the architecture of SharePoint completely changed, and much of that change is associated to a long-term Microsoft vision to deliver SharePoint through the cloud. Whether you are implementing on-premise or purchase SharePoint Online, you will be able to benefit.

I know many folks have done upgrades in past versions of 2003, 2007 and 2010 and have dealt with a lot of work to get through those. There are a mixed bag of reasons of why upgrades were complex. When I was consulting I would also remind customers to design and implement for the future. Every large SharePoint customization should have a path for the future. There is really too much to go into here and not the topic of this posting….

In this blog I am going to discuss some of the big improvements that you should know right off the bat in regards to performing SharePoint 2013 upgrades. Second I am going to discuss how these new upgrade changes apply to SharePoint Online.

The following are some of my notes I have captured from multiple events and presentations.

New Facts About Upgrades for SharePoint 2013

The following are some of the big facts that you should know about SharePoint 2013 Upgrades:

  • Upgrade Approach – The only approach available for SharePoint 2013 upgrades is DB Attach. There is no more In-Place Upgrade (2003, 2007 and 2010) or Gradual Upgrade (2003).
  • Upgrade Versions – It is pretty much the same, you will need to be on the previous version to perform an upgrade; which will require you to be on SharePoint 2010. There are numerous third-party solutions available in the marketplace that can assist you to upgrade from older versions of SharePoint to SharePoint 2013.
  • Requires New Server Instances – Given the fact that you must do DB Attach, you will have to create SharePoint 2013 servers. You cannot upgrade your existing SharePoint 2010 servers.
  • Important >> Site Collections – Upgrading is now focused at the site collection level. Databases will be attached into the new SharePoint 2013 farm however the actual upgrade itself will be done at the site collection, when the site collection is ready. This is a really important point; “when the site collection is ready”. This means SharePoint 2010 site collections will run in full SharePoint 2010 experience on a SharePoint 2013 server until such time it is decided that it will be upgraded.
  • Improvement >> More Well Known There is a new capability for performing an upgrade health check before actually moving forward with an upgrade. The goal is to provide a capability that will tell organizations that an upgrade will fail prior to it actually failing. A health check report will provide detailed warnings and information on issues to remediate. This is extensible and can build own custom rules. These rules that can detect and custom repair operations can be created for specialized scenarios you may need to support.
  • Improvement >> Evaluation Site – This is an exciting capability. There is a new capability that will provide you the ability to spin up a new site collection to preview an upgrade of a site collection before it is actually done. This is a great capability to test and remediate issues before you actually perform an upgrade. This is basically a replacement of the visual upgrade process that was provided in SharePoint 2010 (in my opinion a better solution). These evaluations site collections cannot made be permanent nor is it recommended for long-term usage. There is configuration available to administrators to control the maximum size for an evaluation site (for instance when an administrator did not put on site quotas and you have TBs of data sitting in a single site collection). There is PowerShell available to automate the creation of evaluation site collections (which can come in real handy if you need to do some automation around testing large amounts of site collections). As well an expiration date can be applied to the evaluation site collection to ensure that these sites are not accidently used for too long.
  • Improvement >> Quicker Upgrades – With the focus of moving to Site Collection upgrades versus an entire content database upgrade, significant amounts of time is saved when actually performing an upgrade. This is because you upgrade the farm but all the site collections (which require all processing) can be spread out. This reduces having to do big bang upgrades.
  • Improvement >> Communications – New features have been implemented that will communicate to users via email during an upgrade. There are events and different email templates you can work with to communicate the status of an upgrade. Additionally a system status bar will also be displayed that an upgrade is being performed. Administrators have the ability to customize the messages displayed to the user to give customized instructions and information.
  • Easy User Transition – As I mentioned earlier, upgrades are done at the site collection level and the SharePoint 2010 experience will remain until in effect the site collection is upgraded. This means you can start delivering new SharePoint 2013 solutions in other site collections while continue to run SharePoint 2010 solutions until you have created a proper transition path for the end users. Additionally, this is great for user transition as it will allow end users get user to SharePoint 2013. Organizations have the ability to strategically identify site collections for upgrade to SharePoint 2013 and then select which ones that should go first.
  • Initiating an Upgrade – Upgrades of site collections can be automated through PowerShell, administrators can manually execute them or they can be delegated to site collection admins (which may be a person within the business). Remember there are still controls available to control how site collection admins can do it (max sizes, specific site collection blocks, etc.).
  • Queuing Site Collections – There is a new capability to throttle the number of parallel upgrades. This can become important if you need to coordinate the upgrade of a lot of production site collections. Specifically you do not want to overload your SQL Servers databases. The Web Application processes will wait for processing capacity and database space before executing. Time jobs are responsible for assisting with this. Large, oversized site collections will not be done through the Web Application; it would be done through a timer job. The queue can be managed if you need. Note that each queue is assigned at the content database level. Even if there is a failure (for whatever reason) the upgrade will be re-initiated in the queue. Finally PowerShell is available to place site collections in the queue or even change the throttle.
  • Logging Improvements – Additionally there have been many new improvements for logging. ULS Logs can easily be pulled into Excel, correlations IDs, more error codes and more details are now provided when an error may occur.

Beyond new upgrade capabilities your process for performing an upgrade will change a little bit given these changes. Here are some things to keep in mind.

  • Review the Upgrade charts located here - There are two specific diagrams a workflow for performing an actual upgrade and another for testing.
  • Do not skip doing due diligence. Please audit yourself, know what you have deployed and determine what the best path is to ensure continuity of operations.
  • As discussed above, you will need to build out your SharePoint 2013 farms prior to performing the upgrade. It is also good practice to leave those SharePoint 2010 farms as long as you can until you feel the production migration is complete.
  • Additionally if you are using federated service farms (ie you have a SharePoint 2010 services that provides common services to other SharePoint farms), you MUST upgrade those farms first. Do not worry, an upgraded federated service farm on SharePoint 2013 can communicate with a SharePoint 2010 content farm. However it does not work in the reverse order.
  • Similar to the prior note, even if you do not have a federated service farm, you should upgrade your services first, then bring over content databases and then gradually upgrade sites collections to SharePoint 2013.
  • Do not forget, if you have a ton of customization and third-party tools, you will need to work through the process of moving them to the SharePoint 2013 farm.

SharePoint Online

As you have been reading, you may have noticed a lot of new features that are focused on supporting large SharePoint environments. One would be Office 365 and upgrading SharePoint Online customers. Knowing that:

  • Previously I talked about upgrading federated service farms first and then upgrading the content farms. Well this is exactly what is going to happen with SharePoint Online. The common federated service farms will be upgraded first by building new service farms on SharePoint 2013 and connecting them to SharePoint 2010 content farms. The new SharePoint 2013 content farms will be created and then the content databases will be moved over.
  • Next customers in SharePoint Online will determine when they want to upgrade the Site Collections to use SharePoint 2013. New PowerShell commands are available for SharePoint Online to automate this activity. Both the new Health Check Report and Evaluation Site Collection features are available.
  • Note that with SharePoint Online customers will be required to eventually upgrade all Site Collections to SharePoint 2013 user experience. Even on-premise customers should do this as the goal is not to allow SharePoint 2010 solutions to run forever. This capability is really intended to provide support for transition.

As I mentioned, moving forward it is always good to understand how you are building custom solutions. I know a lot of people are doing this SharePoint. I really believe that you can build a lot of solutions with out of the box features and capabilities. I recommend that if you are building highly customized solution, to investigate building solutions with the new SharePoint Apps model.