New SharePoint and Office 365 Hybrid Search

I am super excited to see the new Hybrid Cloud Search coming. Why? A ton of reasons. The biggest for me are:

• Simplified On-Premises Deployment – If you have ever had to manage SharePoint on-premises before, the search servers require the most care and management. No more search crashing on you because you did not put enough resources to manage it. No more having to worry about timer jobs and full index crawls.
• No More Having to Run Search Servers On-Premises - Moving this entire search workload to the cloud will significantly make your on-premises deployments smaller and easier to manage.

Up to this point, the only way to achieve a hybrid search experience by setting up query federation with SharePoint 2013 on-premises with SharePoint Online. This really did not make your SharePoint on-premises deployments any easier. Now we can do away with that configuration.

Here are some specific technical considerations.

• You now have the ability with your SharePoint 2013 and 2016 on-premises farms to get the entire SharePoint search experience from the cloud. This is being termed as SharePoint Server with Cloud Search service application.
• Scenario: If you have not leveraged SharePoint Online yet, you can use this as an opportunity to remove all your SharePoint on-premises search servers and use the cloud.
• Scenario: If you have move some or a large portion of your SharePoint on-premises to the cloud, you now have the ability to get a unified hybrid search and that entire search experience is delivered by the cloud.
• Scenario: If you have specific reason (i.e. legal, compliance) that you have to keep some search servers on-premises, you can still do that have query federation (picture below) between your on-premises SharePoint and SharePoint Online.
• Hybrid search can go across the following content sources: SharePoint Server 2007, 2010 or 2013, File shares, BCS connectors*. This basically means that user in SharePoint Online can retrieve search results from all these locations.
• Recommend that you get your SharePoint on-premises to SharePoint 2013 or higher to get the most advantage of these features.
• Recommend you seriously considered using search from the cloud. Why? With the amount of data organizations is acquiring and maintaining, using an elastic cloud to deliver those search results will only make your life easier J
• Delve Search experience will also be supported as part of this new search experience. So Delve can find files not just in SharePoint Online, but also in SharePoint on-premises.
• eDiscovery and new SharePoint DLP capabilities will be able to go across SharePoint Online, SharePoint on-premises and OneDrive for Business. This is a big deal.
• Configuration with Azure AD is required. You will do this anyways as an Enterprise Office 365 customer.
• If you need to do customer IFilters, BCS connectors or Partner connectors, those will remain on-premises.
• Cloud will not support: Site collection level schema mapping, Custom security trimming, Custom entity extraction and Content enrichment web service.

Here is a high level logical architecture diagram

Getting the point across that this makes your on-premises deployment so much simpler. It is hard to argue that you SharePoint on-premises life will be made better.

References

https://blogs.office.com/2015/08/24/announcing-availability-of-sharepoint-server-2016-it-preview-and-cloud-hybrid-search/

https://channel9.msdn.com/Events/Ignite/2015/BRK3134

SharePoint 2013 Technical Diagrams Notes

Introduction

SharePoint 2013 Preview Technical Diagrams are now available here - http://technet.microsoft.com/en-us/library/cc263199(v=office.15).aspx

Ever since SharePoint 2007 started publishing these technical diagrams, I have recommended that architects become very familiar with them. I always start here when trying to understand a new major product release for SharePoint. If you search my blog, you will see that I have directly referenced these diagrams when building SharePoint strategies for customers.

The following is a high-level review of the new architecture changes available with SharePoint 2013.

Corporate Portal Diagrams

I reviewed the two new Corporate Portal Diagrams for SharePoint 2013. From a logical architecture perspective, these diagrams do not have any major changes from the SharePoint 2010 versions. These diagrams accurately capture how organizations should build web applications, site collections, application pools, SharePoint services, etc. to support major business initiatives. The diagrams are still a must read for people who are new or who need a refresher to understand how they should be segregating content and business functions across SharePoint.

Extranet Diagram

The new diagram for SharePoint 2013 extranet architecture closely resembles the corporate portal diagrams, however it is not very revealing on the type of information organizations need when making a decision on how to deploy an extranet. Looking back at the SharePoint 2010 Extranet Topologies diagrams (http://technet.microsoft.com/en-us/library/cc263199.aspx), I find that diagram to be much more helpful and the information contained here still holds true with SharePoint 2013. I would recommend reviewing both of them together.

Services in SharePoint 2013 Diagram

I admit this has always been one of my more favorite diagrams. When this was released in SharePoint 2010, it captured a fundamental change in how SharePoint services are configured and delivered. This new architecture was created to support Microsoft’s ability to deliver SharePoint Online as a SaaS solution.

I reviewed this diagram and nothing has significantly changed in regards to sharing services across farms, the logical architecture of services, service groups and service deployment.

In the services table there are a few new services that have been added.

• Access Services – Do not be confused by this. Yes there was Access Services in SharePoint 2010. At this early point, I know that that Access Services for SharePoint 2013 have been changed to be more focused on utilize the new App Architecture. As such, Access Services for SharePoint 2013 is pretty different. Access Services solutions created in SharePoint 2010 will still be supported moving forward, however they will run in a different service.
• App Management Services - This is a new service that will be used specifically for supporting the new internal catalog or the public SharePoint store. Remember that in SharePoint 2013, everything is an app; EVERYTHING. Even everyday SharePoint lists are now called an app. Once you get over the name change, you will find out it makes complete sense and Microsoft has just aligned what is does with how business users talk about technology.
• Machine Translation Service – This is a new one and as of right now, I do not have much information on the purpose of this service other than the description which says it performs automated machine translation.
• Work Management Services – This service provides task aggregation across management systems including SharePoint, Exchange and Project Server. This is huge from a user perspective. One single place to see all of your tasks. No more building content query web parts to find all tasks; this effectively does this plus goes outside the SharePoint boundary to find more tasks. This is a very exciting service.
• Office Web App Services – Is called out in here as a service that is no longer running inside of SharePoint server. Why? Microsoft strategy is to provide Office Web App Service to other enterprise application than just SharePoint and it strategically made sense to move it out of SharePoint.

In the rest of this diagram there are architecture diagrams for how to architect service groups across farms, none of which have changed from SharePoint 2010. If you are not familiar with this stuff, this is a must read and I recommend reading my old posting on it here.

Mobile Architecture Diagram

There is a brand new mobile architecture diagram provided and obviously this is drive by Microsoft’s focus on being a “services and devices” company. This is a pretty simple architecture that basically describes some things you need to think about if you are going to support mobile to your users and discusses some of the mobile capabilities. This can serve as a launch point for you to begin to dive deeper into how you will support mobile for your organization. The following are some high-level observations I had when reading this the first time:

Extranet – If you are not thinking extranet, you need to so you mobile users can access content when they are on a mobile device. They have some diagrams which will get your started thinking about it and additionally how you can use Unified Access Gateway (UAG) as a reverse proxy to help with that.

Mobile Device Management (MDM) – One interesting thing brought up in this diagram is how do you manage mobile devices? If you need something simple, you can leverage Exchange ActiveSync for remote device wipe, password enforcement, etc. If you are looking for application level MDM there are additional solutions out in the marketplace today that provide even more capabilities.

Application Architecture – The new SharePoint 2013 mobile architecture is introduced. They break it down basically into two logical layers: mobile and SharePoint. Some key points are:

• Automatic Mobile Browser Redirection – Is a new capability that can be used to optimize the mobile experience based on the connecting device. This Feature must be active on the site and will be activated by default on numerous site templates. First there is the Classic View which is used to provide backwards capability to mobile devices and will have a SharePoint 2010 mobile browser experience. Then there is the Contemporary View which is geared to support HTML5 browsers. The Contemporary View is several enhanced features for navigation of SharePoint sites. Additionally, Full Site View is available so the SharePoint site page can be viewed as if it were on a desktop browser or a tablet device.
• Office Hub for Windows Phone – Is an application for Windows phone devices that provides enhanced capabilities to access SharePoint content from multiple places in one spot. It also leverages mobile Office.
• Location – There is a new geo-location field type that is available in a SharePoint List. This can make a list location aware to capture latitude and longitude which can be used with map applications. For instance, if a user enters in data on their mobile device, it will capture where it was done from and then can be displayed on a map. Here is some more information about this - http://technet.microsoft.com/en-us/library/fp161355(v=office.15).aspx
• Push Notifications – There is a new capability to allow notifications to be sent from a SharePoint site to registered applications running on a mobile device. The nice thing about this is that Windows Phone Apps can receive notifications without having to poll. Here is some additional reading on the topic - http://msdn.microsoft.com/en-us/library/jj163784(v=office.15).aspx
• Device Channels – This is a really important new capability as device channels allow you to deliver a publishing site geared specifically to support different types of remote devices. Basically the site can mapped to multiple master pages and style sheets and even control what content you want to make available to specific devices. Here is an overview on the new device channels - http://technet.microsoft.com/en-us/library/fp161351(v=office.15).aspx
• Office Web Apps – As mentioned earlier in this posting, Office Web Apps is now a separate standalone server which does not run inside of the SharePoint boundary. Office Web Apps has been improved a lot to support mobile devices. There are Word, Excel and PowerPoint Mobile Viewers.

SharePoint 2013 Upgrade Process Diagrams

There are two upgrade diagrams that have been provided. Here are some high points I walked away with:

• Must be on SharePoint 2010 – To upgrade, you must be on SharePoint 2010 technologies. This means if you are on SharePoint 2003, 2007 or 2010, you will need to upgrade to the appropriate version to get to SharePoint 2013. There are Microsoft migration partners that have solutions to assist with this. I saw many times, this is the big value proposition for using SharePoint Online as this is handled for customers.
• Database Attach Upgrade – Is the only supported method for upgrading. There is no more “in-place” upgrade option. Frankly that is fine because most customers always went down a database-attach upgrade.
• Preparation – much of the preparation activities that we have discussed in the past with SharePoint 2010 hold true with SharePoint 2013. There is a bunch of information you are responsible for gathering.
• Manual Configuration Settings – In the preparation phase is recommended to get a understanding of all the custom configurations that you may have done because not all of them are going to be migrated. This is because not all databases are upgraded. So many custom configurations in central admin such as alternate access methods, time job tweaks, managed paths, incoming/outgoing email settings, certificates, etc. will need to be documented and reconfigured in the new farm.
• Databases That Can Be Upgraded – There is a set of databases that can be upgraded. They are Content, BDC, Managed Metadata, PerformancePoint, Secure Store, Search and User Profile databases.
• Customizations – This is an important task that needs to be completed. I have seen many cases where good software organizations have not implemented a strong configuration management process and the result is an organization may not know about all the customize code that may be implemented. There are numerous ways to find all of them by running PowerShell commands, doing system directory diffs, checking web.config, etc.
• Upgrade Health Checks – There are some new features that are available to site collection administrators that will show you a health check of a site collection before actually upgrading the site collection.
• Evaluation Site Collection – Site Collection Administrators also have the ability to request the site collection be copied into a new site collection to evaluate how the upgrade will affect any customizations they may have. This is helpful so you can remediate issues before you actually perform the upgrade. This is also nice because your site collection will run in a SharePoint 2010 mode until you are ready to actually upgrade it.
• Testing – Just like for SharePoint 2010, the best way to prepare for a migration is to build up your new SP 2013 farm and then multiple practices runs of that upgrade into the new production environment. An entire process is defined in one of the diagrams and is a great place to start.

SharePoint 2013 Search Diagrams

If you are a reader of my blog, I wrote some long postings about the search architecture for both SharePoint 2010 Search (here) and FAST for SharePoint 2010 (here). I am not going to do a deep dive into all these search components and roles because they are basically covered. As many people now know, the FAST search engine is now the core search engine for SharePoint. It will just be referred to as SharePoint Search. Now you will be able to leverage a very powerful search engine out of the box. However many of the advanced enterprise features of search will only be available in the SharePoint Enterprise addition. I am also really excited about this for SharePoint Online because it can leverage FAST too. SharePoint Online will not be able to do Enterprise Search of line of business systems but a Search Farm (which is FAST underneath the hood) can be configured on premise and SharePoint Online can invoke that search and provide the search results in the cloud; pretty exciting.

I highly recommend taking the time to review both of these diagrams. It explains how each of the components interacts with each other. Additionally there is a diagram the goes into how to scale the server farms for the amount of content you will need to index. There is a great, new table in there that shows you how scaling will work. To be honest, the folks who are really serious about search will say it is an art and a table does not always communicate how you will do it. It always comes down to how many items, the types of data sources, custom transformations, query latency, index latency, etc.

SharePoint 2013 App Overview Diagram

This is an area I plan to do a lot more exploration of this coming year on and writing on this blog. Why? This is something we have been waiting for a long time with SharePoint development. There are several ways to look and this. SharePoint Features which we have been writing for years are Apps. This is name change to better communicate our technology to the end users who have to use SharePoint. However the new SharePoint App architecture is way more than that.

I have seen so many things over the years.

• I think one of the biggest challenges people would run into is developing great SharePoint solutions only to find out they incorporated some dependency they should not have, they wrote some high-end code that should not be running in the SharePoint layer, they cannot leverage their solution outside the SharePoint boundary, etc. We want to resolve those problems by helping developers to deploy solutions in way that will keep their SharePoint environment nimble.
• Plus we want to provide third-party vendors quicker access to customers. We want to help customers to quickly acquire third-party solutions.
• Additionally we want to allow customers to leverage commodity based SharePoint Online. As you may know SharePoint Online has restrictions on high-end custom development and if that code where to run in another location, while be highly integrated with SharePoint Online, that is a huge win.

I will thing of many more reason this year on why this is so great J

Now we can achieve this through the new SharePoint App architecture. The old SharePoint Solution architecture where you create a WSP is still around. Nothing has changed there. This is used to create deployment packages and in many cases is used to deploy code that requires full trust. SharePoint Solution packages will continue to be used by third-party vendors or developed internally with such tools as Visual Studio. You can still create Sandbox Solutions which run in a more secure runtime and can be deployed in SharePoint Online.

Now the new Apps framework for SharePoint 2013 is a packaged up in a file called .APP. It is composed of many of the same types of files, AppManifest, embedded Solution.wsp, etc. Once an app is loaded into SharePoint, it is accessible through the App Catalog in SharePoint. This App Catalog can be controlled at an organizational level.

Remember the big point with Apps is, that the custom solution you are writing may or may not actually run in SharePoint. Full trust code is not supported. Your custom solution code itself may run in a different SharePoint farm, on an IIS server as ASP.net pages, ASP.net pages running in Windows Azure, etc. So how does SharePoint access these solutions running outside the SharePoint context? In simple terms we have an IFrame (with some extensions) that external solution is available through. OAuth provides the secure connection for access SharePoint objects from a remote location. We will additionally use a new extended and robust event model and remote client SharePoint library to write integrated, remote code.

Why is this so great? We are going to ensure that custom applications and solutions that are being developed with SharePoint are isolated. No more writing a bunch code and services that should not be running in SharePoint servers. It is great that you can do whatever you want with SharePoint, however this will drive solution management.

So you may be asking where does this get deployed? There are many different options for hosting.

• SharePoint Hosted – This means the app and all the resources run in SharePoint. Remember you server side is not supported however you can write applications with SharePoint’s JavaScript libraries and such.
• Windows Azure Autohosted – This is a model that is only supported in SharePoint Online. In This case you can write an App package that will have code for Azure and SQL Azure embedded into it. When the application is deployed, the azure solution will be automatically deployed for you. You do not have to go to Azure and set anything up at all; it is all handled for you behind the scenes.
• Provider-Hosted – This is the third model where custom code and solutions are hosted in a separate server in your organization, hosted in Azure, hosted in different SharePoint servers, etc.

Once an App package is installed, it can be managed and monitored through the catalog. End users have the ability to select an app to run in their sites (much in the same way as turning on a Feature). If and when an app is updated, the user can decide how they want to upgrade to the new app.

Again I really plan to go much deeper on this in my blog but for right now, these are just some introduction notes and ramblings on how excited I am about this new capability J

Back Up and Recovery Diagram

There is a new diagram that goes into the details of doing your own back-up and recovery for SharePoint 2013. I know many people have become accustomed to using third-party vendors for supporting these operations and I still believe these vendors will continue to provide features above and beyond what is out of the box. However, if you are a do it yourself sort of person, this is a great diagram to review.

Not much has changed in regards to developing back-up procedures for both the SQL Servers and the SharePoint Servers. There are tons of scenarios covered in here, and I recommend reading this if it is important to you.

SharePoint 2013 Database Diagram

Finally the database server diagram has been updated. This is a really really really important diagram to review if you are managing on-premise servers. It goes over all the SharePoint databases, plus provide sizing and scaling guidance. Great information.

SharePoint Search Index Document

I received several questions about indexing of SharePoint content lately. Specifically does SharePoint index the entire document?

First off if FAST is being utilized, yes, the entire document will be indexed.

Second if you are using the out of the box SharePoint search by default it configured to index the content of items that are under 16MB or under. Metadata associated to these documents will still be indexed (so if the big files are videos they will still be searchable).

The max file size for indexing can be upped up to the max file size limit; which is 2GB. Before upping the size, it is recommended that you do some testing and understand any performance ramifications relative to your environment and data. Here are some good references on the topic.

• http://www.toddklindt.com/blog/Lists/Posts/Post.aspx?ID=215
• http://appsense.logimindz.com/?p=27

SharePoint 2010 Independent Market Analysis

Updated - 3/2012 - View new blog posting here

I get asked a lot about reasons to move to SharePoint 2010. I simply state, look at the how the market evaluates SharePoint.
Here is a great link managed by Microsoft that showing Gartner and Forrester reports - http://www.microsoft.com/presspass/itanalyst/default.mspx

As of right now, I recommend reading the following:

• Enterprise Content Management (ECM) – View Report
• Business Intelligence (BI) – View Report
• Enterprise Search – View Report - Just Updated in Nov 2011 and continued leader - View Report
• Social Computing – View Report - Just Updated Aug 2011 (continued to be leader, here is the 2010 report)
• Portal Platform – View Report
• Unified Communications – View Report and View Report
• Email – View Report

I suggest stay up to date with this site to find the latest reports.

SharePoint 2010 Search Configuring for Scale

• Part 1 - SharePoint 2010 Search Architecture Introduction
• Part 2 - Scaling SharePoint 2010 Search
• Part 3 - SharePoint 2010 Search Configuring for Scale

This is the third part in my series on SharePoint 2010 Search Architecture.

SharePoint 2010 Search Scaling Configuration

Now if you have been reading this far you may be think this is really hard to configure. Well it is actually very simple. I am going to skip the step of setting up the SharePoint Search Application instance. I am going to give you a quick introduction to the screens you need to know about to scale SharePoint 2010 Search.

Note the screenshots because are just off my single server development box, so I cannot show you the nice screenshots with different server names.

Search Application Topology Screen

This is a screenshot of the Search Application Topology in Central Admin. Here you can see:

• The Admin Component and what machine it is configured to run on.
• Crawl Components – in this case I only have one and the machine that is configured to run on. Notice the name of the crawl database is shown so you can quickly see which Crawl Component maps to which crawl database.
• Databases – lists all of the search databases associated to this Search Service Application instance.
• Index Partition – Shows the single index that has been created and the query component for that partition.
• You will notice there is a Modify button. Pressing that button will allow us to make changes.

Search Modification Screen

This screen shows the same information as above but there are a few differences

• There is a New button in the top left had corner. This will allow us to create new crawl databases, crawl components, query components/index partitions, index partition mirrors and property databases.
• Note there is an Apply Topology Changes button in the bottom right which allows us to make all of our changes at one time.

New Crawl Component

On this screen you basically just select the server on the farm where you want to install the crawl component. Then you need to select the crawl database for the crawler component; that is it.

Remember you can have multiple crawler components installed on the same machine. It would not make much sense to have two crawler components on the same machine that I using the crawler database.

Reference - http://technet.microsoft.com/en-us/library/ee805950.aspx

New Query Component

This is where you create new query components. Remember when you create a new query component; a new index partition is created. The SharePoint Search Service will have the responsibility to ensure that each index partition has an even distribution of items.

Here you select what server it will be installed on and you select the property database that will be used. The location of the index file is important; it needs to be in a location that is biggest enough to handle the index.

Reference - http://technet.microsoft.com/en-us/library/ee805955.aspx

New Query Component / Index Partition Mirror

To create a mirror, back on the Modify Topology screen, if you hover your mouse over the Query Component link, a dropdown will come up saying “Add Mirror”.

This will allow you to pick a server for where you would like to create a mirror. Remember there is now real point in creating a mirror for an index partition on the same server where the partition resides. The same sort of screen will come up as the as when adding a new query component. The checkbox at the bottom “Set this query component as failover-only” is important. If this is checked, the mirror will only be redundant. If not checked, the mirror will be used as part of the query process.

Reference - http://technet.microsoft.com/en-us/library/ee805953.aspx

New Crawl Database

This is the screen where you create a new crawl database. Here you simply just fill out the basics, not much to it.

One example of why you may need a second is to support dedicated search resources to specific content sources. If you want to do that, you have to make sure to check the checkbox below. If you do not, the new crawl database will simply be used as a new crawl database along with other existing ones you have. After you check the box below, you will need to apply the changes then perform some steps like creating a new content source and host destination rules which will map content sources to crawl databases.

Reference – Crawl Database - http://technet.microsoft.com/en-us/library/ee805952.aspx

Reference – Host Destination Rule - http://technet.microsoft.com/en-us/library/ee805949.aspx

New Property Database

Finally there is an option to create new property databases; nothing special about this. If you were to do this, you may have to go back and reconfigure you existing query components to use this new property database. That will in turn re-create your index partitions underneath the hood and as well require you to create new index partition mirrors. All in all, not a big deal, just a few extra steps.

Reference - http://technet.microsoft.com/en-us/library/ee805954.aspx

Series References:

• SharePoint Server 2010 capacity management: Software boundaries and limits - http://technet.microsoft.com/en-us/library/cc262787.aspx#Search
• SharePoint Server 2010 performance and capacity test results and recommendations - http://www.microsoft.com/downloads/en/details.aspx?familyid=FD1EAC86-AD47-4865-9378-80040D08AC55&displaylang=en
• Manage Search Topology - http://technet.microsoft.com/en-us/library/ee805956.aspx
• Search Architectures for Microsoft SharePoint Server 2010 - http://go.microsoft.com/fwlink/?LinkID=167739
• Design Search Architectures for Microsoft SharePoint Server 2010 - http://go.microsoft.com/fwlink/?LinkID=167742

Scaling SharePoint 2010 Search

• Part 1 - SharePoint 2010 Search Architecture Introduction
• Part 2 - Scaling SharePoint 2010 Search
• Part 3 - SharePoint 2010 Search Configuring for Scale

This is the second part of my series on the architecture of SharePoint 2010 Search.

SharePoint 2010 Search Component Scaling

As I mentioned the SSP was one of the biggest constraints to SharePoint 2007 Search’s ability to scale. Now that is gone, we have a ton more flexibility.

Let’s talk about the components I introduced earlier and how each are commonly scaled.

Crawler – There can be multiple instances of the crawler component within a single search application service. This will allow you to more efficiently crawl content based on the scenario you need to support.

One example would be adding a second crawl component (on a different server) to improve the performance of performing a crawl. As I mentioned the crawl component is stateless and data about what has or has not been crawled is stored in the crawl database. So you can create multiple crawl components that are configured to use the same crawl database which will help with the performance for building the index (i.e. multiple crawlers running in parallel).

Another example would be you want to dedicate higher end machines with crawl components to particular content sources. For instance let’s say there is content to be indexed on SharePoint and in File Shares, and there is significantly more data on the File Share. You have the ability to create new crawler components that has a dedicated crawl database. Using host distribution rules, you can configure a new crawl component and crawl database to only crawl content in the File Shares while others crawl SharePoint content.

Some other guidelines you should be aware of are:

• There should be at least 4 cores dedicated for each crawl component running on a server.
• Not recommended to have more than 16 crawler components in a single Search Service Application.

Crawl Database – Multiple instances of the crawl database can be created to support different scaling scenarios as just mentioned. It is not recommended to have more than 25 million items in a single crawler database and no more than 10 crawl databases per Search Service Application.

Query Component – In SharePoint 2007 we had the ability to run multiple instance of the query component on each load balanced WFE so we had the ability to some scaling. We now have more granular control of how we scale out multiple query components. We still have the ability to create multiple instances of a query component for redundancy purposes.

Index Partition – As I mentioned the Index Partition was added to allow for queries to perform more efficiently. Each query component has an associated partition. Whenever a new partition is created, a new query component must be created as there is a one-to-one relationship between the query component and the index partition. There is an emphasis to ensure that each index partition is evenly balanced with documents (a hash algorithm is used based on the document id).

For example, let’s say you have 2 million documents in your single index and it takes 1 second to return the results. If this were split into two partitions with two query components, the time to complete the search is cut in half because now there are two query components only search 1 million documents.

Couple Notes:

• It is not recommended to exceed more than 20 index partitions within a single Search Application Service instance; even though the hard boundary is 128 index partitions.
• There is a hard limit of 10 million items that can be in any index partition with SharePoint 2010 Search, and it is not recommend exceed 10 partitions (100 million items). If you are exceeding that many items, you will probably look into separating search into multiple search service instances or look into using FAST.

Index Partition Mirror – Earlier I introduced the content of index mirrors as a way to provide redundancy and better performance. The mirror is how it sounds; it is an identical copy of the index. When mirrors are used, you basically now have a one-to-many relationship between the query component and the index partitions but there will only be one primary partition. The mirror partitions can be used a way to provide redundancy so if the machine that is hosting the primary partition was to fail, the mirror will become primary. Also, the mirror can be configured so it can be search against as well to help support scenarios where there is a load of queries being made.

Property Database – There can be more than one property database created in the farm to again support scaling and query performance. It is recommended that once there 25 million items that have been indexed, a new property database should be introduced into the solution architecture. This is because with that many items, a significant amount of metadata will be created and the property database is responsible for managing that metadata for search queries. This is not hard limit; just a recommendation.

It is also recommended to separate the Property and Crawl databases to separate storage to remove I/O contention that can occur when there are both crawls and queries executing at the same time. This is because the property database is highly utilized as part of the querying process. If full crawls are done during off hours and incremental crawls are not very intensive based on the amount, this should not be an issue.

Search Admin Component – There is no capability to scale this; not needed either.

Search Admin Database – Since I mentioned this earlier, I will mention it to be consistent. There is no capability to scale the search admin databases since there can only be one per Search Application Service instance. You can create redundancy at the SQL Server level.

SharePoint 2010 Search Scaling Scenarios

Up to this point you have been consuming a lot of information about what the new SharePoint 2010 Search components are and how they can scale. It is a lot to take in. What I personally like to do is start with the most basic scenario and then scale out from there.

I am going to focus each scenario based on the number of items to be searched.

To save myself time, I am going to re-use several of the diagrams that are provided to us here - http://technet.microsoft.com/en-us/library/cc263199.aspx. Specifically I am using diagram from “Search Architectures for Microsoft SharePoint Server 2010” and “Design Search Architectures for Microsoft SharePoint Server 2010” diagrams. I highly recommend reading both of these in detail once you have finished reading my blog as re-enforcement.

Scenario 1 – 0 to 1 Million Items

This is probably one of the most basic scenarios but according to Microsoft either of the farms can support this many documents.

HOWEVER I would never recommend these two environments for production where a Service Level Agreements (SLAs) might be in place because it is not a redundant. I would expect out of the gate, to minimally have something like below. This is a best practice from SharePoint architecture perspective to have your Web Front Ends (WFEs) and Application Servers on different machines. As well, you typically have the search service dedicated to its own Application Server as to not create contention with other services that may be running in the farm.

I think the point at the end of the day is that a single application server, that running all of the Search components we described earlier, can support up to a 1 million items. So if you are running a small SharePoint production site, where SLAs are not stringent, you should be fine using search on one application server for up to 1 million items.

Scenario 2 – 1 to 10 Million Items

In this scenario is we now have a few million items and at the point where scale must be introduced. This is referred to as a Small Search Farm.

Observations:

• There is a single crawl server with a single crawl component that builds a single index of all content.
• There is a single database server used where the crawl, property and admin databases are hosted.
• There are two query servers and they have been configured to run on each web front end (WFE). This means the WFE needs to have enough space to store the index.
• There is only one index partition created by the crawler where all of the indexed items can be found.
• There is one primary query component and there is one mirror query component. All the queries will go against primary query component and will failover to the second.
• One question you may have is will only having one active query component become performance bottleneck? The answer is it could if you have a website with a lot of users performing concurrent queries. Remember you can configure the query component mirror to accept queries.

Scenario 3 – 10 to 20 Million Items

Next we have a scenario where we have roughly 10 to 20 million items to be indexed and we want to start scaling the architecture. This is the one that many organizations will use to start with because it scaled for redundancy and performance. So even though you may not have that much content right off the bat, you will be able to grow into it.

Observations:

• There are now two crawler components that reside on different machines. The two crawler components that work with a single crawl database in parallel to build up the index. This also adds a level of redundancy if one of the crawl servers were to go down.
• There are now to partitions of the index, mostly based on 10 million item limit for each index partition. The need for two index partitions will require that two query components are created. The primary for each query component is installed on a different Query Server with the query mirror installed on the other machine. The net effect of this is query time will be reduced because the query component does not have to search the entire index; it only has to search half the index.
• The Crawl and Property databases have been split apart onto different servers. This is because contention can be created between them when crawling and querying occurs concurrently. I really think this is probably one of the last things you need to do to improve performance if you have a highly available SQL Server production environment. This will only help when there are database load issues, there are more than 25 million items or there is a significant amount of metadata that has been built up.

Scenario 4 – 20 to 40 Million Items

In this next diagram the amount of content is dialed up to 20 to 40 million items.

Observations:

• This is still actually referred to as a medium farm but it is called a dedicated farm because the Query components are no longer hosted on the web front ends. The query components are not hosted on individual machines.
• Next you will notice there are now four index partitions for the corresponding four query components. As well, the mirror for each query partition is placed on a different query server.
• Next notice there are now two crawl databases and four crawler components. Two crawler components are dedicated to each crawler database. This configuration supports the ability to have different crawler components search different content sources. For instance one set of crawl components may search SharePoint while the other set search MySites, File Shares and Public Exchange Folders. The point is that if you have scaled up to searching this amount of content is likely that you will be breaking about how content is indexed for performance reasons.

Scenario 5 – 40 to 100 Million Items

Last is a fully scaled out SharePoint Search Application service instance.

Observations:

• This is a fully scaled out farm based on the recommended capacity for SharePoint 2010 search, being 100 million items. As you can see there are now ten index partitions.
• There are several crawl components and crawl servers.
• Crawl databases have been broken into multiple database servers.
• Multiple property databases on different database servers.

Personally I am not sure how common you will see something like this. Do I believe more than 100 million items will need to be indexed by SharePoint? Yes – I do believe that will happen in large environments. However you may employ a strategy where you have:

• Multiple Search Application Service instances in the farm with less data being indexed by each.
• There may be a central hosted search farm while smaller farms have their own search configuration.
• And when you need to provide a single user search experience where more than 100 million items need to be searched; FAST can be utilized. I want to point out that even if you have less than 100 million items to be indexed FAST should still be considered. This is because there is a ton of search and usability features available in FAST which is not available in the out of the box SharePoint search.

Closing

To this point I discussed only how to scale based on the number of items to be indexed. However that is not really an accurate way to build a SharePoint 2010 Search architecture. In many cases business rules or other environment factors can drive how search is architected. Here are some examples:

• Connectivity to specific content locations is slower in some cases. What you may do is add dedicated crawler components and databases just for indexing this specific content.
• Content that is indexed must be fresh and full indexes are needed on a regular basis. In this scenario you will again add more crawl components.
• There is a significant amount of users who will query. In this case, you may only have 5 million items but you may have two or even three query components dedicated to their own machines.

Hearing these examples, you should take the information above and start scaling out based on the business requirements and SLAs you need to support.

My personal recommendation is probably start with scenario 3 and make any tweaks to the configuration based on what you need for a production environment that needs to support growth and redundancy. However adding new components later is no big deal either.

Now that you have finished reading this – I highly recommend you read both of these:

• Search Architectures for Microsoft SharePoint Server 2010 - http://go.microsoft.com/fwlink/?LinkID=167739
• Design Search Architectures for Microsoft SharePoint Server 2010 - http://go.microsoft.com/fwlink/?LinkID=167742

SharePoint 2010 Search Architecture Introduction

• Part 1 - SharePoint 2010 Search Architecture Introduction
• Part 2 - Scaling SharePoint 2010 Search
• Part 3 - SharePoint 2010 Search Configuring for Scale

Introduction

Every time there is a major version release of SharePoint there is always a major improvement in the area of Search. Users need to have the ability to search for content and the users demand the same sort of user experience they have with their everyday search engine. Microsoft really stepped up to the plate this time with SharePoint 2010.

SharePoint 2007 introduced tons of new features that were not available in SharePoint 2003. For instance searches could be done across site collections, it actually returned back correct results, scopes, best bets, search analytics, search federation, business data catalog (searching external line of business systems), a search API, etc. Still however there were some challenges. Scale would become an issue because SharePoint 2007 had exponential growth due to its ease of use. The Share Service Provide (SSP) and the way it is architected was a contributing factor. For instance an SSP could only have on crawler which provided no ability to control large amounts of content. As well, users were demanding for the same sort of user experience they have with Bing, Google, etc.

For SharePoint 2010, there is a lot of improvements.

• There is the new Service Architecture of SharePoint 2010 and the removal of the SSP. Read this blog. As you will see this now enable SharePoint 2010 Search to scale and that will be the focus of this blog.
• The ability to index 100 million items.
• Continued support for indexing file shares, external web sites, line of business systems, public exchange folders, etc.
• Boolean search (and, or, not) are supported.
• Range symbols such as =, <, >, <=, and >= can be used.
• Wildcard searches are now supported out of the box.
• Support for property based searches on the metadata (title:“XXX YYY”).
• Improved relevancy mode like Phrase matching and clickthrough counts.
• Refiners which provide the ability filter down the search results using the returned metadata without have to re-run the actual search.
• Did you mean feature which provides suggestions – like in the case the user misspells a word.
• Search suggestions which provides an auto complete based on what the user commonly searches on.
• Search Alerts and RSS Feeds
• Improved query federation
• More extensible search web parts How
• Several new administration features
• Mobile search

Another major Search improvement is Microsoft’s acquisition of FAST Enterprise Search. Microsoft spent over $2 billion to acquire one of the most high enterprise search engines on the market and incorporate it into the SharePoint platform. The goal of this blog is not to do a feature and architecture comparison with FAST. I may do one in the near future. At a high level you should know that FAST has:

• Limitless scalability. Can search and return results in sub-second times over petabytes of data.
• High scale search refiners.
• Extremely powerful and tunable search relevancy model.
• User contextual search results and relevancy.
• Entity extraction.
• Ability to index almost any type of content imaginable.
• Similar search result suggestions.
• Thumbnails and document previewing in search results.
• Visual best bets.

Still getting your arms around the OOB of the box search if you are familiar with SharePoint 2007 search can be daunting task. I was able to pull together a bunch of information and I am going to consolidate this down for you.

1. I will capture the new components for SharePoint 2010 Search.
2. I will then discuss how each component can be scaled.
3. I will then discuss scenarios on SharePoint 2010 Search is scaled.
4. I will actually show you how simple it is do the scaling of SharePoint 2010 Search.

SharePoint 2010 Search Components

Let’s first talk about all the new components and architecture you need to know about right off the bat.

Crawler – You will hear about this a lot; it is commonly referred to as the crawling component or indexer. It is responsible for building indexes. Unlike the previous version of SharePoint the crawl component is stateless; meaning the index that is created is not actually stored in the crawl component. The index is pushed to the appropriate query server.

Crawl Database – As you just learned, the crawling component itself is stateless. State is actually managed in the crawl database which will track what needs to be crawled and what has been crawled.

Query Component – This is the component that will perform a search against an index created by the crawler component. The query component will apply such things as security trimming, best bets, relevancy, removes duplicates, etc. It is also commonly referred to as the query server.

Index Partition – Is a new feature of SharePoint 2010 and is directly correlated to the query component. We now have the ability to break the indexes into multiple partitions to improve the amount of time it takes to perform a search by the query component. For every query component there will be a single index partition that is queried by the query component. Another way of putting it is, every time a query component is created, another index partition is created.

Index Partition Mirror – There is a new capability to create mirrors of the index partitions. These mirrors again provide the ability to provide redundancy and better search result performance.

Property Database – Stores metadata and security information items in the index. The property database will be associated to one or more query components and is user as part of the query process. These properties will be populated as part of the crawling process which creates the index.

Search Admin Component – The admin component that manages the configuration of the instance of the Search Application Service.

Search Admin Database – It is worth noting there is a search administration database and it mostly responsible for managing information associated to the configuration and topology of the SharePoint Search service. There will only ever be one instance of this database for each Search Application Service instance.

Now that I have introduced you to the major components of SharePoint 2010 Search, I will dive into details about the components and how they can be used together to create search solutions.