SharePoint Online Hybrid BCS and Search

Introduction
With the new release of SharePoint Online and SharePoint 2013 we have some new SharePoint scenarios that can be supported. I have always told customer the definition of hybrid sometimes murky. Why because SharePoint can be implemented in so many ways. In many cases ADFS provides a unified user experience providing the user experience they need. Doing application integration using APIs, remote libraries, etc. provides all that is needed. The new SharePoint Apps model is going to really introduce a lot of new ways to create customer hybrid solutions.
Some other new hybrid scenarios that are going to be supported with the new SharePoint Online and SharePoint 2013. Specifically there is the ability to configure BCS and Search to support some real hybrid solutions. This is really exciting.
Please review the following whitepapers that are located here:

• http://technet.microsoft.com/library/jj838715.aspx
• http://www.microsoft.com/en-us/download/details.aspx?id=35593

Hybrid BCS
With the Hybrid BCS, organizations are going to have some real flexibility in you can expose data from the enterprise to SharePoint Online. In the prior version of SharePoint Online based on 2010 technologies, we were limited to working with web services. There is nothing wrong with that, but we would have to create some web services, make it available to SharePoint Online, and then use it. Here are some facts:

• This supports the ability to view, edit and delete data.
• All data sources must be published using OData.
• Using this solution, no data needs to be published out in an unsecure way, require any new ports to be opened, etc. The on-premise SharePoint 2013 server will basically connect, using a reverse proxy, and will publish out an end point which SharePoint Online can connect to.
• ADFS with Federated identities is required.
• There are a few configuration steps that you will need to go through outlined in the document. Additionally there is some configurations that you will need to make on the SharePoint Online administration screens.

Doing this will allow you to bring your line of business data into SharePoint Online. Users can access that data through External lists, External columns, developers can use it through code, workflow can use it to help make decisions through using the external lists, Visio Services has a new feature that allows it to user external lists, etc. I am frankly just happy to see this level of integration and facilitation versus what we had to do in the past to expose data to SharePoint Online.

For more details read here - http://technet.microsoft.com/en-us/library/dn197239.aspx

Hybrid Search
Even more exciting is our new support for setting up Hybrid Search for SharePoint Online. SharePoint 2013 must be installed on-premise and you must again be using federated IDs with the SharePoint Online. I would say this is one of the primary use cases for facilitating a hybrid deployment. Why because we want to allow users to be able to initiate and search for data regardless of where it resides.

Remember that the new SharePoint Online in the cloud does utilize the FAST search technology. However just because that capability is delivered in the cloud, it does not provide the ability to index on-premise locations. We rely on SharePoint 2013 on-premise to basically do a federated search. This means that SharePoint Online Search does not directly index on-premise content; it relies on those on-premise servers.

There are two approaches for configuring Hybrid Search:

• One-way Hybrid – In this scenario you have the ability to perform a search from your on-premise SharePoint 2013 environment to SharePoint Online. This way users will have the ability retrieve content from the SharePoint Online anytime they do a search from on-premise. This is nice if you have a search portal set up on-premise that is indexing your entire enterprise and SharePoint Online is just another content source to index. All those intranets, team sites, My Sites, etc. in SharePoint Online are accessible.
• Two-way Hybrid – In this scenario you will be configuring both your SharePoint 2013 on-premise environment and your SharePoint Online environment to perform search of each other. This is really powerful to allow the end user to find content across the enterprise regardless of where the user is.

For more information read here - http://technet.microsoft.com/en-us/library/dn197172.aspx

SharePoint 2013 Technical Diagrams Notes

Introduction

SharePoint 2013 Preview Technical Diagrams are now available here - http://technet.microsoft.com/en-us/library/cc263199(v=office.15).aspx

Ever since SharePoint 2007 started publishing these technical diagrams, I have recommended that architects become very familiar with them. I always start here when trying to understand a new major product release for SharePoint. If you search my blog, you will see that I have directly referenced these diagrams when building SharePoint strategies for customers.

The following is a high-level review of the new architecture changes available with SharePoint 2013.

Corporate Portal Diagrams

I reviewed the two new Corporate Portal Diagrams for SharePoint 2013. From a logical architecture perspective, these diagrams do not have any major changes from the SharePoint 2010 versions. These diagrams accurately capture how organizations should build web applications, site collections, application pools, SharePoint services, etc. to support major business initiatives. The diagrams are still a must read for people who are new or who need a refresher to understand how they should be segregating content and business functions across SharePoint.

Extranet Diagram

The new diagram for SharePoint 2013 extranet architecture closely resembles the corporate portal diagrams, however it is not very revealing on the type of information organizations need when making a decision on how to deploy an extranet. Looking back at the SharePoint 2010 Extranet Topologies diagrams (http://technet.microsoft.com/en-us/library/cc263199.aspx), I find that diagram to be much more helpful and the information contained here still holds true with SharePoint 2013. I would recommend reviewing both of them together.

Services in SharePoint 2013 Diagram

I admit this has always been one of my more favorite diagrams. When this was released in SharePoint 2010, it captured a fundamental change in how SharePoint services are configured and delivered. This new architecture was created to support Microsoft’s ability to deliver SharePoint Online as a SaaS solution.

I reviewed this diagram and nothing has significantly changed in regards to sharing services across farms, the logical architecture of services, service groups and service deployment.

In the services table there are a few new services that have been added.

• Access Services – Do not be confused by this. Yes there was Access Services in SharePoint 2010. At this early point, I know that that Access Services for SharePoint 2013 have been changed to be more focused on utilize the new App Architecture. As such, Access Services for SharePoint 2013 is pretty different. Access Services solutions created in SharePoint 2010 will still be supported moving forward, however they will run in a different service.
• App Management Services - This is a new service that will be used specifically for supporting the new internal catalog or the public SharePoint store. Remember that in SharePoint 2013, everything is an app; EVERYTHING. Even everyday SharePoint lists are now called an app. Once you get over the name change, you will find out it makes complete sense and Microsoft has just aligned what is does with how business users talk about technology.
• Machine Translation Service – This is a new one and as of right now, I do not have much information on the purpose of this service other than the description which says it performs automated machine translation.
• Work Management Services – This service provides task aggregation across management systems including SharePoint, Exchange and Project Server. This is huge from a user perspective. One single place to see all of your tasks. No more building content query web parts to find all tasks; this effectively does this plus goes outside the SharePoint boundary to find more tasks. This is a very exciting service.
• Office Web App Services – Is called out in here as a service that is no longer running inside of SharePoint server. Why? Microsoft strategy is to provide Office Web App Service to other enterprise application than just SharePoint and it strategically made sense to move it out of SharePoint.

In the rest of this diagram there are architecture diagrams for how to architect service groups across farms, none of which have changed from SharePoint 2010. If you are not familiar with this stuff, this is a must read and I recommend reading my old posting on it here.

Mobile Architecture Diagram

There is a brand new mobile architecture diagram provided and obviously this is drive by Microsoft’s focus on being a “services and devices” company. This is a pretty simple architecture that basically describes some things you need to think about if you are going to support mobile to your users and discusses some of the mobile capabilities. This can serve as a launch point for you to begin to dive deeper into how you will support mobile for your organization. The following are some high-level observations I had when reading this the first time:

Extranet – If you are not thinking extranet, you need to so you mobile users can access content when they are on a mobile device. They have some diagrams which will get your started thinking about it and additionally how you can use Unified Access Gateway (UAG) as a reverse proxy to help with that.

Mobile Device Management (MDM) – One interesting thing brought up in this diagram is how do you manage mobile devices? If you need something simple, you can leverage Exchange ActiveSync for remote device wipe, password enforcement, etc. If you are looking for application level MDM there are additional solutions out in the marketplace today that provide even more capabilities.

Application Architecture – The new SharePoint 2013 mobile architecture is introduced. They break it down basically into two logical layers: mobile and SharePoint. Some key points are:

• Automatic Mobile Browser Redirection – Is a new capability that can be used to optimize the mobile experience based on the connecting device. This Feature must be active on the site and will be activated by default on numerous site templates. First there is the Classic View which is used to provide backwards capability to mobile devices and will have a SharePoint 2010 mobile browser experience. Then there is the Contemporary View which is geared to support HTML5 browsers. The Contemporary View is several enhanced features for navigation of SharePoint sites. Additionally, Full Site View is available so the SharePoint site page can be viewed as if it were on a desktop browser or a tablet device.
• Office Hub for Windows Phone – Is an application for Windows phone devices that provides enhanced capabilities to access SharePoint content from multiple places in one spot. It also leverages mobile Office.
• Location – There is a new geo-location field type that is available in a SharePoint List. This can make a list location aware to capture latitude and longitude which can be used with map applications. For instance, if a user enters in data on their mobile device, it will capture where it was done from and then can be displayed on a map. Here is some more information about this - http://technet.microsoft.com/en-us/library/fp161355(v=office.15).aspx
• Push Notifications – There is a new capability to allow notifications to be sent from a SharePoint site to registered applications running on a mobile device. The nice thing about this is that Windows Phone Apps can receive notifications without having to poll. Here is some additional reading on the topic - http://msdn.microsoft.com/en-us/library/jj163784(v=office.15).aspx
• Device Channels – This is a really important new capability as device channels allow you to deliver a publishing site geared specifically to support different types of remote devices. Basically the site can mapped to multiple master pages and style sheets and even control what content you want to make available to specific devices. Here is an overview on the new device channels - http://technet.microsoft.com/en-us/library/fp161351(v=office.15).aspx
• Office Web Apps – As mentioned earlier in this posting, Office Web Apps is now a separate standalone server which does not run inside of the SharePoint boundary. Office Web Apps has been improved a lot to support mobile devices. There are Word, Excel and PowerPoint Mobile Viewers.

SharePoint 2013 Upgrade Process Diagrams

There are two upgrade diagrams that have been provided. Here are some high points I walked away with:

• Must be on SharePoint 2010 – To upgrade, you must be on SharePoint 2010 technologies. This means if you are on SharePoint 2003, 2007 or 2010, you will need to upgrade to the appropriate version to get to SharePoint 2013. There are Microsoft migration partners that have solutions to assist with this. I saw many times, this is the big value proposition for using SharePoint Online as this is handled for customers.
• Database Attach Upgrade – Is the only supported method for upgrading. There is no more “in-place” upgrade option. Frankly that is fine because most customers always went down a database-attach upgrade.
• Preparation – much of the preparation activities that we have discussed in the past with SharePoint 2010 hold true with SharePoint 2013. There is a bunch of information you are responsible for gathering.
• Manual Configuration Settings – In the preparation phase is recommended to get a understanding of all the custom configurations that you may have done because not all of them are going to be migrated. This is because not all databases are upgraded. So many custom configurations in central admin such as alternate access methods, time job tweaks, managed paths, incoming/outgoing email settings, certificates, etc. will need to be documented and reconfigured in the new farm.
• Databases That Can Be Upgraded – There is a set of databases that can be upgraded. They are Content, BDC, Managed Metadata, PerformancePoint, Secure Store, Search and User Profile databases.
• Customizations – This is an important task that needs to be completed. I have seen many cases where good software organizations have not implemented a strong configuration management process and the result is an organization may not know about all the customize code that may be implemented. There are numerous ways to find all of them by running PowerShell commands, doing system directory diffs, checking web.config, etc.
• Upgrade Health Checks – There are some new features that are available to site collection administrators that will show you a health check of a site collection before actually upgrading the site collection.
• Evaluation Site Collection – Site Collection Administrators also have the ability to request the site collection be copied into a new site collection to evaluate how the upgrade will affect any customizations they may have. This is helpful so you can remediate issues before you actually perform the upgrade. This is also nice because your site collection will run in a SharePoint 2010 mode until you are ready to actually upgrade it.
• Testing – Just like for SharePoint 2010, the best way to prepare for a migration is to build up your new SP 2013 farm and then multiple practices runs of that upgrade into the new production environment. An entire process is defined in one of the diagrams and is a great place to start.

SharePoint 2013 Search Diagrams

If you are a reader of my blog, I wrote some long postings about the search architecture for both SharePoint 2010 Search (here) and FAST for SharePoint 2010 (here). I am not going to do a deep dive into all these search components and roles because they are basically covered. As many people now know, the FAST search engine is now the core search engine for SharePoint. It will just be referred to as SharePoint Search. Now you will be able to leverage a very powerful search engine out of the box. However many of the advanced enterprise features of search will only be available in the SharePoint Enterprise addition. I am also really excited about this for SharePoint Online because it can leverage FAST too. SharePoint Online will not be able to do Enterprise Search of line of business systems but a Search Farm (which is FAST underneath the hood) can be configured on premise and SharePoint Online can invoke that search and provide the search results in the cloud; pretty exciting.

I highly recommend taking the time to review both of these diagrams. It explains how each of the components interacts with each other. Additionally there is a diagram the goes into how to scale the server farms for the amount of content you will need to index. There is a great, new table in there that shows you how scaling will work. To be honest, the folks who are really serious about search will say it is an art and a table does not always communicate how you will do it. It always comes down to how many items, the types of data sources, custom transformations, query latency, index latency, etc.

SharePoint 2013 App Overview Diagram

This is an area I plan to do a lot more exploration of this coming year on and writing on this blog. Why? This is something we have been waiting for a long time with SharePoint development. There are several ways to look and this. SharePoint Features which we have been writing for years are Apps. This is name change to better communicate our technology to the end users who have to use SharePoint. However the new SharePoint App architecture is way more than that.

I have seen so many things over the years.

• I think one of the biggest challenges people would run into is developing great SharePoint solutions only to find out they incorporated some dependency they should not have, they wrote some high-end code that should not be running in the SharePoint layer, they cannot leverage their solution outside the SharePoint boundary, etc. We want to resolve those problems by helping developers to deploy solutions in way that will keep their SharePoint environment nimble.
• Plus we want to provide third-party vendors quicker access to customers. We want to help customers to quickly acquire third-party solutions.
• Additionally we want to allow customers to leverage commodity based SharePoint Online. As you may know SharePoint Online has restrictions on high-end custom development and if that code where to run in another location, while be highly integrated with SharePoint Online, that is a huge win.

I will thing of many more reason this year on why this is so great J

Now we can achieve this through the new SharePoint App architecture. The old SharePoint Solution architecture where you create a WSP is still around. Nothing has changed there. This is used to create deployment packages and in many cases is used to deploy code that requires full trust. SharePoint Solution packages will continue to be used by third-party vendors or developed internally with such tools as Visual Studio. You can still create Sandbox Solutions which run in a more secure runtime and can be deployed in SharePoint Online.

Now the new Apps framework for SharePoint 2013 is a packaged up in a file called .APP. It is composed of many of the same types of files, AppManifest, embedded Solution.wsp, etc. Once an app is loaded into SharePoint, it is accessible through the App Catalog in SharePoint. This App Catalog can be controlled at an organizational level.

Remember the big point with Apps is, that the custom solution you are writing may or may not actually run in SharePoint. Full trust code is not supported. Your custom solution code itself may run in a different SharePoint farm, on an IIS server as ASP.net pages, ASP.net pages running in Windows Azure, etc. So how does SharePoint access these solutions running outside the SharePoint context? In simple terms we have an IFrame (with some extensions) that external solution is available through. OAuth provides the secure connection for access SharePoint objects from a remote location. We will additionally use a new extended and robust event model and remote client SharePoint library to write integrated, remote code.

Why is this so great? We are going to ensure that custom applications and solutions that are being developed with SharePoint are isolated. No more writing a bunch code and services that should not be running in SharePoint servers. It is great that you can do whatever you want with SharePoint, however this will drive solution management.

So you may be asking where does this get deployed? There are many different options for hosting.

• SharePoint Hosted – This means the app and all the resources run in SharePoint. Remember you server side is not supported however you can write applications with SharePoint’s JavaScript libraries and such.
• Windows Azure Autohosted – This is a model that is only supported in SharePoint Online. In This case you can write an App package that will have code for Azure and SQL Azure embedded into it. When the application is deployed, the azure solution will be automatically deployed for you. You do not have to go to Azure and set anything up at all; it is all handled for you behind the scenes.
• Provider-Hosted – This is the third model where custom code and solutions are hosted in a separate server in your organization, hosted in Azure, hosted in different SharePoint servers, etc.

Once an App package is installed, it can be managed and monitored through the catalog. End users have the ability to select an app to run in their sites (much in the same way as turning on a Feature). If and when an app is updated, the user can decide how they want to upgrade to the new app.

Again I really plan to go much deeper on this in my blog but for right now, these are just some introduction notes and ramblings on how excited I am about this new capability J

Back Up and Recovery Diagram

There is a new diagram that goes into the details of doing your own back-up and recovery for SharePoint 2013. I know many people have become accustomed to using third-party vendors for supporting these operations and I still believe these vendors will continue to provide features above and beyond what is out of the box. However, if you are a do it yourself sort of person, this is a great diagram to review.

Not much has changed in regards to developing back-up procedures for both the SQL Servers and the SharePoint Servers. There are tons of scenarios covered in here, and I recommend reading this if it is important to you.

SharePoint 2013 Database Diagram

Finally the database server diagram has been updated. This is a really really really important diagram to review if you are managing on-premise servers. It goes over all the SharePoint databases, plus provide sizing and scaling guidance. Great information.

Updated Independent Analysis on Microsoft Productivity Solutions for Prem and Cloud

There have been a lot of updates with Gartner analysis for the Microsoft Productivity stack and other related stacks that I work with. Specifically SharePoint, Office 365, Lync and BI have all been getting great independent reviews. The important take away that I discuss with my customers is that all these top quartile rated technology is delivered on a single, integrated solution stack. They are NOT hodgepodge solutions brought custom integration.
Many of the reports and newest reports can always be accessed from here - http://www.microsoft.com/presspass/itanalyst/default.mspx
SharePoint and Office 365
As you can see we have been great assessments for enterprise content management, enterprise search with FAST for SharePoint 2010, great recognition as portal provider, social computing and business process analysis with Visio Services and SharePoint workflow. Much of this capability is delivered through the Office 365 cloud as well.

• Magic Quadrant for Enterprise Content Management – October 2011 - G00219745 - http://www.gartner.com/technology/reprints.do?id=1-180UJ6N&ct=111116&st=sb
• MarketScope for Enterprise Search – November 2011 - G00216488 - http://www.gartner.com/technology/reprints.do?id=1-1835DKL&ct=111123&st=sb
• Magic Quadrant for Horizontal Portals – October 2011 - G00219245 - http://www.gartner.com/technology/reprints.do?id=1-17RRLXF&ct=111025&st=sb
• Magic Quadrant for Social Software in the Workplace – August 2011 - G00214779 - http://www.gartner.com/technology/reprints.do?id=1-173SL7W&ct=110826&st=sb
• Magic Quadrant for Business Process Analysis Tools – December 2011 - G00219247 - http://www.gartner.com/technology/reprints.do?id=1-18EUESV&ct=111221&st=sb

SQL Server and BI
As you can see these are very recent reports giving us good reviews in the areas of BI and Data Warehouses. I think the big take away is again total cost of ownership (TCO). We give strong BI solutions and customers do not need to make significant investments to deliver BI through the same SharePoint investments discussed above.

• Magic Quadrant for Business Intelligence Platforms - Feb 2012 - G00225500 - http://www.gartner.com/technology/reprints.do?id=1-196VVFJ&ct=120207&st=sb
• Magic Quadrant for Data Warehouse Database Management Systems – Feb 2012 - G00219281 -http://www.gartner.com/technology/reprints.do?id=1-196VVFG&ct=120207&st=sb

Exchange and Office 365
For email with Exchange either delivered on-premise or through the Office 365 cloud we continue to be evaluated as a leader. This really speaks to our versatility in allowing customers to select the most appropriate place and get the same level of end user experience for business users.

• MarketScope for Email Systems – June 2011 - G00213356 - http://www.gartner.com/technology/media-products/reprints/microsoft/vol14/article22/article22.html
• Magic Quadrant for Secure Email Gateways – August 2011 - G00214080 - http://www.gartner.com/technology/reprints.do?id=1-16XZCJZ&ct=110811&st=sb
• Magic Quadrant for Enterprise Wireless Email Market – December 2011 - G00219736 - http://www.gartner.com/technology/reprints.do?id=1-18KGC8M&ct=120105&st=sb

Lync and Office 365
Microsoft is again recognized as a leader in all these types of solutions with Lync which can additionally be delivered through the Office 365 cloud. Our Unified Communications solutions are completely integrated in Office, Outlook and SharePoint allowing users to quickly transition between tasks they are working on making them highly productive. These capabilities are extremely useful for telework or communications between people spread across large geographies.

• Magic Quadrant for Unified Communications – August 2011 - G00214025 - http://www.gartner.com/technology/reprints.do?id=1-1728DMD&ct=110823&st=sb
• Magic Quadrant for Web Conferencing – December 2011 - G00219266 - http://www.gartner.com/technology/reprints.do?id=1-188LI4E&ct=111207&st=sb
• MarketScope for Enterprise Instant Messaging and Presence – July 2011 - G00213790 - http://www.gartner.com/technology/media-products/reprints/microsoft/vol14/article7/article7.html
• Critical Capabilities for Unified Communications as a Service – March 2011 - G00210950 - http://www.gartner.com/technology/media-products/reprints/microsoft/vol14/article18/article18.html
• Magic Quadrant for Corporate Telephony – September 2011 - G00215455 - http://www.gartner.com/technology/reprints.do?id=1-17DCX5P&ct=110920&st=sb

Architecture Considerations for Moving SharePoint to Office 365 and SharePoint Online

Introduction
Why are customers looking to come to the Office 365 cloud? Customers want an environment that can scale on-demand. Customers do not want to be in the business of managing and patching software . Customers want a solution that give better business continuity and service level agreements to their users. Customers want solutions that are better governed and that will force them to do better governance. Customers want to be in the business of building solutions (i.e. an airplane company should focus on building planes; not writing enterprise applications from the ground up).
When I talk with customers about SharePoint Online, this is where they want to move to.
In this blog I am going to talk about how an organization should be looking at moving to SharePoint Online.

1. In first half we are going to analyze SharePoint Architecture and Governance and review what potential blockers people see when moving to the cloud.
2. In the second half I am going to have a more detailed discussion around how to architecturally plan to move to SharePoint Online.

Why the decision is not always straight forward?
Coming to SharePoint Online in Office 365 may not sometimes be the most straight forward decision for organizations that have made a large investment in SharePoint. Going to SharePoint Online is really easy for organizations that are in it for document management, intranets, OOB features, SharePoint Designer, etc. It gets harder for organizations that have had poor governance of their on-premise environments, made signification investments with custom code, utilize features not available in cloud, have a heavy reliance on third party solutions, etc.
There is a solution to these issues but it requires an organization to take a step back and look at what they have.
Poor governance is probably one of the largest changes organizations have faced with SharePoint. These challenges could have been avoided thought with good forethought and planning. There is not always a one size fit all solution either; SharePoint is powerful platform that can be used to be used to implement a very broad set of business requirements.
SharePoint Architecture Foundations
The following a Venn diagram that I always draw on whiteboard with customers.

I typically say that that:

• Information Architecture – Is content types, taxonomy, topology, site collections, sites, libraries, lists, and solutions that drive the management of this content. The information architecture should be driven by business requirements.
• Logical Architecture – Is the architecture of SharePoint services, web apps, databases, etc. to support the information architecture.
• Physical Architecture – Is the configuration, deployment, farms, etc. that actually host the SharePoint logical architecture.

I admit this is open to interpretation but let’s use this as a foundation for discussion.
Here are some of the biggest problems organizations that have come up when I discuss the diagram above:

• Started with Physical Architecture – This is the first mistake that many seem to make. Smart IT are concerned about how many servers to buy and configure they forget to actually create an environment needed to support their business requirements. What usually happens is the environment is not scaled or organized to support the actual utilization; this is where the trouble starts. This pretty much goes away if you go to the cloud.
• No business requirements and software design best practices – One of the biggest, consistent challenges I see many organizations have with SharePoint is no business requirements. I typically just see organizations create very light documents (a bullet list) of requirements and just start building. I very rarely see organizations actually do software design best practices. There are no use cases, data models, ERDs, UI wireframes, data dictionaries, architecture documents discussing what SharePoint elements they plan to use, etc. Why; because SharePoint is such an easy platform to start building with. I personally believe SharePoint is one of the best solution platforms on the market but sometimes organizations need architect a solution fist. This does not go away with SharePoint Online.
• No Governance – Organizations usually forget to put Governance plans to manage content and solutions running in SharePoint. A solution can be anything from a group of team sites or a complex .NET application in SharePoint. I have even seen organizations create Governance plans but then not manage nor adhere to them. An analogy would be a organizing a ton of boxes in their basement really well (with labels too). However as time goes on, they just keep on stuffing and stuffing the room with more boxes so that entry to the room is blocked. A good SharePoint environment needs care and feeding. The information, logical and physical architectures need to have policies and procedures to manage it. Policies, procedures, service level agreements, etc. must be put into place. With SharePoint Online several of these Governance woes are removed because the Office 365 cloud manages the environment. Still Governance needs to be but in place to manage content and solutions deployed in Office 365.
• Over Regulation – I have also seen the term “SharePoint Governance” used as a crutch. I have seen organizations lock down SharePoint too much. My response to that is was there a requirement to lock it down? If so, the right thing was done like a publishing intranet or Internet site – no user should have rights to do what they want. There should be a locked controlled publishing and branding process. However it is perfectly acceptable to have areas of SharePoint that are pure collaboration. Your information architecture and governance plans will drive business users to put content and solutions in the appropriate areas.

There are more things to think about but I am getting a little preachy J
A Not So Unusual Situation
Here is a very common scenario that backs up what I just described. It is very common for an organization to start a SharePoint environment like the following. As you can see they start with a top level SharePoint site. They create some department level sites and some sub sites for their intranet. All in all this is a pretty good start; right?

However within a few months (after some very heavy business and user adoption) we have the following.

We see such things as:

• Team sites starting to sprawl underneath some of the sub-sites within a department. The challenge is being able to support collaboration in what was meant to initially be a publishing site.
• Navigation, presentation, branding and user experience is not consistent.
• We see custom applications either built from scratch or third party solutions purchased and embedded in sub-sites of a department. The challenge is should these custom solutions be hooked directly here?
• Department level project sites are created. The challenge is that as project sites grow and take on new responsibilities for the business, they need to be elevated and made accessible like other sites.

There tons more. And I have had customers tell me this is not an issue with “SharePoint”. They currently have these same exact challenges with their legacy intranet, portals, etc. So why do organizations keep on having these challenges? I usually point to a lack of attention to Governance and Information Architecture.
So How Should You Be Thinking About This?
What if content and solutions that are to be deployed are delivered within a framework? Not a “novel” idea either. Hopefully we can make it as simple as we can. Instead of just adding more and more sites and applications; have rules that drive where we put things.

For example:

• The Intranet site should remain a dedicated publishing site collection. This would be geared towards business users having read-only access with only a small set of content owners. Content is delivered in a consistent and clean fashion so that every site is the same giving the user a unified user experience.
• Create a separate site collection for team sites with appropriate service level agreements. In the team site collection, sites should be can be dynamic generated from pre-define site templates. Users are given the ability to do pure collaboration, standing up lists, libraries, etc. They should be able to share information to complete everyday business tasks. There would also be an expectation that this area is not uniform and that users can do what they want with these sites. Retention polices can be put in place to discard these sites if they have not been used for a pre-defined period of time.
• Create another site collection for business project sites. These have different support SLAs and Governance. These sites are a little bit more formal in nature with only a subset of people that would have access to manage them. Maybe there are designers from the IT department who have responsibility for building and supporting them. There could be policies and automated procedures to move content out of them to other site collections. Retention policies on these sites would be completely different than team sites and these sites may only be deleted when the project is over (or never deleted).
• For custom applications, instead of embedding the application into a sub-site, place them into their dedicated site collections. Navigation links can be made to the application from other site collections. This would give significantly more flexibility to move it or provide it more resources.
• Instead of creating one massive site collection where all scanned documents or records create multiple site collections and then route content to them based on business rules.

This list I provided is not meant to be an all-encompassing list of possibilities and/or solutions. Hopefully you will see that you need to start vertically and horizontally partitioning solutions and data based on such simple things as business rules, security access, data characteristics, etc. Knowing this will help you identify the types of site collections, sites, features, content types, managed metadata, site templates, permission levels, etc. that you need to configure SharePoint with.
The net result is you would take the previous diagram and start creating management boundaries based on the characteristics of the solutions and data managed in SharePoint. This is why Information Architecture is so critical for Enterprise Content Management system like SharePoint.
Putting everything into a single site collection really puts an organization into a tough spot to scale with the business. Microsoft has been publishing great technical diagrams for SharePoint 2010 (http://technet.microsoft.com/en-us/library/cc263199.aspx) and SharePoint 2007 (http://technet.microsoft.com/en-us/library/cc263199(office.12).aspx). Please review and get to know these diagrams well because they accurately tell everything I am talking about here.
Now some people counter this whole thing with why does SharePoint not give this to me? Why does SharePoint not auto-govern itself? SharePoint absolutely comes with a ton of features and capabilities that support a good governance model like Features, site templates, sandbox, managed metadata service, permission levels, and the list goes on. All of the configuration settings are there and organizations just need to set them as appropriate for their business.
I promise; we are coming back to the SharePoint Online cloud but we need to finish setting the stage.
How You Should Change Your Thinking
The following is a diagram a colleague (http://blogs.msdn.com/b/edhild/) and I commonly discuss with customers. I am shamelessly using it because this simple diagram really helps customers with a basic understanding of Information Architecture with SharePoint. Plus it supports everything that I just discussed. This is what we call the “Arch of SharePoint Data”. This is not all encompassing list, however there are different extremes.

There is publishing which are sites that are managed by a small group of users and read by a large community of read-only users. While on the other side is something like MySites which is my personal area to manage information and data. In between these two extremes are tons of different types of sites. Department sites, project sites, organization sites, custom application sites, team sites, etc. There are too many to even try to draw. Each one of these types of sites has different security, retention, data usage, transaction management, business rules, automation, etc.
I tell customers is that is perfectly ok to have a team sites area that are for pure collaboration where business users can spin-up a site, do some work on it for two months and then move on. Some people call this the “wild wild west” and that is ok. Just do not allow pure collaboration in you publishing area which is probably one of the most common mistakes J.
Coming full circle, you can see this is all about solution and data management. An Information Architecture is going to tell you the data characteristics. This will drive you to put content in one area versus another.
This is the beauty of SharePoint. It allows you to create and manage business workloads based on your specific business and mission. Once you have the Information Architecture nailed down, you can determine the Logical Architecture (services) you need and then the Physical Architecture needed to support this. By doing this, you know that both your Logical and Physical Architectures will be driven off real business requirements.
Where Does this All Fit with SharePoint Online?
Once you build an Information Architecture you will see that many of these partitions can be moved to the cloud. Intranets, team sites, project sites, my sites, light weight custom solutions, etc. can all be moved to the cloud. If you look at the “Arch of SharePoint Data” diagram, depending on your scenarios, there is a good chance that 80% to 90% of your solutions can be moved to the SharePoint Online cloud. For some organizations, they will be able to move everything up to the cloud. For some organizations they will have a hybrid.
Regardless the re-architecture should be a primary task in your SharePoint 2007 migrations to SharePoint 2010 anyways. Going through this exercise will put you in a position to move pieces to the cloud and with time, more and more.
So what are the Gaps with SharePoint Online?
If you want to know the exact answer it is clearly spelled out and completely available for you. Read either the Multi-tenant SharePoint Online Service Description (http://www.microsoft.com/download/en/details.aspx?id=13602) or the Dedicated SharePoint Service Description (http://www.microsoft.com/download/en/details.aspx?id=18128).
However really understanding these gaps depends on the perspective and approach you are taking with SharePoint. As I say a lot, “SharePoint means a lot of different things to different people”. I have seen customers extremely happy with using SharePoint with out of the box features and SharePoint Designer. While I have seen other customers use SharePoint as a full application development platform writing thousands lines of code. Knowing what SharePoint means to you will dictate your approach to the SharePoint Online cloud. The approach that I outlined in the first part of this bog will really help you with that decision process for moving to the cloud.
When you read the SharePoint Online service descriptions for both the multitenant and dedicated you will see that the gaps are really small. However there are some ones you must be aware of as they will affect your decisions on how to move to the cloud. The big ones that most people bring up are full trust code, business intelligence, FAST search, PowerShell, and Central Administration. This list will quickly change and become outdated because more and more features will be released with time. However let me address each one as it stands today:

• Full trust code is usually the first one the first challenges. Right now, the only way to deploy custom code to the SharePoint Online Multitenant cloud is using a Sandbox solution. For the SharePoint Online Dedicated cloud, full trust code is supported but it must adhere to a strict set of rules (which are publically available) and the code will only be deployed within set windows. Why such restrictions and strong governance? Well, for all the obvious reasons that would come up if you were tasked with having to run an extremely large SharePoint environment on-premise. What has been one of the biggest issues with SharePoint 2007 Governance? It was developers writing complex code on SharePoint and disregarding the fact that an error they write may take down other sites in the farm (like the content query web part that retrieved too much data on the home page of the intranet <g>). We need to ensure that there is strong security and good performing code and that there is no possibility the Company A can take down Company B. The only way to achieve that service level agreement is to have an environment for running governed code. The net effect is that there will be limitations but you will have that guaranteed uptime. My golden rule is that all SharePoint development (on-premise and cloud) should begin with Sandbox solutions and only when the customizations cannot run in the Sandbox then build as full trust solutions. Doing this will ensure you have agility to move to the SharePoint Online cloud when you are ready. If you really need to do complex operations and manage data structures consider using services Windows Azure integrated through Business Connectivity Services (BCS), Silverlight, etc. But still that may not always suffice and that is why SharePoint Hybrid implementations will be commonplace for organizations with mature SharePoint deployments. I will cover this in more detail shortly.
• Business Intelligence today in Office 365 is Excel and Visio Services. Other SharePoint business intelligence services like PerformancePoint, SSRS, PowerPivot and Chart Parts are not available right now. Another limitation today is Excel and Visio services only utilize data that is within the SharePoint Online context; it is not able to reach outside (i.e. to back-end databases). More and more capability will be released through the Office 365 cloud; just for the time being Silverlight and Windows Azure can be used to satisfy these requirements.
• FAST Search is not available in the SharePoint Online cloud right now. It is possible to integrate a local FAST farm with SharePoint Online Dedicated; but not Multitenant. Still take comfort in the fact that SharePoint 2010 search made significant jumps forward from the SharePoint 2007 offering and provides a very strong search experience. With time more and more advanced search capabilities will be released in the cloud.
• PowerShell currently is not fully available with SharePoint Online. There are a lot of PowerShell commands available for Exchange Online and user subscription management; however the full set of SharePoint PowerShell commands is not available today.
• Central Administration Site is not available and this should be expected by anyone who understands what cloud architecture. There are administration screens available to some operation that you would normally perform in Central Admin however it is limited and does not give you the granular control. Why? Well this is the cloud. Customers want to come to the environment so they do not need to be in the business of managing every little configuration of SharePoint.

And the reality is with more and more releases of SharePoint Online this gap is going to continue to close. Still, even if it were to completely close, there will be perfectly valid reasons why some SharePoint may remain on-premise thus, creating a SharePoint Hybrid environment.
What is the SharePoint Hybrid Architecture?
SharePoint Hybrid is as simple as it sounds; it is some SharePoint delivered through the cloud and some SharePoint delivered through on-premise. What will drive you to have SharePoint on-premise? All of that has pretty much been covered to this point. Anything from a specific feature to a business policy may keep some SharePoint on-premise. However using the approach I laid out, you will be able to significantly reduce your footprint of SharePoint on-premise and gain the advantages of the cloud.
The great thing about using SharePoint on-premise is that it is the same software being run in the cloud. This means it is very easy to deliver a consistent user experience between on-premise and the cloud. Branding, navigation, security groups and single sign-on can be configured in such a way that the user can go between these two environments and not know it.
What Type of Cloud is SharePoint Online?
One other thing I want to discuss is what type of cloud is SharePoint Online. I really like this picture it really spells it out for people whom are not fully aware of the multiple delivery models for cloud computing.

Starting on the left side, you see on-premises and this is how most organizations run SharePoint. You must own the entire solution; all the way up the computing stack. Next is Infrastructure as a Service (IaaS) which is the cloud environment that is managed all the way up to the virtualization layer. The company is responsible for everything else; including the management of the operating systems. From a SharePoint perspective that means all the software installation, configuration, management, patching, adding new servers to meet demand, load balancing, etc. needs to be managed by the you. IaaS gets you of the business of hosting virtual SharePoint servers.
Platform as a Service (PaaS) is running the environment all the way up to the application and data tiers. Windows Azure is a PaaS cloud. In this cloud you build custom applications, data models and run them through a highly available environment.
Software as a Service (SaaS) is the entire stack delivered in the cloud and this is the delivery model for SharePoint Online. You do not have to install software, manage patches, availability, etc. This environment gets you out of the business of managing software and into the business of building solutions. SaaS does not provide the granular level of control of the SharePoint environment (which we have discussed).
The reality is that organizations and companies need to save costs and SaaS is the cloud delivery framework they want for the long-run. IaaS can run SharePoint 2010 and can be leveraged as a replacement for on-premise complex SharePoint computing. Still it is well recognized by industry that companies want more SaaS solutions.
Supplement with SharePoint Online with Windows Azure
Windows Azure (PaaS) and SharePoint Online (SaaS) can be used together to deliver end-to-end cloud solutions. Companies that have mature SharePoint deployments commonly have:

• Code that runs in full-trust
• Are managing complex data
• Require the ability to do back-end systems integration.

One thing I have been talking with customers about is offloading that code out of SharePoint and into to the Windows Azure (PaaS) cloud. The Windows Azure cloud allows you to develop and deliver custom code, services, complex data (SQL Azure) and connected back-end integration (AppFabric). These complex operations can be connected through Business Connectivity Services (BCS) in SharePoint Online.
This makes a lot of sense too when you take a step back. I have already said that you should develop SharePoint code to the Sandbox first and when there is good reason to deliver outside the Sandbox. Here is a similar question. At what point do you know you should be developing in SharePoint? I fully recognize there is a gray area here.
I say good software development patterns and practices should drive that decision. This is why Windows Azure is so interesting with Office 365 because we can move complex code that cannot run in the SharePoint Sandbox to the Windows Azure cloud. I recognize this is not a perfect rule because some code needs to run in SharePoint as full trust. However this should be part of your design analysis to reduce to your SharePoint on-premise footprint.
There are lots of different ways Windows Azure can be utilized with SharePoint Online. There may be situations where you need:

• To work with data in SharePoint but you have complex relationships in the data model that are not right for SharePoint lists. Use SQL Azure to manage that data and build services and connect via BCS or Silverlight.
• To integrate with line of business applications on premise use custom services deployed in Windows Azure or AppFrabic. Again BCS or Silverlight can connect Windows Azure which is conduit to line of business applications.
• To use custom web services to perform complex computations and logic. Again offload that to Windows Azure.
• As well, reverse the direction. There may be solutions and applications delivered in Windows Azure can utilize SharePoint Online services. There are SharePoint web services, REST services for data, SharePoint Client API, Javascript APIs and Silverlight APIs that can be used as integration points. A simple scenario could be a web page deployed in Windows Azure that needs to manage documents. Instead of building that up from scratch in Windows Azure just connect through SharePoint Online APIs and deliver SharePoint Online services through that custom web page.

At the end of the day both SharePoint Online and Windows Azure can be used to complement each other in the delivery enterprise business solutions through the cloud.

Conclusions
Why did I go through all of this? The answer is simple; to give SharePoint Architects ideas on how to move forward with SharePoint Online. There really should be no blockers as long as you take a realist look at your Information Architecture and assess how you use SharePoint. There will be lots of stuff which can clearly be moved to the cloud, there will be some stuff where it is not appropriate (hybrid) and then there is that gray area. However I really hope that approach I put forth will help you with your thinking into how to significantly reduce, if not completely remove, your SharePoint on-premise footprint.
Thanks
Special thanks to Chris Geier and Stephen Cawood for providing me feedback and advice as I wrote this.

Gartner Rates Microsoft a Leader in Enterprise Search

I am happy to say the Microsoft has again been ranked as a top Enterprise Search provider in the marketplace - http://www.gartner.com/technology/reprints.do?id=1-1835DKL&ct=111123&st=sb. You will see that some Enterprise Search providers dropped in rankings and some were removed.

As you can see, FAST is not slowing down and we have an extremely strong roadmap for it. It is a very fiscally responsible Enterprise Search solution when you consider it as part of your Total Cost of Ownership (TCO) when investing with SharePoint Enterprise. FAST provides tons of added features and capabilities to solutions such as Portals, Sites, ECM, Document Management, BI, Social, Composites, Platform Development, etc. Plus if you know how to configure SharePoint Search; FAST search is configured and managed through the same screens and procedures thus requiring less ramp up time.

Even if an organization just needs a pure Enterprise Search solution, FAST is a great Enterprise Search platform. Having SharePoint in the mix, is great value additive because many other Enterprise Search ends provide you no user interfaces. Yes FAST does have services so that other applications tiers can tap into it. However having a web portal like SharePoint immediately available with numerous web parts to start building search screens and search applications is a huge time and money saver. Plus, when Enterprise Search is made accessible through SharePoint, you get to take advantage of all the other SharePoint features to deliver a more engaging user experience. For instance:

• You can create blogs from the search team to the users
• Wikis to explain to users how to do advanced search
• Stream training videos
• Maintain documents and FAQs
• Create a Search Knowledge Base
• Create support forums
• Manage technical specifications about line of business data that is index
• Etc.

FAST for SharePoint 2010 Case Studies

There is a new FAST Case Study that for FAST for SharePoint 2010.  IMF - http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=4000011274

Here are two other good FAST for SharePoint Case Studies:

• Case Study – General Mills Research (FAST) - http://www.microsoft.com/casestudies/Microsoft-FAST-Search-Server-2010-For-Sharepoint/General-Mills/General-Mills-Frees-More-Time-for-Innovation-with-Research-Focused-Search-Application/4000007255
• Case Study – Mississippi State Department of Transportation (FAST) - http://www.microsoft.com/casestudies/Microsoft-SharePoint-Server-2010/Mississippi-Department-of-Transportation/State-Transportation-Department-Saves-Lives-with-Better-Business-Insight/4000007073

SharePoint Search Index Document

I received several questions about indexing of SharePoint content lately. Specifically does SharePoint index the entire document?

First off if FAST is being utilized, yes, the entire document will be indexed.

Second if you are using the out of the box SharePoint search by default it configured to index the content of items that are under 16MB or under. Metadata associated to these documents will still be indexed (so if the big files are videos they will still be searchable).

The max file size for indexing can be upped up to the max file size limit; which is 2GB. Before upping the size, it is recommended that you do some testing and understand any performance ramifications relative to your environment and data. Here are some good references on the topic.

• http://www.toddklindt.com/blog/Lists/Posts/Post.aspx?ID=215
• http://appsense.logimindz.com/?p=27

SharePoint 2010 Independent Market Analysis

Updated - 3/2012 - View new blog posting here

I get asked a lot about reasons to move to SharePoint 2010. I simply state, look at the how the market evaluates SharePoint.
Here is a great link managed by Microsoft that showing Gartner and Forrester reports - http://www.microsoft.com/presspass/itanalyst/default.mspx

As of right now, I recommend reading the following:

• Enterprise Content Management (ECM) – View Report
• Business Intelligence (BI) – View Report
• Enterprise Search – View Report - Just Updated in Nov 2011 and continued leader - View Report
• Social Computing – View Report - Just Updated Aug 2011 (continued to be leader, here is the 2010 report)
• Portal Platform – View Report
• Unified Communications – View Report and View Report
• Email – View Report

I suggest stay up to date with this site to find the latest reports.

FAST Ampersands in Custom Property Extraction Dictionary

If you are creating a Custom Property Extractor for FAST for SharePoint 2010 be careful because careful to set up your Extraction Dictionary correctly. I recently had a colleague create a new extraction dictionary like in here - http://msdn.microsoft.com/en-us/library/ff795797.aspx

Unfortunately I do not have the logs but my colleague was telling me that FAST was just hanging when trying to execute the FAST crawl. I looked at the modifications he made and took a guess that the ampersands he had throughout the custom property extraction dictionary XML definition could be the issue. I recommended he remove the ampersands and replace with &amp. Well that took care of the issue.

FAST QRProxyService Dead

Issue

I was recently installing FAST for SharePoint 2010 and when I went to check the status after the initial configuration it was saying the QRProxyServices was dead.

Module Name Process Name PID Status

------------------------------ -------------------- -------- --------------

Browser Engine browserengine Running

Config Server configserver 4048 Running

Content Distributor contentdistributor 2464 Running

Enterprise Crawler crawler 5516 Running

FDMWorker fdmworker 5084 Running

RTS Indexer indexer 3792 Running

Indexing Dispatcher indexingdispatcher 3048 Running

Name Service nameservice 2652 Running

Document Processor procserver_1 5584 Running

Document Processor procserver_2 5604 Running

Document Processor procserver_3 5640 Running

Document Processor procserver_4 5664 Running

QRProxy Service qrproxy Dead

QRServer qrserver 4928 Running

SAM Admin samadmin Running

SAM Worker samworker Running

RTS Search search-1 3672 Running

Automatic spellcheck tuning spelltuner 5452 Running

SPRel sprel 5384 Running

RTS Top Dispatcher topfdispatch 4460 Running

WALinkStorerReceiver walinkstorerreceiver 5140 Running

WALookupDB walookupdb0 5076 Running

WebAnalyzer webanalyzer 5232 Running

Error

I then went to the event logs and say the following error.

Service cannot be started. System.IO.FileNotFoundException: Could not load file or assembly ‘Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The system cannot find the file specified

Solution

The resolution was pretty simple, I had to re-install WIF.

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=eb9c345f-e830-40b8-a5fe-ae7a864c4d76&displaylang=en

Part 6 - FAST for SharePoint 2010 References

• Part 1 - FAST for SharePoint 2010 Features and Proposition
• Part 2 – FAST for SharePoint 2010 Logical Architecture
• Part 3 – FAST for SharePoint 2010 Physical Architecture
• Part 4 – FAST for SharePoint 2010 Farm Configuration
• Part 5 – FAST for SharePoint 2010 Service Configuration
• Part 6 – FAST for SharePoint 2010 References

References

Below is a full list of all the resources I used for this series on FAST for SharePoint.

Background References

• SharePoint 2010 Service Architecture - http://www.astaticstate.com/2010/01/sharepoint-2010-service-architecture.html - This will give you a brackground if you do not have one.
• SharePoint 2010 Search Architecture - http://www.astaticstate.com/2010/12/sharepoint-2010-search-architecture.html - This will give you a background on the out of the box SharePoint 2010 Search.
• FAST ESP Series - http://www.astaticstate.com/2010/01/fast-components-introduction.html - This will give you a background of the FAST ESP 5.3 product. The components and architecture are very much the same.

FAST for SharePoint Planning and Architecture References

• FAST Service Blog - http://blogs.msdn.com/b/arnts/archive/2010/01/27/the-two-types-of-search-service-applications-in-a-sharepoint-2010-deployment-with-fast-search-server.aspx - diagram I used to explain the types of FAST services.
• FAST for SharePoint Architecture Blog - http://blogs.technet.com/b/vedant/archive/2009/11/13/information-on-fast-search-server-2010-for-sharepoint.aspx
• FAST Search Server 2010 for SharePoint Resource Center - http://technet.microsoft.com/en-us/enterprisesearch/ee441234.aspx
• FAST for SharePoint Videos - http://technet.microsoft.com/en-us/enterprisesearch/ff960998.aspx
• Deployment and Scalability for FAST Search Server 2010 for SharePoint Video – may be a bit boring but this is a really good one to understand scaling of the FAST for SharePoint Farm - http://technet.microsoft.com/en-us/enterprisesearch/gg213844.aspx
• FAST Search Server 2010 for SharePoint Capacity Planning – I highly recommend reading to really gain a deep understanding of FAST - http://www.microsoft.com/downloads/en/details.aspx?familyid=65B799E3-825C-4398-8CD7-3311D3297997&displaylang=en
• Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint) - http://technet.microsoft.com/en-us/library/ff599530.aspx
• Plan search topology (FAST Search Server 2010 for SharePoint) – Good article that explains what a FAST for SharePoint Farm looks like - http://technet.microsoft.com/en-us/library/ff599528.aspx

FAST for SharePoint Deployment References

• Deployment guide for FAST Search Server 2010 for SharePoint - http://www.microsoft.com/downloads/en/details.aspx?FamilyID=36411da5-ea48-4ab5-8114-21adb21259de&displaylang=en - this is a must read
• deployment.xml reference – Study and learn this - http://technet.microsoft.com/en-us/library/ff354931.aspx
• Deployment options (FAST Search Server 2010 for SharePoint) – Has more examples of the deployment.xml - http://technet.microsoft.com/en-us/library/ff599532.aspx
• FAST Search Server 2010 for SharePoint Capacity Planning – Has several good examples of deployment.xml files - http://www.microsoft.com/downloads/en/details.aspx?familyid=65B799E3-825C-4398-8CD7-3311D3297997&displaylang=en
• Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint) - http://technet.microsoft.com/en-us/library/ff381247.aspx

Part 5 - FAST for SharePoint 2010 Service Configuration

• Part 1 - FAST for SharePoint 2010 Features and Proposition
• Part 2 – FAST for SharePoint 2010 Logical Architecture
• Part 3 – FAST for SharePoint 2010 Physical Architecture
• Part 4 – FAST for SharePoint 2010 Farm Configuration
• Part 5 – FAST for SharePoint 2010 Service Configuration
• Part 6 – FAST for SharePoint 2010 References

In part five of this series, we are going to focus on the SharePoint side of FAST farm configuration.

Configuration of SharePoint Services

After the FAST farm has been set up, you will need to configure the SharePoint 2010 farm to communicate with it. As I mentioned earlier in this blog series this is done through two services:

• FAST Search Connector service – also commonly referred to as the Content SSA.
• FAST Search Query service

Before configuring the search components in SharePoint, there is a file that will help. Go to the server where the FAST Administration Component is installed. There will be a file under c:\FASTSearch called install_info.txt. The values within this file will be used in the configuration of the SharePoint services that communicate with the FAST for SharePoint Farm.

FAST Search Connector service

To set up a new FAST Search Connector service, go to Central Administration as normal and create a new service. Add a New Search Service Application and in the FAST Service Application section, select the FAST Search Connector option.

Next you need to specify a search service account – which in this case will be a SharePoint managed service account. As well you need to specific an application pool.

Next we need to configure some stuff specific to FAST and this is where information from install_info.txt file will assist us.

As I mentioned earlier, the job of the FAST Search Connector service is to send information to the FAST Content Distributor component which will subsequently feed documents to the various processing components on the FAST farm. As you see below, there is a Content Distributor server name.

As well there is a Content Collection Name. The term “collection” is a carryover from the FAST ESP product. A collection is a logical grouping of searchable documents in FAST. If you like to read more about what a FAST collection is read my old blog here. You only have the ability to specific one collection name. The default collection name is “sp”. You may ask why should I care about this? Well you may have multiple SharePoint farms that are feeding content to the same FAST farm and you may want to logically organize content being fed (i.e. intranet content versus extranet content).

Once the FAST Search Connector service has been created, go take a look at the service. You right off the bat that is almost identical to the out of the box search administration, except none crawling links on the left are available. Everything is about crawling.

All you need to do from here is add content sources, set up some crawl schedules and you are off to the races.

Below shows the topology of the FAST Connector service (which is only the topology of the SharePoint side, not the FAST side). There is nothing of real interest other than the fact there is an administration component. The crawl component is not the same as the SharePoint Search. It is used to just support the feeding of content but it is not crawling in the tradition sense. There are no SQL performance considerations you have to account for here.

FAST Search Query service

Next you need to configure the FAST Search Query service. As I mentioned earlier in this blog, this service has two purposes: to send / retrieve query results from FAST and to perform the People Search. To support the people search, this service includes both indexing and querying.

Just like before, add a New Search Service Application but this time select FAST Search Query. Again you will need to specify a service account; you can probably just reuse the one you created for the connector service.

Next you will need to configure two applications pools.

Finally there are four configurations for FAST. All of the information needed to enter here are again located in the install_info.txt file we pulled off the FAST Admin server.

When it is all done, you will have a service that looks like the following. It looks identical to the out of the box SharePoint search. You will notice that there are both Crawling and Query and Result links on the left. Remember there are Crawling links because People Search is part of this service. You will need to configure everything like you normally would for search service.

Finally here again is the search topology for this FAST Search Query service. You see that there are several databases and what not that have been created all to support the crawling and querying.

Once we have completed created both of these services, you will see the two FAST Services in the Service Application list in Central Admin.

Conclusion

I really hope that this blog series will help you to get started in understanding what a FAST for SharePoint deployment would be like before you go off and do one.

Part 4 - FAST for SharePoint 2010 Farm Configuration

• Part 1 - FAST for SharePoint 2010 Features and Proposition
• Part 2 – FAST for SharePoint 2010 Logical Architecture
• Part 3 – FAST for SharePoint 2010 Physical Architecture
• Part 4 – FAST for SharePoint 2010 Farm Configuration
• Part 5 – FAST for SharePoint 2010 Service Configuration
• Part 6 – FAST for SharePoint 2010 References

Part four of this series is going to focus on configuration of FAST for SharePoint.

Configuration of the FAST for SharePoint Farm

Now the next question is how to configure this FAST farm because in many instances you know how to do SharePoint but FAST is a foreign concept. Well the configuration of the FAST for SharePoint 2010 farm is not configured through SharePoint 2010 Administration. At a high level, there is an xml file that you need to create that captures what FAST for SharePoint components are configured on which server in the FAST farm. This xml file basically drives the entire configuration of the FAST farm.

In this part of the series I am going to give an introduction and pointers to information on how to configure your FAST for SharePoint 2010 Farm.

The best resource for you to begin your understanding of a FAST for SharePoint is “Deployment guide for FAST Search Server 2010 for SharePoint” located at - http://go.microsoft.com/fwlink/?LinkId=204984. This whitepaper pretty much has it all. This really just a supplementary with some added details.

Understanding this document will go a long ways in understanding the architecture of FAST for SharePoint. At a high level:

1. There are several service accounts, firewall configuration, IP address work, windows updating, anti-virus and proxy settings you need to do before you can even start the installation.
2. Next I would make sure you have SharePoint 2010 installed and then make sure that the servers that will host the FAST for SharePoint farm have access to the same SQL environment.
3. Next you need to install FAST for SharePoint onto each server in the FAST farm. There is a prerequisites installer that you need to run to make sure all the requirement components are on the machine.
4. Then you need to configure the FAST for SharePoint farm. There are two options: stand-alone or multiple server farm. The stand-alone is as simple as it sounds and you need to go through the configuration steps.
5. The multiple server farm configuration entails the creation of a deployment.xml file that I referred to earlier. After installing the FAST for SharePoint bits on each server in the farm, you will go through a configuration process which will use the deployment.xml file to activate components on that server. We will take a deeper looking into that shortly.
6. Next we need to create the FAST Search Connector service in SharePoint Central Administration which will send SharePoint content to FAST. You will need to configure SSL for the communication between the two.
7. Finally you will need to set up the FAST Search Query service in SharePoint Central Administration which crawls people information and calls the FAST for SharePoint query servers. There is an extra step of this configuration which requires claims authentication to support the call to the FAST query servers.
8. There are several other steps in this whitepaper that layout how to get the SharePoint search centers set up and to test your installation.

For the remaining parts of this blog, I am going to focus on steps 5, 6 and 7.

Deployment XML File

As I mentioned, the deployment.xml file is the key the deployment the FAST for SharePoint farm. There is no nice GUI that will show you the farm and allow you to configure it. In step five you will install the FAST for SharePoint bits on each server of the FAST farm. Then on the machine that will have the admin component installed on it, you need go through the configuration process. The deployment.xml file will be used. Then you will go to each other server in the FAST farm and configure it using the same deployment.xml file.

Now you see the important of this file and why I wanted to focus on it. I was able to find some good references and examples.

First let’s talk about the deployment.xml file. You need to read this - http://technet.microsoft.com/en-us/library/ff354931.aspx - albeit it may be a boring read however it shows how it works. Below is a listing of the components that I provided in the blog with a mapping to the XML nodes.

• Administration component - <admin>
• Document Processing component - <document-processor>
• Content Distributor component - <content-distributor>
• Indexing Dispatcher component - <indexing-dispatcher>
• Web Crawler component - <crawler>
• Web Analyzer component - <webanalyzer>
• Indexing component - <searchengine>
• Query Matching component - <searchengine>
• Query Processing component - <query>

All of the tags described in the deployment.xml reference are important and several do not map directly components. It was not easy at first to gain an understanding of how this works. The best way to do this is to basically open up the deployment.xml reference and then review some examples of deployment.xml files to really understand how it works.

The following are several places where I found examples of the deployment.xml file:

• FAST Search Server 2010 for SharePoint Capacity Planning – has tons of examples that can be used - http://www.microsoft.com/downloads/en/details.aspx?familyid=65B799E3-825C-4398-8CD7-3311D3297997&displaylang=en
• Deployment options (FAST Search Server 2010 for SharePoint) – has a few examples - http://technet.microsoft.com/en-us/library/ff599532.aspx
• Deployment guide for FAST Search Server 2010 for SharePoint – actually has only one example - http://www.microsoft.com/downloads/en/details.aspx?FamilyID=36411da5-ea48-4ab5-8114-21adb21259de&displaylang=en

To save myself time, I am going to pull one of the examples from the FAST Search Server 2010 for SharePoint Capacity Planning document here. I picked this one because it really shows what a FAST Medium size farm that is scaled out. I am not saying this is the best configuration to start off the bat either – read the FAST Search Server 2010 for SharePoint Capacity Planning to find a base farm that best meets your needs and tweak as needed. In many cases a simple three server farm configuration is the best place to start (i.e. a small server farm deployment which was shown earlier in this series). But it would be no fun to talk about J

So here is a picture of a medium farm. As you can see there are:

• This farm has two rows and three columns.
• On the first row, document processing components have been spread across multiple servers to provide good content consumption.
• The first row also really dedicated to indexing as well as the Indexing components will be turned on their along with Query Matching components.
• The second row is dedicated to searching as there are Query Matching components and Query Processing components.
• Note that components such as content distribution and index dispatcher have been turned on in the first row.
• Looking at the very first server in the farm, you may ask why there are query and document processor components are there too? The justification was that they would not be actively referenced by the FAST farm however available if maintenance is being done on the FAST farm.

Now honestly, given all the information I have provided thus far in this blog series, there are a few things I would potentially change in this configuration to ensure that I had a highly available FAST farm. I will get into those changes shortly.

Here is the deployment.xml file for this farm configuration above:

<?xml version="1.0" encoding="utf-8" ?>
  <deployment version="14" modifiedBy="contoso\user"
    modifiedTime="2009-03-14T14:39:17+01:00" comment="M4"
    xmlns=”http://www.microsoft.com/enterprisesearch”
    xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
    xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

  <instanceid>M4</instanceid>

  <connector-databaseconnectionstring>
  [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=M4.jdbc]]>
  </connector-databaseconnectionstring>

  <host name="fs4sp1.contoso.com">
       <admin />
     <query />
     <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
    <document-processor processes="12" />
  </host>

  <host name="fs4sp2.contoso.com">
    <content-distributor />
    <searchengine row="0" column="0" />
    <document-processor processes="12" />
  </host>

  <host name="fs4sp3.contoso.com">
    <content-distributor />
    <indexing-dispatcher />
    <searchengine row="0" column="1" /> 
    <document-processor processes="12" />
  </host>

  <host name="fs4sp4.contoso.com">
    <indexing-dispatcher />
    <searchengine row="0" column="2" />
    <document-processor processes="12" />
  </host>

  <host name="fs4sp5.contoso.com">
    <query />
    <searchengine row="1" column="0" />
  </host>

  <host name="fs4sp6.contoso.com">
    <query />
    <searchengine row="1" column="1" /> 
  </host>

  <host name="fs4sp7.contoso.com">
    <query />
    <searchengine row="1" column="2" />
  </host>

  <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="none" search="true" />
  </searchcluster>

</deployment>

Now looking at that deployment.xml file, here are some notes that will help you better understand it:

• The <deployment> tag is a wrapper tag. It has some attributes for you to manage the version, last modified date, etc. Highly recommend you use these attributes for configuration management reasons.
• The <instanceid> tag can be used by SCOM.
• The <connector-databaseconnectionstring> is the location where you specify JDBC connection strings if that connector is being used.
• The <host> tag is a specific server in the farm. Within the <host> tag is where you identify all of the components that will turned on in a specific server. This where <admin>, <document-processor>, <content-distributor>, <indexing-dispatcher>, <crawler>, <webanalyzer>, <searchengine> and <query> are defined. You can read the specification about these tags as they are pretty straight forward in their configuration.
• The <searchengine> tag is the most important component tag when comes to understanding how many Indexing and Query Matching servers there are. In the <searchengine> tag, you specify the row number and column number however you do not specify here if the Indexing and/or Query Matching components are turned on. This is done in correlation with the <searchcluster> and <row> tags.
• The <searchcluster> tag is an important tag that is used to wrap the <row> tag. Earlier in this series I presented a diagram that discusses columns and rows. These <row> tags are used to define the number of Indexing and Query Matching component rows that are being used in the farm configuration.
• A <row> tag can be defined as a primary, secondary or none using the index attributed. There can only be one primary index row. Marking the <row> tag as primary indicates this is the primary row of servers with Indexing components. Marking the <row> tag as secondary means the row is redundant index row. Finally, marking the <row> tag with the value of “none” means there is no Indexing components on this row.
• The <row> tag also has an attribute called search which can be true or false. If marked true, Query Matching component is deployed on that row.
• So here are some examples of the <row> tag which find critical to understanding the deployment. There are more permutations however these are the ones you will run up against the most:
• <row id="0" index="primary" search="true" /> - this means this is a primary row with the Indexing component. As well there are Query Matching components on that row.
• <row id="0" index="primary" search="false" /> - this means this row is completely dedicated to indexing.
• <row id="1" index="secondary" search="true" /> - this means this second row has redundant Indexing components and the primary purpose of this row is for Query Matching components.
• <row id="2" index="none" search="true" /> - this means this row is completely dedicated to Query Matching components only.
• The <row> tag is tightly correlated back to the <searchengine> tag. Having a <searchengine> tag defined for <host> tag indicates that row will have Indexing and/or Query Matching components. As you can see, the <row> tag actually controls which one is active on that server.

As I mentioned earlier, I believe there are some changes I would make to this farm to ensure there is some redundancy. For instance I would change the <row id="1" index="none" search="true" /> to be <row id="1" index="secondary" search="true" />. The reason being is if the index server were to fail, the query server in that columns would redundantly become the index server until the other server were fixed. Now you could experience some query performance issues while this is going on if you have high volumes of content that are being consumed.

The way the redundancy works is FIXML document (which is the searchable representation of every document in the index) will be replicated to the second row. However an index will not be actually built from the FIXML document. When an indexing component fails, a new index will be built using the FIXML documents on the redundant Index row.

Once you start to get the hang of it, this deployment.xml file is not too hard.

When you have completed an installation on a specific machine, run the following command

nctrl status

This will return the status of all the FAST components that are running on that machine.

Another thing you can do is go into the services console and check out the FAST services.

In the next part of the series I am going to focus on the SharePoint side of the configuration.