SharePoint Online Hybrid BCS and Search

Introduction
With the new release of SharePoint Online and SharePoint 2013 we have some new SharePoint scenarios that can be supported. I have always told customer the definition of hybrid sometimes murky. Why because SharePoint can be implemented in so many ways. In many cases ADFS provides a unified user experience providing the user experience they need. Doing application integration using APIs, remote libraries, etc. provides all that is needed. The new SharePoint Apps model is going to really introduce a lot of new ways to create customer hybrid solutions.
Some other new hybrid scenarios that are going to be supported with the new SharePoint Online and SharePoint 2013. Specifically there is the ability to configure BCS and Search to support some real hybrid solutions. This is really exciting.
Please review the following whitepapers that are located here:

• http://technet.microsoft.com/library/jj838715.aspx
• http://www.microsoft.com/en-us/download/details.aspx?id=35593

Hybrid BCS
With the Hybrid BCS, organizations are going to have some real flexibility in you can expose data from the enterprise to SharePoint Online. In the prior version of SharePoint Online based on 2010 technologies, we were limited to working with web services. There is nothing wrong with that, but we would have to create some web services, make it available to SharePoint Online, and then use it. Here are some facts:

• This supports the ability to view, edit and delete data.
• All data sources must be published using OData.
• Using this solution, no data needs to be published out in an unsecure way, require any new ports to be opened, etc. The on-premise SharePoint 2013 server will basically connect, using a reverse proxy, and will publish out an end point which SharePoint Online can connect to.
• ADFS with Federated identities is required.
• There are a few configuration steps that you will need to go through outlined in the document. Additionally there is some configurations that you will need to make on the SharePoint Online administration screens.

Doing this will allow you to bring your line of business data into SharePoint Online. Users can access that data through External lists, External columns, developers can use it through code, workflow can use it to help make decisions through using the external lists, Visio Services has a new feature that allows it to user external lists, etc. I am frankly just happy to see this level of integration and facilitation versus what we had to do in the past to expose data to SharePoint Online.

For more details read here - http://technet.microsoft.com/en-us/library/dn197239.aspx

Hybrid Search
Even more exciting is our new support for setting up Hybrid Search for SharePoint Online. SharePoint 2013 must be installed on-premise and you must again be using federated IDs with the SharePoint Online. I would say this is one of the primary use cases for facilitating a hybrid deployment. Why because we want to allow users to be able to initiate and search for data regardless of where it resides.

Remember that the new SharePoint Online in the cloud does utilize the FAST search technology. However just because that capability is delivered in the cloud, it does not provide the ability to index on-premise locations. We rely on SharePoint 2013 on-premise to basically do a federated search. This means that SharePoint Online Search does not directly index on-premise content; it relies on those on-premise servers.

There are two approaches for configuring Hybrid Search:

• One-way Hybrid – In this scenario you have the ability to perform a search from your on-premise SharePoint 2013 environment to SharePoint Online. This way users will have the ability retrieve content from the SharePoint Online anytime they do a search from on-premise. This is nice if you have a search portal set up on-premise that is indexing your entire enterprise and SharePoint Online is just another content source to index. All those intranets, team sites, My Sites, etc. in SharePoint Online are accessible.
• Two-way Hybrid – In this scenario you will be configuring both your SharePoint 2013 on-premise environment and your SharePoint Online environment to perform search of each other. This is really powerful to allow the end user to find content across the enterprise regardless of where the user is.

For more information read here - http://technet.microsoft.com/en-us/library/dn197172.aspx

SharePoint 2013 Technical Diagrams Notes

Introduction

SharePoint 2013 Preview Technical Diagrams are now available here - http://technet.microsoft.com/en-us/library/cc263199(v=office.15).aspx

Ever since SharePoint 2007 started publishing these technical diagrams, I have recommended that architects become very familiar with them. I always start here when trying to understand a new major product release for SharePoint. If you search my blog, you will see that I have directly referenced these diagrams when building SharePoint strategies for customers.

The following is a high-level review of the new architecture changes available with SharePoint 2013.

Corporate Portal Diagrams

I reviewed the two new Corporate Portal Diagrams for SharePoint 2013. From a logical architecture perspective, these diagrams do not have any major changes from the SharePoint 2010 versions. These diagrams accurately capture how organizations should build web applications, site collections, application pools, SharePoint services, etc. to support major business initiatives. The diagrams are still a must read for people who are new or who need a refresher to understand how they should be segregating content and business functions across SharePoint.

Extranet Diagram

The new diagram for SharePoint 2013 extranet architecture closely resembles the corporate portal diagrams, however it is not very revealing on the type of information organizations need when making a decision on how to deploy an extranet. Looking back at the SharePoint 2010 Extranet Topologies diagrams (http://technet.microsoft.com/en-us/library/cc263199.aspx), I find that diagram to be much more helpful and the information contained here still holds true with SharePoint 2013. I would recommend reviewing both of them together.

Services in SharePoint 2013 Diagram

I admit this has always been one of my more favorite diagrams. When this was released in SharePoint 2010, it captured a fundamental change in how SharePoint services are configured and delivered. This new architecture was created to support Microsoft’s ability to deliver SharePoint Online as a SaaS solution.

I reviewed this diagram and nothing has significantly changed in regards to sharing services across farms, the logical architecture of services, service groups and service deployment.

In the services table there are a few new services that have been added.

• Access Services – Do not be confused by this. Yes there was Access Services in SharePoint 2010. At this early point, I know that that Access Services for SharePoint 2013 have been changed to be more focused on utilize the new App Architecture. As such, Access Services for SharePoint 2013 is pretty different. Access Services solutions created in SharePoint 2010 will still be supported moving forward, however they will run in a different service.
• App Management Services - This is a new service that will be used specifically for supporting the new internal catalog or the public SharePoint store. Remember that in SharePoint 2013, everything is an app; EVERYTHING. Even everyday SharePoint lists are now called an app. Once you get over the name change, you will find out it makes complete sense and Microsoft has just aligned what is does with how business users talk about technology.
• Machine Translation Service – This is a new one and as of right now, I do not have much information on the purpose of this service other than the description which says it performs automated machine translation.
• Work Management Services – This service provides task aggregation across management systems including SharePoint, Exchange and Project Server. This is huge from a user perspective. One single place to see all of your tasks. No more building content query web parts to find all tasks; this effectively does this plus goes outside the SharePoint boundary to find more tasks. This is a very exciting service.
• Office Web App Services – Is called out in here as a service that is no longer running inside of SharePoint server. Why? Microsoft strategy is to provide Office Web App Service to other enterprise application than just SharePoint and it strategically made sense to move it out of SharePoint.

In the rest of this diagram there are architecture diagrams for how to architect service groups across farms, none of which have changed from SharePoint 2010. If you are not familiar with this stuff, this is a must read and I recommend reading my old posting on it here.

Mobile Architecture Diagram

There is a brand new mobile architecture diagram provided and obviously this is drive by Microsoft’s focus on being a “services and devices” company. This is a pretty simple architecture that basically describes some things you need to think about if you are going to support mobile to your users and discusses some of the mobile capabilities. This can serve as a launch point for you to begin to dive deeper into how you will support mobile for your organization. The following are some high-level observations I had when reading this the first time:

Extranet – If you are not thinking extranet, you need to so you mobile users can access content when they are on a mobile device. They have some diagrams which will get your started thinking about it and additionally how you can use Unified Access Gateway (UAG) as a reverse proxy to help with that.

Mobile Device Management (MDM) – One interesting thing brought up in this diagram is how do you manage mobile devices? If you need something simple, you can leverage Exchange ActiveSync for remote device wipe, password enforcement, etc. If you are looking for application level MDM there are additional solutions out in the marketplace today that provide even more capabilities.

Application Architecture – The new SharePoint 2013 mobile architecture is introduced. They break it down basically into two logical layers: mobile and SharePoint. Some key points are:

• Automatic Mobile Browser Redirection – Is a new capability that can be used to optimize the mobile experience based on the connecting device. This Feature must be active on the site and will be activated by default on numerous site templates. First there is the Classic View which is used to provide backwards capability to mobile devices and will have a SharePoint 2010 mobile browser experience. Then there is the Contemporary View which is geared to support HTML5 browsers. The Contemporary View is several enhanced features for navigation of SharePoint sites. Additionally, Full Site View is available so the SharePoint site page can be viewed as if it were on a desktop browser or a tablet device.
• Office Hub for Windows Phone – Is an application for Windows phone devices that provides enhanced capabilities to access SharePoint content from multiple places in one spot. It also leverages mobile Office.
• Location – There is a new geo-location field type that is available in a SharePoint List. This can make a list location aware to capture latitude and longitude which can be used with map applications. For instance, if a user enters in data on their mobile device, it will capture where it was done from and then can be displayed on a map. Here is some more information about this - http://technet.microsoft.com/en-us/library/fp161355(v=office.15).aspx
• Push Notifications – There is a new capability to allow notifications to be sent from a SharePoint site to registered applications running on a mobile device. The nice thing about this is that Windows Phone Apps can receive notifications without having to poll. Here is some additional reading on the topic - http://msdn.microsoft.com/en-us/library/jj163784(v=office.15).aspx
• Device Channels – This is a really important new capability as device channels allow you to deliver a publishing site geared specifically to support different types of remote devices. Basically the site can mapped to multiple master pages and style sheets and even control what content you want to make available to specific devices. Here is an overview on the new device channels - http://technet.microsoft.com/en-us/library/fp161351(v=office.15).aspx
• Office Web Apps – As mentioned earlier in this posting, Office Web Apps is now a separate standalone server which does not run inside of the SharePoint boundary. Office Web Apps has been improved a lot to support mobile devices. There are Word, Excel and PowerPoint Mobile Viewers.

SharePoint 2013 Upgrade Process Diagrams

There are two upgrade diagrams that have been provided. Here are some high points I walked away with:

• Must be on SharePoint 2010 – To upgrade, you must be on SharePoint 2010 technologies. This means if you are on SharePoint 2003, 2007 or 2010, you will need to upgrade to the appropriate version to get to SharePoint 2013. There are Microsoft migration partners that have solutions to assist with this. I saw many times, this is the big value proposition for using SharePoint Online as this is handled for customers.
• Database Attach Upgrade – Is the only supported method for upgrading. There is no more “in-place” upgrade option. Frankly that is fine because most customers always went down a database-attach upgrade.
• Preparation – much of the preparation activities that we have discussed in the past with SharePoint 2010 hold true with SharePoint 2013. There is a bunch of information you are responsible for gathering.
• Manual Configuration Settings – In the preparation phase is recommended to get a understanding of all the custom configurations that you may have done because not all of them are going to be migrated. This is because not all databases are upgraded. So many custom configurations in central admin such as alternate access methods, time job tweaks, managed paths, incoming/outgoing email settings, certificates, etc. will need to be documented and reconfigured in the new farm.
• Databases That Can Be Upgraded – There is a set of databases that can be upgraded. They are Content, BDC, Managed Metadata, PerformancePoint, Secure Store, Search and User Profile databases.
• Customizations – This is an important task that needs to be completed. I have seen many cases where good software organizations have not implemented a strong configuration management process and the result is an organization may not know about all the customize code that may be implemented. There are numerous ways to find all of them by running PowerShell commands, doing system directory diffs, checking web.config, etc.
• Upgrade Health Checks – There are some new features that are available to site collection administrators that will show you a health check of a site collection before actually upgrading the site collection.
• Evaluation Site Collection – Site Collection Administrators also have the ability to request the site collection be copied into a new site collection to evaluate how the upgrade will affect any customizations they may have. This is helpful so you can remediate issues before you actually perform the upgrade. This is also nice because your site collection will run in a SharePoint 2010 mode until you are ready to actually upgrade it.
• Testing – Just like for SharePoint 2010, the best way to prepare for a migration is to build up your new SP 2013 farm and then multiple practices runs of that upgrade into the new production environment. An entire process is defined in one of the diagrams and is a great place to start.

SharePoint 2013 Search Diagrams

If you are a reader of my blog, I wrote some long postings about the search architecture for both SharePoint 2010 Search (here) and FAST for SharePoint 2010 (here). I am not going to do a deep dive into all these search components and roles because they are basically covered. As many people now know, the FAST search engine is now the core search engine for SharePoint. It will just be referred to as SharePoint Search. Now you will be able to leverage a very powerful search engine out of the box. However many of the advanced enterprise features of search will only be available in the SharePoint Enterprise addition. I am also really excited about this for SharePoint Online because it can leverage FAST too. SharePoint Online will not be able to do Enterprise Search of line of business systems but a Search Farm (which is FAST underneath the hood) can be configured on premise and SharePoint Online can invoke that search and provide the search results in the cloud; pretty exciting.

I highly recommend taking the time to review both of these diagrams. It explains how each of the components interacts with each other. Additionally there is a diagram the goes into how to scale the server farms for the amount of content you will need to index. There is a great, new table in there that shows you how scaling will work. To be honest, the folks who are really serious about search will say it is an art and a table does not always communicate how you will do it. It always comes down to how many items, the types of data sources, custom transformations, query latency, index latency, etc.

SharePoint 2013 App Overview Diagram

This is an area I plan to do a lot more exploration of this coming year on and writing on this blog. Why? This is something we have been waiting for a long time with SharePoint development. There are several ways to look and this. SharePoint Features which we have been writing for years are Apps. This is name change to better communicate our technology to the end users who have to use SharePoint. However the new SharePoint App architecture is way more than that.

I have seen so many things over the years.

• I think one of the biggest challenges people would run into is developing great SharePoint solutions only to find out they incorporated some dependency they should not have, they wrote some high-end code that should not be running in the SharePoint layer, they cannot leverage their solution outside the SharePoint boundary, etc. We want to resolve those problems by helping developers to deploy solutions in way that will keep their SharePoint environment nimble.
• Plus we want to provide third-party vendors quicker access to customers. We want to help customers to quickly acquire third-party solutions.
• Additionally we want to allow customers to leverage commodity based SharePoint Online. As you may know SharePoint Online has restrictions on high-end custom development and if that code where to run in another location, while be highly integrated with SharePoint Online, that is a huge win.

I will thing of many more reason this year on why this is so great J

Now we can achieve this through the new SharePoint App architecture. The old SharePoint Solution architecture where you create a WSP is still around. Nothing has changed there. This is used to create deployment packages and in many cases is used to deploy code that requires full trust. SharePoint Solution packages will continue to be used by third-party vendors or developed internally with such tools as Visual Studio. You can still create Sandbox Solutions which run in a more secure runtime and can be deployed in SharePoint Online.

Now the new Apps framework for SharePoint 2013 is a packaged up in a file called .APP. It is composed of many of the same types of files, AppManifest, embedded Solution.wsp, etc. Once an app is loaded into SharePoint, it is accessible through the App Catalog in SharePoint. This App Catalog can be controlled at an organizational level.

Remember the big point with Apps is, that the custom solution you are writing may or may not actually run in SharePoint. Full trust code is not supported. Your custom solution code itself may run in a different SharePoint farm, on an IIS server as ASP.net pages, ASP.net pages running in Windows Azure, etc. So how does SharePoint access these solutions running outside the SharePoint context? In simple terms we have an IFrame (with some extensions) that external solution is available through. OAuth provides the secure connection for access SharePoint objects from a remote location. We will additionally use a new extended and robust event model and remote client SharePoint library to write integrated, remote code.

Why is this so great? We are going to ensure that custom applications and solutions that are being developed with SharePoint are isolated. No more writing a bunch code and services that should not be running in SharePoint servers. It is great that you can do whatever you want with SharePoint, however this will drive solution management.

So you may be asking where does this get deployed? There are many different options for hosting.

• SharePoint Hosted – This means the app and all the resources run in SharePoint. Remember you server side is not supported however you can write applications with SharePoint’s JavaScript libraries and such.
• Windows Azure Autohosted – This is a model that is only supported in SharePoint Online. In This case you can write an App package that will have code for Azure and SQL Azure embedded into it. When the application is deployed, the azure solution will be automatically deployed for you. You do not have to go to Azure and set anything up at all; it is all handled for you behind the scenes.
• Provider-Hosted – This is the third model where custom code and solutions are hosted in a separate server in your organization, hosted in Azure, hosted in different SharePoint servers, etc.

Once an App package is installed, it can be managed and monitored through the catalog. End users have the ability to select an app to run in their sites (much in the same way as turning on a Feature). If and when an app is updated, the user can decide how they want to upgrade to the new app.

Again I really plan to go much deeper on this in my blog but for right now, these are just some introduction notes and ramblings on how excited I am about this new capability J

Back Up and Recovery Diagram

There is a new diagram that goes into the details of doing your own back-up and recovery for SharePoint 2013. I know many people have become accustomed to using third-party vendors for supporting these operations and I still believe these vendors will continue to provide features above and beyond what is out of the box. However, if you are a do it yourself sort of person, this is a great diagram to review.

Not much has changed in regards to developing back-up procedures for both the SQL Servers and the SharePoint Servers. There are tons of scenarios covered in here, and I recommend reading this if it is important to you.

SharePoint 2013 Database Diagram

Finally the database server diagram has been updated. This is a really really really important diagram to review if you are managing on-premise servers. It goes over all the SharePoint databases, plus provide sizing and scaling guidance. Great information.

Gartner Rates Microsoft a Leader in Enterprise Search

I am happy to say the Microsoft has again been ranked as a top Enterprise Search provider in the marketplace - http://www.gartner.com/technology/reprints.do?id=1-1835DKL&ct=111123&st=sb. You will see that some Enterprise Search providers dropped in rankings and some were removed.

As you can see, FAST is not slowing down and we have an extremely strong roadmap for it. It is a very fiscally responsible Enterprise Search solution when you consider it as part of your Total Cost of Ownership (TCO) when investing with SharePoint Enterprise. FAST provides tons of added features and capabilities to solutions such as Portals, Sites, ECM, Document Management, BI, Social, Composites, Platform Development, etc. Plus if you know how to configure SharePoint Search; FAST search is configured and managed through the same screens and procedures thus requiring less ramp up time.

Even if an organization just needs a pure Enterprise Search solution, FAST is a great Enterprise Search platform. Having SharePoint in the mix, is great value additive because many other Enterprise Search ends provide you no user interfaces. Yes FAST does have services so that other applications tiers can tap into it. However having a web portal like SharePoint immediately available with numerous web parts to start building search screens and search applications is a huge time and money saver. Plus, when Enterprise Search is made accessible through SharePoint, you get to take advantage of all the other SharePoint features to deliver a more engaging user experience. For instance:

• You can create blogs from the search team to the users
• Wikis to explain to users how to do advanced search
• Stream training videos
• Maintain documents and FAQs
• Create a Search Knowledge Base
• Create support forums
• Manage technical specifications about line of business data that is index
• Etc.

SharePoint 2010 Independent Market Analysis

Updated - 3/2012 - View new blog posting here

I get asked a lot about reasons to move to SharePoint 2010. I simply state, look at the how the market evaluates SharePoint.
Here is a great link managed by Microsoft that showing Gartner and Forrester reports - http://www.microsoft.com/presspass/itanalyst/default.mspx

As of right now, I recommend reading the following:

• Enterprise Content Management (ECM) – View Report
• Business Intelligence (BI) – View Report
• Enterprise Search – View Report - Just Updated in Nov 2011 and continued leader - View Report
• Social Computing – View Report - Just Updated Aug 2011 (continued to be leader, here is the 2010 report)
• Portal Platform – View Report
• Unified Communications – View Report and View Report
• Email – View Report

I suggest stay up to date with this site to find the latest reports.

FAST Ampersands in Custom Property Extraction Dictionary

If you are creating a Custom Property Extractor for FAST for SharePoint 2010 be careful because careful to set up your Extraction Dictionary correctly. I recently had a colleague create a new extraction dictionary like in here - http://msdn.microsoft.com/en-us/library/ff795797.aspx

Unfortunately I do not have the logs but my colleague was telling me that FAST was just hanging when trying to execute the FAST crawl. I looked at the modifications he made and took a guess that the ampersands he had throughout the custom property extraction dictionary XML definition could be the issue. I recommended he remove the ampersands and replace with &amp. Well that took care of the issue.

Part 1 - FAST for SharePoint 2010 Features and Proposition

• Part 1 - FAST for SharePoint 2010 Features and Proposition
• Part 2 – FAST for SharePoint 2010 Logical Architecture
• Part 3 – FAST for SharePoint 2010 Physical Architecture
• Part 4 – FAST for SharePoint 2010 Farm Configuration
• Part 5 – FAST for SharePoint 2010 Service Configuration
• Part 6 – FAST for SharePoint 2010 References

Introduction

FAST for SharePoint 2010 (FS4SP) is one of the newest and coolest features of SharePoint 2010. FAST was a major acquisition for Microsoft and it is one of the top Enterprise Search engines in the Gartner quartiles. Combined together, Microsoft really has a best of breed platform for managing content and building enterprise solutions.

FAST ESP 5.3 was the last version of the product before it was integrated with SharePoint 2010. The legacy product is still available in different formats from Microsoft because they want to continue to support existing clients and there are scenarios where FAST ESP 5.3 is a better fit. It has been rebranded as FAST Search for Internet Sites (FSIS) and FAST Search for Internal Applications (FSIA). I am not going to go into either of these. If you want an understanding of FAST ESP 5.3 I have written a blog series about it here which goes into the architecture.

After writing that series over a year ago, I have been wondering how is FAST integrated into SharePoint 2010. I really wanted to understand this from an architecture perspective so I can provide good guidance on how to set up SharePoint 2010 to use FAST. When I set out to do this, it took me awhile to really find the information but I was able to piece it together. I have listed all the references I have used at the end of the series. What you will soon find out is the architecture of FAST did not really change.

Features

The following are some of the additional features of FS4SP that are added on top of SharePoint Search you should know about right off the bat:

• Advanced Content Processing – Extract and create metadata by using content within the documents. This will improve search results, relevancy, sorting and refinement. Plus this reduces the human workload to actually create the metadata.
• Advanced Sorting – Provides ability to sort results based on any managed properties or rank profiles that are available to the current user.
• Business Intelligence Indexing Connector – Supports the ability to index such things as Excel workbooks, SSRS reports, etc. So for instance there may data within the report, a title in a pie chart, etc. which will now appear in a search result.
• Contextual Search – Have the ability to customize search results and refinement options based on the user profile or the audience that is performing the search. This is a major feature when it comes to building relevant search results to the user.
• Tunable Relevance with Multiple Rank Profiles – Ability to create rank profiles that incorporate things such as freshness, authority and quality to provide more relevant results. This is important because these rankings are not part of the index and are applied to the query; making this tunable. There is the ability to identify authoritative pages, you can identify documents for promotion (and associate that promotion based on the user context).
• Deep Refinement – SharePoint Search only performs a shallow refiner which only allows refinement for the first 50 results in the original query. FAST provides deep refinement which is based on statistical aggregation of managed property values within an entire result set. Because FS4SP provides a deep refiner, the exact count of documents in refiners.
• Extensible Search Platform – Build complex search solutions, search driven applications, etc. Personally this is where I see lots opportunity to use search in ways you have not thought about before.
• Extreme Scale Search – The ability to search millions upon millions of documents with sub second time speed. There is no scale boundary when it comes to FAST.
• Rich Web Indexing – Ability to index dynamic HTML and javascript content with custom indexing connector.
• Similar Results – When results are returned, there will be a link called Similar Results which the user can click. Basically when a user clicks on the link the search is re-defined and re-executed to include documents that are similar to the result.
• Result Collapsing – FS4SP can perform a checksum on data within the index and will collapse results into a single result where possible. Now documents that are popular and stored in many places across the organization will not be shown multiple times. The user has the ability to click a link to see where all of the duplicates are stored.
• Thumbnails and Previews – Review content quickly with thumbnail and preview images in search results. One example is the PowerPoint previewer. In reality this feature is given to us through Office Web Apps however it is bundled as part of the FAST solution offering.
• Two Way Synonyms – SharePoint Search supports the usage of synonyms but they are only applied to keywords that in the query. However FS4SP also supports the ability to apply synonyms to the documents itself so if the query has a keyword with a synonym but the document only has the synonym, that document will be returned in the result set.
• Managed Property – With FAST, you have the ability to create metadata mappings and create rules for that metadata and how it will be used to provide a better search experience. Specifically this identifies if the property can be used for sorting (i.e. can the property be sorted in the search results), filtering (i.e. can the property be used to filter), as a refiner (i.e. can be used with deep refiners), priority (i.e. used in ranking algorithm against other documents), and dynamic/static summaries (i.e. dynamic summary will display a hit-highlighted summary of the managed property in the result).
• Property Extraction – FAST will identify key information like people names, company names, geographic names/location, etc. within a document and then use that data to provide more relevant search results.
• Rank Profiles – Are a feature of FS4SP which control how relevancy is calculated for all items that are indexed. Rank Profiles can be aligned to type of user to provide them a more relevant search experience or even let users select a rank profile based on what they are trying to accomplish. Rank Profiles in FS4SP provide more features such as freshness, proximity, authority, query authority, context, managed properties
• Visual Best Bets – Provide the ability to return rich, editorialized results based on keywords. This can as well be tied contextually to the type of user viewing the best bet.
• Linguistics – Supports the ability for language variations to be used to allow users to find relevant information. What this helps with is finding relevant search based on words and phrases that may not be identical between the query and the indexed item. Such features as character normalization, normalization of stemming variations and suggested spelling corrections are part of the linguistic processing. Linguistic processing is part of both item processing (before the document is indexed) and as well as for the queries that are submitted by the user.
• Multiple Language Support – FAST support well over 80 languages - http://technet.microsoft.com/en-us/library/ff793350.aspx

Value Proposition

I will be honest, all those features sound great, but taking a step back you really need to say to yourself how will a power search engine help your organization? I actually wrote about this topic from my perspective when I was starting to learn about Enterprise Search - http://www.astaticstate.com/2009/11/why-is-fast-enterprise-search-important.html. I highly recommend that you review this posting understand why Enterprise Search is so important. Upon reading this, you will see how important having FAST is whether you have massive amounts of content or you are just a medium sized organization with 1 million items. At the end of the day:

• Users are bombarded with information that is located in numerous locations.
• Users are not aware, educated or trained on how to find that data.
• In many instances documents do not have good descriptive data making them hard to find.
• Companies have challenges with retention of people and technology. They are looking for ways keep that data available to the organization.
• Companies have challenges on boarding new personnel.
• Companies have challenges communicating across geographical boundaries where information is not shared.
• Users are challenged with finding the most right information from authoritative experts.
• Organizations have challenges with customer relationships, satisfaction and retention.
• Companies are not enabled to bring the offline office community online. Documents are not recognized in the way they are actually used within the organization.
• Companies have challenges with effectively sharing or selling data they have.
• Organizations have challenges with knowledge management as information is managed in silos.
• Organizations are not responsive to their employees and customers.
• Companies and organizations have challenges bridging together business processes the span across geographies, organizational hierarchies and technologies.

Really the list can go on and on. The utilization of FAST in SharePoint, regardless of how big or small the content source(s) is will ultimately improve a company’s ability to react to these challenges.

SharePoint 2010 Enterprise Search Hands on Labs

Do you want to really start learning the difference between SharePoint 2010 Search and FAST for SharePoint? No better way to do it other than some good self-paced training and some hands on labs.

Here are the following:

• SharePoint 2010 Enterprise Search Developer Training – Videos and presentations - http://msdn.microsoft.com/en-us/sharepoint/ff960975.aspx
• SharePoint 2010 Enterprise Search Labs - http://www.microsoft.com/downloads/en/details.aspx?FamilyID=b17a747f-3b47-4712-a06c-73252774e60a
• SharePoint 2010 Demos: Enterprise Search Demos & Labs - http://www.microsoft.com/downloads/en/details.aspx?FamilyID=9fcbcc96-02b7-4292-b3b7-c04b236accde

SharePoint 2010 Capacity Planning

• SharePoint 2010 Service Architecture
• SharePoint 2010 Logical Architecture
• SharePoint 2010 Architecture and Design Models
• SharePoint 2010 Performance and Capacity Case Studies
• SharePoint 2010 Capacity Planning
• SharePoint 2010 Cache Strategies
• SharePoint 2010 Databases
• SharePoint 2010 Physical Topology
• Simple Real World Examples

Introduction

One of the most important things for planning, managing and governing a SharePoint 2010 environment is to understand the boundaries and thresholds that affect performance and maintenance. Capacity Planning is extremely important when architecting a new SharePoint 2010 environment. A lot of times people designing SharePoint environments or creating SharePoint solutions completely forget about how Capacity Planning is directly tied to use cases and business scenarios. Knowing where your limits are and how you plan to grow will affect both your physical topology and logical topology.

For SharePoint 2007 we had a Capacity Planning tool (old blog I wrote). The old tool did its job at the time but there is no new one as of yet. However Microsoft has actually put out a ton of information that you can use for SharePoint 2010 Capacity Planning.

In this blog I will touch on:

• Web Site and Site Collection Capacity Planning
  
• Content Database Capacity Planning
  
• List and Library Capacity Planning
  
• Search Capacity Planning
  
• Security Capacity Planning
  
• Social Networking Capacity Planning
  
• User Profile Service and Social Networking Capacity Planning
  
• Web Analytics Capacity Planning
  
• Workflow Capacity Planning
  
• Excel Services Capacity Planning
  
• PerformancePoint Service Capacity Planning
  
• InfoPath Services Capacity Planning
  
• Visio Services Capacity Planning
  
• Word Automation Services Capacity Planning
  
• Office Web Application Services Capacity Planning
  
• Access Services Capacity Planning
  

Resources

The following are the resources that I used for this analysis

• SharePoint 2010 Capacity Planning (Boundaries and Limits) Whitepaper (View or Download). This is a really good whitepaper which should be used as a baseline for architecting your SharePoint 2010 environments. I know there will be creative solutions over the long run that will allow you to exceed these however this is the best place to start. I actually enjoyed reading through this.
  
• SharePoint Server 2010 performance and capacity test results and recommendations (View or Download). This is a set of several detailed whitepapers that go into the nuts and bolts of how SharePoint 2010 was tested and provides tons of very detailed recommendations.
  

All this information in this blog comes from these whitepapers and these are basically my notes.

Since these are just notes – I have not spent any time on grammar J

Web Application Limits

• Try to limit to 300 content databases per web application. This is an important one because a well managed SharePoint environment will not have one big content database; but instead have many to facilitate back-up and recovery procedures. If you plan to have lots of content databases, the recommendation is to use Powershell to manage the web application instead of using the management interface.
  
• Zones have a hard limit of five (default, intranet, extranet, internet and custom); nothing new there.
  
• It is recommended that you do not exceed 10 managed paths per web application without testing.
  

Thoughts: Not much new here other than the most important one everyone should know – do not have one big massive content database for your web application.

Web Server and Application Pools

• It is recommended to have no more than 10 Application Pools per web server. This is mostly driven by amount of RAM and usage by the users.
  

Content Database Limits

• Content databases should not exceed 200GB of data. Again SharePoint 2010 does scale up to 1 TB but that is only recommended for large, single site repositories.
  
• Remote BLOB Storage (RBS) cannot exceed 20 milliseconds to access the first byte of data.
  

Thoughts: I can say this over and over but you need to plan to keep your content database sizes down and do not let them exceed recommended limits. RBS is good for performance but you still think about not letting your content databases store TBs of data. That is a business requirements issue.

Site Collection Limits

• It is recommended to not exceed 250,000 sites per Site Collection. Basically the deal is that there is performance overhead when both creating and deleting sites. If you have a highly dynamic SharePoint web site, where this many sites are created and deleted on a regular basis, you will experience performance issues across the board.
  
• Site Collections should not exceed more than 100 GB. Again the issue is to make backup and restore procedures run quickly.
  

Thoughts: If this becomes an issue you probably have too many people with rights to create sites. Another way I can see this happening is if you create some sort of automated site provisioning process that just creates too many sites. Your SharePoint Topology should plan for not exceeding these limits.

SQL Service Column Limits

• As we mentioned before, we need to limit the number of row wrapping to 6 rows. According to SQL Server the threshold is 64 columns per row, so you should not exceed more than 384 columns in your SharePoint list or library definition.
  
• As mentioned before, the amount data stored SharePoint item per row cannot be more that 8,000 bytes.
  
• This whitepaper goes into specifics the size limits by SharePoint column type.
  

Thoughts: I had never really thought about this in the past. It will become a more important factor in your SharePoint design given the amount that can now be stored in SharePoint.

Web Page Limits

• It is not recommended to exceed more than 25 web parts per SharePoint web page.
  

Thoughts: If you have than many on your screen you are trying to display too much or your have made your web parts way too granular.

List and Library Limits

Understanding SharePoint list and library capacity is extremely important, especially with the new improvements to SharePoint 2010.

Here are some specifics:

• For each row in a SharePoint List or Document Library List Item you have a hard maximum of 8,000 bytes. What this basically means is the amount of data stored in the columns of a list item cannot total more than 8,000 bytes.
  
• The maximum File Size is 2GB. The default is 50 MB.
  
• It is recommended to not exceed 30,000,000 documents per document library.
  
• It is recommended to not exceed 30,000,000 items per SharePoint list.
  
• Should not exceed more than 6 table rows in the Content database to store a list or document library item. What this basically means is that if your list or library definition has lots of columns, SharePoint will break that storage of that data across rows. There is fixed amount of columns in a SQL table and if there more list columns that SQL table columns the data for the list item will be stored across rows (referred to as row wrapping). So this will only become an issue for you if you have an extreme amount of columns in your SharePoint list definitions.
  
• The SharePoint list user interface only allows 100 items to be processed in bulk at any one time.
  
• SharePoint List queries have a maximum of 8 joins allowed per query. If there are more than 8 joins in the query, it will be blocked.
  
• List view threshold by default will show 5,000 items. Going beyond this you will experience poor performing queries and overall throughput will decrease. List view threshold for auditors/administrators is 20,000 items.
  
• It is not recommended to exceed 2,000 subsites for each website. Performance will degrade on several of the out of the box pages, controls and web parts when 2,000 subsites are exceeded in a single website.
  
• It is recommended to not exceed 10 users simultaneously editing the same document. If there are more than 10, you will receive performance issues when committing the document to SharePoint. The hard boundary is 99 users.
  
• When extracting datasheet view into Excel, there is a hard limit of 50,000 items that can be exported.
  
• A SharePoint Workspace has a hard limit of not containing more than 30,000 items for synchronization.
  

Now are some other recommendations that you should be made aware of:

• A basic way for calculating the potential size of the list is the following. First estimate the average size of a document to be stored. Multiply that against the number of versions of that document. They multiple that by 20% for the data stored associated to the content database. It is also recommended to add in an appropriate buffer.
  
• When designing how content will be stored in lists, you need to consider if storing data in a single list, across multiple lists or even across lists in different site collections. There is a detailed discussion of this in "Single list, multiple lists, or multiple site collections" section in DesigningLargeListsMaximizingListPerformance.docx.
  
• Even through list and libraries can store large amount of content, we still need to get it back out. Microsoft recommends that when retrieving data from large lists using lists views or CAML queries, they be partitioned across folders, indexes or both. If not, the best and most efficient way to retrieve items is through search.
  
• Lots of permissions applied to a library will degrade performance. It is worse if you are doing item level permissions on large amounts of documents. There is a configuration will limit 50,000 unique permissions to a list. However it is recommended to lower this to 5,000.
  
• I mentioned row wrapping earlier and why it is used. It is recommended on for lists with large amount of data, to not exceed 1 or 2 additional rows of wrapping.
  
• Lookup columns can again be another source for performance issues. Lookup columns in a list view will result in a join being applied to the query which increases the complexity. By default each list view will support up to 8 lookup columns. It is strongly recommended to not exceed this number as it will cause significant throughput decrease for queries in a view and use lots of SQL resources.
  
• List indexes have little effect to list performance but can insert and update operations if there are lots of them. Up to 20 indexes can be applied per list. It is recommended to only use indexes when they are needed.
  
• There are three basic ways to access list data: list views, content query web parts and search. List views access data directly off SQL Server which incur slower query performance and higher load on the SQL Server. They also generate the most HTML which can slow down page rendering. However list views provide the best user experience when it comes to managing data in a list. Content query web parts display a configured view of data that is cached in the portal site map provider. It will generate the least amount of HTML and is cached, so overall performance will be better. Search web parts offer the best performance as it is optimized for queries large amounts of data. However the data returned in search is only as recent as the last crawl.
  
• Microsoft has several types of lists they have identified and when designing your SharePoint site, it may be good to classify in this manner so you can plan and govern better of the long run.
  
• There are unstructured document libraries which are basically team libraries with tens to hundreds of documents. They are highly utilized with expensive operations but the volume is low. It is important to make sure they libraries stay under 5,000 items.
  
• There are collaborative large lists which can have thousands of documents that can be edited, like in a knowledge management solution. Typically these lists grow significantly faster than anticipated and requiring lots of administration.
  
• There are structured large repositories which may have thousands to hundreds of thousands of documents. Typically these are departmental archives with automated processes to publish the documentation in a controlled manner. Almost all of the interactions with the list are read only.
  
• Finally there are large scale archives which contain millions of documents spread across multiple lists. Typically there are is a low amount of reads and updates and the purpose is for long term storage for compliance requirements.
  
• Content Organizers are a new feature that will route content to configured libraries, folders or sites. Users can submit data and not be concerned about the rules for storing content. It can support evenly storing data within a library to better manage performance.
  
• Metadata Navigation is a new feature that allows users to search for content in a tree based on the data associated to the document. This feature will use the most efficient query possible to return items instead of allowing the user to freely search and filter for content.
  
• Throttling of SharePoint list is also new. This can be configured at the web application so that inefficient operations by a user will not affect performance of the entire farm.
  
• Another new feature is compound indexes so indexes can be built across multiple columns instead of just one.
  
• The developer dashboard is another new feature that can help when understanding potential performance issues with a page. When it is turned on, tons of statistics are available, including database queries and load times.
  
• Allow object model is a new configuration that allows for object model to override the list view thresholds. This allows for developers to create web parts that query for content that do not adhere to list view thresholds, which is a good thing.
  
• Daily time window is another new configuration that when turned on, allows queries to be run that do not adhere to list view thresholds.
  
• You need to be aware that if the list will grow very large and exceed thresholds, several operations will be blocked, prevented or limited. For the details read the "Differences between large lists and regular lists" section in DesigningLargeListsMaximizingListPerformance.docx.
  
• Microsoft testing of extremely large document libraries (millions of documents) generally concluded that most of the bottlenecks would occur at the SQL Server level. To get around this bottleneck you should split content across multiple instances of SQL Server.
  

Thoughts: I am a broken record on this topic. Yes SharePoint 2010 resolved the challenges of storing mass amounts of data in SharePoint lists. However I still believe quantities of list data, especially "highly relational data", should be stored in a relational database and then exposed to SharePoint through the appropriate means.

Security Limits

• It is recommended to not exceed having a user belong to more than 5,000 SharePoint groups. There are no severe performance penalties if you go over, just a recommendation. If you do go over the user's token is bigger, it takes longer to cache access control list (ACLs) for search, and increases the amount of time check security.
  
• It is recommended to not exceed 2,000,000 users per site collection. Again you can exceed this number however there are long term management issues with that many users and it is recommended that you use PowerShell.
  
• It is recommended to not exceed 5,000 users or external groups (AD group) per SharePoint Group. The more users, the longer it takes to validate permissions or render a screen.
  
• It is not recommended to exceed more than 10,000 SharePoint Groups per Site Collection.
  
• It is not recommended to exceed 5,000 permission items in an access control list (ACL).
  

Thoughts: It is important to keep in mind for very large organizations and when planning to pull in data from external directory services (which may not always be Active Directory).

SharePoint Search Limits (not FAST)

When it comes to search, this is just such a big topic. I highly recommend reading all of "Estimate performance and capacity requirements for SharePoint Server 2010 Search" (SearchforSPServer2010CapacityPlanningDoc.docx). Understanding the search architecture is critical for sizing and capacity management. I am going to go over some high level concepts to get you started:

• First when planning you need to understand what data needs to be searchable and how much of it is there. You need to know how available the data must be. You need to know how fresh the data must be kept and many people will be searching for data simultaneously. This actually goes a lot deeper and I have blogged about scaling FAST 5.3 here. The concepts are the same.
  
• For SharePoint 2010 there is a search life-cycle that you need to understand. There is Index Acquisition which is where full crawls for data are performed which is dependent on the size and access to the content is being crawled. Next there is Index Maintenance which is incremental crawls of all content. Finally there is Index Cleanup is basically when content sources change or removed from the crawler.
  
• Next there is the query service which is responsible for querying for data. It is important to know both the query latency and query throughput. You want to decrease latency which is the amount of time it takes for data to become searchable. You want to increase throughput which is the amount of data which can be searched.
  
• There are really just so many strategies for configuration your SharePoint logical and physical topology based on your requirements. Here are some simple things to think about. First never run both the query and search services on the same machine. Second, if you have the hardware, it is also recommended to run your query service on a separate machine which will decrease latency and increase throughput. Third it is good to have multiple search servers which build the indexes so there is redundancy when there is a failure or if you need to dedicate resources to search specific content sources.
  
• There some calculations which you can do to estimate the amount of space you need to indexes that will be built. Again, read this whitepaper for further details.
  

Some general thresholds and boundaries are:

• It is recommended to not have more than 20 SharePoint search service applications deployed to a single farm.
  
• For each SharePoint search service, do not exceed 10 crawl databases which store crawl data. The optimal configuration is 4 crawl databases per Search service. Each crawl databases should not exceed 25 million items.
  
• It is recommended to not exceed 16 total crawl components per service application.
  
• It is recommended to not exceed 20 index partitions for a search service. The limit is 128 index partitions. Partitioning an index allows for smaller indexes to created which can be searched faster, however having too many can have opposite effects.
  
• It is recommended to not exceed 10 million items per search index. The overall recommended limit for all indexes in a search service is 100 million items.
  
• It is recommended to not exceed 100 million crawl log items per search service.
  
• For each search service, it is not recommended to exceed 10 property databases. The property database contains metadata for items in each index partition. There is a hard limit of 128 property databases.
  
• There is a hard limit of 128 query components per search application.
  
• For search scopes, you should not exceed 100 scope rules per scope. As well you should not exceed 600 scope rules per search service. Exceeding these thresholds will affect performance. As well, you should not exceed more than 200 scopes per web site, which again can affect performance.
  
• It is recommended to not exceed 25 display groups per site. Display groups are used for grouped display of scopes through the user interface. Exceeding this number will affect performance.
  
• For alerts, there is a limit of 1M per search application.
  
• For search content sources it is not recommended to exceed 50 data sources but the top threshold is 500 data sources.
  
• There is a threshold for running 20 concurrent crawls within the same search service; exceeding this will cause performance issues.
  
• During crawling, each search application will support up to 500,000 properties.
  
• Recommended not to exceed 100 crawl impact rules per farm.
  
• Recommended not to exceed 100 crawl rules per search service. This is because the administrative screen performance will degrade.
  
• There is a threshold of 100,000 managed properties per search services. Managed properties are used in search queries. Basically the crawled properties are mapped to the managed properties.
  
• It is not recommended to exceed 100 mappings per managed properties. Exceeding this will degrade crawl speed and query performance.
  
• Each search service will support 100 URL removals per operations.
  
• It is recommended to have only 1 top level authoritative page and minimize second and third level pages.
  
• It is recommended not to exceed 200 keywords per site collection but the maximum is 5,000 per site collection. The only real effect of exceeding the recommendation is performance of the administration page.
  
• There is a limit of 10,000 metadata properties that can be crawled per item crawled.
  

Thoughts: Microsoft has put a lot of emphasis into this. Even though FAST is a very compelling, the out of the box SharePoint search has been dramatically improved.

User Profile Service and Social Networking limits

• The user profiles service supports up to 2 million user profiles with full social networking features. This basically means this is the limit of profiles that can be imported into the people profiles repository.
  
• There is a limit of 500 million total social tags, notes and ratings in the social database. Exceeding this runs the risk of significant performance issues.
  
• Microsoft's testing found that there was linear increase in throughput up to 8 WFEs. Past that there was no improvement. Scaling can be further achieved when separating the content and services database onto two separate database services.
  
• Most of the bottlenecks that will be experienced are typically due to WFEs. They found no bottlenecks associated to any of the application servers when evaluating social networking features.
  
• It is expected that separate web site with dedicated resources is created for MySites and the new social networking features. When planning to support performance this is not just maintaining a user's profile information, connections, newsfeeds, etc. It is also storage of documentation, collaboration, etc.
  
• User Profile service does have caching. The user profile service job is to take data from whatever location where you are loading from, and it will save that data in SharePoint profile database. Once the user profile cache is loaded, WFEs request for profile data is returned from the application server without requiring a call to the SQL database. I guess Microsoft's strategy is that profile information is not really dynamic data and they want to keep transactions down.
  
• I am not going to go in the details of the Outlook Social Connector however there is very detailed information about the performance of this in the Social Computing Capacity Planning whitepaper.
  

Thoughts: Much of this is associated to the new social networking features of SharePoint 2010.

Business Connectivity Services limits

• There is a hard limit of 5,000 external content type definitions that can be loaded into memory at any given time.
  
• There is a limit of 500 external system connections that can either be active or open at any given time. The default is set to 200 and it does not matter what type of connection is being used either.
  
• The default is 2,000 database items that can be returned in a request when using the database connector. The boundary is 1 million.
  
• There are three variables that you must always consider: the number of items in the external list, the number of columns per item and the size of each item.
  
• Profile pages display data from external content type data. The performance of these pages is driven by the complexity of the associations to external systems.
  
• The internal process for bringing external data into SharePoint via BCS is pretty simple. There is load (queries external source and loads into SharePoint), process (applies sort, filter, group processing) and render (display data onto page). BCS does not have in-memory cache for external items. Data has to go through load, process and render each time an external list is refreshed. Knowing this you need to make sure you control the amount of data that is processed at any given time.
  
• Microsoft's recommendation is to keep the number of items to be processed as possible by reducing the amount of items returned from external systems. It is recommended to keep the number of returned between 100 to 500 rows and it is recommended to not exceed 2.000 rows. It is recommended to use filters to ensure you work within these guidelines. More can be returned if needed but this needs to be done by an administrator.
  
• Rendering external list can be intensive for both the WFE and the application server. It is recommended to keep the number of items being displayed at any time to around 30. Note that the number of items that are rendered is not the same amount of items that were processed. The number of items rendered is controlled by the external list view that is on a SharePoint page.
  
• It is recommended to reduce the number of columns from external list. Obviously a large number of columns will affect performance.
  
• When rendering it is recommended not to use large-sized columns in list views. Columns larger that 1KB should not be utilized in a view. However performance is more affected by the number of items and not the size. So it is recommended always try to keep the number of items lower for better results.
  
• When designing an external list which is using BCS, make sure the default view will be the view the user needs to see the most. If the user needs to sort or filter the view, that will require data go back through the load, process and render process.
  
• For a profile page, the number of associations is the key for good performance. It is recommended to not exceed two associations. There will be lower performance of both throughput and latency when large number of items in an association.
  
• Diagnostic logging of BCS can become a factor for performance and it is recommended to lower when not doing testing.
  
• Performance of the external system will have performance implications to BCS and you need to sure those systems perform well.
  

Thoughts: I personally think is an area that has not been explored enough. Understand that BCS is another layer that the data must pass through. In Microsoft's detailed testing results for BCS they only tested WCF Web Services using .NET data types and SQL Server 2008 databases. As we know in an enterprise architecture not everything will be SQL Server or web service based. Data resides in formats all over the organization. Performance will be driven by factors out of scope of this blog.

Web Analytics limits

• There are three basic categories of analytics – traffic, search and inventory. Reports can be aggregated at the site, site collection, and web application level. The web analytics service mostly utilizes the application servers and SQL server, so that is where most of the capacity planning needs to be.
  
• At a high-level this is how the data is gathered. On each web server usage data is gathered and there is a Usage time job then calls the Logging Web Service to submit the data. The data is stored in a staging database for seven days. The web analytics components clean and process the submitted data and then every 24 hours the data is aggregated from the staging database and written to a reporting database. The aggregated data is stored for 25 months (default).
  
• Performance of the logging web service is directly related to the number of application servers on the farm. However the performance of the web analytics components is dependent on the performance of the analytics and reporting databases.
  
• Other areas of performance challenges are number of queries run each day, number of unique users each day, total number of unique assets clicked each day and existing data size in the reporting database. Basically the point is, the more data and more interactive your SharePoint farm is, the more performance overhead there will be to process that data for reporting purposes.
  

Workflow limits

• Workflow throughput can be affected by numerous factors such as the number of users, the type of workflow, the complexity of the workflow, dependencies on external calls and frequency on the number of users accessing the workflow. Note that workflows that use data from the content database or are registered for lots of events will run slower. So if you reference a large SharePoint list or you call out to an external database obviously that will take a little more time.
  
• Microsoft's testing found that throughput for workflows topped out at three to four WFEs added.
  
• There are some administrative settings that can be set which will affect the performance of workflows.
  
• Postpone Threshold is the number of maximum workflows that can run concurrently against a single content database. Subsequent ones will be placed into a queue. The default max is 15 workflows. It is important to understand this is the number of workflows that can be in progress at one time and it not the total number workflows that can be in progress.
  
• Batch Size is an exception to the Postpone Threshold limit and can pick up larger amounts of workflow items from the queue. The default batch size is 100 and this can be set per service instance.
  
• Another configuration is the Timer Job Frequency which is the frequency that the workflow time service runs. This service is responsible for picking up items off the workflow queue based on the interval. The default is every 5 minutes.
  
• I found this interesting but you should consider how all three of these configurations are used together to affect workflow performance. Workflow throughput is impacted based on how quickly operations get out of the queue and then processed. In the example I read, if there are 1,000 work items in the queue, it will take the time job 10 runs to execute all of them (Batch Size), which would take close to 50 minutes to execute (Timer Job Frequency). Modifying the Batch Size would increase that at the expense of taking up lots of processing resources.
  
• Another thing to make sure you do not do is use the same list for workflow history and task lists. Lots of items will be written to these lists as time goes on. Allowing these lists to get very large will affect performance.
  
• To keep the task lists from not getting too large there is a workflow job (autoclean) that will delete workflow instances and associated tasks that have been in a completed state for greater than 60 days. It is recommended that if historical information is needed past that, to write the data to other locations as part of the workflow. Workflow history items are not part of this autoclean job and a script should be written to purge those items.
  
• Another maintenance trick I learned which can affect performance is removing workflow columns. If you have a list with like 50,000 items in it, and you remove the workflow status column will cause database operation proportional to each item in the list, which can flood your SharePoint environment with transactions. It is recommended to just turn off the workflow so no new instances can be created and may do the operation during off hours.
  
• Another performance consideration when building workflows is make sure you are not going to violate SharePoint's database locking for modifying data at the same time. You need to make sure users and/or other workflows do not try to access the same thing at the same time.
  

Thoughts: I found the discussion on how to configure the workflow service very interesting because I had really never thought about that when I had built workflows in MOSS 2007. I can see how you may want to configure this very differently based on the site where the workflow is running. They had a creative example that was discussed where if you want to dedicate running workflows to a specific machine in the topology, you could try to lower the postpone threshold to force all workflows onto the queue so the batch job would pick them up on a specific machine. You would have to set the batch size to be high so that it picks up the work items before another machine. If you want to make sure the processing is shared across all the machines, you could try increasing the postpone threshold so items never sit around for a long time.

Excel Services limits

• Excel Services performance is directly correlated to the size and complexity workbooks that are hosted within SharePoint.
  
• Excel Services is stateful meaning that workbooks will remain in memory between user interactions.
  
• Excel Services have most effect on both the application servers and the WFEs. Excel Services will not highly utilize SQL Server because the workbook is read as a binary blob and used within the Excel Service. Microsoft's finding was that bottlenecks only happened on either the application or WFE servers, which can be scaled out by upping CPU memory or adding new Excel Services machines to the topology.
  
• Within Central Admin there are several configurations which can be used to better manage performance.
  
• Maximum Private Bytes is a percentage value Excel Services can use of memory on the machine. If the machine is dedicated to Excel Services, you can up this percentage. This is one of the main bottlenecks identified by Microsoft.
  
• Memory Cache Threshold is another configuration available. Excel Services will cache unused objects in memory and by default it will use 90% of the Maximum Private Bytes. Increasing this number gives a better chance that the requested workbook would already be in memory when the user accesses it. However lowering this number will improve performance of other services if the application server is not just hosting Excel Services.
  
• Maximum Unused Object Age is a configuration that helps keep objects in memory as long as possible but you will want to again modify this if other services are hosted on the application server.
  
• There are also settings which are not global to Excel Services however they are specific to the trusted locations where Excel Services are being used. For instance there is max workbook size, max chart/image size, session timeout, volatile function cache lifetime and allow external data all of which can be used to better manager performance.
  

Thoughts: For SharePoint 2010 Excel Services I would say there is more anticipation for a more feature rich solution than what was available for SharePoint 2007. It is very hard in my opinion to come up with a one size fits all sort of assessment for performance when it comes to Excel Services. This is because each hosted workbook can be dramatically different than the other.

InfoPath Service limits

• There are numerous factors that affect the throughput of InfoPath Services, some of which are: number of users, type/complexity/frequency of user operations, number of postbacks per operation, and performance of data connections. When building and deploying forms, this should always be accounted for.
  
• In the design of the form, try to optimize the first request to the form template. Try to exclude custom onLoad events, queries or business logic that occur when opening the form. You should try to make these operations as user driven as possible. What we want to do is delay the creation of session state data in the database until an actual post occurs. If the form only have one post, meaning when it is submitted, no session state for the form will ever be created.
  
• If the InfoPath form needs to be saved to a document library, it is better to submit the form into the document library rather than saving it. The reason is a submit operation triggers only one post but a Save will trigger two posts.
  
• Document library forms can support more throughput that InfoPath list forms.
  
• Form complexity such as the number of controls, rules, encapsulated data, etc. dramatically affects performance. The more complex, the more pressure will be put on the WFEs to support greater throughput.
  
• It is recommended to minimize the amount of controls displayed at any given time. If you do have a lot of controls that have to be displayed, it is recommended to put them into a secondary view so that you reduce the first page loading we talked about earlier. This is not just to reduce the amount of HTML generated; there can be a lot of presentation rules which can slow performance down.
  
• Large amounts of complex XML can cause more performance issues. Be careful of using attachment controls as the binary of the attachment will actually be embedded into the XML of the InfoPath form, making it very large.
  
• Database locks prevent multiple users from making changes to the same set of data. Since an InfoPath form is more than a piece of data in SharePoint, a lock is placed on it while it is edited. So it will not support concurrent usage of the same form well.
  
• It is recommended to return filtered data directly to the InfoPath form instead of filtering it within SharePoint.
  
• When reviewing the results of Microsoft's test, there was always more performance overhead when the form has web service submits.
  

Thoughts: From a performance perspective, not much has changed from a performance perspective. Yes - InfoPath 2010 form services will be faster than SharePoint 2007 – but the design concepts to create forms that perform well has not really changed. Your design of the forms drives performance and you should not lean on adding more WFEs to solve your problems.

Additional References:

• http://blogs.msdn.com/b/infopath/archive/2009/10/27/infopath-2010-performance-highlights.aspx
  
• http://blogs.msdn.com/b/infopath/archive/2008/05/09/designing-browser-enabled-forms-for-performance-in-infopath-forms-services.aspx
  

Visio Services limits

• There are three basic factors that affect performance: drawing size, complexity and data connectivity.
  
• Typically Visio Services are tough on the WFEs which will require you to scale up or out.
  
• The initial file size of Visio web drawings is 50MB. This can be changed by the administrator however larger files will increase memory required for Visio services, increase in CPU usage, reduction in service requests per second, increased latency, and increased SharePoint farm network load.
  
• There is a configuration called Visio Web drawing recalculation timeout which changes the maximum amount of time it will spend recalculating a drawing after a data refresh. The default it 120 seconds. Increasing the timeouts will increase CPU and memory, reduce application requests per second and increase latency across all documents.
  
• For Visio services there is a Minimum cache age setting which is used for data connected diagrams. This setting is between 0 to 24 hours. Setting this to a lower value will reduce throughput and increase latency because recalculations have to occur more often. This subsequently reduces CPU and memory.
  
• As well there is a Maximum cache age for non-data connected diagrams that determines how long to keep the diagram in memory. Increasing this value will decrease latency for diagrams that are used a lot. However increasing the maximum cache age will increase latency and slow down throughput for all items that have not been cached because the cached items consume available memory.
  

Thoughts: I found this section interesting to read because it gave me insight into how Visio Services is actually working. I had heard the Visio services can be data driven. When reading this you need to focus on caching as part of your design and understand the types of documents that you plan to post.

PerformancePoint Service limits

• The factors that most affect throughput are number of users, type of user interaction, frequency of use, number of postbacks in an operation and data connection performance.
  
• When a scorecard uses an Excel data sources, there is a limit 1 million cells per query.
  
• In a dashboard that is using Excel as a data source, the max number of columns is 15 and 60,000 rows.
  
• It is recommended that if you do have business logic or complex controls, the demand on WFEs will increase and adding more WFEs will alleviate those issues.
  

Additional Resource

• http://technet.microsoft.com/en-us/library/bb660539(office.12).aspx
  

Word Automation Services limits

• The input file size cannot exceed 512MB.
  
• The frequency of the conversions can be configured by minutes. A lower number will allow the timer job to run faster. The default is every 15 minutes but the recommendation is every 1 minute.
  
• There is a threshold for the number of conversions to start per conversion process. Obviously if the value is high, this can cause intermittent failures.
  
• The conversion job size can support 100,000 items at one time. A large number of conversion items increase the amount of time it takes to execute.
  
• The total number of conversion processes is directly related to the number of core processors. You should have not more conversion processors running than you have of processors. The reason why you should know this is because word automation services can at times full leverage a processing core and cause issues if other services are hosted on the application server. There is a property of word automation services called Total Active Conversion Processes which can be used to "throttle down" word automation services such that it does not use too many processing cores per application server.
  
• The work automation service database size cannot exceed 2 million conversion items. Items are not deleted from the database automatically after processing; so administrators will have to write jobs to remove the history. Exceeding 2 million items can cause performance challenges the longer the word automation service runs.
  
• When scaling out word automation services it is recommended to add more application servers and potentially add a new SQL Server which has the work automated services database dedicated to it. Another scaling recommendation would be to create dedicated word automation application servers.
  
• There is a fundamental performance limitation with converting PDF and XPS file types. Adding servers will not always solve the challenge.
  

Office Web Application Services limits

• This includes the Word Web App, PowerPoint Web App and OneNote Web App. There are several different perspectives that you have to look at each of these. Specifically you should focus on viewing versus editing because the resources required are quite different.
  
• The drivers for performance are expected concurrent users and what type of operations are going to be done. Microsoft's initial recommendations are 100 daily users with average of 10 concurrent can be supported by 1 WFE and 1 App Server. Going to 1000 daily and 30 concurrent would require 2 WFE and 2 App servers. Going to 10,000 daily and 300 concurrent would require 4 WFE and 3 app servers.
  
• Heavy Word Web App viewing will typically require more CPUs to the application servers.
  
• Heavy Word Web App editing and OneNote Web App viewing/editing require more CPU to the WFEs.
  
• Heavy PowerPoint Web App viewing requires more CPUs to the app servers and more memory to the WFEs.
  
• Heavy PowerPoint Web App editing requires more memory to be added to the application services.
  
• If you did not know what this is this feature enables presenters to broadcast a slide show from Microsoft PowerPoint 2010 to remote viewers over the web through SharePoint. It is positioned as a low-infrastructure presentation capability. Heaving PowerPoint Broadcast will require the WFEs to have more CPU. It is recommended that if you have extremely PowerPoint Broadcast feature to create an environment to support that. The reason being is web browsers will hit the server every second to get the latest changes to the slidedeck.
  
• Much of the limits associated to OneNote Services are directly tied to the limits for list and libraries. This is because each section in OneNote is stored as folders and documents in a library.
  
• The maximum size for each section of an OneNote section is again driven by the file size limits for lists and libraries.
  
• If there are embedded images, files, etc. in OneNote, that are greater than 100KB, they will be split out into their own binary files within the SharePoint library.
  

Access Services limits

• The performance of Access Services is dependent on the other applications that are hosted within the service. It would be recommended that Access Services to be dedicated to a service if there is lots of data to be managed.
  
• The amount of data in the tables and the size of the queries will have the most impact to performance of Access Services. It is recommended to limit the size and complexity of queries and well as control the amount of data that flows through. This can be done in Central admin. First the query can be configured by controlling the max sources per query and max records per table. Second results can be configured by max columns per query, max rows per query and allowing out joints. Third processing complexity can be configured by max calculation columns per query and max order by clauses per query.
  
• There is another configuration called All Non-Remotable Queries. Way Access Services work under the hood is that is augments SharePoint query operations support Access Service features, like working with large amounts of data. If there are performance challenges, you can change this configuration so that Access Services just uses the SharePoint query operations – subsequently making data fetches less complex (at the same time not as robust).
  
• Access Services is stateful. It will maintain in memory cursors and record sets between user interactions. Microsoft's testing did not determine that this would be a bottleneck.
  
• There are no special hard requirements for WFE or application servers to support Access Services. It is recommended to increase capacity by scaling up your existing servers or scale out by adding more servers into the SharePoint topology.
  
• To support more users it is suggested to add more CPUs to the servers and more Access Services computers if needed. Microsoft's testing found out that if you add too many Access Service machines the WFEs can become a bottleneck to the Access Services. This tends to happen when the four Access Service machines have been added.
  
• Reporting services are utilized with Access Services. It is recommended to create a dedicated machine to support it if processing of reports is taking a long time.
  
• There are several performance counters that can be used to determine performance that are in the whitepaper.
  

Thoughts: Again reading over this gave me some great initial insight into the architecture of Access Services. I actually intrigued to see how this will be utilized with organizations.

New SharePoint 2010 Features Part 2 – Enterprise Data Access and Management Features

• Introduction
  
• Enterprise Data Access and Management Features
  
• Development Features
  
• Enterprise Content Management Features
  
• SharePoint Office and BI Features
  
• Administration Features
  

Enterprise Data Access and Management Features

In this section I will discuss many of the new features associated to working with enterprise data with SharePoint 2010. SharePoint 2010 is much more compelling with than SharePoint 2007 with more robust out of the box functionality that allows you to integrate with enterprise data.

Business Connectivity Services (BCS)

Business Connectivity Services (BCS) is probably one of the top five most important new releases for SharePoint 2010. BCS is the enhancement to SharePoint 2007's Business Data Catalog (BDC) which was used to provide read-only access to Line of Business (LOB) data across the enterprise into SharePoint 2007. Note if you see BDC mentioned in the context of SharePoint 2010, it now means Business Data Connectivity.

BCS for SharePoint 2010 supports the follow:

• Supports full Create, Read, Update, and Delete (CRUD). SharePoint 2007's BDC just supported only read operations.
  
• Supports integration with databases and web services and additionally WCF, DLLS and custom domain services.
  
• Provides an extensible model provider which provides a pluggable framework for which developers can write to other data source types that are not provided out of the box with BCS.
  
• BCS supports batch processing for read operations unlike its predecessor which could only support processing a single item at a time.
  
• Can now work with BLOB data type (i.e. streaming BLOBs of data from external systems).
  
• It can support read/write of complex data types and not just primitive types.
  
• Rich API that can be used by custom applications to use data services provided by BCS.
  
• Tools like SharePoint Designer 2010 can be used to build solutions using a declarative model which does not require a developer to write code.
  
• Users can effectively consume and work with data from multiple external data sources in a user interface that they are familiar with such as SharePoint lists, web parts, Outlook contacts/tasks/appointments, MS Word documents, etc.
  
• BCS has a robust cache mechanism for supporting users to work with data provide by BCS offline. Solutions written in correlation with MS Outlook 2010 and SharePoint Workspace 2010 and work with data while offline and then be synchronized when back online.
  
• BCS can be integrated with SharePoint Enterprise Search to allow users to search external SharePoint data.
  

Business Connectivity Service Thoughts

I am really excited to see BCS leveraged as a solution to build composite applications that expose enterprise data instead of building expensive, custom applications. Typically the cost of building these custom composite applications come in the form of heavy design, development and maintenance. In many instances the demise of these custom applications is that a significant investment is made in engineering of the solution but the flexibility and scalability does not meet the future business demand. BCS removes some of these challenges in that it is a framework provided by SharePoint to expose enterprise data so organizations, so they can focus on solving existing and reacting to new business requirements instead of trying to build an enterprise data access framework.

On a side note - I also believe that SharePoint 2007's BDC did exactly what it was supposed to do. SharePoint 2007's BDC was originally intended to be an enterprise search solution that could expose data from web services or databases in the result search of a search. However this was confused when BDC web parts were created to expose enterprise data. When these web parts were utilized business users typically asked to perform full CRUD operations (FYI, it was actually possible to support CRUD with SharePoint 2007 BDC but it was not great solution). Microsoft stepped up to the plate and solved both problems. First they acquired FAST and integrated it in the SharePoint solution architecture and we now have a true best in class enterprise search solution to work with which SharePont 2007's BDC cannot even compete with. Second they created the Business Connectivity Services which can support full CRUD operations which is fully integrated with the SharePoint Foundation.

Enterprise Search (Reference)

SharePoint 2010 includes significant improvements in regards to enterprise search. The biggest is Microsoft's acquisition of FAST (one of the top three enterprise solutions in the marketplace) which has subsequently been integrated into SharePoint 2010. FAST is an extremely rich and deep application that should be dived into separately. Basically you now have two options SharePoint 2010 Search and FAST for SharePoint 2010.

Let's first understand at a glance what in general the improvements to search for SharePoint 2010 are:

• Better Keyword Query Syntax has been incorporated into SharePoint 2010 Search. Such new features as Boolean, proximity, synonym, wildcard operators, prefix matching for free-text, property restriction queries and property operators for non-text fields have been incorporated. What this basically means is creating complex queries to run against the search service is much easier in SharePoint 2010. Note that the new Keyword Query Syntax is supported for both SharePoint 2010 Search and FAST for SharePoint 2010.
  
• The search web parts are unified so they support both SharePoint 2010 Search and FAST for SharePoint 2010. This will ensure there is some consistency in configuration, management and user experience regardless of which search engine you use. Note that the search web parts are built upon the federated search object model.
  
• The search results page new includes a new refinement panel that gives the user a summary about the search results and provides them the ability to quickly filter results based on the taxonomy of the data.
  
• New connector frameworks for both SharePoint 2010 Search and FAST for SharePoint 2010 have been incorporated to support indexing data external to SharePoint. SharePoint 2010 Search utilizes the new Business Connectivity Services (BCS) while FAST has its own highly scalable connector framework. Both of the connector frameworks are significantly better than the BDC used for SharePoint 2007.
  
• Relevance has been enhanced for SharePoint 2010 Search while FAST provides an extremely rich and advanced relevance model. For SharePoint 2010 Search, we now have the ability to create customized ranking models to improve result relevancy by collection, content source and data verticals.
  
• Federated Search is again supported by both search options for SharePoint 2010. If you are not familiar with it, what Federated Search supports is the ability to display search results that were crawled by other search services such as Bing, Google, etc.
  

Enterprise Search Thoughts

I personally believe that FAST for SharePoint 2010 is a game changer and one of the compelling reasons why organizations should upgrade as soon as possible. The features and functionality of it are unparalleled and it is a mature product that has been injected into SharePoint. Content and data stored in or out of SharePoint will be easily accessible utilizing highly advanced relevance models. To learn more about Enterprise Search and FAST in general to understand the value proposition of it, please read this blog series.

List Enhancements (Reference)

There are several new features that have been introduced into SharePoint 2010 to support their initiative to better handler larger quantities of data:

• List throttling – To ensure that open ended queries for list data to not effect performance of the site for other users. Thresholds can be set to ensure web application level.
  
• Referential Integrity – In previous versions of SharePoint it was possible to use lookup fields or custom events to enforce referential integrity betweens items between lists. With SharePoint 2010, referential integrity can be set up directly between two lists and SharePoint will manage it from there forth.
  
• Enforcing Uniqueness in Column Values – There is now the ability to select which column in a list will define uniqueness. This does not have to be the internal ID column of the list.
  

List Enhancements Thoughts

The ability to efficiently and effectively handle list with large amounts of data is a very important new feature to SharePoint 2010. In SharePoint 2007 it was possible to create efficient solutions that had SharePoint lists with more than 2,000 items, however complete custom code solutions had to be written. I still have some hesitancy to make sure that data management be well thought out before dumping extremely large amounts of data into a SharePoint list (my old blog entry). If data is highly relational or must be reported on, it is probably best to use SQL Server and then expose the data to SharePoint using the new Business Connectivity Services.

Still the new ability to manage very large quantities of data is not just for SharePoint Lists, it is also for managing large values of documents in a document library. It is very exciting to see this and will be very interesting to see how SharePoint lists evolve as a new enterprise resource for data storage.

Microsoft Sync Framework and SharePoint Workspace 2010 (Reference)

Microsoft Sync Framework is a unified synchronization architecture that allows for bidirectional communications between SharePoint and Line of Business systems. This framework as a specific focus on synchronizing SharePoint data stored in either lists or document libraries. This architecture is used as the backbone for SharePoint Workspace 2010 which is the replacement for Microsoft Groove 2007. What this tool supports is the ability to work with data stored within SharePoint in a disconnected environment. This includes lists, document libraries and even data within BCS.

Microsoft Sync Framework Thoughts

This again is another very exciting release for SharePoint 2010 and provides a tool that no CMS competitors are providing. Users now can work while travel, we can ensure they have the latest information locally, etc. Groove was an initial release given this is Microsoft's second release of this technology I have high expectations that it will be a very effective tool.

User Profiles and Social Data (Reference)

There are several aspects to the new features that have been integrated into SharePoint 2010 to support social computing. There are new APIs discussed in this article about activity feeds, social data and user profiles. Activity Feeds API supports the sharing of data between users. Social Data is a new API that has been created to facilitate social tagging, rating and comments for content that is stored within SharePoint. The User Profiles API has been augmented to support both organization and user profiles.

The new social computing features of SharePoint 2010 support scenarios such as (this is not an entire list):

• The ability to find commonalities between two user profiles.
  
• Providing the ability for users modify profile data
  
• Adding links to your profile.
  
• Adding colleagues to your profile.
  
• Social Data Statistics Web Part that displays social statistics such as each URL that has been tagged and their terms, the number of times that term has been used in a tag, each user who has added a social tag.
  
• The ability to approve who is marking you as a colleague.
  
• Ability to send content to colleagues.
  
• Ability to pull in external social computing data like from LinkedIn.
  
• Ability to create organizational charts based on enterprise data.
  
• Ability to rank, tag and comment content within SharePoint and use that information to the most relevant information to users based on the data's actual usefulness.
  

User Profiles and Social Data Thoughts

At first glance this does not really seem like much however this new foundation provides a foundation for building social computing business solutions that was significantly lacking in SharePoint 2007. I cannot tell you how many clients wanted to incorporate social computing into their Internet, Extranets and Intranets. Common use cases were who is the SME for what I need and who do I contact them? What content is considered to have the best information? And so on. To support this in SharePoint 2010 typically highly customize solutions had to be created.

I know when I first started doing SharePoint 2007 projects I was little skeptical of social computing and how it applied to the business world. However client after client I saw that this was significantly important to companies and I even saw how it would be important to me at my company. Today I rely on the fact that I have been at a company for several years, I know everyone, I have seen people come and go, I have seen lots of projects and I know where to go find something or who I need to call to find it. But if I am new, how do I find what I need? We have talked about search, but that is only one piece of the puzzle. There is still an element of finding information which cannot be gleaned by a search engine. The data indexed is only as good as the metadata that supports it, and if there is no metadata, it is not relevant.

Another scenario is very geographically dispersed or remote environment can highly benefit from the adoption of business social computing. We are so used be able to crowd around the water-cooler, well we can now do that with SharePoint. With the new Social computing features we can find out who is doing what and when and if you should chime in with some information that stop that person from potentially making a costly business decision.

Now we are empowered with SharePoint 2010 to come up with these new business social computing solutions and I am truly excited to see how this is used over the few months.