FAST Introduction and SharePoint Search Evolution

There is a lot of information that is coming out from the SharePoint 2010 conference and one of the biggest ones is the integration of FAST into SharePoint 2010. What is FAST? FAST is an enterprise search engine that Microsoft acquired and they have placed a significant investment into. The most important thing you should know right off the bat is FAST does not equal SharePoint. FAST is an enterprise search platform which can be used as the search engine for SharePoint. Up to this point Microsoft has not provided a way to search for content across the enterprise. What we have done to compensate for this is build custom applications or purchase products like FAST and Google Appliances to do enterprise search.

This is what I have seen with the evolution of search solutions in the context of SharePoint. SharePoint 2001, nothing to really discuss but with SharePoint 2003 we started to get a taste of what we wanted for Search. We found that the search did not really work well in SharePoint 2003 (cross site searching did not work) and many customers who were using SharePoint 2003 said it simply did not work. It did basic text searching of content within SharePoint but it was missing key things like relevancy. This created a small market of third party vendors who creates search solutions for SharePoint. Remember, at this time Google had become the search engine of choice, as every day business users would just say go Google something and get the answer. Problem was we did not have the same search engine that we could use internally with a company, organization or enterprise. As result FAST, Google, Autonomy, etc. created enterprise search solutions that could be used within a company enterprise and that many these features that were required by the business user.

Then SharePoint 2007 came out with Enterprise Search. It was a significant improvement over what we had with SharePoint 2003 but it was still far off from being an enterprise search solution. They improved the user interface, allowed for target content taxonomy searching, they added a relevancy model, best bests, synonyms, administrative features, reporting, an API we can build customizations to, added security using an access control list (ACL), and business data search using the business data catalog (BDC). All the stuff needed when creating an enterprise search platform. We now had the ability to search for data inside and outside of SharePoint, we could rank the search results based on who you were, we could analyze searches to improve the user experience, etc. however it still seemed to fall short. The core problem I go back to is users are expecting that Google experience; and not just doing text searching. SharePoint tried to solve some of that but in the end it fell short.

One thing that had always been the most interesting is the introduction of the business data catalog (BDC) to provide a single result set of data from multiple disparate data sources. This was the most interesting search feature for me when SharePoint 2007 came out. This is where they tried to become an enterprise search engine because you go to one place, you enter something to search on, and you query against many different places but get back a single result set. I personally was able to use it successfully to index custom SQL databases of HR related data for several clients. So when they searched for a person, they were able to get more information about that person other than just information stored in Active Directory. Now the BDC had lots of limitations including only able to call databases, stored procedures and web services, no ability to do data transformation, an API that was very hard to develop with and had limitied scalability.

With the introduction of FAST as part of the Microsoft stack, they really have a true enterprise search engine. FAST has a significant amount of features and functionality, which I have not even touched upon. In my next blog, I intend to write about some of these core features and capabilities that are needed for an enterprise search solution and how they are used to meet your business users needs to find the data.

For more information on the value proposition of FAST, I have written the following two blogs:

• Why is a FAST Enterprise Project Important
• What is a FAST Enterprise Search Project

BDC Crawl Missing Security Identifier

SOLUTION - We rebuild the SSP

Error: I ran across the interesting little error with the BDC which I did not expect. You may know that in the Share Service Provider (SSP) you need to give user permissions to the BDC. Well you may come across this error when you index a content source that is using the BDC:

The parameter is incorrect. (Could not create a security identifier for the identity 'XXX\YYY'. This identity may have been deleted.)

Symptoms: If you view the BDC content source crawl log there will be no errors however this error will be listed as an error in the Crawl Settings for search itself. This error will be created for every record that was indexed.

Cause: Well I found out that the user account in question had been deleted from Active Directory.

Resolution: To resolve the issue I had to remove the user account from the Business Data Catalog permissions as well as from all the BDC applications and entities permissions. Once I did that, run a full index, and everything will work.

Background: The first question I had was why are there no errors on the BDC content source? Knowing what I know about the BDC I suspect this is what occurred. First, there are connection errors to the BDC data source (SQL Server in this case); for instance there were no access issues to the database or the database objects that were being used in the Application Definition File (ADF). Second, knowing this I can only assume there is an error with the way the index is being built. Basically the error is in reference to the validation of Active Directory account that has permissions to the BDC application and its entities.

Deeper Dive: Digging a little deeper the following is the root of the issue. Many developers create an ADF using a tool or by hand and may not add an AccessControlList node. These nodes get added to the ADF when permissions are added in Central Administration. The AccessControlList can be added to both the LobSystem and Entity nodes.

The AccessControlList nodes must be added to the ADF prior to it being crawled. This is why if an administrator goes to Central Administration and gives a user or a group permission to the BDC, they must do a new full index of the BDC content source.

The following is an example the AccessControlList. In this example, account XXX\YYYY is being given full rights. This XML can be added to both the LobSystem and Entity node.

<AccessControlList>
  <AccessControlEntry Principal="XXX\YYYY">
    <Right BdcRight="Edit" />
    <Right BdcRight="Execute" />
    <Right BdcRight="SetPermissions" />
    <Right BdcRight="SelectableInClients" />
  </AccessControlEntry>
</AccessControlList>

Avoiding this Error: The error is occurring because the account XXX\YYYY does not exist in Active Directory account. It is recommend that permissions through a Active Directory group to minimize the occurrence of this error.

SharePoint Customizing Display of Profile Data

We recently took over the management and support of an Intranet site. One of the first things our list to knock is to fix the way profile data is being displayed. A couple of the issues we have are:

• Users log in with their AD accounts but there are several data elements that need to originate from different data sources.
• mySites are turned on and we need to fix currently provisioned sites to show this extra data. As well, all new sites need to need to appear correctly.
• There is also the userdisp.aspx page which displays profile data. We need to make sure that is in sync with data displayed on the mySites.

We did some research and found some really good stuff out there that will help us get this problem resolved.

• Using BDC to Augment Profile Data – here is a good guide that shows you how to augment data being pulled with the BDC.
• mySites Redirection – this is a pretty good article that shows how to do redirection of the userdisp.aspx to the mySites page. We are not going to take this approach but worth noting.
• Customizing mySites 1 and customizing mySites 2 – both of these are really solid articles that provide similar approaches for overriding the presentation of mySites. Both entail Feature Stapling to existing sites and using FeatureSiteTemplateAssociation for new sites. This helps get around not messing around with the Microsoft owned site template.
• Customizing User Information Page – this is a three part series on how to modify the userdisp.aspx page and customize it.

Between all of these you should be able to construct a good end-to-end solution to override and display of profile data.

Also take a good look at this Code Plex Project. Really cool.