2007 Products Nearing End of Support

I saw that there was a reminder going around that some software was coming end of extended support in Calendar Year 2017.  For instance:

• Office 365 ProPlus client using 2013 rich client is coming to an end 2/18/17.  This means you need to finish your transitions to Office 365 ProPlus 2016 client as soon as possible.
• Exchange 2007 will be ending 4/11/2017.
• Office 2007, Project Server 2007, and SharePoint 2007 are also coming to end of extended support.  I still see customers with SharePoint 2007 sitting out there.
• OCS R2 (2007) has a little bit more time, 1/9/18.

I just call this out because in my mind, these 2007 releases was the first year where you could start seeing Microsoft’s vision of bringing together Exchange, SharePoint and OCS into a single collaboration platform.  That little presence “jellybean” was the start of something great.  Seeing what we have now with Office 365 where productivity is delivered as a SaaS service where Exchange, SharePoint, OneDrive, Skype, Office, Groups, Teams, etc., etc. are delivered in a fully integrated and seamless user experience brings a smile to my face. 

Here are some references:https://support.microsoft.com/en-us/lifecycle/selectindex
https://support.microsoft.com/en-us/help/3198497/office-2007-approaching-end-of-extended-support
https://blogs.technet.microsoft.com/exchange/2016/04/11/exchange-server-2007-t-1-year-and-counting/

InfoPath Being Retired

Announcement: Well there was a big announcement in the SharePoint world that hit me personally – InfoPath is being retired - http://blogs.office.com/2014/01/31/update-on-infopath-and-sharepoint-forms/

Reflection: I had been waiting for public announcement and it was pretty obvious it was coming down when we saw that no new additions or investments (of any importance) we made into InfoPath 2013 and InfoPath Form Services.

Now InfoPath has a special place in my heart and I will say that my love for InfoPath put me where I am today. Really. When I first started at RDA Corp back in 2005 I was just a .NET engineer. My first project I was thrown into an engagement where I had to learn InfoPath 2003, K2.net 2003, BizTalk 2004 and SharePoint 2003 in a week and make them work. That project change my life. I built this solution (with some smart people) which eventually was the finalist for solution of the year at the Microsoft World Wide Partner conference. It had tons of InfoPath components which were hooked on top of a VAX mainframe. I subsequently got hooked into the developer community in ways I had not done before. I starting doing more and more InfoPath and K2 projects and then I got really hooked into SharePoint; the rest is history. I was able to get a job as a SharePoint TSP at Microsoft which has transcended into a career doing Office 365. I can say, it was InfoPath that sparked my love for the Microsoft productivity stack to build business solutions. I started saying “why do I need to build solutions from scratch when I can take all the Microsoft solutions together and build something”.

Over the years on this blog I had wrote extensively on how to do development, best practices and build solutions for InfoPath - http://www.astaticstate.com/search/label/InfoPath. My first blogs in September 2007 were focused on InfoPath. I thought it was a great solution with SharePoint to build business forms and automate them using simple or complex business processes. Reflecting back, I found that InfoPath did a lot of great things for standard to moderately complex forms but when you brush up against the edges (which is what I had to do a lot as a consultant) you found tough things you would be able to resolve with a custom web form. I would always be put into these debates with colleagues on this topic.

Where Do We Go From Here: I believe Microsoft product team is making the right decision. InfoPath had its day, it is a great concept behind it to create forms quickly, extract the XML, shred it and sent it to a database. However today there are so many development frameworks for rapid development and Microsoft is going to continue to create new solutions that will help customers build forms and process automated solutions. Support for InfoPath is not going away anytime soon, they are keeping support through April 2023 (not a typo) and I still say InfoPath is a great solution, just nothing new is being introduced. They noted they plan to create some migration scenarios in Q4 of CY2014, so I will be keeping an eye out for that.

Architecture Considerations for Moving SharePoint to Office 365 and SharePoint Online

Introduction
Why are customers looking to come to the Office 365 cloud? Customers want an environment that can scale on-demand. Customers do not want to be in the business of managing and patching software . Customers want a solution that give better business continuity and service level agreements to their users. Customers want solutions that are better governed and that will force them to do better governance. Customers want to be in the business of building solutions (i.e. an airplane company should focus on building planes; not writing enterprise applications from the ground up).
When I talk with customers about SharePoint Online, this is where they want to move to.
In this blog I am going to talk about how an organization should be looking at moving to SharePoint Online.

1. In first half we are going to analyze SharePoint Architecture and Governance and review what potential blockers people see when moving to the cloud.
2. In the second half I am going to have a more detailed discussion around how to architecturally plan to move to SharePoint Online.

Why the decision is not always straight forward?
Coming to SharePoint Online in Office 365 may not sometimes be the most straight forward decision for organizations that have made a large investment in SharePoint. Going to SharePoint Online is really easy for organizations that are in it for document management, intranets, OOB features, SharePoint Designer, etc. It gets harder for organizations that have had poor governance of their on-premise environments, made signification investments with custom code, utilize features not available in cloud, have a heavy reliance on third party solutions, etc.
There is a solution to these issues but it requires an organization to take a step back and look at what they have.
Poor governance is probably one of the largest changes organizations have faced with SharePoint. These challenges could have been avoided thought with good forethought and planning. There is not always a one size fit all solution either; SharePoint is powerful platform that can be used to be used to implement a very broad set of business requirements.
SharePoint Architecture Foundations
The following a Venn diagram that I always draw on whiteboard with customers.

I typically say that that:

• Information Architecture – Is content types, taxonomy, topology, site collections, sites, libraries, lists, and solutions that drive the management of this content. The information architecture should be driven by business requirements.
• Logical Architecture – Is the architecture of SharePoint services, web apps, databases, etc. to support the information architecture.
• Physical Architecture – Is the configuration, deployment, farms, etc. that actually host the SharePoint logical architecture.

I admit this is open to interpretation but let’s use this as a foundation for discussion.
Here are some of the biggest problems organizations that have come up when I discuss the diagram above:

• Started with Physical Architecture – This is the first mistake that many seem to make. Smart IT are concerned about how many servers to buy and configure they forget to actually create an environment needed to support their business requirements. What usually happens is the environment is not scaled or organized to support the actual utilization; this is where the trouble starts. This pretty much goes away if you go to the cloud.
• No business requirements and software design best practices – One of the biggest, consistent challenges I see many organizations have with SharePoint is no business requirements. I typically just see organizations create very light documents (a bullet list) of requirements and just start building. I very rarely see organizations actually do software design best practices. There are no use cases, data models, ERDs, UI wireframes, data dictionaries, architecture documents discussing what SharePoint elements they plan to use, etc. Why; because SharePoint is such an easy platform to start building with. I personally believe SharePoint is one of the best solution platforms on the market but sometimes organizations need architect a solution fist. This does not go away with SharePoint Online.
• No Governance – Organizations usually forget to put Governance plans to manage content and solutions running in SharePoint. A solution can be anything from a group of team sites or a complex .NET application in SharePoint. I have even seen organizations create Governance plans but then not manage nor adhere to them. An analogy would be a organizing a ton of boxes in their basement really well (with labels too). However as time goes on, they just keep on stuffing and stuffing the room with more boxes so that entry to the room is blocked. A good SharePoint environment needs care and feeding. The information, logical and physical architectures need to have policies and procedures to manage it. Policies, procedures, service level agreements, etc. must be put into place. With SharePoint Online several of these Governance woes are removed because the Office 365 cloud manages the environment. Still Governance needs to be but in place to manage content and solutions deployed in Office 365.
• Over Regulation – I have also seen the term “SharePoint Governance” used as a crutch. I have seen organizations lock down SharePoint too much. My response to that is was there a requirement to lock it down? If so, the right thing was done like a publishing intranet or Internet site – no user should have rights to do what they want. There should be a locked controlled publishing and branding process. However it is perfectly acceptable to have areas of SharePoint that are pure collaboration. Your information architecture and governance plans will drive business users to put content and solutions in the appropriate areas.

There are more things to think about but I am getting a little preachy J
A Not So Unusual Situation
Here is a very common scenario that backs up what I just described. It is very common for an organization to start a SharePoint environment like the following. As you can see they start with a top level SharePoint site. They create some department level sites and some sub sites for their intranet. All in all this is a pretty good start; right?

However within a few months (after some very heavy business and user adoption) we have the following.

We see such things as:

• Team sites starting to sprawl underneath some of the sub-sites within a department. The challenge is being able to support collaboration in what was meant to initially be a publishing site.
• Navigation, presentation, branding and user experience is not consistent.
• We see custom applications either built from scratch or third party solutions purchased and embedded in sub-sites of a department. The challenge is should these custom solutions be hooked directly here?
• Department level project sites are created. The challenge is that as project sites grow and take on new responsibilities for the business, they need to be elevated and made accessible like other sites.

There tons more. And I have had customers tell me this is not an issue with “SharePoint”. They currently have these same exact challenges with their legacy intranet, portals, etc. So why do organizations keep on having these challenges? I usually point to a lack of attention to Governance and Information Architecture.
So How Should You Be Thinking About This?
What if content and solutions that are to be deployed are delivered within a framework? Not a “novel” idea either. Hopefully we can make it as simple as we can. Instead of just adding more and more sites and applications; have rules that drive where we put things.

For example:

• The Intranet site should remain a dedicated publishing site collection. This would be geared towards business users having read-only access with only a small set of content owners. Content is delivered in a consistent and clean fashion so that every site is the same giving the user a unified user experience.
• Create a separate site collection for team sites with appropriate service level agreements. In the team site collection, sites should be can be dynamic generated from pre-define site templates. Users are given the ability to do pure collaboration, standing up lists, libraries, etc. They should be able to share information to complete everyday business tasks. There would also be an expectation that this area is not uniform and that users can do what they want with these sites. Retention polices can be put in place to discard these sites if they have not been used for a pre-defined period of time.
• Create another site collection for business project sites. These have different support SLAs and Governance. These sites are a little bit more formal in nature with only a subset of people that would have access to manage them. Maybe there are designers from the IT department who have responsibility for building and supporting them. There could be policies and automated procedures to move content out of them to other site collections. Retention policies on these sites would be completely different than team sites and these sites may only be deleted when the project is over (or never deleted).
• For custom applications, instead of embedding the application into a sub-site, place them into their dedicated site collections. Navigation links can be made to the application from other site collections. This would give significantly more flexibility to move it or provide it more resources.
• Instead of creating one massive site collection where all scanned documents or records create multiple site collections and then route content to them based on business rules.

This list I provided is not meant to be an all-encompassing list of possibilities and/or solutions. Hopefully you will see that you need to start vertically and horizontally partitioning solutions and data based on such simple things as business rules, security access, data characteristics, etc. Knowing this will help you identify the types of site collections, sites, features, content types, managed metadata, site templates, permission levels, etc. that you need to configure SharePoint with.
The net result is you would take the previous diagram and start creating management boundaries based on the characteristics of the solutions and data managed in SharePoint. This is why Information Architecture is so critical for Enterprise Content Management system like SharePoint.
Putting everything into a single site collection really puts an organization into a tough spot to scale with the business. Microsoft has been publishing great technical diagrams for SharePoint 2010 (http://technet.microsoft.com/en-us/library/cc263199.aspx) and SharePoint 2007 (http://technet.microsoft.com/en-us/library/cc263199(office.12).aspx). Please review and get to know these diagrams well because they accurately tell everything I am talking about here.
Now some people counter this whole thing with why does SharePoint not give this to me? Why does SharePoint not auto-govern itself? SharePoint absolutely comes with a ton of features and capabilities that support a good governance model like Features, site templates, sandbox, managed metadata service, permission levels, and the list goes on. All of the configuration settings are there and organizations just need to set them as appropriate for their business.
I promise; we are coming back to the SharePoint Online cloud but we need to finish setting the stage.
How You Should Change Your Thinking
The following is a diagram a colleague (http://blogs.msdn.com/b/edhild/) and I commonly discuss with customers. I am shamelessly using it because this simple diagram really helps customers with a basic understanding of Information Architecture with SharePoint. Plus it supports everything that I just discussed. This is what we call the “Arch of SharePoint Data”. This is not all encompassing list, however there are different extremes.

There is publishing which are sites that are managed by a small group of users and read by a large community of read-only users. While on the other side is something like MySites which is my personal area to manage information and data. In between these two extremes are tons of different types of sites. Department sites, project sites, organization sites, custom application sites, team sites, etc. There are too many to even try to draw. Each one of these types of sites has different security, retention, data usage, transaction management, business rules, automation, etc.
I tell customers is that is perfectly ok to have a team sites area that are for pure collaboration where business users can spin-up a site, do some work on it for two months and then move on. Some people call this the “wild wild west” and that is ok. Just do not allow pure collaboration in you publishing area which is probably one of the most common mistakes J.
Coming full circle, you can see this is all about solution and data management. An Information Architecture is going to tell you the data characteristics. This will drive you to put content in one area versus another.
This is the beauty of SharePoint. It allows you to create and manage business workloads based on your specific business and mission. Once you have the Information Architecture nailed down, you can determine the Logical Architecture (services) you need and then the Physical Architecture needed to support this. By doing this, you know that both your Logical and Physical Architectures will be driven off real business requirements.
Where Does this All Fit with SharePoint Online?
Once you build an Information Architecture you will see that many of these partitions can be moved to the cloud. Intranets, team sites, project sites, my sites, light weight custom solutions, etc. can all be moved to the cloud. If you look at the “Arch of SharePoint Data” diagram, depending on your scenarios, there is a good chance that 80% to 90% of your solutions can be moved to the SharePoint Online cloud. For some organizations, they will be able to move everything up to the cloud. For some organizations they will have a hybrid.
Regardless the re-architecture should be a primary task in your SharePoint 2007 migrations to SharePoint 2010 anyways. Going through this exercise will put you in a position to move pieces to the cloud and with time, more and more.
So what are the Gaps with SharePoint Online?
If you want to know the exact answer it is clearly spelled out and completely available for you. Read either the Multi-tenant SharePoint Online Service Description (http://www.microsoft.com/download/en/details.aspx?id=13602) or the Dedicated SharePoint Service Description (http://www.microsoft.com/download/en/details.aspx?id=18128).
However really understanding these gaps depends on the perspective and approach you are taking with SharePoint. As I say a lot, “SharePoint means a lot of different things to different people”. I have seen customers extremely happy with using SharePoint with out of the box features and SharePoint Designer. While I have seen other customers use SharePoint as a full application development platform writing thousands lines of code. Knowing what SharePoint means to you will dictate your approach to the SharePoint Online cloud. The approach that I outlined in the first part of this bog will really help you with that decision process for moving to the cloud.
When you read the SharePoint Online service descriptions for both the multitenant and dedicated you will see that the gaps are really small. However there are some ones you must be aware of as they will affect your decisions on how to move to the cloud. The big ones that most people bring up are full trust code, business intelligence, FAST search, PowerShell, and Central Administration. This list will quickly change and become outdated because more and more features will be released with time. However let me address each one as it stands today:

• Full trust code is usually the first one the first challenges. Right now, the only way to deploy custom code to the SharePoint Online Multitenant cloud is using a Sandbox solution. For the SharePoint Online Dedicated cloud, full trust code is supported but it must adhere to a strict set of rules (which are publically available) and the code will only be deployed within set windows. Why such restrictions and strong governance? Well, for all the obvious reasons that would come up if you were tasked with having to run an extremely large SharePoint environment on-premise. What has been one of the biggest issues with SharePoint 2007 Governance? It was developers writing complex code on SharePoint and disregarding the fact that an error they write may take down other sites in the farm (like the content query web part that retrieved too much data on the home page of the intranet <g>). We need to ensure that there is strong security and good performing code and that there is no possibility the Company A can take down Company B. The only way to achieve that service level agreement is to have an environment for running governed code. The net effect is that there will be limitations but you will have that guaranteed uptime. My golden rule is that all SharePoint development (on-premise and cloud) should begin with Sandbox solutions and only when the customizations cannot run in the Sandbox then build as full trust solutions. Doing this will ensure you have agility to move to the SharePoint Online cloud when you are ready. If you really need to do complex operations and manage data structures consider using services Windows Azure integrated through Business Connectivity Services (BCS), Silverlight, etc. But still that may not always suffice and that is why SharePoint Hybrid implementations will be commonplace for organizations with mature SharePoint deployments. I will cover this in more detail shortly.
• Business Intelligence today in Office 365 is Excel and Visio Services. Other SharePoint business intelligence services like PerformancePoint, SSRS, PowerPivot and Chart Parts are not available right now. Another limitation today is Excel and Visio services only utilize data that is within the SharePoint Online context; it is not able to reach outside (i.e. to back-end databases). More and more capability will be released through the Office 365 cloud; just for the time being Silverlight and Windows Azure can be used to satisfy these requirements.
• FAST Search is not available in the SharePoint Online cloud right now. It is possible to integrate a local FAST farm with SharePoint Online Dedicated; but not Multitenant. Still take comfort in the fact that SharePoint 2010 search made significant jumps forward from the SharePoint 2007 offering and provides a very strong search experience. With time more and more advanced search capabilities will be released in the cloud.
• PowerShell currently is not fully available with SharePoint Online. There are a lot of PowerShell commands available for Exchange Online and user subscription management; however the full set of SharePoint PowerShell commands is not available today.
• Central Administration Site is not available and this should be expected by anyone who understands what cloud architecture. There are administration screens available to some operation that you would normally perform in Central Admin however it is limited and does not give you the granular control. Why? Well this is the cloud. Customers want to come to the environment so they do not need to be in the business of managing every little configuration of SharePoint.

And the reality is with more and more releases of SharePoint Online this gap is going to continue to close. Still, even if it were to completely close, there will be perfectly valid reasons why some SharePoint may remain on-premise thus, creating a SharePoint Hybrid environment.
What is the SharePoint Hybrid Architecture?
SharePoint Hybrid is as simple as it sounds; it is some SharePoint delivered through the cloud and some SharePoint delivered through on-premise. What will drive you to have SharePoint on-premise? All of that has pretty much been covered to this point. Anything from a specific feature to a business policy may keep some SharePoint on-premise. However using the approach I laid out, you will be able to significantly reduce your footprint of SharePoint on-premise and gain the advantages of the cloud.
The great thing about using SharePoint on-premise is that it is the same software being run in the cloud. This means it is very easy to deliver a consistent user experience between on-premise and the cloud. Branding, navigation, security groups and single sign-on can be configured in such a way that the user can go between these two environments and not know it.
What Type of Cloud is SharePoint Online?
One other thing I want to discuss is what type of cloud is SharePoint Online. I really like this picture it really spells it out for people whom are not fully aware of the multiple delivery models for cloud computing.

Starting on the left side, you see on-premises and this is how most organizations run SharePoint. You must own the entire solution; all the way up the computing stack. Next is Infrastructure as a Service (IaaS) which is the cloud environment that is managed all the way up to the virtualization layer. The company is responsible for everything else; including the management of the operating systems. From a SharePoint perspective that means all the software installation, configuration, management, patching, adding new servers to meet demand, load balancing, etc. needs to be managed by the you. IaaS gets you of the business of hosting virtual SharePoint servers.
Platform as a Service (PaaS) is running the environment all the way up to the application and data tiers. Windows Azure is a PaaS cloud. In this cloud you build custom applications, data models and run them through a highly available environment.
Software as a Service (SaaS) is the entire stack delivered in the cloud and this is the delivery model for SharePoint Online. You do not have to install software, manage patches, availability, etc. This environment gets you out of the business of managing software and into the business of building solutions. SaaS does not provide the granular level of control of the SharePoint environment (which we have discussed).
The reality is that organizations and companies need to save costs and SaaS is the cloud delivery framework they want for the long-run. IaaS can run SharePoint 2010 and can be leveraged as a replacement for on-premise complex SharePoint computing. Still it is well recognized by industry that companies want more SaaS solutions.
Supplement with SharePoint Online with Windows Azure
Windows Azure (PaaS) and SharePoint Online (SaaS) can be used together to deliver end-to-end cloud solutions. Companies that have mature SharePoint deployments commonly have:

• Code that runs in full-trust
• Are managing complex data
• Require the ability to do back-end systems integration.

One thing I have been talking with customers about is offloading that code out of SharePoint and into to the Windows Azure (PaaS) cloud. The Windows Azure cloud allows you to develop and deliver custom code, services, complex data (SQL Azure) and connected back-end integration (AppFabric). These complex operations can be connected through Business Connectivity Services (BCS) in SharePoint Online.
This makes a lot of sense too when you take a step back. I have already said that you should develop SharePoint code to the Sandbox first and when there is good reason to deliver outside the Sandbox. Here is a similar question. At what point do you know you should be developing in SharePoint? I fully recognize there is a gray area here.
I say good software development patterns and practices should drive that decision. This is why Windows Azure is so interesting with Office 365 because we can move complex code that cannot run in the SharePoint Sandbox to the Windows Azure cloud. I recognize this is not a perfect rule because some code needs to run in SharePoint as full trust. However this should be part of your design analysis to reduce to your SharePoint on-premise footprint.
There are lots of different ways Windows Azure can be utilized with SharePoint Online. There may be situations where you need:

• To work with data in SharePoint but you have complex relationships in the data model that are not right for SharePoint lists. Use SQL Azure to manage that data and build services and connect via BCS or Silverlight.
• To integrate with line of business applications on premise use custom services deployed in Windows Azure or AppFrabic. Again BCS or Silverlight can connect Windows Azure which is conduit to line of business applications.
• To use custom web services to perform complex computations and logic. Again offload that to Windows Azure.
• As well, reverse the direction. There may be solutions and applications delivered in Windows Azure can utilize SharePoint Online services. There are SharePoint web services, REST services for data, SharePoint Client API, Javascript APIs and Silverlight APIs that can be used as integration points. A simple scenario could be a web page deployed in Windows Azure that needs to manage documents. Instead of building that up from scratch in Windows Azure just connect through SharePoint Online APIs and deliver SharePoint Online services through that custom web page.

At the end of the day both SharePoint Online and Windows Azure can be used to complement each other in the delivery enterprise business solutions through the cloud.

Conclusions
Why did I go through all of this? The answer is simple; to give SharePoint Architects ideas on how to move forward with SharePoint Online. There really should be no blockers as long as you take a realist look at your Information Architecture and assess how you use SharePoint. There will be lots of stuff which can clearly be moved to the cloud, there will be some stuff where it is not appropriate (hybrid) and then there is that gray area. However I really hope that approach I put forth will help you with your thinking into how to significantly reduce, if not completely remove, your SharePoint on-premise footprint.
Thanks
Special thanks to Chris Geier and Stephen Cawood for providing me feedback and advice as I wrote this.

FAST ESP SharePoint Connector

Series

• Why is FAST Enterprise Search Important Part 1
• What is a FAST Enterprise Search Project Part 2
• FAST ESP Components Introduction
• Scaling FAST ESP Enterprise Search
• FAST ESP SharePoint Connector
• SharePoint 2010 Search (Coming Soon)

Introduction

After introducing the components and providing a preview of design considerations for scaling a FAST ESP implementation, let us take a look at how FAST EST works with SharePoint today. In this post I will introduce you to the architecture of the FAST SharePoint Connector and explain how content is fed, processed, stored and queried. We will cover considerations and strategies for a successful implementation.

In the next set of posting, we will discuss in detail what has been planned for SharePoint 2010.

Note, if you have no FAST ESP experience or training, you must read this blog to understand some of the concepts.

FAST Connector for SharePoint Today

FAST supports both SharePoint 2003 and 2007 in the same manner it would support any other enterprise application that FAST would index. FAST provides an API (Java, .Net, C++) and the FAST Content Connector Toolkit, which facilitates the building for Connector applications. The SharePoint Connector is built on these frameworks to feed content from SharePoint into FAST.

There are three features of the FAST SharePoint Connector you should be aware of:

1. It will index sites, lists, list items, files and associated metadata from SharePoint.
2. It can incrementally retrieve content from SharePoint.
3. It will capture SharePoint item permissions and incorporate it into the access control list.

Architecture

The architecture of the FAST SharePoint Connector is pretty simple and well contained.

1. A custom web service will be installed into the SharePoint farm. This web service will be accessible just like the out of the box web services provided in SharePoint. Side note: if you are interested in writing out your own custom web service for SharePoint, read this blog .
2. The FAST SharePoint Connector must be installed on a machine that can access the SharePoint web services and is able to connect to FAST ESP Content Distributors. It really does not matter where this is installed as long as it can make the required connections. That being said, the Connector could be installed on either the SharePoint Farm or on the FAST Farm.
3. The Windows Authentication Proxy must be installed onto the FAST Farm.

Basic Processing Flow

The installed components work together to retrieve content from SharePoint and make it searchable within FAST. Here is the process:

1. The SharePoint Connector calls the FAST SharePoint Web Service to retrieve content.
2. The FAST Connector connects to the FAST Content Distributor and sends along the SharePoint data.
3. The FAST Windows Authentication Proxy “may” be used to get additional SharePoint data.
4. The document processors process the content into FIXML documents so an index can be built.

Now let’s dive a little bit deeper into some of the details about how this works.

Incremental Loading

The FAST SharePoint Connector will perform incremental loading of content. The first time it will be heavy because all of the SharePoint data will be loaded. However, subsequent content loads will only retrieve changes. The interval for incremental loading is configurable.

Incremental Loading Strategy

If you need to completely reload the data, you must clear the Collection the documents were fed to. Doing this has ramifications that you should be aware of. The most important one that comes to mind is that Collection can have documents from other locations. If so, all of that content will have to be re-indexed too! That can be a big deal. So it is important organize your Collections and potentially anticipate if you will ever have to do this.

You may be wondering how you can control the amount of data that is indexed at any given time. Well there are probably many ways but here are some options that come to mind first.

Create multiple pipeline instances for processing SharePoint data, then configure the pipelines to include or exclude specific URLs within SharePoint. I might create a dedicated pipeline for processing content in areas where I know there will be lots of updates. For instance, collaboration or project sites will have data updated on a regular basis. I would then configure that pipeline to refresh the interval on a regular basis. The advantage of doing this is that a smaller subset of data that is regularly updated will be polled more frequently. Then I would create a dedicated pipeline for a publishing site where data is updated less frequently. The interval between getting data may be longer.

Another thing I may take into consideration is pushing data to different collections. For instance, you can have dedicated collections for intranet, extranet and Internet (remember I am not talking about SharePoint collections, I am talking about FAST collections). Typically in the SharePoint world you logically group data into different content databases, shared service providers and even different hardware. It may be good to maintain that logical separation knowing that it is recommended to feed content into different FAST Collections based on these logical boundaries. Doing this will also give more control over the Search Profiles and what Collections people have access to.

Document Feeding

When SharePoint data is read through the web service by the SharePoint Connector, both metadata and security information will be sent to the Connector. However, depending on the configuration you set, SharePoint files may or may not be part of that pay load. By default, a reference to the file (a URL) will be part of the information sent from the SharePoint Connector to the Content Distributor(s). During Document Processing, the Windows Authentication Proxy will use this reference to retrieve the actual document from SharePoint. You have the ability to change this configuration and send the file as a BLOB.

Document Feeding Strategy

Why are you provided with this option? Mostly for flexibility reasons. If you pass the files by reference:

• The Connector is going to perform more quickly because it has less data to work with.
• Your network I/O will be better utilized because the document will not have to be passed twice as it will only be retrieved one from SharePoint. This is a big deal if you have large files.

If you choose to pass the file immediately:

• Document Processing will be quicker because it does not have to go out a retrieve the file from SharePoint.
• The machines where Document Processing is located do not need to have access to the SharePoint sites because all of the content is available.

Processing SharePoint Data

Working with SharePoint data is not really different than working with other data that may come into FAST. But we need to be aware of some strategies you may want to employ. First, you should know how the data type mapping from SharePoint to FAST will he handled. Second, metadata from columns in SharePoint will be mapped into the fields within the Index Profile. This mapping is based on the unique name of the field. For example, if you have a column called Last Update Date in SharePoint, in the FAST index profile there must be a column called lastupdatedate (notice it is lower case, no spaces and no special characters). If this is the case, the data from the SharePoint column will be automatically mapped into that index field and become searchable. Note the SharePoint column data is not mapped; Document Processing will discard the data.

If you have a good understanding of SharePoint, this will raise a red flag for you. This is because you know that columns of SharePoint data can be added in an ad-hoc fashion. The question on your mind is how can data made searchable if it is not mapped to a field? When you learn FAST there is a concept called Scope fields. Scope fields take metadata and store them in a structured format (similar to XML) in a single field in the index. Scope fields are specifically provided to support storing of index data without having to know the schema of that data in advance. When you store data in a Scope field you have the ability to query it back out using their FQL language (similar to writing an XPath query).

Processing SharePoint Data Strategy

There are some considerations that you must take into account. First, you can add new fields into the Index Profile to match all of the data that you want to bring into SharePoint. This is fine; however, if you need to add a field this is considered to be a “warm update.” After making the change, only new documents will have the data but all previously indexed documents will not. For old documents to have the data, they must be completely re-processed. This will require you to clear the collection and completely re-index (discussed above). A second consideration is that using Scope fields to support querying all column data has a query performance penalty.

Here are some additional recommendations:

• Come up with a hybrid approach where important SharePoint columns are mapped to a field in the Index Profile. Then allow all other columns to be indexed automatically into a Scope field. This is a common practice. This will give you good query performance on most common columns of data and still allow you to access to all other column data.
• Earlier, we mentioned potentially creating separate Collections for publishing sites versus collaboration sites. In that scenario, do not turn on Scope fields for the publishing Collection because the metadata should be very well defined. This way you can get better query performance. All you need to do is either add new fields that map directly to SharePoint columns or add document processing stage(s) that will save the data in existing index fields.

Wrap Up

This post provided you with some insight into how FAST ESP indexes data from SharePoint. Hopefully you will take these factors into consideration before you start to index your content. This is why we say it is so important to understand the life-cycle of the data you are indexing - because it will influence your approach.

Scaling FAST ESP Enterprise Search

Series

• Why is FAST Enterprise Search Important Part 1
• What is a FAST Enterprise Search Project Part 2
• FAST ESP Components Introduction
• Scaling FAST ESP Enterprise Search
• FAST ESP SharePoint Connector
• SharePoint 2010 Search (Coming Soon)

Introduction

In the first part of this series I introduced all of the components and their role for FAST ESP 5.3. Now I want to discuss how these components should be scaled and the associated design decisions.

Just as with a SharePoint installation, typically the first thing a client wants to know is how much hardware they need to buy and what the licensing costs will be. So in the SharePoint world we have a multitude of questions that we ask up front to gather this information, which is usually easy to get since SharePoint is an application server. For FAST you need to focus on understanding the content that will be indexed, which can be pretty challenging at times. This is because you need a full understanding of the characteristics and lifecycle of that data. This is usually not very well documented, especially with legacy systems.

In this post I am going to expand upon the previous one by focusing on how to scale the components of FAST based on business rules.

General Comment

FAST can be implemented on a single machine or may be completely scaled out with each component having own dedicated hardware. FAST is linearly scalable, meaning that doubling the hardware will double the capacity. This is unlike SharePoint 2007, where there can be decreasing performance (for example, with more than four WFEs). FAST best practices also state that better performance can be achieved by using multiple small machines instead of one big machine.

Requirements for Scaling

Some initial questions you need answered right off the bat for scaling are:

• How many documents are going to be processed (content volume)?
• What is the size of the documents that will be fed into FAST?
• What is the estimated total number of documents that are going to be searchable (which is different than number of documents that could be processed)?
• How would the documents be described (format, percentage of content to be indexed, type, metadata, etc.)?
• Will data be continually fed or pulled in periodically?
• What is the life-cycle of the data to be indexed?
• What is the acceptable amount of time between a document being fed into FAST and being made searchable to a user (Index latency)?
• How long should it take for search results to be returned to the user (Search latency)?
• What are the expected peaks of queries that must be handled? Specifically getting to understand how many queries per second that must be handled.
• What features are required for document search (spelling checking, lemmatization, entity extraction, categorization, ranking, navigation, etc.)?
• What sort of service level agreement exists in case there is a failure at any point in the component architecture? Example, must search continue to work at all times (even on stale data)?
• What can the network bandwidth support, specifically for copying large amounts of data?
• What storage capacity can be supported?
• What service level agreement must be maintained for business owners and users?

Here is a picture of the components that make up FAST.

Connectors

Connector applications will have to be scaled appropriately to support the feeding of content into FAST. Multiple servers may need to be used depending on what is fed into FAST and how often. Content may have to be sent in batches or run based on schedule.

Make sure you do not create bottlenecks when you have multiple Connector applications running. It's probably best to create a schedule of when content is fed into FAST.

Collections

I mention Collections here because we introduced them earlier but there is no scaling for them. They are just a logical grouping of searchable documents.

Content Distributors

Having multiple Content Distributor machines will provide fault tolerance but will not improve performance as they cannot be load balanced. When a Connector application connects, you have the ability to provide 1 to n Content Distributors and when one fails, the Connector will simply connect to the next one available.

If you need to support a high volume of content processing (or continuous feeding) you can use multiple Content Distributors and dedicate them to specific Connector applications. Remember Content Distributors are simply a pass through and do not have special hardware requirements outside the FAST basic recommendations.

Feeding Proxy

A Feeding Proxy (not enabled by default) can be placed in front of the Content Distributors. This will replicate submitted content to another installation (like a cold back up). This can provide a very high level of fault tolerance for your entire installation. The Connector application(s) would instead connect to the Feeding Proxy instead of directly to the Content Distributors.

Note: if a Feeding Proxy fails, the non-failing Feeding Proxy will buffer the new content until the failing Feeding Proxy comes back online.

Document Processors

Multiple Document Processors can provide fault tolerance and increased document throughput. All of the Document Processors will pass content to only one Content Distributor at a time. Should a Content Distributor fail, that Document Processor will connect to the next available Content Distributor. The Content Distributor is responsible for sending content across Document Processors, which increases throughput of content.

Document Processors have the biggest impacts on CPU and RAM.

Index Nodes

Multiple Index servers can be used to provide fault tolerance, capacity and performance. Basically you create a “matrix” of index servers. The more rows you create, the more fault tolerance you provide. The more columns, the more content is created and the better performance will be realized. Below is a diagram depicting the Index server matrix (a node is the same thing as a server).

First, let’s talk about fault tolerance. Fault tolerance is created by adding more rows to the index. The first row in a column is the Index Master and all subsequent Index servers in the column are Backups. The Index Dispatcher will push all of the FIXML files to each index server in the column. However, the Master will build an index file. If Master goes down, the next Backup will be made to the Master and the index partitions will have to be rebuilt with the FIXML already on that machine.

Adding new columns to the matrix will increase capacity and can help indexing performance. Performance is improved because we can decrease the index latency (amount of time for a document to become searchable) because there are more indexing services running.

Index Servers will require machines with high RAM (building the Index), Disk (store the index), and Net/IO (send the index to Search Server). You need enough memory to build and enough space to store the index being built. If there is not enough space for new content, it will not be indexed. That is why it is extremely important to know the size of the content that is being fed into FAST. Net/IO is needed to receive processed content and to push the index to Search Servers.

Index Partitions

Let’s go ahead and dive a little deeper into how indexing works as there are ways to improve performance. Within each Index node there are Index Partitions (by default there are three). Indexes are built by Index Partitions. The goal of using Index Partitions is to have low index latency (time it takes for a document to become visible in search) and support real time indexing with high volumes of content.

The first Index Partition should be the smallest with the other partitions being increasingly larger. The size of the Index Partition is controlled by a document count. This document count states how many documents can be stored within each Index Partition. New FIXML documents are always added into the first Index Partition (smallest) for indexing. Once a document count threshold is met for an Index Partition, the documents within that Index Partition will be merged with the next Index Partition. This merging will cause a re-index of that partition.

The goal of all is to not index large amounts of documents at one time. Instead index only what needs to be indexed. Once a document has been indexed it is highly likely that the document will not need to be re-indexed on a regular basis. As mentioned, new documents will be added to the first partition and that document will be immediately indexed. Since the first partition is significantly smaller than the other partitions, indexing is not an expensive operation. Once the count of documents goes over the threshold in the first partition, the documents will be merged into the next partition and re-indexed within that next partition. Obviously that operation will be a little more expensive but it will be less frequent and you are not re-indexing content on a regular basis. More Index Partitions can be added and the documents counts for each Index Partition can be adjusted. This can be strategically aligned with the schedules of when content is fed into the FAST.

You may be wondering how updated documents are handled across Index Partitions. This is handled through a process called Index Blacklisting. Based on what was just discussed it possible that different versions of the same document could spread across Index Partitions. The Index Blacklisting process (which runs on an interval) will suppress old versions of a document from search results. When an Index Partition is re-indexed and both of the versions reside on the same Index Partition only then will the old version be discarded. An interesting thing to know is that the index can become fragmented when there are multiple versions of the FIXML residing on the Index Server. This can lead to query performance issues, disk space issues, and increased time requirements needed to rebuild the entire index server. It is important to determine how often content is updated and develop a strategy to align that to the Index Partitions.

One last side note - a document’s Collection does not dictate where a document will be stored in an Index Partition. Documents from the same Collection will be stored across all Index Nodes. Remember, a Collection can be thought of as a piece of metadata.

Index Dispatchers

Multiple Index Dispatchers can provide fault tolerance and better scaling. It will be more scalable as the documents created by Document Processors will be load balanced between the available Index Dispatchers. It is not common to need to scale here unless you want to add Fault Tolerance.

The Index Dispatcher has the responsibility of sending processed documents to the correct Index columns in a round robin fashion (the actual policy is beyond the scope of this article).

Note: Index Dispatchers are a pass through and do not have special hardware requirements outside the FAST basic recommendations.

Search Nodes

Search Servers typically require machines with good CPU, RAM and disk to store the index. Multiple Search Nodes (servers) can provide fault tolerance, better query performance and support for more volume. Like the Index Nodes, Search Nodes are arranged into a “matrix.” The number of Search versus Index rows do not have to match. However, there must be at least one Search Node for each Index Node column.

More rows add fault tolerance and better query performance. Adding more Search Nodes rows will directly improve query performance. This is because the Top Level Dispatchers will balance queries across the Search nodes in a column. Additionally, having many Search rows provides fault tolerance in case one goes down.

It is possible to have both Search and Index roles installed on the same machine, but this not a good practice for the following reasons: First, indexing is an expensive operation and can directly affect query performance. When the roles are on separate machines, the index will be replicated to each Search node in the respective column. Then the Search node will search on the local index file. Secondly, more fault tolerance is introduced by having the Index and Search nodes on separate machines. If the Index nodes were to all fail, search results can still be returned because the Search nodes will find results using the local index file. The only downside is the results will be stale. However, that is acceptable in many cases.

If you are familiar with MOSS 2007 Search this is not new stuff. In MOSS 2007 you typically set up a single index server and run all the query services on each WFE server to achieve the performance and redundancy that was just discussed.

Query/Result (QR) Server

QR Servers require good CPU and RAM to support the query request from applications. Adding more QR servers will provide you with better query performance and provide redundancy. Performance of the QR server can be directly affected by the features that have been activated. If it takes a long time to prepare a query or to prepare the results, having multiple QR servers can be helpful.

Search Front End Servers (SFE)

SFE are the front end applications that make query requests. The scaling of these applications is beyond the scope of this article.

Administration Server

If you are looking to have a truly fault tolerant environment, it is recommended that you set u pa separate second Admin server. However, this is not usually done. One of the more important Administrative services that can be scaled is the CORBA Name Service. This service ensures that components will be able to resolve each other.

Index and Query Expansion

Another concept I will introduce you to is Index and Query expansion. This can affect performance and storage capacity and should be considered as part of your planning efforts. Expansion is specifically related to lemmatization, which is a linguistic feature typically implemented. Lemmatization provides the ability to search for alternate forms of a word (walk >> walked, walks, walking, etc.). Using lemmatization will improve the search experience, but at a cost of disk space, content processing and query processing. Typically, you will not use lemmatization when content is small, very structured, and requires exact text matching. Lemmatization really helps when there is lots of rich, verbose and unstructured structured content.

There are three ways to approach lemmatization: index expansion, query expansion and by reduction. Index expansion will store the lemma values in the index which will have an impact to the size of the index, but querying will be faster. Query expansion will have the opposite affect: making the index smaller but impacting query performance. Expansion by reduction will store the reduced lemma value in the index and the query terms will be reduced to the same lemma value. This is out of scope of this article, but Expansion by reduction is basically a middle ground.

Applying What We Learned

This is a lot to consume for a newcomer to FAST, let alone Enterprise Search. But I wanted to introduce you to some of the considerations you need to be thinking about when designing a FAST implementation. The goal is a low index latency (time to make a document searchable) and to maximize query per seconds. Some things you should consider right off the bat are:

• Establish dedicated machines for searching and indexing. Rows can always be added to increase capacity or to improve performance.
• Try to process documents and index during periods when you know there will be a low amount of queries.
• Try to use machines with high disk speed, 8 GB RAM, and multiple cores.
• Only use the features that you need (for example, turn off lemmatization if you do not need it).
• Optimize the document processing so that it is efficient.

Depending on the business rules you have to support, you will have to scale the components to different machines. Here is an example:

In this small farm, the first machine is dedicated to receiving content and then processing that content. The second machine is devoted to building indexes that can be searched. The third machine is devoted to accepting queries, performing searches, and returning query results.

Some benefits of this configuration are:

• A level of redundancy is created in that Search results can still be returned if the other components should fail (which is important).
• Hardware is dedicated for searching and indexing.
• It is easy to scale by adding more search and index nodes.

Some drawbacks of this configuration are:

• Latency will be introduced when copying the index from the Index Node to the Search Node.
• There is no redundancy for each specific component.

Here is another strategy for FAST configuration:

In this case both the Index and Search Nodes are placed onto the same machine. Why would you do that? Because there could be data that when updated must be available very quickly. We have reduced that time because the index does not have to be copied between machines. Obviously, issues can arise in this configuration if indexing was to occur on a regular basis and if there are lots of users concurrently running queries.

We know that FAST can run perfectly well on one or two servers and that can even make sense in some environments. Still, more servers will be required to support the proper level of redundancy and performance required by production users. Simply take the information provided here and scale the environment based on your business requirements. Knowing the answers to the questions presented at the beginning of this posting is critical and will ensure that you have a successful and sustainable implementation.

FAST ESP Components Introduction

Series

• Why is FAST Enterprise Search Important Part 1
• What is a FAST Enterprise Search Project Part 2
• FAST ESP Components Introduction
• Scaling FAST ESP Enterprise Search
• FAST ESP SharePoint Connector
• SharePoint 2010 Search (Coming Soon)

Introduction

I am going to provide an introduction in the architecture of FAST ESP 5.3. As many of you know, FAST is part of the enterprise MOSS 2010 CAL but not for MOSS 2007. I will delve into the future architecture of FAST with MOSS later in this series. Right now I wanted to introduce the architecture of FAST as it is today. Many of the concepts will carry over for MOSS 2010 so having a good understanding of FAST will be valuable.

In this blog I am going to start by giving you an introduction to the various different servers, components, and logical architecture that make up a FAST ESP 5.3 implementation. This blog is meant to provide a tip of the iceberg view of FAST.

In this blog there will be a technical focus on understanding how content is actually made searchable with FAST. One of the first things you should understand is the purpose of each component and its role in making content searchable.

MOSS 2007 Search Architecture

I am going to assume the reader is familiar with SharePoint and is learning FAST from a SharePoint perspective. Some of the Enterprise Search concepts with MOSS 2007 are similar to FAST. Let me touch on these so you have a frame of reference before we jump into FAST. In MOSS:

• There is an index server that builds the index.
• There is a physical index file that is pushed to each WFE server.
• There is a query service that runs on the WFE that queries the index file.
• There are SSPs which are to host search services and control what content is indexed.
• The BDC is used to add external data (from a database and web services) to the index.
• There is an access control list which is used to manage security to items.

FAST ESP 5.3

The most current version of FAST is called ESP 5.3. If you are familiar with FAST and the value proposition (further reading) you will know that it is a really powerful engine for crawling content and providing relevant results to users.

Let us look at FAST ESP 5.3 components from a 20,000 foot level (If you are new to Enterprise Search please review this blog first). A simple way of looking at FAST is to think of it as a big ETL project. First you need to get content from various locations across your enterprise. Then you need to process and format the content so it can be indexed. Once content is indexed, you need to quickly return results in a relevant matter. There are numerous components within FAST that perform this processing, indexing and searching:

• Connectors – Applications that can feed content into FAST.
• Content Distributors – Connecters feed content to Content Distributors which send the content for document processing.
• Document Processors – Processing workflows that create searchable documents from submitted content.
• Collections – Logical grouping of searchable documents.
• Index Dispatchers – Routes processed documents to the appropriate Index Node.
• Index Node – Builds searchable indexes from processed documents.
• Search Node – Queries built indexes for matching documents.
• Top Level Dispatcher (TLD) – Manages communications and performance between the QR Server and multiple Search Nodes.
• Query/Result (QR) Servers – Prepares queries to be sent to the Search Nodes and refines search results returned.
• Search Front End Servers (SFE) – Applications where a user makes a search request.
• Administration Server – Administrative features, configuration management, etc. for the enterprise installation.

The diagram below depicts how all of these components interact with each other at a high level. This should give you a general idea of the life-cycle of a document.

Connectors

Connectors are also commonly referred to as content feeders because that is exactly what they do. Whether Connectors do it by pushing or pulling content to FAST, they are built on a common API (available in Java, C# and C++) which submits content to FAST. Think of Connectors as standalone applications that submit content into FAST. FAST provides several Connector applications. Those that you should be immediately familiar with include:

• Enterprise Crawler – an extremely powerful crawler for web content.
• File Traverser – This app can crawl file directories and supports over 270 file formats out of the box. It also provides extensive support for consuming XML content.
• JDBC Connector – This app is used for submitting structured data/content from databases into FAST.
• SharePoint Connector – For the SharePoint people reading this blog, there is a connector for SharePoint 2003 and 2007. We will discuss details later in this series.

This is by no means the only list of Connectors. Many FAST partners have built product FAST connectors for many of the well-known enterprise servers that we work with today. You also have the ability to create your own feeder applications using their API.

Collections

Collections are one of the fundamental concepts you need to know about when implementing FAST. Content is always submitted to Collections, which are logical groupings of searchable documents. In FAST terminology, a “document” is anything that has been indexed. This can be a file, some web content, a database record, or something else that can be processed by FAST. Each document has fields which are populated based on document processing rules. FAST has a very powerful relevancy model and fields are used to provide more relevant results to users.

Why would you create Collections? Some examples are you may want to create different collections for different type of content that you are indexing (Internet versus intranet). You may want to index content differently based on business rules and relevance models.

We will touch more on Collections later when discussing Document Processing.

Note if you have a SharePoint background you may want to say a Collection is similar to a SSP but they are not the same. SSP is a concept unique to SharePoint.

Content Distributors

When Connectors feed content to FAST, they must provide two things: the Collection (which is the logical destination for the content) and the Content Distributor. The purpose of Content Distributors is to provide fault tolerance and increased Document Processing throughput for FAST. They are responsible for routing content directly to the Document Process servers. Specifically, the Content Distributor is responsible for sending content to a Collection which has a Document Processing pipeline mapped to Collection. Content will only pass through the Content Distributor and it will never be modified.

Document Processors

Content processing occurs within Document Processing servers of FAST. Pipelines in the servers support an end-to-end process that transforms submitted content into a FAST searchable document. Within each pipeline there are stages which perform actual tasks. FAST comes with numerous pipelines already preconfigured with stages. It is possible to write your own stages in Python and then add them into existing pipelines. You can also create your own pipelines for custom content processing. The goal is to create a document that is in a proprietary format called FIXML. FIXML is a physical file that is used by the FAST index servers to build an index that can be searched. A common comparison made is a pipeline and it stages to an ETL job such as in SQL SSIS.

When a Collection is defined, a single pipeline will be assigned to perform the Document Processing for that Collection. Pipelines can be reused between Collections but it is important to note that a Collection will only have one defined pipeline. This is because some pipelines are geared for processing unstructured content (web) versus structured content (database and XML). Also, documents are uniquely identified by the internal ID along with the Collection name the document belongs to. It is possible that the same document could be in the index more than once but in different Collections. They are not considered to be the same document (even though the document source is the same) because it “may” have passed through a different document processing pipeline.

Note there is a significant amount of work (beyond the scope of this blog) that is performed on the Document Processor servers, including everything from extracting complex entities to applying linguistics.

Index Dispatchers

Before data is sent to the Index nodes, a component called an Index Dispatcher sends the FIXML to the correct Index node. The job of the Index Dispatcher is to hide the topology of the Index nodes from the Document Processing servers. I will discuss details of the Index Server topology later.

Index Nodes

The Index nodes are responsible for building binary indexes from FIXML files created by the Document Processors. Searches are completed against the built indexes, not the FIXML files. There may be many Index Nodes to support fault tolerance, performance or the amount of content that must be indexed. We will dive into this in the next section.

Search Nodes

Search Nodes are the processes that perform queries against Index Nodes. There will always be at least one Search Node for every Index Node. The Search Node will only search the Index Node that it is assigned to. We will go into the details of topology of Index and Search nodes when we discuss scaling FAST.

Within a Search Node, there is a process called fSearch which is created for each index partition within the Index Node. The fSearch processsearches the index partition for matching documents. Then a single process within the Search Node called FDispatch takes all the results from each index partition and merges them into a single result that is ranked and sorted appropriately based on the rules specified in the Index Profile.

We will not go into all the details of the Index Profile, but since it was mentioned I will give you a highlight of it. The Index Profile (XML file) defines the search index schema for the search cluster. It defines document fields, document processing features, search features and results features. Almost every server in the search cluster configuration uses this file in some way.

Note that a Search Node is very similar in concept to the Query Service in MOSS 2007. The Query Service has the responsibility of searching the index file that is built by the Index service.

Query/Result (QR) Server

QR Server is responsible for preparing queries to be sent to the Search Nodes and refines the results before they are returned to the calling Search Front End Server (SFE). Query transformation includes spell checking, query-side lemmatization, query-side synonym expansion, anti-phrasing, stop work removal. It is applied to ensure that the best possible query is submitted. Some of this processing can be controlled by providing parameters with the query.

QR Server is also responsible for preparing the results for the calling SFE. Results transformation will perform result side duplication removal, build document clusters and build shallow navigators based on the query parameters that were given to the QR Server.

Note much of the configuration for the QR server is managed in the Index Profile (mentioned in the previous section).

Top Level Dispatcher (TLD)

Between the Search Nodes and the QR server is the Top Level Dispatcher (TLD). This must be installed anytime there is more than one Search Node. The TLD is responsible for distributing queries across the Search Nodes to improve query performance. The TLD is also responsible for merging search results from each FDispatch in each Search Node into a consolidated result set.

Search Front End Servers (SFE)

Search Front End Server (SFE) is simply the front end application that calls out to the FAST server. FAST provides an SFE application but it is not recommended for production use. A majority of the time, SFEs are custom built or integrated into existing applications. There is an API that is available in C#, Java and C++.

For MOSS 2007, there is a set of web parts available on CodePlex that allows you to display results of FAST ESP 5.3. This will be integrated into SharePoint 2010.

Administration Server

There are several administrative components to FAST such as COBRA Name Service, License Manager, Resource Service, Log Server, Config Server, Cache Manager, Admin Server, etc. Going into their details is beyond the scope of this article.

Round Up

Hopefully this was a good introduction to the major components and their role for FAST ESP 5.3. In the next post, I am going to discuss how these components are scaled and the design decisions you should consider as part of a FAST deployment.

SharePoint Web Configuration Options

Background

We recently had some questions come up on what are the best practices for pushing out custom configurations into SharePoint, especially across the farm. Many of the custom SharePoint solutions you write will need to push out a custom configuration to a web.config (database connection string) or even have to push out custom configuration file (like a log4net.config).

After some discuss here are some approaches with some considerations for each.

Use a Feature

Whatever you do we recommend using a Feature to do the deployment. Going through an alternate method or even manual can introduce errors. SharePoint Features are meant to provide a way to deploy files and solutions across your entire SharePoint farm.

Provision to 12 Hive

Scenario – How do you deploy a custom configuration file?

Approach - One solution is to provision a file to the 12 hive of SharePoint. In the manifest.xml of the WSP solution package use the <RootFiles> element to push the file to a specific location. For instance <RootFile Location="CONFIG\MyCustomConfig\Foo.xml" />. This will create a folder call MyCustomConfig under CONFIG in the 12 hive and place a custom config file there. You can use this approach to push the file anywhere inside of the 12 hive.

Considerations:

• This file will be accessible to all web applications. This actually will be a good approach for many solutions.
• Your calling solutions within SharePoint (let's say a web part) will have to be able to access these deployed files. You will need to have proper Code Access Security FileIOPermissions to access the files. We recommend keeping your security level a minimum and adding a CodeAccessSecurity tag to you manifest.xml of the web part only. For more information on how to deploy Code Access Security with web parts, please read this - http://www.k2distillery.com/2009/06/deploy-web-part-as-feature-with-cas.html.
• If you like this approach but you need to deploy out custom configurations per web application you can do the following. Create a separate Feature for each web application and set the Feature scope=WebApplication. Then deploy and activate the Feature to the appropriate web application. However calling applications (web part for example) will have to know which directory to go to within the CONFIG directory.

Feature Activation

Scenario 1 – How do you deploy a custom configuration value (like a database connection string)?

Approach – A well accepted approach is to use SPFeatureReceiver in a Feature to push out the changes. One approach would be to use the SPWebConfigModification object to push on the changes on activation (FeatureActivating) and remove the changes on deactivation (FeatureDeactivating). For more information on SPWebConfigModification please read this detailed blog on the topic http://www.crsw.com/mark/Lists/Posts/Post.aspx?ID=32. As well read this http://weblogs.asp.net/wesleybakker/archive/2009/01/21/web.config-modifications-with-a-sharepoint-feature.aspx

Scenario 2 – How do you deploy a custom configuration file?

Approach – Again use SPFeatureReceiver however you will have to write some different code that will push the file to the location of your choice. Here is an example to get started from http://geekswithblogs.net/bjackett/archive/2009/12/01/deploy-files-to-sharepoint-web-application-virtual-directories-at-feature-again.aspx

Considerations

• Requires custom code.
• Small note - if new web servers are added, the configuration values will not be pushed to the new web servers unless the Feature is deactivated and then activated again.

Deploy to 12 Hive Bin

Scenario – How do you deploy a custom configuration file?

Approach - We found a trick described in this blog that will push a file into the bin of the web application - http://oricode.wordpress.com/2008/02/27/deploy-a-xml-file-in-your-sharepoint-solution-to-the-web-application-folder/

Considerations

• This directory is really meant for binaries. We do not consider this a good approach.

Final Thoughts

All of the solutions listed here we believe to be valid solutions however you need to use the right one based on the scenario you need to support.

FAST Introduction and SharePoint Search Evolution

There is a lot of information that is coming out from the SharePoint 2010 conference and one of the biggest ones is the integration of FAST into SharePoint 2010. What is FAST? FAST is an enterprise search engine that Microsoft acquired and they have placed a significant investment into. The most important thing you should know right off the bat is FAST does not equal SharePoint. FAST is an enterprise search platform which can be used as the search engine for SharePoint. Up to this point Microsoft has not provided a way to search for content across the enterprise. What we have done to compensate for this is build custom applications or purchase products like FAST and Google Appliances to do enterprise search.

This is what I have seen with the evolution of search solutions in the context of SharePoint. SharePoint 2001, nothing to really discuss but with SharePoint 2003 we started to get a taste of what we wanted for Search. We found that the search did not really work well in SharePoint 2003 (cross site searching did not work) and many customers who were using SharePoint 2003 said it simply did not work. It did basic text searching of content within SharePoint but it was missing key things like relevancy. This created a small market of third party vendors who creates search solutions for SharePoint. Remember, at this time Google had become the search engine of choice, as every day business users would just say go Google something and get the answer. Problem was we did not have the same search engine that we could use internally with a company, organization or enterprise. As result FAST, Google, Autonomy, etc. created enterprise search solutions that could be used within a company enterprise and that many these features that were required by the business user.

Then SharePoint 2007 came out with Enterprise Search. It was a significant improvement over what we had with SharePoint 2003 but it was still far off from being an enterprise search solution. They improved the user interface, allowed for target content taxonomy searching, they added a relevancy model, best bests, synonyms, administrative features, reporting, an API we can build customizations to, added security using an access control list (ACL), and business data search using the business data catalog (BDC). All the stuff needed when creating an enterprise search platform. We now had the ability to search for data inside and outside of SharePoint, we could rank the search results based on who you were, we could analyze searches to improve the user experience, etc. however it still seemed to fall short. The core problem I go back to is users are expecting that Google experience; and not just doing text searching. SharePoint tried to solve some of that but in the end it fell short.

One thing that had always been the most interesting is the introduction of the business data catalog (BDC) to provide a single result set of data from multiple disparate data sources. This was the most interesting search feature for me when SharePoint 2007 came out. This is where they tried to become an enterprise search engine because you go to one place, you enter something to search on, and you query against many different places but get back a single result set. I personally was able to use it successfully to index custom SQL databases of HR related data for several clients. So when they searched for a person, they were able to get more information about that person other than just information stored in Active Directory. Now the BDC had lots of limitations including only able to call databases, stored procedures and web services, no ability to do data transformation, an API that was very hard to develop with and had limitied scalability.

With the introduction of FAST as part of the Microsoft stack, they really have a true enterprise search engine. FAST has a significant amount of features and functionality, which I have not even touched upon. In my next blog, I intend to write about some of these core features and capabilities that are needed for an enterprise search solution and how they are used to meet your business users needs to find the data.

For more information on the value proposition of FAST, I have written the following two blogs:

• Why is a FAST Enterprise Project Important
• What is a FAST Enterprise Search Project

SharePoint GB 2057 Localization

I was recently asked to dig around into an issue with an international SharePoint site we are setting up. I personally have little experience with globalization other having to read about it to pass a MS certification test.

There are language packs for SharePoint which are used to support configurable text for globalization. Well the issue was how is LCID 2057 for England handled? The English language pack supports 1033, which is for US English. LCID 2057 is a considered a sub language of 1033. So, would it be possible to create a unique resx file for GB that maps to 2057? After digging and stumbling around, the answer is it is not possible.

The only resolution would be in the web application set the regional settings to LCID 2057 (GB), and then modify the resx for US English (1033) in that specific web application.

This is what I was able to find out:

• There is only a language pack for English (1033).
  
• It is possible to have formatted text, like dates, formatted to 2057. It is possible to change the locale to 2057 by accessing the SPWeb.Locale. You can try to change the locale through the SharePoint Regional Settings screen in Site Settings, but you will not see a GB option, only US. Another way to change the locale is to go to Webs table in the site collection database; HOWEVER that is not supported by Microsoft.
  
• In the Webs table you will see another column called language. What I was able to find out is that the value in this column MUST correspond to a language pack that has been installed. Otherwise SharePoint will bomb. So setting Language = 2057 and Locale = 2057 will not work. However Language = 1033 and Locale = 2057 will work. What this will do is make sure that things like dates are formatted correctly. The reason why it fails is because in several places, including the 12 hive, SharePoint is building a relative path to resources installed when the Language Pack was installed. You will 1033 folders throughout the 12 hive. So if the Language is set to 2057, it will start looking for a 2057 folders and things will start breaking. At this point I said, it would not be possible to create a dedicated unique resx file for GB. Bummer.
  

Here are some references:

• http://www.i-arif.com/blog/post/2009/01/17/What-is-1033-.aspx
  
• http://nickgrattan.wordpress.com/2007/09/01/regional-settings-and-central-administration/
  
• http://grounding.co.za/blogs/brett/archive/2008/04/09/sharepoint-locale-id-lcid-table.aspx
  
• http://lumbgaps.blogspot.com/2007/08/sharepoint-2007-set-mysite-locale.html
  
• http://vspug.com/mirjam/2008/04/29/changing-the-language-of-an-existing-sharepoint-site/
  
• http://blogs.msdn.com/brianwilson/archive/2008/07/12/how-to-install-multiple-language-packs-for-office-sharepoint-server-2007.aspx