Thursday, January 7, 2010

Scaling FAST ESP Enterprise Search

Series

Introduction

In the first part of this series I introduced all of the components and their role for FAST ESP 5.3. Now I want to discuss how these components should be scaled and the associated design decisions.

Just as with a SharePoint installation, typically the first thing a client wants to know is how much hardware they need to buy and what the licensing costs will be. So in the SharePoint world we have a multitude of questions that we ask up front to gather this information, which is usually easy to get since SharePoint is an application server. For FAST you need to focus on understanding the content that will be indexed, which can be pretty challenging at times. This is because you need a full understanding of the characteristics and lifecycle of that data. This is usually not very well documented, especially with legacy systems.

In this post I am going to expand upon the previous one by focusing on how to scale the components of FAST based on business rules.

General Comment

FAST can be implemented on a single machine or may be completely scaled out with each component having own dedicated hardware. FAST is linearly scalable, meaning that doubling the hardware will double the capacity. This is unlike SharePoint 2007, where there can be decreasing performance (for example, with more than four WFEs). FAST best practices also state that better performance can be achieved by using multiple small machines instead of one big machine.

Requirements for Scaling

Some initial questions you need answered right off the bat for scaling are:

  • How many documents are going to be processed (content volume)?
  • What is the size of the documents that will be fed into FAST?
  • What is the estimated total number of documents that are going to be searchable (which is different than number of documents that could be processed)?
  • How would the documents be described (format, percentage of content to be indexed, type, metadata, etc.)?
  • Will data be continually fed or pulled in periodically?
  • What is the life-cycle of the data to be indexed?
  • What is the acceptable amount of time between a document being fed into FAST and being made searchable to a user (Index latency)?
  • How long should it take for search results to be returned to the user (Search latency)?
  • What are the expected peaks of queries that must be handled? Specifically getting to understand how many queries per second that must be handled.
  • What features are required for document search (spelling checking, lemmatization, entity extraction, categorization, ranking, navigation, etc.)?
  • What sort of service level agreement exists in case there is a failure at any point in the component architecture? Example, must search continue to work at all times (even on stale data)?
  • What can the network bandwidth support, specifically for copying large amounts of data?
  • What storage capacity can be supported?
  • What service level agreement must be maintained for business owners and users?

Here is a picture of the components that make up FAST.

components

Connectors

Connector applications will have to be scaled appropriately to support the feeding of content into FAST. Multiple servers may need to be used depending on what is fed into FAST and how often. Content may have to be sent in batches or run based on schedule.

Make sure you do not create bottlenecks when you have multiple Connector applications running. It's probably best to create a schedule of when content is fed into FAST.

Collections

I mention Collections here because we introduced them earlier but there is no scaling for them. They are just a logical grouping of searchable documents.

Content Distributors

Having multiple Content Distributor machines will provide fault tolerance but will not improve performance as they cannot be load balanced. When a Connector application connects, you have the ability to provide 1 to n Content Distributors and when one fails, the Connector will simply connect to the next one available.

If you need to support a high volume of content processing (or continuous feeding) you can use multiple Content Distributors and dedicate them to specific Connector applications. Remember Content Distributors are simply a pass through and do not have special hardware requirements outside the FAST basic recommendations.

Feeding Proxy

A Feeding Proxy (not enabled by default) can be placed in front of the Content Distributors. This will replicate submitted content to another installation (like a cold back up). This can provide a very high level of fault tolerance for your entire installation. The Connector application(s) would instead connect to the Feeding Proxy instead of directly to the Content Distributors.

Note: if a Feeding Proxy fails, the non-failing Feeding Proxy will buffer the new content until the failing Feeding Proxy comes back online.

Document Processors

Multiple Document Processors can provide fault tolerance and increased document throughput. All of the Document Processors will pass content to only one Content Distributor at a time. Should a Content Distributor fail, that Document Processor will connect to the next available Content Distributor. The Content Distributor is responsible for sending content across Document Processors, which increases throughput of content.

Document Processors have the biggest impacts on CPU and RAM.

Index Nodes

Multiple Index servers can be used to provide fault tolerance, capacity and performance. Basically you create a “matrix” of index servers. The more rows you create, the more fault tolerance you provide. The more columns, the more content is created and the better performance will be realized. Below is a diagram depicting the Index server matrix (a node is the same thing as a server).

IndexNodes

First, let’s talk about fault tolerance. Fault tolerance is created by adding more rows to the index. The first row in a column is the Index Master and all subsequent Index servers in the column are Backups. The Index Dispatcher will push all of the FIXML files to each index server in the column. However, the Master will build an index file. If Master goes down, the next Backup will be made to the Master and the index partitions will have to be rebuilt with the FIXML already on that machine.

Adding new columns to the matrix will increase capacity and can help indexing performance. Performance is improved because we can decrease the index latency (amount of time for a document to become searchable) because there are more indexing services running.

Index Servers will require machines with high RAM (building the Index), Disk (store the index), and Net/IO (send the index to Search Server). You need enough memory to build and enough space to store the index being built. If there is not enough space for new content, it will not be indexed. That is why it is extremely important to know the size of the content that is being fed into FAST. Net/IO is needed to receive processed content and to push the index to Search Servers.

Index Partitions

Let’s go ahead and dive a little deeper into how indexing works as there are ways to improve performance. Within each Index node there are Index Partitions (by default there are three). Indexes are built by Index Partitions. The goal of using Index Partitions is to have low index latency (time it takes for a document to become visible in search) and support real time indexing with high volumes of content.

The first Index Partition should be the smallest with the other partitions being increasingly larger. The size of the Index Partition is controlled by a document count. This document count states how many documents can be stored within each Index Partition. New FIXML documents are always added into the first Index Partition (smallest) for indexing. Once a document count threshold is met for an Index Partition, the documents within that Index Partition will be merged with the next Index Partition. This merging will cause a re-index of that partition.

The goal of all is to not index large amounts of documents at one time. Instead index only what needs to be indexed. Once a document has been indexed it is highly likely that the document will not need to be re-indexed on a regular basis. As mentioned, new documents will be added to the first partition and that document will be immediately indexed. Since the first partition is significantly smaller than the other partitions, indexing is not an expensive operation. Once the count of documents goes over the threshold in the first partition, the documents will be merged into the next partition and re-indexed within that next partition. Obviously that operation will be a little more expensive but it will be less frequent and you are not re-indexing content on a regular basis. More Index Partitions can be added and the documents counts for each Index Partition can be adjusted. This can be strategically aligned with the schedules of when content is fed into the FAST.

You may be wondering how updated documents are handled across Index Partitions. This is handled through a process called Index Blacklisting. Based on what was just discussed it possible that different versions of the same document could spread across Index Partitions. The Index Blacklisting process (which runs on an interval) will suppress old versions of a document from search results. When an Index Partition is re-indexed and both of the versions reside on the same Index Partition only then will the old version be discarded. An interesting thing to know is that the index can become fragmented when there are multiple versions of the FIXML residing on the Index Server. This can lead to query performance issues, disk space issues, and increased time requirements needed to rebuild the entire index server. It is important to determine how often content is updated and develop a strategy to align that to the Index Partitions.

One last side note - a document’s Collection does not dictate where a document will be stored in an Index Partition. Documents from the same Collection will be stored across all Index Nodes. Remember, a Collection can be thought of as a piece of metadata.

Index Dispatchers

Multiple Index Dispatchers can provide fault tolerance and better scaling. It will be more scalable as the documents created by Document Processors will be load balanced between the available Index Dispatchers. It is not common to need to scale here unless you want to add Fault Tolerance.

The Index Dispatcher has the responsibility of sending processed documents to the correct Index columns in a round robin fashion (the actual policy is beyond the scope of this article).

Note: Index Dispatchers are a pass through and do not have special hardware requirements outside the FAST basic recommendations.

Search Nodes

Search Servers typically require machines with good CPU, RAM and disk to store the index. Multiple Search Nodes (servers) can provide fault tolerance, better query performance and support for more volume. Like the Index Nodes, Search Nodes are arranged into a “matrix.” The number of Search versus Index rows do not have to match. However, there must be at least one Search Node for each Index Node column.

SearchNodes

More rows add fault tolerance and better query performance. Adding more Search Nodes rows will directly improve query performance. This is because the Top Level Dispatchers will balance queries across the Search nodes in a column. Additionally, having many Search rows provides fault tolerance in case one goes down.

It is possible to have both Search and Index roles installed on the same machine, but this not a good practice for the following reasons: First, indexing is an expensive operation and can directly affect query performance. When the roles are on separate machines, the index will be replicated to each Search node in the respective column. Then the Search node will search on the local index file. Secondly, more fault tolerance is introduced by having the Index and Search nodes on separate machines. If the Index nodes were to all fail, search results can still be returned because the Search nodes will find results using the local index file. The only downside is the results will be stale. However, that is acceptable in many cases.

If you are familiar with MOSS 2007 Search this is not new stuff. In MOSS 2007 you typically set up a single index server and run all the query services on each WFE server to achieve the performance and redundancy that was just discussed.

Query/Result (QR) Server

QR Servers require good CPU and RAM to support the query request from applications. Adding more QR servers will provide you with better query performance and provide redundancy. Performance of the QR server can be directly affected by the features that have been activated. If it takes a long time to prepare a query or to prepare the results, having multiple QR servers can be helpful.

Search Front End Servers (SFE)

SFE are the front end applications that make query requests. The scaling of these applications is beyond the scope of this article.

Administration Server

If you are looking to have a truly fault tolerant environment, it is recommended that you set u pa separate second Admin server. However, this is not usually done. One of the more important Administrative services that can be scaled is the CORBA Name Service. This service ensures that components will be able to resolve each other.

Index and Query Expansion

Another concept I will introduce you to is Index and Query expansion. This can affect performance and storage capacity and should be considered as part of your planning efforts. Expansion is specifically related to lemmatization, which is a linguistic feature typically implemented. Lemmatization provides the ability to search for alternate forms of a word (walk >> walked, walks, walking, etc.). Using lemmatization will improve the search experience, but at a cost of disk space, content processing and query processing. Typically, you will not use lemmatization when content is small, very structured, and requires exact text matching. Lemmatization really helps when there is lots of rich, verbose and unstructured structured content.

There are three ways to approach lemmatization: index expansion, query expansion and by reduction. Index expansion will store the lemma values in the index which will have an impact to the size of the index, but querying will be faster. Query expansion will have the opposite affect: making the index smaller but impacting query performance. Expansion by reduction will store the reduced lemma value in the index and the query terms will be reduced to the same lemma value. This is out of scope of this article, but Expansion by reduction is basically a middle ground.

Applying What We Learned

This is a lot to consume for a newcomer to FAST, let alone Enterprise Search. But I wanted to introduce you to some of the considerations you need to be thinking about when designing a FAST implementation. The goal is a low index latency (time to make a document searchable) and to maximize query per seconds. Some things you should consider right off the bat are:

  • Establish dedicated machines for searching and indexing. Rows can always be added to increase capacity or to improve performance.
  • Try to process documents and index during periods when you know there will be a low amount of queries.
  • Try to use machines with high disk speed, 8 GB RAM, and multiple cores.
  • Only use the features that you need (for example, turn off lemmatization if you do not need it).
  • Optimize the document processing so that it is efficient.

Depending on the business rules you have to support, you will have to scale the components to different machines. Here is an example:

Servers1

In this small farm, the first machine is dedicated to receiving content and then processing that content. The second machine is devoted to building indexes that can be searched. The third machine is devoted to accepting queries, performing searches, and returning query results.

Some benefits of this configuration are:

  • A level of redundancy is created in that Search results can still be returned if the other components should fail (which is important).
  • Hardware is dedicated for searching and indexing.
  • It is easy to scale by adding more search and index nodes.

Some drawbacks of this configuration are:

  • Latency will be introduced when copying the index from the Index Node to the Search Node.
  • There is no redundancy for each specific component.

Here is another strategy for FAST configuration:

Servers2

In this case both the Index and Search Nodes are placed onto the same machine. Why would you do that? Because there could be data that when updated must be available very quickly. We have reduced that time because the index does not have to be copied between machines. Obviously, issues can arise in this configuration if indexing was to occur on a regular basis and if there are lots of users concurrently running queries.

We know that FAST can run perfectly well on one or two servers and that can even make sense in some environments. Still, more servers will be required to support the proper level of redundancy and performance required by production users. Simply take the information provided here and scale the environment based on your business requirements. Knowing the answers to the questions presented at the beginning of this posting is critical and will ensure that you have a successful and sustainable implementation.

4 comments:

Unknown said...

Great, practical post and series, Jason. I look forward to more.

I posted a more general exploration of the topic of scale and performance here: http://blogs.msdn.com/enterprisesearch/archive/2009/05/22/thinking-big-search-scale-and-performance-on-a-budget.aspx

(btw - the last 2 graphics in your last post didn't come through)

Nate

Jason Apergis said...

Nate,

Thanks for the feedback. I have two more planned. One on SharePoint indexing today with MOSS and FAST ESP. Second is vision with SharePoint 2010 with want informatoin I have been able to get. Unfortunately much of it is high level information.

Fixed the two graphics as well...

Thanks,
Jason

Lux said...

Jason,

Great post! It's tough to find a lot of good information on scaling FAST for the enterprise. I work for a large Fortune 500 company and I am responsible for the Search platform here. It's been a real challenge to scale FAST correctly and takes a lot of work - and a lot of consulting engagements to get it right sometimes. What is your experience with FAST? Looking to meet other folks that I can share ideas / techniques with.

Glad I found your article,

Jeff

Jason Apergis said...

Lux,

Thanks for the feedback. I have also written some recent blogs on the following

FAST for SharePoint 2010 - http://www.astaticstate.com/2011/01/part-1-fast-for-sharepoint-2010.html

SharePoint 2010 Search - http://www.astaticstate.com/2010/12/sharepoint-2010-search-architecture.html


Since coming to Microsoft I have moved more into an architecture role.

I have seen FAST installation that do scale to massive sizes and it does become an art to understand the characteristics of each FAST component and know where to scale based on the nature of your data. I always say that FAST is like a big ETL project. Knowing the life-cycle of the data to be indexed and the user requirements always drive how you scale FAST.

Currently I work for Microsoft Federal but I used to work in the commercial space as a consultant and have some contacts that could help you. You can provide an email address here to contact you at OR you can go to LinkedIN, add me as a contact and I can refer you to some contacts through that web site.

Thanks,
Jason