Sunday, February 17, 2008

BlackPearl MOSS Governance Case Study

1) Introduction


One of the biggest things we run into when coming in and talking with a new MOSS client is how is governance enforced within MOSS? Some of their experiences may be:

  • With SharePoint 2003 and WSS 2.0 we stood up sites everywhere and no one could navigate or search for content.
  • Team sites, workspaces, etc. are not organized making it impossible to find content and have sprawled everywhere.
  • Documents and content have no classifications.
  • Document libraries are full of disorganized content.
  • Documents and content have no business life-cycle management.
  • Etc.

MOSS 2007 has given us tons new features and functionality to assist with governance of content. The following site is a good resource to become familiar with MOSS Governance. However the out of the box functionality can fall short specifically in terms of workflow. These shortcomings have been documented before (here) but these shortcomings become even more compelling if a Use Case is put against it.

2) Use Case

The following is a simple use case for managing an opportunity/sales process.

  1. Account Executive identifies a business opportunity.
  2. Account Executive goes to screen and enters in required data.
  3. A site is created for the company.
  4. A site is created for the opportunity.
  5. Account Executive fills out Opportunity Assessment and SOW Templates.
  6. Account Executive initiates workflow that notifies participants (Business Development Manager, Program Director and Project Manager).
  7. All participants review the document and mark as Approved.
    1. A single Deny requires Account Executive resubmission.
  8. When complete, all participants are notified of the end status.

3) End Solution Vision

The goal of this implantation is to create functionality in MOSS that will manage both the topology and the taxonomy of content. The first thing that comes to mind is using Content Types and Workflows to manage this solution. Content Types would be used to classify, store associated metadata and have workflows to manage the life-cycle of the document. To name a few benefits doing this through Content Type is the content data can be centrally defined, re-useable across the entire site and allow for more targeted searching with Enterprise Search.

We will see that this vision cannot be easily achieved with SharePoint Designer or Visual Studio and the case for using K2 BlackPearl will be overwhelming.

4) MOSS WF Shortcomings

The first problem you will encounter is that you cannot use SharePoint Designer. SharePoint Designer is a great tool for managing the workflow for documents but it is stuck on the list you created it for and cannot be associated to a Content Type which is a major shortcoming. In this use case as new sites are created. Creating workflows on the document library itself will not work there will be a new document library for each site.

Changing the use case to not use sub sites (or Document Workspaces) would not solve the problem as the single document library will become a dumping ground for documents. There are documents that are used to support the creation of Opportunity or SOW documents, for instance there are project plans, budgets, requirements documents, whitepaper references, etc. It is important to keep this content close to each other in an organized fashion and creating a Document Workspace fits the bill. If you did not use a Document Workspace you would not be able to take advantage of other collaboration tools like task lists, calendars, custom lists, discussion boards, etc. because these tools would be shared among all the documents in the single library creating a rat's nest of content.

Second is that SharePoint Designer does not support anything other than modifying the metadata of a document, sending emails, and a few other things. You cannot use SharePoint Designer to provision sites.

Third there will probably be a need to access CRM data which has company names, past opportunities, contact information, etc. and none of this is accessible using SharePoint designer as it can connect to external data sources.

Fourth some may try to retrieve external data or provision sites creating custom WF Activity Library in SharePoint Designer. Creating the custom code to do these operations is nothing out of the ordinary however getting the activities to hook into the SharePoint Designer and passing values around between the activities is a whole different issue you will run into.

At this point we can pretty much say SharePoint Designer 2007 will not be useful and its position as a tool for workflow will fall short over the long run. Next we would look into using Visual Studio to create a SharePoint Workflow Foundation (WF). Simply put, creating workflow in Visual Studio is a complete custom coding effort. All of the things needed for this solution can be done with Visual Studio however there level of effort increases and there is zero infrastructure around it (other than you do not have to build a host server for the workflow definition as it is hosted by SharePoint as a Feature).

WF workflows in SharePoint revolve around managing tasks in a Task List; that is it. Tasks are created in a task list which will lead the user to an asp.net or InfoPath form where a user can take an action. From what we have seen with custom WF projects the effort is increased to do the simplest of things. For instance when writing WF workflows there is no destination user and rules infrastructure. For instance it is not out of the box to define rules like:

  • All destination users must approve but the first to deny will move the workflow to another state.
  • All destination users must approve task in a serial manner one after another.
  • If a certain amount of destination users approve, then the task is completed and moves to the next state.
  • Destination user assignment is determined based on process instance state data and consequently the user profile must be retrieved from an external system.
  • Etc.

All of this logic would have to be built from scratch.

As well, there is no framework around the actions that users can take. It would be coded into the workflow to look for a custom column in the task list to check for a specific value and then move the workflow from one state to another. There is no security around the completion of the task to ensure that the right person is completing it either.

As well building multistage processes with different forms is cumbersome. There are things like initiation, association and task forms. When I have played with them in the past using InfoPath there is no way to get at the XML data easily (except on the initiation form). What you will quickly start doing is not storing any data at all in the workflow, externalizing all of it completely defeating some of the purpose around using a tool like InfoPath. Since you will start externalizing all of your data you will need to create custom libraries to persist data and abstract in such a way that your workflow and your forms (whether asp.net or InfoPath) can both use it.

There is no reporting around the WF workflows either (SharePoint Designer too). You can go to the properties of an item and see all the logging that were created for the item but that is it. There is no way to aggregate the data for the items to create business contextual reports. As well, since the data is associated to the SharePoint item, if that item were to ever be deleted all of the workflow histories are deleted. Knowing this you will be forced to externalize all of your business reporting data and make sure you persist this data during workflow execution.

As well, there is no central administration or infrastructure for the workflows at all. There is no way out of the box way to handle security, no way to search for workflow instances globally to manage them, no infrastructure for managing configuration parameters, and the list goes on and on…

5) BlackPearl SharePoint Workflow Solution

I must first stress that BlackPearl workflow is so much more than SharePoint workflow. It is a workflow server built completely on top of Windows Workflow Foundation that can be used to create workflows for SharePoint, WinForms, ASP.net, around COTS, etc.

While trying to figure out how I wanted to do this solution I had two options. I could either use BlackPearl's SharePoint Integration which will deploy a workflow as a Feature in the same fashion as creating a workflow in Visual Studio or I could have used the SharePoint Events Integration which will listen for events at a location (like a list) to initiation a workflow. Given that I wanted to create a workflow that would be associated to Content Type and I wanted a workflow a user could action either from within SharePoint or Office using the BlackPearl SharePoint Integration was a no brainer.

Knowing the Use Case stated above these are the projects in my solution:

  • SharePoint Project:
    • Feature with Content Types (with standard document templates) and custom Fields.
    • Feature with custom document library with custom Content Types Associated to them.
    • Simple Custom Web Part to display list of Sub Sites (not out of the box with WSS 3 but extremely simple).
    • Custom Site Definition for Company level sites (Sub Site Web part included in definition).
    • Custom Site Definition for Opportunity level sites (had custom document library and Forms library with XML configuration added).
  • K2 BlackPearl SmartObject Project:
    • Two SmartObjects using the Dynamic SQL service to retrieve data from CRM.
    • One SmartObject to retrieve data from Active Directory.
  • K2 BackPearl Process Project:
    • InfoPath Process to initiate site provisioning.
    • SharePoint Integration process for Opportunity Content Types.
    • SharePoint Integration process for SOW Content Types.
  • SQL Reporting Services Project:
    • Reports that used the SmartObject Data Provider to retrieve data from custom SmartObjects and out of the box SmartObjects to display status of all workflows.

That is in a nutshell is the solution, lets dive a little deeper into each.

The SharePoint Project basically encompasses all of the standard development needed for SharePoint. I made it a point to create the custom content types, document lists, and site definitions as I believe in making solution highly re-deployable. I am not a fan of creating templates or doing stuff in SharePoint Designer unless I have to. I look at CAML code the same way I look at DDL code I would write for defining a database. It must be repeatable and have the ability to be built from scratch, on demand otherwise you will encounter maintainability issues down the road.

The SmartObject Project allows me to harvest data from disparate sources and provide the data to all levels of the solution using a uniform interface, API, and deployment methodology. A whole separate discussion is required for the SmartObject framework however this provided me the ability to work with data from different locations without having to write a single line of code. As you will see I was able to use this data in my InfoPath form through their out of the box web services and I was able to use the drag and drop environment within the K2 Process to access this data. If I needed to write any custom code I would just use its API to make a connection to the server, set some data into a SmartObject instance and call a method like save.

The first process looks like the following. This process is started by a user going to an InfoPath form. The InfoPath form uses SmartObjects too display past opportunities for the company and allows for the selection of participants of future Opportunity and SOW workflows (Account Execs, Directors and Project Managers). There are events to create sites based on the custom definitions created earlier. As well, there are other events to save the XML from the InfoPath form into the provisioned site.


The following screenshots show the two workflows for the Opportunity and SOW workflows. They are both very similar in nature. Notice the first thing each one does is load in the XML from the InfoPath form used during the site provisioning process. From the XML we retrieve primary key information like the company and opportunity ID to link the workflow to external data. I am not concerned about the other data as the InfoPath form is hooked into SmartObjects which I read from dynamically in the process configuration. Otherwise the rest is simple.

  • Metadata of the document is updated from the CRM via a SmartObject.
  • Users are configured dynamically into the process to complete tasks.
  • Email notifications are sent.
  • Environment variables are used to configure all parameters such as library names, sites, etc.
  • Only about five lines of custom code are written to set some process level data.
  • The workflows are deployed to the custom Content Types.



Finally the reporting services process uses the SmartObject Data Provider to query data from the CRM, K2 database and Active Directory. I was able to write a single SQL statement that joined all of these data sources together to create reports that shows all of the active process instances and whom they are assigned to. The K2 workspace provides several out of the box reports that do the same and I could have used their report generate tool. However I wanted to create a more customized report with more tactical data that business decision makes would want to view. Then I added simply added the reporting services viewer web part into SharePoint site and the reports were now available to everyone. I could even import this custom report into the K2 Workspace if I wanted to.


6) Conclusions

The creation and refinement of the workflows only took me day once I had laid out the pattern that I wanted. That only really took me about two days so the level of effort to create this solution is around three days (creating the custom document library definition was a whole different headache associated to poor MOSS documentation). Now, we have this nailed down, future document workflows can be laid out in this fashion in significantly less time.

As you can see I was able to create processes that can be used to manage both the topology and taxonomy of content within MOSS with almost no custom code. Yes, no custom code and I have a solution where I am hooked into external data sources, I have custom UIs, I am calling WSS Services, etc. K2 BlackPearl provides tons of events and wizards that you can use to drive more sorts of variations of workflows in MOSS. At your fingers you have event wizards to:

  • Create, Delete or Modify Sites and Workspaces
  • Create, Delete or Modify lists and libraries.
  • Create and delete both SharePoint Users and Groups.
  • Manage Permissions for sites, lists, folders, items.
  • Manage documents (upload, download, check in/out/undo, move/copy/delete, and modify metadata).
  • Create, copy, delete, retrieve and update any list items.
  • SharePoint search for list or library data.
  • Records management sending documents, creating and releasing holds.
  • Publishing wizards to create, copy, more, delete, check in, update/copy page content, create/update/delete reusable content.
  • Out of the box BDC integration with several of the SmartObjects.
  • Ability to create complex InfoPath workflows.
  • Etc.

If that is not enough, you can create your own event wizards using their API. This list does not include the entire infrastructure and management tools that come for free when using BlackPearl, I just cannot cover it all here.

As you can see an overwhelming case can be made use K2 BlackPearl when doing workflow automation within SharePoint. I classify BlackPearl as a framework in which I can use to build all my SharePoint solutions.

No comments: