K2.net Process Sizing and Performance Best Practices

This is the third of a series of best practices I considered when starting a new workflow. Note that manye of these Best Practices are K2.net 2003 specific. They will be re-evaluated with the new BlackPearl release.

1) Process Sizing Considerations
1.1) Use the K2.net Estimator When Designing a Process
K2.net has some spreadsheets that will help you understand how much database storage will be required for process instances. Specifically things that must be considered are:

• What are the number of process and activity level fields, the data types and size?
• If audit trails are used for data fields and how many times can the data be updated?
• How many destination users will be assigned for each activity?

1.2) Understand Process and Activity Level Fields
From a general programming position process level fields should be though of as global fields defined at a top of a class while activity level fields are local fields to a method. Process and Activity level fields are not limited to just using primitive data types. Custom objects can be used if they are serialized into a data field. Important Note: Activity level fields can have multiple instances if many slots have been created. A unique activity instance will always be created for an activity slot. This will result in the data value being created and stored for each individual activity instance. Use activity data fields when you need to pass data between activities and events or when each destination user requires their own unique value. This will have to be balanced with having too many process level fields that could be similar to one another.

1.3) Number of Destination Users
“A high number of destination users for activities with client events can cause severe performance issues on the K2.net Server. This comes into play when a large number of users access the Worklist simultaneously (or calls from the K2.netROM API to user worklists) and a high volume of data has to be returned to the K2.net Server from the database. In conjunction with the "Data on Demand" it can be alleviated. With the arrival of K2.net 2003 Service Pack 3 (SP3), there is now an alternative way to handle this. SP3 introduced the option to create a single activity instance for an activity when a Destination Queue is used as the activity destination. The new feature creates a single Activity Instance for the Destination Queue. K2.net Server will create only one activity instance which is visible to all users within the Destination Queue. When one of the users opens the item it will no longer be visible in the other user's tasks lists but will have been assigned to the user who opened it. The implications however are that only one slot is available. The advantage is that only one activity instance is created in the database significantly reducing database overhead as well.” [1]

1.4) Keep Audit Trail
For data fields checking the keep audit trail checkbox will save every value every time the data field changes. If the data field is big (like xml) this will fill up the K2 database every quickly.

1.5) Data on Demand is Lazy Loading
“Data on Demand is a feature which minimizes the load placed on server resources when a large volume of data from the K2.net database is requested by the K2.net server and worklist. By default, when a process instance is loaded by the K2.net server all the data fields are loaded, regardless of whether they are needed at that time or not. This creates a resource drain as not all the data fields are required at the same time to perform a step within the process. However, since all the data has been loaded, the system must manage the data contained within memory even though only a small portion of the data may be affected at that time. To make use of the "Data on Demand" feature, it must be enabled on a field-by-field basis from within K2.net Studio. When "Data on Demand" is active, only the fields required at that time will be returned by the server when the request for data is sent through to the server. In other words, the data must be "demanded" as an explicit call for it to be loaded into server memory and passed to the client application. “ [1]

2) Externalize Large Data Requests for Open Worklist Items
When using the K2.net ROM be careful to not make unfiltered requests for worklist items to the K2.net server as this can be resource intensive. For the OpenWorklist method create a WorklistCritera object with a filter.

“For large volume scenarios where process/activity data fields need to be utilized to filter or sort a worklist, it is recommended that the specific data fields required in the search be stored in a database table external to K2.net, in addition to the native K2.net data field storage. The external table should contain a field for the K2.net "Process Instance ID" if querying process datafields or a "Process Instance ID" and an "Event Instance ID" if tracking activity level data. A query can then be constructed that joins the external table data with the internal K2.net _Worklist table, or more appropriately a read- only view on the _Worklist table (see below for a sample view). Please note the read-only view of the worklist table is the recommended interface to querying the K2.net Worklist. This query should then become the basis for the custom worklist application, ensuring that the fields necessary to process a worklist item in an application (generally the platform, serial number and URL for the event which is stored in the "Data" field), are accessible.” [1]

3) Server Events in Multi-Destination Activities
Be careful to not place a server event into an activity that has multiple slots for destination users unless the code in the event must be executed for each destination user. This can become a bottleneck by flooding external applications with hundreds of simultaneous calls or it could even introduce errors like insert duplicate records into a database.

4) Use Destination Queues for AD Groups
When Active Directory (AD) Groups are used as destination users for activities, destination queues should be used. K2.net will poll active directory to check for changes in the group membership. This will allow K2.net to remove a destination user whom has been removed from the group creating more security around who can access the K2.net process instance. As well if a user is added a new activity instance for this destination user will be created. The rate in which K2.net will poll active directory is configurable. [1]

5) Understand Exception Handling
Exceptions will be raised in the following order from within K2.net: [1]

• Code Level Try-Catch Block
• Event Exceptions Code Block
• Activity Exceptions Code Block
• Process Exceptions Code Block

Exceptions that are handled in a try/catch block will not be logged to the K2.net database. It is recommended that the error may need to be handled locally but a custom error can be thrown which will continue up the stack and be caught at the Process Level. At that location it can be determined if the error should be result in stopping the process instance or continue letting it move along. This is a good location to do global logging of exceptions.

Reference - K2.net 2003 Process Design best practices - http://kb.K2.networkflow.com/articles/kb000161.aspx

K2.net Architecture Best Practices

This is the second of a series of best practices I considered when starting a new workflow. Note that manye of these Best Practices are K2.net 2003 specific. They will be re-evaluated with the new BlackPearl release.

1) Understand When to Use K2.net
K2.net provides the layer of human-to-human and human-to-system process definitions. K2.net typically compliments the middle layer business logic in a traditional three-tiered architecture. BizTalk workflow is used for system-to-system business workflow definition and specializes as a data broker. The native pattern of BizTalk is to serve as an Enterprise Bus for data between line of business applications. Integration Services (SSIS) for SQL Server 2005 can be used as the “poor man’s” BizTalk but should be limited in its usage as an ETL facility. K2.net can be the hub for human related workflows to start BizTalk or SSIS data workflows.

2) Decouple External Data Request with XML
Using XML to pass data between K2.net and external actors will ensure for a more configurable and manageable process over its lifetime.

3) Understand Integration with Line of Business Systems
Versioning should be accounted for whenever interacting with anything external to K2.net. When integrating with external applications, the introduction of a version number should be considered especially when process instances may have a long life-cycle.

4) Design for Long Running Processes
When designing long running and large processes it is highly recommended that the process is broken up into several smaller processes. This allows for flexibility and manageability of the process definitions if things change while a process instance is in flight. Use the IPC server event and create a master child or a daisy chain of process that as a whole make up on large process. Note this is not recommending that processes are used as a sub function.

5) Design with Maintainability in Mind
A reality is that processes change with time due to evolving business rules. Locate all places where there is a possibility of change and consider what types of changes will be made and how they can be resolved. Understand the service level agreement of the process; for instance can a process instance be stopped and started over if there is a change request? If not, understand where a configuration can be introduced. A configuration is not just the usage of the string table or configuration file. A web service, database, sub-process, etc. could be introduced to allow for more flexibility in handling changes.

6) Handling Volatile Business Rules
Line Rules are no more than standard if statements in structured programming; if this do that. If Line Rules are going to be volatile and it is required they are up to date at all times it is recommended that those rules be externalized. Use web services, BizTalk Business Rule Engine, BlackPearl Business Rule Engine (TBD), Database, etc.

K2.net General Best Practices

This is the first of a series of best practices I considered when starting a new workflow. Note that manye of these Best Practices are K2.net 2003 specific. They will be re-evaluated with the new BlackPearl release.

1) Start with Use Cases & Process Flow Diagram
Prior to doing your workflow, start with the creation of use cases as they can be used to design a workflow. Use tools like Visio or K2 Studio itself to create process flows to show to the business users. With BlackPearl, the Visio diagrams can be imported into K2.

2) Plan on Reporting
With K2.net 2003 the Workspace website provides several out of the box reports on the health of all processes. They can be used to support development and administrative activities. They sometimes do not do well with business users as low level implementation details are exposed which can become confusing. Consider storing key business data externally in reporting database using your reporting engine of choice. The K2 database is optimized for managing a state machine not for doing business reporting. BlackPearl provides a wizard to design reports I would still recommend doing externally reporting.

3) Maximize Use of Code Accelerators (Wizards)
Use the K2 Event Wizards to their fullest extent. If you do not know the code, but need to write some custom code, use an Event wizard to generate the code and then re-use it. It is not a short coming of K2.ne that a developer still needs to go into the generated code; it is advantage that much of this code is already built and is completely extendable. For instance, there are events for email, SQL, Data Manipulation, InfoPath, SharePoint, Web Services, BizTalk, Exchange, etc. Good .net developers will want to take the code generated and generalize it in a code library. That is fine approach but should be balanced with maximizing K2.net to create things quickly.

4) Code Modules
Code modules provide a quick way to centralize re-usable code that can be used in all processes defined in a solution. They are best used when created utility methods and functions. Singleton classes tend to work well with code modules. If full object orientated libraries are needed it is better to create an external class library in Visual Studio.

5) Using External Libraries Versioning
External code library should be considered if there is a need for complex classes. If the classes can be used in other context outside of K2.net, it is recommended the class definitions be absolutely externalized. Note that referenced K2.net libraries will be exported with the K2.net process definition to the K2.net server (the dll is serialized into the database).

There are considerations that must be thought of before placing the external DLL that K2.net will use in the GAC. Even though the DLL is versioned in the GAC there is no version mapping to the K2.net process definition that has been exported to the K2.net server. If the code has volatile business rules being computed with long running processes, creating an external service to access that code library is the suggested best practice.

Typically it is just best to create an external library in Visual Studio and import that directly in the K2.net process but there are still some considerations. Note that configuration files will have to be manually pushed to the production server and there can only be one config file for the K2 Server requiring all of the external DLLs so share the same config.

Finally errors in external libraries can be particularly hard to debug without logging. They are even harder or impossible to repair using the K2.net Service Manager without going through a lot of effort.

6) Use String Table for Configuration Values
Use the K2.net string table to place all configurations for your processes. Complex configurations (repeating data) cannot be managed inside of string table. A configuration file can be created in the K2.net bin directory it is recommend that an external database be used to retrieve configuration or process metadata if it is particularly complex.

7) Provide Users with Multiple Ways to Access Assigned Tasks
Provide users with multiple ways to complete the tasks that they have been assigned. Relying completely on email as a way to distribute links to an InfoPath or ASP.net form is not good as the email can be lost or deleted.

8) Using K2.net ROM in Server Event
The K2.net ROM should never be used in a server event when it is accessing the current process instance. Attempting to use the K2.net ROM to operate on the current process instance from within the execution of the current process instance can result in inconsistent behavior and/or crashing of the K2.net Server service. This because is the K2.net server locks the processes that is currently executing and cannot connect back to itself with the external K2.net ROM as described. [1] It is alright to use the K2.net ROM to create new process instances (similar to using an IPC sever event) or to finish the event of a different process. The golden rule is to not use the K2.net ROM on the current process instance itself, only on other process instances.

9) Design with Testing in Mind
Having a K2.net development and QA server should be highly considered. It is possible to create unit tests using Visual Studio 2005 test projects the can unit test an entire workflow and all of its permutations. This is done by using the K2.net ROM to create process instances and mimic user events that push the workflow through. [2]

10) Use Source Control and Define File Hierarchy
It is highly recommended the version control be used to manage K2.net solutions. With BlackPearl integrated into Visual Studio working with source control becomes much easier!

11) Activity Logic Considerations
11.1) Line Rule Must be True to Continue
A common mistake when designing K2.net process is to have activities that will stop a K2.net process instance. If the activity has no lines extending from it or none that will result with a true value the K2.net process instance will finish and cannot be restarted again. It is suggested to pseudo code your line rules on paper to avoid this issue.

11.2) Line Rule Custom Code
Custom code in the Line Rules should only have code that is used determined if something is true or false; do not embed something like a write operation. All line rules are executed NOT matter what and other line rules will be affected. In general, whenever you put custom code into an event handler for line rules, events, preceding, succeeding, etc. make sure the code placed in there is for that operation only.

11.3) Graceful Cancel out of a Workflow
Most workflows will require paths to gracefully cancel or stop a workflow. Make sure this is incorporated into the design of the process definition from the beginning.

11.4) Design for Delegation
Make sure that there is functionality to delegate or escalate so process instances can be re-assigned by the users themselves (even though this will be done through the K2 Service Manager). When delegating, events that are places before the client event will be re-executed. Ensure that there are no problems with iterating over the same code more than once; otherwise move the event.

12) Logging
Ensure Logging is turned on in the K2.net server and logging has been implemented within code.

13) Unique Name in Folio
Processes allow for a Folio Name which is a readable name for a K2.net process instance; use them.

14) Do not Let K2 Spam
Escalations with email can quickly become a spam machine. Ensure that the setting for the duration for escalations is configurable through the string table or a database. As well, create a configuration to turn off escalations in the string table and use it in the event handler for the escalation.

Automation testing or simulation with K2.net 2003

I initially published this at http://k2underground.com/blogs/articles/archive/2007/08/21/automation-testing-or-simulation-with-k2-net-2003.aspx, but I figured this would be a good starting point for my K2.net Blog.

1. Purpose

This article will show how to create automated unit tests which can be used to exercise K2.net workflow process in both ASP.net and InfoPath. Creating automated testing or simulation with K2.net 2003 may seem difficult to do but is easy when using Visual Studio 2005 Test Projects. Creating unit tests with Visual Studio is no different than creating a custom web page that uses the K2ROM to finish a worklist item that has been assigned to a user. In this case instead of embedding code into an .aspx code behind we are going to put the code into a test method.

2. Create a K2.net Process
The following K2.net process is a standard approve/deny process. This particular screenshot shows an approval process using ASP.net. This process would be initiated by a custom ASP.net page or a K2.net 2003 SmartForm PlanPage. There would then be a second web page which the Manager would use to approve the process instance. Finally an email would be sent out based on the Manager’s decision.

An assumption is made that the reader knows how to create a basic workflow. The specific details of setting destination users, configuring line rules, configuring email, etc. will not be covered.

2.1 Adding a Test ID Data Field
The Visual Studio Test Project will require an identifier to correlate the K2.net process instance with the test instance. K2.net generates an identifier called a Serial Number that uniquely identifies every process, activity and event instance. To achieve this we will add a Data Field to the Process by going to the Properties of the Process, clicking on Data Fields and adding a string type called AutomatedTestID. The value will be generated by the Unit Test and set through the K2ROM which will be discussed later.

2.2 Add Approval Data Field
For the purposes of this article we will add an Approve Data Field to the Manager Approval activity. This will be used by the manager to approve or deny the workflow.

3. Create Visual Studio Test Project
Start Visual Studio 2005 and select File, then new Project. In the New Project window select the programming language of choice (in this case C#) and the select the Test option. Within this select Test Project and place the project in a location.

3.1 Configuring the Test Project
The following will be created for you by default.

First rename the default class UnitTest1 to SimpleWorkflowTest by right clicking the filename in the Solution Explorer and selecting rename. After renaming the file you will be prompted to rename all references; select yes.

Next add a reference to the K2ROM by right clicking the References node in the Solution Explorer and selecting Add Reference. In the Add Reference window select the Browse tab and go to \Program Files\K2.net 2003\Bin and select the K2ROM.dll.

The resulting project should look like the following.

3.2 Create Test Methods

To simulate this workflow we need to create two test methods. Note the methods must be decorated with the [TestMethod] attribute and the class with [TestClass]. If these are not present neither the class nor its methods will be used when the unit test is executed. First rename TestMethod1() to SimpleApprovalTest(). Next add two method stubs one called StartSimpleWorkflow() and the other called ManagerApproval(). Notice that both of these methods have not been decorated with [TestMethod] attribute. They will be executed by Visual Studio because SimpleApprovalTest() is the entry point and has the [TestMethod] attribute.

3.3 SimpleApprovalTest Method
Once we have our stub set up, a unique identifier needs to be generated for the unit test instance which will be used to correlate to the K2.net process instance. Modify SimpleApprovalTest() to generate a GUID and pass that value down into both StartSimpleWorkflow() and ManagerApproval().

[TestMethod]
public void SimpleApprovalTest() {
//Create Test Instance GUID
Guid testInstance = new Guid();

//Start the process
StartSimpleWorkflow(testInstance);

//Give K2 server time to create process instance
System.Threading.Thread.Sleep(2000);

//Approve the process
ManagerApproval(testInstance);
}

Both of these methods will need to have their signatures modified to accept the GUID.

private void StartSimpleWorkflow(Guid testInstance) {

}

private void ManagerApproval(Guid testInstance) {

}

Note the System.Threading.Thread.Sleep(2000) was added to give K2.net server a little time between activities. Depending on your testing server performance, this value may need to be modified or this line could be completely removed.

3.4 StartSimpleWorkflow Test Method
Add the following statement using SourceCode.K2ROM; in the class.

Add the following code to the StartSimpleWorkflow(Guid testInstance) method which will create a new process instance using the K2ROM. This method opens a connection to the K2.net server, sets the GUID to the process instance and folio name, and finally starts the process.

private void StartSimpleWorkflow(Guid testInstance) {
string connectionString;
string k2Server;
string processName;
Connection conn = new Connection();

try {
//Recommend moving thses values to app.config file
connectionString = "CONNECTION STRING";
k2Server = "SERVER NAME";
processName = "SimpleApproval\\SimpleApproval";

//Open K2 Connection
conn.Open(k2Server, connectionString);

//Create Process Instance
ProcessInstance process = conn.CreateProcessInstance(processName);

//Set the test instance id
process.DataFields["AutomatedTestID"].Value = testInstance.ToString();

//Set K2 Folio Name
process.Folio = "Test Simple Approval " + testInstance.ToString();

//Start the process
conn.StartProcessInstance(process);
}
catch (Exception ex) {
// failed validation
Assert.Fail("The process failed: " + ex.Message);
}
finally {
conn.Close();
}
}

3.5 ManagerApproval Test Method
In this method a connection is made to the K2.net server and the Worklist for the Manager is opened. This is done by using the WorklistCriteria object to query for the WorklistItem that has test instance GUID set in its process data field. This is done by using a WorklistCriteria with a filter. When the WorklistItem is returned the activity instance data field is set to “approve” and the WorklistItem is subsequently finished.

private void ManagerApproval(Guid testInstance) {
string connectionString;
string k2Server;
Connection conn = new Connection();
Worklist workList;
WorklistCriteria workListCriteria = new WorklistCriteria();

try {
//Recommend moving thses values to app.config file
connectionString = "CONNECTION STRING";
k2Server = "SERVER NAME";

//Open K2 Connection
conn.Open(k2Server, connectionString);

//Retrieve the test manager's worklist items
workListCriteria.AddFilterField(WCLogical.And, WCField.ProcessData, "AutomatedTestID", WCCompare.Equal, testInstance.ToString());
workList = conn.OpenWorklist(workListCriteria);

//Find the worklist item with the test identifier in it
foreach (WorklistItem workListItem in workList) {

//Set the process for approval
workListItem.ActivityInstanceDestination.DataFields["Approve"].Value = "Approve";

//Finish the worklist item
workListItem.Finish();
}
}
catch (Exception ex) {
// failed validation
Assert.Fail("The process failed: " + ex.Message);
}
finally {
conn.Close();
}
}

3.6 More Test Methods
More test methods could be created in a similar fashion. For instance a similar test method could be created for the denial path. As well if the workflow had multiple steps every step in the process could have a method created and chained to one another to fully test the entire workflow and every possible path it could take.

4. Deployment
It is highly recommended that this not be done on a production K2.net server. A K2.net testing server should be used to run the unit tests on. The reason is that these test process instances will be intermingled with production instances. Having these instances in there will subsequently throw off or dilute reports provided in the K2.net Workspace.

Deployment of the SimpleWorkflow would be the same as any other K2.net process using the Export functionality in K2.net Studio.

5. InfoPath Testing
It is possible to test InfoPath process in K2.net visual studio as well. In the Visual Studio Test Project, add a reference to the InfoPathService web service.

Change the StartSimpleWorkflow(Guid testInstance) method and remove all of the code associated to the K2ROM. Instead call the SubmitInfoPathData method of the InfoPathService web service. Generate the XML for the InfoPath form and pass that into SubmitInfoPathData webmethod. To get a sample of this XML, open an InfoPath that has been K2.net enabled and save it locally. Then open the .Xml file in a text editor and copy the Xml into the Visual Studio. That Xml string can be passed into SubmitInfoPathData.

Next in ManagerApproval(Guid testInstance) remove the following line of code.

workListItem.ActivityInstanceDestination.DataFields["Approve"].Value = "Approve";

Instead use this line of code to retrieve the XML for the InfoPath Form.

string infoPathXml = workListItem.ActivityInstanceDestination.XmlFields["K2InfoPathSchema"].Value;

Once the Xml has been retrieved, modify it (setting it to Approved) and call SubmitInfoPathData webmethod again passing in the Xml. Do not set the Xml back into workListItem.ActivityInstanceDestination.XmlFields["K2InfoPathSchema"] because this would not accurately test the process as InfoPath will always call the web service.

6. Conclusions
Very sophisticated testing processes can be done in this manner testing multiple steps and every permutations of the workflow. Load testing can be done as well by creating multiple unit test instances to identify bottlenecks in the workflow process. Bottlenecks could be sometimes be hardware specific, a connection to an external system may be too slow, tables in a database require indexes, etc. Using this methodology it is possible to completely test your K2.net processes without writing any ASP.net or InfoPath front-ends. This will create a highly decoupled solution and is considered a best practice.