Untyped Messages and WCF Services in a real scenario

Context:

This project was my first big Biztalk experience. It started more than two years ago (is still in Biztalk 2006 R2) and most of the work was made based on pre-conceived ideas of what was good, what was easy, what was trustable.
The project kept on growing and at a certain point the desire to trash it all and restart was too strong. This is an example of how to slowly modify a monster-like orchestration. The images and names aren’t always the same because they are not from a case-study, they were all taken from a real project. It consisted of several orchestrations and this post was made when each one of them was in a different stage of transformation so that all situations are represented.

---

In this project we had to deal with a bunch of different XML files, different types, different schedules, and, worst of all, different actions on the destination server.



We could make this by using maps on the send port, but the tracking of the messages was critical so we needed to keep track of the message on every step of the journey. So, an orchestration was the choice.

To reunite everything, the receive port that collects them (from FTP, SAP idocs, etc.) is the entry of the orchestration. The Message schema in the orchestration is System.Xml.XmlDocument so all schemas are accepted with no distinction.
A Decide component help us creating the basic structure for it, with a lot of conditions of type

OriginalMessage(BTS.MessageType) == http://Company/Schemas/BusinessPartnerActivityTypes#ActivityTypes

and a two-way destination port prepared for send/receive schemas of type System.Xml.XmlDocument, we could make it by repeating a lot of components.

For each flow we make

1. Transform Xml (Received message) to original schema

FCBusinessPartnerActivityTypes = OriginalMessage;

2. Map original schema to service schema


3. Transform ServiceSchema to Xml (port schema)

XmlMessage = SMUVASActivityType;

4. Send to Service

5. Receive Response

6. And after some flows made the general picture would be something like

How to make it in a lighter way?

The first step is to remove the transformation.

1. Declare a string variable for the map, called mapName

1.1. Use an expression editor to select the map based on the message type

1.2. Use the Fully Qualified Name of the map

mapName="";
if(OriginalMessage(BTS.MessageType) == "http://Company/Schemas/OrderTypes#OrderTypes"){ mapName="Company.Biztalk.SmuasCommercialModule.Messaging.map_FCOrderTypes_to_SMUVASOrderTypes";
}
if(OriginalMessage(BTS.MessageType) == "http://Company/Schemas/OrderStatus#OrderStatus"){
mapName="Company.Biztalk.SmuasCommercialModule.Messaging.map_FCOrderStatus_to_SMUVASOrderStatus";
}
…

2. Then in a message assignment use the transform method.

mapSystemType = System.Type.GetType(mapName);
transform(XmlMessage) = mapSystemType(OriginalMessage);

3. The orchestration will be much simpler.

The next optimization is to remove the multiple actions.

At the if that defines the map, you can also define the action. Something like

if(OriginalMessage(BTS.MessageType) == "http://Company/Schemas/Banks#Banks"){
mapName="Company.Biztalk.SmuasFinancialModule.Messaging.map_FCBanks_to_SMUVASBanks"; action="http://SmuAs/Services/BackendServices/CommonBackendSyncService/ICommonBackendSyncService/SetBankList";
}

And when the request message is created, give it the correspondent action.

mapSystemType = System.Type.GetType(mapName);
transform(RequestMessage) = mapSystemType(OriginalMessage);
RequestMessage(WCF.Action)= action;

The look of the orchestration is much lighter and pleasant to work with.

Using Business Rules Engine (I)

Making business rules without messages

The most common way of using business rules is by evaluating a XML message with XPath and return an XML message with more info. The problem is that most of the examples are orchestration dependents: by demanding a compiled DLL with the schemas to be used, they limit the possibilities.
In a project with many orchestrations an easy solution is to make a solution with the schemas, compile, define the rules based on that DLL, and then apply them in the remaining solutions. If it is a small project where schema and orchestration could be on the same DLL, a first compilation is needed for BRE to access the schema. Only then the orchestration can add the rule.

But, what if the message is not important in any direction? What if you just want to use the same rule in all orchestrations without referring each and every DLL?
Scenario: ten distinct schemas. One flow in and one flow out for each. The same tracking rules apply on each direction, depending only on the schema. Here I found a great way of corrupting the principles of the rules.
I recommend this blog because it was the first were I could find any comprehensible info about BRE.

Basic: Policies have versions, on each version one or more rules. By activating different versions different rules can be in use. A rule is divided in condition and actions. A straight-forward "If-Then".



Usually each orchestration has its own rule and each rule is for an orchestration, a direct match.
The trick is to make a single rule, the if always true, and the orchestration will receive information that can be used or not.

The advantage of business rules engine is the low processing time and the changeable ease (copy version, edit, publish and deploy). There are no big processing advantages in using rules this way, but the changing method is still easy and no deployments required.

Making the if always true

Like always, just use the condition 1==1 or something that obvious.
On the actions list, we’ll use an ArrayList to return a list of words.

Go to facts explorer window

Select .NET classes

Right-click and browse .NET Assemblies

Select the mscorlib library

Press OK



The library will become available.



Find ArrayList and the Add() method, and drag it to actions pane as many time as you’ll need.
Fill in with the key strings and publish the version.



Add to the orchestration variables list one of type ArrayList.
Go to Orchestration View and right click Variables folder for New Variable
At new variable define name (in this example rulesArray) and type (<.NET Class…>, mscorlib, ArrayList)
Create another variable, of type Boolean called generateAck.
Add Call Rules component and use that variable as destination for the output.



And then at a Expression editor, search for the value and do as you please. In this example, only for the flows listed we’ll generate Ack. The fixed value is the message type that the orchestration mandatorily knows.



For instance we are searching for Purchase Orders, abbreviated PO. Since the rulesArray includes that value, ANY orchestration that has Purchase Orders will set generateAcks as true and therefore record Acks, if the proper code is made.

RFC related Web Service doesn't seem to update

Situation:

You are exposing a RFC through Biztalk as a Web Service.
An update was made on the RFC and on development environment, and works well. When passing to test/production environment still refers to the old version.

Solution:

You can even delete the application, it has nothing to do with that DLL. The problem is that for applications related to RFC, Biztalk generates another DLL, refering only to the RFC schema. Supposing that all RFC's are SAP-made and unchangeable, those DLL's are not updated when you reinstall your app.

The only way to avoid that problem is:

Delete that DLL at a folder similar to C:\Program Files\Microsoft BizTalk Adapter v2.0 for mySAP Business Suite\Bin

Install your app MSI

Restart application, Host Instance and local IIS

Thanks to http://timrayburn.net/blog/learning-adapters-sap/

Isolated Host Instance Active Messages

Sometimes it has happened that our BizTalk 2006 R2 development and test servers end up with many active instances running on the Isolated Host Instance.

We can't terminate these instances using the administration console.

I found this solution on the web, using a database stored procedure (bts_CleanupMsgBox) that needs to be created first using a DB Schema file that's installed with BizTalk Server.

Don't forget to run this SP using "exec bts_CleanupMsgBox" instead of the "Execute" command from SQL Server Management Studio!!

PS: Always backup you databases before running bts_CleanupMsgBox!

XSLT in Any situation

XSLT: The Basics

Here are some basics XSLT commands that will be useful for the next step.

Setting a variable

Variables can only be set once. For loops consider recursive functions (a.k.a. templates).
MyVariable=5
<xsl:variable name="MyVariable">5</xsl:variable>

Reading a variable

<xsl:value-of select="@MyVariable" />

Read element

Read an element called Element from a node Node2 inside a node Node1
<xsl:value-of select="/Root/Node1/Node2/Element" />
Remember to close the tag

Setting a variable with the value of an element

<xsl:variable name="ValueOfElement">
<xsl:value-of select="/Root/Node1/Node2/Element" />
</xsl:variable>

Read attribute

Read an attribute called Attribute from a node Node4 inside a node Node3
<xsl:value-of select="/Root/Node3/Node4/@Attribute" />

Setting a variable with the value of an attribute

<xsl:variable name="ValueOfAttribute">
<xsl:value-of select="/Root/Node3/Node4/@Attribute" />
</xsl:variable>

If condition

Remember that XSLT is used to construct a XML document so the return of a value is not mandatory. The easiest way to think of it is like a print command, not like a function.
if(MyVariable>10) print MyVariable (> is a reserved char so we use &gt; instead)
<xsl:if test="@MyVariable &gt; 10">
<xsl:value-of select="@MyVariable" />
</xsl:if>

When the comparition is not against variables or numbers (string comparition for instance) the method is very similar, just delimit the string with ' '.
if(MyVariable="test") print "this is a test"
<xsl:if test="@MyVariable='10'">
this is a test
</xsl:if>

If, Elseif, Else

Does not exist an equivalent approach. We use a kind of switch instead.

if(MyVariable=10) print 1;
else if (MyVariable=10) print 10;
else print 100;


<xsl:choose>
<xsl:when test="@MyVariable &lt; 10">1</xsl:when> <!-- if --!>
<xsl:when test="@MyVariable = 10">10</xsl:when> <!-- else if --!>
<xsl:otherwise>100</xsl:otherwise> <!-- else --!>
</xsl:choose>

<Any> and <AnyAttribute>

A schema can be created without knowing the name of fields to fill. The menu Add Element has the option <Any>, and the same applies for Add Attribute (<AnyAttribute>). Like this, you can add virtually any node or attribute even after compilation. Be sure to define 0 at minimum occurrencies or it will fail while you don't have values to fill.

How to Map to <Any>

Create a Script Functoid of type "Inline XSLT". On the code area write the new tag and the desired value.

<NewField>
42
</NewField>

<NewField>
<xsl:value-of select="@MyVariable" />
</NewField>

<NewField>
<xsl:choose>
<xsl:when test="@MyVariable='X'">1</xsl:when>
<xsl:otherwise>0</xsl:otherwise>
</xsl:choose>
</NewField>

To give a namespace instead of writing directly the element, write the tag element:
<xsl:element name="ns2:ElementName">

and so on...

How to Map to <AnyAttribute>

Create a Script Functoid of type "Inline XSLT". On the code area write the new tag and the desired value between xsl:attribute tags.


<xsl:attribute name="NewAttribute">
<xsl:value-of select="$MyVariable" />
</xsl:attribute>
or any other case.

The Scripting Fuctoid is not linked to source schema, but be careful on linking Scripting Functoid to the destination schema. Point to the desired parent node.

Pipelines in Biztalk

Creating pipelines in ports: knowing when to use PassThru Pipelines, XML Pipelines, and when to use Custom Pipelines.

A PassThru pipeline is straightforward. On a ReceiveLocation it receives a file without further treatment, validation or field promotion. The clear text message can be captured by a Send Port with filter by Receive Port.

A XML Pipeline is more complex. It analyses the received file and determines its structure. Properties are promoted and therefore more kinds of filters become available (specially the useful namespace).

In a XML pipeline there are no limitations. The schemas can by converted, envelopes can be splitted, all options are available under the properties section. To avoid time-consuming configurations in that area – and reduce error-proneness – a Custom Pipeline should be made. It stores the configuration inside the DLL and thus can be reused in another port, or even exported to another environment without dues.

The same logic applies to SendPipelines. Usually a PassThruTransmit is enough because the message is already known in Biztalk. If a format change is needed (to Flat File i.e.) an XMLTransmit with the adequate assembler is the only option. And to place that configuration at development level, a Custom Pipeline.


Hint: For the Flat File Assembler there is no need to select a schema. According to this MSDN page If no schema is specified, run-time schema discovery is done. It was tested and confirmed. It makes development a lot easier:

by reducing the number of pipelines. There's no need for multiple assembler pipelines. Instead of one pipeline for each schema, a single pipeline can be used by the entire application.

by making them faster to upgrade. When refering schemas from external DLL's, the assembly version is required. In the past if that DLL was changed (it happens in our dev environment on a weekly basis) we had to manually correct each map and pipeline. Now the pipelines are not a problem.

Mapping several elements to a single element

Frequently we have one block of XML that we want to split to diferent blocks. For instance, when exporting to SQL stored procedures. It is easy to do. Sometimes the opposite happens and the tables are imported from SQL: we have the information divided by several blocks on one XML and we want it grouped.

Let's imagine a simple scenario: one table included the name of a person, the other the professional information. How can we group it to a single element?

We use XSLT.

Supposing that one of the segments has all the keys, the trick is to loop that segment to read the keys and pull the information from all. This way the order of the elements is not important, the association is made by key.

This template compiles the information:

 <xsl:template name="NameValueTemplate">
  <xsl:param name="param1" />
  <xsl:for-each select="//ElementKey">
   <xsl:if test="Id/text()=$param1">
    <xsl:attribute name="Id">xsl:value-of select="Id/text()" /></xsl:attribute>
    <xsl:element name="Name"><xsl:value-of select="Name/text()" /></xsl:element>
   </xsl:if>
  </xsl:for-each>
  <xsl:for-each select="//ElementInfo">
   <xsl:if test="$Id=$param1">
    <xsl:element name="Mail"><xsl:value-of select="Mail/text()" /></xsl:element>
    <xsl:element name="Job"><xsl:value-of select="Job/text()" /></xsl:element>
    <xsl:element name="Phone"><xsl:value-of select="Phone/text()"/></xsl:element>
   </xsl:if>
  </xsl:for-each>
 </xsl:template>


And this is how it is called:

 <xsl:template match="/s0:Root">
  <ns0:Root>
   <xsl:for-each select="//ElementKey">
    <xsl:element name="Element">
     <xsl:call-template name="NameValueTemplate">
      <xsl:with-param name="param1"><xsl:value-of select="Id/text()"/></xsl:with-param>
     </xsl:call-template>
    </xsl:element>
   </xsl:for-each>
  </ns0:Root>
 </xsl:template>

This is it. It includes attributes and elements to include a broad range of situations. More complex cases can be made from this.

WCF service publishing wizard settings

Did you ever had to republish a web service over and over again? Is it fun to go through the wizard and add the methods one by one?

We were having that problem. The idea was to make a connection between SAP and Sharepoint through web services created by Biztalk. An RFC in SAP receives the request and returns the values to Sharepoint that will use them to create a page. To comunication was quick and stable, but I was losing my mind and time creating the services one by one. Let's see the process step by step with focus on Biztalk:

1. the new RFC is created in SAP (not my problem)
2. Biztalk imports reference with the SAP Adapter (easy)
3. build and deploy dll (too easy)

4. recreate service through Biztalk WCF Publishing Wizard
4.1 select Transport Type
4.2 enable Metadata endpoint
4.3 select Biztalk application
4.4 select "publish schemas"
4.5 rename Web Service
4.6 rename Web Method
4.7 define schema type for request and for response, enlarging both the columns to read the schema type
4.8 repeat 4.6 and 4.7 for every single RFC, old or new...
4.9 define namespace
4.10 define the same service location with overwrite
4.11 allowing anonymous access
4.12 press Create

5 update reference on client
6 do some new methods
7 test

The best way to reduce time on Biztalk is never to press the Finish button. Going back the settings are all there and you can deploy it again.

But there is a better way... I googled for this situation. On this blessed link I found the answers for all my problems. Well, at least for this one.
So, the deployment stores the configuration inside wwwroot. And more! It is understandable and can be edited!

I simply made a batch file to call the wizard with the parameter

cd C:\Program Files\Microsoft BizTalk Server 2006\
BtsWcfServicePublishingWizard -WcfServiceDescription=C:\Inetpub\wwwroot\BizTalkWcfService\App_Data\Temp\WcfServiceDescription.xml

and now my work is just with steps 4.3, 4.4, 4.6, 4.7, 4.9, 4.12. The annoying configuration of all old methods, and more than half the checkbox clicks are made for me.

Question: Shouldn't the wizard allow the loading of a xml file?

Registering adapters while publishing services

The Messaging Engine failed to register the adapter for "WCF-BasicHttp" for the receive location "*/*.svc". Please verify that the receive location exists, and that the isolated adapter runs under an account that has access to the BizTalk databases.

This time the problem is not on BizTalk but on IIS. The Application Pool for this website must be running not under a predefined service, but on an account with permissions over Biztalk.

Solution:

Create a new Application Pool for Biztalk, and on the Identity Tab select Configurable and insert the login. Associate the Web Site to this new app and refresh the svc page.

"Registering multiple adapter types within the same process is not a supported scenario" error

From MSDN:

Setting Up HTTP and SOAP Receive

If it is required to run both HTTP and SOAP receive functions on the same Web server, additional configuration is required to avoid the following error:

"The Messaging Engine failed to register an adapter "SOAP". Details: "Registering multiple adapter types within the same process is not a supported scenario. For e.g. HTTP and SOAP receive adapters cannot co-exist in the same process"

To avoid this error, create two separate application pools (one for HTTP adapter and one for SOAP adapter).
Note
Both application pools may use the same BizTalk Server 2004 Isolated Host User identity.

Note
The actual host configuration used should be dictated by security and isolation requirements of the production environment.