Pipelines in Biztalk

Creating pipelines in ports: knowing when to use PassThru Pipelines, XML Pipelines, and when to use Custom Pipelines.

A PassThru pipeline is straightforward. On a ReceiveLocation it receives a file without further treatment, validation or field promotion. The clear text message can be captured by a Send Port with filter by Receive Port.

A XML Pipeline is more complex. It analyses the received file and determines its structure. Properties are promoted and therefore more kinds of filters become available (specially the useful namespace).

In a XML pipeline there are no limitations. The schemas can by converted, envelopes can be splitted, all options are available under the properties section. To avoid time-consuming configurations in that area – and reduce error-proneness – a Custom Pipeline should be made. It stores the configuration inside the DLL and thus can be reused in another port, or even exported to another environment without dues.

The same logic applies to SendPipelines. Usually a PassThruTransmit is enough because the message is already known in Biztalk. If a format change is needed (to Flat File i.e.) an XMLTransmit with the adequate assembler is the only option. And to place that configuration at development level, a Custom Pipeline.


Hint: For the Flat File Assembler there is no need to select a schema. According to this MSDN page If no schema is specified, run-time schema discovery is done. It was tested and confirmed. It makes development a lot easier:

by reducing the number of pipelines. There's no need for multiple assembler pipelines. Instead of one pipeline for each schema, a single pipeline can be used by the entire application.

by making them faster to upgrade. When refering schemas from external DLL's, the assembly version is required. In the past if that DLL was changed (it happens in our dev environment on a weekly basis) we had to manually correct each map and pipeline. Now the pipelines are not a problem.

Mapping several elements to a single element

Frequently we have one block of XML that we want to split to diferent blocks. For instance, when exporting to SQL stored procedures. It is easy to do. Sometimes the opposite happens and the tables are imported from SQL: we have the information divided by several blocks on one XML and we want it grouped.

Let's imagine a simple scenario: one table included the name of a person, the other the professional information. How can we group it to a single element?

We use XSLT.

Supposing that one of the segments has all the keys, the trick is to loop that segment to read the keys and pull the information from all. This way the order of the elements is not important, the association is made by key.

This template compiles the information:

 <xsl:template name="NameValueTemplate">
  <xsl:param name="param1" />
  <xsl:for-each select="//ElementKey">
   <xsl:if test="Id/text()=$param1">
    <xsl:attribute name="Id">xsl:value-of select="Id/text()" /></xsl:attribute>
    <xsl:element name="Name"><xsl:value-of select="Name/text()" /></xsl:element>
   </xsl:if>
  </xsl:for-each>
  <xsl:for-each select="//ElementInfo">
   <xsl:if test="$Id=$param1">
    <xsl:element name="Mail"><xsl:value-of select="Mail/text()" /></xsl:element>
    <xsl:element name="Job"><xsl:value-of select="Job/text()" /></xsl:element>
    <xsl:element name="Phone"><xsl:value-of select="Phone/text()"/></xsl:element>
   </xsl:if>
  </xsl:for-each>
 </xsl:template>


And this is how it is called:

 <xsl:template match="/s0:Root">
  <ns0:Root>
   <xsl:for-each select="//ElementKey">
    <xsl:element name="Element">
     <xsl:call-template name="NameValueTemplate">
      <xsl:with-param name="param1"><xsl:value-of select="Id/text()"/></xsl:with-param>
     </xsl:call-template>
    </xsl:element>
   </xsl:for-each>
  </ns0:Root>
 </xsl:template>

This is it. It includes attributes and elements to include a broad range of situations. More complex cases can be made from this.

WCF service publishing wizard settings

Did you ever had to republish a web service over and over again? Is it fun to go through the wizard and add the methods one by one?

We were having that problem. The idea was to make a connection between SAP and Sharepoint through web services created by Biztalk. An RFC in SAP receives the request and returns the values to Sharepoint that will use them to create a page. To comunication was quick and stable, but I was losing my mind and time creating the services one by one. Let's see the process step by step with focus on Biztalk:

1. the new RFC is created in SAP (not my problem)
2. Biztalk imports reference with the SAP Adapter (easy)
3. build and deploy dll (too easy)

4. recreate service through Biztalk WCF Publishing Wizard
4.1 select Transport Type
4.2 enable Metadata endpoint
4.3 select Biztalk application
4.4 select "publish schemas"
4.5 rename Web Service
4.6 rename Web Method
4.7 define schema type for request and for response, enlarging both the columns to read the schema type
4.8 repeat 4.6 and 4.7 for every single RFC, old or new...
4.9 define namespace
4.10 define the same service location with overwrite
4.11 allowing anonymous access
4.12 press Create

5 update reference on client
6 do some new methods
7 test

The best way to reduce time on Biztalk is never to press the Finish button. Going back the settings are all there and you can deploy it again.

But there is a better way... I googled for this situation. On this blessed link I found the answers for all my problems. Well, at least for this one.
So, the deployment stores the configuration inside wwwroot. And more! It is understandable and can be edited!

I simply made a batch file to call the wizard with the parameter

cd C:\Program Files\Microsoft BizTalk Server 2006\
BtsWcfServicePublishingWizard -WcfServiceDescription=C:\Inetpub\wwwroot\BizTalkWcfService\App_Data\Temp\WcfServiceDescription.xml

and now my work is just with steps 4.3, 4.4, 4.6, 4.7, 4.9, 4.12. The annoying configuration of all old methods, and more than half the checkbox clicks are made for me.

Question: Shouldn't the wizard allow the loading of a xml file?

Registering adapters while publishing services

The Messaging Engine failed to register the adapter for "WCF-BasicHttp" for the receive location "*/*.svc". Please verify that the receive location exists, and that the isolated adapter runs under an account that has access to the BizTalk databases.

This time the problem is not on BizTalk but on IIS. The Application Pool for this website must be running not under a predefined service, but on an account with permissions over Biztalk.

Solution:

Create a new Application Pool for Biztalk, and on the Identity Tab select Configurable and insert the login. Associate the Web Site to this new app and refresh the svc page.

"Registering multiple adapter types within the same process is not a supported scenario" error

From MSDN:

Setting Up HTTP and SOAP Receive

If it is required to run both HTTP and SOAP receive functions on the same Web server, additional configuration is required to avoid the following error:

"The Messaging Engine failed to register an adapter "SOAP". Details: "Registering multiple adapter types within the same process is not a supported scenario. For e.g. HTTP and SOAP receive adapters cannot co-exist in the same process"

To avoid this error, create two separate application pools (one for HTTP adapter and one for SOAP adapter).
Note
Both application pools may use the same BizTalk Server 2004 Isolated Host User identity.

Note
The actual host configuration used should be dictated by security and isolation requirements of the production environment.

Business Rules deployment: «Database associated with deployment driver does not match the database ":" specified during product configuration»

If you ever see this dreaded error message, don't fear!
First, follow each and every step suggested here: SOLVED: The BizTalk Rules Engine Deployment Riddle.
If it still doesn't work, then find the following tables in your BizTalkMgmtDb: adm_Group, adm_OtherDatabases and adm_OtherBackupDatabases.
In the first, fill the columns RuleEngineDbServerName and RuleEngineDbName.
In the other two, add a new line with the DefaultDatabaseName = RuleEngine DB, and the other columns with your installation-specific information.
Et voilá!

BizTalk Maintenance

Excellent article on Biztalk maintenance:

The BizTalk Ops Team - Maintaining a Healthy, Responsive and Available BizTalk Environment

Biztalk Performance tunning blog on MSDN

There is a useful blog about Biztalk performance tunning on MSDN:
Biztalk Performance

Defining new filenames for send handler

How to create more ellaborated filename?

Following the list of predefined file names given on my previous post, how can you get more than what Biztalk gives you?

Solution:

You can change the SourceFileName. In the construct message block of the orchestration use something like the following:

Message_2 = Message_1;
Message_2(FILE.ReceivedFileName) = "your_value"+ToString(Message_1.id);

And then when you are defining the send port use File Name as usually: %SourceFileName%

List of variables for naming files at in send handler

How to change file name

This isn't a trouble, but a useful information that should be on every biztalk programmer manual.

The name of the file generated, mainly for archiving porposes, sometimes should be more than a GUId. This is a complete list of all the variables that can be used at the send handler. This is only valid for writing files, and some of them are only available when the input comes from a file also.

Macro name	Substitute value
%datetime%	Coordinated Universal Time (UTC) date time in the format YYYY-MM-DDThhmmss (for example, 1997-07-12T103508).
%datetime_bts2000%	UTC date time in the format YYYYMMDDhhmmsss, where sss means seconds and milliseconds (for example, 199707121035234 means 1997/07/12, 10:35:23 and 400 milliseconds).
%datetime.tz%	Local date time plus time zone from GMT in the format YYYY-MM-DDThhmmssTZD, (for example, 1997-07-12T103508+800).
%DestinationParty%	Name of the destination party. The value comes from message the context property BTS.DestinationParty.
%DestinationPartyID%	Identifier of the destination party (GUID). The value comes from the message context property BTS.DestinationPartyID.
%DestinationPartyQualifier%	Qualifier of the destination party. The value comes from the message context property BTS.DestinationPartyQualifier.
%MessageID%	Globally unique identifier (GUID) of the message in BizTalk Server. The value comes directly from the message context property BTS.MessageID.
%SourceFileName%	Name of the file from where the File adapter read the message. The file name includes extension and excludes the file path, for example, foo.xml. When substituting this property, the File adapter extracts the file name from the absolute file path stored in the FILE.ReceivedFileName context property. If the context property does not have a value, for example, if message was received on an adapter other than File adapter, then the macro will not be substituted and will remain in the file name as is (for example, C:\Drop\%SourceFileName%).
%SourceParty%	Name of the source party from which the File adapter received the message.
%SourcePartyID%	Identifier of the source party (GUID). The value comes from the message context property BTS.SourcePartyID.
%SourcePartyQualifier%	Qualifier of the source party from which the File adapter received the message.
%time%	UTC time in the format hhmmss.
%time.tz%	Local time plus time zone from GMT in the format hhmmssTZD (for example, 124525+530).