Challenges in determining the Message Pattern using tracking database

Introduction

In complex BizTalk environment, it is important to visualize the patterns of how messages are processed within the system or integrate with multiple disparate systems to handle the message transfer from one point to another. It could be a simple direct transfer between the two ports, or through a complex messaging route passing through orchestrations to perform certain validations, etc. It is helpful to BizTalk Administrators or Business users to track the flow of messages in different stages. For this purpose, the BizTalk Admin Console has the capability to view the workflow of messages, although that feature will mainly be of use for BizTalk Administrators.

BizTalk360 has the functionality to allow users to view the graphical flow of the messages within your BizTalk Group. However, the challenge with the Graphical Message Flow Viewer is that, for instance, if there are 100 messages passing through the system at a point of time, each message is tracked at the service instance level with a unique id. The problem with this approach is that every single transaction is logged into the database. In an environment where there are lots of messages passing through the BizTalk system, there is every chance that the database will get filled up quickly.

Message pattern is to visualize interactive message flows with additional capabilities like the number of messages processed and Average Execution Time in the artifacts.

Message Pattern using Tracking Data

In BizTalk360 v8.0 introduces the “Messaging Patterns” capability to visualize a list of unique message flows that are being processed by the system. The key difference from Graphical Message Flow Viewer is that, in the Message Flow list, only the unique messaging patterns get recorded. As in the previous case, if there are 100 messages passing through the system at a point of time, not all the 100 message flows are recorded in the Messaging patterns. Only the unique message patterns are recorded, which could be just, say, 15 or 20.

For Messaging Patterns to work, a backend service (called Mapping Data) is responsible for keeping track of the unique message flows. That Mapping data service will collect the tracked message in the DTA DB.

Note: Make sure you have enabled Tracking at the ‘Pipeline’ level for the unique data flow data to be captured.

message-patterns-biztalk360

Challenges in determining the Message Patterns

Determining the message flows using tracking data has some challenges. As BizTalk Developers and Administrators might be aware that the BizTalk DTADb has its own challenges. For example, few customers, face performance issues with the tracking database when the BizTalk360 Analytics service fetches message flows using the MessageInout table;

  1. Query Time out

    In the initial version of Message Pattern, we have seen that query timeout issues have happened while retrieving message service instance data.

  2. Temp Database Growth

    To address this issue of query time outs, since BizTalk360 Version 8.5, the relevant data is fetched in a temporary table in the tracking database. However, with a few customers, this approach causes temp database growth.

Alternate Technique (BizTalkMgmt Database)

To overcome this challenge, in BizTalk360 9v.0 Phase2, we bring an alternative mechanism to determine the message patterns. The alternate way of finding the message patterns uses the BizTalk Management database relational data. It has the advantage that here the flow is determined without enabling the pipeline tracking at the artifact level.

Message Flows have been determined by the following models:

  1. Publish-Subscribe Model

    This method is used to determine the message flow between ports without any orchestration being involved. It uses the Filter property in send ports to understand the start and endpoint of the flow.
    send-port-propertiessend-receive-port

  2. Orchestration Port Binding Model

    The Orchestration Port Binding model uses the orchestration port binding relational data to determine the message flows. Using BizTalk Artifact tables (adm_ReceiveLocation, bts_receiveport, bts_sendport, bts_pipeline, bts_application) with orchestration port and orchestration port binding.

SELECT 'Pipeline' as fromServiceType,FROMPL.Name as fromServiceName,   
RP.nvcName as fromPortName, 
RP.uidGUID as fromUIdServiceId, 
RPAPP.nvcName as fromAppName,
							   
'Orchestration' as toServiceType, ORCH.nvcFullName as toServiceName, ORCH.uidGUID as toUIdServiceId, OP.nvcName as toPortName, '' as toAPPName					    
								
FROM [dbo].[bts_orchestration] ORCH
                                
INNER JOIN  [dbo].[bts_orchestration_port] OP ON ORCH.nID = OP.nOrchestrationID
                                
INNER JOIN [dbo].bts_porttype PT ON PT.nID = OP.nPortTypeID
                                
INNER JOIN  [dbo].[bts_orchestration_port_binding] OPB ON OPB.nOrcPortID = OP.nID                                
                                
LEFT OUTER JOIN [dbo].[bts_receiveport] RP ON RP.nID = OPB.nReceivePortID
                                
LEFT OUTER JOIN adm_ReceiveLocation RL ON RL.ReceivePortId = RP.nID
                                
LEFT OUTER JOIN bts_pipeline FROMPL ON FROMPL.Id = RL.ReceivePipelineId                               
                                
LEFT OUTER JOIN bts_application RPAPP ON RPAPP.nID = RP.nApplicationID
WHERE OP.nPolarity = 2 AND RP.nvcName 
IS NOT NULL AND (OP.nvcName  IS NOT NULL AND RP.nvcName IS NOT NULL)

orchestration-port-bindingflow

Message Pattern Configuration

Users can choose the message flow determination technique by enabling or disabling the Dynamic pattern option.  When Dynamic pattern option is enabled, then the Message pattern will be determined using the tracking database.  If it is disabled, it will use the BizTalk Management database to find the message patterns for the BizTalk Group.

message-pattern-configurationFew Points to be remembered to use this new mechanism (Dynamic Pattern – Disabled).

  • Pipeline tracking needs to be enabled to see the message count and average execution time
  • This mechanism will be able to determine the possible flows. In case a Port uses other filter criteria than Port Names, they are not determined

Other Improvements

In the upcoming version of BizTalk360 v9.0 phase 2, we are bringing management capabilities like hiding and deleting of message flows.

The Hide Operation can be used on the message patterns that are decommissioned or less frequently used. The Delete Operation can be applied on the flows that are no longer used in BizTalk Group.

messaging-patterns-hide

Conclusion

We hope these improvements will bring a new way to determine the message flows in your BizTalk environments. You can expect these improvements to be available in the upcoming version of 9.0 phase 2. We are constantly improving on customer feedback. Please share your requests through the feedback portal

Author: Senthil Kumar Palanisamy

Senthil Kumar Palanisamy is the Technical Lead at BizTalk360 having 15 years of experience in Microsoft Technologies. Worked various products across domains like Health Care, Energy and Retail.

Back to Top