BizTalk Application Insights in depth – Part 1

|  Posted: May 26, 2017  |  Categories: BizTalk Server

In my previous blog, I explained about installing BTS 2016 feature pack 1 and configuring it for Application Insights integration. In this article, I want to go a bit deeper and try to demonstrate –

You can download the entire article as a PDF document.
BizTalk Application Insights in depth
  • Nature of  tracking data sent to Application Insights
  • Structure of the data
  • Querying  data in Application Insights
  • Some practical examples of getting sensible analytics for BizTalk interfaces.

I am hoping to give a jump start to someone who wants to use the Application Insights for BizTalk Server 2016.

Tracking data

As you know the term tracking data in BizTalk refers to different types of data emitted from different artifacts. It could be in/out events from ports and orchestrations, pipeline components, it could be system context properties, could be custom properties tracked from custom property schemas, could be message body in various artifacts, could be events fired from rule engine etc. So we would like to know, whether we will be able to get all this data in Application Insights or is it just a subset. I will try to answer this question based on the POC I have created.

POC I created is pretty simple. It has one receive port which receives an order XML file, processes that in an orchestration and send it to two different send ports. It can be pictorially represented as below.

XML transmit receive pipeline

  • I have enabled pipeline tracking on XML receive and XML transport pipelines.

biztalk pipeline properties

  • Enabled track message body and analytics in Receive ports (If you want to know about Analytics option please refer to my first article)

Receive port properties

  • Enabled the Track Events, Track Message Bodies, and Analytics on orchestration.

biztalk orchestration properties

  • Enabled the Analytics on Send Port

send port properties

Note: I enabled a different level of tracking at different artifacts to see if it has an impact on the analytics data sent to Application Insights. Later I realized that different tracking levels do not have any impact on the analytics data.

Analytics Data in Application Insights

I placed a single file into the receive location and started observing the events pushed to Application Insights. In general, Applications integrated with Application Insights can send data belonging to various categories, such as traces, customEvents, pageViews, requests, dependencies, exceptions, availabilityResults, customMetrics, band browserTimings. With BizTalk, I have observed that data belongs to “CustomEvents” category. Following are the custom events which are ingested from my BizTalk interface.

application insights

  • There are two events for a receive port.
  • There is an event for every logical port inside the orchestration. And hence we can see three events in total for orchestration.
  • There are two events for each send port.

All these events can be related to events logged into “Tracked events” query results which are shown below.

biztalk application query results

Structure of a BizTalk custom event

In the previous section, we saw that our BizTalk interface emitted various custom events for ports and orchestration. In this section, we will look into the structure of data which is captured in a custom event.

Event Metadata

Event metadata is the list of values which defines an event. Following are the event metadata in one of the custom events.

biztalk client details

Custom Dimensions

Custom dimensions consist of the service instance details and context properties promoted in the messaging instance. Hence we can observe two different kinds of data under custom dimensions.

Service instance properties: These are the values specific to service instance associated with the messaging event.

biztalk service instance properties

Context properties:  All the context properties which are non-integer type will be listed under the custom dimensions.

context properties

Custom Measurements

As per my observation, custom measurements only contain the context properties of integer type.
custom measurements

Since there is no proper documentation regarding this, I tried to prove this theory by creating three custom properties in a property schema and promoted the fields in the incoming message. Following is the property schema that I defined.

biztalk property schema

I observed that PartyID and AskPrice properties which are of type string and decimal respectively are moved to Custom Dimensions section. Property Quantity which is of type integer is moved to Custom measurement.

Querying data

As discussed in above section all the BizTalk events are tracked under the customEvents category. Hence our query will start with customEvents.

querying data in biztalk

Query language in Application Insights is very straightforward and yet very powerful. If you want to find out all the construct of this query language please refer this link Application Insights Analytics Reference.

In this section, I would like to cover some concepts or techniques which are relevant for querying BizTalk events.

You can download the entire article as a PDF document.
BizTalk Application Insights in depth

Convert the context property values to specific types

In Application Insights, the context property values are stored as dynamic types. When you directly use them into queries especially in aggregations, you will receive a type casting exception as shown below.

converting property values

To overcome this error, you will need to convert the context property to a specific type as shown below.

custom events

Easy way to bring a context property key into the query

Since context properties are a combination of namespace and property name, it will be a bit of an effort to type them in the queries that we create. To bring the context property on to the query page easily, follow steps as below.

  • Query for custom events and navigate to the property you are interested in the results section.
  • When you hover the mouse on the desired property, we will get two buttons. ‘+’ for inclusion and ‘-’ for exclusion.

property context key

Selecting specific fields

If you already know app insights query language, this tip is not so special. But if you are new to it and trying to find out how to select a column, you will face some difficulty as I did. The main reason for this is there is no construct called “select”. Instead, you will have to use something called “project”. Below is an example query.

selecting specific fields

Some useful sample queries.

In this section, I will try to list some queries which I found useful.

Message count by port names

query 


customEvents
| where customDimensions.Direction == "Receive" 
| summarize count() by tostring(customDimensions.["PortName (http_//schemas.microsoft.com/BizTalk/2003/messagetracking-properties)"])

Chart

biztalk analytics

Messaging Volume by schema

Query

customEvents
| where customDimensions.Direction == "Receive" 
| summarize count() by tostring(customDimensions.["MessageType (http_//schemas.microsoft.com/BizTalk/2003/system-properties)"])

Chart

biztalk server analytics

Analytics with custom context properties.

Ability to generate analytics reports based on the custom promoted properties is a very powerful feature which really makes using application insights interesting. As I explained in previous sections I have created a custom property schema to track PartId, Quantity and AskPrice fields. Now we will see some example reports based on this.

Total quantity by part id

Query

customEvents
| where customDimensions.PortType == "ReceivePort" 
| where customDimensions.Direction == "Send" 
|summarize sum(toint(customMeasurements.["Quantity (https_//SampleBizTalkApplication.PropertySchema)"])) by PartId = tostring(customDimensions.["PartID (https_//SampleBizTalkApplication.PropertySchema)"])

Chart
biztalk analytics data

Total sales over period of time

Query

customEvents
| where customDimensions.PortType == "ReceivePort" 
| where customDimensions.Direction == "Send" 
| summarize sum(todouble(customDimensions.["AskPrice (https_//SampleBizTalkApplication.PropertySchema)"])) by bin( timestamp,10m)

Chart
sales chart with biztalk server

Pinning charts to Azure dashboard

All the charts that you have created can be pinned to an Azure dashboard and you can club these charts with other application dashboards as well. My dashboard with the charts that we created looks as below.

azure dashboard

Summary

In summary BizTalk analytics option which is introduced in BizTalk Server 2016 Feature Pack 1 is useful to get analytics out of tracking data. I would like to conclude by stating following points.

  • Only the tracked messaging events, service instance information and context properties of associated service instance are sent to App iInsightsby analytics feature. Message body, pipeline events, business rule engine events etc. are not being pushed out.
  • Under messaging events, I was unable to find the Transmission Failure events for send ports. This will be useful for getting metrics on failure rates. If you agree with this observation please vote here.
  • The different level of tracking on ports and orchestrations does not have an impact on the data being transmitted to app insights.
  • Orchestration failures/suspended events are not pushed to application insights. It would be good if Microsoft provides an extensible feature to push exceptions from orchestrations. If you agree please vote here
  • There is no control on what context properties published to app insights. It is all or nothing scenario. It would be good to have control on it. Especially when you are promoting and tracking business data. If you agree please vote here.
  • Ability to perform analytics based on context property values can turn out to be a powerful feature for BizTalk implementations.
You can download the entire article as a PDF document.
BizTalk Application Insights in depth
Author: Srinivasa Mahendrakar

Technical Lead at BizTalk360 UK - I am an Integration consultant with more than 11 years of experience in design and development of On-premises and Cloud based EAI and B2B solutions using Microsoft Technologies.

One Platform Operations, Monitoring and Analytics Software
BizTalk360

microsoft biztalk

Learn more

Over 500 customers across 30+ countries depend on BizTalk360

ServiceBus360

Azure service bus

Learn more

Start managing your Azure Service Bus namespaces in minutes

One Platform - Operations, Monitoring and Analytics Software
BizTalk360

microsoft biztalk

Learn more

Over 500 customers across 30+ countries depend on BizTalk360

One Platform - Operations, Monitoring and Analytics Software
ServiceBus360

Azure service bus

Learn more

Start managing your Azure Service Bus namespaces in minutes

Back to Top