Category: SCOM APM

Wow! Was it always here?

In the previous post, I shared my experience on how to configure a SCOM APM to get the most from it. In this post, I will write about a SCOM APM Transaction and what an excellent feature it is. At the end of this short article, I am sure, you will discover that if you use it right. By investment in a small amount of time, you will decrease the downtime of your organization.

I have seen many administrators using the transaction feature to monitor only business workflows. Although this is good, you should use the transaction to monitor endpoints or resources that are used by the application, so if something goes wrong, you will quickly figure out what causes to do so.

So, how you achieve it?

It begins by investigating monitored endpoints or resources that are used by the application we want to monitor, and what triggers them. Once we know it, we continue by creating a transaction to monitor in a SCOM APM transaction wizard for each monitored endpoint or resource based on a trigger. In addition, we end it with a big smile 🙂

To clarify my point, I will use a ridiculous simple metaphor. Let us assume that a store has four departments. Each department is a “transaction.” And each has only one unique supplier. Therefore, whenever one of them has a problem, let’s say, a series of bad products, a manager quickly contacts the supplier that is responsible for it.

For each application type, there are several transaction types you can choose to monitor

Application type Transaction types for System Center 2012 Transaction types for System Center 2012 SP1
ASP.NET Web application
  • ASP.NET webpage
  • ASP.NET web service
  • Function
  • ASP.NET webpage
  • ASP.NET MVC page
  • ASP.NET web service
  • WCF method
  • Function
ASP.NET Web service
  • ASP.NET webpage
  • ASP.NET web service
  • Function
  • ASP.NET webpage
  • ASP.NET MVC page
  • ASP.NET web service
  • WCF method
  • Function
WCF service Not available
  • ASP.NET webpage
  • ASP.NET MVC page
  • ASP.NET web service
  • WCF method
  • Function
Windows service Not available
  • WCF method
  • Function

This table is from a Microsoft documentation about .NET Application Performance Monitoring Template. I recommend to read it. You will find there all the necessary technical information that you need to use the transaction feature.

If you have any questions, I will be happy to help you.

SCOM APM, What? Why? How?

The APM is an abbreviation of Application Performance Monitoring. It monitors application code. For example, when you click on a button in an application the APM runs in the background and tracks your activity.

The APM can help us catch bugs easily. For instance, a development team that I work with had an unpredictable application behavior. They tried many ways to find what caused this bug. After two years of searching, they ask me to install SCOM APM in a production environment. It took only several days to catch the bug and figure out what caused it.

I had many successful SCOM APM cases like this one. Therefore, I was surprised to hear from my counterparts that the SCOM APM is not a favorite SCOM feature. I think that the reason is a lack of knowledge. You can destroy the monitored application if you do not know how the SCOM APM works. Therefore, a typical SCOM administrator that has a system administrator background would prefer not to implement the SCOM APM while a SCOM administrator with a developer background would probably know how to tune the APM so that it return meaningful data with low-cost effect on the application.

Having said this, this post is all about how the SCOM APM works so you could tune the SCOM APM wisely to experience the power of the SCOM APM.

It is important to understand all the different APM tools using similar resources to monitor an application code. Currently, I will focus on a DotNet application and SCOM APM tool, but under the hoods, this tool works very similarly to other APM tools.

SCOM APM uses the .Net profiling API. It is a part of .Net Framework and basically it allows you to observe the .NET runtime in action. The .NET profiling API is COM-based. Using the profiling API involves creating an in-process COM server that implements a single interface. Each interface method represents a single event from  one of the following categories:

  • CLR startup and shutdown
  • Application domain creation/shutdown
  • Assembly loads/unloads
  • Module loads/unloads
  • Class loads/unloads
  • COM interop VTable creation/destruction
  • Just-in-time (JIT) compilation and code-pitching events
  • Thread creation/destruction/suspends
  • Remoting activity
  • Exception handling
  • Managed/unmanaged code transitions
  • Garbage collection and managed heap objects
  • Method entry/exit


In other words, the SCOM APM uses a COM server object DLL that implements the required profiling interface. The next task is to force the CLR into loading the COM server and invoking the methods. This trick is accomplished by configuring environment variables. The SCOM APM configures these variables in the registry of the preferred application.

Reg

There are two main variables in the registry: COR_PROFILER and Cor_Enable_Profiling. These are used by CLR for loading the COM Server object. The loading process begins with CLR check of the Cor_Enable_Profiling (it can have one of the two values 1 – enable or 0 – disable loading the COM server object). If the value is 1 the CLR will attempt to load the COM server as specified in the COR_PROFILER environment variable. In the SCOM APM case, COM server object is the StubProfiler.dll, as you can see with OleViewCOMObject tool:

COMObj

Once the CLR loads the StubProfiler.dll it will implement the required profiling interface and from this moment and on, every event will notify StubProfiler. The StubProflier will pass the data to the SCOM Agent where all the magic happens.

ProfilingArch
Profiling architecture

Let’s go deeper by using Sysinternals Process Explorer tool; it is an excellent tool that can help us investigate the In-Process mechanism, in this screenshot you see the SCOM Agent managed DLLs inside the w3wp.exe:

ProcessExplorer.png

After that we understand the In-Process feature, we can go forward to investigate these DLLs managed code. For this we can use ILSpay tool, it is an excellent tool for reflection of .NET code:

ILSPY

I won’t go through all of what it does, although, it is important to mention that the main class is InstrumentationClass. It is responsible for the monitor functions. These functions are marked as may interest the user either for performance or exception analyze, as well as, to get all monitored function arguments and collecting event data that will be sent to SCOM APM AppDiagonstics. To mark functions the SCOM APM uses a configuration file it is a PMonitor.config file. It contains entry points, application, and all the data producers definitions. Note! A wrong configuration can cause an unpredictable behavior of the monitored application because each producer must have a proper implementation in InstrumentationClass class, as you can see here: Producer.png

After understanding this you ready to tweak the SCOM APM configuration.

In the next post, we go forward on all possible configurations in SCOM APM template, and how you can decrease the effect on the application on the one hand and the other side returns a meaningful data.