If you (partial or temporary) need to stop storage to the data warehouse, you can just override the default overrides (again) to set the Drop Items parameter to true. This will, after propagation to the management server, cause the items to be dropped (and not stored into the data warehouse database).

Note that while this is possible, I assume it is a non supported configuration

1. Manually enable the proxy agent setting for all Exchange 2010 servers (now and in the future). Which means an Exchange 2010 server will not be discovered until we actually do.
2. Enable the proxy agent for all agents.

Counting at the moment around 60 percent of the agents already has the proxy functionality enabled. So what’s the advantage of not setting this setting on default for all agents? Looking at security, you have to enable this setting already for all security important servers (AD, Exchange, ISA, Citrix, etc.). And since we have no knowledge of when an Exchange server is connected to our environment, we decided to enable it for all agents.

This is the script to do it:

$rootMS="RMS.TEST.LOCAL"

#——————————————————————————-
# Add operations manager snapin and connect to the root management server.
#——————————————————————————-
add-pssnapin "Microsoft.EnterpriseManagement.OperationsManager.Client";
set-location "OperationsManagerMonitoring::";
new-managementGroupConnection -ConnectionString:$rootMS;

## set proxy enabled for all agents where it is disabled
$NoProxy = get-agent | where {$_.ProxyingEnabled -match "False"}
$NoProxy | foreach {$_.ProxyingEnabled = $true}
$NoProxy | foreach {$_.ApplyChanges()}

See also Kevin Holman’s blog

When we looked at the results, we did see some strange things. First, the number of state changes were equal for a lot of monitors.

An example of this:

Having exact the same number of state changes for two different monitors? When we looked into these monitors, they were disabled by default in the management pack were the monitor was defined. Digging further showed that there was no override present in the system, which would enable this monitor.

We now had a problem while from our top 50, 48 of them were monitors that were default disabled, without any override to enable the monitor.

This turns out to be a bug in the agent software. The moment the agent (re-) initializes either by starting or coming out of maintenance mode it will detect the monitor and initialize it. When realizing it should (default) disable the monitor, it will send a state change for the the disabled monitor.

This is also the case for monitors that are default enabled, but are disabled using a custom override. Unfortunately a fix for this issue is not as easy as it sounds (according to Microsoft support) and a fix will not be realized in the R2 version of SCOM.

So if you have tuned a lot (like us) and find these monitors, just skip them. You can’t fix this one

Below is a list of monitors we found that were default disabled during this investigation and that we now exclude from the query.

If you want to exclude any (default) disabled monitor that you found, exclude it in the query as shown below:

use OperationsManager
go

select distinct top 50 count(sce.StateId) as NumStateChanges, m.MonitorName, mt.typename AS TargetClass
from StateChangeEvent sce with (nolock)
join state s with (nolock) on sce.StateId = s.StateId
join monitor m with (nolock) on s.MonitorId = m.MonitorId
join managedtype mt with (nolock) on m.TargetManagedEntityType = mt.ManagedTypeId
where m.IsUnitMonitor = 1
and m.MonitorName not in (

‘Microsoft.Windows.Server.2003.LogicalDisk.AvgDiskSecPerWrite’,

‘Microsoft.Windows.Server.2003.LogicalDisk.AvgDiskSecPerRead’

)
group by m.MonitorName,mt.typename
order by NumStateChanges desc

We still had config churn, but running the queries form Kevin Holman did not point us to the reason for those config churns. But we got some information with a support case.

First we can retrieve exact what is changed, using this query (run on against de data warehouse database):

use OperationsManagerDW
go

select * from dbo.ManagedEntityProperty
where DWCreatedDateTime > dateadd(hh,-24,getutcdate())
order by DWCreatedDateTime, ManagedEntityRowId

This will result in output similar to this:

It will give you all properties that are changed within the last 24 hours and what exactly is changed. Now when you “click” on a PropertyXML value or DeltaXml, a new windows will be opened showing you exact which properties there are and which are changed.

But now we don’t have any idea were to find this in a management pack, but we will get there. From the above output, take the ManagedEntityRowId and place this in the next query:

use OperationsManagerDW
go

select * from ManagedEntity
where ManagedEntityRowId = 121403

This will result in output similar to this:

The ManagedEntityGuid is what we need here. Place it in the next query (which will run against the operations database):

use OperationsManager
go

select * from BaseManagedEntity
where BaseManagedEntityId = ‘3B9F6E60-02B5-8369-859F-8047093CE33F‘

The result is:

The next thing we need is the BaseManagedTypeId

use OperationsManager
go
select * from ManagedType
where ManagedTYpeId = ‘10C1C7F7-BA0F-5F9B-C74A-79A891170934‘

Which results in:

Here you can find the TypeName that is discovered (Microsoft.SQLServer.Database in this case). Use the ManagementPackId to get the actual management pack:

Use OperationsManager
Go
Select * from ManagementPack where ManagementPackId = ‘BCD6DCCF-C46C-A1F5-3C8D-BB4E99E2A6A3‘

And the final result will be:

So we now know that the property is of type “Microsoft.SQLServer.Database” and that it is discovered in the “Microsoft.SQLServer.Library” management pack (aka “Microsoft SQL Server Core Library”).

Note that if you are only interested in the actual management pack name, you can also use this query (which uses the ManagedEntityGuid from the first query against the operations database):

use OperationsManager
Select * from ManagementPack
where ManagementPackId = (
    select ManagementPackId from ManagedType
    where ManagedTYpeId = (select BaseManagedTypeId from BaseManagedEntity
        where BaseManagedEntityId = ‘3B9F6E60-02B5-8369-859F-8047093CE33F‘
    )
)

This will result in the same output as the last screenshot (but now you don’t know the type for the data). When you have this information, you can look up the corresponding discoveries so you can fine tune them if required.

Below is the final XML file for your management pack called jamaAgent.Update.xml:

<?xml version="1.0" encoding="utf-8"?><ManagementPack ContentReadable="true" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <Manifest>
    <Identity>
      <ID>jamaAgent.Update</ID>
      <Version>0.4.0.0</Version>
    </Identity>
    <Name>jamaAgent.Update</Name>
    <References>
      <Reference Alias="MSWL">
        <ID>Microsoft.Windows.Library</ID>
        <Version>6.1.7221.0</Version>
        <PublicKeyToken>31bf3856ad364e35</PublicKeyToken>
      </Reference>
      <Reference Alias="MSSCL">
        <ID>Microsoft.SystemCenter.Library</ID>
        <Version>6.1.7221.0</Version>
        <PublicKeyToken>31bf3856ad364e35</PublicKeyToken>
      </Reference>
    </References>
  </Manifest>
  <Monitoring>
    <Tasks>
      <Task ID="jamaAgent.Update.ConsoleTask.AgentUpdate" Accessibility="Public" Enabled="true" Target="MSSCL!Microsoft.SystemCenter.HealthService" Timeout="300" Remotable="true">
        <Category>Custom</Category>
        <WriteAction ID="PA" TypeID="MSWL!Microsoft.Windows.ScriptWriteAction">
          <ScriptName>jamaTaskUpdateAgent.vbs</ScriptName>
          <Arguments />
          <ScriptBody><![CDATA[
'-------------------------------------------------------------------------------
' File   : jamaTaskUpdateAgent.vbs
' Use    : Script for the jama Agent Update task.
' SVN    : Revision: 136
'          Date: 2011-04-12 09:14:33 +0200 (Tue, 12 Apr 2011)
'
' Note(s): 1) ---
'-------------------------------------------------------------------------------
option explicit
on error goto 0
setlocale("en-us")

const INT_RETRIES           = 5
const INT_DEFAULT_WAIT_TIME = 120
const STR_DOWNLOAD_PATH     = "/files/opsmgr/updates/agent/"

jamaMain()

'-------------------------------------------------------------------------------
' jamaMain
'
' Use    : Main entry for this script.
' Input  : ---
' Returns: ---
' Note(s): 1) ---
'-------------------------------------------------------------------------------
function jamaMain()
    dim jdf
    dim iMinutesToWait
    dim bForceUpdate

    if(not jdfLoadFramework(jdf, null, null, null, null, null)) then
        wscript.echo "The Jama Distribution Framework could not be loaded. No update is performed."
    else
        wscript.echo "Jama Distribution Framework loaded" & vbNewLine & vbNewLine & jdf.GetInfoString(wscript.fullname)
        '---------------------------------------------------------------------------
        ' Your code here!
        '---------------------------------------------------------------------------
        if(jamaInitialize(iMinutesToWait, bForceUpdate)) then
            if((bForceUpdate = false) and jamaRestartRequired()) then
                wscript.echo "Update could not be scheduled due to dependent services." & vbNewLine & _
                             "Use the force option to override this behaviour and force the update to run."
            else
                if(jamaScheduleUpdateIn(jdf, iMinutesToWait) = true) then
                    wscript.echo "Update is scheduled to run in " & iMinutesToWait & " minutes."
                else
                    wscript.echo "Update could not be scheduled. No update will be performed!"
                end if
            end if
        else
            wscript.echo "Initilization failed. No update will be performed!"
        end if
    end if
end function

'-------------------------------------------------------------------------------
' jamaInitialize
'
' Use    : Initialize the script.
' Input  : iMinutesToWait - integer - Number of minutes to wait before
'                                     activating the schedule (output)
' Returns: Boolean - TRUE  - No errors detected.
'                    FALSE - An error was detected.
' Note(s): 1) ---
'-------------------------------------------------------------------------------
function jamaInitialize(byref iMinutesToWait, byref bForceUpdate)
    dim colNamedArguments
    dim bResult
    dim bFound

    bResult = false
    bFound  = false
    bForceUpdate = false
    set colNamedArguments = WScript.Arguments.Named

    if(colNamedArguments.Exists("minutes")) then
        iMinutesToWait = cint(lcase(colNamedArguments.Item("minutes")))
        bFound         = true
        bResult        = true
    end if

    if(colNamedArguments.Exists("force")) then
        if(lcase(colNamedArguments.Item("force")) = "true") then
            bForceUpdate = true
        end if
    end if

    if(not bFound) then
        iMinutesToWait = INT_DEFAULT_WAIT_TIME
        bResult        = true
    end if

    jamaInitialize = bResult
end function

'-------------------------------------------------------------------------------
' jamaRestartRequired
'
' Use    : Checks if an update of the agent will trigger a restart or reboot.
' Input  : ---
' Returns: Boolean - TRUE  - A restart is required.
'                    FALSE - A restart is not required.
' Note(s): 1) If the dependency could not be determined, this function will
'             return true.
'          2) For windows 2000 this function will always return true.
'          3) When true on a 2003 server, a reboot is required but not forced.
'          4) When true on 2008 or higher, the dependent services could be
'             restarted by the installer service.
'
'-------------------------------------------------------------------------------
function jamaRestartRequired()
    dim strCmd
    dim iResult
    dim bResult
    dim fso
    dim fh
    dim strTempFile
    dim strLine

    strTempFile = "jamaTaskUpdateAgent.$$$"
    bResult = true
    strCmd  = "tasklist /fo csv /m EventCommon.dll /FI ""imagename ne HealthService.exe"" /FI ""imagename ne MonitoringHost.exe"" > " & strTempFile
    jamaRun(strCmd)

    set fso = CreateObject("scripting.filesystemobject")
    if(fso.FileExists(strTempFile)) then
        on error resume next
            err.clear
            set fh = fso.OpenTextFile(strTempFile, 1)                           ' Open for reading.
            if(err.number = 0) then
                do while(not fh.AtEndOfStream)
                    strLine = lcase(fh.ReadLine)
                    if(instr(strLine, "info: no tasks") > 0) then               ' No dependencies found.
                        bResult = false
                    end if
                loop

                set fh = fso.GetFile(strTempFile)
                fh.delete(true)                                                 ' delete file.
            end if
        on error goto 0
    end if

    jamaRestartRequired = bResult
end function

function jamaRun(byval strCmd)
    dim objShell
    dim iResult

    iResult      = 0
    set objShell = wscript.createObject("wscript.shell")
    iResult      = objShell.run("cmd /c " & strCmd, 0, true) ' hidden and wait for result
    set objShell = nothing
    jamaRun      = iResult

    jamaRun = iResult
end function

'-------------------------------------------------------------------------------
' jamaScheduleUpdateIn
'
' Use    : Retrieve and schedule the required update.
' Input  : jdf            - object  - JDF object
'          iMinutesToWait - integer - Minutes to wait for the schedule.
' Returns: Boolean - TRUE  - No errors.
'                    FALSE - An error was detected.
' Note(s): 1) ---
'-------------------------------------------------------------------------------
function jamaScheduleUpdateIn(byref jdf, iMinutesToWait)
    dim strSourceFile
    dim strTargetFile
    dim strTargetDir
    dim iCount
    dim iResult
    dim bResult
    dim strCmd
    dim fso

    iCount = 0
    bResult = false
    set fso = CreateObject("scripting.filesystemobject")
    strSourceFile = "jamaAgentUpdate" & jdf.Platform & ".exe"
    strTargetDir  = jdf.ExpandString("$JDF_IN_DIR$" & "jamaAgentUpdate\")
    if(jdf.CreateDirectory(strTargetDir)) then
        strTargetFile = strTargetDir & strSourceFile
        bResult = fso.FileExists(strTargetFile)
        if(not bResult) then
            do
                iResult = jdf.GetFile("jdfBaseDistribution", STR_DOWNLOAD_PATH & strSourceFile, strTargetFile)
                iCount  = iCount + 1
            loop while((iCount < INT_RETRIES) and (iResult <> 0))
        end if
        if(iResult = 0) then
            bResult = fso.FileExists(strTargetFile)
            if(bResult) then
                strCmd  = strTargetFile & " /verysilent"
                bResult = jdf.ScheduleTaskIn(strCmd, iMinutesToWait)
            end if
        end if
    end if
    jamaScheduleUpdateIn = bResult
end function

'-------------------------------------------------------------------------------
' From template.vbs
'-------------------------------------------------------------------------------
'-------------------------------------------------------------------------------
' jdfLoadFramework
'
' Use    : Load and initialize the JDF framework.
' Input  : objJDF               - object - Object passed back with framework.
'          bInitialize          - bool   - true or null for initializing.
'          strVersion           - string - Minimum framework version required.
'          bForceVersion        - bool   - true, false or null.
'          strCustomerId        - string - Customer id or null.
'          strDefaultUploadPath - string - Default upload path.
' Returns: bool - TRUE  - Initialization of the framework succeeded.
'                 FALSE - Initialization of the framework failed.
' Note(s): 1) strVersion = null
'                 Any version of the framework will be accepted. The value of
'                 strForceVersion will be ignored.
'
'          2) strVersion = "x.x.x"
'                 Integer values, seperated by dots, e.g. : "3.5.11"
'
'          3) bForceVersion = null/false
'                 The given version is a minimum version required for the
'                 framework.
'
'          4) bForceVersion = true
'                 The framework version must exactly match with the given
'                 version number.
'
'          5) strCustomerId = null
'                 The customer id will be retrieved from the registry. See
'                 the documentatiohn for more information.
'
'          6) strDefaultPath = null
'                 The default upload path will be set to the root: "/". This
'                 argument can hold JDF variables (both default and custom
'                 provided in the jdf.jdp file). See the documentation for more
'                 information.
'
'          7) Normal use is:
'                 jdfLoadFramework(jdf, null, null, null, null, null)
'                 where jdf is the object you pass to the function.
'-------------------------------------------------------------------------------
function jdfLoadFramework(byref objJDF, byval bInitialize, byval strVersion, byval bForceVersion, byval strCustomerId, byval strDefaultUploadPath)
    dim bResult, fso, strFrameworkFile, objReg, strResult

    bResult   = false
    strResult = ""
    on error resume next
        err.clear
        set objReg=GetObject("winmgmts:{impersonationLevel=impersonate}!\\.\root\default:StdRegProv")
        if(err.number = 0) then
            objReg.GetStringValue  &H80000002, "SOFTWARE\Company\jama\jdf", "path", strResult     ' If you use another registry location, change this value!
            if((err.number = 0) and (strResult <> "") and (not isnull(strResult))) then           ' We got something, so try it.
                strFrameworkFile = strResult & "jdf.wsc"                                          ' This now should be the full path to the framework file.
                set fso = CreateObject("Scripting.FileSystemObject")
                if(err.number = 0) then
                    if(fso.FileExists(strFrameworkFile)) then
                        set objJDF = GetObject("script:" & strFrameworkFile)                      ' File exist, try to load it.
                        if(err.number = 0) then                                                   ' Framework found and object loaded.
                            if(bInitialize = false) then                                          ' No initialization.
                                bResult = false
                            else                                                                  ' Initialize the framework for use.
                                bResult = objJDF.InitializeFramework(strVersion, bForceVersion, wscript.scriptname, strCustomerId, strDefaultUploadPath)
                            end if
                        end if
                    end if
                end if
            end if
        end if
    on error goto 0
    jdfLoadFramework = bResult
end function
]]></ScriptBody>
          <TimeoutSeconds>300</TimeoutSeconds>
        </WriteAction>
      </Task>
    </Tasks>
  </Monitoring>
  <LanguagePacks>
    <LanguagePack ID="ENU" IsDefault="false">
      <DisplayStrings>
        <DisplayString ElementID="jamaAgent.Update">
          <Name>jamaAgent.Update</Name>
        </DisplayString>
        <DisplayString ElementID="jamaAgent.Update.ConsoleTask.AgentUpdate">
          <Name>jama Agent Update</Name>
        </DisplayString>
      </DisplayStrings>
    </LanguagePack>
  </LanguagePacks>
</ManagementPack>

Now the script in this management pack expects a few things.

const STR_DOWNLOAD_PATH = "/files/opsmgr/updates/agent/"

This constant is used to specify the path on the remote SCP server, where the update files can be retrieved. If you use another location, change the value of the variable.

strSourceFile = "jamaAgentUpdate" & jdf.Platform & ".exe"

This value is used to generate the name of the update to retrieve. We have re-packaged the two update files required for the agent update into a single executable (see also this article from Kevin Holman for more information). You can download an archive with three packages (for all suported platforms) on this location. If you create you own, just make sure the final name of the update package corresponds with the final strSourceFile variable value:

jamaAgentUpdateIA64.exe –> For the itanium platform
jamaAgentUpdateX64.exe –> For the x64 platform
jamaAgentUpdateX86.exe –> For the x86 platform

What we actually do is very simple.

• We detect the platform we are running on and construct the correct file name of the file to retrieve (this update package contains the required .msp files for the agent).
• We retrieve the update package.
• We schedule the update package to run in ## minutes and to run unattended.
• We hope for the best

If something goes wrong with the installation, the failure can be found in the log file created by the update. There are two logfiles:

%TEMP%\jamaAgentUpdate.<platform>.log
%TEMP%\jamaAgentUpdate.ENU.<platform>.log

Also the package itself will again detect we if we are running on the correct platform and only start the update if the package is the correct platform version.

As you can see in the .XML file, we use the jdfBaseDistribution account for retrieving the file. So you don’t need to configure the SCP server for every customer you have. Just make the file available for the jdfBaseDistribution account. The task itself is targeted against the health service. Select the “Agent By Version” leaf in the console, to get an overview of all your agents and their version:

If you select a health service, the task pane will show you the new update task:

Note that the task is not version aware, so it will always be available and will run. So you can do an update over a current installed agent (which does work, without issues). After selecting the task, the tasks pane will be shown and you can change the arguments for the script.

If you don’t change anything, the task will try to schedule the update to run in 120 minutes. If you require the another time, use the /minutes:## argument. The task will be scheduled ## minutes in the future. So if I want the update to run within 5 minutes:

The first thing the task does is checking for dependencies (for more information about the dependencies, see this link). If no dependencies are found, the job is scheduled ## minutes in the future (or 120 if no override is given):

If a dependency is found, the tasks will not schedule the update:

But as the output shows you can override the dependency detection, by adding the /force:true option in the argument list. If set the update will always run, even if there are dependencies detected.

After the update is finished, the agent should reflect the new version information:

You now can update your manual installed agent, using the operations console

Transferring files

The idea behind the framework is to write an set of scripts, that would make it possible to transfer files to and from the SCOM agent. The main trigger for this wish was our installed base, which are manual installed agents only. Updating these agents manually is a long job for every CU update. If we could just schedule an update from the operations console and sit-back, that would be great. But in order to do that, we need to be able to get the required update files to the SCOM agent. Keep in mind that we do not have a single customer but multiple customers, all with (or without) their own file distribution method. Being able to distribute and update from the operations console itself would be ideal.

But transferring files should also be secure. Our solution: Secure Copy, based on private / public key pairs for data transfers.

And since we have multiple customers, we need to make sure that Customer A is not able to see or change any data we retrieve from Customer B (and vice versa). Therefor we have to implement a secure way to copy files to and from the SCOM agent and separate data based on customers. Our solution: A Secure Copy implementation, that is capable of creating virtual directory structures, based on a customer (secure copy) account.

We did not want to change any firewall rules (with the exception of adding a new management server). So we basically wanted to use communication from the SCOM agent on TCP port 5723 and only initiate communication from the SCOM agent. Our solution: A Secure Copy server configured to listen on port 5723 for secure copy clients.

Finally it needed to be implemented on all our agents, not just a subset of agents. Our solution: A Microsoft Windows Scripting Component, written in vbscript, which will run on all Windows versions we support (Windows 2000 and higher).

So if we draw this in a high overview, it would look like this:

And yes it looks very simple So the first thing we need is a secure copy server.

Secure Copy server

The secure copy server can be any implementation, as long as it is capable of implementing private/public key pairs and able to generate virtual directories. We use the bitvise winsshd service. Now before we start implementing this service, we need to make some rules.

1. Only logins with private/public ssh key pairs are allowed.
2. Only secure copy is allowed (no sftp or ssh).
3. There will only be one general (shared) account and this account has read only access only to it own virtual directories. We call the shared account: jdfBaseDistribution This will prevent accounts from uploading data to a “shared” location. This is also the account that will be used to distribute the framework.
4. Data will be compressed as much as possible, but the framework will not rely on the secure copy implementation and will “zip” the files to upload into a “package”.
5. When a “package” is downloaded, it is expected to be a “zip” file, which can be extracted at the SCOM agent side by the framework.
6. Files (not “packages”) that will be downloaded, must be compressed files to reduce network traffic (this is not forced by the framework).
7. Customer accounts will have the naming convention: jdfCustomer_<CustomerName>
8. The root (virtual) directory for every account will always be empty. All directories will be read only, with the exception of the “upload” directory. But it will not be possible to remove files from the “upload” directory. The following structure will be used for every customer

 

Off course you can implement your own directories and rules for read/write, but these are the rules we use to explain the framework. If you start with these, the examples and code will work correctly.

 

Note: The shared account (jdfBaseDistribution) will not have a /upload directory!

A seperate account for each customer

We also need an account for every customer we support. Each customer will have the same folder layout (as show above), but this will be virtual directories. This will result in the same “view” for every customer, but data will be uploaded and downloaded from customer specific locations on the Secure Copy server.

Note: Files can not be deleted from the upload directory.

As can be seen, the account jdfCustomer_JAMA has four virtual mount paths. And with the exception of the root path (which should never be used), all data is ‘re-directed’ to a directory specific for that account. So if we create a second account (jdfCustomer_JAMA00), you can see that it has its own ‘re-directed’ directories.

This results in the same view for both customers (the Virtual mount path), but data being stored and retrieved from physical different locations.

We only allow the secure copy protocol and only logins with private/public key pairs. On how to configure winsshd with private/public key pairs, see this link.

ssh key pairs

For each customer we need to create an ssh public/private key pair. We use puttygen.exe to create the key pairs. The public key is saved as “jdfCustomer_<CustomerName>.public” and the private key is saved as “jdfCustomer_<CustomerName>.private”.

Note: You can not use a password phrase for the private key file (not supported by the framework). So be sure to only distribute the private key file to the correct customer.

When creating the keys with puttygen.exe, you should use the default Paramters, as shown below:

When your implementation of the secure copy server is working correctly you can continue with the next step: configuring the framework. Just make sure you can use the secure copy server with private/public key pairs before you continue.

The JDF Framework

The JDF Framework itself needs to be configured for use in your environment. What is required is fully described in the documentation of the framework. Although I tried to create a generic framework, still some environment dependent settings are required. But first you should download the framework

The framework can be downloaded here

The documentation can be downloaded here

What’s next

If you have setup the framework, you can start testing it. The framework download includes two examples on how to use the framework. The first thing we created with this framework was the ability to update our manual installed agents from the console (we have this working in our test environment). In my next blog I will create a management pack which will update your (manual) installed agents to CU4. Just make sure you got the framework up and running If you fail to do so just write a comment and I will try to answer your questions.

SCOM’s main benefit of monitoring imho is it’s ability to discover what is running on a server and based on that information start to monitor the server with the appropriate rules. When you follow Microsoft’s best practices you’ll first perform a lightweight discovery to create a very basic class and have the more heavy discoveries run against that basic class. This is pretty good stuff actually. it helps quite a lot for the performance of an agent as it will only run heavy discoveries if the server has an application role and never run on servers which have nothing to do with that application.

However, I’ve recently found out a drawback with this 2 step discovery, which I can probably explain the best with a real world example:

Discover the windows domain controllers on “windows computers” (the management pack from where this discovery runs in is an exception. usually it’s in the application mp itself; apparently MS thought of domain controllers being basic info. similar discoveries for workstation and member servers can be found in this mp as well). For this discovery a wmi query is used to determine if the “windows computer” is a domain controller as well (SELECT NumberOfProcessors FROM Win32_ComputerSystem WHERE DomainRole > 3; if this returns something, it’s a dc)

When it is a “windows domain controller” it will run a few other discoveries to determine more info.

Just by looking at the classes you can imagine it’s not really lightweight anymore.

So far so good, on all my windows computer I run a simple query and if that query returns something SCOM will also run a script that founds more interesting stuff about the DC.

But here’s the catch with this kind of discovery. Suppose I don’t need a certain DC anymore, but I still need to keep the server as it’s running some application I still need to use and monitor. What will happen? The lightweight discovery will do its job. It will correctly determine that the server is not a “windows domain controller” anymore and as a result it won’t run the script-discovery anymore.

You might ask, why is that bad, we didn’t want that, did we? Yes you are correct, we didn’t want to run this discovery against servers that aren’t DC’s, but SCOM doesn’t unlearn the discovered classes automatically. Because this discovery never runs again SCOM never unlearns this server doesn’t have the “Active Directory Domain Controller Computer Role” anymore. And this is the class that is used for targetting rules and monitors. So allthough SCOM knows the server isn’t a “windows domain controller” anymore, it still is monitoring the “Active Directory Domain Controller Computer Role”. This will result in quite a lot of noise (script errors, ldap failures, etc).

For now, there’s just a workaround available. You will need to override the 2nd discovery for that particular server. As the first discovery doesn’t include this server as an object of class, you can’t override the discovery for a “specific object of class: Windows Domain Controller”. You’ll need to create a group and include the server object. Then use the override the object discovery “for a group…” and choose the group you’ve just created.

What’s the point of disabling a discovery that didn’t run anyway? Well now you can go to powershell and run the “Remove-DisabledMonitoringObject” cmdlet. This will remove the discovered object classes for this discovery and all of the monitoring attached to those classes.

Discoveries make SCOM stand out from other monitoring tools, but it needs to work both ways. Finding out this took me about 1 day. And that’s just 1 issue with 1 server (DNS was also installed on this server and had the same issue). Loads of servers might change role without me knowing about it and when it’s not being reported to me I’ll just have extra noise in SCOM. I’m just not sure if this can be picked up within SCOM itself or that the “un-discovery” needs to be done by the mp’s logic. For the AD part it needs to be picked up by Microsoft anyway, but if the logic is build in the management pack then it will have an impact on all the custom build mp’s by all you SCOM authors out there.

1. Targeting – If you target the MP to a class, all servers in that class will get the MP (even if all rules in the MP are disabled by default).
2. Scheduling – There is no way to schedule a delivery to a remote server. As soon as the MP is imported and the RMS detects the new configuration, the MP will be distributed.

Both issues can result in very high network traffic, if not taken into account. Now the first one we can slightly control, by targeting at specific classes. The more specific the class the better. Of course we wanted our files to be distributed to all computers The second one is only under control by controlling the time of the MP import. Of course this is not ideal, but till now we had nothing.

What we do is creating a management pack, that will hold a script. In that script, all other files we want to distribute are placed in a comment section at the bottom of the script, after being converted to a hex notation. It looks something like this:

‘<BEGIN_FILE>jamaMaintenanceMode.vbs
’272D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D
’0A272046696C6520202020202020203A206A616D614D61696E
’74206C6F6720656E74727920746F207075742074686520636F
’61696E74656E616E6365206D6F64652E0D0A2720246376735F
’652E7662732C7620312E3420323030392F30382F3331203130
’2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D

At the other side (the agent side) we reconstruct the files again, and place the files on a fixed location. Now we have distributed the files using SCOM! The solution can distribute both text based files as binary files, but binary files will double in size when converted to the hex notation. So distributing a 500K binary file, will end up with a 1MB script to be distributed (and a 2MB script on the agent side, were it is stored in unicode).

So in short it is a nice method for distributing either small files or to a very limited set of targets. Distributing large files to all agents isn’t a good idea with this method (but it will work).

What do you need:

The above files and directory structure.  You should create this directory structure before continuing:

include_binary – This is the location were you need to place your binary files, which need to be distributed.

include_text – This is the location were you need to place your text files, which need to be distributed.

output – This directory is used to store the generated code.

The files you need can be found here!

Now importing this MP’s without editing, results in a management pack doing nothing. So if you want to distribute a file, this is how:

1. You should edit the BuildTargetScript.cmd file for every change in your distribution you want to be delivered:

   

   The script that will generate the files, will check for this version number. If it is not equal to what is found in the registry it will recreate the files.  The STR_TARGET_DIR variable will hold your target directory on the target machines. This variable is used to check if the files can be generated correctly on your local desktop.

2. You should also change the jamaTextDistribution.vbs

   

   These two constants will be combined to create your target directory for the files:

   %SystemDrive%\STR_BASE_DIR\STR_DISTRIBUTION_DIR

   All files will be placed inside the above directory (this combined directory, must be equal to the STR_TARGET_DIR form the BuildTargetScript.cmd file above).

   Secondly you should change were in the registry you would like to store the version number:

   

   You only have to do this once. If the key can not be found, it will be created.

    

3. Finally you have to change the jamaDistribution.Distribution.xml file that will be distributed. Running the BuildTargetScript.cmd file will generate an ouput file in the output directory. The complete contents of that file needs to be inserted into this .xml file. Below you can find the location where inside the .xml file you need to paste the contents of the output file:

    

4. Now import the two management packs and your file(s) will be distributed.  NOTE: Default the script will run every hour, but if all files are present, the impact is minimal.

What’s next?

Well since we can now get any file at the agent side we can start building a complete file distribution system (which we will ) and after that we can finally automate the update of our manual installed agents and fully automate the manual installation!

In order for this monitor to work you need to have a static record in WINS which doesn’t exist in DNS (this is not mentioned in the knowledge, but it runs a nslookup, so make sure it doesn’t resolve by using dns) and configure the monitor to lookup this HostName. Default the monitor looks up “PlaceHolder”, so you could just create a Wins record named “PlaceHolder”.

I had done this, but i still received errors and when i ran the nslookup query i did receive a valid response for placeholder, so the wins connector does work.

The knowledge mention something about a debug flag to get some helpful troubleshooting information to solve the issue. The description of the debug events i got:

DNS.TTL.vbs : Starting DNS.TTL.vbs Host:PlaceHolder Server:xxx.xxx.xxx.xxx

DNS.TTL.vbs : Writing Property Bag . State=False ttl1:0 ttl2:0 Authority Flag:

It’s not helpful at all and even worse it’s wrong as well. The vbscript file’s name is ttl.vbs, so look for this in the “system center management” folder. Also the parameters are wrong so manually running it fails as well.

To run the script manually on the dns server open a commandline box and run:

path.to.ttl.vbs>cscript /nologo ttl.vbs <hostname> <dnsserverip> false

<hostname>= static wins entry, which doesnt exist in dns (placeholder)  (The script says to fill in fqdn, but this is incorrect as well).

<dnsserverip> = listening ip address of the dns server 

bolean = debug flag. When you set this to true you get the worthless debug information in scom, so keep it on false

When you save the output to an xml file and open it you’ll get something like this.

<Collection>

  <DataItem type=”System.PropertyBagData” time=”2010-08-25T17:18:49.1162249+02:00” sourceHealthServiceId=”BA0AF2AD-5058-0DA0-D5D0-BF3CDD878B88“>

    <ConversionType>StateData</ConversionType>

    <Property Name=”state” VariantType=”8“>ERROR</Property>

  </DataItem>

</Collection>

i’ve modified the script so it will run show the stdout for the nslookup and show a line that it has exited the regex compare function (which it shouldn’t when it functions ok).

Just save the script below to a temp location and run it from there. When you run this vbs and all goes well you should just get this as output:

std_output:

————
Got answer:
    HEADER:
        opcode = QUERY, id = 1, rcode = NOERROR
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 1,  authority records = 0,  additional = 0

    QUESTIONS:
        xxxxxxxxxxx, type = PTR, class = IN
    ANSWERS:
    ->  xxxxxxxxxxxxxxxxx

        name = xxxxxxxxxxxxxxxxxx
        ttl = 168 (2 mins 48 secs)

————
Server:  xxxxxxxxxxxxxxxxxx
Address:  xxxxxxxxxxxxxxxxxx

————
Got answer:
    HEADER:
        opcode = QUERY, id = 2, rcode = NOERROR
        header flags:  response, auth. answer, want recursion, recursion avail.
        questions = 1,  answers = 1,  authority records = 0,  additional = 0

    QUESTIONS:
        xxxxxxxxxxxxxx, type = A, class = IN
    ANSWERS:
    ->  xxxxxxxxxxxx

        internet address = xxxxxxxxxxxx
        ttl = 1200 (20 mins)

————
Name:    xxxxxxxxxxxxxxx
Address:  xxxxxxxxxxxx

When it fails it will log a line before this output:

exit function at regex: 16

This means the WINS_LOOKUP_REGEX array fails at  “”^\s*ttl = “,_” or the line after that.

I couldn’t be bothered to figure out the exact regular expression mismatch, rewrite the WINS_LOOKUP_REGEX array, disable the monitor and create a new one with a new script. I’ve just disabled this monitor as it’s just gives me incorrect information.

Modified script:

'
' Microsoft Corporation
' Copyright (c) Microsoft Corporation. All rights reserved.
'
' ttl.vbs
'
' Determine if a wins connector is healthy.
'
' Parameters -
'                       TargetComputer	The FQDN of the computer targeted by the script.
'			Server - the listening ip
'                      	DebugFlag          True / False

Option Explicit

SetLocale("en-us")

Const DNS_TRACEEVENTNUMBER	= 1125
Const DNS_SCRIPTNAME = "DNS.TTL.vbs"
Const SCOM_ERROR=1
Const SCOM_WARNING=2
Const SCOM_INFORMATIONAL=4
Const SCOM_PB_STATEDATA = 3
Const NSLOOKUP_PATH = "%SystemRoot%\system32\nslookup.exe" 

Dim ImagePath, oWMI, rc, oArgs, oAPI, oDiscoveryData, oInst, SourceID, ManagedEntityId, TargetComputer, OSVersion, oDebugFlag
Dim TTL1 , TTL2, AuthorityFlag
dim host,server,bolDebug
dim objAPI  ,boolWins , oPropertyBag
Dim  sCommand,iErrCode, sOutput, sError,m_sNetshPath ,aSubMatches,  oShell, objArgs

Dim WINS_LOOKUP_REGEX

WINS_LOOKUP_REGEX = Array( _
                                 ".*\r\n",_
                                 ".*\r\n",_
                                 ".*\r\n",_
                                 ".*\r\n",_
                                 ".*\r\n",_
			      "^[\s]*questions = [0-9]*,",_
			      "^[\s]*answers = ",_
			      "[0-9]*,",_
			      "^[\s]*authority records = ",_
			      "[0-9]*",_
                                 ".*\r\n",_
                                 ".*\r\n",_
                                 ".*\r\n",_
                                 ".*\r\n",_
                                 ".*\r\n",_
                                 ".*\r\n",_
                                 "^\s*ttl = ",_
                                 "[0-9]*.*" )         

'***************
'
' start here.
'
'***************
On Error Resume Next

Set objAPI = CreateObject("MOM.ScriptAPI")
If Err.Number <> 0 Then
  Wscript.Quit
end if
Set oPropertyBag = objAPI.CreateTypedPropertyBag(3)
If Err.Number <> 0 Then
  ThrowErrorAndExit "CreateStateDataTypedPropertyBag failed. code = " & Err.Number
end if

Set objArgs = WScript.Arguments
If objArgs.Count <> 3 Then
    Call objAPI.LogScriptEvent( DNS_SCRIPTNAME & " <host>  <computername>  [debug [true | false]")
    wscript.Quit
End If 

host = objArgs(0)
server = objArgs(1)
bolDebug= objArgs(2)

Set oShell = CreateObject("WScript.Shell")
boolWins=cbool(false)
TTL1=0
TTL2=0

Set oShell = CreateObject("WScript.Shell")

trace "Starting " & DNS_SCRIPTNAME & " Host:" & host  & " Server:" & server 

sOutput=ExecuteCmd("-debug -querytype=a " +host + " "+ server ,NSLOOKUP_PATH  ,true)
If  LCase(sOutput) <> "error" Then

	If   GetSubMatches( WINS_LOOKUP_REGEX, sOutput, sOutput, aSubMatches) Then

		TTL1=cint(aSubMatches(8) )
		AuthorityFlag=cint(aSubMatches(4) )

    end if
end if

if 	AuthorityFlag=1 then
	Wscript.sleep (1020)
	sOutput=ExecuteCmd("-debug -querytype=a " +host + " "+ server ,NSLOOKUP_PATH  ,true)
	If  LCase(sOutput) <> "error" Then

		If   GetSubMatches( WINS_LOOKUP_REGEX, sOutput, sOutput, aSubMatches) Then

			TTL2=cint(aSubMatches(8) )
			if ttl1>ttl2 then
				 boolWins=cbool(true)
			end if
		end if
    end if
end if	

wscript.echo ""
wscript.echo "std_output:"
wscript.echo sOutput 

trace "Writing Property Bag . State=" & cstr(boolWins) & " ttl1:" & cstr(ttl1) & " ttl2:" & cstr(ttl2) & " Authority Flag:" & cstr(	AuthorityFlag)

if boolWins=true then
	oPropertyBag.AddValue "state", "OK"
else
	oPropertyBag.AddValue "state", "ERROR"
end if
objAPI.AddItem(oPropertyBag)
If Err.Number <> 0 Then ThrowErrorAndExit "Error adding state data to property bag. code = " & Err.Number
objAPI.ReturnItems
If Err.Number <> 0 Then ThrowErrorAndExit "Error returning property bag data. code = " & Err.Number  

Set objAPI = Nothing
Set oPropertyBag = Nothing

Wscript.Quit

'*******************************************************

Sub ThrowErrorAndExit(Message)

   Err.Clear
   Call oAPI.LogScriptEvent(DNS_SCRIPTNAME, DNS_TRACEEVENTNUMBER, SCOM_ERROR, Message)
   WScript.Quit

End Sub

Sub Trace(Message)
   If (bolDebug) Then
      Call objAPI.LogScriptEvent(DNS_SCRIPTNAME, DNS_TRACEEVENTNUMBER, SCOM_INFORMATIONAL, Message)
   End If

End Sub

Function GetSubMatches(ByVal aRegexes, ByVal sText, ByRef sRemainingText, ByRef aCapturedSubMatches)
  Dim oRegex
  Set oRegex = New RegExp
  oRegex.Global = False

  Dim oMatches
  Dim oMatch
  Dim sPattern
  Dim aSubMatches()
  aCapturedSubMatches = aSubMatches

  GetSubMatches = False

  Dim i
  Dim lSubMatchCount

  lSubMatchCount = 0
  sRemainingText = sText

  dim intCount
  intCount = 0

  For i = 0 To UBound(aRegexes)
    sPattern = aRegexes(i)
    oRegex.Pattern = "^" & sPattern
    Set oMatches = oRegex.Execute(sRemainingText)
    if oMatches.Count = 0 then
        wscript.echo "exit function at regex: " & intCount
    end if
    If oMatches.Count <> 1 Then
      sRemainingText = sText
      Exit Function
    End If

    Set oMatch = oMatches(0)
    sRemainingText = Mid(sRemainingText, oMatch.Length + 1)

    ' save output If odd line, or only line.

    If i Mod 2 = 1 Then
      lSubMatchCount = lSubMatchCount + 1
      ReDim Preserve aSubMatches(lSubMatchCount - 1)
      aSubMatches(lSubMatchCount - 1) = oMatch.Value
    elseIf UBound(aRegexes)=0 Then
      lSubMatchCount = lSubMatchCount + 1
      ReDim Preserve aSubMatches(lSubMatchCount - 1)
      aSubMatches(lSubMatchCount - 1) = oMatch.Value
    End If
    intCount = intCount + 1
  Next

  GetSubMatches = True
  aCapturedSubMatches = aSubMatches

End Function

Function ExecuteCmd(strOptionToUse, strCmdToUse, boolReadOutput)
Dim ncControlcommand
Dim oShell
Dim curDir
Dim strExecOut

Set oShell = CreateObject("WScript.Shell")
curDir = oShell.CurrentDirectory
ncControlcommand =  "cmd.exe /C """ & QuoteWrap(strCmdToUse) & " " & strOptionToUse & " " &"""" 

IF boolReadOutput Then
    strExecOut = RunCmd(ncControlcommand,true)
Else
    strExecOut = RunCmd(ncControlcommand,false)
End If
ExecuteCmd = strExecOut
End Function

Function RunCmd(CmdString, boolGetOutPut)
    Dim wshshell
    Dim oExec
    Dim output
    Dim strOutPut

    Set wshshell = CreateObject("WScript.Shell")
    Set oExec = wshshell.Exec(CmdString)
    Set output = oExec.StdOut
    Do While oExec.Status = 0
         WScript.Sleep 100
         if output.AtEndOfStream = false then
            IF boolGetOutPut Then
                    strOutPut = strOutPut & output.ReadAll
                End IF
         else
              exit Do
         End If
    Loop
    IF boolGetOutPut Then
        strOutPut = strOutPut & output.ReadAll
    Else
        strOutPut = "1"
    End IF    

    If oExec.ExitCode <> 0 Then
         strOutPut = "Error"

    End If

    Set wshshell = Nothing
    RunCmd = strOutPut
End Function

Function QuoteWrap(myString)
      If (myString <> "") And (left(mySTring,1) <> Chr(34)) And (Right(myString,1) <> Chr(34)) Then
            QuoteWrap = Chr(34) & myString & Chr(34)
      Else
            QuoteWrap = myString
      End If
End Function

Function IsValidObject(ByVal oObject)
  IsValidObject = False

  If IsObject(oObject) Then
    If Not oObject Is Nothing Then
      IsValidObject = True
    End If
  End If
End Function

Heartbeat mechanisme: An agent sends out a heartbeat on the monitoring port 5723 to a management servers. The management server (rms actually) keeps track of the last received heartbeat of each agent in its management group. When an agent’s heartbeat hasn’t been received for X time alert “heartbeat failure” alert is generated. The heartbeat failure alert in turn triggers a ping on the agent-managed server’s FQDN. When this fails as well another alert will be generated, the “server unreachable” alert.

This blog is about the “X” time to wait before a heartbeat failure alert will be generated. X isn’t actually a direct configurable time. It’s a combination of 2 settings. The agent setting “the heartbeat interval” in seconds and the server setting “number of missed heartbeats allowed”. The default for this setting is 60 seconds and 3 missed heartbeats allowed. So “X” is default 60×3 = 3 minutes.

I assume the default is ok in most cases, but if you have a large and complex network or reboot servers without setting maintenance it might generate a lot of noise as well. So here’s what we did to find the optimal setting for X without starting to experiment with settings themselves.

By running the query below you can see the number of alerts for “Health Service HeartBeat Failure” and “Failed to Connect to Computer” with a closed status.

Select alertstringname, count(*) as Number_of_alerts from AlertView
where ResolutionState = 255
and (alertstringname = ‘Health Service HeartBeat Failure’ or alertstringname = ‘Failed to Connect to Computer’)
group by AlertStringNAme
order by 2 DESC

When you haven’t changed retention for closed alerts it gives the number for about 1 week. For our environment it turned out to be around 3000 heartbeat alerts, which is about a weekly alert for each server. However most of these alerts are gone before someone looked at the “problem”.

In this post i’ve given some queries to identify the auto closing alerts already. I modified the query a bit to only see heartbeat failure and failed to connect to computer alerts.

By running the query below you can see the number of heartbeat failure alerts which were closed within 2 minutes after creation.

Select alertstringname, count(*) as Number_of_alerts from AlertView
where ResolutionState = 255
and ResolvedBy =’system’
and (alertstringname = ‘Health Service HeartBeat Failure’ or alertstringname = ‘Failed to Connect to Computer’)
and DATEDIFF(MI,TimeRaised,TimeResolved) <= 2
group by AlertStringNAme
order by 2 DESC

The result of this query in my environment was 1450 alerts were auto-closed within the first 2 minutes. So if X would have been 5, it probably would have prevented 1450 alerts.

I’ve plotted the X versus the expected number of heartbeat failures. Please note i left out quite a lot of values for X, but i haven’t adjusted the scale for this and after 1 hour i still would get a few heartbeat failures.

So what’s the optimal X for this environment?

Actually you still can’t say what the setting should be. It still depends on what is acceptable for your environment as we’re talking about how fast you can detect whether a server is down or not. Setting X to 60 would give us the least of heartbeats, but it wouldn’t make any sense either. I believe finding a balance between noise and when we have to take a look is more important, i’d say the optimal X for my environment is 7-8. This will leave about 800 heartbeats alerts weekly, but this is acceptable for us.

Also note you might miss unexpected reboots whatever the value for X is. If it’s important not to miss them, just pick up the event about unexpected reboot from the system eventlog by an alert rule and make that alert critical.