JAMA00

SCOM 2007 R2

Archive for the ‘Agent’ Category

State changes from disabled monitors

Posted by MarcKlaver on May 18, 2011

While we try to reduce the number of state changes, we stumbled into a bug in the agent software. We were investigating our top 50 of state change monitors, using Kevin Holman’s queries.

When we looked at the results, we did see some strange things. First, the number of state changes were equal for a lot of monitors.

An example of this:

image

Having exact the same number of state changes for two different monitors? When we looked into these monitors, they were disabled by default in the management pack were the monitor was defined. Digging further showed that there was no override present in the system, which would enable this monitor.

We now had a problem while from our top 50, 48 of them were monitors that were default disabled, without any override to enable the monitor.

This turns out to be a bug in the agent software. The moment the agent (re-) initializes either by starting or coming out of maintenance mode it will detect the monitor and initialize it. When realizing it should (default) disable the monitor, it will send a state change for the the disabled monitor.

This is also the case for monitors that are default enabled, but are disabled using a custom override. Unfortunately a fix for this issue is not as easy as it sounds (according to Microsoft support) and a fix will not be realized in the R2 version of SCOM.

So if you have tuned a lot (like us) and find these monitors, just skip them. You can’t fix this one :)

Below is a list of monitors we found that were default disabled during this investigation and that we now exclude from the query.

Microsoft.Windows.Server.2003.LogicalDisk.AvgDiskSecPerWrite
Microsoft.Windows.Server.2003.LogicalDisk.AvgDiskSecPerRead
Microsoft.Windows.Server.2003.NetworkAdapter.NetworkAdapterConnectionHealth
Microsoft.Windows.Server.2008.LogicalDisk.AvgDiskSecPerWrite
Microsoft.Windows.Server.2008.LogicalDisk.AvgDiskSecPerRead
Microsoft.SystemCenter.Ping
Microsoft.SystemCenter.AgentManagement.EndToEndEventMonitorError
Microsoft.SystemCenter.HealthService.ConfigurationStateWarningLevel
Microsoft.SystemCenter.HealthService.ConfigurationProcessing
Microsoft.SystemCenter.HealthService.Security.DataIntegrityCheck
Microsoft.SystemCenter.HealthService.ConfigurationStateCriticalLevel
Microsoft.SystemCenter.AgentManagement.EndToEndEventMonitorWarning
SMSv4_dependent_service_running__Background_Intelligent_Transfer_Service_16_Rule.AdvancedAlertCriteriaMonitor
SMS_2003_dependent_service_running__Background_Intelligent_Transfer_Service_13_Rule.AdvancedAlertCriteriaMonitor
Microsoft.SQLServer.2005.DBFile.DiskFreeSpace
Microsoft.SQLServer.2005.Database.Configuration.RecoveryModel
Microsoft.SQLServer.2005.Database.TransactionLogSizeMegabytesMonitor
Microsoft.SQLServer.2005.Database.Configuration.TrustWorthy
Microsoft.SQLServer.2005.Database.Configuration.AutoUpdateStatAsync
Microsoft.SQLServer.2005.Database.Configuration.TornPageDetection
Microsoft.SQLServer.2005.Database.DBSizePercentMonitor
Microsoft.SQLServer.2005.Database.Configuration.AutoCreateStat
Microsoft.SQLServer.2005.Database.Configuration.DBChaining
Microsoft.SQLServer.2005.Database.TransactionLogSizePercentMonitor
Microsoft.SQLServer.2005.Database.DBSizeMegabytesMonitor
Microsoft.SQLServer.2005.Database.Configuration.AutoUpdateSet
Microsoft.SQLServer.2005.Database.Configuration.AutoShrink
Microsoft.SQLServer.2005.Database.DBSizePercentageChangeMonitor
Microsoft.SQLServer.2005.Database.Configuration.AutoClose
Microsoft.Windows.Server.2003.TerminalServerRole.InactiveSessions
Microsoft.Windows.Server.2003.TerminalServerRole.ActiveSessions
Microsoft.Windows.Server.2003.TerminalServerRole.CPUPerSession
Microsoft.Windows.InternetInformationServices.2003.WebSite.WebServiceCurrentISAPIExtensionRequests.Monitor
Microsoft.Windows.InternetInformationServices.2003.WebSite.WebServiceCurrentConnections.Monitor
Microsoft.Windows.InternetInformationServices.2003.WebSite.WebServiceBytesTotalSec.Monitor
Microsoft.Windows.InternetInformationServices.2003.WebSite.WebServiceBytesSentSec.Monitor
Microsoft.Windows.InternetInformationServices.2003.WebSite.WebServiceBytesReceivedSec.Monitor
Microsoft.Windows.InternetInformationServices.2003.WebSite.WebServiceISAPIExtensionRequestsSec.Monitor
Microsoft.Windows.InternetInformationServices.2003.WebServer.WebServiceISAPIExtensionRequestsSec.Monitor
Microsoft.Windows.InternetInformationServices.2003.WebServer.ASP.NETRequestsQueued.Monitor
Microsoft.Windows.InternetInformationServices.2003.WebServer.ASP.NETRequestsCurrent.Monitor
Microsoft.Windows.InternetInformationServices.2003.WebServer.ASP.NETWorkerProcessRestarts.Monitor
Microsoft.Windows.InternetInformationServices.2003.WebServer.WebServiceBytesReceivedSec.Monitor

 

If you want to exclude any (default) disabled monitor that you found, exclude it in the query as shown below:

use OperationsManager
go

select distinct top 50 count(sce.StateId) as NumStateChanges, m.MonitorName, mt.typename AS TargetClass
from StateChangeEvent sce with (nolock)
join state s with (nolock) on sce.StateId = s.StateId
join monitor m with (nolock) on s.MonitorId = m.MonitorId
join managedtype mt with (nolock) on m.TargetManagedEntityType = mt.ManagedTypeId
where m.IsUnitMonitor = 1
and m.MonitorName not in (

‘Microsoft.Windows.Server.2003.LogicalDisk.AvgDiskSecPerWrite’,

‘Microsoft.Windows.Server.2003.LogicalDisk.AvgDiskSecPerRead’

)
group by m.MonitorName,mt.typename
order by NumStateChanges desc

Posted in Agent | Leave a Comment »

Updating manual installed agents from the console

Posted by MarcKlaver on May 9, 2011

We have created a management pack, that will update the manual installed agents, from a task in the operations console. However, before you can use this management pack you should have implemented the JDF framework for file distribution. This management pack depends on the framework, so if you are not capable of setting up the framework, this management pack is useless to you.

 

Below is the final XML file for your management pack called jamaAgent.Update.xml:

<?xml version="1.0" encoding="utf-8"?><ManagementPack ContentReadable="true" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <Manifest>
    <Identity>
      <ID>jamaAgent.Update</ID>
      <Version>0.4.0.0</Version>
    </Identity>
    <Name>jamaAgent.Update</Name>
    <References>
      <Reference Alias="MSWL">
        <ID>Microsoft.Windows.Library</ID>
        <Version>6.1.7221.0</Version>
        <PublicKeyToken>31bf3856ad364e35</PublicKeyToken>
      </Reference>
      <Reference Alias="MSSCL">
        <ID>Microsoft.SystemCenter.Library</ID>
        <Version>6.1.7221.0</Version>
        <PublicKeyToken>31bf3856ad364e35</PublicKeyToken>
      </Reference>
    </References>
  </Manifest>
  <Monitoring>
    <Tasks>
      <Task ID="jamaAgent.Update.ConsoleTask.AgentUpdate" Accessibility="Public" Enabled="true" Target="MSSCL!Microsoft.SystemCenter.HealthService" Timeout="300" Remotable="true">
        <Category>Custom</Category>
        <WriteAction ID="PA" TypeID="MSWL!Microsoft.Windows.ScriptWriteAction">
          <ScriptName>jamaTaskUpdateAgent.vbs</ScriptName>
          <Arguments />
          <ScriptBody><![CDATA[
‘——————————————————————————-
‘ File   : jamaTaskUpdateAgent.vbs
‘ Use    : Script for the jama Agent Update task.
‘ SVN    : Revision: 136
‘          Date: 2011-04-12 09:14:33 +0200 (Tue, 12 Apr 2011)

‘ Note(s): 1) —
‘——————————————————————————-
option explicit
on error goto 0
setlocale("en-us")

const INT_RETRIES           = 5
const INT_DEFAULT_WAIT_TIME = 120
const STR_DOWNLOAD_PATH     = "/files/opsmgr/updates/agent/"

jamaMain()

‘——————————————————————————-
‘ jamaMain

‘ Use    : Main entry for this script.
‘ Input  : —
‘ Returns: —
‘ Note(s): 1) —
‘——————————————————————————-
function jamaMain()
    dim jdf
    dim iMinutesToWait
    dim bForceUpdate

    if(not jdfLoadFramework(jdf, null, null, null, null, null)) then
        wscript.echo "The Jama Distribution Framework could not be loaded. No update is performed."
    else
        wscript.echo "Jama Distribution Framework loaded" & vbNewLine & vbNewLine & jdf.GetInfoString(wscript.fullname)
        ‘—————————————————————————
        ‘ Your code here!
        ‘—————————————————————————
        if(jamaInitialize(iMinutesToWait, bForceUpdate)) then
            if((bForceUpdate = false) and jamaRestartRequired()) then
                wscript.echo "Update could not be scheduled due to dependent services." & vbNewLine & _
                             "Use the force option to override this behaviour and force the update to run."
            else
                if(jamaScheduleUpdateIn(jdf, iMinutesToWait) = true) then
                    wscript.echo "Update is scheduled to run in " & iMinutesToWait & " minutes."
                else
                    wscript.echo "Update could not be scheduled. No update will be performed!"
                end if
            end if
        else
            wscript.echo "Initilization failed. No update will be performed!"
        end if
    end if
end function

‘——————————————————————————-
‘ jamaInitialize

‘ Use    : Initialize the script.
‘ Input  : iMinutesToWait – integer – Number of minutes to wait before
‘                                     activating the schedule (output)
‘ Returns: Boolean – TRUE  – No errors detected.
‘                    FALSE – An error was detected.
‘ Note(s): 1) —
‘——————————————————————————-
function jamaInitialize(byref iMinutesToWait, byref bForceUpdate)
    dim colNamedArguments
    dim bResult
    dim bFound

    bResult = false
    bFound  = false
    bForceUpdate = false
    set colNamedArguments = WScript.Arguments.Named

    if(colNamedArguments.Exists("minutes")) then
        iMinutesToWait = cint(lcase(colNamedArguments.Item("minutes")))
        bFound         = true
        bResult        = true
    end if

    if(colNamedArguments.Exists("force")) then
        if(lcase(colNamedArguments.Item("force")) = "true") then
            bForceUpdate = true
        end if
    end if

    if(not bFound) then
        iMinutesToWait = INT_DEFAULT_WAIT_TIME
        bResult        = true
    end if

    jamaInitialize = bResult
end function

‘——————————————————————————-
‘ jamaRestartRequired

‘ Use    : Checks if an update of the agent will trigger a restart or reboot.
‘ Input  : —
‘ Returns: Boolean – TRUE  – A restart is required.
‘                    FALSE – A restart is not required.
‘ Note(s): 1) If the dependency could not be determined, this function will
‘             return true.
‘          2) For windows 2000 this function will always return true.
‘          3) When true on a 2003 server, a reboot is required but not forced.
‘          4) When true on 2008 or higher, the dependent services could be
‘             restarted by the installer service.

‘——————————————————————————-
function jamaRestartRequired()
    dim strCmd
    dim iResult
    dim bResult
    dim fso
    dim fh
    dim strTempFile
    dim strLine

    strTempFile = "jamaTaskUpdateAgent.$$$"
    bResult = true
    strCmd  = "tasklist /fo csv /m EventCommon.dll /FI ""imagename ne HealthService.exe"" /FI ""imagename ne MonitoringHost.exe"" > " & strTempFile
    jamaRun(strCmd)

    set fso = CreateObject("scripting.filesystemobject")
    if(fso.FileExists(strTempFile)) then
        on error resume next
            err.clear
            set fh = fso.OpenTextFile(strTempFile, 1)                           ‘ Open for reading.
            if(err.number = 0) then
                do while(not fh.AtEndOfStream)
                    strLine = lcase(fh.ReadLine)
                    if(instr(strLine, "info: no tasks") > 0) then               ‘ No dependencies found.
                        bResult = false
                    end if
                loop

                set fh = fso.GetFile(strTempFile)
                fh.delete(true)                                                 ‘ delete file.
            end if
        on error goto 0
    end if

    jamaRestartRequired = bResult
end function

function jamaRun(byval strCmd)
    dim objShell
    dim iResult

    iResult      = 0
    set objShell = wscript.createObject("wscript.shell")
    iResult      = objShell.run("cmd /c " & strCmd, 0, true) ‘ hidden and wait for result
    set objShell = nothing
    jamaRun      = iResult

    jamaRun = iResult
end function

‘——————————————————————————-
‘ jamaScheduleUpdateIn

‘ Use    : Retrieve and schedule the required update.
‘ Input  : jdf            – object  – JDF object
‘          iMinutesToWait – integer – Minutes to wait for the schedule.
‘ Returns: Boolean – TRUE  – No errors.
‘                    FALSE – An error was detected.
‘ Note(s): 1) —
‘——————————————————————————-
function jamaScheduleUpdateIn(byref jdf, iMinutesToWait)
    dim strSourceFile
    dim strTargetFile
    dim strTargetDir
    dim iCount
    dim iResult
    dim bResult
    dim strCmd
    dim fso

    iCount = 0
    bResult = false
    set fso = CreateObject("scripting.filesystemobject")
    strSourceFile = "jamaAgentUpdate" & jdf.Platform & ".exe"
    strTargetDir  = jdf.ExpandString("$JDF_IN_DIR$" & "jamaAgentUpdate\")
    if(jdf.CreateDirectory(strTargetDir)) then
        strTargetFile = strTargetDir & strSourceFile
        bResult = fso.FileExists(strTargetFile)
        if(not bResult) then
            do
                iResult = jdf.GetFile("jdfBaseDistribution", STR_DOWNLOAD_PATH & strSourceFile, strTargetFile)
                iCount  = iCount + 1
            loop while((iCount < INT_RETRIES) and (iResult <> 0))
        end if
        if(iResult = 0) then
            bResult = fso.FileExists(strTargetFile)
            if(bResult) then
                strCmd  = strTargetFile & " /verysilent"
                bResult = jdf.ScheduleTaskIn(strCmd, iMinutesToWait)
            end if
        end if
    end if
    jamaScheduleUpdateIn = bResult
end function

‘——————————————————————————-
‘ From template.vbs
‘——————————————————————————-
‘——————————————————————————-
‘ jdfLoadFramework

‘ Use    : Load and initialize the JDF framework.
‘ Input  : objJDF               – object – Object passed back with framework.
‘          bInitialize          – bool   – true or null for initializing.
‘          strVersion           – string – Minimum framework version required.
‘          bForceVersion        – bool   – true, false or null.
‘          strCustomerId        – string – Customer id or null.
‘          strDefaultUploadPath – string – Default upload path.
‘ Returns: bool – TRUE  – Initialization of the framework succeeded.
‘                 FALSE – Initialization of the framework failed.
‘ Note(s): 1) strVersion = null
‘                 Any version of the framework will be accepted. The value of
‘                 strForceVersion will be ignored.

‘          2) strVersion = "x.x.x"
‘                 Integer values, seperated by dots, e.g. : "3.5.11"

‘          3) bForceVersion = null/false
‘                 The given version is a minimum version required for the
‘                 framework.

‘          4) bForceVersion = true
‘                 The framework version must exactly match with the given
‘                 version number.

‘          5) strCustomerId = null
‘                 The customer id will be retrieved from the registry. See
‘                 the documentatiohn for more information.

‘          6) strDefaultPath = null
‘                 The default upload path will be set to the root: "/". This
‘                 argument can hold JDF variables (both default and custom
‘                 provided in the jdf.jdp file). See the documentation for more
‘                 information.

‘          7) Normal use is:
‘                 jdfLoadFramework(jdf, null, null, null, null, null)
‘                 where jdf is the object you pass to the function.
‘——————————————————————————-
function jdfLoadFramework(byref objJDF, byval bInitialize, byval strVersion, byval bForceVersion, byval strCustomerId, byval strDefaultUploadPath)
    dim bResult, fso, strFrameworkFile, objReg, strResult

    bResult   = false
    strResult = ""
    on error resume next
        err.clear
        set objReg=GetObject("winmgmts:{impersonationLevel=impersonate}!\\.\root\default:StdRegProv")
        if(err.number = 0) then
            objReg.GetStringValue  &H80000002, "SOFTWARE\Company\jama\jdf", "path", strResult     ‘ If you use another registry location, change this value!
            if((err.number = 0) and (strResult <> "") and (not isnull(strResult))) then           ‘ We got something, so try it.
                strFrameworkFile = strResult & "jdf.wsc"                                          ‘ This now should be the full path to the framework file.
                set fso = CreateObject("Scripting.FileSystemObject")
                if(err.number = 0) then
                    if(fso.FileExists(strFrameworkFile)) then
                        set objJDF = GetObject("script:" & strFrameworkFile)                      ‘ File exist, try to load it.
                        if(err.number = 0) then                                                   ‘ Framework found and object loaded.
                            if(bInitialize = false) then                                          ‘ No initialization.
                                bResult = false
                            else                                                                  ‘ Initialize the framework for use.
                                bResult = objJDF.InitializeFramework(strVersion, bForceVersion, wscript.scriptname, strCustomerId, strDefaultUploadPath)
                            end if
                        end if
                    end if
                end if
            end if
        end if
    on error goto 0
    jdfLoadFramework = bResult
end function
]]></ScriptBody>
          <TimeoutSeconds>300</TimeoutSeconds>
        </WriteAction>
      </Task>
    </Tasks>
  </Monitoring>
  <LanguagePacks>
    <LanguagePack ID="ENU" IsDefault="false">
      <DisplayStrings>
        <DisplayString ElementID="jamaAgent.Update">
          <Name>jamaAgent.Update</Name>
        </DisplayString>
        <DisplayString ElementID="jamaAgent.Update.ConsoleTask.AgentUpdate">
          <Name>jama Agent Update</Name>
        </DisplayString>
      </DisplayStrings>
    </LanguagePack>
  </LanguagePacks>
</ManagementPack>

Now the script in this management pack expects a few things.

 

const STR_DOWNLOAD_PATH = "/files/opsmgr/updates/agent/"

This constant is used to specify the path on the remote SCP server, where the update files can be retrieved. If you use another location, change the value of the variable.

 

strSourceFile = "jamaAgentUpdate" & jdf.Platform & ".exe"

This value is used to generate the name of the update to retrieve. We have re-packaged the two update files required for the agent update into a single executable (see also this article from Kevin Holman for more information). You can download an archive with three packages (for all suported platforms) on this location. If you create you own, just make sure the final name of the update package corresponds with the final strSourceFile variable value:

jamaAgentUpdateIA64.exe –> For the itanium platform
jamaAgentUpdateX64.exe –> For the x64 platform
jamaAgentUpdateX86.exe –> For the x86 platform

 

What we actually do is very simple.

  • We detect the platform we are running on and construct the correct file name of the file to retrieve (this update package contains the required .msp files for the agent).
  • We retrieve the update package.
  • We schedule the update package to run in ## minutes and to run unattended.
  • We hope for the best :)

If something goes wrong with the installation, the failure can be found in the log file created by the update. There are two logfiles:

%TEMP%\jamaAgentUpdate.<platform>.log
%TEMP%\jamaAgentUpdate.ENU.<platform>.log

Also the package itself will again detect we if we are running on the correct platform and only start the update if the package is the correct platform version.

 

As you can see in the .XML file, we use the jdfBaseDistribution account for retrieving the file. So you don’t need to configure the SCP server for every customer you have. Just make the file available for the jdfBaseDistribution account. The task itself is targeted against the health service. Select the “Agent By Version” leaf in the console, to get an overview of all your agents and their version:

image

If you select a health service, the task pane will show you the new update task:

image

Note that the task is not version aware, so it will always be available and will run. So you can do an update over a current installed agent (which does work, without issues). After selecting the task, the tasks pane will be shown and you can change the arguments for the script.

image

If you don’t change anything, the task will try to schedule the update to run in 120 minutes. If you require the another time, use the /minutes:## argument. The task will be scheduled ## minutes in the future. So if I want the update to run within 5 minutes:

image

 

 

The first thing the task does is checking for dependencies (for more information about the dependencies, see this link). If no dependencies are found, the job is scheduled ## minutes in the future (or 120 if no override is given):

image

 

 

If a dependency is found, the tasks will not schedule the update:

image

But as the output shows you can override the dependency detection, by adding the /force:true option in the argument list. If set the update will always run, even if there are dependencies detected.

 

After the update is finished, the agent should reflect the new version information:

image

You now can update your manual installed agent, using the operations console :)

Posted in Agent | Leave a Comment »

 
Follow

Get every new post delivered to your Inbox.