Hostmon and VM - Memory Issue

[jromariz]

Hi,

We are facing some memory issues with hostmon running on a Virtual Machine. We´ve configured a VM with 4Gb of memory. From time to time we need to restart the machine because the memory is exhausted by hostmon. This keeps happening every week or so.

Any ideas?


Thanks for any help.

Regards,

Jromariz.

[KS-Soft]

Could you please provide some information about your system?
- HostMonitor version?
- Windows?
- Service pack?
- Do you use ODBC logging or ODBC test method? If yes, what ODBC driver do you use?
- Do you have installed some antivirus monitors, personal firewall, content monitoring software? Non stanard winsock components?

Regards
Alex

[CX-FR]

I would like to revive this post if ok, otherwise pls copy to a new one.

We have a similar issue. Every week Hostmonitor flips and fails every single test, sending a few hundred mails in the process, more and more often until you reboot server or restart service.

After a month of trying to find the problem, I have these 'facts':
- It happens on a Windows 2008 R2 x64, not on Windows 2003
- Private bytes for the process hostmon.exe.*32 grows. From about 60MB until it starts to act erratically at around 150MB


To answer the questions posted to the original poster:
- Hostmonitor version is 8.68 (problem was there in at least 8.64)¨
- Windows 2008 R2 x64, latest patches and sp's
- ODBC, 32-bit system DSN with SQLSRV32.DLL version 6.01.7600.16385
- SQL 2005 x64(9.0.4053) on remote system
- Kaspersky Admin 8.0.2090 is running on same server
- WSUS 3.0 sp2 as well
- and Spiceworks
- server is a vm(2GB RAM, 2vCPU) running on VMWare vSphere 4.0 u1

My question is, must we go back to Windows 2003? Scheduled service restart? Or is it possible to fix?

All help greatly appreciated,
/Fredrik

[CX-FR]

Oh and maybe worth noting is that we autogenerate a custom HTML-report every minute for a website to display status.
We have 255 tests with a load of 21 tests/sec and the audit is green (except for OS)

[KS-Soft]

ODBC, 32-bit system DSN with SQLSRV32.DLL version 6.01.7600.16385

So you are using ODBC logging or ODBC Query tests? Could you disable them for testing?

Private bytes for the process hostmon.exe.*32 grows. From about 60MB until it starts to act erratically at around 150MB

150MB of memory is not a problem.
When problem appers again, could you check Handles, Threads, GDI and USER objects used by hostmon.exe process? You may use standard Windows Task Manager to check these resources.
What is the total resource usage on the system?

Kaspersky Admin 8.0.2090 is running on same server

Some version of Kaspersky leads to problems. We have the following statistics:
- KAV versions 7.0.0.125 and 8.0.0.454 work fine and do not lead to any problems
- KAV 2010 (version 9.0.0.736) and KAV 2011 (version 11.0.0.232) leads to some problems with test results
- KAV 2011 (version 11.0.1.400) worked without any problems until some database updates around July 15, 2010

Could you try to disable Kaspersky Antivirus for testing?

My question is, must we go back to Windows 2003? Scheduled service restart? Or is it possible to fix?

The most important is to find what exactly tests, actions, function or software leads to the problem. Then we can fix it (if problem caused by HostMonitor itself), disable some option (if problem caused by some Windows bug) or uninstall/update/change 3rd party software (if problem caused by some 3rd party software).

Regards
Alex

[CX-FR]

Thank you for the reply, Alex.

So you are using ODBC logging or ODBC Query tests? Could you disable them for testing?

Done. Set logging to None.

could you check Handles, Threads, GDI and USER objects used by hostmon.exe process?

Will do. I restarted the service yesterday so it will take up to a week for it to fail. I will then get back with results.

What is the total resource usage on the system?

Quite balanced. Never out of resources as far as I have seen.

Could you try to disable Kaspersky Antivirus for testing?

I'm sorry I wasn't very clear on this. There is no antivirus, only the administration software for our Kaspersky solution.
Though it uses a network agent to communicate.

[CX-FR]

This morning Hostmonitor started to stutter again. Please see pictures below and advice.

These are views of Task Manager and Resource Manager.





...and the history of the process' private bytes



Last but not least an interesting view from process explorer showing a svchost process, almost all activity in this process is derived from nsisvc.dll, and always connected to Hostmonitors troubles:

[CX-FR]

After a restart of the hostmon service, the same process looks like this:



So a wild stab in the dark is that Hostmonitor is the culprit here.
What do you think?

[KS-Soft]

We are working on this issue but its very hard to find reason of such problems, it can take months of work without any good result
Especially if problem caused by some 3rd party components (DLL, Performance Counter, etc).
We tried to find some information about problem related to nsisvc.dll - nothing, I assume it works fine.

BTW: We are not sure that memory leak within HostMonitor is the problem in your case. 150-200MB used by HostMonitor should not lead to any problems with tests or RCC connection. There is no leak in Handles, User or GDI objects. No problem with threads...

Could you send your configuration files to us? If we are lucky, we will be able to reproduce the problem then we will disable tests, actions and other options one by one and check when problem disappear. This way we can narrow down search field...


On the other hand, if most of tested servers located in your private network, HostMonitor will not be able to reach them.
Then there is another possible testing method: could you start RMA on the same system where HostMonitor is running and use this agent to perform some test methods? E.g. perform all Performance Counters, WMI, NT Event Log, ODBC Query using agent? Its pretty easy to change agent for set of tests using special View: create View that will display all tests of some kind, select all items, click Edit button and change "Test by" parameter.

If HostMonitor will work fine while RMA will experiance problems, we will know there are something wrong with one of these several test methods. Otherwise, other test methods should be checked.

Actually you can install several RMA into different folders on the same system, using different TCP ports for each agent. Then setup HostMonitor to use different RMA for different kind of tests. E.g. use one RMA to perform WMI and NT Event Log tests, another RMA to perform CPU Usage, Process and Performance Counter tests, use 3rd RMA to perform IMAP, POP3, SMTP, HTTP and URL checks.


Also, could you please setup test to check virtual address space used by HostMonitor?
You may use WMI test and query like Select VirtualSize from Win32_Process where Caption='hostmon.exe'

Regards
Alex