Wildcards and log-files with Text-log method

All questions related to installations, configurations and maintenance of Advanced Host Monitor (including additional tools such as RMA for Windows, RMA Manager, Web Servie, RCC).
Post Reply
User avatar
GijsbertVanOeveren
Posts: 11
Joined: Thu Jan 03, 2008 3:03 am

Wildcards and log-files with Text-log method

Post by GijsbertVanOeveren »

An application I use generates log files that all have the same file name but all have a different extention.
e.g.
  • log.23634
    log.56745
    log.28764
    etc.
These extention-numbers correspond with the id of running processes.

I tried to use wildcards to monitor these logfiles with the Text-Log method:

Code: Select all

File: c:\logfiles\log.*
Look for: String errormessage
The result is that I only get a warning when the errormessage is found in all logfiles instead of after a single occurrence.

Is there a possibility to monitor all files and fire a warning if the criterium is experienced in just 1 of the files?

Kind regards,

Gijsbert
KS-Soft Europe
Posts: 2832
Joined: Tue May 16, 2006 4:41 am
Contact:

Re: Wildcards and log-files with Text-log method

Post by KS-Soft Europe »

GijsbertVanOeveren wrote:The result is that I only get a warning when the errormessage is found in all logfiles instead of after a single occurrence.
In fact, "Text Log" test method does not process all the files. If you specify the wildcards in the name of the file, the 1st matched file will be checked.
Also, "Text Log" test method works in a way, when it warns you only if the string is found in a NEW record. It means that if log file already has "bad' records when you started HostMonitor, you will not receive any alerts; but if any new "bad" record will be added to the log file while HostMonitor is running, you will receive a warning about the new problem.
If you just want to check specific string in any part of the file, you may use "Compare Files" test method, which provides "alert when file contains a specified string" option: http://www.ks-soft.net/hostmon.eng/mfra ... tm#filecmp
GijsbertVanOeveren wrote:Is there a possibility to monitor all files and fire a warning if the criterium is experienced in just 1 of the files?
Lets clarify it a bit, because I do not completely understand your needs. Do you mean, you want to check all files and fire alert when the criterium is experienced in only one file (if two of more files match the criteria we do not fire alert) or when the criterium is experienced in at least one file?

Both solutions might be implemented using "Shell script" test method. You just should create a simple .bat file, that use "for" command to enumerate all necessaryfiles in a folder, and "type filename | find "String errormessage"" commands to check if the file contains the string.
I can assist you developing such a script, if you like. Or you may try to create it by yourself.
http://www.ks-soft.net/hostmon.eng/tests.htm#chkShell
Please note: The following rules must be obeyed when creating a script:
===============================
The script or external program must write to stdout (standard output stream) single result string. This string should contain 3 parts separated by colon (:).

1. First obligatory part - marker "scriptres" tells to HostMonitor or RMA that this string is the result string.

2. Second obligatory part represents the test status, it can take one of the following values (case insensitive):
Status string Comment
Host is alive status means " script executed successfully, target system correctly responds"
No answer script executed successfully, target system does not respond
Unknown status means "test cannot be performed for some reason"
Unknown host use this status when script cannot resolve target host name into IP address
Ok script executed, result satisfies (some) requirements
Bad script executed, result does not satisfy (some) requirements
Bad contents use this status if script found some error in monitored file, web page, etc

3. Third optional part contains Reply value, HostMonitor displays this value in Reply field, writes to log files, uses to displays charts (Log Analyzer), etc. If you want Log Analyzer to process Reply values correctly and display charts, use one of the following formats for Reply value:
* decimal_number (like "123", "0", "6456.45". as decimal separator use symbol that is specified on your system, usually a dot (.) or a comma (,))
* decimal_number + space + "Kb" (like "512 Kb", "64 Kb")
* decimal_number + space + "Mb" (like "1024 Mb", "5 Mb")
* decimal_number + space + "Gb" (like "12 Gb", "4 Gb")
* decimal_number + space + "%" (like "50 %", "99 %")
* decimal_number + space + "ms" (like "100 ms", "5400 ms")

Several examples:
String printed by script "Status" of the test "Reply" field of the test
scriptres:Host is alive:1000 ms Host is alive 1000 ms
scriptres:Unknown host: Unknown host
scriptres:Ok:300 Kb Ok 300 Kb
scriptres:Bad:90 % Bad 90 %

That's it. Everything else is entirely up to you
===============================

Regards,
Max
User avatar
GijsbertVanOeveren
Posts: 11
Joined: Thu Jan 03, 2008 3:03 am

Thanks!

Post by GijsbertVanOeveren »

Wow Max,

Thank you for this very elaborate answer!
Indeed I ment the situation when then criterium is experienced in at least one of the files checked.

I am going to work this out and will let you know about the results.

Kind regards,

Gijsbert
User avatar
GijsbertVanOeveren
Posts: 11
Joined: Thu Jan 03, 2008 3:03 am

Batch files

Post by GijsbertVanOeveren »

Hi Max,

I have managed to create batch files to check the appropriate files.
Below is a transcript of the two files.
These scripts are the manual versions with 'help' info and other parts that will be stripped when used in the HostMonitor of course.

file_runcheck.bat

Code: Select all

@echo off

IF "%1"=="" GOTO HELP
IF "%2"=="" GOTO HELP

SET VAR1=
SET TMPVAR=

FOR %%f IN (%1) DO FOR /F "tokens=1 delims= " %%e in ('type %%f ^| find "%2"') DO CALL file_processcheck.bat %%f 

IF "%VAR1%"=="" GOTO NONEFOUND

ECHO ScriptRes:Bad:Errors found in files: %VAR1%
SET VAR1=
SET TMPVAR=
GOTO END

:NONEFOUND
ECHO No instances found
GOTO END

:HELP
ECHO Usage: file_runcheck {filepath to search, may include wildcard} {word to search for}
GOTO END

:END


file_processcheck.bat

Code: Select all

@echo off

IF "%TMPVAR%"=="" GOTO GETFIRST
IF "%1"=="%TMPVAR%" GOTO GETNEXT
GOTO :GETSECOND

:GETFIRST
SET VAR1=%1
GOTO :GETNEXT

:GETSECOND
SET VAR1=%VAR1%, %1
GOTO :GETNEXT

:GETNEXT
SET TMPVAR=%1
You mention though that the script should write to the 'stdout', which to my knowledge is a unix of linux thing. How is that on a Windows system? Just an echo from the script?

One more question about the 'TextLog' test method: you mention that it checks the first matched file when using wildcards. What will it see as 'first matched': oldest or newest or maybe another criterium?

Thanks for your time and input!

Kind regards,

Gijsbert
KS-Soft Europe
Posts: 2832
Joined: Tue May 16, 2006 4:41 am
Contact:

Re: Batch files

Post by KS-Soft Europe »

GijsbertVanOeveren wrote:I have managed to create batch files to check the appropriate files.
Below is a transcript of the two files.
Great job! :-)
GijsbertVanOeveren wrote:You mention though that the script should write to the 'stdout', which to my knowledge is a unix of linux thing. How is that on a Windows system? Just an echo from the script?
Correct. You should use "echo" command. You have properly write the following command: ECHO ScriptRes:Bad:Errors found in files: %VAR1%
However, to make HostMonitor handle the result of scripts properly, you should change the bottom part of the file_runcheck.bat script in the following way:

Code: Select all

...

:NONEFOUND 
ECHO ScriptRes:Ok:No instances found 
GOTO END 

:HELP 
ECHO ScriptRes:Unknown:Usage: file_runcheck {filepath to search, may include wildcard} {word to search for} 
GOTO END 

:END 
Script should cover different statuses, at least "Bad" and "Good".

There are several script examples, that might be useful for you:
http://www.ks-soft.net/cgi-bin/phpBB/vi ... php?t=3436
http://www.ks-soft.net/cgi-bin/phpBB/vi ... php?t=3532
http://www.ks-soft.net/cgi-bin/phpBB/vi ... php?t=4139
http://www.ks-soft.net/cgi-bin/phpBB/vi ... php?t=4372
GijsbertVanOeveren wrote:One more question about the 'TextLog' test method: you mention that it checks the first matched file when using wildcards. What will it see as 'first matched': oldest or newest or maybe another criterium?
In most cases it uses the oldest file.

Regards,
Max
User avatar
GijsbertVanOeveren
Posts: 11
Joined: Thu Jan 03, 2008 3:03 am

Integrated file

Post by GijsbertVanOeveren »

Hi Max,

Since I wanted to have the two batchfiles in just one, I had to change the code. In HostMonitor I could not find how to have one batch script cal another one, so I made one back script out of it.

Code: Select all

@echo off

:--------------------------------------------------------
:: Jump to Help if not all parameters have been given
	IF "%1"=="" GOTO HELP
	IF "%2"=="" GOTO HELP
:--------------------------------------------------------


:--------------------------------------------------------
:: Initiate variables
:--------------------------------------------------------
	SET ConcatFiles=
	SET ProcessFile=
	SET PreviousFile=
	SET ProcessingResult=
:--------------------------------------------------------


:--------------------------------------------------------
:: Loop through the files (%1) and search for the string (%2)
:--------------------------------------------------------
	FOR %%f IN (%1) DO FOR /F "tokens=1 delims= " %%e IN ('type %%f ^| find "%2"') DO (
		SET ProcessFile=%%f
		CALL :CHECKITEMS
	)
:--------------------------------------------------------


:--------------------------------------------------------
:: Looping is done, now jump to build the report
:--------------------------------------------------------
	GOTO :BUILDREPORT
:--------------------------------------------------------


:--------------------------------------------------------
:: This section is called from the loop
:--------------------------------------------------------
	:CHECKITEMS

		IF "%PreviousFile%"=="" GOTO GETFIRST
		IF "%ProcessFile%"=="%PreviousFile%" GOTO TRYNEXT
		GOTO :GETSECOND

		:GETFIRST
		SET ConcatFiles=%ProcessFile%
		GOTO :TRYNEXT

		:GETSECOND
		SET ConcatFiles=%ConcatFiles%, %ProcessFile%
		GOTO :TRYNEXT

		:TRYNEXT
		SET PreviousFile=%ProcessFile%

	GOTO :EOF	
:--------------------------------------------------------


:--------------------------------------------------------
:: Build the report
	:BUILDREPORT

	IF "%ConcatFiles%"=="" (
		SET ProcessingResult=Ok:No instances found
	) ELSE (
		SET ProcessingResult=Bad:Errors found in files: %ConcatFiles%
	)

	ECHO ScriptRes:%ProcessingResult%
:--------------------------------------------------------


:--------------------------------------------------------
:: Clear the variables that have been used
	SET ConcatFiles=
	SET ProcessFile=
	SET PreviousFile=
	SET ProcessingResult=
	GOTO END
:--------------------------------------------------------


:--------------------------------------------------------
:HELP
	ECHO Usage: integrated_file_runcheck {filepath to search, may include wildcard} {word to search for}
	GOTO END
:--------------------------------------------------------


:--------------------------------------------------------
:END
:--------------------------------------------------------
KS-Soft Europe
Posts: 2832
Joined: Tue May 16, 2006 4:41 am
Contact:

Re: Integrated file

Post by KS-Soft Europe »

GijsbertVanOeveren wrote:Since I wanted to have the two batchfiles in just one, I had to change the code. In HostMonitor I could not find how to have one batch script cal another one, so I made one back script out of it.
You are right. HostMonitor is unable to invoke the script, that is stored in "Script Manager", from another script. Of course, it is possible to call certain script from filesystem directly (e.g. "call C:\temp\some_script.bat"), but it in not pretty convenient and not much flexible.

So, what can I say... Well done! :-)
Except one thing. I would suggest to add "Unknown" status to the :Help section, like the following:

Code: Select all

:HELP 
ECHO ScriptRes:Unknown:Usage: file_runcheck {filepath to search, may include wildcard} {word to search for}
GOTO END  
Thank you for your time and contribution. :-)

Regards,
Max
User avatar
GijsbertVanOeveren
Posts: 11
Joined: Thu Jan 03, 2008 3:03 am

Even more complete

Post by GijsbertVanOeveren »

Hi Max,

I implemented your suggestion.
And to make it even better I added two more things:
- the possibility to search for a string of two words
- the checked files will be renamed, so they won't be checked again

Code: Select all

@echo off

:--------------------------------------------------------
:: Jump to Help if not all parameters have been given
	IF "%1"=="" GOTO HELP
	IF "%2"=="" GOTO HELP
:--------------------------------------------------------

:--------------------------------------------------------
:: Initiate variables
:--------------------------------------------------------
	SET SearchString=
	SET ConcatFiles=
	SET ProcessFile=
	SET PreviousFile=
	SET ProcessingResult=
	SET TmpDrivePath=
	SET TmpFile=
	:: Check if a string of two words is given to search for
	IF "%3"=="" (
		SET SearchString=%2
	) ELSE (
		SET SearchString=%2 %3
	)
:--------------------------------------------------------

:--------------------------------------------------------
:: Loop through the files (%1) and search for the string (%SearchString%)
:--------------------------------------------------------
	FOR %%f IN (%1) DO FOR /F "tokens=1 delims= " %%e IN ('type %%f ^| find "%SearchString%"') DO (
		SET ProcessFile=%%f
		SET TmpDrivePath=%%~dpf
		SET TmpFile=%%~nxf
		CALL :CHECKITEMS
	)
:--------------------------------------------------------

:--------------------------------------------------------
:: Looping is done, now jump to build the report
:--------------------------------------------------------
	GOTO :MAKEREPORT
:--------------------------------------------------------

:--------------------------------------------------------
:: This section is called from the loop
:--------------------------------------------------------
	:CHECKITEMS

		IF "%PreviousFile%"=="" GOTO GETFIRST
		IF "%ProcessFile%"=="%PreviousFile%" GOTO TRYNEXT
		GOTO :GETSECOND

		:GETFIRST
		COPY %ProcessFile% %TmpDrivePath%CHECKED_%TmpFile%_CHECKED > NUL
		SET ConcatFiles=%ProcessFile%
		GOTO :TRYNEXT

		:GETSECOND
		COPY %ProcessFile% %TmpDrivePath%CHECKED_%TmpFile%_CHECKED > NUL
		DEL %PreviousFile%
		SET ConcatFiles=%ConcatFiles%, %ProcessFile%
		GOTO :TRYNEXT

		:TRYNEXT
		SET PreviousFile=%ProcessFile%

	GOTO :EOF	
:--------------------------------------------------------

:--------------------------------------------------------
:: Build the report
	:MAKEREPORT

	IF "%ConcatFiles%"=="" (
		SET ProcessingResult=Ok:No instances found
	) ELSE (
		SET ProcessingResult=Bad:Errors found in files: %ConcatFiles%
	)

	ECHO ScriptRes:%ProcessingResult%
:--------------------------------------------------------

:--------------------------------------------------------
:: Clear the variables that have been used and delete the last processed file
	IF NOT "%ProcessFile%"=="" DEL %ProcessFile%
	SET SearchString=
	SET ConcatFiles=
	SET ProcessFile=
	SET PreviousFile=
	SET ProcessingResult=
	SET TmpDrivePath=
	SET TmpFile=
	GOTO END
:--------------------------------------------------------

:--------------------------------------------------------
:HELP
	ECHO ScriptRes:Unknown:Usage: integrated_file_runcheck {filepath to search, may include wildcard} {word to search for} 
	GOTO END
:--------------------------------------------------------

:--------------------------------------------------------
:END
:--------------------------------------------------------
I implemented this in the HostMonitor as a Shell Script Test and it works as planned.

At your service,
Cheers!

Gijsbert
KS-Soft Europe
Posts: 2832
Joined: Tue May 16, 2006 4:41 am
Contact:

Re: Even more complete

Post by KS-Soft Europe »

GijsbertVanOeveren wrote:I implemented this in the HostMonitor as a Shell Script Test and it works as planned.
Great job, Gijsbert. Wel done! I can allow myself to encourage you to post this awesome script into the "Library" branch of our forum:
http://www.ks-soft.net/cgi-bin/phpBB/viewforum.php?f=7

There is a special branch, where you can share your ideas, scripts, utilities useful information with HostMonitor's community.

Thank you very much.

Regards,
Max
User avatar
GijsbertVanOeveren
Posts: 11
Joined: Thu Jan 03, 2008 3:03 am

Final posting

Post by GijsbertVanOeveren »

Hi Max,

As you requested I posted the script in the other forum (http://www.ks-soft.net/cgi-bin/phpBB/vi ... 9127#19127).
I also changed a little bit in the code which now allows search strings up to 8 words :D
Please feel free to modify the accompanying text if you find anything unclear.

Thanks for your help and encouragements!

Gijsbert
KS-Soft Europe
Posts: 2832
Joined: Tue May 16, 2006 4:41 am
Contact:

Re: Final posting

Post by KS-Soft Europe »

GijsbertVanOeveren wrote:I also changed a little bit in the code which now allows search strings up to 8 words :D
Wow! 8 words. In fact, I think you may pass the several (8 or more) words as a second parameter. You just should wrap the second parameter into the quotation marks (""). For instance:

Code: Select all

Params: C:\*.txt "error message multi words"
Within the scipt body, you may use %2 to get the second parameter with surrounding quotation marks (e.g.: "error message multi words") or use %~2 that removes any surrounding quotation marks (e.g.: error message multi words).
GijsbertVanOeveren wrote:Thanks for your help and encouragements!
You are welcome. :-)
Actually, we have to say "thank you" to you as well! We really appreciate you found the time to make it work by yourself. Most of our customers prefer to utilize "cooked" solution.

Regards,
Max
User avatar
GijsbertVanOeveren
Posts: 11
Joined: Thu Jan 03, 2008 3:03 am

Quotation Marks

Post by GijsbertVanOeveren »

Thanks Max,

I adjusted the code and added the use of %~2 as a second option.
The fun thing about the first method is that you do not have to use quotation marks.

I tried to find a way to have both methods active by finding out if quotation marks are used. This is not hard to do by using this code:

Code: Select all

SET TMP1=%2
ECHO %TMP1:~0,1%
which will show a " which means that quotation marks are used.
It seems to be impossible though to put it in a VAR now and use it!
The code

Code: Select all

SET TMP1=%2
SET TMP2=%TMP1:~0,1%
can be set, but asking

Code: Select all

IF !%TMP2%!==!"! SET QUOT=Quotations Used
generates an error.
I would love to continue with something like

Code: Select all

IF NOT "%QUOT%"=="Quotations Used" (
{use method 1} 
) ELSE (
{use method 2}
)
The only thing I can come up with is determining if there is a difference between the string-lengths of %2 and %~2.
If not, no quotation marks are used, if the difference is 2 then the string is a quoted string.
But this seems quite cumbersome to me... like this:

Code: Select all

@echo off

SET TMP1=%1
SET TMP2=%~1

echo Parameter: %TMP1%
Call :STRLEN %TMP1%
SET LenVar1=%RESULT%
ECHO Length: %LenVar1%
SET RESULT=

echo Parameter: %TMP2%
Call :STRLEN %TMP2%
SET LenVar2=%RESULT%
ECHO Length: %LenVar2%
SET RESULT=

SET /A NewVal=%LenVar1%-%LenVar2%
ECHO Difference in length: %NewVal%
IF %NewVal%==2 (
	ECHO The String: %TMP1% has quotation marks!
) ELSE (
	ECHO The String: %TMP1% has NO quotation marks.
)



:STRLEN
SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
SET STR=%1 & SET STR=!STR:~0,-1!
SET LEN=0
SET POS=8192

:STRLEN_LOOP
SET /A POS /= 2
IF NOT "!STR!" == "" (
    IF NOT "!STR:~%POS%,1!" == "" (
        SET /A LEN += %POS% + 1
        SET STR=!STR:~%POS%!
        SET STR=!STR:~1!
    )
    GOTO :STRLEN_LOOP
)
ENDLOCAL & SET RESULT=%LEN%

GOTO :EOF
Do you have any simpler suggestions?

Kind regards

Gijsbert
Last edited by GijsbertVanOeveren on Wed Jan 16, 2008 2:05 am, edited 1 time in total.
KS-Soft Europe
Posts: 2832
Joined: Tue May 16, 2006 4:41 am
Contact:

Re: Quotation Marks

Post by KS-Soft Europe »

GijsbertVanOeveren wrote:Do you have any simpler suggestions?
Actually, I do not. In fact, this is a difficult matter. Handling quotation marks in scripts is a real headache. So, I do not think it worth to continue with this script because it becomes too complex and dificult to understand. Let's leave your first script as the base script. What do you think?
Anyway, I do not think someone needs to seek more than 8 words indeed. ;-)

Regards,
Max
Post Reply