Retrieving Grid Logs

As M3 has evolved it has become more complex. There are more components to monitor and though the grid has improved and provides easy ways to look at the logs there are a lot of logs to look at and a lot of information to digest.

Taking a look at IFL there’s what, 1, 2, 3, 4…oh really…far too many components to examine. And I am far too lazy to go through each one 🙂

They aren’t a great example because there isn’t much going on in the warning/error department, however some other customers have a lot in a lot of the grid applications.

The other issue is that it starts to get hard to correlate events between the different grid applications. One particular issue for a customer had a key event which caused a cascade of warnings and errors through other grid applications making the logs look pretty scary.

Around the 13.2 / 13.3 timeframe Infor started adding more rest interfaces to depreciate the old grid scripting tool. In 13.3 we have enough functionality to retrieve the grid application logs themselves via rest calls – which is exactly what I was looking for.

I wrote a quick little Powershell script which loops through the nodes and will retrieve the notes, warning and error events from the logs and will then dump them out to the console. I redirect that output to a html file and then I can copy the data in to Excel and filter and sort as I see fit – even create scatter graphs to look for correlation.

This is an example of dumping the data out to a HTML file.

Then I can extract portions of the data in Excel or apply filtering and sorting to examine things of interest.

Which can then be plotted on scatter graphs and pivot tables.

This graph, though it’s not terribly interesting – it plots the number of events per grid application on a timeline.

The filtering options within the powershell script are fairly limited at the moment as we can do more complex filtering and sorting of data within Excel but it’s enough for me to cut down some of the noise.

The script is pretty simple to use and you will be prompted for credentials:

Call getNodeLogs.ps1 -GridUrl [-htmlOutput [-funkyStuff [-GreaterThan ‘yyyy-MM-dd HH:mm:ss’] [-SystemName ] [-LastDays ]]]

= <the URL of the grid>. Eg. https://ifldev-bel1.cloud.infor.com:63906

-htmlOutput = if present, it will use the powershell ConvertTo-HTML to dump the table of data in a html table.

-funkyStuff = this turns on the following filtering options

-GreaterThan ‘<yyyy-MM-dd HH:mm:ss>’    = filter the logs so we output anything greater than or equal to the date/time entered

– SystemName <system name>     = the system name to filter on, eg. M3Auto (it is case sensitive)

– LastDays <number of days>    = the we will only return records <today>  – <number of days>

An example of a call may be

.\getNodeLogs.ps1 -GridUrl https://ifldev-bel1.cloud.infor.com:63906 -funkyStuff -LastDays 7 -htmlOutput >d:\data.html

 

Script is below.

#
# Retrieve the grids logs
# it will prompt you for grid credentials
# Works on 13.3 and 13.4
#
#	Usage:
#		getNodeLogs.ps1 -GridUrl https://ifldev-bel1.cloud.infor.com:63906
#
#	-htmlOutput output to html
#	-funkyStuff enables some of the more complex filtering
#		-GreaterThan 'yyyy-MM-dd HH:mm:ss' greater than or equal to date
#		-SystemName <name> eg. M3Auto (case sensitive)
#		-LastDays <days> number of days prior to DateTime::Now
#
#	Example
#		.\getNodeLogs.ps1 -GridUrl https://ifldev-bel1.cloud.infor.com:63906 -funkyStuff -GreaterThan '2017-09-20' -SystemName M3Auto -htmlOutput >d:\data.html
#	writes an html to d:\data which is greater than or equal to '2017-09-20' related to the M3Auto system

param([Parameter(Mandatory = $true)][string]$GridUrl, [switch]$htmlOutput, [switch]$funkyStuff, [string]$GreaterThan, [string]$SystemName, [string]$LastDays)

$credentials = Get-Credential

$baseURL = $GridUrl + '/grid/rest/nodes/'

$processedLogs = New-Object System.Collections.Generic.List[System.Object]

$lastDate

function getdate($entry)
{
	$parsedDate = [DateTime]::MinValue

	if($entry.Length -gt 18)
	{
		$extractedDate = $entry.Substring(0,19)
		$result = [DateTime]::TryParse($extractedDate,[ref]$parsedDate)
	}

	return $parsedDate
}

function getRawDate($entry)
{
	$result = ""

	$parsedDate = getdate($entry)

	if($parsedDate -ne [DateTime]::MinValue)
	{
		$position = $entry.IndexOf("Z")

		if($position -ne -1)
		{
			if($entry.Length -gt $position)
			{
				$result = $entry.Substring(0, ($position + 1))
			}
		}
	}

	return $result
}

function getLogEntry($entry)
{
	$result = $entry;

	$rawDate = getRawDate($entry)

	if($rawDate.Length -gt 0)
	{
		$result = $entry.Substring($rawDate.Length)
	}

	return $result
}

function createLogObject
{
	Param($Name, $EntryType, $entry)
	$result = New-Object PSObject

	Add-Member -InputObject $result -Name Name -MemberType NoteProperty -Value $Name
	Add-Member -InputObject $result -Name EntryType -MemberType NoteProperty -Value $EntryType

	$newDate = getdate($entry)

	$logEntry = getLogEntry($entry)
	$rawDate = getRawDate($entry)

	if($newDate -ne [DateTime]::MinValue)
	{
		$lastDate = $newDate
		$processedLogs.Add($result)
	}

	Add-Member -InputObject $result -Name LogDate -MemberType NoteProperty -Value $lastDate
	Add-Member -InputObject $result -Name LogDateRaw -MemberType NoteProperty -Value $rawDate
	Add-Member -InputObject $result -Name LogEntry -MemberType NoteProperty -Value $logEntry

	if($newDate -eq [DateTime]::MinValue)
	{
		if($processedLogs.Count -gt 0)
		{
			$separator = [Environment]::NewLine
			# if($htmlOutput -eq $true)
			# {
				# $separator = "::"
			# }

			$processedLogs[$processedLogs.Count - 1].LogEntry += ($separator + $entry)
		}
	}

	#return $result
}

function update($log)
{
	$processedEntries  = New-Object System.Collections.Generic.List[System.Object]
	#$log.Log | ForEach-Object { if($_.Length -gt 31) { $newDate = getdate($_); $lastDate; if($newDate -ne [DateTime]::MinValue) { $lastDate = $newDate};$entry = New-Object PSObject; Add-Member -InputObject $entry -Name LogDate -MemberType NoteProperty -Value $lastDate; Add-Member -InputObject $entry -Name LogDateRaw -MemberType NoteProperty -Value $_.Substring(0,31); Add-Member -InputObject $entry -MemberType NoteProperty -Name LogEntry -Value $_}}; $processedEntries.Add($entry);
	$log.Log | ForEach-Object { if($_.Length -gt 31) { $newDate = getdate($_); $rawDate = getRawDate($_); $logEntry = getLogEntry($_); $lastDate; if($newDate -ne [DateTime]::MinValue) { $lastDate = $newDate};$entry = New-Object PSObject; Add-Member -InputObject $entry -Name LogDate -MemberType NoteProperty -Value $lastDate; Add-Member -InputObject $entry -Name LogDateRaw -MemberType NoteProperty -Value $rawDate; Add-Member -InputObject $entry -MemberType NoteProperty -Name LogEntry -Value $logEntry}}; $processedEntries.Add($entry);
	Add-Member -InputObject $log -Name ProcessedLogs -MemberType NoteProperty -Value $processedEntries
}

function processLogEntry
{
	Param($Name, $EntryType, $log)
	$log.Log | ForEach-Object { $newObject = createLogObject $Name $EntryType $_}
}

$nodes = Invoke-RestMethod -Uri $baseURL -Credential $credentials
#$nodes | Format-Table

$notes = $nodes | Where-Object {$_.loggerErrorCount -gt 0 -or $_.loggerWarningCount -gt 0} | select @{n='node';e={$baseURL + $_.jvmID + "/log?filter=NOTE"}},name
$errors = $nodes | Where-Object {$_.loggerErrorCount -gt 0} | select @{n='node';e={$baseURL + $_.jvmID + "/log?filter=ERROR"}},name
$warnings = $nodes | Where-Object {$_.loggerWarningCount -gt 0} | select @{n='node';e={$baseURL + $_.jvmID + "/log?filter=WARN"}},name

if($funkyStuff -eq $false)
{
	if($htmlOutput -eq $false)
	{
		$errors | ForEach-Object { Write-Output $_.name;Write-Output "=ERROR======E"; Invoke-RestMethod -Uri $_.node -Credential $credentials;Write-Output "===========E" }
		$warnings | ForEach-Object { Write-Output $_.name;Write-Output "=WARN=======W"; Invoke-RestMethod -Uri $_.node -Credential $credentials;Write-Output "===========W" }
		$notes | ForEach-Object { Write-Output $_.name;Write-Output "=NOTE=======N"; Invoke-RestMethod -Uri $_.node -Credential $credentials;Write-Output "===========N" }
	}
	else
	{
		Write-Output "<html><body>"

		$errors | ForEach-Object 	{ 		Write-Output "
<h1>";		Write-Output $_.name;		Write-Output "</h1>
";		Write-Output "
<h2>Errors</h2>
<table border=1>
<tr>
<td>";		((Invoke-RestMethod -Uri $_.node -Credential $credentials) | %{$_ -split "`n"}  | %{$_ + "
"});		Write-Output "</td>
<tr></table>
" 	}

		Write-Output "</body></html>"

	}
}
else
{
	$logEntries = New-Object System.Collections.Generic.List[System.Object]
	#$errors | ForEach-Object { $log = New-Object PSObject; Add-Member -InputObject $log -MemberType NoteProperty -Name Name -Value $_.name; Add-Member -InputObject $log -MemberType NoteProperty -Name Log -Value (Invoke-RestMethod -Uri $_.node -Credential $credentials);$logEntries.Add($log); }
	$errors | ForEach-Object { $log = New-Object PSObject; Add-Member -InputObject $log -MemberType NoteProperty -Name EntryType -Value 'ERROR'; Add-Member -InputObject $log -MemberType NoteProperty -Name Name -Value $_.name; Add-Member -InputObject $log -MemberType NoteProperty -Name Log -Value ((Invoke-RestMethod -Uri $_.node -Credential $credentials) | %{$_ -split "`n"});$logEntries.Add($log); }
	$warnings | ForEach-Object { $log = New-Object PSObject; Add-Member -InputObject $log -MemberType NoteProperty -Name EntryType -Value 'WARN'; Add-Member -InputObject $log -MemberType NoteProperty -Name Name -Value $_.name; Add-Member -InputObject $log -MemberType NoteProperty -Name Log -Value ((Invoke-RestMethod -Uri $_.node -Credential $credentials) | %{$_ -split "`n"}) ;$logEntries.Add($log); }
	$notes | ForEach-Object { $log = New-Object PSObject; Add-Member -InputObject $log -MemberType NoteProperty -Name EntryType -Value 'NOTE'; Add-Member -InputObject $log -MemberType NoteProperty -Name Name -Value $_.name; Add-Member -InputObject $log -MemberType NoteProperty -Name Log -Value ((Invoke-RestMethod -Uri $_.node -Credential $credentials) | %{$_ -split "`n"}) ;$logEntries.Add($log); }

	#$logEntries | ForEach-Object { update($_) }

	$logEntries | ForEach-Object { processLogEntry $_.Name $_.EntryType $_ }

	$result = $processedLogs

	$greaterThanDate = [DateTime]::MinValue
	$validDate = [DateTime]::TryParse($GreaterThan,[ref]$greaterThanDate)

	if($true -eq $validDate)
	{
		$result = $result | where {$_.LogDate -ge $greaterThanDate}
	}

	if($false -eq [String]::IsNullOrEmpty($SystemName))
	{
		$result = $result | where {$_.Name -eq $SystemName}
	}

	if($false -eq [String]::IsNullOrEmpty($LastDays))
	{
		$days = 0
		$validDays = [Int32]::TryParse($LastDays, [ref]$days)

		if($validDays -eq $true)
		{
			$days *= -1
			$filterDate = [DateTime]::Now.AddDays($days)
			$result = $result | where {$_.LogDate -ge $filterDate}
		}
	}

	if($htmlOutput -eq $false)
	{
		$result | Format-Table
	}
	else
	{
		$result | ConvertTo-Html
	}

}

And as it is something I see myself changing, I’ve dropped it on to github so people can grab any updates.

https://github.com/potatoit/PSM3-getNodeLogs

This entry was posted in Development, M3 / MoveX, Misc. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s