Enterprise Session Monitor for SAS: Administration Guide

The basic functionality of ESM is described in the ESM User Guide. This document covers the Superuser and Administrative functionality of the ESM application, available to advanced users and administrators of the application.

Note: It is recommended that you review the document titled Functional Overview of ESM before continuing with the tasks outlined here.

Basic Operation¶

Starting and stopping the ESM Server¶

UNIX based installations¶

Starting ESM Server¶

Change directory (cd) to the path of the ESM Server binaries e.g. /opt/esm/esm-server/bin
Run the following command to start ESM: ./esm-server.sh start

The server will now launch and a message similar to the following should be displayed:

[user@esmserver bin]$ ./esm-server.sh start
Waiting for domain1 to start ..............................
Successfully started the domain : domain1
domain  Location: /opt/esm/esm-server/esm-core/glassfish/domains/domain1
Log File: /opt/esm/esm-server/esm-core/glassfish/domains/domain1/logs/server.log
Admin Port: 14848
Command start-domain executed successfully.

Stopping ESM Server¶

Change directory (cd) to the path of the ESM Server binaries e.g. /opt/esm/esm-server/bin
Run the following command to stop ESM: ./esm-server.sh stop

The server will now stop and a message similar to the following should be displayed:

[user@esmserver bin]$ ./esm-server.sh stop
Waiting for the domain to stop ....
Command stop-domain executed successfully.

Should you encounter any issues stopping ESM server, a force-stop command can be issued as follows: ./esm-server.sh force-stop
Force stop identifies the pid under which the server is running and issues a kill -9 command. This command should only be used if the normal stop command fails to work.

Restarting ESM Server¶

Stopping and starting ESM server can be achieved using the restart option on the ESM Server script
Change directory (cd) to the path of the ESM Server binaries e.g. /opt/esm/esm-server/bin
Run the following command to start ESM: ./esm-server.sh restart

The server will now stop and immediately start. A message similar to the following should be displayed:

[user@esmserver bin]$ ./esm-server.sh restart
Waiting for the domain to stop ....
Command stop-domain executed successfully.
Waiting for domain1 to start .............................
Successfully started the domain : domain1
domain  Location: /opt/esm/esm-server/esm-core/glassfish/domains/domain1
Log File: /opt/esm/esm-server/esm-core/glassfish/domains/domain1/logs/server.log
Admin Port: 14848
Command start-domain executed successfully.

Windows based installations¶

Using Windows Service status¶

If the ESM Server is installed as a service, the service status can be verified using the Windows Services manager. To check the service status, do the following:

Log on to the server using the account with the required privileges
Click Start
In the search box type services to filter the list of applications
Right click Services and click Run as Administrator
In the services window which appears, scroll down the list to the “ESM Web Server" entry. This should have a status of Started
The service can be started and stopped from this interface by right clicking and changing the status

Using the Command Line¶

Note: cmd.exe does not support UNC paths. Use Windows Powershell to issue the following commands when the ESM Server installation path is not mapped to a drive letter.

Click Start
In the search box type Powershell to filter the list of applications
Right click the Windows Powershell program and click Run as Administrator
At the command prompt, navigate to the location of the installed esm agent binaries. e.g. c:\esm\esm-server\bin

If ESM Server is installed and running as a Windows Service, the service can be stopped by issuing the "stop-service" command:

C:\esm\esm-server\bin> .\esm-server.bat stop-service
Stopping ESM Server service...

Note: If not installed/running as a Windows Service, issue the "stop" command:

C:\esm\esm-server\bin> .\esm-server.bat stop
Stopping ESM Server ...
Waiting for the domain to stop ..
Command stop-domain executed successfully.

To start the ESM Server Windows Service from the command line:

C:\esm\esm-server\bin> .\esm-server.bat start-service
Starting ESM Server service...

Note: To start ESM Server independently of the Windows Service manager, issue the "start" command:

C:\esm\esm-server\bin> .\esm-server.bat start
Starting ESM Server ...
Waiting for domain1 to start .........................................
Successfully started the domain : domain1
domain  Location: C:\esm\esm-server\esm-core\glassfish\domains\domain1
Log File: C:\esm\esm-server\esm-core\glassfish\domains\domain1\logs\server.log
Admin Port: 14848
Command start-domain executed successfully.

Starting and stopping agents¶

UNIX based nodes¶

Each ESM Agent can be started with the following command:

    /filesystem/esm/esm-agent/bin/esm-agent.sh start

Each ESM Agent can be stopped with the following command:

    /filesystem/esm/esm-agent/bin/esm-agent.sh stop

The status of each ESM Agent can be checked with the following command:

    /filesystem/esm/esm-agent/bin/esm-agent.sh status

Windows based nodes¶

Note: If the ESM Agents are installed as a Service under Windows, then the same procedure for starting the ESM Server, using the Windows Services Manager, can be used to start and stop the ESM Agents on each machine.

This section covers the start/stop procedure for Agents which are not installed as a Service under Windows:

Log on to the server using the nominated account with the required privileges (see installation requirements for more details)
Navigate to the location of the installed ESM Agent binaries. e.g.
\filesystem\esm\esm-agent\bin
To start the service, right click the esm-agent.bat file and select “Run as Administrator”
At the prompt type start to start the ESM Agent Service
The ESM Agent service will now start

Debugging¶

Inspecting the ESM Server Log files¶

UNIX based installations¶

The ESM Server log file is located under the logs path of the installation directory. E.g.

/opt/esm/esm-server/logs.

By default, minimal logging is enabled for ESM Server to ensure smooth running of production environments. More detailed logging can be enabled with the assistance of Boemska Support. For further details please email support@boemskats.com

The Java logfiles for the ESM Server JVM are located in the following directory:

[ESM Server Home]/esm-core/glassfish/domains/domain1/logs/server.log

Inspecting the ESM Agent Log Files¶

The logfiles for the ESM Agents are located in the following directory:

UNIX¶

$ESHMOME/esm-agent/logs

Windows¶

%ESMHOME%\esm-agent\logs

To increase or decrease the verbosity of the logs by adjusting the logging level, please contact Boemska support.

The ESM Admin Settings Interface¶

Many of the settings in ESM can be tuned through the ESM GUI. This is what the Admin Settings interface looks like:

ESM Admin Settings

Agent Configuration¶

The ESM Admin Settings interface shows a list of all ESM Agents that the system currently has a configuration for. Clicking on a row in the list loads the configuration for the selected agent into the Update Agent Settings pane.

The following settings can be set from this pane:

Events Folder¶

This is the directory on the node that the Agent monitors for trigger (Event) files created by SAS sessions configured with ESM. The initial value is set upon initial connection by the Agent, using the ESMEVENTS variable set in the Agent's esmconfig file.

The location can be changed here for debugging purposes without restarting the Agent. When the ESMEVENTS variable is reconfigured in esmconfig, it must also be updated here. Alternatively, te node configuration can be removed entirely to allow the agent to set its default values.

Note: For ESM to function correctly, the Agents directory constructed by the SAS code located in the esmtags.sas file must, for each node, match the directory configured here. Changing this setting from the default is generally not recommended.

Data Collection Interval¶

This setting changes the resolution of the data collected by the agent on that node. The default interval is 2 seconds, which is the lowest acceptable value. Increasing this number will lower the resolution of the data collected, therefore decreasing the amount of data stored in the database. This can also improve UI performance when looking at data over longer intervals, or when working with web browsers less capable of rendering a high number of data points.

Data Read Ratio¶

This setting dictates how often the sizes of a Session's SASWork and SASUtil directories are measured. The default ratio of 5, means that the size of these directories is measured every fifth Data Collection Interval (see above). With a default Data Collection Interval of 2 seconds, and a ratio of 5, these directories are sized for each session every 10 seconds. With a data collection Interval of 5 seconds and a ratio of 12, the size of these directories will be checked once per minute.

Reporting Interval¶

This setting changes how often the Agent sends a data packet to the Server. With the default setting of 0, the agent sends its data to the server after every Collection Interval. This interval can be increased to lower the frequency of network calls being made if desired.

The default setting of 0 is recommended.

Sum CPU cores¶

This setting allows the overall CPU usage on the Server to be presented using the same scale as a Session graph. For example:

With this setting disabled, on a node with 20 available threads, a 100% saturated CPU core will appear to consume 5% of the overall node CPU. The node CPU axis will be fixed at a maximum of 100%.
With this setting enabled, on a node with 20 available threads, a 100% saturated CPU core will appear to be consuming 100% of the overall node CPU. Five 100% saturated CPU cores will appear as 500% CPU consumption on the node. The node graph will go to a maximum of 2,000%.

This setting is generally not recommended.

Disable Work and Util tracking¶

This setting disables the size tracking for the SASWork and SASUtil directories for that agent.

Disable log file parsing¶

This setting disables the parsing of Batch Job log files for Warnings and Errors.

Updating the ESM License¶

Using a web browser, navigate to http://localhost:18080/esm-1.0/ where 18080 is the port you configured ESM Server on. If accessing from a machine other than the server, you must specify the hostname or IP address of the ESM Server in place of localhost. You should be presented with a screen requesting a license file
Click Browse to navigate and select your ESM license file.
Click Upload, then Renew License.
The License Holder and Expiry Date should be updated with the new license details.
Click Refresh. You should see the main ESM Web Application screen.

Database maintenance¶

Scheduling Database Maintenance¶

ESM has a built-in configurable scheduler designed to ease the maintenance of the ESM database. It allows the ESM Administrator to adjust the time of day that the maintenance routines are triggered.

ESM Database Maintenence Scheduler

Changing the Perform Database Maintenance time, and clicking the Save button, will update the time at which the daily maintenance of the database is performed. It is advised that this is set to a time of day when the number of concurrently active jobs will be at its minimum.
Note: To avoid clashing with other batch routines that may be scheduled to start 'on the hour' or a similar rounded interval, the scheduler will delay its maintenance routine for a random time between 1 and 30 minutes before the database maintenance jobs are triggered.

Other Database Settings¶

The 'Database Maintenence' section of the ESM Admin Settings screen also allows the following settings to be adjusted:

Keep last NN days¶

This setting is the main method of limiting the size of the ESM database. It controls for how long the detailed time-series metrics will be retained by ESM. At the default setting of 14 days, the system retains 14 days of detail data before converting it into cost items and discarding the time-series data.

On most systems, changing this setting should have a linear impact on the size of the database.

Maximum Query Timespan¶

This setting allows the administrator to tune the performance of ESM and restrict Historical View queries to a maximum timespan. This setting affects both database performance and the rendering performance of the Historical View data at the front end.

Discard following process types¶

This setting allows certain process types to be ignored in the cost item calculations that ESM makes when discarding time series data. When applied to certain session types that typically form a shared cost (such as most SYS processes), this can improve database performance.

Discard after N days¶

This setting controls how long ESM keeps the timeseries data for sessions that are typically discarded. Early discarding of the data for certain process types can decrease database size and improve performance.

Superuser features of ESM¶

This section describes the features of ESM which are restricted to Advanced Users and Administrators of the application.

User Roles in ESM¶

Users in ESM can be separated into the following 'Roles':

General Users
Privileged Users
The Administrator

General Users do not require any authentication within ESM. Privileged Users and the Administrator are prompted for credentials when they attempt to use the advanced features of the application. The roles are hierarchial: Privileged Users and Administrators are able to access all of the functionality available to General Users, and the Administrator is able to access all of the functionality available to Privileged Users.

Managing Privileged Users¶

Privileged Users in ESM are managed by the ESM Administrator. This is done via the Privileged Users Settings page, accessed via the Admin Settings interface.

ESM Privileged Users tab

To add a user to the list of Privileged Users, click the 'Add User' button. When asked, enter the credentials you wish to be used by that user.

ESM Privileged Users Add

Clicking the Edit User button will allow for the password of an existing user to be changed. Clicking Delete User will remove that user from the Privileged Users list.

Changing the ESM Administrator Password¶

The overriding password for the ESM Administrator is stored in a master password file and secured using filesystem-level security.

To change the ESM Administrator username and password, log on to the machine where the ESM Server is running as a suitably privileged user. Then, edit the admin.txt plaintext file, located in

[esm home]/esm-core/glassfish/domains/domain1/config

The format of admin.txt should as follows:

esmadminusername
esmadminpassword

Note: When saving the password file, ensure that the correct permissions continue to be applied to it. It can only be read by the ESM system account, and written by a designated user / administrator of the underlying operating system.

Advanced User Capabilities¶

Session Management¶

Required role: Privileged User**

Note: The Session Management functionality of ESM is still considered experimental. In order for it to function properly, permissions need to be further configured at the Operating System level and made available to the system identity under which the ESM Agent is executing.

If permitted by the Operating System, the ESM Agent can be used to terminate unresponsive, orphaned or unwanted SAS sessions. Once a session is identified, it can be terminated using a menu accessed by right-clicking on the session in the List Portlet in the Live View.

ESM Terminate Session

The menu presents the following options:

Terminate Process

This sends the process the equivalent of a SIGTERM signal for UNIX platforms, and a taskkill command on Windows platforms. Once the process is terminated, the Agent ensures that the temporary directories related to that process have been removed.

Force Terminate Process

This sends the process the equivalent of a SIGKILL signal for UNIX platforms, and a taskkill /f command on Windows platforms. Once the process is terminated, the Agent ensures that the temporary directories related to that process have been removed.

Force Terminate Process leaving temporary directories intact

This sends the process the equivalent of a SIGKILL signal for UNIX platforms, and a taskkill /f command on Windows platforms. Once the process is terminated, the Agent leaves any temporary directories that may be related to that process intact.

Note: If a process is known by ESM to be a System (SYS) process, or if it is not an instance of the sas binary executing, the process can not be acted upon. This limitation is imposed as a security precaution.

All actions performed using the Session Management menu are audited by ESM. A history of all Session Management actions can be seen in the Privileged Users section of the ESM Admin Settings screen.

SAS Batch Logfile retrieval¶

Required role: General User, optionally restricted to Privileged User**

When ESM is passed a 'logfile' option for a Session, such as in the case of a Batch Job, ESM will continually monitor that logfile for occurences of WARNING and ERROR messages. When either a Warning or Error message occurs, it is plotted against the timeline of that session in real time as a flag. Hovering over the flag using the mouse displays the warning or error text. Clicking on the flag retrieves the logfile in question, opens it in a new tab and navigates to the line where the error occured.

If this behaviour is configured as restricted to Privileged Users, suitable credentials must be entered prior to the logfile being retrieved.

Usage Chargeback calculation¶

Required role: Administrator**

When the timeseries data for a session is archived by the database, a 'cost item' is created in place of the timeseries data. Cost items are held in the database and calculated on a hourly basis. Each cost item is marked with the name of the session or Job, the name of the user under which it executed, and the amount of resource that the session consumed during that hour.

The ESM Classification Interface allows those cost items to be grouped into categories, so that overall system usage per category can be calculated for a given time period.

ESM Classification Interface

To access the ESM Classification Interface, click the Open Classification Interface button at the bottom of the Admin Settings tab. You will be presented with the following settings:

Item Limit¶

This number controls the maximum number of cost items that are shown in the Unclassified Items list. By default it is set to a conservative number of 2000. If, after setting the Target Confidence and Billing Period the Unclassified Items list still contains 2,000 unique items, this limit can be increased. The number of items shown are always sorted in descending order, therefore a limited list always contains the specified number of most significant cost items.

Target Confidence¶

Like the Item Limit, this setting also controls the maximum number of cost items that are presented in the Unclassified Items list. Whereas the Item Limit setting is a Count of the total number of cost items presented for classification, the Target Confidence setting limits the cost items shown according to their significance.

For example, if the database contains a list contains 15 cost items, where the first 6 items make up 90% of the total Cost and the last 9 items make up the other 20%, then setting a Target confidence of 90% will mean that only those 6 items will be presented to the user for categorisation, and the other 9 items will be considered as insignificant. If items 7 and 8 together add up to another 5% of cost, then setting the Target confidence to 95% will include the first 8 items in the list of items to be classified, ignoring the remaining 8 that make up the smallest 5% of cost.

Billing Period¶

This setting allows the Cost Items to be filtered according to the period during which they occurred. Any cost items that sit outside the specified date range are ignored.

Allocating Cost¶

ESM groups the cost items into unique job name / user combinations and calculates their share of the overall CPU consumed, taking into account the filter settings above. In order for the desired Target Confidence to be achieved, each of the cost items needs to be classified into Department.

Departments are created by clicking the Add Department button, which will prompt the user for a Department Name to be added to the list. When all departments have been added, their cost buckets can be opened by clicking each department in the Choose Department list.

From this point, each of the cost items in the Unclassified Items list must be allocated into a Department. This can be done by dragging and dropping each item in the list into a Departmental Bucket, or by selecting multiple items in the Unclassified Items list and clicking the (+) button on the department's cost bucket item. In addition, the Unclassified Items list can be filtered on Job Name or Executing User patterns, and clicking Select All and then (+) will assign all items appearing in a filtered list to the selected Departmental bucket.

The Total CPU Accounted For is designed to act as an indicator of progress. Clicking the Show Report button at any time will show a report detailing the breakdown of costs per department, along with a list of the top Users and Processes responsible for those costs. Clicking the Print Report button in this screen will present the report in a format which can be printed and reviewed by the party responsible for the system usage in each Department.

The username/jobname lookup lists created during the classification process persist in ESM across billing periods. While the first classification exercise may be arduous, subsequent instances will only require the classification of cost items which have newly entered the list of significant cost items, as defined by the Item Limit and Target Confidence settings.