Managing Global settings

Clicking on the admin settings menu button will prompt you for you ESM credentials. Once you have entered your ESM username and password you will be presented with the Admin overview screen.

The overview screen displays a list of all of the agents that have been configured to talk to the ESM server and provides an interface for updating agent configuration settings.

Update agent settings

Selecting an agent from the top table list of agents will populate the Update Agent Settings form automatically.

Setting	Default Value	Editable	Description
Agent Hostname	N/A	No	Hostname of the agent for who's settings we are viewing/editing.
App Version	N/A	No	The version of the agent that this node is running.
Runner Version	N/A	No	The version of the runner that this agent is running.
Cores	N/A	No	The number of cores that the agent is reporting that the node has.
Events folder	$ESM_AGENT_HOME/events/$HOSTNAME	Yes	This is the directory on the node that the Agent monitors for trigger (Event) files created by SAS sessions configured with ESM. The initial value is set upon initial connection by the Agent, using the ESMEVENTS variable set in the Agent's esmconfig file. The location can be changed here for debugging purposes without restarting the Agent. When the ESMEVENTS variable is reconfigured in esmconfig, it must also be updated here. Alternatively, te node configuration can be removed entirely to allow the agent to set its default values. Note: For ESM to function correctly, the Agents directory constructed by the SAS code located in the esmtags.sas file must, for each node, match the directory configured here. Changing this setting from the default is generally not recommended.
Data collecting interval	2	Yes	This setting changes the resolution of the data collected by the agent on that node. The default interval is 2 seconds, which is the lowest acceptable value. Increasing this number will lower the resolution of the data collected, therefore decreasing the amount of data stored in the database. This can also improve UI performance when looking at data over longer intervals, or when working with web browsers less capable of rendering a high number of data points.
Disk read ratio	5	Yes	This setting dictates how often the sizes of a Session's SASWork and SASUtil directories are measured. The default ratio of 5, means that the size of these directories is measured every fifth Data Collection Interval (see above). With a default Data Collection Interval of 2 seconds, and a ratio of 5, these directories are sized for each session every 10 seconds. With a data collection Interval of 5 seconds and a ratio of 12, the size of these directories will be checked once per minute.
Reporting interval	0	Yes	This setting changes how often the Agent sends a data packet to the Server. With the default setting of 0, the agent sends its data to the server after every Collection Interval. This interval can be increased to lower the frequency of network calls being made if desired. The default setting of 0 is recommended.
Sum CPU cores (set max load to cores x 100%)	False	Yes	This setting allows the overall CPU usage on the Server to be presented using the same scale as a Session graph. For example: With this setting disabled, on a node with 20 available threads, a 100% saturated CPU core will appear to consume 5% of the overall node CPU. The node CPU axis will be fixed at a maximum of 100%. With this setting enabled, on a node with 20 available threads, a 100% saturated CPU core will appear to be consuming 100% of the overall node CPU. Five 100% saturated CPU cores will appear as 500% CPU consumption on the node. The node graph will go to a maximum of 2,000%. This setting is generally not recommended.

Database Maintenance

ESM has a built-in configurable scheduler designed to ease the maintenance of the ESM database. It allows the ESM Administrator to adjust the time of day that the maintenance routines are triggered.

Changing the Perform Database Maintenance time, and clicking the Save button, will update the time at which the daily maintenance of the database is performed. It is advised that this is set to a time of day when the number of concurrently active jobs will be at its minimum.

Note

To avoid clashing with other batch routines that may be scheduled to start 'on the hour' or a similar rounded interval, the scheduler will delay its maintenance routine for a random time between 1 and 30 minutes before the database maintenance jobs are triggered.

Classification Interface

When the timeseries data for a session is archived by the database, a 'cost item' is created in place of the timeseries data. Cost items are held in the database and calculated on a hourly basis. Each cost item is marked with the name of the session or Job, the name of the user under which it executed, and the amount of resource that the session consumed during that hour.

The ESM Classification Interface allows those cost items to be grouped into categories, so that overall system usage per category can be calculated for a given time period.

To access the ESM Classification Interface, click the Open Classification Interface button at the bottom of the Admin Settings tab. You will be presented with the following settings:

Item Limit This number controls the maximum number of cost items that are shown in the Unclassified Items list. By default it is set to a conservative number of 2000. If, after setting the Target Confidence and Billing Period the Unclassified Items list still contains 2,000 unique items, this limit can be increased. The number of items shown are always sorted in descending order, therefore a limited list always contains the specified number of most significant cost items.

Target Confidence Like the Item Limit, this setting also controls the maximum number of cost items that are presented in the Unclassified Items list. Whereas the Item Limit setting is a Count of the total number of cost items presented for classification, the Target Confidence setting limits the cost items shown according to their significance.

For example, if the database contains 15 cost items, where the first 6 items make up 90% of the total Cost and the last 9 items make up the other 20%, then setting a Target confidence of 90% will mean that only those 6 items will be presented to the user for categorization, and the other 9 items will be considered as insignificant. If items 7 and 8 together add up to another 5% of cost, then setting the Target confidence to 95% will include the first 8 items in the list of items to be classified, ignoring the remaining 8 that make up the smallest 5% of cost.

Billing Period This setting allows the Cost Items to be filtered according to the period during which they occurred. Any cost items that sit outside the specified date range are ignored.

Allocating Cost ESM groups the cost items into unique job name / user combinations and calculates their share of the overall CPU consumed, taking into account the filter settings above. In order for the desired Target Confidence to be achieved, each of the cost items needs to be classified into Department.

Departments are created by clicking the Add Department button, which will prompt the user for a Department Name to be added to the list. When all departments have been added, their cost buckets can be opened by clicking each department in the Choose Department list.

From this point, each of the cost items in the Unclassified Items list must be allocated into a Department. This can be done by dragging and dropping each item in the list into a Departmental Bucket, or by selecting multiple items in the Unclassified Items list and clicking the (+) button on the department's cost bucket item. In addition, the Unclassified Items list can be filtered on Job Name or Executing User patterns, and clicking Select All and then (+) will assign all items appearing in a filtered list to the selected Departmental bucket.

The Total CPU Accounted For is designed to act as an indicator of progress. Clicking the Show Report button at any time will show a report detailing the breakdown of costs per department, along with a list of the top Users and Processes responsible for those costs. Clicking the Print Report button in this screen will present the report in a format which can be printed and reviewed by the party responsible for the system usage in each Department.

The username/jobname lookup lists created during the classification process persist in ESM across billing periods. While the first classification exercise may be arduous, subsequent instances will only require the classification of cost items which have newly entered the list of significant cost items, as defined by the Item Limit and Target Confidence settings.

License Interface

The license interface allows an ESM administrator to update the ESM license through the user interface. For more information about updating the ESM license please see the ESM Administration Guide section on updating the ESM license.

Privileged Users

Users in ESM can be separated into the following 'Roles':

• General Users
• Privileged Users
• The Administrator

General Users do not require any authentication within ESM. Privileged Users and the Administrator are prompted for credentials when they attempt to use the advanced features of the application. The roles are hierarchial: Privileged Users and Administrators are able to access all of the functionality available to General Users, and the Administrator is able to access all of the functionality available to Privileged Users.

Privileged Users in ESM are managed by the ESM Administrator. This is done via the Privileged Users Settings page, accessed via the Admin Settings interface.

To add a user to the list of Privileged Users, click the 'Add Privileged User' button. This will change the focus to the username field further down the screen where you'll be prompted to create a password for the user and assign the permissions for the user:

Sessions

If permitted by the Operating System and ESM's configuration, the ESM Agent can be used to terminate unresponsive, orphaned or unwanted SAS sessions. Once a session is identified, it can be terminated using a menu accessed by right-clicking on the session in the List Portlet in the Live View.

The menu presents the following options:

• Terminate Process

This sends the process the equivalent of a SIGTERM signal for UNIX platforms, and a taskkill command on Windows platforms. Once the process is terminated, the Agent ensures that the temporary directories related to that process have been removed.

• Force Terminate Process

This sends the process the equivalent of a SIGKILL signal for UNIX platforms, and a taskkill /f command on Windows platforms. Once the process is terminated, the Agent ensures that the temporary directories related to that process have been removed.

• Force Terminate Process leaving temporary directories intact

This sends the process the equivalent of a SIGKILL signal for UNIX platforms, and a taskkill /f command on Windows platforms. Once the process is terminated, the Agent leaves any temporary directories that may be related to that process intact.

Note

If a process is known by ESM to be a System (SYS) process, or if it is not an instance of the sas binary executing, the process can not be acted upon. This limitation is imposed as a security precaution¹.

Important

All actions performed using the Session Management menu are audited by ESM. A history of all Session Management actions can be seen in the Privileged Users section of the ESM Admin Settings screen.

Logs

When ESM is passed a 'log file' option for a Session, such as in the case of a Batch Job, ESM will continually monitor that log file for occurrences of WARNING and ERROR messages. When either a Warning or Error message occurs, it is plotted against the timeline of that session in real time as a flag. Hovering over the flag using the mouse displays the warning or error text. Clicking on the flag retrieves the log file in question, opens it in a new tab and navigates to the line where the error occurred.

If this behaviour is configured as restricted to Privileged Users, suitable credentials must be entered prior to the logfile being retrieved.

Settings

Changing the ESM Administrator Password

The overriding password for the ESM Administrator is stored in a master password file and secured using filesystem-level security.

To change the ESM Administrator username and password, log on to the machine where the ESM Server is running as a suitably privileged user. Then, edit the admin.txt plaintext file, located in

[esm home]/esm-core/glassfish/domains/domain1/config

The format of admin.txt should as follows:

esmadminusername
esmadminpassword

Note: When saving the password file, ensure that the correct permissions continue to be applied to it. It can only be read by the ESM system account, and written by a designated user / administrator of the underlying operating system.

Log file markers

ESM can be configured to automatically scan log files that it is aware of to detect events that users might want to know about such as ERROR and WARN events in SAS. These log file markers are used to annotate the process portlet charts.

Log file marker rules are can be defined in two ways, through the ESM user interface or via a JSON import mechanism².

UI Adding and Editing

Clicking Add or Edit while highlighting an existing log file marker pattern will bring up the Add/Edit regex dialog box. From this screen you can create or modify ESM log file marker patterns.

Regex parameters

The regex parameters are used to define the rules which are applied when the log file parser is run.

• Filename pattern - When log files are found to be associated with a session the filename rule is applied first. This ensures that only those rules which match the file name pattern are applied when reading the contents of the file. This is done for performance reasons.

  The file name in this context includes the full path to the file. If you wish to just distinguish between files of different names you can use the .* regex to start the filename, for example, .*WorkspaceServer_\d{4}-\d{2}-\d{2}_.*[0-9]+.log. If you wish the rule that you are creating or editing to apply to all files, leave this field blank

• Hostname pattern - After the file name pattern is run, the hostname of the server from which the session is associated is checked. The hostname pattern accepts a valid regex pattern or, if you wish for the rule to applied to all host, can be left blank.

• Log line filter - For each line of the applicable log files that are read by the agent this is the rule that gets applied. Parts or all of the line can be used in either the tag text or the tool tip text by capturing the relevant data using named regular expression capture groups. For example, capturing the whole line in a capture group called line could be done with (?<line>.*). The variable ${line} would then be available for both the tag text and tooltip text properties.

• Tag text - The tag text is the value that is visible in the chart annotation (see below). Valid values can be from a named capture group from the log line filter accessed using the notation ${cgname}where cgname is the name of the capture group, hard coded text, or blank.

  We recommend keeping these values has short as possible so as not to clutter the charts. You'll notice that the default log marker patterns shipped with ESM are a single character like E or W.

• Tooltip text - The Tooltip text is the the text that you see when you hover your mouse over the chart annotation. In most cases this will be the value from a named capture group from the log line filter. For example, if you had a capture group called msg in your log line filter, the value of the tooltip text might be: ${msg}.

• Marker color - The marker color defines the background color of the chart annotations. A valid color-hex code is required³.

Sample Input

This are is where you can test the rules that you have configured with sample data.

• Sample file name - This is an example of a full file path, including the file name, that you wish to test the filename pattern.
• Sample hostname - This is an example of a hostname to test the hostname pattern filter. The hostnames that are passed to this filter will match the hostnames that you see in the live view and that were configured during the agent installation process. For a SAS system, this should match the value of the &syshostname macro variable.
• Sample log data - This is the data that we are going to test the log line rule, tooltip text rule and the tag text rule. This area should be populated with a sample for the type of log file that you wish to test. It should contain one or more lines that you wish to match

Sample Output

Having defined a rule (at a minimum the log line filter, tag text, tooltip text, and marker color) and provided some sample data, the bottom half of the add/edit regex dialog provides an example of what a chart will look like based on the sample input and the rules that you have defined.

JSON Import / Export

The log marker settings import export functionality is available starting in ESM 2020.4. It provides a simple way of importing and exporting large and complex log marker rule sets between ESM installations.

Export

Clicking on the Export button will automatically download an export.json file containing all of the rules that you have in configured and saved within your environment. The file is downloaded to your browser's default downloads directory.

Import

Clicking on the Import button will bring up an Open File dialog prompting you to select a file to upload.

File format

The ESM log marker upload parser accepts an array of JSON objects. Each object in the array must contain the following properties (all properties are required by can be set to null):

Special regex characters, for example, \s, used to denote a space will need to be escaped within the JSON string. Special characters are escaped by using backslack: \\s.

• fileName - This is the filename pattern. Specify null for the rule to apply to all filenames
• hostnameRegex - This is the hostname pattern. Specify null for the rule to apply to all hostnames
• searchRegex - This is the log line filter. A valid JSON escaped regular expression string must be provided for this field.
• fixedOutputTitle - This is the tag text field. A string value must be provided for this field.
• outputRegex - This is the tooltip text field. A string value must be provided for this field. This field can contain variables in the form of ${name} where name is a named capture group in the searchRegex string.
• color - This is the color value for the rule. Must be a string value, starting with a # and be a valid hexadecimal color value.

An example of a valid file might be:

[
    {
        "fileName":".*ObjectSpawner_\\d{4}-\\d{2}-\\d{2}_.*[0-9]+.log", 
        "hostnameRegex":null, 
        "outputRegex":"${msg}", 
        "fixedOutputTitle":"E", 
        "color":"#FF9999", 
        "searchRegex":"^(?<timestamp>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2},\\d{3})\\sERROR\\s[^:]+:\\w+\\s-\\s(?<msg>.*)"
    },
    {
        "fileName":".*ObjectSpawner_\\d{4}-\\d{2}-\\d{2}_.*[0-9]+.log", 
        "hostnameRegex":null, 
        "outputRegex":"${msg}", 
        "fixedOutputTitle":"W", 
        "color":"#F5ED0F", 
        "searchRegex":"^(?<timestamp>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2},\\d{3})\\sERROR\\s[^:]+:\\w+\\s-\\s(?<msg>.*)"
    }
]

---

1. For more information about administering ESM's global settings or for more advanced ESM Administration options please see the ESM Administration Guide ↩

2. The log marker settings import functionality is available in ESM 2020.4 and later. ↩

3. A great site for finding colors that you like can be found at https://www.color-hex.com/. For your assistance and reference, the ESM default ERROR color is #FF9999 and WARN is #F5ED0F. ↩