Advanced ESM Administration Tasks

Disabling Windows Short Names to enable SASWORK Monitoring

If SASWORK is configured to point to a volume with short names enabled, and you wish to monitor SASWORK with ESM (recommended) you must change your SASWORK configuration to use a volume with short names disabled.

  1. Determine the new location for SASWORK. This guide assumes a drive letter of E:

  2. Query the short name status of drive E: by issuing the following command at a command prompt with elevated/Administrator privileges:

fsutil.exe 8dot3name query E: 
This should produce the following output:
C:\WINDOWS\system32>fsutil.exe 8dot3name query E:
The volume state is: 0 (8dot3 name creation is enabled).
The registry state is: 2 (Per volume setting - the default).
Based on the above two settings, 8dot3 name creation is enabled on E:. If the status returns 8dot3 name creation is enabled on E:, issue the following command to disable it:
C:\WINDOWS\system32>fsutil 8dot3name set E: 1
Successfully disabled 8dot3name generation on E:

  1. Now create a saswork directory (e.g E:\saswork)

  2. Now change the SAS Server configuration to set SASWORK to point to this location. It is recommended to change this setting in the Application Server root sasv9_usermods.cfg file. For example:

    C:\SAS\Lev1\SASApp\sasv9_usermods.cfg
    
    add the following entry to this file:
    -WORK "E:\saswork"
    

  3. Restart the SAS Object Spawners to pick up the new configuration

  4. Start a new SAS Session and re-issue the saswork option query: %put %sysfunc(getoption(work)); If the change has taken effect, you should see output in the SAS log similar to the following:

    E:\saswork\_TD1592_WIN2008R2N1_
    

Installing the ESM Agent as a systemd service on RHEL7

RHEL7 comes with systemd replacing sysvinit and upstart from RHEL6. Systemd lets us configure control the ESM agent on machine startup and if the agent stops.

To configure the ESM agent as a systemd service, create a file in /etc/systemd/system called esm-agent.service:

Update paths

Remember to change the reference to ##ESMHOME## below to point to real install path (eg. /opt/esm/esm-agent).

[Unit]
Description=ESM agent control
After=network.target

[Service]
ExecStart= ##ESMHOME##/bin/esm-agent-nonforking.sh start
ExecStartPost= ##ESMHOME##/modules/esm-module-control.sh start
Restart=on-failure
Type=simple
TimeoutStartSec=10
TimeoutStopSec=5
RestartSec=2
ExecStop=  ##ESMHOME##/bin/esm-agent-nonforking.sh stop
ExecStop=  ##ESMHOME##/modules/esm-module-control.sh stop
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

After creating the file, you can run the following commands as root (or sudo):

  • run systemctl daemon-reload command to make systemd re-read the config
  • run systemctl start esm-agent to have upstart run the agent for you
  • run systemctl enable esm-agent to have the agent start as a service on every subsequent boot
  • run systemctl status esm-agent to see the output of the last start

Installing the ESM agent as an upstart service in RHEL 6

RHEL6.x comes with Upstart as well as sysvinit, which lets us configure it to control the esm agent on machine startup and if the agent stops.

To configure the ESM agent to run as an upstart service, create a file in /etc/init called esm-agent.conf. Remember to change the env ESMHOME to point to real install path.

#!upstart
description "Boemska ESM Agent"

# Set this to location of installed agent
env ESMHOME=/pub/esm/agent/esm-agent

start on runlevel [2345]
stop on runlevel [!2345]

# Be nice and set some limits on open file handles
limit nofile 4096 4096

# But also make sure data gets collected when needed most
nice -15

# Return output about connecting to ESM server to the console
console output

# Restart automatically if agent stops but no more than 3x in the space of a minute
respawn
respawn limit 3 60

# Go
exec $ESMHOME/bin/esm-agent-nonforking.sh start

Following that:

run the initctl --verbose reload-configuration command to make upstart re-read the config run start esm-agent to have upstart run the agent for you

Configuring LDAP authentication

Enabling LDAP authentication in ESM allows developers to view the log files and manage their own sessions in a authenticated, audited way. When enabled, users will be able to authenticate using their normal Active Directory or LDAP account to view the log files or kill sessions where the username of the authenticated user matches the username of the owner of the session.

LDAP integration for ESM takes place on the ESM server. Depending on whether you are connecting to either Active Directory or another LDAP provider such as OpenLDAP, add a file called either ad.properties or ldap.properties in the following (relative) path of your ESM server installation: esm-core/glassfish/domains/domain1/config.

Active Directory configuration properties

An ad.properties file could should contain, at a minimum, the properties marked as required below:

Name Value Notes
ad.url ldap://<host>:<port>/ Required The URL of the Active Directory server that you are connecting to.
ad.domain boemska.local Required The name of the domain that you are connecting to.
ad.searchFilter Optional An AD search filter is used to define criteria to provide more efficient and effective searches (MS docs)

LDAP configuration properties

An ldap.properties file could should contain, at a minimum, the properties marked as required below:

Name Value Notes
ldap.url ldap://<host>:<port> Required The URL of the LDAP server that you are connecting to.
ldap.userSearchBase ou=users Required Example:
ldap.userSearchFilter (uid={0}) Required The {0} is important as this represent the value that is to be searched for.
ldap.root /dc=company,dc.com Optional. The root can be specified using the /dc=company,dc.com on the connection string URL.
ldap.userDn CN=ESM Binder,cn=Users,dc=boemska,dc=local Optional. If the LDAP server that you are connecting to doesn't accept annonymous binds then you must specify an account with which to perform the initial LDAP server bind.
ldap.password Optional. This is the valid password that corresponds to the user that is specified in ldap.userDn.

Triggerfile formats

When communicating with the agent through the triggerfile mechanism, processes can output triggerfiles for consumption by the agent as either a space-separated array of quoted values, or in JSON format. The space-separated array depends on field order and is considered legacy, but remains around 40% faster to parse and therefore continues to be supported for high-stress environments with little available overhead.

The following are examples of each format.

Warning

For legacy reasons, the ESM Agent matches triggerfiles based on hostname and will disregard any files where the hostname element of the filename does not match the agent hostname exactly. This check may be removed in a future release.

new event triggerfile

File name: new_12345_thishostname (where 12345 is the PID of the process to be monitored and thishostname is the agent recognised hostname)

File content (json)

{
   "owner": "thisUser",
   "sasUuid": "234801532270870",
   "hostname" : "mysashost",
   "queue": "defaultQueue",
   "jobName": "Lev1_SASAppTest",
   "workFolder": "/data/saswork/SAS_workC02600003039_thishostname",
   "utilFolder": "/data/sasutil/tempUtil_12345_thishostname",
   "logFile": "/data/logs/someWorkspaceLog.log",
   "esmType": "WS"
}

tag event triggerfile

File name: tag_12345_thishostname_123 (where 12345 is the PID of the process to be monitored, thishostname is the agent recognised hostname and 123 is an identifier that ensures a unique filename for the triggerfile, often an autoincremented counter)

File content (json)

{
   "text": "flagText",
   "tooltip": "tooltipText with more detail",
   "color": "#FAAD39"
}

highlightStart event triggerfile

File name: highlightStart_12345_thishostname (where 12345 is the PID of the process to be monitored and thishostname is the agent recognised hostname)

File content (json)

{
   "uuid": "234801532270870"
}

highlightEnd event triggerfile

File name: highlightEnd_hostname_234801532270870 (where thishostname is the agent recognised hostname and 234801532270870 is the uuid

File content (json)

{
    "uuid":"234801532270870", 
    "text":"myJobName.sas", 
    "returnCode":0, 
    "flow":"1670:thisUser:some_daily_flow:some_daily_subflow:myJobName"
}

ESM Agent Events directories

ESM Agent Configuration Reference

With ESM agent deployment there is a file called config.properties which can be found in the bin folder of the ESM agent installation. This configuration file is used to control the behaviour of the ESM agent. A full list of the ESM agent configuration properties, a description of the property, and the default value, can be found below:

Property Description Default value
monitorAllSessions Used to enable active process discovery False
monitoringInterval Used in conjunction with monitorAllSessions to specify the interval, in seconds, at which the host's process table is scanned 10
threadsIndexer The number of threads dedicated to periodic node-level directory sizing 5
processMonitorThreads The number of threads dedicated to measuring process performance 1
processIOMOnitorThreads The number of threads dedicated to measuring per-process temp directories 1
threadsCrawler unknown 5
maxiothroughput The ceiling limit for IO reads and writes per-second in bytes. This is a work around for a SIGAR underflow issue. 263-1
maxBufferSize The number of process measurements kept in the local agent cache if the server is uncontactable. 250000
maxSessionsCount The maximum number of processes that can be tracked by the agent 5000
startTimeVarianceTolerance This is a tolerance limit in milliseconds for hardware clock drift when identifying unique processes by pid and start time. 2000
maxEventFileSize The maximum size of a file, in bytes, of a file that will be read from the ESMEVENTS folder. If a file exceeds this size, the ESM agent will delete the file without reading it and log an error in the agent log file. 10485760
dontReportAgent Disable sending of data from this agent to the configured server False
regexTimeout The number of milliseconds before a regex lookup is considered catastrophic (in L notation) 100L
ignoreExistingSessionsOnRestart Disable the retrieval of existing session list from the server False
containerIdDisable Disable cgroup-based identification of the process container True
disableProcessIOCollection Disable per-process IO metrics collection False
disableProcessMemoryCollection Disable per-process memory metrics collection False
logEventsRateLimit Limit the number of events that are generated per-log for each iteration of the agent metric collection cycle. Events exceeding this number are skipped. 100
containerIdRegex The regular expression that is used to extract the container ID from cgroups. \\d+:\\w+:(\\/docker\\/|\\/kubepods\\/.*\\/)(.*)
containerIdRegexGroup The number of the capture group, defined in containerIdRegex that contains the actual container ID to be passed to the server when registering a pod. 2
logDiscoveryDisable Disable discovery of log files that are defined in the processLogfilePattern property of the filterSpec.yaml file False
disableCopyTruncate By default, if a log file reduces in size from when it was last read, the ESM agent assumes that the log has been rotated, and will start reading the log file from the beginning. Setting this value to True will override this functionality and the log will wait until the file is bigger than the last read size. False
useCumulativeIOCounters Use alternate SIGAR counters for measuring disk IO. You should set this to True by default on Windows. False
disableProcessStateCollection Do not collect process state information (# of threads, status, etc.). This data is not exposed on some operating systems including Solaris and generates an error. False
disableProcessFDCollection Do not collect process file descriptor information (# of threads, status, etc.). This data is not exposed on some operating systems including Solaris and generates an error. False
disableUsingPsForProcessOwner The ESM agent primarily uses a SIGAR function to get the process owner information. As a fallback, the process table is looked up directly. Setting this value to true, disables the fallback method. If SIGAR is unable to find the username and the fallback has been disabled, the username for a process may be blank. False
totalIOThroughputOnly Do not try and collect separated read and write IO information. This data is not exposed on some operating systems including AIX and can generate an error if not set to True. False
autodiscoveredTempRemoval When a process' temporary directories (SAS work & UTIL) are automatically discovered, this sets the default behavior for what should happen to those directories when the associated PID is terminated through the ESM user interface. True
serverMode When an ESM agent is in serverMode a hard coded, internal filterSpec.yaml file is used to only discover ESM server and ESM database processes. When an ESM agent is in server mode, the number of cores of the running server are not counted for licensing purposes. False
logThreads The ESM agent uses multiple threads to scan log files. You can control the number of threads that the ESM agent uses to read log files. 4