Advanced ESM configuration

Web Server reverse proxy configuration

In certain deployments, you may wish to configure the SAS Enterprise Session Monitor so that it interacts with an existing web server deployment. This section contains a sample configuration for running SAS Enterprise Session Monitor behind a ngnix web server:

Note

You will need to update the value of the server_name for your environment.

server {
    server_name esm-server.sasdemo.com;
    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass http://127.0.0.1:18080/;
        access_log  /var/log/nginx/esm.access.log  main;
        error_log /var/log/nginx/esm.error.log debug;
    }
}

Per-session tracking of SAS UTILLOC size

SAS multi-threaded procedures can make use of a disk paging location referred to as UTILLOC. ESM is designed with the capability to track the size of this UTILLOC usage of each individual session, but to enable this functionality, some extra configuration is required.

Unlike the SAS WORK location, which is a directory created at session startup that is unique to each individual SAS session, SAS UTIL directories are created on-the-fly by SAS procedures as they are required. Therefore, unlike the WORK system option which holds the full path to each session's individual SASWORK directory, the UTILLOC system option points to the parent directory which can contain any number of UTIL directories belonging to a number of other sessions. This makes periodic size measurements of a session's individual set of UTIL directories challenging.

One way of addressing this issue is to create a per-session UTILLOC subdirectory for each session before it is started, using it's usermods initialisation script, and removing the directory after the session has finished executing. This results in all of a given session's UTIL directories being created under the same parent directory, making it more efficient to size and easier to clean up, and the UTILLOC option pointing to a directory dedicated to that particular session.

An example way of doing this for a Workspace Server would be:

Unix

  1. Add the following to the WorkspaceServer_usermods.sh before the set -A USERMODS_OPTIONS= line modifying the values of ESMHOME and UTILROOT for your environment:
    # set a unique subdirectory for the util loc
    UTILROOT=/utilloc
    export ESMUTILLOC=$UTILROOT/$(hostname -s)_$$
    mkdir $ESMUTILLOC;
    export ESMUNQUTIL="-utilloc $ESMUTILLOC"
    
  2. Append the following to the USERMODS_OPTIONS[${#USERMODS_OPTIONS[*]}]= line in WorkspaceServer_usermods.sh:
    $ESMUNQUTIL
    
  3. Add the following immediately after the "%SAS_COMMAND%" %CMD_OPTIONS% in WorkspaceServer.bat:
    if [ -d $ESMUTILLOC ]; then rmdir $ESMUTILLOC; fi
    

Windows

  1. Add the following to the WorkspaceServer_usermods.bat before the Set USERMODS_OPTIONS= line modifying the values of ESMHOME and UTILROOT for your environment:
    set ESMHOME=C:\esm\esm-agent
    set UTILROOT=C:\SAS\Util
    for /f %%i in ('%ESMHOME%\jre\bin\java -cp %ESMHOME%\conf\ UUIDGen') do set RANDGUID=%%i
    echo %UTILROOT%\util_%RANDGUID%
    setx ESMUTILOC %UTILROOT%\util_%RANDGUID%
    if not exist %ESMUTILOC% mkdir %ESMUTILOC%
    
  2. Append the following to the Set USERMODS_OPTIONS= line in WorkspaceServer_usermods.bat:
    -utilloc %ESMUTILOC%
    
  3. Add the following immediately after the "%SAS_COMMAND%" %CMD_OPTIONS% in WorkspaceServer.bat:
    rmdir %ESMUTILOC% /s /q 
    

Warning

Unlike the other configuration changes presented in this guide, this change alters the standard SAS behaviour and is therefore not included as part of the standard ESM Configurator workflow. This change should be tested and documented. Use at your own risk.**

Real time monitoring of Workspace Server logs

It is sometimes desirable to be made aware of ERROR and WARNING messages as soon as they are emitted to the Workspace session log, often before they have appeared in the client UI. It may also be useful to review log messages from a previous Workspace session execution, long after that session has completed. In these instances, ESM can be configured to monitor SAS Workspace Server logs and to highlight log messages such as ERROR and WARNING messages in real time.

To configure this functionality the following steps need to be taken:

  1. Enable and configure logging for the SAS Workspace Server (not enabled by default)
  2. Modify ESM Workspace Server configuration for SAS Workspace Server logging
  3. (Optional) Simplify formatting of the SAS Workspace Server log
  4. Log file management

Configure SAS Workspace Server Logging

SAS product documentation provides instructions for enabling Workspace Server logging.

Modify ESM Workspace Server configuration for SAS Workspace Server logging

In order for ESM to track the SAS Workspace Server logs that are now being created, the names of those logs need to be communicated to the ESM Agent.

The original configuration for SAS Workspace Servers involves adding an ESM configuration block to the WorkspaceServer_usermods file. Add the following lines inside the ESM CONFIG block, just before the ESM CONFIG END comment:

Slashes matter

The trailing slashes in the below LOGPATH variables are important. Don't forget to add them

export LOGPATH=/opt/sas/config/Lev1/SASApp/WorkspaceServer/Logs/
export ESM_APPSERVER_NAME=$(basename $APPSERVER_ROOT)
set LOGPATH=D:\SAS\Config\Lev1\SASApp\WorkspaceServer\Logs\
for /D %%i in (%APPSERVER_ROOT%) do set ESM_APPSERVER_NAME=%%~ni

(Optional) Simplify formatting of the SAS Workspace Server log

This step is optional

This step is not required by ESM. It makes the generated logfiles smaller in size, lessening the impact of enabling the logging on available storage. It also makes them easier to read.

Proceed with caution

This step makes a change to the logconfig.trace.xml. Modifications to this file, if not completed correctly, may cause issues on your system.

It is highly recommend that you familiarise yourself with the documentation for the SAS Logging Facility for your version of SAS, and ensure that you take a backup of any SAS configuration file before making changes.

As noted in the documentation on the SAS Support Site the primary purpose of enabling Workspace Server logging is to provide SAS Technical Support with high-level logging messages which can be used by SAS Technical Support to help resolve issue. As part of this information, SAS prepends a large amount of data to the beginning of each line of the SAS log:

2020-03-31T23:38:35,973 INFO  [00000006] :chris -  Linux LIN X64 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05UTC 2019
2020-03-31T23:38:35,973 INFO  [00000006] :chris -       x86_64 CentOS Linux release 7.6.1810 (Core)
2020-03-31T23:38:35,973 INFO  [00000006] :chris -
2020-03-31T23:38:35,973 INFO  [00000006] :chris - You are running SAS 9. Some SAS 8 files will be automatically converted
2020-03-31T23:38:35,974 INFO  [00000006] :chris - by the V9 engine; others are incompatible.  Please see
2020-03-31T23:38:35,974 INFO  [00000006] :chris - http://support.sas.com/rnd/migration/planning/platform/64bit.html

For most purposes, the bulk of this information is not relevant and makes the log file significantly larger and more difficult to read. A small modification to line 14 of the logconfig.trace.xml file removes all of the prepended information, leaving a more traditional and easier to read SAS log:

Modify:

<param name="ConversionPattern" value="%d %-5p [%t] %X{Client.ID}:%u - %m"/>

To:

<param name="ConversionPattern" value="%m"/>

Diffing the two files should yield results like below:

--- a/logconfig.trace.xml
+++ b/logconfig.xml
@@ -11,7 +11,7 @@
       </rollingPolicy>
       <layout>
          <param name="HeaderPattern" value="Host: '%S{hostname}', OS: '%S{os_family}', Release: '%S{os_release}', SAS
-         <param name="ConversionPattern" value="%d %-5p [%t] %X{Client.ID}:%u - %m"/>
+         <param name="ConversionPattern" value="%m"/>
       </layout>
    </appender>

A further small modification to Application message logger of the logconfig.trace.xml file to remove all of the Debug messages making the SAS log even easier to read:

Modify:

   <!-- Application message logger -->
   <logger name="App" immutability="true">
      <level value="Debug"/>
   </logger>

To:

   <!-- Application message logger -->
   <logger name="App" immutability="true">
      <level value="Info"/>
   </logger>

Diffing the two files should yield results like below:

--- a/logconfig.trace.xml
+++ b/logconfig.xml
@@ -11,7 +11,7 @@
   <logger name="App" immutability="true">
-      <level value="Debug"/>
+      <level value="Info"/>
   </logger>

Log file management

With a high number or interactive users or Workspace sessions, you may find that, even if you reduce the size of the log file the number and size of the log repositories can quickly grow. By default SAS creates workspace server logs in a Logs folder within the Workspace folder of your SAS configuration. If filling up this storage location with logfiles poses an issue, consider implementing one or more of the following:

  1. Redirection of log files to another disk. This is most easily accomplished by creating a symbolic link for the Logs directory to an alternate location. Redirecting the files in this way requires fewer SAS configuration file changes.
  2. Regularly delete old logs. For example, schedule a simple script to perform a housekeeping task on your system which deletes files older than a certain age.

The following delete files older than 14 days:

find /opt/sas/config/Lev1/SASApp/WorkspaceServer/Logs -type f -mtime +14 -exec rm -f {} \;
ForFiles /p "D:\SAS\Config\Lev1\SASApp\WorkspaceServer\Logs" /s /d -14 /c "cmd /c del @file"

Warning

The above commands are destructive and assume familiarity with system level maintenance routines.

Real time monitoring of Connect Server logs

It is sometimes desirable to be made aware of ERROR and WARNING messages as soon as they are emitted to the Connect Server session log, often before they have appeared in the client UI. It may also be useful to review log messages from a previous Connect Server session execution, long after that session has completed. In these instances, ESM can be configured to monitor SAS Connect Server logs and to highlight log messages such as ERROR and WARNING messages in real time.

To configure this functionality the following steps need to be taken:

  1. Enable and configure logging for the SAS Connect Server (not enabled by default)
  2. Modify ESM Connect Server configuration for SAS Connect Server logging
  3. (Optional) Simplify formatting of the SAS Connect Server log
  4. Log file management

Configure SAS Connect Server Logging

SAS product documentation provides instructions for enabling Connect Server logging.

Modify ESM Connect Server configuration for SAS Connect Server logging

In order for ESM to track the SAS Connect Server logs that are now being created, the names of those logs need to be communicated to the ESM Agent.

The original configuration for SAS Connect Server involves adding an ESM configuration block to the ConnectServer_usermods file. Add the following lines inside the ESM CONFIG block, just before the ESM CONFIG END comment:

Slashes matter

The trailing slashes in the below LOGPATH variables are important. Don't forget to add them

export LOGPATH=/opt/sas/config/Lev1/SASApp/ConnectServer/Logs/
export ESM_APPSERVER_NAME=$(basename $APPSERVER_ROOT)
set LOGPATH=D:\SAS\Config\Lev1\SASApp\ConnectServer\Logs\
for /D %%i in (%APPSERVER_ROOT%) do set ESM_APPSERVER_NAME=%%~ni

(Optional) Simplify formatting of the SAS Connect Server log

This step is optional

This step is not required by ESM. It makes the generated logfiles smaller in size, lessening the impact of enabling the logging on available storage. It also makes them easier to read.

Proceed with caution

This step makes a change to the logconfig.trace.xml. Modifications to this file, if not completed correctly, may cause issues on your system.

It is highly recommend that you familiarise yourself with the documentation for the SAS Logging Facility for your version of SAS, and ensure that you take a backup of any SAS configuration file before making changes.

As noted in the documentation on the SAS Support Site the primary purpose of enabling Connect Server logging is to provide SAS Technical Support with high-level logging messages which can be used by SAS Technical Support to help resolve issue. For most purposes, the bulk of this information is not relevant and makes the log file significantly larger and more difficult to read. A small modification to line 14 of the logconfig.trace.xml file removes all of the prepended information, leaving a more traditional and easier to read SAS log:

For the rolling file appender, modify the layout of the lines written to the log

<param name="ConversionPattern" value="TRACE"/>

To:

<param name="ConversionPattern" value="%m"/>

For the rolling file appender, modify the App.Connect and App.WQ loggers from Trace to Info:

    <logger name="App.Connect">
      <level value="Trace"/>
    </logger>

    <logger name="App.WQ">
      <level value="Trace"/>
    </logger>    

To:

    <logger name="App.Connect">
      <level value="Info"/>
    </logger>

    <logger name="App.WQ">
      <level value="Info"/>
    </logger>   

Diffing the two files should yield results like below:

--- a/logconfig.trace.xml.orig
+++ b/logconfig.trace.xml
@@ -11,7 +11,7 @@
       </rollingPolicy>
       <layout>
-         <param name="ConversionPattern" value="TRACE"/>
+         <param name="ConversionPattern" value="%m"/>
       </layout>
    </appender>

@@ -104,12 +104,12 @@ _ENDDISARM

    <!-- Connect Application logger -->
    <logger name="App.Connect">
-      <level value="Trace"/>
+      <level value="Info"/>
    </logger>

    <!-- WQ logger -->
    <logger name="App.WQ">
-      <level value="Trace"/>
+      <level value="Info"/>
    </logger>

Log file management

With a high number or interactive users or Connect sessions, you may find that, even if you reduce the size of the log file the number and size of the log repositories can quickly grow. By default SAS creates workspace server logs in a Logs folder within the Connect Server folder of your SAS configuration. If filling up this storage location with logfiles poses an issue, consider implementing one or more of the following:

  1. Redirection of log files to another disk. This is most easily accomplished by creating a symbolic link for the Logs directory to an alternate location. Redirecting the files in this way requires fewer SAS configuration file changes.
  2. Regularly delete old logs. For example, schedule a simple script to perform a housekeeping task on your system which deletes files older than a certain age.

The following delete files older than 14 days:

find /opt/sas/config/Lev1/SASApp/ConnectServer/Logs -type f -mtime +14 -exec rm -f {} \;
ForFiles /p "D:\SAS\Config\Lev1\SASApp\ConnectServer\Logs" /s /d -14 /c "cmd /c del @file"

Warning

The above commands are destructive and assume familiarity with system level maintenance routines.

Configuring Error and Warning event generation using ESM Logfile Markers

By default, ESM is configured to detect Errors and Warnings from SAS logs in the default SAS format, as generated by the Batch Server and SAS Foundation sessions. Following the steps above to reduce the size of the log file will simplify the logfiles generated by the Workspace Server SAS Logging facility so that they follow the same format, and ESM will begin to detect Error and Warning messages from the Workspace Server logs without any further configuration.

If you have previously configured Workspace Server logging without updating the logfile format, and do not wish to update that configuration, Workspace Server logs will still be available for viewing through the ESM interface. However some changes to the ESM log marker configuration will be required to enable the detection of Error and Warning messages that are prepended with a timestamp, or any othe properties specified in logconfig.xml.

Changes to the Log Marker configuration are made using the Log Markers configuration screen in the ESM Admin Settings section.

Active Process Discovery

ESM has two methods of process discovery that can be configured: active and passive. By default, only the passive method is enabled. During passive process discovery the ESM agents watch a configured directory (the ESMEVENTS folder) for a new event trigger file. When active process discovery is enabled the ESM agent actively scans the node's process table for processes that match a configured set of rules that are specified in a file called filterSpec.yaml which is found in the ESM agent's bin folder. Any processes that match the configured rules are added the the list of processes that are monitored by the ESM agent. Active and passive process discovery can run simultaneously.

Active vs Passive

The passive process discovery method guarantees that ESM creates a record of a process having run on system, even if that session was so short-lived that no metrics were captured about the session. The way that the passive process monitoring works requires the process that is going to be monitored be able to create the new event trigger file itself. Functions for SAS, shell scripts, and Python currently exist to provide this capability.

There are certain deployment scenarios where we might want to monitor processes that do not have the capability to create ESM new event files, or we may not want to touch the configuration of the environment to include the capability to generate the new event trigger files. In this scenario, we would want to use the active process discovery mechanism to monitor processes.

Active process discovery is enabled by uncommenting and setting the value of the monitorAllSessions to True. The monitorAllSessions property enables the active process discovery mode. monitorAllSessions is used in conjunction with monitoringInterval which specifies, in seconds, the interval at which the ESM agent scans the node's process table for processes and applies the rules specified in the filterSpec.yaml file.

filterSpec.yaml

The filterSpec.yaml file contains one or more process filter blocks. Each block contains a number of different properties that describe a set of rules that are applied to each process in the scanned node's process table. The blocks in the filterSpec file are parsed in sequential order and the first rule that matches a process is the rule that is applied.

filterSpec:
  - processFilterRegex: ^\/(.+\/)*(?<prog>[^\s]+)
    userFilterRegex: '' 
    ignoreMatching: False
    esmPattern:
      processName: ${prog}
      processType: unk
      processQueue: 
An example filterSpec.yaml file.

Property Sub-property Description Accepts variables? Example
processFilterRegex A regular expression that is applied against each process. If the regular expression matches, the rule block is tried. If it doesn't, the next rule in the file is applied. No ^\/(.+\/)*(?<prog>[^\s]+)
userFilterRegex A regular expression that is applied against the user of each process. You can specify '' to apply to all users. No ^svc_.*
processLogfile A regular expression that is applied against all of the file descriptors of a matched process. If a file is found, that file is passed as the associated log file of the process. Unix only. No .*WorkspaceServer_\w+.*
ignoreMatching A binary (True or False) option that controls whether if a process matches the rule block, the process is added to ESM. No False
esmPattern processName This is the name of the session as we want to see it in ESM. This can be hardcoded or you can use the results of a named capture group from the processFilterRegex using the special variable format ${var}. Yes ${prog}
esmPattern processType This is the name of the "Category" as we want to see it in ESM. This value can be hardcoded or you can use the results of a named capture group from the processFilterRegex using the special variable format ${var}. Yes SAS1
esmPattern processQueue This is the name of the "Queue" as we want to see it in ESM. This value can be hardcoded or you can use the results of a named capture group from the processFilterRegex using the special variable format ${var}. Yes normal

  1. There are a number of keywords that have a special meaning within ESM including: Batch, STP, WS, CS, GRID, Foundation, SPRE, viya, CMP. You should avoid using these for customer processes unless you are aware of the implications. Please contact technical support for more information: support@sas.com.