Advanced SAS Enterprise Session Monitor Configuration

Web Server reverse proxy configuration¶

In certain deployments, you may wish to configure the SAS Enterprise Session Monitor so that it interacts with an existing web server deployment. This section contains a sample configuration for running SAS Enterprise Session Monitor behind a ngnix web server:

Note

You will need to update the value of the server_name for your environment.

server {
    server_name esm-server.sasdemo.com;
    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass http://127.0.0.1:18080/;
        access_log  /var/log/nginx/esm.access.log  main;
        error_log /var/log/nginx/esm.error.log debug;
    }
}

Per-session tracking of SAS UTILLOC size¶

SAS multi-threaded procedures can make use of a disk paging location referred to as UTILLOC. SAS Enterprise Session Monitor is designed with the capability to track the size of this UTILLOC usage of each individual session, but to enable this functionality, some extra configuration is required.

Unlike the SAS WORK location, which is a directory created at session startup that is unique to each individual SAS session, SAS UTIL directories are created on-the-fly by SAS procedures as they are required. Therefore, unlike the WORK system option which holds the full path to each session's individual SASWORK directory, the UTILLOC system option points to the parent directory which can contain any number of UTIL directories belonging to a number of other sessions. This makes periodic size measurements of a session's individual set of UTIL directories challenging.

One way of addressing this issue is to create a per-session UTILLOC subdirectory for each session before it is started, using it's usermods initialisation script, and removing the directory after the session has finished executing. This results in all of a given session's UTIL directories being created under the same parent directory, making it more efficient to size and easier to clean up, and the UTILLOC option pointing to a directory dedicated to that particular session.

An example way of doing this for a Workspace Server would be:

Unix¶

Add the following to the WorkspaceServer_usermods.sh before the set -A USERMODS_OPTIONS= line modifying the values of ESMHOME and UTILROOT for your environment:

# set a unique subdirectory for the util loc
UTILROOT=/utilloc
export ESMUTILLOC=$UTILROOT/$(hostname -s)_$$
mkdir $ESMUTILLOC;
export ESMUNQUTIL="-utilloc $ESMUTILLOC"

Append the $ESMUNQUTIL variable to the setting of the USERMODS_OPTIONS line in WorkspaceServer_usermods.sh:
```
USERMODS_OPTIONS[${#USERMODS_OPTIONS[*]}]=$ESMUNQUTIL
```
Remove the exec from the eval exec $cmd line towards the bottom of the file. After the change, it should read:
```
eval $cmd
```
Add the following immediately after the eval $cmd in WorkspaceServer.bat:
```
if [ -d "$ESMUTILLOC" ]; then rmdir "$ESMUTILLOC"; fi
```

Windows¶

Add the following to the WorkspaceServer_usermods.bat before the Set USERMODS_OPTIONS= line modifying the values of ESMHOME and UTILROOT for your environment:

set ESMHOME=C:\esm\esm-agent
set UTILROOT=C:\SAS\Util
for /f %%i in ('%ESMHOME%\jre\bin\java -cp %ESMHOME%\conf\ UUIDGen') do set RANDGUID=%%i
echo %UTILROOT%\util_%RANDGUID%
setx ESMUTILOC %UTILROOT%\util_%RANDGUID%
if not exist %ESMUTILOC% mkdir %ESMUTILOC%

Append the following to the Set USERMODS_OPTIONS= line in WorkspaceServer_usermods.bat:
```
-utilloc %ESMUTILOC%
```
Add the following immediately after the "%SAS_COMMAND%" %CMD_OPTIONS% in WorkspaceServer.bat:
```
rmdir %ESMUTILOC% /s /q 
```

Warning

Unlike the other configuration changes presented in this guide, this change alters the standard SAS behaviour and is therefore not included as part of the standard SAS Enterprise Session Monitor Configurator workflow. This change should be tested and documented. Use at your own risk.**

Real-time monitoring of Workspace Server logs¶

It is sometimes desirable to be made aware of ERROR and WARNING messages as soon as they are emitted to the Workspace session log, often before they have appeared in the client UI. It may also be useful to review log messages from a previous Workspace session execution, long after that session has completed. In these instances, SAS Enterprise Session Monitor can be configured to monitor SAS Workspace Server logs and to highlight log messages such as ERROR and WARNING messages in real time.

To configure this functionality the following steps need to be taken:

Enable and configure logging for the SAS Workspace Server (not enabled by default)
Modify SAS Enterprise Session Monitor Workspace Server configuration for SAS Workspace Server logging
(Optional) Simplify formatting of the SAS Workspace Server log
Log file management

Configure SAS Workspace Server Logging¶

SAS product documentation provides instructions for enabling Workspace Server logging.

Modify SAS Enterprise Session Monitor Workspace Server configuration for SAS Workspace Server logging¶

In order for SAS Enterprise Session Monitor to track the SAS Workspace Server logs that are now being created, the names of those logs need to be communicated to the SAS Enterprise Session Monitor agent.

The original configuration for SAS Workspace Servers involves adding a SAS Enterprise Session Monitor configuration block to the WorkspaceServer_usermods file. Add the following lines inside the ESM CONFIG block, just before the ESM CONFIG END comment:

Slashes matter

The trailing slashes in the below LOGPATH variables are important. Don't forget to add them

UnixWindows

export LOGPATH=/opt/sas/config/Lev1/SASApp/WorkspaceServer/Logs/
export ESM_APPSERVER_NAME=$(basename $APPSERVER_ROOT)

set LOGPATH=D:\SAS\Config\Lev1\SASApp\WorkspaceServer\Logs\
for /D %%i in (%APPSERVER_ROOT%) do set ESM_APPSERVER_NAME=%%~ni

(Optional) Simplify formatting of the SAS Workspace Server log¶

This step is optional

This step is not required by SAS Enterprise Session Monitor. It makes the generated logfiles smaller in size, lessening the impact of enabling the logging on available storage. It also makes them easier to read.

Proceed with caution

This step makes a change to the logconfig.trace.xml. Modifications to this file, if not completed correctly, may cause issues on your system.

It is highly recommended that you familiarise yourself with the documentation for the SAS Logging Facility for your version of SAS, and ensure that you take a backup of any SAS configuration file before making changes.

As noted in the documentation on the SAS Support Site the primary purpose of enabling Workspace Server logging is to provide SAS Technical Support with high-level logging messages which can be used by SAS Technical Support to help resolve issue. As part of this information, SAS prepends a large amount of data to the beginning of each line of the SAS log:

2020-03-31T23:38:35,973 INFO  [00000006] :chris -  Linux LIN X64 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05UTC 2019
2020-03-31T23:38:35,973 INFO  [00000006] :chris -       x86_64 CentOS Linux release 7.6.1810 (Core)
2020-03-31T23:38:35,973 INFO  [00000006] :chris -
2020-03-31T23:38:35,973 INFO  [00000006] :chris - You are running SAS 9. Some SAS 8 files will be automatically converted
2020-03-31T23:38:35,974 INFO  [00000006] :chris - by the V9 engine; others are incompatible.  Please see
2020-03-31T23:38:35,974 INFO  [00000006] :chris - http://support.sas.com/rnd/migration/planning/platform/64bit.html

For most purposes, the bulk of this information is not relevant and makes the log file significantly larger and more difficult to read. A small modification to line 14 of the logconfig.trace.xml file removes all of the prepended information, leaving a more traditional and easier to read SAS log:

Modify:

<param name="ConversionPattern" value="%d %-5p [%t] %X{Client.ID}:%u - %m"/>

To:

<param name="ConversionPattern" value="%m"/>

Diffing the two files should yield results like below:

--- a/logconfig.trace.xml
+++ b/logconfig.xml
@@ -11,7 +11,7 @@
       </rollingPolicy>
       <layout>
          <param name="HeaderPattern" value="Host: '%S{hostname}', OS: '%S{os_family}', Release: '%S{os_release}', SAS
-         <param name="ConversionPattern" value="%d %-5p [%t] %X{Client.ID}:%u - %m"/>
+         <param name="ConversionPattern" value="%m"/>
       </layout>
    </appender>

A further small modification to Application message logger of the logconfig.trace.xml file to remove all of the Debug messages making the SAS log even easier to read:

Modify:

   <!-- Application message logger -->
   <logger name="App" immutability="true">
      <level value="Debug"/>
   </logger>

To:

   <!-- Application message logger -->
   <logger name="App" immutability="true">
      <level value="Info"/>
   </logger>

Diffing the two files should yield results like below:

--- a/logconfig.trace.xml
+++ b/logconfig.xml
@@ -11,7 +11,7 @@
   <logger name="App" immutability="true">
-      <level value="Debug"/>
+      <level value="Info"/>
   </logger>

Log file management¶

With a high number or interactive users or Workspace sessions, you may find that, even if you reduce the size of the log file the number and size of the log repositories can quickly grow. By default SAS creates workspace server logs in a Logs folder within the Workspace folder of your SAS configuration. If filling up this storage location with logfiles poses an issue, consider implementing one or more of the following:

Redirection of log files to another disk. This is most easily accomplished by creating a symbolic link for the Logs directory to an alternate location. Redirecting the files in this way requires fewer SAS configuration file changes.
Regularly delete old logs. For example, schedule a simple script to perform a housekeeping task on your system which deletes files older than a certain age.

The following delete files older than 14 days:

Unix

```sh

find /opt/sas/config/Lev1/SASApp/WorkspaceServer/Logs -type f -mtime +14 -exec rm -f {} \;

=== "Windows"

    ```bat
ForFiles /p "D:\SAS\Config\Lev1\SASApp\WorkspaceServer\Logs" /s /d -14 /c "cmd /c del @file"

Warning

The above commands are destructive and assume familiarity with system-level maintenance routines.

Real-time monitoring of SAS/CONNECT server logs¶

It is sometimes desirable to be made aware of ERROR and WARNING messages as soon as they are emitted to the SAS/CONNECT server session log, often before they have appeared in the client UI. It may also be useful to review log messages from a previous SAS/CONNECT server session execution, long after that session has completed. In these instances, SAS Enterprise Session Monitor can be configured to monitor SAS/CONNECT server logs and to highlight log messages such as ERROR and WARNING messages in real time.

To configure this functionality the following steps need to be taken:

Enable and configure logging for the SAS/CONNECT server (not enabled by default)
Modify the SAS Enterprise Session Monitor SAS/CONNECT server configuration for SAS/CONNECT server logging
(Optional) Simplify formatting of the SAS/CONNECT server log
Log file management

Configure SAS/CONNECT Server Logging¶

SAS product documentation provides instructions for enabling SAS/CONNECT server logging.

Modify the SAS Enterprise Session Monitor SAS/CONNECT server configuration for SAS/CONNECT server logging¶

In order for SAS Enterprise Session Monitor to track the SAS/CONNECT server logs that are now being created, the names of those logs need to be communicated to the SAS Enterprise Session Monitor agent.

The original configuration for SAS/CONNECT server involves adding a SAS Enterprise Session Monitor configuration block to the ConnectServer_usermods file. Add the following lines inside the ESM CONFIG block, just before the ESM CONFIG END comment:

Slashes matter

The trailing slashes in the below LOGPATH variables are important. Don't forget to add them

Unix

```sh

export LOGPATH=/opt/sas/config/Lev1/SASApp/ConnectServer/Logs/ export ESM_APPSERVER_NAME=$(basename $APPSERVER_ROOT)

=== "Windows"

    ```bat
set LOGPATH=D:\SAS\Config\Lev1\SASApp\ConnectServer\Logs\
for /D %%i in (%APPSERVER_ROOT%) do set ESM_APPSERVER_NAME=%%~ni

(Optional) Simplify formatting of the SAS/CONNECT server log¶

This step is optional

This step is not required by SAS Enterprise Session Monitor. It makes the generated logfiles smaller in size, lessening the impact of enabling the logging on available storage. It also makes them easier to read.

Proceed with caution

This step makes a change to the logconfig.trace.xml. Modifications to this file, if not completed correctly, may cause issues on your system.

It is highly recommended that you familiarise yourself with the documentation for the SAS Logging Facility for your version of SAS, and ensure that you take a backup of any SAS configuration file before making changes.

As noted in the documentation on the SAS Support Site the primary purpose of enabling SAS/CONNECT server logging is to provide SAS Technical Support with high-level logging messages which can be used by SAS Technical Support to help resolve issue. For most purposes, the bulk of this information is not relevant and makes the log file significantly larger and more difficult to read. A small modification to line 14 of the logconfig.trace.xml file removes all of the prepended information, leaving a more traditional and easier to read SAS log:

For the rolling file appender, modify the layout of the lines written to the log

<param name="ConversionPattern" value="TRACE"/>

To:

<param name="ConversionPattern" value="%m"/>

For the rolling file appender, modify the App.Connect and App.WQ loggers from Trace to Info:

    <logger name="App.Connect">
      <level value="Trace"/>
    </logger>

    <logger name="App.WQ">
      <level value="Trace"/>
    </logger>

To:

    <logger name="App.Connect">
      <level value="Info"/>
    </logger>

    <logger name="App.WQ">
      <level value="Info"/>
    </logger>

Diffing the two files should yield results like below:

--- a/logconfig.trace.xml.orig
+++ b/logconfig.trace.xml
@@ -11,7 +11,7 @@
       </rollingPolicy>
       <layout>
-         <param name="ConversionPattern" value="TRACE"/>
+         <param name="ConversionPattern" value="%m"/>
       </layout>
    </appender>

@@ -104,12 +104,12 @@ _ENDDISARM

    <!-- Connect Application logger -->
    <logger name="App.Connect">
-      <level value="Trace"/>
+      <level value="Info"/>
    </logger>

    <!-- WQ logger -->
    <logger name="App.WQ">
-      <level value="Trace"/>
+      <level value="Info"/>
    </logger>

Log file management¶

With a high number or interactive users or Connect sessions, you may find that, even if you reduce the size of the log file the number and size of the log repositories can quickly grow. By default SAS creates workspace server logs in a Logs folder within the SAS/CONNECT server folder of your SAS configuration. If filling up this storage location with logfiles poses an issue, consider implementing one or more of the following:

Redirection of log files to another disk. This is most easily accomplished by creating a symbolic link for the Logs directory to an alternate location. Redirecting the files in this way requires fewer SAS configuration file changes.
Regularly delete old logs. For example, schedule a simple script to perform a housekeeping task on your system which deletes files older than a certain age.

The following delete files older than 14 days:

Unix

```sh

find /opt/sas/config/Lev1/SASApp/ConnectServer/Logs -type f -mtime +14 -exec rm -f {} \;

=== "Windows"

    ```bat
ForFiles /p "D:\SAS\Config\Lev1\SASApp\ConnectServer\Logs" /s /d -14 /c "cmd /c del @file"

Warning

The above commands are destructive and assume familiarity with system-level maintenance routines.

Configuring Error and Warning event generation using SAS Enterprise Session Monitor Logfile Markers¶

By default, SAS Enterprise Session Monitor is configured to detect Errors and Warnings from SAS logs in the default SAS format, as generated by the Batch Server and SAS Foundation sessions. Following the steps above to reduce the size of the log file will simplify the logfiles generated by the Workspace Server SAS Logging facility so that they follow the same format, and SAS Enterprise Session Monitor will begin to detect Error and Warning messages from the Workspace Server logs without any further configuration.

If you have previously configured Workspace Server logging without updating the logfile format, and do not wish to update that configuration, Workspace Server logs will still be available for viewing through the SAS Enterprise Session Monitor interface. However some changes to the SAS Enterprise Session Monitor log marker configuration will be required to enable the detection of Error and Warning messages that are prepended with a timestamp, or any other properties specified in logconfig.xml.

Changes to the Log Marker configuration are made using the Log Markers configuration screen in the SAS Enterprise Session Monitor Admin Settings section.

Active Process Discovery¶

SAS Enterprise Session Monitor has two methods of process discovery that can be configured: active and passive. By default, only the passive method is enabled. During passive process discovery the SAS Enterprise Session Monitor agents watch a configured directory (the ESMEVENTS folder) for a new event trigger file. When active process discovery is enabled the SAS Enterprise Session Monitor agent actively scans the node's process table for processes that match a configured set of rules that are specified in a file called filterSpec.yaml which is found in the SAS Enterprise Session Monitor agent's bin folder. Any processes that match the configured rules are added to the list of processes that are monitored by the SAS Enterprise Session Monitor agent. Active and passive process discovery can run simultaneously.

Active vs Passive¶

The passive process discovery method guarantees that SAS Enterprise Session Monitor creates a record of a process having run on system, even if that session was so short-lived that no metrics were captured about the session. The way that the passive process monitoring works requires the process that is going to be monitored to be able to create the new event trigger file itself. Functions for SAS, shell scripts, and Python currently exist to provide this capability.

There are certain deployment scenarios where we might want to monitor processes that do not have the capability to create SAS Enterprise Session Monitor new event files, or we may not want to touch the configuration of the environment to include the capability to generate the new event trigger files. In this scenario, we would want to use the active process discovery mechanism to monitor processes.

Active process discovery is enabled by uncommenting and setting the value of the monitorAllSessions to True. The monitorAllSessions property enables the active process discovery mode. monitorAllSessions is used in conjunction with monitoringInterval which specifies, in seconds, the interval at which the SAS Enterprise Session Monitor agent scans the node's process table for processes and applies the rules specified in the filterSpec.yaml file.

filterSpec.yaml¶

The filterSpec.yaml file contains one or more process filter blocks. Each block contains a number of different properties that describe a set of rules that are applied to each process in the scanned node's process table. The blocks in the filterSpec file are parsed in sequential order and the first rule that matches a process is the rule that is applied.

filterSpec:
  - processFilterRegex: ^\/(.+\/)*(?<prog>[^\s]+)
    userFilterRegex: '' 
    ignoreMatching: False
    esmPattern:
      processName: ${prog}
      processType: unk
      processQueue:

An example filterSpec.yaml file.

Property	Sub-property	Description	Accepts variables?	Example
`processFilterRegex`		A regular expression that is applied against each process. If the regular expression matches, the rule block is tried. If it doesn't, the next rule in the file is applied.	No	`^\/(.+\/)*(?<prog>[^\s]+)`
`userFilterRegex`		A regular expression that is applied against the user of each process. You can specify `''` to apply to all users.	No	`^svc_.*`
`processLogfile`		A regular expression that is applied against all of the file descriptors of a matched process. If a file is found, that file is passed as the associated log file of the process. Unix only.	No	`.WorkspaceServer_\w+.`
`ignoreMatching`		A binary (`True` or `False`) option that controls whether if a process matches the rule block, the process is added to SAS Enterprise Session Monitor.	No	`False`
`esmPattern`	`processName`	This is the name of the session as we want to see it in SAS Enterprise Session Monitor. This can be hardcoded or you can use the results of a named capture group from the `processFilterRegex` using the special variable format `${var}`.	Yes	`${prog}`
`esmPattern`	`processType`	This is the name of the "Category" as we want to see it in SAS Enterprise Session Monitor. This value can be hardcoded or you can use the results of a named capture group from the `processFilterRegex` using the special variable format `${var}`.	Yes	`SAS`¹
`esmPattern`	`processQueue`	This is the name of the "Queue" as we want to see it in SAS Enterprise Session Monitor. This value can be hardcoded or you can use the results of a named capture group from the `processFilterRegex` using the special variable format `${var}`.	Yes	`normal`

There are a number of keywords that have a special meaning within SAS Enterprise Session Monitor including: Batch, STP, WS, CS, GRID, Foundation, SPRE, viya, CMP. You should avoid using these for customer processes unless you are aware of the implications. Please contact technical support for more information: support@sas.com. ↩