Maintenance Utility
Arguments
| Option |
Description |
-h OR --help |
Show this help message and exit |
--verbose |
When enabled, additional logger is displayed. |
--check-partitions |
Check the ESM database for issues with the partitioning process. |
--fix-partitions |
Fix the ESM database for issues with the partitioning process. |
--no-check |
When fixing partitions, skip the check of the partitions. |
--migrate-session-id |
Update the session ID used by ESM between 2020.x versions and later versions of the software. |
--temp-dir |
The folder where temporary data will be stored. A transient folder will be created in this location. |
--migrate-diskio |
Migrate existing server device Disk IO metrics from Java byte arrays to JSON objectsfor use outside of ESM. |
--extract-data |
Extract data from the ESM database to be used for data analysis purposes. |
--obfuscate-users |
Used with --extract-data. Obfuscate username data from the data extract files |
--obfuscate-hosts |
Used with --extract-data. Obfuscate hostname data from the data extract. |
--max-age |
Used with --extract-data. The max age of the data to be retrieved in days. A value of 30 would mean data only data 30 days old or less being extracted. |
--compress-data |
Used with --extract-data. Specifies whether output datasets should be compressed. Default is True. You can just create CSV files, and not compress to a tar.gz file by specifying --compress-data=False. |
--server-data-only |
used with --extract-data. Specifies whether to just export data at the server |
--these-sessions |
used with --extract-data. Specifies a comma separated list of session data to export. |
--high-detail-output |
used with --extract-data. Specifies whether to only export very granular level data. |
--high-detail-start |
used with --extract-data and --high-detail-output |
--high-detail-end |
used with --extract-data and --high-detail-output |
Configuration Reference
| Setting |
Sub-Setting |
Type |
Description |
settings |
esmHome |
String |
The location of the"esm-server" of your SAS Enterprise Session Monitor Server installation. |
settings |
domain_xml_path |
String |
For 2022.x and below Server versions this is the location of the domain.xml within the Enterprise Session Monitor Server installation. The file contains key environment information required by the utility. |
settings |
setenv |
String |
This is the path to the setenv.sh script file which contains key environment information required by the utility. |
partitioned_tables |
name |
String |
The name of the parent table. |
partitioned_tables |
partition |
String |
The name of the column on which the table is partitioned. |
partitioned_tables |
retention |
String |
The retention period of that table. This value should match what you have configured within the SAS Enterprise Session Monitor user interface. |
partitioned_tables |
schema |
String |
The schema in which the table is located. |
aggregations |
|
String |
List of table suffixes where aggregation rules exist |
logLevel |
|
String |
Sets the logLevel of the application. This value is overridden if the --verbose command line option is passed. |
keepSourceFiles |
|
String |
Specifies whether or not to remove the CSV files are they have been added to the main archive. Default is True. |
splitSizeMB |
|
Integer |
Value, in MB, after which the created compressed TAR file will be split into parts. For example, if the the value was 10 and there was a 100MB file, you would get ten 10MB files with an extension of _part[1-10].
This is used because of limits uploading files to SharePoint using the Web Interface. |
Data Dictionary
esm_node.csv[.gz]
| Column Name |
Data Type |
Description |
Example |
| hostname |
string |
The name of the server for which the measures apply |
server1 |
| esm_group |
string |
The name of the group for which this host applies. The grouping of servers is done through the "admin settings" section of the ESM user interface. Typically, this would be the name of the environment to which this host belongs. |
Production |
| core_count |
numeric |
The number of physical cores addressable by the operating system. If this machine is a virtual machine, this is the number of virtual cores. |
8 |
esm_session.csv[.gz]
| Column Name |
Data Type |
Description |
Example |
| id |
numeric |
Unique identifier for a session within this ESM server instance |
123456 |
| avg_cpu |
numeric |
The average CPU utilization for this session as a percentage * 100.0 |
12.678 |
| end_time |
Datetime |
The finishing time of the session in UTC |
2022-03-06 21:28:38.229 |
| esm_type |
string |
ESM classification for the session |
WS |
| hostname |
string |
The name of the server for which the measures apply |
server1 |
| job_name |
string |
The name of the session. This could be program name of another configured name based on other session properties like, Lev or application server context. |
Lev1_SASApp |
| log_file |
string |
Full path to the log file associated with this session |
/var/logs/workspace/workspace_server_1234.log |
| max_mem |
numeric |
Peak memory utilization for the session in MB |
41.25 |
| max_util |
numeric |
If specified, the peak size of the SAS Util location for this session. If esm_type is CASC or CASW this is the in-memory CAS cache utilization. Measured in MB. |
165.66 |
| max_work |
numeric |
If specified, the peak size of the SAS Work location for this session. If esm_type is CASC or CASW this is the total disk based CAS cache utilization. Measured in MB. |
165.66 |
| esm_user |
string |
The username of the user that owns this session |
user1 |
| pid |
numeric |
The process ID of this session. |
1234 |
| queue |
string |
In a SAS Grid deployment this value is derived from the LSB_QUEUE environment variable. In a Viya 3.x environment, it is often used to identify tenants in a multi-tenant environment. |
default |
| sas_uuid |
string |
Unique identifier for a SAS Program generated by ESM. |
891273494 |
| start_time |
Datetime |
The starting time of the session in UTC |
2022-03-06 21:26:38.229 |
| util_folder |
string |
The full path to the Util folder of the session if specified. |
/tmp/SAS_Util_1234 |
| work_folder |
string |
The full path to the Work folder of the session if specified. |
/tmp/SAS_Work_1234 |
| total_bytes_read |
string |
The total number of MB read by the session during its lifetime |
234.67 |
| total_bytes_written |
string |
The total number of MB written by the session during its lifetime |
123.44 |
| avg_memory |
numeric |
The average amount of memory used by the session in MB |
34.23 |
| total_log_lines |
numeric |
The total number of log lines written by the process |
1231123 |
| total_duration |
numeric |
The total runtime of the session in Milliseconds |
1232.01 |
| peak_cpu |
numeric |
The peak CPU utilization of the process as a percentage of a single CPU core * 100.0 |
99.02 |
| sum_cpu |
numeric |
To total amount of CPU utilization used |
|
| processor |
numeric |
The ID of the process onto which the process was originally scheduled. |
|
| priority |
numeric |
The value that is used by the operating system's kernel to schedule tasks to the processor |
|
| nice |
numeric |
A user-space value that is used to control the priority of a process |
|
| ppid |
numeric |
The process ID of the parent process to this session. |
|
esm_job.csv[.gz]
| Column Name |
Data Type |
Description |
Example |
| from_date |
Datetime |
The starting time of the session in UTC |
2022-03-06 21:28:38.229 |
| hostname |
string |
The name of the server for which the measures apply |
server1 |
| pid |
numeric |
The process ID of this session. |
|
| to_date |
Datetime |
The finishing time of the session in UTC |
2022-03-06 21:28:38.229 |
| avg_cpu |
numeric |
The average CPU utilization for this session as a percentage * 100.0 |
12.678 |
| avg_mem |
numeric |
The average amount of memory used by the session in MB |
34.23 |
| color |
|
|
|
| description |
|
|
|
| flow |
string |
This value is typically taken from the LSB_JOBNAME environment variable. |
1:user1:MAIN_FLOW:DEPARTMENT_SUBFLOW:STAGE_FLOW:my-big_program.sas |
| job_name |
string |
The name of the session. This could be program name of another configured name based on other session properties like, Lev or application server context. |
my-big_program.sas |
| max_cpu |
|
|
|
| max_mem |
numeric |
Peak memory utilization for the session in MB |
41.25 |
| max_util |
numeric |
If specified, the peak size of the SAS Util location for this session. If esm_type is CASC or CASW this is the in-memory CAS cache utilization. Measured in MB. |
165.66 |
| max_work |
numeric |
If specified, the peak size of the SAS Work location for this session. If esm_type is CASC or CASW this is the total disk based CAS cache utilization. Measured in MB. |
165.66 |
| queue |
|
|
|
| return_code |
numeric |
The exit code of the job. See here for details of codes. |
0 |
| sum_cpu |
numeric |
To total amount of CPU utilization used |
|
| esm_type |
string |
ESM classification for the session |
Batch |
| esm_user |
string |
The username of the user that owns this session |
user1 |
| uuid |
|
|
|
| total_bytes_read |
string |
The total number of MB read by the session during its lifetime |
234.67 |
| total_bytes_written |
string |
The total number of MB written by the session during its lifetime |
123.44 |
| total_log_lines |
numeric |
The total number of log lines written by the process |
1231123 |
| total_duration |
numeric |
The total runtime of the session in Miliseconds |
2000.0 |
| peak_cpu |
|
|
|
esm_events_cas_actions.csv[.gz]
| Column Name |
Data Type |
Description |
Example |
| session_id |
numeric |
Unique identifier for a session within this ESM server instance. Joins to "id" on esm_session. |
123456 |
| hostname |
string |
The name of the server for which the measures apply |
server1 |
| esm_user |
string |
The username of the user that owns this session |
user1 |
| action_class |
string |
The CAS action group |
simple |
| action_name |
string |
The CAS action name |
summary |
| action_duration |
numeric |
The length of time, in miliseconds that the action took to complete |
123.67 |
| action_start |
datetime |
The start time of the CAS action (UTC) |
2022-03-06 21:28:38.229 |
| action_end |
datetime |
The end time of the CAS action (UTC) |
2022-03-06 21:28:38.229 |
esm_events_sas_procs.csv[.gz]
| Column Name |
Data Type |
Description |
Example |
| session_id |
numeric |
Unique identifier for a session within this ESM server instance. Joins to "id" on esm_session. |
123456 |
| hostname |
string |
The name of the server for which the measures apply |
server1 |
| esm_user |
string |
The username of the user that owns this session |
2022-03-06 21:28:38.229 |
| start_timestamp |
datetime |
The start time of the SAS procedure (UTC) |
|
| end_timestamp |
datetime |
The end time of the SAS procedure (UTC) |
2022-03-06 21:28:38.229 |
| procedure |
string |
The name of the SAS procedure executed |
PROC SQL |
| real_time |
numeric |
The amount of time that the procedure took to run |
14.24 |
esm_events_data_access.csv[.gz]
| Column Name |
Data Type |
Description |
Example |
| session_id |
numeric |
Unique identifier for a session within this ESM server instance. Joins to "id" on esm_session. |
123456 |
| time_stamp |
datetime |
The datetime that the table was opened to perform the action (read/write) |
2022-03-06 21:28:38.229 |
| hostname |
string |
The name of the server for which the measures apply |
server1 |
| table |
string |
The name of the table within the SAS library |
MYDATATABLE |
| libref |
string |
The SAS library reference used within the SAS code |
MYLIB |
| engine |
string |
The access engine that was used to access the data. |
V9 |
| mode |
string |
Whether the table was opened for input (read) or output (write) |
OUTPUT |
| path |
string |
For SAS data sets, the folder in which the file is located, for database tables, the server name / IP from where the table came |
/path/to/my/data |
esm_events_logevents.csv[.gz]
| Column Name |
Data Type |
Description |
Example |
| session_id |
numeric |
Unique identifier for a session within this ESM server instance. Joins to "id" on esm_session. |
123456 |
| time_stamp |
datetime |
The datetime that the table was opened to perform the action (read/write) |
2022-03-06 21:28:38.229 |
| pod_id |
string |
|
|
| container_id |
string |
|
|
| hostname |
string |
The name of the server for which the measures apply |
server1 |
| esm_user |
string |
The username of the user that owns this session |
2022-03-06 21:28:38.229 |
| event_type |
string |
The type of event. Default is 'logEvent' |
logEvent |
| event_level |
string |
The level of the log event (error or warning) |
error |
| message |
string |
The contents of the log message |
This is an error message |
| line |
numeric |
The line number within the log file from which the error came |
123 |
| log_file_name |
string |
Full path to the log file where the error came from |
/path/to/the/logfile.log |
| Column Name |
Data Type |
Description |
Example |
| session_id |
numeric |
Unique identifier for a session within this ESM server instance. Joins to "id" on esm_session. |
123456 |
| time_stamp |
datetime |
The datetime that the table was opened to perform the action (read/write) |
2022-03-06 21:28:38.229 |
| pod_id |
string |
|
|
| container_id |
string |
|
|
| esm_user |
string |
The name of the server for which the measures apply |
user1 |
| hostname |
string |
The username of the user that owns this session |
server1 |
| event_type |
string |
The type of event. Default is 'logEvent' |
tag |
| event_level |
string |
The level of the log event (error or warning) |
tag |
| chart_annotation |
string |
The text that appears on the ESM resource utilization charts |
Tag 1 |
| message |
numeric |
The text that appears when you hover over the tag within ESM |
This is a tag in a program |
esm_servers_stats_agg_min[.gz]
Unless otherwise stated, each of the numeric measures is an average (using the Postgres AVG() function) of the max detail data collected for that minute.
| Column Name |
Data Type |
Description |
Example |
| hostname |
string |
The name of the server for which the measures apply |
server1 |
| time_stamp |
datetime |
The minute for which the measures apply. (UTC) |
2022-03-06 21:16:00 |
| buff_cache_mem |
numeric |
The amount, in MB, of physical memory that is being used by the operating system as cache for speeding up file-system operations. |
6138.562333 |
| cpu_usage |
numeric |
The percentage of overall CPU utilization * 100.0 |
13.117667 |
| cpu_wait_usage |
numeric |
The percentage of overall CPU wait * 100.0. |
0.498000 |
| mem_info |
numeric |
The amount of "physical" memory used. |
34.998078 |
| swap_mem_used |
numeric |
The amount of virtual memory used (physical + swap). |
29849.643667 |
| total_memory |
numeric |
The total amount of available memory in MB as addressable by the operating system |
65467.28 |
| total_swap |
numeric |
Linux: The size of the SWAP space used.
Windows: The total amount of Virtual Memory (Physical memory + Page File). |
79542 |
esm_process_stats_agg_min.csv[.gz]
Unless otherwise stated, each of the numeric measures is an average (using the Postgres AVG() function) of the max detail data collected for that minute.
| Column Name |
Data Type |
Description |
Example |
| hostname |
string |
The name of the server for which the measures apply |
server1 |
| pid |
int |
The process ID of this session. |
1234 |
| time_stamp |
datetime |
The minute for which the measures apply. (UTC) |
2022-03-06 21:16:00 |
| bytes_read |
numeric |
|
|
| bytes_written |
numeric |
|
|
| cpu_usage |
numeric |
The percentage of overall CPU utilization * 100.0. This measure is as a % of one core. |
|
| dead |
boolean |
|
|
| job_name |
string |
The name of the session. This could be program name of another configured name based on other session properties like, Lev or application server context. |
|
| log_line |
numeric |
|
|
| mem_actual |
numeric |
The virtual memory allocated to the session. |
|
| mem_info |
numeric |
The actual memory used by the session. |
|
| server_name |
numeric |
Unused |
|
| temp_size |
numeric |
The size of the SAS work area or in-memory CAS cache. |
|
| esm_type |
string |
ESM classification for the session |
|
| esm_user |
string |
The username of the user that owns this session |
|
| util_size |
numeric |
The size of the SAS UTIL area for the session or the size of the disk-based CAS cache. |
|
| total_bytes_read |
string |
The total number of MB read by the session during its lifetime |
234.67 |
| total_bytes_written |
string |
The total number of MB written by the session during its lifetime |
123.44 |
| total_log_lines |
numeric |
The total number of log lines written by the process |
1231123 |
| total_duration |
numeric |
The total runtime of the session in Miliseconds |
2000.0 |
| avg_memory |
numeric |
|
|
| peak_cpu |
numeric |
|
|
| cpu_io_ratio |
numeric |
|
|
| session_id |
int |
|
|
| threads |
numeric |
|
|
| major_faults |
numeric |
|
|
| minor_faults |
numeric |
|
|
| share |
numeric |
|
|
| user_cpu |
numeric |
|
|
| sys_cpu |
numeric |
|
|
esm_servers_diskio_agg_min.csv[.gz]
| Column Name |
Data Type |
Description |
Example |
| hostname |
string |
The name of the server for which the measures apply |
server1 |
| time_stamp |
datetime |
The time when the measurement was taken |
2022-03-07 00:00:00 |
| diskname |
string |
The name of the disk device |
sdb |
| read_kb |
numeric |
The average throughput rate for the device for the previous minute |
15.125770 |
| write_kb |
numeric |
The average write throughput rate for the device for the previous minute |
44.862402 |
| diskqueue |
numeric |
The average disk queue length for the device for the previous minute |
0.010277 |
DONOTSEND_user_hash_mapping.csv[.gz]
| Column Name |
Data Type |
Description |
Example |
| username |
string |
The actual username that was captured in ESM |
chris |
| hash |
string |
The obfuscated username as it will appear in reports based upon this data. |
user1 |
DONOTSEND_node_hash_mapping.csv[.gz]
| Column Name |
Data Type |
Description |
Example |
| hostname |
string |
The actual name of the node that was captured in ESM |
vexample01sas01.test.sas.com |
| hash |
string |
The obfuscated username as it will appear in reports based upon this data. |
server1 |
Example Usage
Export data to a tar.gz file in /tmp for the past 30 days from the per-minute aggregate tables obfuscating usernames:
./esm-maintenance --extract-data --temp-dir /tmp --obfuscate-users --max-age 30
Export data to a tar.gz file in /tmp from all ESM process level tables for sessions with IDs 1234, 4567, and 8967:
./esm-maintenance --extract-data --temp-dir /tmp --these-sessions 1234,4567,8967
Export data to a tar.gz file in /tmp at 2s granularity for servers and processes between 18:00 UTC and 18:10 UTC on 08MAR2022
./esm-maintenance --extract-data --temp-dir /tmp --high-detail-output --high-detail-start "2022-03-08 18:00:00" --high-detail-end "2022-03-08 18:10:00"
Export data to a tar.gz file in /tmp at 2s granularity data for just servers between 18:00 UTC and 18:10 UTC on 08MAR2022
./esm-maintenance --extract-data --temp-dir /tmp --server-data-only --high-detail-output --high-detail-start "2022-03-08 18:00:00" --high-detail-end "2022-03-08 18:10:00"
Last update:
May 23, 2023
Created:
April 24, 2023