ESM Administration Guide for Unix¶

ESM Server¶

System requirements¶

Hardware Requirements¶

Provisioning a standalone host to run the ESM Server component is advisable. The following host specifications are recommended:

Minimum 2 CPUs or vCPUs (4 recommended)
8GB RAM
2GB system Storage for the ESM Server installation
50 – 100GB Storage for the underlying ESM Database tablespace

Larger deployments with an increased number of end users, jobs or nodes may require more resource. If in doubt, contact Boemska Support for assistance when sizing the ESM Server host.

Supported Versions of Unix/Linux¶

64 bit Linux (RedHat / CentOS EL6 or later recommended)

Supported Databases¶

A standalone database installation is no longer required.

Note on supported databases

As of March 2018, support for third party databases is being phased out in favour of the embedded ESM Database (currently based on PostgreSQL 9.5). The embedded ESM database ships with the ESM Server binaries, and provisioning of a standalone database component is no longer required. For legacy documentation on MySQL or Oracle based deployment methods please contact Boemska Support.

Installation Requirements¶

Ports¶

If a firewall is configured, the following ports will need to be opened for communication between the ESM Agent(s) and the ESM Server. The default ports are listed below. Custom ports can be specified during installation if required:

Port	Description	Notes
18080	Default HTTP port for ESM communication	Required for 2020.3.x and earlier.
18082	Default gRPC port for ESM communication	Required for ESM 2020.4.x
14848	Default Glassfish administration port	For debug purposes
15432	Default Database JDBC connection port	When configuring DB replication

In addition, the following ports should be opened for communication between the ESM Server and user(s) web browsers. The default port(s) are listed below but custom ports can be specified during the installation if required:

Port	Description	Mandatory	Notes
18080	Default HTTP port for ESM communication		Default user interface port for 2020.3.x and 2020.4.x
18181	Default HTTPS port for ESM communication		For embedded HTTPS

Correct functioning of embedded HTTPS requires manual configuration of certificates. As an alternative, configuration with a separate reverse proxy is recommended. For details contact Boemska Support.

If the ESM Embedded Database is clustered across multiple hosts, more detailed firewall configuration may be required.

User Accounts¶

An Operating System account under which to run the ESM Server is required. ESM requires an account with the following properties:

A centrally managed or non-expiring password (recommended)
Read/Write access to the target installation directory and subdirectories

Warning

Do not install the ESM Server as the root user. The Operating System username chosen for the ESM user account also cannot be the same as the Database account username specified during the configuration of the embedded ESM database, which by default is esm. For a vanilla installation it is therefore recommended that the name esm is avoided when provisioning an Operating System user for the ESM application to run under. A username starting with esm, such as esmuser or esmserver, is acceptable.

Server Installation¶

Obtain installation package¶

Log on to the server using the nominated account with the required installation privileges (see installation requirements for more details)
Download the appropriate ESM Server installation package for your Linux/Unix operating system. Installation packages can be obtained from the Boemska Downloads site at https://downloads.boemskats.com/esm or by contacting your Boemska representative.
Unpack the archive to a suitable temporary location. This guide assumes a target location of /pub/esm/esm-server-installer

tar xzf ./esm-server-x64-linux-v2020.x.tar.gz -C /pub/esm

Note

When un-tarring as the root user the default behaviour is to apply the --same-owner option. This can cause files to be shown with an unmatched UID and GID. To prevent this behaviour, add the following option to the end of the above command: --no-same-owner.

Running the Installation Wizard¶

Change directory to the destination of the unpacked installation package and execute the setup.sh script to start the ESM Server Installation Wizard.

cd /pub/esm/esm-server-installer/
./setup.sh

The ESM Server installation wizard will now start. You should be greeted with the Terms of License, which you can accept by typing 'y':

$ ./setup.sh 

 Terms of License and User Agreement 

Please review the following Terms of License before installing the ESM Agent.

     By typing 'y' you confirm that:

i)   there is a valid license agreement in place which duly authorises you to use this
ii)  you have read the Software License Agreement and you understand its terms and
iii) you agree to be bound by the terms and conditions of the Software License
     software and describes your rights and obligations as a licensee of this software

     You should not proceed and select 'n' if you cannot confirm any of the
     statements above.

If you accept the Terms of License above please type 'y' to continue.  y

You will be presented with a choice of database. Select 1) Embedded:

=============================================================================
 ESM Server Installer Started.
 Current time:  Sun Jul  8 20:34:22 UTC 2018
=============================================================================


1) Embedded (pgsql, most common option)
2) MySQL
3) Oracle
4) PostgreSQL
Please choose the database which will be used: 1
Embedded database selected.

Embedded database hostname set to localhost.

Confirm the port and JDBC URL for the embedded defaults.

Enter the database port number for Embedded database: [default: 15432]

Port 15432 is ok.
Enter the JDBC connection URL for Embedded database: [default: jdbc:postgresql://localhost:15432/esm]

Specify the username and password for the user which will be created in the ESM database and which the ESM application will use to connect. Note the Warning in the User Accounts section above.
```
Enter the Embedded database password (type 'none' for blank password): [default: esm]
esmdb123 
(again) Confirm the database password: 
esmdb123
```

Next, specify the filesystem paths where ESM will be deployed. For a single filesystem deployment, the defaults are same; if a separate partition is provisioned for data storage, specify it in the Embedded database data directory step:

Please specify esm-server install path: [default: /home/esmuser]

Enter the Embedded database installation directory: [default: /home/esmuser/esm-server/pgsql]

Selected installation directory for Embedded is /home/esmuser/esm-server/pgsql.

Enter the Embedded database data directory: [default: /home/esmuser/esm-server/pgsql/data]

Selected data dir for Embedded is /home/esmuser/esm-server/pgsql/data.

Enter the Embedded database logs directory: [default: /home/esmuser/esm-server/pgsql/logs]

Selected logs dir for Embedded is /home/esmuser/esm-server/pgsql/logs.

At this stage the database will be deployed, and the SQL output from the schema generation will most likely fill the terminal screen. The output should look something like this (significant output highlighted):

Extracting embedded postgres database to /home/esmuser/esm-server/pgsql ...
Initializing embedded database...
The files belonging to this database system will be owned by user "esmuser".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/esmuser/esm-server/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    ./pg_ctl -D /home/esmuser/esm-server/pgsql/data -l logfile start

Appending database tunning parameters to postgresql.conf file...
waiting for server to start.... done
server started
LOG:  database system was shut down at 2018-07-08 20:34:57 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
CREATE ROLE
GRANT
Checking database connection...
Embedded database successfully checked.
<...>

This will be followed by around 600 lines of SQL output, trailing with something like the following:

20:35:02,046  INFO SchemaExport:268 - schema export complete
20:35:02,046  INFO DriverManagerConnectionProvider:170 - cleaning up connection pool: 
                jdbc:postgresql://localhost:15432/esm
Jul 08, 2018 8:35:02 PM com.acedigital.esm.server.HibernateUtil exportSchema
INFO: SCHEMA EXPORT DONE...
waiting for server to shut down.... done
server stopped

Next, configure the ports on which the ESM Server application will bind to. You may want to cange the ESM Server port, but should accept the defaults for all other ports unless more than one instance of ESM Server is to be installed on this machine.

Enter the port that esm server will use: [default: 18080]

Port 18080 is ok.
Enter the port that esm admin will use: [default: 14848]

Port 14848 is ok.
Enter the port that esm will use for secure connection: [default: 18181]

Port 18181 is ok.
Enter the port that esm admin will use for JMS service: [default: 17676]

Port 17676 is ok.
Enter the port that esm admin will use for EJB service: [default: 13700]

Port 13700 is ok.

Specify the username and password for the ESM Administrator account, which will be used to configure ESM from the Web interface:
```
Enter the esm admin settings username: [default: esmadm]

Enter the esm admin settings password: [default: Orion123]
```
Keep the ESM Administrator username and password details safe.

The ESM server will now be extracted and installed. On successful completion, you should see a message similar to the following. Enter 'y' to start the ESM Server:

Extracting server to /home/esmuser ...
Configuring server installation...
Configuring server database connection...
Configuring esm admin user...
Exporting selected database...
Exporting database scripts...

-----------------------------------------------------------------------------
   ESM server installation complete:
   Successfully installed to: /home/esmuser/esm-server
-----------------------------------------------------------------------------

   NOTE: Make sure that port 18080 is open for agents connections.

You can start ESM Server by running this command:
   /home/esmuser/esm-server/bin/esm-server.sh start

You can access to your server at:
   http://esmsrv:18080/esm/

 Your esm administration username: esmadm
 Your esm administration password: XXXXXX

Do you want to start it now? (y/n):  y
Starting embedded database...
waiting for server to start.... done
server started
Waiting for domain1 to start ....
Successfully started the domain : domain1
domain  Location: /home/esmuser/esm-server/esm-core/glassfish/domains/domain1
Log File: /home/esmuser/esm-server/esm-core/glassfish/domains/domain1/logs/server.log
Admin Port: 14848
Command start-domain executed successfully.

Note

It will take around 30 to 60 seconds before the web interface is accessible, as the ESM Web Application is deployed into its Application Server for the first time. During this deployment, a java process owned by the ESM installation user will consume a lot of CPU. Once this process has subsided, the application will be ready and accessible via the URL provided.

Validating connections¶

Click on Admin Settings button from the main ribbon (top right)
Enter the ESM Administrator username and password you configured during installation
Verify that the application is connecting to the database correctly, by checking the reported Database Size: The size of the database should be reported, showing a good connection between the ESM Server and the ESM Database and normal operation of the ESM Server application.

Note

This concludes the ESM Server Installation and Configuration steps. Now proceed with the ESM Agent installation and configuration steps for each node to be monitored, as outlined in the ESM Agent Installation Guide for Unix.

Server Administration¶

Starting and Stopping the ESM Server¶

Starting ESM Server¶

Change directory (cd) to the path of the ESM Server binaries e.g. /pub/esm/esm-server/bin
Run the following command to start ESM:
```
./esm-server.sh start
```

The server will now launch and a message similar to the following should be displayed:

[esmuser@esmsrv bin]$ ./esm-server.sh start
Starting embedded database...
waiting for server to start.... done
server started
Waiting for domain1 to start ...................
Successfully started the domain : domain1
domain  Location: /home/esmuser/esm-server/esm-core/glassfish/domains/domain1
Log File: /home/esmuser/esm-server/esm-core/glassfish/domains/domain1/logs/server.log
Admin Port: 14848
Command start-domain executed successfully.

Stopping ESM Server¶

Change directory (cd) to the path of the ESM Server binaries e.g. /pub/esm/esm-server/bin
Run the following command to stop ESM:
```
./esm-server.sh stop
```

The server will now stop and a message similar to the following should be displayed:

[user@esmserver bin]$ ./esm-server.sh stop
Waiting for the domain to stop .
Command stop-domain executed successfully.
Stopping embedded database...
waiting for server to shut down.... done
server stopped

Should you encounter any issues stopping ESM server, a force-stop command can be issued as follows:
```
./esm-server.sh force-stop
```
Force stop identifies the pid under which the server is running and issues a SIGKILL signal. This command should only be used if the normal stop command fails to stop the server.

Restarting ESM Server¶

Stopping and starting ESM server can be achieved using the restart option on the ESM Server script

Change directory (cd) to the path of the ESM Server binaries e.g. /pub/esm/esm-server/bin
Run the following command to start ESM:
```
./esm-server.sh restart
```

The server will now stop and immediately start. A message similar to the following should be displayed:

Waiting for the domain to stop .
Command stop-domain executed successfully.
Stopping embedded database...
waiting for server to shut down.... done
server stopped
Starting embedded database...
waiting for server to start.... done
server started
Waiting for domain1 to start ...................
Successfully started the domain : domain1
domain  Location: /home/esmuser/esm-server/esm-core/glassfish/domains/domain1
Log File: /home/esmuser/esm-server/esm-core/glassfish/domains/domain1/logs/server.log
Admin Port: 14848
Command start-domain executed successfully.

Uninstalling ESM Server¶

Removal of the ESM Server application is simple:

Ensure ESM Server is not running by issuing the stop server command. See section titled Stopping ESM Server for further details
Delete the installation directory and contents.

This concludes the removal procedure for ESM Server.

ESM Agent¶

Important note

After completing the tasks here, further configuration of SAS will be required before session-level detail is reported. The further configuration steps required are covered in Configuring SAS for ESM.

For Grid or multi-node deployments of ESM, it is recommended to install the ESM Agent to a clustered file system, accessible by all nodes in the grid. The installation procedure only needs to be run once on the first grid node to generate the agent installation directory. The ESM Agent is then started from each subsequent grid node from the generated installation directory.

For single node or non-clustered file systems, the ESM Agent can be installed to a local file system.

System Requirements¶

Supported versions of Linux/Unix¶

64 bit Linux (RHEL4 or later recommended)
AIX v5.2 to 7.2
HPUX 11i
Solaris v8 or later

Supported versions of SAS®¶

ESM is compatible with all recent versions of SAS. The minimum recommended version of SAS is SAS v9.1.3; however, as ESM Agents monitor sessions at the process level, ESM is able to monitor earlier versions of SAS where necessary. Some custom configuration may be required.

Installation Requirements¶

Approximately 750 MB disk space (including logs)
A user account with sudoer privilege, or suitable privileges to run the ESM Agent sufficiently. The following should be considered when choosing or configuring a system user account for ESM:
- The user account will need Read/Write access to the target installation directory and sub directories
- To use ESM to measure the size of SASWORK and SASUTIL directories of user-owned SAS sessions, the account under which the ESM Agent process is executing must have the privileges necessary to read those directories.
- To use ESM to tail logfiles generated by system services, userspace sessions or batch jobs, the agent process must have the privileges necessary to read the logfiles written by those processes.
- To use ESM to terminate SAS sessions from the ESM interface, the account under which the agent service is executing must have the privileges necessary to do so at the operating system level.
A running ESM Server. During installation the agent will validate connectivity to the target server. Please install, configure and start an ESM Server instance before installing the agent
The hostname or IP address and port number of the ESM Server
A modern Web Browser. Although ESM loads and works under IE8, usable performance is not guaranteed.
If a firewall is configured, the default TCP port 18080 (or the custom configured port selected during server installation) will need to be opened for communication to the target ESM server

Agent Installation¶

Getting the ESM Agent Installation package¶

Download the appropriate ESM agent installation package for your Linux/Unix operating system. Installation packages can be obtained by signing in to the Boemska Downloads site at https://downloads.boemskats.com/esm or by contacting your Boemska representative. If you have not received your logon details or require further help, please contact Boemska Support on support@boemskats.com.

First/Primary node installation¶

Log on to the server using the nominated account with the required installation privileges (see installation requirements for more details).

Unpack the installation package and navigate to its contents, which will be a subdirectory called esm-agent-installer:

esminstaller at node1 in /home/esminstaller
$  tar xf /pub/esm/esm-agent-x64-linux-v1.0-branch_build.tar.gz -C /pub/esm
esminstaller at node1 in /home/esminstaller
$ cd /pub/esm/esm-agent-installer
esminstaller at node1 in /pub/esm/esm-agent-installer
$

Execute ./setup.sh to start the ESM Agent Installation Wizard. You should be greeted with the Terms of License, which you can accept by typing 'y':

$ ./setup.sh 

 Terms of License and User Agreement 

Please review the following Terms of License before installing the ESM Agent.

     By typing 'y' you confirm that:

i)   there is a valid license agreement in place which duly authorises you to use this
ii)  you have read the Software License Agreement and you understand its terms and
iii) you agree to be bound by the terms and conditions of the Software License
     software and describes your rights and obligations as a licensee of this software

     You should not proceed and select 'n' if you cannot confirm any of the
     statements above.

If you accept the Terms of License above please type 'y' to continue.  y

Specify the Agent installation path:

=============================================================================
 ESM Agent Installer Started.
 Current time:  Mon  2 Jul 16:47:57 BST 2018
=============================================================================

Please, insert agent install path: [default: /home/nik]

/pub/esm

The installer will automatically create an esm-agent sub folder within the selected destination folder named esm-agent, which in this example will be /pub/esm/esm-agent

Enter the hostname or IP address of the ESM Server into which this Agent will report, and the port which the server is listening on. You may hit enter to accept the defaults:

What is the hostname or IP address of the ESM server: [default: localhost]
apps.boemskats.com
What port is the ESM server configured to communicate on: [default: 18080]
18080

For the next step, the format in which SAS Foundation reports the hostname for each node is needed. This can be sourced by either running the following command from a shell:

echo 'data _null_; file STDOUT; put "&syshostname."; run;' | \
     /pub/sas/SASFoundation/9.4/sas -stdio 2>/dev/null

or running the following code from an existing SAS session:

%put &syshostname.;

The Installer will prompt for the format which matches the output from either of the above.

How is the value of the &syshostname macro variable formatted within the target environment?
1) node1.boemskats.com
2) node1
3) Set hostname manually
Chosen format: 2
You have chosen node1 as your hostname format.

Configure the block devices which ESM will monitor. The script will attempt to detect available block devices using the lsblk command, but this does not always detect all devices for which metrics are available. If a device is not shown in the list of defaults, but is visible under /proc/diskstats or by running iostat -x, the agent should generally be able to report on it. Ensure all devices are punctuated by a trailing semicolon, including the last device in the list.
```
Please specify the storage devices you would like to be monitored.
When adding more than one, ensure they are separated by a semicolon ';'.
To disable disk monitoring, type 'none'.

You can also type 'auto' or 'autodiscover' if you want ESM to try to automatically
find eligible disk devices, but this is not always reliable.

Recommended disks to monitor, according to lsblk -l: [default: sda;vda; ]
sda;
Selected disks: "sda;"
```

The ESM Agent will now install. On completion, a message similar to the following will be displayed:

Extracting ESM Agent
Moving ESM Agent to /pub/esm
Configuring ESM Agent...
Configuring ESM Agent default events folder...

Checking connection between ESM Agent and ESM Server...
Connection to apps.boemskats.com:18080 succeeded!

-----------------------------------------------------------------------------
   ESM Agent installation complete: 
   Successfully installed to: /pub/esm/esm-agent
-----------------------------------------------------------------------------

Following installation, you will be prompted to start the ESM Agent. Before doing so, consider the user account requirements specified in the Installation Requirements section of this document. If the current user meets those requirements, proceed as instructed:

You can start ESM Agent by running this command:
   /pub/esm/esm-agent/bin/esm-agent.sh start

Do you want to start it now? (y/n):  y

The ESM Agent should now start:

Checking connection between ESM Agent and ESM Server...
Connection to apps.boemskats.com:18080 succeeded!
Starting ESM Agent...

Log file: '/pub/esm/esm-agent/logs/localhost_wrapper.log'

Agent started. (pid 12968)

Use a web browser to connect to the ESM Server web client to verify the agent has started and is reporting in. You should see a node with the specified agent hostname listed in the Live view, and clicking the node should load the live graph.

If the ESM Agent failed to install or start, please refer to the Troubleshooting section of this document for further information.

Installation on subsequent nodes¶

To monitor subsequent GRID nodes, simply starting the ESM Agent from the shared filesystem on each node should be sufficient.

/filesystem/esm/esm-agent/bin/esm-agent.sh start

To monitor nodes that do not share an identically mounted filesystem, repeat the installation steps for each node to be monitored.

Reminder of a previous Note

Once the ESM Agent installation is complete for all nodes, further configuration of SAS is required for full integration. This is detailed in the document titled Configuring SAS for ESM.

Agent Administration¶

Starting and Stopping the ESM Agent Service¶

The ESM Agent can be started with the following command:

Note

The ESM agent should be run under an account with sufficient permissions to inspect and manage the processes being monitored. On a typical multi-user system, the ESM Agent is therefore started and stopped by prefixing the sudo command to the start and stop commands below.

/filesystem/esm/esm-agent/bin/esm-agent.sh start

Checking connection between ESM Agent and ESM Server...
Connection to apps.boemskats.com:18080 succeeded!
Starting ESM Agent...
Log file: '/pub/esm/esm-agent/logs/node1_wrapper.log'
Agent started. (pid 3327)

The ESM Agent can be stopped with the following command:

/filesystem/esm/esm-agent/bin/esm-agent.sh stop

Stopping ESM Agent...
Stopped ESM Agent.

The status of the ESM Agent can be checked with the following command:

/filesystem/esm/esm-agent/bin/esm-agent.sh status
ESM Agent is not running.

Uninstall the ESM Agent¶

First, ensure the ESM Agent is stopped with the following command

/filesystem/esm/esm-agent/bin/esm-agent.sh stop

You may now delete the ESM Agent installation directory if required. If the ESM Agent was configured to start using something like init.d, systemd or upstart, you may need to update your startup configuration. Otherwise no further action is required.

Warning

If removing ESM permanently, remember to remove any ESM specific configuration from your SAS Configuration scripts. Depending on the version of ESM you have installed, the integration routines may not fail gracefully if the ESM agent is deleted before your SAS environment is deconfigured.