Log files are an important component of troubleshooting attacks and obtaining information about breaches. All ESXi hosts run a syslog service, which logs messages from the VMkernel and other system components to local files or to a remote host.
To increase the security of the host, take the following measures.
- Configure persistent logging to a datastore. By default, the logs on ESXi hosts are stored in the in-memory file system. Therefore, they are lost when you reboot the host, and only 24 hours of log data is stored. When you enable persistent logging, you have a dedicated activity record for the host.
- Remote logging to a central host allows you to gather log files on a central host. From that host, you can monitor all hosts with a single tool, do aggregate analysis, and search log data. This approach facilitates monitoring and reveals information about coordinated attacks on multiple hosts.
- Configure the remote secure syslog on ESXi hosts by using ESXCLI or PowerCLI, or by using an API client.
- Query the syslog configuration to make sure that the syslog server and port are valid.
See the vSphere Monitoring and Performance documentation for information about syslog setup, and for additional information on ESXi log files.
Configure Syslog on ESXi Hosts
You can use the vSphere Client, the VMware Host Client, or the esxcli system syslog command to configure the syslog service.
For information about using the esxcli system syslog command and other ESXCLI commands, see Getting Started with ESXCLI. For details how to open the ESXi firewall for the port specified in each remote host specification, see Configuring the ESXi Firewall .
Procedure
- Browse to the host in the vSphere Client inventory.
- Click Configure.
- Under System, click Advanced System Settings.
- Click Edit.
- Filter for syslog.
- To set up logging globally and configure various advanced settings, see ESXi Syslog Options.
- (Optional) To overwrite the default log size and log rotation for any of the logs:
- Click the name of the log that you want to customize.
- Enter the number of rotations and the log size you want.
- Click OK.
Results
Changes to the syslog options take effect.
Note: Syslog parameter settings that you define by using the vSphere Client or VMware Host Client are effective immediately. However, most settings you define by using ESXCLI require an additional command to take effect. For more details, see
ESXi Syslog Options.
ESXi Syslog Options
You can define the behavior of ESXi syslog files and transmissions by using a set of syslog options.
Apart from the base settings, such as Syslog.global.logHost, starting from ESXi 7.0 Update 1, a list of advanced options is available for customizations, and NIAP compliance.
Note: Always configure persistent storage before you set any of the audit record parameters or the
Syslog.global.logDir parameter.
Note: All audit record settings, beginning with
Syslog.global.auditRecord, take effect immediately. However, for other settings that you define by using ESXCLI, make sure to run the
esxcli system syslog reload command to enable the changes.
| Option | ESXCLI command | Description |
|---|---|---|
| Syslog.global.logHost | esxcli system syslog config set --loghost=<str> |
Defines a comma-delimited list of remote hosts and specifications for message transmissions. If the loghost=<str> field is blank, no logs are forwarded. While no hard limit to the number of remote hosts to receive syslog messages exists, good practice is to keep the number of remote hosts to five or less. The format of a remote host specification is: protocol://hostname|ipv4|'['ipv6']'[:port]. The protocol must be one of TCP, UDP, or SSL. The value of a port can be any decimal number from 1 through 65535. If a port is not provided, SSL and TCP use 1514. UDP uses 514. For example: ssl://hostName1:1514. |
| Syslog.global.defaultRotate | esxcli system syslog config set --default-rotate=<long> | Maximum number of old log files to keep. You can set this number globally and for individual subloggers (see Syslog.global.defaultSize). |
| Syslog.global.defaultSize | esxcli system syslog config set --default-size=<long> | Default size of log files, in KiB. After a file reaches the default size, the syslog service creates a new file. You can set this number globally and for individual subloggers. |
| Syslog.global.logDir | esxcli system syslog config set --logdir=<str> | Directory where logs reside. The directory can be on mounted NFS or VMFS volumes. Only the /scratch directory on the local file system is persistent across reboots. Specify the directory as [datastorename] path_to_file, where the path is relative to the root of the volume backing the datastore. For example, the path [storage1] /systemlogs maps to the path /vmfs/volumes/storage1/systemlogs. |
| Syslog.global.logDirUnique | esxcli system syslog config set --logdir-unique=<bool> | Specifies the ESXi host name to be concatenated to the value of Syslog.global.logDir. It is critical that you enable this setting when multiple ESXi hosts log to a shared file system. Selecting this option creates a subdirectory with the name of the ESXi host under the directory specified by Syslog.global.LogDir. A unique directory is useful if the same NFS directory is used by multiple ESXi hosts. |
| Syslog.global.certificate.checkSSLCerts | esxcli system syslog config set --check-ssl-certs=<bool> | Enforces checking of SSL certificates when transmitting messages to remote hosts. |
| Option | ESXCLI command | Description |
|---|---|---|
| Syslog.global.auditRecord.storageCapacity | esxcli system auditrecords local set --size=<long> | Specifies the capacity of the audit record storage directory located on the ESXi host, in MiB. You cannot decrease the capacity of the audit record storage. You can increase the capacity before or after the audit record storage is enabled (see Syslog.global.auditRecord.storageEnable). |
| Syslog.global.auditRecord.remoteEnable | esxcli system auditrecords remote enable | Enables sending audit records to remote hosts. Remote hosts are specified by using the Syslog.global.logHost parameter. |
| Syslog.global.auditRecord.storageDirectory | esxcli system auditrecords local set --directory=<dir> | Creates an audit record storage directory and unless specified, sets /scratch/auditLog as the default location. You must not manually create an audit record storage directory and you cannot change the audit record storage directory while audit record storage is enabled (see Syslog.global.auditRecord.storageEnable). |
| Syslog.global.auditRecord.storageEnable | esxcli system auditrecords local enable | Enables the storage of audit records on an ESXi host. If the audit record storage directory does not exist, it is created with the capacity specified by Syslog.global.auditRecord.storageCapacity. |
| Syslog.global.certificate.checkCRL | esxcli system syslog config set --crl-check=<bool> | Enables checking the revocation status of all the certificates in an SSL certificate chain. Enables verification of X.509 CRLs, which are not checked by default in compliance with industry conventions. A NIAP-validated configuration requires CRL checks. Due to implementation limitations, if CRL checks are enabled, then all certificates in a certificate chain must provide a CRL link. Do not enable the crl-check option for installations not related to certification, because of the difficulty in properly configuring an environment that uses CRL checks. |
| Syslog.global.certificate.strictX509Compliance | esxcli system syslog config set --x509-strict=<bool> | Enables strict compliance with X.509. Performs additional validity checks on CA root certificates during verification. These checks are generally not performed, as CA roots are inherently trusted, and might cause incompatibilities with existing, misconfigured CA roots. A NIAP-validated configuration requires even CA roots to pass validations. Do not enable the x509-strict option for installations not related to certification, because of the difficulty in properly configuring an environment that uses CRL checks. |
| Syslog.global.droppedMsgs.fileRotate | esxcli system syslog config set --drop-log-rotate=<long> | Specifies the number of old dropped message log files to keep. |
| Syslog.global.droppedMsgs.fileSize | esxcli system syslog config set --drop-log-size=<long> | Specifies the size of each dropped message log file before switching to a new one, in KiB. |
| Syslog.global.logCheckSSLCerts | esxcli system syslog config set --check-ssl-certs=<bool> | Enforces checking of SSL certificates when transmitting messages to remote hosts.
Note: Deprecated. Use
Syslog.global.certificate.checkSSLCerts in ESXi 7.0 Update 1 and later.
|
| Syslog.global.logFilters | esxcli system syslog config logfilter [add | remove | set] ... | Specifies one or more log filtering specifications. Each log filter must be separated by a double vertical bar "||". The format of a log filter is: numLogs | ident | logRegexp. numLogssets the maximum number of log entries for the specified log messages. After reaching this number, the specified log messages are filtered and ignored. ident specifies one or more system components to apply the filter to the log messages that these components generate. logRegexp specifies a case-sensitive phrase with Python regular expression syntax to filter the log messages by their content. |
| Syslog.global.logFiltersEnable | Enables the use of log filters. | |
| Syslog.global.logLevel | esxcli system syslog config set --log-level=<str> | Specifies the log filtering level. You must change this parameter only when troubleshooting an issue with the syslog daemon. You can use the values debug for the most detailed level, info for the default detail level, warning for only warnings or errors, or error, only for errors. |
| Syslog.global.msgQueueDropMark | esxcli system syslog config --queue-drop-mark=<long>) | Specifies the percent of the message queue capacity at which messages are dropped. |
| Syslog.global.remoteHost.connectRetryDelay | esxcli system syslog config set --default-timeout=<long> | Specifies the delay before retrying to connect to a remote host after a connection attempt fails, in seconds. |
| Syslog.global.remoteHost.maxMsgLen | esxcli system syslog config set --remote-host-max-msg-len=<long> | For the TCP and SSL protocols, this parameter specifies the maximum length of a syslog transmission before truncation occurs, in bytes. The default maximum length for remote host messages is 1 KiB. You can increase the maximum message length to up to 16 KiB. However, raising this value above 1 KiB does not ensure that long transmissions arrive untruncated to a syslog collector. For example, when the syslog infrastructure that issues a message is external to ESXi. This setting does not affect the UDP protocol. RFC 5426 sets the maximum message transmission length for the UDP protocol to 480 bytes for IPV4 and 1180 bytes for IPV6. Because of this restriction, and because UDP packets can be arbitrary dropped by the networking infrastructure, the use of UDP for transmitting critical syslog messages is not recommended. |
| Syslog.global.vsanBacking | esxcli system syslog config set --vsan-backing=<bool> | Allows log files and the audit record storage directory to be placed on a vSAN cluster. However, enabling this parameter might cause the ESXi host to become unresponsive. |
ESXi Log File Locations
ESXi records host activity in log files, using a syslog facility.
| Component | Location | Purpose |
|---|---|---|
| Authentication | /var/log/auth.log | Contains all events related to authentication for the local system. |
| ESXi host agent log | /var/log/hostd.log | Contains information about the agent that manages and configures the ESXi host and its virtual machines. |
| Shell log | /var/log/shell.log | Contains a record of all commands typed into the ESXi Shell and shell events (for example, when the shell was enabled). |
| System messages | /var/log/syslog.log | Contains all general log messages and can be used for troubleshooting. This information was formerly located in the messages log file. |
| vCenter Server agent log | /var/log/vpxa.log | Contains information about the agent that communicates with vCenter Server (if the host is managed by vCenter Server). |
| Virtual machines | The same directory as the affected virtual machine's configuration files, named vmware.log and vmware*.log. For example, /vmfs/volumes/datastore/virtual machine/vmware.log | Contains virtual machine power events, system failure information, tools status and activity, time sync, virtual hardware changes, vMotion migrations, machine clones, and so on. |
| VMkernel | /var/log/vmkernel.log | Records activities related to virtual machines and ESXi. |
| VMkernel summary | /var/log/vmksummary.log | Used to determine uptime and availability statistics for ESXi (comma separated). |
| VMkernel warnings | /var/log/vmkwarning.log | Records activities related to virtual machines and ESXi. |
| Quick Boot | /var/log/loadESX.log | Contains all events related to restarting an ESXi host through Quick Boot. |
| Trusted infrastructure agent | /var/run/log/kmxa.log | Records activities related to the Client Service on the ESXi Trusted Host. |
| Key Provider Service | /var/run/log/kmxd.log | Records activities related to the vSphere Trust Authority Key Provider Service. |
| Attestation Service | /var/run/log/attestd.log | Records activities related to the vSphere Trust Authority Attestation Service. |
| ESX Token Service | /var/run/log/esxtokend.log | Records activities related to the vSphere Trust Authority ESX Token Service. |
| ESX API Forwarder | /var/run/log/esxapiadapter.log | Records activities related to the vSphere Trust Authority API forwarder. |
