Rotating Trace Log to Manage its Size

Without a log rotation policy in place, the tracelog can keep growing in size until it consumes all remaining disk space. Hence, it is important to implement log rotation. As tracelog files are text files, you may want to compress them after rotation to free up disk space.

SingleStore supports two approaches to trace log rotation: via a built-in function to rotate and manage trace log files, or you can build your own custom solution using something like logrotate.

Built-In Trace Log Rotation

The SingleStore provided trace log rotation feature includes settings for:

  • rotating based on log size

  • rotating based on time

  • deleting logs based on age

You can enable log rotation based on log size or time on disk or both. You can separately configure whether to delete old logs and the number of days on the system at which to delete them.

To enable trace log rotation, set either or both of the tracelog_rotation_size and tracelog_rotation_time engine variables to a valid value other than 0 -- the default values are 0 (disabled).

If you set both engine variables, rotation will happen on either of triggers;, i.e. either the file exceeds a particular size or the time has come to rotate. For most use cases, setting either one or other is sufficient.

Trace Log Rotation Engine Variables

The following engine variables control the behavior of the built-in trace log rotation feature.

tracelog_rotation_size specifies the log file size threshold in bytes. Once that size is exceeded the log is rotated. The minimum value is 65536 (65 Kilobytes).

tracelog_rotation_time specifies the log file rotation time threshold in seconds. The minimum value is 60 seconds.

tracelog_retention_period controls the amount of time in days to keep rotated logs on the file system. SingleStore will delete rotated logs if the log last modification date is past the threshold. This variable is settable both at startup and runtime.

Warning

If tracelog_retention_period is not enabled, trace log files will eventually fill the filesystem. You should have a process in place for processing these files.

tracelog_rotation_size and tracelog_rotation_time are used to control the rotation strategy. These variables are only settable at startup, via the sdb-admin update-config command.

Examples of Trace Log Rotation Configuration

This table shows various trace log rotation settings and their effects.

Engine Variable Settings

Trace Log Rotation Behavior

tracelog_rotation_size = 0

tracelog_rotation_time = 0

tracelog_retention_period = 0

Trace log rotation disabled.

tracelog_rotation_size = 0

tracelog_rotation_time = 0

tracelog_retention_period = 1

Trace log rotation disabled.

tracelog_rotation_size = 128000

tracelog_rotation_time = 0

tracelog_retention_period = 0

Trace log rotation enabled. Logs rotated when they reach 128000 bytes. Logs remain in the system directory indefinitely (SingleStore will not remove them, and the filesystem will eventually fill with log files).

tracelog_rotation_size = 0

tracelog_rotation_time = 86400

tracelog_retention_period = 0

Trace log rotation enabled. Logs rotated every day (86400 seconds = 24 hours). Logs remain in the system directory indefinitely (SingleStore will not remove them, and the filesystem will eventually fill with log files.).

tracelog_rotation_size = 0

tracelog_rotation_time = 86400

tracelog_retention_period = 1

Trace log rotation enabled. Logs rotated every day (86400 seconds = 24 hours). Logs are deleted automatically after they are 1 day old.

tracelog_rotation_size = 128000

tracelog_rotation_time = 86400

tracelog_retention_period = 1

Trace log rotation enabled. Logs rotated when they reach 128000 bytes or once per day, whichever happens first. Logs are deleted automatically after they are 1 day old.

Using logrotate

The memsql.log file can be rotated by moving the memsql.log file and then sending SIGHUP to the memsqld process. This will trigger the server to reopen memsql.log and continue writing.

To automate tracelog rotation, you may use the logrotate utility. The logrotate utility runs with the default /etc/logrotate.conf configuration. If you have custom configurations, you can place them in /etc/logrotate.d/; the include directive in the logrotate.conf file makes sure that the custom configuration files are read as well. Alternatively, you can specify the necessary configuration files as arguments to the logrotate command. When multiple configuration files are specified, files that appear later in the list override the options in the earlier ones, so the order in which the files are listed matters. For more information about logrotate, refer to the logrotate man page.

Example logrotate Configuration File

The following logrotate configuration file rotates two log files: memsql.log and query.log. It represents a generic configuration, which can be modified to suit your requirements.

Note: The example paths use the default folder location for SingleStore with an example ID value in the folder name for a Master Aggregator node. You will need to replace them with the values for your node. Also, this example assumes you are logged in as the Linux root user when running logrotate.

cat > /var/tmp/tracelog-logrotate.conf <<EOF
/var/lib/memsql/master-3306-MI4478312f/tracelogs/memsql.log /var/lib/memsql/master-3306-MI4478312f/tracelogs/query.log {
daily
rotate 7
missingok
compress
sharedscripts
postrotate
# Send SIGHUP to both memsqld processes
killall -q -s1 memsqld
endscript
}
EOF

The directives used in the above example indicate the following:

  • daily: Log files are rotated every day; alternately, you can rotate them weekly or monthly.

  • rotate [count]: Specifies the number of times log files are rotated before being deleted. In the example, log files are rotated 7 times before being deleted. You can specify the number of times you want to rotate the old log files before deleting them. If run with the --mail option, the log files will be rotated the specified number of times before being emailed, rather than being removed.

  • missingok: If a specified log file is missing, this directive ensures that logrotate rotates the next file in the queue without throwing an error.

  • compress: Old versions of log files are compressed with gzip by default.

  • sharedscripts: With this directive specified, the postrotate script is run only one time (after the old logs have been compressed), not each time a log file is rotated.

  • postrotate/endscript: The lines between these directives are executed after the log file is rotated. The example uses the killall -q -s1 memsqld command, which allows for a new tracelog to be generated and written to the next trace.

Scheduling logrotate

The logrotate process is typically run as a daily cron job by the cron daemon. The package manager used by the system creates a default schedule in /etc/cron.daily/logrotate that runs logrotate with the default /etc/logrotate.conf configuration. If it is not automatically created, you can manually create one and add the following code snippet.

#! /bin/sh
/user/sbin/logrotate /etc/logrotate.conf

You can also create cron jobs in the cron.hourly, cron.weekly, or cron.monthly folders. The jobs located in these folders and the cron.daily folder are run by the cron service, per the instructions in the /etc/crontab file. If you want to run logrotate with a custom schedule, create a cron job and place it under /etc/cron.d/.

Note that the cron jobs stored in /etc/cron.d/ and /etc/crontab directories are system cron jobs. System cron jobs are defined in shared directories on the filesystem. You need sudo privileges to define system cron jobs.

Cron jobs are defined using the unix-cron format in both /etc/cron.d/ and /etc/crontab directories. The schedule consists of a string of five values indicating when the cron job must be executed. These values represent date and time fields, including minute, hour, day of month, and day of week, in that order. The string value is followed by the username under which the cron job must run and then the command to execute. For example, the following cron job issues logrotate using the /etc/tracelog-logrotate.conf configuration every Saturday at five minutes past midnight.

5 0 * * 6 root /usr/sbin/logrotate /etc/tracelog-logrotate.conf

Manually Running logrotate

To manually run logrotate, use the following command and specify the configuration file.

logrotate -f /var/tmp/tracelog-logrotate.conf

To list the compressed log files, run the following command.

ls /var/lib/memsql/master-3306-MI4478312f/tracelogs/*.gz

Alternatives to logrotate

As an alternative to logrotate, you can configure SingleStore to log using the syslog daemon, such as journald, and then implement log rotation. Since logging is a host-level activity, you can use a logging system that is ideal for your environment.

Last modified: January 18, 2024

Was this article helpful?