Rotating Trace Log to Manage its Size
Without a log rotation policy in place, the tracelog can keep growing in size until it consumes all remaining disk space. Hence, it is important to implement log rotation. The memsql.log file can be rotated by moving the memsql.log file and then sending SIGHUP to the memsqld process. This will trigger the server to reopen memsql.log and continue writing. As tracelog files are text files, you may want to compress them after rotation to free up disk space.
Using logrotate
To automate tracelog rotation, you may use the logrotate utility. The logrotate utility runs with the default /etc/logrotate.conf configuration. If you have custom configurations, you can place them in /etc/logrotate.d/; the include directive in the logrotate.conf file makes sure that the custom configuration files are read as well. Alternatively, you can specify the necessary configuration files as arguments to the logrotate command. When multiple configuration files are specified, files that appear later in the list override the options in the earlier ones, so the order in which the files are listed matters. For more information about logrotate, refer to the logrotate man page.
Example logrotate Configuration File
The following logrotate configuration file rotates two log files: memsql.log and query.log. It represents a generic configuration, which can be modified to suit your requirements.
Note: The example paths use the default folder location for SingleStoreDB with an example ID value in the folder name for a Master Aggregator node. You will need to replace them with the values for your node. Also, this example assumes you are logged in as the Linux root user when running logrotate.
cat > /var/tmp/tracelog-logrotate.conf <<EOF
/var/lib/memsql/master-3306-MI4478312f/tracelogs/memsql.log /var/lib/memsql/master-3306-MI4478312f/tracelogs/query.log {
daily
rotate 7
missingok
compress
sharedscripts
postrotate
# Send SIGHUP to both memsqld processes
killall -q -s1 memsqld
endscript
}
EOF
The directives used in the above example indicate the following:
daily: Log files are rotated every day; alternately, you can rotate them weekly or monthly.rotate[count]: Specifies the number of times log files are rotated before being deleted. In the example, log files are rotated 7 times before being deleted. You can specify the number of times you want to rotate the old log files before deleting them. If run with the--mailoption, the log files will be rotated the specified number of times before being emailed, rather than being removed.missingok: If a specified log file is missing, this directive ensures thatlogrotaterotates the next file in the queue without throwing an error.compress: Old versions of log files are compressed with gzip by default.sharedscripts: With this directive specified, thepostrotatescript is run only one time (after the old logs have been compressed), not each time a log file is rotated.postrotate/endscript: The lines between these directives are executed after the log file is rotated. The example uses thekillall -q -s1 memsqldcommand, which allows for a new tracelog to be generated and written to the next trace.
Scheduling logrotate
The logrotate process is typically run as a daily cron job by the cron daemon. The package manager used by the system creates a default schedule in /etc/cron.daily/logrotate that runs logrotate with the default /etc/logrotate.conf configuration. If it is not automatically created, you can manually create one and add the following code snippet.
#! /bin/sh /user/sbin/logrotate /etc/logrotate.conf
You can also create cron jobs in the cron.hourly, cron.weekly, or cron.monthly folders. The jobs located in these folders and the cron.daily folder are run by the cron service, per the instructions in the /etc/crontab file. If you want to run logrotate with a custom schedule, create a cron job and place it under /etc/cron.d/.
Note that the cron jobs stored in /etc/cron.d/ and /etc/crontab directories are system cron jobs. System cron jobs are defined in shared directories on the filesystem. You need sudo privileges to define system cron jobs.
Cron jobs are defined using the unix-cron format in both /etc/cron.d/ and /etc/crontab directories. The schedule consists of a string of five values indicating when the cron job must be executed. These values represent date and time fields, including minute, hour, day of month, and day of week, in that order. The string value is followed by the username under which the cron job must run and then the command to execute. For example, the following cron job issues logrotate using the /etc/tracelog-logrotate.conf configuration every Saturday at five minutes past midnight.
5 0 * * 6 root /usr/sbin/logrotate /etc/tracelog-logrotate.conf
Manually Running logrotate
To manually run logrotate, use the following command and specify the configuration file.
logrotate -f /var/tmp/tracelog-logrotate.conf
To list the compressed log files, run the following command.
ls /var/lib/memsql/master-3306-MI4478312f/tracelogs/*.gz
Alternatives to logrotate
As an alternative to logrotate, you can configure SingleStoreDB to log using the syslog daemon, such as journald, and then implement log rotation. Since logging is a host-level activity, you can use a logging system that is ideal for your environment.