# Rotating Trace Log to Manage its Size

Without a log rotation policy in place, the tracelog can keep growing in size until it consumes all remaining disk space. Hence, it is important to implement log rotation. As tracelog files are text files, you may want to compress them after rotation to free up disk space.

SingleStore supports two approaches to trace log rotation: via a built-in function to rotate and manage trace log files, or you can [build your own custom solution](https://docs.singlestore.com/#section-idm4611540200651232849883358409.md) using something like `logrotate`.

## Built-In Trace Log Rotation

The SingleStore provided trace log rotation feature includes settings for:

* rotating based on log size
* rotating based on time
* deleting logs based on age

You can enable log rotation based on log size or time on disk or both. You can separately configure whether to delete old logs and the number of days on the system at which to delete them.

To enable trace log rotation, set either or both of the `tracelog_rotation_size` and `tracelog_rotation_time` engine variables to a valid value other than 0 -- the default values are 0 (disabled).

If you set both engine variables, rotation will happen on either of triggers;, i.e. either the file exceeds a particular size or the time has come to rotate. For most use cases, setting either one or other is sufficient.

## Trace Log Rotation Engine Variables

The following engine variables control the behavior of the built-in trace log rotation feature.

`tracelog_rotation_size` specifies the log file size threshold in bytes. Once that size is exceeded the log is rotated. The minimum value is 65536 (65 Kilobytes).

`tracelog_rotation_time` specifies the log file rotation time threshold in seconds. The minimum value is 60 seconds.

`tracelog_retention_period` controls the amount of time in days to keep rotated logs on the file system. SingleStore will delete rotated logs if the log last modification date is past the threshold. This variable is settable both at startup and runtime.

> **⚠️ Warning**: If `tracelog_retention_period` is not enabled, trace log files will eventually fill the filesystem. You should have a process in place for processing these files.

`tracelog_rotation_size` and `tracelog_rotation_time` are used to control the rotation strategy. These variables are only settable at startup, via the `sdb-admin update-config` command.

## Examples of Trace Log Rotation Configuration

This table shows various trace log rotation settings and their effects.

| Engine Variable Settings                                                                         | Trace Log Rotation Behavior                                                                                                                                                                                                 |
| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tracelog_rotation_size = 0``tracelog_rotation_time = 0``tracelog_retention_period = 0`          | Trace log rotation disabled.                                                                                                                                                                                                |
| `tracelog_rotation_size = 0``tracelog_rotation_time = 0``tracelog_retention_period = 1`          | Trace log rotation disabled.                                                                                                                                                                                                |
| `tracelog_rotation_size = 128000``tracelog_rotation_time = 0``tracelog_retention_period = 0`     | Trace log rotation enabled. Logs rotated when they reach 128000 bytes. Logs remain in the system directory indefinitely (SingleStorewill not remove them, and the filesystem will eventually fill with log files).          |
| `tracelog_rotation_size = 0``tracelog_rotation_time = 86400``tracelog_retention_period = 0`      | Trace log rotation enabled. Logs rotated every day (86400 seconds = 24 hours). Logs remain in the system directory indefinitely (SingleStorewill not remove them, and the filesystem will eventually fill with log files.). |
| `tracelog_rotation_size = 0``tracelog_rotation_time = 86400``tracelog_retention_period = 1`      | Trace log rotation enabled. Logs rotated every day (86400 seconds = 24 hours). Logs are deleted automatically after they are 1 day old.                                                                                     |
| `tracelog_rotation_size = 128000``tracelog_rotation_time = 86400``tracelog_retention_period = 1` | Trace log rotation enabled. Logs rotated when they reach 128000 bytes or once per day, whichever happens first. Logs are deleted automatically after they are 1 day old.                                                    |

## Using `logrotate`

The `memsql.log` file can be rotated by moving the `memsql.log` file and then sending `SIGHUP` to the `memsqld` process. This will trigger the server to reopen `memsql.log` and continue writing.

To automate `tracelog` rotation, you may use the [`logrotate`](https://linux.die.net/man/8/logrotate) utility. The `logrotate` utility runs with the default `/etc/logrotate.conf` configuration. If you have custom configurations, you can place them in `/etc/logrotate.d/`; the `include` directive in the `logrotate.conf` file makes sure that the custom configuration files are read as well. Alternatively, you can specify the necessary configuration files as arguments to the `logrotate` command. When multiple configuration files are specified, files that appear later in the list override the options in the earlier ones, so the order in which the files are listed matters. For more information about `logrotate`, refer to the [`logrotate`](https://linux.die.net/man/8/logrotate) man page.

## Example `logrotate` Configuration File

The following `logrotate` configuration file rotates two log files: `memsql.log` and `query.log`. It represents a generic configuration, which can be modified to suit your requirements.

**Note**: The example paths use the default folder location for SingleStore with an example ID value in the folder name for a Master Aggregator node. You will need to replace them with the values for your node. Also, this example assumes you are logged in as the Linux `root` user when running `logrotate`.

```shell
cat > /var/tmp/tracelog-logrotate.conf <<EOF
/var/lib/memsql/master-3306-MI4478312f/tracelogs/memsql.log /var/lib/memsql/master-3306-MI4478312f/tracelogs/query.log {
         daily
         rotate 7
         missingok
         compress
         sharedscripts
         postrotate
             # Send SIGHUP to both memsqld processes
             killall -q -s1 memsqld
         endscript
}
EOF

```

The directives used in the above example indicate the following:

* `daily`: Log files are rotated every day; alternately, you can rotate them weekly or monthly.
* `rotate` \[count]: Specifies the number of times log files are rotated before being deleted. In the example, log files are rotated 7 times before being deleted. You can specify the number of times you want to rotate the old log files before deleting them. If run with the `--mail` option, the log files will be rotated the specified number of times before being emailed, rather than being removed.
* `missingok`: If a specified log file is missing, this directive ensures that `logrotate` rotates the next file in the queue without throwing an error.
* `compress`: Old versions of log files are compressed with gzip by default.
* `sharedscripts`: With this directive specified, the `postrotate` script is run only one time (after the old logs have been compressed), not each time a log file is rotated.
* `postrotate/endscript`: The lines between these directives are executed after the log file is rotated. The example uses the `killall -q -s1 memsqld` command, which allows for a new tracelog to be generated and written to the next trace.

## Scheduling `logrotate`

The `logrotate` process is typically run as a daily `cron` job by the `cron` daemon. The package manager used by the system creates a default schedule in `/etc/cron.daily/logrotate` that runs `logrotate` with the default `/etc/logrotate.conf` configuration. If it is not automatically created, you can manually create one and add the following code snippet.

```
#! /bin/sh
/user/sbin/logrotate /etc/logrotate.conf

```

You can also create `cron` jobs in the `cron.hourly`, `cron.weekly`, or `cron.monthly` folders. The jobs located in these folders and the `cron.daily` folder are run by the `cron` service, per the instructions in the `/etc/crontab` file. If you want to run `logrotate` with a custom schedule, create a `cron` job and place it under `/etc/cron.d/`.

Note that the `cron` jobs stored in `/etc/cron.d/` and `/etc/crontab` directories are system `cron` jobs. System `cron` jobs are defined in shared directories on the filesystem. You need `sudo` privileges to define system `cron` jobs.

`Cron` jobs are defined using the [unix-cron](https://man7.org/linux/man-pages/man5/crontab.5.html) format in both `/etc/cron.d/` and `/etc/crontab` directories. The schedule consists of a string of five values indicating when the `cron` job must be executed. These values represent date and time fields, including minute, hour, day of month, and day of week, in that order. The string value is followed by the username under which the `cron` job must run and then the command to execute. For example, the following `cron` job issues `logrotate` using the `/etc/tracelog-logrotate.conf` configuration every Saturday at five minutes past midnight.

```
5 0 * * 6 root /usr/sbin/logrotate /etc/tracelog-logrotate.conf
```

## Manually Running `logrotate`

To manually run `logrotate`, use the following command and specify the configuration file.

```shell
logrotate -f /var/tmp/tracelog-logrotate.conf
```

To list the compressed log files, run the following command.

```shell
ls /var/lib/memsql/master-3306-MI4478312f/tracelogs/*.gz
```

## Alternatives to `logrotate`

As an alternative to `logrotate`, you can configure SingleStore to log using the `syslog` daemon, such as `journald`, and then implement log rotation. Since logging is a host-level activity, you can use a logging system that is ideal for your environment.

***

Modified at: January 18, 2024

Source: [/db/v9.1/reference/troubleshooting-reference/trace-log/rotate/](https://docs.singlestore.com/db/v9.1/reference/troubleshooting-reference/trace-log/rotate/)

(An index of the documentation is available at /llms.txt)
