SingleStore DB

Location and Management

The tracelog is located in the tracelogs directory with the file name memsql.log. When the server is started, it opens this file in append mode and begins to log messages.

You can rotate the log by moving the memsql.log file and then sending SIGHUP to the memsqld process. This will trigger the server to reopen memsql.log and continue writing.

SingleStore DB manages code compilation with a separate process. This process also maintains a tracelog named command.log.

Use the logrotate utility combined with a configuration file to automatically manage SingleStore DB log rotation. The example shown below rotates two log files. The example showcases a generic configuration file, which you can modify to suit your requirements.

Note: The example paths use the default folder location for SingleStore DB with an example ID value in the folder name for a master aggregator node. You should replace them with the values for your node. Also, this example assumes you are logged in as root when running logrotate.

# cat > /var/tmp/logrotate.conf <<EOF
/var/lib/memsql/master-3306-MI4478312f/tracelogs/memsql.log /var/lib/memsql/master-3306-MI4478312f/tracelogs/query.log {
         daily
         rotate 7
         missingok
         compress
         sharedscripts
         postrotate
             # Send SIGHUP to both memsqld processes
             killall -q -s1 memsqld
         endscript
}
EOF

The directives used in the above example indicate the following:

  • daily - Log files are rotated every day; alternately, you can rotate them weekly or monthly.

  • rotate [count] - Species the number of times log files are rotated before being deleted. In the example, log files are rotated 7 times before being deleted. You can specify the number of times you want to rotate the old log files before deleting them. If is run with the --mail option, the log files will be rotated the specified number of times before being emailed, rather than being removed.

  • missingok - If a specified log file is missing, this directive ensures that logrotate rotates the next file in the queue without throwing an error.

  • compress - Old versions of log files are compressed with gzip, by default.

  • sharedscripts - With this directive specified, the postrotate script used in the file is run only once after a log rotation and not once for each log that is rotated.

  • postrotate/endscript- The lines between these directives are executed after the log file is rotated. The example uses the killall -q -s1 memsqld command, which allows for a new tracelog to be generated and written to the next trace.

Manually run logrotate and point to the configuration file created above.

# logrotate -f /var/tmp/logrotate.conf

List the compressed log files.

# ls /var/lib/memsql/master-3306-MI4478312f/tracelogs/*.gz

For more information about logrotate, see the logrotate manpage.

As an alternative to logrotate, you can configure SingleStore DB to log trace using syslog daemon, such as journald, and then implement log rotation. Since logging is a host-level activity, you can use a logging system that is ideal for your environment.

memsql.log Log Format

The log entries in the memsql.log file adhere to a predefined syntax.

<time-since-server-startup> <timestamp> <logging-level> <message>

time-since-server-startup: As the name indicates, this value provides the time since the server startup in microseconds.

timestamp: Specifies the node’s local system time when an operation was executed. The timestamp is printed in %Y-%m-%d %H:%M:%S.%f format.

logging-level: Specifies the logging level as INFO, ERROR, WARN, or FAILURE.

  • INFO: Provides informational messages about routine application operations and state.

  • ERROR: Indicates error events that might slow down the application or stop the application from functioning.

  • WARN: Designates potentially harmful events and may warrant further investigation.

  • FAILURE: Indicates the failure of a critical service or an application.

message: Depending on the logging level of each entry, the message and its components may vary for each log entry.

Note

INFO and WARN messages largely demonstrate the normal functioning of an application and should be used for debugging and root cause analysis. Although these messages can be used to study the issues and behavior of an application, it is important to understand that their occurrences are largely expected.

Below is the sample log entry in the memsql.log file.

120 2021-08-10 07:44:28.737 INFO: Log opened

Where:

  • 120 is the time-since-server-startup

  • 2021-08-10 is the timestamp

  • INFO is the logging-level

  • Log opened is the message

More examples:

01959106 2021-08-10 07:44:30.696   INFO: Initializing OpenSSL
01960040 2021-08-10 07:44:30.697   INFO: MemSQL version hash: b31088a4e045e89608495932c5d7dd1b987848ec (Mon Aug 9 20:55:17 2021 +0000)
...
02053854 2021-08-10 07:44:30.791   INFO: ./memsqld: ready for connections.
02053897 2021-08-10 07:44:30.791   INFO: Version:  '7.5.7'  Socket:  '/var/lib/memsql/306d09a6-b845-4ea9-8b6e-a521d87fcb6a/data/memsql.sock'  Port:  '3306'
...
05088719 2021-08-10 07:44:33.826  ERROR: Thread 115058: RefreshMetadata: Metadata refresh and replication management postponed until node can identify itself.
...
05110359 2021-08-10 07:44:33.847  ERROR: ProcessNetworkEvents Heartbeat connection error reading RPC (2 = End of file)
...
11827443 2021-08-10 07:44:40.564   WARN: Attempted to set network write buffer size to 0x800000 bytes, but the value is capped at 0x34000 bytes.
...