Trace Log Location and Management

Location

SingleStore maintains a tracelog with the file name memsql.log, which contains the consolidated general/error log for reporting INFO, WARNING, ERROR, and FAILURE messages. When the SingleStore (memsqld) server is started, it opens this file in append mode and begins to log messages. The memsql.log file typically resides under the /var/lib/memsql/<nodeID>/tracelogs directory. The path to the tracelogs directory can, however, be changed using the tracelogsdir engine variable.

In addition to the memsql.log file, SingleStore maintains a command.log file for the code compilation process, which is separate from the memsqld process. The command.log file contains information about commands that require forking to start a background process and information that is helpful for tracing errors/warnings from the external process that runs LLVM for code-generation. The command.log file is also located in the tracelogs directory.

Note that SingleStore maintains both memsql.log and command.log files by default.

Log Format

The log entries in the memsql.log file adhere to a predefined syntax.

<time-since-server-startup> <timestamp> <logging-level> <message>

time-since-server-startup: As the name indicates, this value provides the time since the server startup in microseconds.

timestamp: Specifies the node’s local system time when an operation was executed. The timestamp is printed in %Y-%m-%d %H:%M:%S.%f format.

logging-level: Specifies the logging level as INFO, ERROR, WARN, or FAILURE.

INFO: Provides informational messages about routine application operations and state.
ERROR: Indicates error events that might slow down the application or stop the application from functioning.
WARN: Designates potentially harmful events and may warrant further investigation.
FAILURE: Indicates the failure of a critical service or an application.

message: Depending on the logging level of each entry, the message and its components may vary for each log entry.

Note

INFO and WARN messages largely demonstrate the normal functioning of an application and should be used for debugging and root cause analysis. Although these messages can be used to study the issues and behavior of an application, it is important to understand that their occurrences are largely expected.

Below is the sample log entry in the memsql.log file.

120 2021-08-10 07:44:28.737 INFO: Log opened

Where:

120 is the time-since-server-startup
2021-08-10 is the timestamp
INFO is the logging-level
Log opened is the message

More examples:

01959106 2021-08-10 07:44:30.696   INFO: Initializing OpenSSL
01960040 2021-08-10 07:44:30.697   INFO: MemSQL version hash: b31088a4e045e89608495932c5d7dd1b987848ec (Mon Aug 9 20:55:17 2021 +0000)
...
02053854 2021-08-10 07:44:30.791   INFO: ./memsqld: ready for connections.
02053897 2021-08-10 07:44:30.791   INFO: Version:  '7.5.7'  Socket:  '/var/lib/memsql/306d09a6-b845-4ea9-8b6e-a521d87fcb6a/data/memsql.sock'  Port:  '3306'
...
05088719 2021-08-10 07:44:33.826  ERROR: Thread 115058: RefreshMetadata: Metadata refresh and replication management postponed until node can identify itself.
...
05110359 2021-08-10 07:44:33.847  ERROR: ProcessNetworkEvents Heartbeat connection error reading RPC (2 = End of file)
...
11827443 2021-08-10 07:44:40.564   WARN: Attempted to set network write buffer size to 0x800000 bytes, but the value is capped at 0x34000 bytes.
...

Configuration Parameters

minimal_disk_space: Defines the minimal disk space in MB, which, when reached on the disk where the data directories (such as logs, snapshots, blobs, plancache, auditlogs, and tracelogs) are mounted, SingleStore will halt new write queries. As tracelog files can potentially grow in size until the disk space limit, it is important to monitor this configuration.
tracelogsdir: The path to the tracelog destination directory, which contains memsql.log and command.log files, among others. By default, this variable points to the /var/lib/memsql/<nodeID>/tracelogs directory.
tracelog_level_*: When a tracelog_level_* variable is set, each component traces to a separate memsql_$COMPONENT.log file. Many of these files will be empty and all of them are meant for internal SingleStore use only.