Trace Log Location and Management
Location
SingleStoreDB maintains a tracelog with the file name memsql.log, which contains the consolidated general/error log for reporting INFO, WARNING, ERROR, and FAILURE messages. When the SingleStoreDB (memsqld) server is started, it opens this file in append mode and begins to log messages. The memsql.log file typically resides under the /var/lib/memsql/<nodeID>/tracelogs directory. The path to the tracelogs directory can, however, be changed using the tracelogsdir engine variable.
In addition to the memsql.log file, SingleStoreDB maintains a command.log file for the code compilation process, which is separate from the memsqld process. The command.log file contains information about commands that require forking to start a background process and information that is helpful for tracing errors/warnings from the external process that runs LLVM for code-generation. The command.log file is also located in the tracelogs directory.
Note that SingleStoreDB maintains both memsql.log and command.log files by default.
Log Format
The log entries in the memsql.log file adhere to a predefined syntax.
<time-since-server-startup> <timestamp> <logging-level> <message>
time-since-server-startup: As the name indicates, this value provides the time since the server startup in microseconds.
timestamp: Specifies the node’s local system time when an operation was executed. The timestamp is printed in %Y-%m-%d %H:%M:%S.%f format.
logging-level: Specifies the logging level as INFO, ERROR, WARN, or FAILURE.
INFO: Provides informational messages about routine application operations and state.ERROR: Indicates error events that might slow down the application or stop the application from functioning.WARN: Designates potentially harmful events and may warrant further investigation.FAILURE: Indicates the failure of a critical service or an application.
message: Depending on the logging level of each entry, the message and its components may vary for each log entry.
Note
INFO and WARN messages largely demonstrate the normal functioning of an application and should be used for debugging and root cause analysis. Although these messages can be used to study the issues and behavior of an application, it is important to understand that their occurrences are largely expected.
Below is the sample log entry in the memsql.log file.
120 2021-08-10 07:44:28.737 INFO: Log opened
Where:
120is thetime-since-server-startup2021-08-10is thetimestampINFOis thelogging-levelLog openedis themessage
More examples:
01959106 2021-08-10 07:44:30.696 INFO: Initializing OpenSSL 01960040 2021-08-10 07:44:30.697 INFO: MemSQL version hash: b31088a4e045e89608495932c5d7dd1b987848ec (Mon Aug 9 20:55:17 2021 +0000) ... 02053854 2021-08-10 07:44:30.791 INFO: ./memsqld: ready for connections. 02053897 2021-08-10 07:44:30.791 INFO: Version: '7.5.7' Socket: '/var/lib/memsql/306d09a6-b845-4ea9-8b6e-a521d87fcb6a/data/memsql.sock' Port: '3306' ... 05088719 2021-08-10 07:44:33.826 ERROR: Thread 115058: RefreshMetadata: Metadata refresh and replication management postponed until node can identify itself. ... 05110359 2021-08-10 07:44:33.847 ERROR: ProcessNetworkEvents Heartbeat connection error reading RPC (2 = End of file) ... 11827443 2021-08-10 07:44:40.564 WARN: Attempted to set network write buffer size to 0x800000 bytes, but the value is capped at 0x34000 bytes. ...
Configuration Parameters
minimal_disk_space: Defines the minimal disk space in MB, which, when reached on the disk where the data directories (such as logs, snapshots, blobs, plancache, auditlogs, and tracelogs) are mounted, SingleStore will halt new write queries. As tracelog files can potentially grow in size until the disk space limit, it is important to monitor this configuration.tracelogsdir: The path to the tracelog destination directory, which containsmemsql.logandcommand.logfiles, among others. By default, this variable points to the/var/lib/memsql/<nodeID>/tracelogsdirectory.