# Trace Log Location and Management ## Location SingleStore maintains a tracelog with the file name `memsql.log`, which contains the consolidated general/error log for reporting INFO, WARNING, ERROR, and FAILURE messages. When the SingleStore (`memsqld`) server is started, it opens this file in append mode and begins to log messages. The `memsql.log` file typically resides under the `/var/lib/memsql//tracelogs` directory. The path to the tracelogs directory can, however, be changed using the `tracelogsdir` engine variable. In addition to the `memsql.log` file, SingleStore maintains a `command.log` file for the code compilation process, which is separate from the `memsqld` process. The `command.log` file contains information about commands that require forking to start a background process and information that is helpful for tracing errors/warnings from the external process that runs LLVM for code-generation. The `command.log` file is also located in the `tracelogs` directory. Note that SingleStore maintains both `memsql.log` and `command.log` files by default. ## Log Format The log entries in the `memsql.log` file adhere to a predefined syntax. ``` ``` `time-since-server-startup`: As the name indicates, this value provides the time since the server startup in microseconds. `timestamp`: Specifies the node’s local system time when an operation was executed. The timestamp is printed in `%Y-%m-%d %H:%M:%S.%f` format. `logging-level`: Specifies the logging level as INFO, ERROR, WARN, or FAILURE. * `INFO`: Provides informational messages about routine application operations and state. * `ERROR`: Indicates error events that might slow down the application or stop the application from functioning. * `WARN`: Designates potentially harmful events and may warrant further investigation. * `FAILURE`: Indicates the failure of a critical service or an application. `message`: Depending on the logging level of each entry, the message and its components may vary for each log entry. > **📝 Note**: `INFO` and `WARN` messages largely demonstrate the normal functioning of an application and should be used for debugging and root cause analysis. Although these messages can be used to study the issues and behavior of an application, it is important to understand that their occurrences are largely expected. Below is the sample log entry in the memsql.log file. ``` 120 2021-08-10 07:44:28.737 INFO: Log opened ``` Where: * `120` is the `time-since-server-startup` * `2021-08-10` is the `timestamp` * `INFO` is the `logging-level` * `Log opened` is the `message` **More examples**: ``` 01959106 2021-08-10 07:44:30.696   INFO: Initializing OpenSSL 01960040 2021-08-10 07:44:30.697   INFO: MemSQL version hash: b31088a4e045e89608495932c5d7dd1b987848ec (Mon Aug 9 20:55:17 2021 +0000) ... 02053854 2021-08-10 07:44:30.791   INFO: ./memsqld: ready for connections. 02053897 2021-08-10 07:44:30.791   INFO: Version:  '7.5.7'  Socket:  '/var/lib/memsql/306d09a6-b845-4ea9-8b6e-a521d87fcb6a/data/memsql.sock'  Port:  '3306' ... 05088719 2021-08-10 07:44:33.826  ERROR: Thread 115058: RefreshMetadata: Metadata refresh and replication management postponed until node can identify itself. ... 05110359 2021-08-10 07:44:33.847  ERROR: ProcessNetworkEvents Heartbeat connection error reading RPC (2 = End of file) ... 11827443 2021-08-10 07:44:40.564   WARN: Attempted to set network write buffer size to 0x800000 bytes, but the value is capped at 0x34000 bytes. ... ``` ## Configuration Parameters * `minimal_disk_space`: Defines the minimal disk space in MB, which, when reached on the disk where the data directories (such as logs, snapshots, blobs, plancache, auditlogs, and tracelogs) are mounted, SingleStore will halt new write queries. As tracelog files can potentially grow in size until the disk space limit, it is important to monitor this configuration. * `tracelogsdir`: The path to the tracelog destination directory, which contains `memsql.log` and `command.log` files, among others. By default, this variable points to the `/var/lib/memsql//tracelogs` directory. * `tracelog_level_*`: When a `tracelog_level_*` variable is set, each component traces to a separate `memsql_$COMPONENT.log` file. Many of these files will be empty and all of them are meant for internal SingleStore use only. *** Modified at: January 26, 2024 Source: [/db/v9.1/reference/troubleshooting-reference/trace-log/location-and-management/](https://docs.singlestore.com/db/v9.1/reference/troubleshooting-reference/trace-log/location-and-management/) (An index of the documentation is available at /llms.txt)