Troubleshooting Audit Logging

You may encounter problems with the audit logging feature during initial configuration, log file rotation, or general use. This section provides troubleshooting information for common error scenarios.

Note

When SingleStore is unable to write log entries for any reason, it will retry the operation every 10 seconds for 30 attempts. Additionally, all errors are written in a node’s tracelog. The tracelog should be used as the primary source of information while troubleshooting.

Configuration Issues

If you specify any invalid configuration value in your memsql.cnf file, the node won’t be able to start, and an error message will be printed to the console.

To fix a configuration problem, review your memsql.cnf file on the node and ensure that a valid audit logging variable has been specified.

Failed to Write Audit Log Record

The most common audit logging error you may encounter is a failure to write an entry in the audit log. When SingleStore is unable to write an entry to a log for any node in a cluster, the database will not process incoming queries until the problem is resolved. Therefore, the causes of this error are particularly important to understand.

This error writes the following message in the affected node’s tracelog: Failed to write audit log record., followed by a more detailed message of the cause. If your database becomes unresponsive and this error message is present, ensure that the following conditions are not present.

Available Disk Space

If your destination directory runs out of available disk space, SingleStore will be unable to write new log files. To fix this problem, either specify a new destination directory that has sufficient space, or make space available on the current destination directory.

Destination Directory Access

The specified destination directory may not be accessible for many reasons. The most common cases are described below:

  • If your destination directory’s permissions are misconfigured, SingleStore will not be able to write log files. Either specify a new destination directory that has sufficient permissions, or change the permissions on your current destination directory.

  • If your destination directory doesn’t exist or was removed, SingleStore will not be able to write log files. This problem may also be the root cause of more specific log file errors. Ensure that the destination directory actually exists.

  • If your destination directory is located on a network share and the node loses network connectivity, SingleStore will not be able to write log files. This condition usually resolves itself when network access is restored. If a node reconnects to the cluster but the destination directory is still inaccessible, you may need to restart the node.

Last modified: May 29, 2024

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined
  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined
  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
    cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
    --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
    --bundle undefined \
    --new-bundle-format -
    Verified OK