AWS Proxy Settings

Navigate to Settings > AWS Proxy Settings, enter the following configuration details.

1. Proxy Host: Enter the S3 proxy host name.

2. Proxy Port: Enter the S3 proxy port.

3. Proxy User Id: Enter the S3 proxy user ID.

4. Proxy Password: Enter the S3 proxy password.

Select Apply to save the settings.

AWS Credentials

Ingest can access AWS services using either IAM roles or access keys when deployed on-premises. You must configure the access method and credentials in Ingest.

To access AWS services from an on-premises SingleStore Flow server, navigate to Settings > AWS Credentials, and perform the following steps.

1. Enable Use EC2 IAM Role, if you are using.

2. Select the AWS Region from the list.

3. Enter the AWS access key ID in AWS Key and AWS Secret Key to access the S3 service (if installed on-premises). For installations using IAM roles, these keys are not required. Note: Keys are encrypted within Ingest.

4. If you are using KMS, enter the KMS Key in AWS Key. Note: Keys are encrypted within Ingest.

Select Apply to confirm and save the details.

AWS Recovery

Ingest supports high availability, which means it automatically saves the current configuration and execution state to S3. This enables recovery of the Ingest instance (including its current state) if it is catastrophically lost.

SingleStore Flow keeps a backup of every successful job execution in S3 that makes the latest version available for recovery. To configure it, navigate to Settings > AWS Recovery, and perform the following steps.

1. In Ingest Instance Name, enter a name for the current instance of Ingest.

2. Check Enable Recovery.

3. In S3 Recovery Location, enter the destination for recovery data in S3, for example: s3://your_bucket_name/your_folder_name/Ingestyour__name.

4. In S3 KMS Id, enter the KMS ID to encrypt logs in S3. This is optional but recommended. When provided, logs are encrypted using KMS encryption.

Select Apply to save the settings.

import json
import boto3

def lambda_handler(event, context):
    freeGb = 100
    cloudwatch = boto3.client('logs')
    response = cloudwatch.get_log_events(
        logGroupName='Oracle_LogGroup',
        logStreamName='Oracle_LogStream',
        startFromHead=False,
        limit=100
    )
    for i in response['events']:
        msg = json.loads(i['message'])
        if msg['type'] == "Status" and msg['diskFreeGB'] < freeGb:
            sns = boto3.client('sns')
            sns.publish(
                TopicArn='arn:aws:sns:us-west-2:689564010160:LambdaTrigger',
                Message='Your free disk size is getting low, please contact the concerned team!'
            )

AWS CloudWatch Logs

Ingest comes pre-configured with remote monitoring capabilities that leverage existing AWS technologies, such as CloudWatch Logs and Events. CloudWatch, along with other AWS services, can monitor the execution of Ingest and trigger appropriate alarms in case of errors or failures. The events from Ingest application flows to CloudWatch Logs. These events provide detailed application statistics that can be used for custom monitoring.

In addition to integration with CloudWatch, Ingest writes internal logs directly to S3, including execution and error logs from the Ingest console. Navigate to Settings > AWS CloudWatch Logs, and perform the following steps.

1. Check Enable CloudWatch Logs and metrics to get notified. AWS CloudWatch events can be used for alerting by writing custom Lambda functions.

2. Enter the name of the CloudWatch Log Group. This needs to be created first in the AWS console.To create a CloudWatch Log Group, refer to CreateLogGroup for more information.

3. Enter the name of the CloudWatch Log Stream. This also needs to be created first in the AWS console. To create a CloudWatch Log Stream, refer to CreateLogStream for more information.

4. Select Apply to save the settings.

Refer to Appendix for more information.

AWS CloudWatch Metrics

Navigate to Settings AWS CloudWatch Metrics, and perform the following steps.

1. Check Enable CloudWatch Metrics if required.

2. In CloudWatch Metrics Namespace, enter the name of CloudWatch metric. Refer to View available metrics for more information.

3. Select Apply to save the settings.

The events that Ingest pushes to AWS CloudWatch Logs and the metrics console are as follows. Refer to Appendix and Appendix for more information.

AWS SNS Notifications

Navigate to Settings > AWS SNS Notifications, and perform the following steps.

1. Check Enable SNS notifications and subscribe to the SNS topic via the AWS console.

2. In SNS Topic, enter the Topic ARN or SNS Topic Name. To create an SNS Topic, refer to Creating an Amazon SNS Topic for more information.

3. Select Apply to save the settings.

Refer to Appendix for more information.

AWS S3 Logging

Navigate to Settings > AWS S3 Logging, and perform the following steps.

1. Check Enable S3 Logging if you want to record data to S3 (console/execution logs).

2. In S3 Log Location, enter the destination of the logging data in S3, for example: s3://your_bucket_name/your_folder_name.

3. In S3 KMS Id, enter the KMS ID to encrypt logs in S3. This is optional but recommended. When provided, logs are encrypted using KMS encryption.

4. Select Apply to save the settings.

Metadata Settings

Navigate to Settings > Metadata Settings, and perform the following steps.

1. Check Enable Operational Metadata, if required.

2. Select metadata database from the Database Type list: AWS Aurora.

3. In Host Name, enter the IP address or hostname of the database server.

4. In Port, enter the port number on which the database server is listening.

5. In Database Name, enter the name of your database.

6. In Database Schema, enter the schema name of your database.

7. In User Id, enter a valid database user ID to be used with Ingest.

8. Enter the Password and confirm it by re-entering it in the Confirm Password field.

9. JDBC Options are optional and can be used to extend the JDBC URL for accessing the database.

10. Select Apply to save the settings.