Environment Preparation

Create an AWS Account: To prepare the environment for Flow in AWS, you need an AWS account. If you already have an AWS account, proceed to the next step. If not, refer to How do I create and activate a new AWS account? to create one.
Create an IAM User: SingleStore recommends creating a separate IAM user for managing all AWS services. Do not use the root user for any tasks. Refer to Setting up your AWS account to create an IAM admin user.
Create and Assign Policies to the User: Use the AWS Management Console to create a customer-managed policy and attach it to the IAM user based on their role. This policy enables the IAM user to sign in directly to the AWS Management Console with the assigned permissions.
Sign in to AWS:
1. As an IAM User: Sign in to the AWS Management Console using your Account ID or account alias, along with your username and password. Refer to Sign in to the AWS Management Console for more information.
2. AWS SSO: Sign in using IAM Identity Center (AWS SSO). Refer to How to create and manage users within AWS IAM Identity Center for more information.
Create a VPC: A Virtual Private Cloud (VPC) is a dedicated virtual network for your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. You can launch your Flow application and related AWS resources, such as Amazon EC2 instances, into this VPC. Refer to Configure a virtual private cloud for instructions on creating a VPC.
Create a Private Subnet in Your VPC: Since Ingest needs to be set up in your VPC, SingleStore recommends creating a new private subnet within the VPC for Flow. Refer to Create a VPC to create a subnet.
Create a Security Group: A security group acts as a virtual firewall for your instances that controls inbound and outbound traffic. Security groups apply at the instance level, not the subnet level. Each instance in a subnet can be assigned different security groups. SingleStore does not recommend using the default security group, which AWS assigns if you don’t specify a security group at launch. You can add rules to each security group to control inbound and outbound traffic. Refer to Control traffic to your AWS resources using security groups for more information.
Configure Security Group Rules: You can add or remove rules to a security group to authorize or revoke inbound or outbound access. A rule applies either to inbound traffic (ingress) or outbound traffic (egress). You can grant access to a specific CIDR range or to another security group in your VPC or in a peer VPC (with a VPC peering connection).
Create an IAM Role: Flow uses an IAM role assigned to the EC2 instance where the application is hosted. The EC2 role must have the necessary policies attached. Refer to Create a role to delegate permissions to an IAM user for instructions on creating an IAM role for Flow. Attach the required policies to the newly created IAM role.
Assign the Role to Users or Groups: The IAM role must be assigned to an AWS Directory Service user or group. The role must have a trust relationship with AWS Directory Service. Refer to Assigning users or groups to an existing IAM role to assign users or groups to an IAM role.
Create Access Key ID and Secret Access Key: Flow uses an access key ID and secret access key to connect to AWS services from an on-premises server. SingleStore recommends generating a set of access keys for the Flow user account. The following are the steps to create access keys from the admin user account:
1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.
2. In the navigation pane, select Users.
3. Select the separateSingleStore Flow user whose access keys you want to manage, then select the Security credentials tab.
4. In Access keys, select Create access key. Then, select Download .csv file to save the access key ID and secret access key to a CSV file. Store this file securely, as the secret access key cannot be accessed again after closing the dialog.
5. After downloading the CSV file, select Close. The access key is active by default, and you can use it. Refer to AWS security credentials for more information.
  
  Note: SingleStore recommends rotating access keys every 90 days. Refer to Data Security and Encryption for more information.
Create an Auto-Scaling Group: When deploying Ingest in a high-availability (HA) environment, SingleStore recommends configuring your EC2 instances with an Auto Scaling Group. Follow the steps mentioned in Create an Auto Scaling group using the Amazon EC2 launch wizard to launch an Auto Scaling group via the AWS Console. The recommended parameters for configuring the Auto Scaling group are:
1. Select the instance type recommended by SingleStore under the Prerequisites section.
2. Set the minimum number of instances to 2 for a high-availability deployment.
3. Select the IAM role as SingleConnectEc2Role.
4. Configure the storage as recommended under the Additional AWS Services section.
5. Select the security group created for Flow in the previous steps.
6. Select the key pair for the launch configuration.
7. Refer to Create an Auto Scaling group using the Amazon EC2 launch wizard for remaining steps.

Create an EC2 System

Refer to Get started with Amazon EC2 for more information.

Optimize Usage of AWS Resources

Tagging AWS Resources

AWS allows customers to assign metadata to their AWS resources through tags. SingleStore highly recommends tagging all AWS resources created for and by SingleStore Flow to manage and organize resources, control access, track costs, and automate processes.

For clarity and organization, SingleStore recommends to use tags with names specific to the instances being created. For example, for a Flow instance replicating a production database server for Billing and Finance, the tag name must reflect the database name it is dedicated to, such as SingleStoreIngest_BFS_EC2_Prod. Similarly, for a user acceptance testing (UAT) environment, the tag name can be SingleStoreIngest_BFS_EC2_UAT. This approach helps customers differentiate between various AWS resources in their environment. Use consistent tag names across all services.

SingleStore recommends tagging AWS EC2 services with unique and identifiable tag names. Refer to Tag your Amazon EC2 resources for more information on tagging AWS EC2 resources.

AWS Identity and Access Management (IAM) for SingleStore Flow

AWS IAM roles delegate access to AWS resources. With IAM roles, you can establish trust relationships between your trusting account and other trusted AWS accounts. The trusting account owns the resource to be accessed, while the trusted account contains the users who need access.

SingleStore’s Recommendations:

Create an IAM User (e.g., SingleStore_Flow_User). Do NOT use the root user account to set up the application. Refer to Create an IAM user in your AWS account for creating an IAM user.
Create an IAM Role (e.g., SingleStore_Flow_EC2Role). Refer to IAM role creation for creating IAM roles.
Create an IAM Policy (e.g., SingleStore_Flow_Policy) and assign custom policies (provided below) to the EC2 role. Refer to Creating IAM policies for creating a policy.
Instead of defining permissions for individual SingleStore Flow IAM users, it is more convenient to create groups based on job functions (e.g., administrators, developers, accounting, etc.). Define the relevant permissions for each group and assign IAM users to those groups. All users in an IAM group inherit the permissions assigned to the group. This way, you can make changes for everyone in the group in one place.
Only grant the minimal required permissions to the IAM role. The SingleStore Flow user requires basic permissions for S3, CloudWatch, SNS and DynamoDB.
SingleStore Flow needs access to the following AWS services: S3, EC2, SNS, and CloudWatch with the minimum privileges outlined below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "1",
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::<bucket_name>"
    },
    {
      "Sid": "2",
      "Action": [
        "ec2:AcceptVpcEndpointConnections",
        "ec2:AcceptVpcPeeringConnection",
        "ec2:AssociateIamInstanceProfile",
        "ec2:CreateTags",
        "ec2:DescribeTags",
        "ec2:RebootInstances"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:ec2:<region>:<account_id>:instance/<ec2_instance_id>"
    },
    {
      "Sid": "4",
      "Action": [
        "sns:Publish"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:sns::<region>:<account_id>:<sns_name>"
    },
        {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "dynamodb:CreateTable",
        "dynamodb:PutItem",
        "dynamodb:Update*",
        "dynamodb:Get*",
        "dynamodb:Scan"
      ],
      "Resource": "arn:aws:dynamodb::<region>:table/SingleConnectTable"
    },
  ]
}

Roles Defined for SingleStore Flow

The following are the roles and permissions required to launch and manage the Flow suite of applications.

Role	Type	Permissions/Policies	Purpose
EC2Admin	AWS Custom Role for EC2	List-DescribeInstanceStatus Directory Service List,Write-DescribeDirectories,CreateComputer Systems Manager List,Read,Write ListAssociations, ListInstanceAssociations, DescribeAssociation, DescribeDocument, GetDeployablePatchSnapshotForInstance, GetDocument, GetManifest, GetParameters, PutComplianceItems, PutInventory, UpdateAssociationStatus, UpdateInstanceAssociationStatus, UpdateInstanceInformation	Create and Manage EC2 instance
DBAdmin	AWS Custom Role	cloudwatch:DeleteAlarms cloudwatch:Describe* cloudwatch:DisableAlarmActions cloudwatch:EnableAlarmActions cloudwatch:Get* cloudwatch:List* cloudwatch:PutMetricAlarm dynamodb:CreateTable dynamodb:BatchGetItem dynamodb:BatchWriteItem dynamodb:ConditionCheckItem dynamodb:PutItem dynamodb:DescribeTable dynamodb:DeleteItem dynamodb:GetItem dynamodb:Scan dynamodb:Query dynamodb:UpdateItem ec2:DescribeAccountAttributes ec2:DescribeAddresses ec2:DescribeAvailabilityZones ec2:DescribeInternetGateways ec2:DescribeSecurityGroups ec2:DescribeSubnets ec2:DescribeVpcs iam:ListRoles iam:GetRole kms:ListKeys logs:DescribeLogGroups logs:DescribeLogStreams logs:FilterLogEvents logs:GetLogEvents logs:Create* logs:PutLogEvents sns:Get sns:List* sns:SetTopicAttributes	Manage DB access and privileges
Network Admin	Custom Role	autoscaling:Describe, directconnect:, ec2:AcceptVpcEndpointConnections, ec2:AllocateAddress, ec2:AssignIpv6Addresses, ec2:AssignPrivateIpAddresses, ec2:AssociateAddress, ec2:AssociateDhcpOptions, ec2:AssociateRouteTable, ec2:AssociateSubnetCidrBlock, ec2:AssociateVpcCidrBlock, ec2:AttachInternetGateway, ec2:AttachNetworkInterface, ec2:AttachVpnGateway, ec2:CreateCarrierGateway, ec2:CreateCustomerGateway, ec2:CreateDefaultSubnet, ec2:CreateDefaultVpc, ec2:CreateDhcpOptions, ec2:CreateEgressOnlyInternetGateway, ec2:CreateFlowLogs, ec2:CreateInternetGateway, ec2:CreateNatGateway, ec2:CreateNetworkAcl, ec2:CreateNetworkAclEntry, ec2:CreateNetworkInterface, ec2:CreateNetworkInterfacePermission, ec2:CreatePlacementGroup, ec2:CreateRoute, ec2:CreateRouteTable, ec2:CreateSecurityGroup, ec2:CreateSubnet, ec2:CreateTags, ec2:CreateVpc, ec2:CreateVpcEndpoint, ec2:CreateVpcEndpointConnectionNotification, ec2:CreateVpcEndpointServiceConfiguration, ec2:CreateVpnConnection, ec2:CreateVpnConnectionRoute, ec2:CreateVpnGateway, ec2:DeleteCarrierGateway, ec2:DeleteEgressOnlyInternetGateway, ec2:DeleteFlowLogs, ec2:DeleteNatGateway, ec2:DeleteNetworkInterface, ec2:DeleteNetworkInterfacePermission, ec2:DeletePlacementGroup, ec2:DeleteSubnet, ec2:DeleteTags, ec2:DeleteVpc, ec2:DeleteVpcEndpointConnectionNotifications, ec2:DeleteVpcEndpointServiceConfigurations, ec2:DeleteVpcEndpoints, ec2:DeleteVpnConnection, ec2:DeleteVpnConnectionRoute, ec2:DeleteVpnGateway, ec2:DescribeAccountAttributes, ec2:DescribeAddresses, ec2:DescribeAvailabilityZones, ec2:DescribeCarrierGateways, ec2:DescribeClassicLinkInstances, ec2:DescribeCustomerGateways, ec2:DescribeDhcpOptions, ec2:DescribeEgressOnlyInternetGateways, ec2:DescribeFlowLogs, ec2:DescribeInstances, ec2:DescribeInternetGateways, ec2:DescribeKeyPairs, ec2:DescribeMovingAddresses, ec2:DescribeNatGateways, ec2:DescribeNetworkAcls, ec2:DescribeNetworkInterfaceAttribute, ec2:DescribeNetworkInterfacePermissions, ec2:DescribeNetworkInterfaces, ec2:DescribePlacementGroups, ec2:DescribePrefixLists, ec2:DescribeRouteTables, ec2:DescribeSecurityGroupReferences, ec2:DescribeSecurityGroupRules, ec2:DescribeSecurityGroups, ec2:DescribeStaleSecurityGroups, ec2:DescribeSubnets, ec2:DescribeTags, ec2:DescribeVpcAttribute, ec2:DescribeVpcClassicLink, ec2:DescribeVpcClassicLinkDnsSupport, ec2:DescribeVpcEndpointConnectionNotifications, ec2:DescribeVpcEndpointConnections, ec2:DescribeVpcEndpointServiceConfigurations, ec2:DescribeVpcEndpointServicePermissions, ec2:DescribeVpcEndpointServices, ec2:DescribeVpcEndpoints, ec2:DescribeVpcPeeringConnections, ec2:DescribeVpcs, ec2:DescribeVpnConnections, ec2:DescribeVpnGateways, ec2:DescribePublicIpv4Pools, ec2:DescribeIpv6Pools, ec2:DetachInternetGateway, ec2:DetachNetworkInterface, ec2:DetachVpnGateway, ec2:DisableVgwRoutePropagation, ec2:DisableVpcClassicLinkDnsSupport, ec2:DisassociateAddress, ec2:DisassociateRouteTable, ec2:DisassociateSubnetCidrBlock, ec2:DisassociateVpcCidrBlock, ec2:EnableVgwRoutePropagation, ec2:EnableVpcClassicLinkDnsSupport, ec2:ModifyNetworkInterfaceAttribute, ec2:ModifySecurityGroupRules, ec2:ModifySubnetAttribute, ec2:ModifyVpcAttribute, ec2:ModifyVpcEndpoint, ec2:ModifyVpcEndpointConnectionNotification, ec2:ModifyVpcEndpointServiceConfiguration, ec2:ModifyVpcEndpointServicePermissions, ec2:ModifyVpcPeeringConnectionOptions, ec2:ModifyVpcTenancy, ec2:MoveAddressToVpc, ec2:RejectVpcEndpointConnections, ec2:ReleaseAddress, ec2:ReplaceNetworkAclAssociation, ec2:ReplaceNetworkAclEntry, ec2:ReplaceRoute, ec2:ReplaceRouteTableAssociation, ec2:ResetNetworkInterfaceAttribute, ec2:RestoreAddressToClassic, ec2:UnassignIpv6Addresses, ec2:UnassignPrivateIpAddresses, ec2:UpdateSecurityGroupRuleDescriptionsEgress, ec2:UpdateSecurityGroupRuleDescriptionsIngress, logs:DescribeLogGroups, logs:DescribeLogStreams, logs:GetLogEvents, route53:, route53domains:, sns:CreateTopic, sns:ListSubscriptionsByTopic, sns:ListTopics, ec2:AcceptVpcPeeringConnection, ec2:AttachClassicLinkVpc, ec2:AuthorizeSecurityGroupEgress, ec2:AuthorizeSecurityGroupIngress, ec2:CreateVpcPeeringConnection, ec2:DeleteCustomerGateway, ec2:DeleteDhcpOptions, ec2:DeleteInternetGateway, ec2:DeleteNetworkAcl, ec2:DeleteNetworkAclEntry, ec2:DeleteRoute, ec2:DeleteRouteTable, ec2:DeleteSecurityGroup, ec2:DeleteVolume, ec2:DeleteVpcPeeringConnection, ec2:DetachClassicLinkVpc, ec2:DisableVpcClassicLink, ec2:EnableVpcClassicLink, ec2:GetConsoleScreenshot, ec2:RejectVpcPeeringConnection, ec2:RevokeSecurityGroupEgress, ec2:RevokeSecurityGroupIngress, ec2:CreateLocalGatewayRoute, ec2:CreateLocalGatewayRouteTableVpcAssociation, ec2:DeleteLocalGatewayRoute, ec2:DeleteLocalGatewayRouteTableVpcAssociation, ec2:DescribeLocalGatewayRouteTableVirtualInterfaceGroupAssociations, ec2:DescribeLocalGatewayRouteTableVpcAssociations, ec2:DescribeLocalGatewayRouteTables, ec2:DescribeLocalGatewayVirtualInterfaceGroups, ec2:DescribeLocalGatewayVirtualInterfaces, ec2:DescribeLocalGateways, ec2:SearchLocalGatewayRoutes, s3:GetBucketLocation, s3:GetBucketWebsite, s3:ListBucket, iam:GetRole, iam:ListRoles, iam:PassRole, ec2:AcceptTransitGatewayVpcAttachment, ec2:AssociateTransitGatewayRouteTable, ec2:CreateTransitGateway, ec2:CreateTransitGatewayRoute, ec2:CreateTransitGatewayRouteTable, ec2:CreateTransitGatewayVpcAttachment, ec2:DeleteTransitGateway, ec2:DeleteTransitGatewayRoute, ec2:DeleteTransitGatewayRouteTable, ec2:DeleteTransitGatewayVpcAttachment, ec2:DescribeTransitGatewayAttachments, ec2:DescribeTransitGatewayRouteTables, ec2:DescribeTransitGatewayVpcAttachments, ec2:DescribeTransitGateways, ec2:DisableTransitGatewayRouteTablePropagation, ec2:DisassociateTransitGatewayRouteTable, ec2:EnableTransitGatewayRouteTablePropagation, ec2:ExportTransitGatewayRoutes, ec2:GetTransitGatewayAttachmentPropagations, ec2:GetTransitGatewayRouteTableAssociations, ec2:GetTransitGatewayRouteTablePropagations, ec2:ModifyTransitGateway, ec2:ModifyTransitGatewayVpcAttachment, ec2:RejectTransitGatewayVpcAttachment, ec2:ReplaceTransitGatewayRoute, ec2:SearchTransitGatewayRoutes	Manage Network access and firewall settings
SingleStore FlowAdmin	Custom Role	s3:ListBucket, s3:PutObject, s3:GetObject, s3:DeleteObject, s3:GetBucketLocation, s3:PutObjectAcl, secretsmanager:GetSecretValue, secretsmanager:DescribeSecret, secretsmanager:PutSecretValue, secretsmanager:UpdateSecret	Able to manage SingleStore Flow configurations
AmazonS3	Resource Based Policy	s3:PutObject, s3:GetObject, s3:DeleteObject, s3:GetBucketLocation, s3:PutObjectAclResource: arn:aws:s3:::<bucket-name>, arn:aws:s3:::<bucket-name>/*	To manage bucket level permissions, resource-based policy for S3 must be applied to restrict the bucket level access. The policy is attached to the bucket, but the policy controls access to both the bucket and the objects in it.
AmazonEC2	Resource Based Policy	ec2:AcceptVpcEndpointConnections, ec2:AcceptVpcPeeringConnection, ec2:AssociateIamInstanceProfile, ec2:CreateTags, ec2:DescribeTags, ec2:RebootInstancesResource: arn:aws:ec2:<ec2_instance_id>	To manage instance level permissions, resource-based policy for EC2 must be applied to restrict the access for the EC2 instance.

Recommended Network ACL Rules for EC2

The following table presents the recommended rules for your EC2. These rules block all traffic except for what is explicitly required.

The EC2 security group must include the necessary inbound and outbound rules as per the following:

Inbound
Rule #	Source IP	Protocol	Port	Allow / Deny	Comments
1	Custom IP which requires access to the Ingest application	TCP	80	ALLOW	Allows inbound HTTP traffic only from known/custom IPv4 addresses.
2	Public IPv4 address range of your home network	TCP	22	ALLOW	Allows inbound SSH traffic from your home network over the Internet gateway.
3	Public IPv4 address range of your home network	TCP	3389	ALLOW	Allows inbound RDP traffic from your home network over the Internet gateway.
4	0.0.0.0/0	all	all	DENY	Denies all inbound IPv4 traffic not handled by a preceding rule (not modifiable).
Outbound
Rule #	Dest IP	Protocol	Port	Allow / Deny	Comments
1	Source database host IP address	TCP	Custom port (port specific to source database ports)	ALLOW	Allows connections to the source database.
2	SingleStore cluster host IP address	TCP	8080 (port specific to the SingleStore destination database)	ALLOW	Allows connections to the SingleStore destination database.
3	0.0.0.0/0	all	all	DENY	Denies all outbound IPv4 traffic not handled by a preceding rule (not modifiable).

To open ports on the Amazon Console, refer to What is Amazon EC2? and follow the steps to allow inbound traffic to your Amazon instance.

To open ports on Windows Server, refer to Opening Ports in the Firewall and follow the steps to allow inbound traffic to your server.

Outbound Connections

Ingest connects to any source endpoints outside its VPC using NAT, VPN, or API gateways.

A NAT gateway is a Network Address Translation (NAT) service. You can use a NAT gateway to allow instances in a private subnet to connect to services outside your VPC, while preventing external services from initiating a connection with those instances. Refer to NAT gateways for more information.

To connect the VPC to a remote network and enable source/destination endpoint connections, use AWS VPN. Refer to NAT gateways for more information.

Additional AWS Services

SingleStore Flow uses several AWS resources to fulfill user requirements. The costs of these services are separate from Flow charges and are billed by AWS to your account.

The following list provides an overview of other billable services within Flow. Use the AWS Pricing Calculator to estimate the cost of additional AWS resources.

A sample estimate for a high availability setup with a source data volume of 100 GB is provided for reference. Note that

Note: Not all services are mandatory, and the size and number of services varies for each customer environment. The sample is for reference purposes only. All AWS services have service limits. Check for sufficient resources before launching services, and if necessary, refer to Request a quota increase to request an increase in quota. Refer to Service endpoints and quotas to check the service limits for each service.

Service	Mandatory	Billing Type	Service Limits
AWS EC2	Y	Pay-as-you-go	Check EC2 quota here
Additional EBS storage attached to EC2	Y	Based on size
AWS S3	N	Pay-as-you-go	Check Amazon S3 quota here
AWS CloudWatch Logs and metrics	N	Pay-as-you-go	Check EC2 quota here
AWS SNS	N	Pay-as-you-go	Check AWS SNS quota here
AWS Dynamo DB (5 WCUs /5 RCUs)	N	Pay-as-you-go	Check Dynamo DB quota here
AWS Lambda	N	Pay-as-you-go	Check AWS Lambda quota here
AWS KMS	N	Pay-as-you-go	Check Amazon KMS quota here

SingleStore recommends to use the following mentioned instance types for EC2 with EBS volumes attached.

EC2 Instance Type	SingleStore Flow Enterprise Edition	Recommended EBS volumes	EBS Volume Type
t3.large	Volume < 100 GB	100 GB	General Purpose SSD (gp2) Volumes
t3.xlarge	Volume >100 and < 300 GB	500 GB	General Purpose SSD (gp2) Volumes
t3.2xlarge	Volume > 300 GB and < 1 TB	500 GB	General Purpose SSD (gp2) Volumes

IMDS Settings and Recommendations

Flow uses the latest version of the AWS SDK in each release. It uses IMDSv2 for all API calls to AWS services. SingleStore recommends disabling IMDS after deployment if required. Refer to Use the Instance Metadata Service to access instance metadata for more information on how to disable IMDS.

To modify this using the AWS CLI, run the following command.

Shell

aws ec2 modify-instance-metadata-options \
--instance-id <instance-id> \
--http-endpoint disabled

Data Security and Encryption

SingleStore ensures data security through various mechanisms, including encryption.

With AWS KMS, Ingest uses a customer-specified KMS key to encrypt customer data on AWS services such as EC2, Secrets Manager, and DynamoDB. You can configure the customer KMS ID in SingleStore Flow, which is used to encrypt data stored on these AWS services.

Ingest also supports server-side encryption using AES-256. Amazon S3 server-side encryption employs 256-bit Advanced Encryption Standard (AES-256), one of the strongest block ciphers available, to encrypt your data. Flow supports this encryption by default.

Ingest does not store any data outside the customer's designated environment. It stores data in the following AWS services, depending on customer requirements:

Amazon EC2: Used only for temporary staging and pipeline configuration. Refer to Data protection in Amazon EC2 for enabling encryption.
EBS Storage: Used only for temporary staging and pipeline configuration. Refer to Amazon EBS data security for enabling encryption.
Amazon Aurora DB: Used for exporting pipeline metadata. Refer to Encrypting Amazon Aurora resources for enabling encryption.
Amazon DynamoDB: Used when configured for High Availability. Refer to DynamoDB encryption at rest for enabling encryption.
AWS Secrets Manager: Used for storing all pipeline credentials. Refer to AWS Key Management Service for encryption details.

For non-AWS destination endpoints:

SingleStore: Data encryption is handled within SingleStore.

Manage Access Keys

Flow uses an access key and secret key to authenticate to AWS services like S3, CloudWatch, DynamoDB, and others. It requires the AWS access key ID and secret key to access S3 and other services from on-premises. AWS IAM roles are used when installing on an EC2 instance.

For security reasons, SingleStore recommends rotating access keys and KMS keys periodically, for example, every 90 days. After generating new keys, update them in Ingest’s configuration. Follow these steps:

Open the Ingest instance that needs the new key in a web browser.
Go to the Schedule tab and stop the replication schedule for Ingest by disabling Schedule.
Navigate to Connections > Destination File System.
Enter the new Access Key and Secret Access Key and select Apply.
Once the keys are saved, resume the replication by enabling Schedule.

Refer to Rotate AWS KMS keys for more information on key rotation.

The IAM role for Flow must have the recommended policies attached. Refer to AWS Identity and Access Management (IAM) for SingleStore Flow for the list of policies and permissions.

Key Rotation

SingleStore recommends rotating all keys configured in Ingest every 90 days for security reasons. This includes credentials for all source and destination endpoints. The following are some references for AWS services for more details:

Follow the recommendations for all non-AWS sources and destinations:

External Applications	Reference for Key Rotation
Oracle	Oracle password rotation
Microsoft SQL Server	SQL Server password rotation
MySQL	MySQL password rotation
PostgreSQL	PostgreSQL password rotation

Configure Data Encryption

SingleStore follows AWS recommendations for encrypting data both at rest and in transit. This is achieved by creating keys and certificates used for encryption.

AWS Secrets Manager uses encryption through AWS KMS. Refer to Secret encryption and decryption in AWS Secrets Manager for more information.

Encryption In-Transit

Flow uses SSL to establish secure connections (e.g., with AWS services, databases, etc.) for data flow that ensures secure communication in transit.

SSL involves managing security certificates. It is important to keep these certificates active at all times to ensure uninterrupted service.

AWS Certificate Manager (ACM) handles the complexity of creating and managing public SSL/TLS certificates. Customers can set up notifications to be alerted before the certificate expiration date and can renew certificates in advance to ensure uninterrupted service. Refer to Managed certificate renewal in AWS Certificate Manager for more information on managing ACM.

Testing the Connections

Verify whether the connectivity to remote services is available.

To test the remote connections, you need the Telnet utility, which must be enabled from the Control Panel under Turn On Windows Features.

Follow these steps:

Open the Start menu and select Run.
Type CMD and select OK.
At the command prompt, run the following command:

telnet <IP-address-or-hostname> <port>

For example:
Shell
```
telnet 192.168.1.1 8081
```
If the connection is unsuccessful, an error message is displayed.

If the command prompt window is blank with only the cursor visible, the connection is successful, and the service is available.

Connection error to source or destination database server.

If there is a connectivity issue with the source or destination database, check if the SingleStore Flow server can reach the remote host and port.

You can test the connection to the IP address and port using the following command:

telnet <IP-address-or-hostname> <port>

Alternatively, run the PowerShell command to verify the connection:

tnc <IP-address-or-hostname> <port>

Unable to start Windows service

Error: Unable to start the Windows service 'SingleStore Ingest'.

Resolution: If Java is not installed or the system path is not updated, the Ingest service fails to start. Install Java 21 or add the Java path to the system path. To verify, open CMD and run the following command.

java -version

If the response is ‘unable to recognize command’, check the Java path in the environment variables under 'Path' and update it to the correct path.

Application not able to launch

Error: The SingleStore Ingest service is installed and started, but the application does not launch in the browser.

Resolution: SingleStore Flow requires Java 21 to function. Please install the correct version of Java and restart the service.

If Java 11 is installed, the Ingest service starts, but the page displays an error message.

To verify the Java version, open CMD and run the following command.

Shell

java -version

Expected result: java version "any_build", where "any_build" could be, for example:

openjdk version "21.0.5" 2024-10-15 LTS
OpenJDK Runtime Environment Corretto-21.0.5.11.1 (build 21.0.5+11-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.5.11.1 (build 21.0.5+11-LTS, mixed mode, sharing)

If the Java version displayed is earlier than 21, uninstall Java and install Java 21.

Grants not available on the database

Error: Cannot open database 'demo' requested by the login.

Resolution: The user does not have the necessary grants to connect to the database. Apply the correct grants to the user and try again.

Login failed for user

Error: Login failed for user 'Demo'.

Resolution: The user does not exist, or there is a typo in the username. Alternatively, the password may be incorrect. Verify the credentials and try again.