# AWS Environment Setup The following are the steps to prepare an environment for SingleStore Flow in AWS: 1. **Create an AWS Account**: To prepare the environment for Flow in AWS, you need an AWS account. If you already have an AWS account, proceed to the next step. If not, refer to [How do I create and activate a new AWS account?](https://repost.aws/knowledge-center/create-and-activate-aws-account) to create one. 2. **Create an IAM User**: SingleStore recommends creating a separate IAM user for managing all AWS services. Do not use the root user for any tasks. Refer to [Setting up your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started-account-iam.html#create-an-admin) to create an IAM admin user. 3. **Create and Assign Policies to the User**: Use the AWS Management Console to create a customer-managed policy and attach it to the IAM user based on their role. This policy enables the IAM user to sign in directly to the AWS Management Console with the assigned permissions. 4. **Sign in to AWS**: 1. **As an IAM User**: Sign in to the AWS Management Console using your Account ID or account alias, along with your username and password. Refer to [Sign in to the AWS Management Console](https://docs.aws.amazon.com/signin/latest/userguide/how-to-sign-in.html) for more information. 2. **AWS SSO**: Sign in using IAM Identity Center (AWS SSO). Refer to [How to create and manage users within AWS IAM Identity Center](https://aws.amazon.com/blogs/security/how-to-create-and-manage-users-within-aws-sso/) for more information. 5. **Create a VPC**: A Virtual Private Cloud (VPC) is a dedicated virtual network for your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. You can launch your Flow application and related AWS resources, such as Amazon EC2 instances, into this VPC. Refer to [Configure a virtual private cloud](https://docs.aws.amazon.com/vpc/latest/userguide/configure-your-vpc.html) for instructions on creating a VPC. 6. **Create a Private Subnet in Your VPC**: Since Ingest needs to be set up in your VPC, SingleStore recommends creating a new private subnet within the VPC for Flow. Refer to [Create a VPC](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#AddaSubnet) to create a subnet. 7. **Create a Security Group**: A security group acts as a virtual firewall for your instances that controls inbound and outbound traffic. Security groups apply at the instance level, not the subnet level. Each instance in a subnet can be assigned different security groups. SingleStore does not recommend using the default security group, which AWS assigns if you don’t specify a security group at launch. You can add rules to each security group to control inbound and outbound traffic. Refer to [Control traffic to your AWS resources using security groups](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html) for more information. 8. **Configure Security Group Rules**: You can add or remove rules to a security group to authorize or revoke inbound or outbound access. A rule applies either to inbound traffic (ingress) or outbound traffic (egress). You can grant access to a specific CIDR range or to another security group in your VPC or in a peer VPC (with a VPC peering connection). 9. **Create an IAM Role**: Flow uses an IAM role assigned to the EC2 instance where the application is hosted. The EC2 role must have the necessary policies attached. Refer to [Create a role to delegate permissions to an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) for instructions on creating an IAM role for Flow. Attach the [required policies](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest/aws-environment-setup/#aws-identity-and-access-management-iam-for-singlestore-flow-2.md) to the newly created IAM role. 10. **Assign the Role to Users or Groups**: The IAM role must be assigned to an AWS Directory Service user or group. The role must have a trust relationship with AWS Directory Service. Refer to [Assigning users or groups to an existing IAM role](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/assign_role.html) to assign users or groups to an IAM role. 11. **Create Access Key ID and Secret Access Key**: Flow uses an access key ID and secret access key to connect to AWS services from an on-premises server. SingleStore recommends generating a set of access keys for the Flow user account. The following are the steps to create access keys from the admin user account: 1. Sign in to the AWS Management Console and open the IAM console at . 2. In the navigation pane, select **Users**. 3. Select the **separateSingleStore Flow** user whose access keys you want to manage, then select the **Security credentials** tab. 4. In **Access keys**, select **Create access key**. Then, select **Download .csv** **file** to save the access key ID and secret access key to a CSV file. Store this file securely, as the secret access key cannot be accessed again after closing the dialog. 5. After downloading the CSV file, select **Close**. The access key is active by default, and you can use it. Refer to [AWS security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html#access-keys-and-secret-access-keys) for more information. **Note**: SingleStore recommends rotating access keys every 90 days. Refer to [Data Security and Encryption](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest/aws-environment-setup/#data-security-and-encryption-2.md) for more information. 12. **Create an Auto-Scaling Group**: When deploying Ingest in a high-availability (HA) environment, SingleStore recommends configuring your EC2 instances with an Auto Scaling Group. Follow the steps mentioned in [Create an Auto Scaling group using the Amazon EC2 launch wizard](https://docs.aws.amazon.com/autoscaling/ec2/userguide/create-asg-ec2-wizard.html) to launch an Auto Scaling group via the AWS Console. The recommended parameters for configuring the Auto Scaling group are: 1. Select the instance type recommended by SingleStore under the [Prerequisites](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest.md) section. 2. Set the minimum number of instances to 2 for a high-availability deployment. 3. Select the IAM role as SingleConnectEc2Role. 4. Configure the storage as recommended under the [Additional AWS Services](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest/aws-environment-setup/#additional-aws-services-2.md) section. 5. Select the security group created for Flow in the previous steps. 6. Select the key pair for the launch configuration. 7. Refer to [Create an Auto Scaling group using the Amazon EC2 launch wizard](https://docs.aws.amazon.com/autoscaling/ec2/userguide/create-asg-ec2-wizard.html) for remaining steps. ## Create an EC2 System Refer to [Get started with Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html) for more information. ## Optimize Usage of AWS Resources ## Tagging AWS Resources AWS allows customers to assign metadata to their AWS resources through tags. SingleStore highly recommends tagging all AWS resources created for and by SingleStore Flow to manage and organize resources, control access, track costs, and automate processes. For clarity and organization, SingleStore recommends to use tags with names specific to the instances being created. For example, for a Flow instance replicating a production database server for Billing and Finance, the tag name must reflect the database name it is dedicated to, such as `SingleStoreIngest_BFS_EC2_Prod`. Similarly, for a user acceptance testing (UAT) environment, the tag name can be `SingleStoreIngest_BFS_EC2_UAT`. This approach helps customers differentiate between various AWS resources in their environment. Use consistent tag names across all services. SingleStore recommends tagging AWS EC2 services with unique and identifiable tag names. Refer to [Tag your Amazon EC2 resources](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) for more information on tagging AWS EC2 resources. ## AWS Identity and Access Management (IAM) for SingleStore Flow AWS IAM roles delegate access to AWS resources. With IAM roles, you can establish trust relationships between your trusting account and other trusted AWS accounts. The trusting account owns the resource to be accessed, while the trusted account contains the users who need access. **SingleStore’s Recommendations:** 1. Create an IAM User (e.g., `SingleStore_Flow_User`). Do **NOT** use the root user account to set up the application. Refer to [Create an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) for creating an IAM user. 2. Create an IAM Role (e.g., `SingleStore_Flow_EC2Role`). Refer to [IAM role creation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html) for creating IAM roles. 3. Create an IAM Policy (e.g., `SingleStore_Flow_Policy`) and assign custom policies (provided below) to the EC2 role. Refer to [Creating IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html#access_policies_create-start) for creating a policy. 4. Instead of defining permissions for individual SingleStore Flow IAM users, it is more convenient to create groups based on job functions (e.g., administrators, developers, accounting, etc.). Define the relevant permissions for each group and assign IAM users to those groups. All users in an IAM group inherit the permissions assigned to the group. This way, you can make changes for everyone in the group in one place. 5. Only grant the minimal required permissions to the IAM role. The SingleStore Flow user requires basic permissions for S3, CloudWatch, SNS and DynamoDB. 6. SingleStore Flow needs access to the following AWS services: S3, EC2, SNS, and CloudWatch with the minimum privileges outlined below. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "1", "Action": [ "s3:DeleteObject", "s3:GetObject", "s3:ListBucket", "s3:PutObject" ], "Effect": "Allow", "Resource": "arn:aws:s3:::" }, { "Sid": "2", "Action": [ "ec2:AcceptVpcEndpointConnections", "ec2:AcceptVpcPeeringConnection", "ec2:AssociateIamInstanceProfile", "ec2:CreateTags", "ec2:DescribeTags", "ec2:RebootInstances" ], "Effect": "Allow", "Resource": "arn:aws:ec2:::instance/" }, { "Sid": "4", "Action": [ "sns:Publish" ], "Effect": "Allow", "Resource": "arn:aws:sns::::" }, { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "dynamodb:CreateTable", "dynamodb:PutItem", "dynamodb:Update*", "dynamodb:Get*", "dynamodb:Scan" ], "Resource": "arn:aws:dynamodb:::table/SingleConnectTable" }, ] } ``` ## Roles Defined for SingleStore Flow The following are the roles and permissions required to launch and manage the Flow suite of applications. | **Role** | **Type** | **Permissions/Policies** | **Purpose** | | -------------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | EC2Admin | AWS Custom Role for EC2 | List-DescribeInstanceStatusDirectory ServiceList,Write-DescribeDirectories,CreateComputerSystems ManagerList,Read,WriteListAssociations,ListInstanceAssociations,DescribeAssociation,DescribeDocument,GetDeployablePatchSnapshotForInstance,GetDocument,GetManifest,GetParameters,PutComplianceItems,PutInventory,UpdateAssociationStatus,UpdateInstanceAssociationStatus,UpdateInstanceInformation | Create and Manage EC2 instance | | DBAdmin | AWS Custom Role | cloudwatch:DeleteAlarmscloudwatch:Describe\*cloudwatch:DisableAlarmActionscloudwatch:EnableAlarmActionscloudwatch:Get\*cloudwatch:List\*cloudwatch:PutMetricAlarmdynamodb:CreateTabledynamodb:BatchGetItemdynamodb:BatchWriteItemdynamodb:ConditionCheckItemdynamodb:PutItemdynamodb:DescribeTabledynamodb:DeleteItemdynamodb:GetItemdynamodb:Scandynamodb:Querydynamodb:UpdateItemec2:DescribeAccountAttributesec2:DescribeAddressesec2:DescribeAvailabilityZonesec2:DescribeInternetGatewaysec2:DescribeSecurityGroupsec2:DescribeSubnetsec2:DescribeVpcsiam:ListRolesiam:GetRolekms:ListKeyslogs:DescribeLogGroupslogs:DescribeLogStreamslogs:FilterLogEventslogs:GetLogEventslogs:Create\*logs:PutLogEventssns:Getsns:List\*sns:SetTopicAttributes | Manage DB access and privileges | | Network Admin | Custom Role | autoscaling:Describe\*,directconnect:\*,ec2:AcceptVpcEndpointConnections,ec2:AllocateAddress,ec2:AssignIpv6Addresses,ec2:AssignPrivateIpAddresses,ec2:AssociateAddress,ec2:AssociateDhcpOptions,ec2:AssociateRouteTable,ec2:AssociateSubnetCidrBlock,ec2:AssociateVpcCidrBlock,ec2:AttachInternetGateway,ec2:AttachNetworkInterface,ec2:AttachVpnGateway,ec2:CreateCarrierGateway,ec2:CreateCustomerGateway,ec2:CreateDefaultSubnet,ec2:CreateDefaultVpc,ec2:CreateDhcpOptions,ec2:CreateEgressOnlyInternetGateway,ec2:CreateFlowLogs,ec2:CreateInternetGateway,ec2:CreateNatGateway,ec2:CreateNetworkAcl,ec2:CreateNetworkAclEntry,ec2:CreateNetworkInterface,ec2:CreateNetworkInterfacePermission,ec2:CreatePlacementGroup,ec2:CreateRoute,ec2:CreateRouteTable,ec2:CreateSecurityGroup,ec2:CreateSubnet,ec2:CreateTags,ec2:CreateVpc,ec2:CreateVpcEndpoint,ec2:CreateVpcEndpointConnectionNotification,ec2:CreateVpcEndpointServiceConfiguration,ec2:CreateVpnConnection,ec2:CreateVpnConnectionRoute,ec2:CreateVpnGateway,ec2:DeleteCarrierGateway,ec2:DeleteEgressOnlyInternetGateway,ec2:DeleteFlowLogs,ec2:DeleteNatGateway,ec2:DeleteNetworkInterface,ec2:DeleteNetworkInterfacePermission,ec2:DeletePlacementGroup,ec2:DeleteSubnet,ec2:DeleteTags,ec2:DeleteVpc,ec2:DeleteVpcEndpointConnectionNotifications,ec2:DeleteVpcEndpointServiceConfigurations,ec2:DeleteVpcEndpoints,ec2:DeleteVpnConnection,ec2:DeleteVpnConnectionRoute,ec2:DeleteVpnGateway,ec2:DescribeAccountAttributes,ec2:DescribeAddresses,ec2:DescribeAvailabilityZones,ec2:DescribeCarrierGateways,ec2:DescribeClassicLinkInstances,ec2:DescribeCustomerGateways,ec2:DescribeDhcpOptions,ec2:DescribeEgressOnlyInternetGateways,ec2:DescribeFlowLogs,ec2:DescribeInstances,ec2:DescribeInternetGateways,ec2:DescribeKeyPairs,ec2:DescribeMovingAddresses,ec2:DescribeNatGateways,ec2:DescribeNetworkAcls,ec2:DescribeNetworkInterfaceAttribute,ec2:DescribeNetworkInterfacePermissions,ec2:DescribeNetworkInterfaces,ec2:DescribePlacementGroups,ec2:DescribePrefixLists,ec2:DescribeRouteTables,ec2:DescribeSecurityGroupReferences,ec2:DescribeSecurityGroupRules,ec2:DescribeSecurityGroups,ec2:DescribeStaleSecurityGroups,ec2:DescribeSubnets,ec2:DescribeTags,ec2:DescribeVpcAttribute,ec2:DescribeVpcClassicLink,ec2:DescribeVpcClassicLinkDnsSupport,ec2:DescribeVpcEndpointConnectionNotifications,ec2:DescribeVpcEndpointConnections,ec2:DescribeVpcEndpointServiceConfigurations,ec2:DescribeVpcEndpointServicePermissions,ec2:DescribeVpcEndpointServices,ec2:DescribeVpcEndpoints,ec2:DescribeVpcPeeringConnections,ec2:DescribeVpcs,ec2:DescribeVpnConnections,ec2:DescribeVpnGateways,ec2:DescribePublicIpv4Pools,ec2:DescribeIpv6Pools,ec2:DetachInternetGateway,ec2:DetachNetworkInterface,ec2:DetachVpnGateway,ec2:DisableVgwRoutePropagation,ec2:DisableVpcClassicLinkDnsSupport,ec2:DisassociateAddress,ec2:DisassociateRouteTable,ec2:DisassociateSubnetCidrBlock,ec2:DisassociateVpcCidrBlock,ec2:EnableVgwRoutePropagation,ec2:EnableVpcClassicLinkDnsSupport,ec2:ModifyNetworkInterfaceAttribute,ec2:ModifySecurityGroupRules,ec2:ModifySubnetAttribute,ec2:ModifyVpcAttribute,ec2:ModifyVpcEndpoint,ec2:ModifyVpcEndpointConnectionNotification,ec2:ModifyVpcEndpointServiceConfiguration,ec2:ModifyVpcEndpointServicePermissions,ec2:ModifyVpcPeeringConnectionOptions,ec2:ModifyVpcTenancy,ec2:MoveAddressToVpc,ec2:RejectVpcEndpointConnections,ec2:ReleaseAddress,ec2:ReplaceNetworkAclAssociation,ec2:ReplaceNetworkAclEntry,ec2:ReplaceRoute,ec2:ReplaceRouteTableAssociation,ec2:ResetNetworkInterfaceAttribute,ec2:RestoreAddressToClassic,ec2:UnassignIpv6Addresses,ec2:UnassignPrivateIpAddresses,ec2:UpdateSecurityGroupRuleDescriptionsEgress,ec2:UpdateSecurityGroupRuleDescriptionsIngress,logs:DescribeLogGroups,logs:DescribeLogStreams,logs:GetLogEvents,route53:\*,route53domains:\*,sns:CreateTopic,sns:ListSubscriptionsByTopic,sns:ListTopics,ec2:AcceptVpcPeeringConnection,ec2:AttachClassicLinkVpc,ec2:AuthorizeSecurityGroupEgress,ec2:AuthorizeSecurityGroupIngress,ec2:CreateVpcPeeringConnection,ec2:DeleteCustomerGateway,ec2:DeleteDhcpOptions,ec2:DeleteInternetGateway,ec2:DeleteNetworkAcl,ec2:DeleteNetworkAclEntry,ec2:DeleteRoute,ec2:DeleteRouteTable,ec2:DeleteSecurityGroup,ec2:DeleteVolume,ec2:DeleteVpcPeeringConnection,ec2:DetachClassicLinkVpc,ec2:DisableVpcClassicLink,ec2:EnableVpcClassicLink,ec2:GetConsoleScreenshot,ec2:RejectVpcPeeringConnection,ec2:RevokeSecurityGroupEgress,ec2:RevokeSecurityGroupIngress,ec2:CreateLocalGatewayRoute,ec2:CreateLocalGatewayRouteTableVpcAssociation,ec2:DeleteLocalGatewayRoute,ec2:DeleteLocalGatewayRouteTableVpcAssociation,ec2:DescribeLocalGatewayRouteTableVirtualInterfaceGroupAssociations,ec2:DescribeLocalGatewayRouteTableVpcAssociations,ec2:DescribeLocalGatewayRouteTables,ec2:DescribeLocalGatewayVirtualInterfaceGroups,ec2:DescribeLocalGatewayVirtualInterfaces,ec2:DescribeLocalGateways,ec2:SearchLocalGatewayRoutes,s3:GetBucketLocation,s3:GetBucketWebsite,s3:ListBucket,iam:GetRole,iam:ListRoles,iam:PassRole,ec2:AcceptTransitGatewayVpcAttachment,ec2:AssociateTransitGatewayRouteTable,ec2:CreateTransitGateway,ec2:CreateTransitGatewayRoute,ec2:CreateTransitGatewayRouteTable,ec2:CreateTransitGatewayVpcAttachment,ec2:DeleteTransitGateway,ec2:DeleteTransitGatewayRoute,ec2:DeleteTransitGatewayRouteTable,ec2:DeleteTransitGatewayVpcAttachment,ec2:DescribeTransitGatewayAttachments,ec2:DescribeTransitGatewayRouteTables,ec2:DescribeTransitGatewayVpcAttachments,ec2:DescribeTransitGateways,ec2:DisableTransitGatewayRouteTablePropagation,ec2:DisassociateTransitGatewayRouteTable,ec2:EnableTransitGatewayRouteTablePropagation,ec2:ExportTransitGatewayRoutes,ec2:GetTransitGatewayAttachmentPropagations,ec2:GetTransitGatewayRouteTableAssociations,ec2:GetTransitGatewayRouteTablePropagations,ec2:ModifyTransitGateway,ec2:ModifyTransitGatewayVpcAttachment,ec2:RejectTransitGatewayVpcAttachment,ec2:ReplaceTransitGatewayRoute,ec2:SearchTransitGatewayRoutes | Manage Network access and firewall settings | | SingleStoreFlowAdmin | Custom Role | s3:ListBucket,s3:PutObject,s3:GetObject,s3:DeleteObject,s3:GetBucketLocation,s3:PutObjectAcl,secretsmanager:GetSecretValue,secretsmanager:DescribeSecret,secretsmanager:PutSecretValue,secretsmanager:UpdateSecret | Able to manageSingleStoreFlowconfigurations | | AmazonS3 | Resource Based Policy | s3:PutObject,s3:GetObject,s3:DeleteObject,s3:GetBucketLocation,s3:PutObjectAclResource:arn:aws:s3:::\,arn:aws:s3:::\/\* | To manage bucket level permissions, resource-based policy for S3 must be applied to restrict the bucket level access. The policy is attached to the bucket, but the policy controls access to both the bucket and the objects in it. | | AmazonEC2 | Resource Based Policy | ec2:AcceptVpcEndpointConnections,ec2:AcceptVpcPeeringConnection,ec2:AssociateIamInstanceProfile,ec2:CreateTags,ec2:DescribeTags,ec2:RebootInstancesResource: arn:aws:ec2:\ | To manage instance level permissions, resource-based policy for EC2 must be applied to restrict the access for the EC2 instance. | ## Recommended Network ACL Rules for EC2 The following table presents the recommended rules for your EC2. These rules block all traffic except for what is explicitly required. The EC2 security group must include the necessary inbound and outbound rules as per the following: | **Inbound** | | | | | | | ------------ | ------------------------------------------------------- | ------------ | --------------------------------------------------------- | ------------------- | ---------------------------------------------------------------------------------- | | **Rule #** | **Source IP** | **Protocol** | **Port** | **Allow /****Deny** | **Comments** | | 1 | Custom IP which requires access to theIngestapplication | TCP | 80 | ALLOW | Allows inbound HTTP traffic only from known/custom IPv4 addresses. | | 2 | Public IPv4 address range of your home network | TCP | 22 | ALLOW | Allows inbound SSH traffic from your home network over the Internet gateway. | | 3 | Public IPv4 address range of your home network | TCP | 3389 | ALLOW | Allows inbound RDP traffic from your home network over the Internet gateway. | | 4 | 0.0.0.0/0 | all | all | DENY | Denies all inbound IPv4 traffic not handled by a preceding rule (not modifiable). | | **Outbound** | | | | | | | **Rule #** | **Dest IP** | **Protocol** | **Port** | **Allow / Deny** | **Comments** | | 1 | Source database host IP address | TCP | Custom port(port specific to source database ports) | ALLOW | Allows connections to the source database. | | 2 | SingleStoreclusterhost IP address | TCP | 8080(port specific to theSingleStoredestination database) | ALLOW | Allows connections to theSingleStoredestination database. | | 3 | 0.0.0.0/0 | all | all | DENY | Denies all outbound IPv4 traffic not handled by a preceding rule (not modifiable). | To open ports on the Amazon Console, refer to [What is Amazon EC2?](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) and follow the steps to allow inbound traffic to your Amazon instance. To open ports on Windows Server, refer to Opening Ports in the Firewall and follow the steps to allow inbound traffic to your server. ## Outbound Connections Ingest connects to any source endpoints outside its VPC using NAT, VPN, or API gateways. A NAT gateway is a Network Address Translation (NAT) service. You can use a NAT gateway to allow instances in a private subnet to connect to services outside your VPC, while preventing external services from initiating a connection with those instances. Refer to [NAT gateways](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) for more information. To connect the VPC to a remote network and enable source/destination endpoint connections, use AWS VPN. Refer to [NAT gateways](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) for more information. ## Additional AWS Services SingleStore Flow uses several AWS resources to fulfill user requirements. The costs of these services are separate from Flow charges and are billed by AWS to your account. The following list provides an overview of other billable services within Flow. Use the [AWS Pricing Calculator](https://calculator.aws/#/) to estimate the cost of additional AWS resources. A sample estimate for a high availability setup with a source data volume of 100 GB is provided for reference. Note that **Note**: Not all services are mandatory, and the size and number of services varies for each customer environment. The sample is for reference purposes only. All AWS services have service limits. Check for sufficient resources before launching services, and if necessary, refer to [Request a quota increase](https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html) to request an increase in quota. Refer to [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) to check the service limits for each service. | **Service** | **Mandatory** | **Billing Type** | **Service Limits** | | -------------------------------------- | ------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------- | | AWS EC2 | Y | Pay-as-you-go | [Check EC2 quota here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html) | | Additional EBS storage attached to EC2 | Y | Based on size | | | AWS S3 | N | Pay-as-you-go | [Check Amazon S3 quota here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/BucketRestrictions.html) | | AWS CloudWatch Logs and metrics | N | Pay-as-you-go | [Check EC2 quota here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html) | | AWS SNS | N | Pay-as-you-go | [Check AWS SNS quota here](https://docs.aws.amazon.com/general/latest/gr/sns.html) | | AWS Dynamo DB(5 WCUs /5 RCUs) | N | Pay-as-you-go | [Check Dynamo DB quota here](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ServiceQuotas.html) | | AWS Lambda | N | Pay-as-you-go | [Check AWS Lambda quota here](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html) | | AWS KMS | N | Pay-as-you-go | [Check Amazon KMS quota here](https://docs.aws.amazon.com/kms/latest/developerguide/resource-limits.html) | SingleStore recommends to use the following mentioned instance types for EC2 with EBS volumes attached. | **EC2 Instance Type** | **SingleStoreFlow Enterprise Edition** | **Recommended EBS volumes** | **EBS Volume Type** | | --------------------- | -------------------------------------- | --------------------------- | --------------------------------- | | t3.large | Volume < 100 GB | 100 GB | General Purpose SSD (gp2) Volumes | | t3.xlarge | Volume >100 and < 300 GB | 500 GB | General Purpose SSD (gp2) Volumes | | t3.2xlarge | Volume > 300 GB and < 1 TB | 500 GB | General Purpose SSD (gp2) Volumes | ## IMDS Settings and Recommendations Flow uses the latest version of the AWS SDK in each release. It uses IMDSv2 for all API calls to AWS services. SingleStore recommends disabling IMDS after deployment if required. Refer to [Use the Instance Metadata Service to access instance metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html) for more information on how to disable IMDS. To modify this using the AWS CLI, run the following command. ```shell aws ec2 modify-instance-metadata-options \ --instance-id \ --http-endpoint disabled ``` ## Data Security and Encryption SingleStore ensures data security through various mechanisms, including encryption. With AWS KMS, Ingest uses a customer-specified KMS key to encrypt customer data on AWS services such as EC2, Secrets Manager, and DynamoDB. You can configure the customer KMS ID in SingleStore Flow, which is used to encrypt data stored on these AWS services. Ingest also supports server-side encryption using AES-256. Amazon S3 server-side encryption employs 256-bit Advanced Encryption Standard (AES-256), one of the strongest block ciphers available, to encrypt your data. Flow supports this encryption by default. Ingest does not store any data outside the customer's designated environment. It stores data in the following AWS services, depending on customer requirements: * **Amazon EC2**: Used only for temporary staging and pipeline configuration. Refer to [Data protection in Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/data-protection.html) for enabling encryption. * **EBS Storage**: Used only for temporary staging and pipeline configuration. Refer to [Amazon EBS data security](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/data-protection.html#ebs-data-security) for enabling encryption. * **Amazon Aurora DB**: Used for exporting pipeline metadata. Refer to [Encrypting Amazon Aurora resources](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Overview.Encryption.html) for enabling encryption. * **Amazon DynamoDB**: Used when configured for High Availability. Refer to [DynamoDB encryption at rest](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EncryptionAtRest.html) for enabling encryption. * **AWS Secrets Manager**: Used for storing all pipeline credentials. Refer to [AWS Key Management Service](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) for encryption details. For non-AWS destination endpoints: * **SingleStore**: [Data encryption](https://docs.singlestore.com/db/v8.9/security/encryption/) is handled within SingleStore. ## Manage Access Keys Flow uses an access key and secret key to authenticate to AWS services like S3, CloudWatch, DynamoDB, and others. It requires the AWS access key ID and secret key to access S3 and other services from on-premises. AWS IAM roles are used when installing on an EC2 instance. For security reasons, SingleStore recommends rotating access keys and KMS keys periodically, for example, every 90 days. After generating new keys, update them in Ingest’s configuration. Follow these steps: * Open the Ingest instance that needs the new key in a web browser. * Go to the **Schedule** tab and stop the replication schedule for Ingest by disabling **Schedule**. * Navigate to **Connections > Destination File System**. * Enter the new **Access Key** and **Secret Access Key** and select **Apply**. * Once the keys are saved, resume the replication by enabling **Schedule**. Refer to [Rotate AWS KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html) for more information on key rotation. The IAM role for Flow must have the recommended policies attached. Refer to [AWS Identity and Access Management (IAM) for SingleStore Flow](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest/aws-environment-setup/#aws-identity-and-access-management-iam-for-singlestore-flow-3.md) for the list of policies and permissions. ## Key Rotation SingleStore recommends rotating all keys configured in Ingest every 90 days for security reasons. This includes credentials for all source and destination endpoints. The following are some references for AWS services for more details: * [Rotate AWS KMS keys](https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html) * [Encryption key rotation](https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-db-encryption.html#working-with-key-rotation) Follow the recommendations for all non-AWS sources and destinations: | **External Applications** | **Reference for Key Rotation** | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Oracle | [Oracle password rotation](https://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_4003.htm) | | Microsoft SQL Server | [SQL Server password rotation](https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/rotate-always-encrypted-keys-using-ssms?view=sql-server-ver16) | | MySQL | [MySQL password rotation](https://dev.mysql.com/doc/refman/5.7/en/innodb-data-encryption.html#:~:text=To%20rotate%20the%20master%20encryption,could%20arise%20from%20concurrent%20execution.) | | PostgreSQL | [PostgreSQL password rotation](https://www.postgresql.org/docs/8.0/sql-alteruser.html) | ## Configure Data Encryption SingleStore follows AWS recommendations for encrypting data both at rest and in transit. This is achieved by creating keys and certificates used for encryption. AWS Secrets Manager uses encryption through AWS KMS. Refer to [Secret encryption and decryption in AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/security-encryption.html) for more information. ## Encryption In-Transit Flow uses SSL to establish secure connections (e.g., with AWS services, databases, etc.) for data flow that ensures secure communication in transit. SSL involves managing security certificates. It is important to keep these certificates active at all times to ensure uninterrupted service. AWS Certificate Manager (ACM) handles the complexity of creating and managing public SSL/TLS certificates. Customers can set up notifications to be alerted before the certificate expiration date and can renew certificates in advance to ensure uninterrupted service. Refer to [Managed certificate renewal in AWS Certificate Manager](https://docs.aws.amazon.com/acm/latest/userguide/managed-renewal.html) for more information on managing ACM. ## Testing the Connections **Verify whether the connectivity to remote services is available.** To test the remote connections, you need the Telnet utility, which must be enabled from the Control Panel under **Turn On Windows Features**. Follow these steps: 1. Open the **Start** menu and select **Run**. 2. Type `CMD` and select **OK**. 3. At the command prompt, run the following command: `telnet ` For example: ```shell telnet 192.168.1.1 8081 ``` If the connection is unsuccessful, an error message is displayed. If the command prompt window is blank with only the cursor visible, the connection is successful, and the service is available. **Connection error to source or destination database server.** If there is a connectivity issue with the source or destination database, check if the SingleStore Flow server can reach the remote host and port. You can test the connection to the IP address and port using the following command: `telnet ` Alternatively, run the PowerShell command to verify the connection: `tnc ` **Unable to start Windows service** ***Error***: Unable to start the Windows service 'SingleStore Ingest'. ***Resolution***: If Java is not installed or the system path is not updated, the Ingest service fails to start. Install Java 21 or add the Java path to the system path. To verify, open `CMD` and run the following command. ``` java -version ``` If the response is ‘`unable to recognize command`’, check the Java path in the environment variables under '`Path`' and update it to the correct path. **Application not able to launch** ***Error***: The SingleStore Ingest service is installed and started, but the application does not launch in the browser. ***Resolution***: SingleStore Flow requires Java 21 to function. Please install the correct version of Java and restart the service. If Java 11 is installed, the Ingest service starts, but the page displays an error message. To verify the Java version, open `CMD` and run the following command. ```shell java -version ``` Expected result: `java version "any_build"`, where "`any_build`" could be, for example: * `openjdk version "21.0.5" 2024-10-15 LTS` * `OpenJDK Runtime Environment Corretto-21.0.5.11.1 (build 21.0.5+11-LTS)` * `OpenJDK 64-Bit Server VM Corretto-21.0.5.11.1 (build 21.0.5+11-LTS, mixed mode, sharing)` If the Java version displayed is earlier than 21, uninstall Java and install Java 21. **Grants not available on the database** ***Error***: Cannot open database '`demo`' requested by the login. ***Resolution***: The user does not have the necessary grants to connect to the database. Apply the correct grants to the user and try again. **Login failed for user** ***Error***: Login failed for user '`Demo`'. ***Resolution***: The user does not exist, or there is a typo in the username. Alternatively, the password may be incorrect. Verify the credentials and try again. *** Modified at: January 27, 2026 Source: [/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest/aws-environment-setup/](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest/aws-environment-setup/) (An index of the documentation is available at /llms.txt)