Consul-AWS

consul-aws syncs services between a Consul cluster's datacenter and an AWS CloudMap namespace. This tutorial describes configuring, and using consul-aws to sync services one, or two-way between Consul and AWS CloudMap.

Prequisites

For this tutorial, you will need:

• curl: v7.64.1
• jq jq-1.6

Installation

Begin by downloading the latest consul-aws binary from the HashiCorp releases directory. If the provided $OS and $ARCH variables do not resolve as expected in your local environment, edit and set these variables as preferred.

$ ARCH=amd64 && \
OS=$(uname | tr '[:upper:'] '[:lower:]') && \
DL_URL=$(curl -s https://api.releases.hashicorp.com/v1/releases/consul-aws/latest | jq -r --arg arch $ARCH --arg os $OS '.builds[] | select(.arch==$arch and .os==$os) | .url') && \
FILENAME=$(echo $DL_URL | awk -F'/' '{print $NF}') && \
curl -L -o $FILENAME $DL_URL 2&>/dev/null && \
tar -xvf $FILENAME

Update your $PATH variable to the current working directory to use consul-aws.

$ export $PATH=$PATH:$(pwd)

Authentication

consul-aws requires access to Consul and AWS for uni- or bi-directional syncing.

consul-aws accepts the CLI flag -token, or the environment variable CONSUL_HTTP_TOKEN to connect to a Consul cluster.

For AWS, consul-aws uses AWS's default credential provider chain to discover AWS credentials. The default provider chain searches for AWS credentials in this order:

1. Environment variables.
2. A shared credentials file.
3. An AWS IAM role, when an application is hosted on an Amazon EC2 instance.

Configuration

Two subcommands are available with consul-aws:

• version: displays version number
• sync-catalog: syncs services in the consul catalog

The version subcommand contains no flags. The sync-catalog subcommand provides the following flags:

• A set of parameters to connect to your Consul datacenter, including-http-addr, -token, -ca-file, -client-cert, among others.
• -aws-namespace-id: (REQUIRED) The AWS namespace to sync with Consul services.
• -aws-service-prefix: A prefix to prepend to services written to AWS from Consul. When not set, the services use no prefix.
• -consul-service-prefix: A prefix to prepend to all services written to Consul from AWS CloudMap. If not set then services will have no prefix.
• -to-aws: When true, Consul services sync to AWS CloudMap (defaults to false).
• -to-consul: When true, AWS services sync to Consul (defaults to false).
• -aws-pull-interval: The time between syncing from AWS CloudMap. Accepts a sequence of decimal numbers with optional fractional values, each with a unit suffix, such as "300ms", "10s", "1.5m" (defaults to 30s).
• -aws-dns-ttl: DNS TTL for services created in AWS CloudMap, in seconds (defaults to 60).

Sync Consul services one-way to AWS CloudMap

Using the -to-aws command line flag begins syncing services to an AWS CloudMap namespace. When passing -aws-service-prefix, imported services from Consul share a common identifier. One strategy when syncing from multiple Consul clusters, such as an HCP Consul Dedicated cluster, is using the Consul cluster's name in the prefix.

$ consul-aws -aws-namespace-id ns-hjrgt3bapp7phzff -to-aws -consul-service-prefix my_hcp_consul_cluster_

After the command starts, consul-aws begins importing services from Consul to AWS CloudMap. A service in Consul with the nameweb identifies as my_hcp_consul_cluster_web in AWS CloudMap.

Consul sets the following properties for services synced to AWS CloudMap:

• Description: "Imported from Consul"
• Record types: A and SRV
• DNS routing policy: Multivalue answer routing

Sync AWS CloudMap namespaces one-way to Consul

To sync services from an AWS CloudMap namespace to Consul, the -to-consul flag enables syncing from AWS to Consul. If you are using HCP Consul Dedicated or hosting services across multiple cloud providers, one strategy for prefixing is adding aws somewhere in the prefix. This lets you quickly differentiate services imported with this tool, and other hosted Consul services. The prefix is fully customizable to your choosing.

$ consul-aws -aws-namespace-id ns-hjrgt3bapp7phzff -to-consul -aws-service-prefix my_aws_account_

After the command starts, consul-aws begins importing services from the CloudMap namespace to Consul. A service in AWS CloudMap with the name redis identifies as my_cloudmap_namespace_redis in Consul.

Consul sets the following properties for services synced from AWS CloudMap:

• Tag: aws
• Meta-Data: includes aws as the source set, the aws-id, the aws-namespace and custom attributes the instance uses in AWS CloudMap
• Node: the node name is consul-aws

Bidirectional syncing

To enable bidirectional sync, between Consul and AWS CloudMap, provide -to-consul and -to-aws, including the optional -aws-service-prefix and -consul-service-prefix flags:

$ consul-aws -aws-namespace-id ns-hjrgt3bapp7phzff -to-consul -aws-service-prefix my_cloudmap_namespace_ -to-aws -consul-service-prefix hcp_consul_

After the command starts, consul-aws syncs services in both directions between Consul from AWS CloudMap.

Next steps

You have completed this tutorial. Uni- or bidirectional sync is active between Consul and AWS CloudMap, with service discovery available across Consul and AWS CloudMap.

Try setting up consul-aws using a Consul ACL token for authentication. The Secure Consul with ACLs tutorial provides instructions on configuring ACLs for your cluster.