Setup HashiCorp Vault HA Cluster with Integrated Storage (Raft) and AWS KMS Auto Unseal on CentOS 7
Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.
A modern system requires access to a multitude of secrets: database credentials, API keys for external services, credentials for service-oriented architecture communication, etc. Understanding who is accessing what secrets is already very difficult and platform-specific. Adding on key rolling, secure storage, and detailed audit logs is almost impossible without a custom solution. This is where Vault steps in.
The key features of Vault are:
Secure Secret Storage: Arbitrary key/value secrets can be stored in Vault. Vault encrypts these secrets prior to writing them to persistent storage, so gaining access to the raw storage isn't enough to access your secrets. Vault can write to disk, Consul, and more.
Dynamic Secrets: Vault can generate secrets on-demand for some systems, such as AWS or SQL databases. For example, when an application needs to access an S3 bucket, it asks Vault for credentials, and Vault will generate an AWS keypair with valid permissions on demand. After creating these dynamic secrets, Vault will also automatically revoke them after the lease is up.
Data Encryption: Vault can encrypt and decrypt data without storing it. This allows security teams to define encryption parameters and developers to store encrypted data in a location such as SQL without having to design their own encryption methods.
Leasing and Renewal: All secrets in Vault have a lease associated with them. At the end of the lease, Vault will automatically revoke that secret. Clients are able to renew leases via built-in renew APIs.
Revocation: Vault has built-in support for secret revocation. Vault can revoke not only single secrets, but a tree of secrets, for example all secrets read by a specific user, or all secrets of a particular type. Revocation assists in key rolling as well as locking down systems in the case of an intrusion.
#
1. Architecture Diagram#
2. System Requirements#
2.1. HashiCorp Vault NodesComponent | Description |
---|---|
Number of VMs | 3 |
CPU | 2 Cores |
Memory | 4 GB |
Disk Size | 20 GB SSD |
Operating System | CentOS 7 x64 |
File System | XFS |
Privileges | ROOT access prefered |
#
2.2. IP AllocationComponent | Description |
---|---|
VM IPs | 10.101.15.101 - 10.101.15.103 |
Virtual IP (Floating IP) | 10.101.15.100 (For on-premises deployments ONLY) |
#
2.3. DNS EntriesIP | Hostname | FQDN |
---|---|---|
10.101.15.100 | vault | vault.cluster.local |
10.101.15.101 | vault-1 | vault-1.cluster.local |
10.101.15.102 | vault-2 | vault-2.cluster.local |
10.101.15.103 | vault-3 | vault-3.cluster.local |
#
2.4. AWS KMS Informationimportant
The provided AWS credentials must have permissions to perform kms:DescribeKey, kms:Encrypt, and kms:Decrypt actions on the given KMS ARN.
Component | Description |
---|---|
AWS_ACCESS_KEY_ID | AKIAUF222X2TAMFCVONW |
AWS_SECRET_ACCESS_KEY | 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 |
AWS_REGION | eu-west-2 |
KMS_KEY_ID | 3t6265bd-31c0-456d-a4cb-a4d24dc28c1d |
KMS ARN | arn:aws:kms:eu-west-2:285216434864:key/3t6265bd-31c0-456d-a4cb-a4d24dc28c1d |
#
3. Install and Configure a HashiCorp Vault HA Cluster#
3.1. Install prerequisites on ALL nodes3.1.1. Set server hostname.
3.1.2. Install prerequisites.
3.1.3. Synchronize server time with default NTP servers. If you have your own NTP servers, please make sure to update the /etc/chrony.conf
3.1.4. Start and enable chronyd service.
3.1.5. Display time synchronization status.
3.1.6. Disable File Access Time Logging and enable Combat Fragmentation to enhance XFS file system performance. Add noatime,nodiratime,allocsize=64m
to all XFS volumes under /etc/fstab
.
3.1.7. Tweaking the system for high concurrency and security.
3.1.8. Reload all sysctl variables without rebooting the server.
3.1.9. Create Local DNS records.
3.1.10. The servers need to be restarted before continuing further.
#
3.2. HashiCorp Vault common configurations on ALL nodes3.2.1. Configure YUM repository for Vault.
3.2.2. Install Vault.
3.2.3. Enable command auto-completion.
3.2.4. Copy the correct SSL certificate, SSL key and the CA certificate files under /opt/vault/tls
important
- To generate a custom CA and CA signed SSL certificates, please follow this guide.
- Other than the *.cluster.local domain, you must have 'localhost' as a Subject Alternative Name (SAN) since keepalived uses it for health checks.
3.2.5. Open necessary firewall ports.
3.2.6. Configure AWS KMS Auto Unseal
important
If you are running Vault on an AWS EC2 instance, make sure to create an AWS role with the following policy and attach it to all Vault running EC2 nodes.
important
If you are running Vault on an ON-PREMISE instance, make sure to create an IAM user with the following policy and configure a systemd drop-in with necessary environment variables.
#
3.3. Configurations on VAULT-1 node.3.3.1. Add the configurations under /etc/vault.d/vault.hcl
3.3.2. Set correct permissions.
3.3.3. Start and enable vault.service.
3.3.4. If there are any errors, please check systemd logs.
#
3.4. Configurations on VAULT-2 node.3.4.1. Add the configurations under /etc/vault.d/vault.hcl
3.4.2. Set correct permissions.
3.4.3. Start and enable vault.service.
3.4.4. If there are any errors, please check systemd logs.
#
3.5. Configurations on VAULT-3 node.3.5.1. Add the configurations under /etc/vault.d/vault.hcl
3.5.2. Set correct permissions.
3.5.3. Start and enable vault.service.
3.5.4. If there are any errors, please check systemd logs.
#
3.6. Initialize the Vault cluster on VAULT-1 node with KMS auto-unseal.3.6.1. Before initializing the cluster, make sure to check cluster status.
You should get an output like below.
3.6.2. Initialize the vault cluster.
If the command succeeded, you would get an output like below. Please make sure to record it.
3.6.3. Verify if the cluster is initialized and unsealed.
You should get an output like below.
#
3.7. Join the VAULT-2 and VAULT-3 nodes to the cluster.3.7.1. When you initialize the VAULT-1 node, the other nodes should automatically initialize and unseal using raft replication. Please verify if the other cluster nodes are initialized and unsealed.
You should get an output like below.
3.7.2. If the VAULT-2 and VAULT-3 cluster nodes are still uninitialized and sealed, please restart the vault service.
3.7.3. Run the following command on any node and verify if the cluster is active.
You should get an output like below.
3.7.4. If there are any errors, please check systemd logs.
#
4. Configure Cluster High Availability using Keepalived and Floating IPsimportant
- If you are using a cloud load balancer, you can SKIP this step. You have to use load balancer health checks instead.
- HashiCorp Vault health check URL https://localhost:8200/v1/sys/health
#
4.1. Install and configure Keepalived on ALL nodes.4.1.1. Install Keepalived package.
4.1.2. Allow VRRP traffic on the firewall.
4.1.3. Create the health check script under /usr/libexec/keepalived
4.1.4. Provide necessary permissions to the health check script.
4.1.5. Configure keepalived.
4.1.6. Start and enable keepalived.service.
4.1.7. Check if the floating IP is assigned to a node.
#
5. Maintenance#
5.1. Backup and restore a vault cluster using raft snapshots.5.1.1. Create a snapshot of raft storage.
5.1.2. Restore a snapshot of raft storage.
#
5.2. Migrate a vault cluster using raft snapshots.important
- If you want to clone/migrate a vault cluster to a new cluster with a new AWS KMS key, please follow these steps.
- Let's assume the NEW KMS_KEY_ID and KMS ARN values are "5m6268td-43c0-45d4-a4nc-a6d34mc38t2c" and "arn:aws:kms:eu-west-2:285216434864:key/5m6268td-43c0-45d4-a4nc-a6d34mc38t2c" respectively.
5.2.1. Please run these commands on any node of OLD vault cluster.
5.2.1.1. Create a snapshot of raft storage of the old vault cluster.
5.2.2. Please run these commands on NEW vault cluster.
5.2.2.1. To restore the cluster, you must have access to both OLD and NEW AWS KMS keys. Please create/update the IAM policy and attach it to the corresponding IAM User/Role.
5.2.2.2. Please make sure to use the NEW KMS key in the /etc/vault.d/vault.hcl.
5.2.2.3. Initialize the NEW vault cluster.
If the command succeeded, you would get an output like below. Please make sure to record it.
5.2.2.4. Verify if the cluster is initialized and unsealed.
5.2.2.5. Now you are ready to import vault backup.
5.2.2.6. Verify if the cluster is initialized and unsealed.
5.2.2.7. Remove the OLD KMS ARN entry from IAM policy.