Scale Master Nodes in AWS Clusters

Problem

How to scale/replace master nodes in a Multi-Master AWS PMK cluster due to one of the following reasons: maintenance, OS crash, or hardware failure or AMI update?

Environment

  • Platform9 Managed Kubernetes - All Versions

  • AWS Clusters

Procedure

  • Ensure the etcd quorum of the cluster by checking the etcd cluster's health:

$ sudo docker cp etcd:/usr/local/bin/etcdctl /opt/pf9/pf9-kube/bin
$ export PATH=$PATH:/opt/pf9/pf9-kube/bin
$ etcdctl member list -w=table

$ etcdctl endpoint health --cluster -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"

$ etcdctl endpoint status --cluster -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"
# /opt/pf9/pf9-kube/bin/etcdctl --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key member list -w=table

# /opt/pf9/pf9-kube/bin/etcdctl --cacert=/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --cert=/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key=/etc/pf9/kube.d/certs/etcdctl/etcd/request.key endpoint health --cluster -w=table

  • Currently, Platform9 UI does not support scaling up/down the master nodes for an AWS cluster. But this can be achieved using Qbert APIs.

Info

To get X-Auth-Token use the following steps.

# curl --request POST \
--url https://<FQDN>/qbert/v4/<project_uuid>/clusters/<cluster_uuid>/detach \
--header "X-Auth-Token: {X-Auth-Token}" \
--header "Content-Type: application/json" \
--data '[
{
"uuid": "<master_node_UUID>",
"isMaster": "1"
}
]'

  • Once the node is detached from the cluster, delete the same master node from the AWS EC2 console so that AWS auto-scales a new node and attaches it to the cluster.

  • The progress can be seen by clicking on Infrastructure --> Clusters --> Node Health --> Select the newly added node --> Click on "View All Steps".

  • The final step is to verify the status of nodes using the command kubectl get nodes -w


© 2025 Platform9. All Rights Reserved

On This Page
Scale Master Nodes in AWS ClustersProblemProcedure