Decommission a Data Node

[Charles Herring]

Replicas of data are kept on data nodes. If a failure of a node occurs, the cluster will recover. The following steps are for gracefully migrating data off of a data node and removing it from the cluster.

1. Check that the Cluster is in a healthy state by running sudo docker exec -i cassandra nodetool status.
   1. Make sure all listed Statuses are “Up” and All State are “Normal”.  This will display as UN.
2. Execute sudo bash.
3. Execute WF_CASSANDRA_PW=$(cat /witfooprecinct/cassandra.password | jq -r '.password')
4. On the data node to be removed exec the following command to notify it to start replicating its data to other nodes in the cluster:docker exec cassandra nodetool -u cassandra -pw $WF_CASSANDRA_PW decommission -f
5. It will take some time for the replication to occur. To check status run sudo docker exec -i cassandra nodetool -u cassandra -pw $WF_CASSANDRA_PW netstats.
   1. During the process, MODE will be “Leaving”. Once complete, the MODE will be “Decommissioned”.
6. Once the MODE is “Decommissioned” the node can be safely powered down.
7. IMPORTANT – Notify WitFoo support via Email or Slack that you have decommissioned a data node so the licensing portal can be updated.
   1. Provide the IP address of the decommissioned appliance.

[Author]

Posts