Replicas of data are kept on data nodes. If a failure of a node occurs, the cluster will recover. The following steps are for gracefully migrating data off of a data node and removing it from the cluster.
1) Check that the Cluster is in a healthy state by running sudo docker exec -i cassandra nodetool ring. Make sure all listed Statuses are “Up” and All State are “Normal”.
2) On the data node to be removed exec the following command to notify it to start replicating its data to other nodes in the cluster: sudo docker exec -i cassandra nodetool decommission.
3) It will take some time for the replication to occur. To check status run sudo docker exec -i cassandra nodetool netstats. During the process, MODE will be “Leaving”. Once complete, the MODE will be “Decommissioned”.
4) Once the MODE is “Decommissioned” the node can be safely powered down.
5) Notify WitFoo support via Email or Slack that you have decommissioned a data node so the licensing portal can be updated.