Redis: Gossip is backbone for healthy cluster
If you have ever setup Redis cluster you would notice that there is no single node responsible to manage cluster or about health of the cluster.
CLUSTER INFO command used to check cluster health and information can be executed on any master and salve node and will provide the same information, which means every node maintains this information. This got me curious on how Redis clusters are managed and how node failures are handled.
Gossiping is the core
Redis Cluster uses a simple Gossip protocol in order to quickly spread information through any connected node. Heartbeat packets carry information on their own, but they also contain a special header for gossip data.
This section contains information about a few random nodes among the set of nodes known to the sender. The number of entries included in the gossip header is proportional to the size of the cluster (usually 1/10 of the nodes, but may be empty as well in some cases). The trick is: almost any heartbeat packet contains data about some (a few) nodes of the cluster from the point of view of the sender. This is how information propagates quickly at large scale without requiring an exponential number of messages exchanged.
This communication happens through Redis Cluster Bus. Every node is connected to every other node through a dedicated tcp connection, thus forming a fully connected network. This node-to-node communication channel is used exclusively for cluster operations (e.g. configuration update, failure detection, failover authorization) and its links are kept separate from normal client connections.
Nodes exchange data over the bus using a binary protocol
Cluster heartbeat: PING / PONG packets
Under normal conditions, nodes of the cluster continuously exchange PING and PONG packets over the bus (remember we’re talking about raw binary packets here, not to be confused with the PING client command).
This packet flow constitutes the heartbeat mechanism of the cluster, a means of information propagation fundamental to some key features such as, for example, node auto-discovery and failure detection.
In order to avoid exchanging too many packets on the network, usually a node will ping only a few (not all) randomly chosen nodes every second. each node in the cluster will always try to ping every other node that didn’t send a PING or received a PONG for longer than half this timeout value.
When a new node joins the cluster it will send MEET packet which is similar to PING and receiving node will register the node as new node for further communication.
Data replication
Redis cluster is a combination for master and slave nodes..
Data replication from master to slave is async operation and does not impact client operations. Clients only deals with master nodes.
When a master fails or is found to be unreachable by the majority of the cluster as determined by the nodes’ communication via the gossip port, the remaining masters hold a vote and elect one of the failing masters’ slaves to take its place. If the new master does not have a slave after the election, one of the redundant slaves can be reshuffled to replicate the new master using replicas migration. When the failing master eventually rejoins the cluster, it will join as a slave and begin to replicate another master.
Replica migration
In Redis Cluster it is possible to reconfigure a replica to replicate with a different master at any time just using the following command:
CLUSTER REPLICATE <master-node-id>
The reason why you may want to let your cluster replicas to move from one master to another under certain condition, is that usually the Redis Cluster is as resistant to failures as the number of replicas attached to a given master. This will be taken care by Redis cluster itself without manual intervention.
Consider a cluster where every master has a single replica can’t continue operations if the master and its replica fail at the same time, simply because there is no other instance to have a copy of the hash slots the master was serving. However while net-splits are likely to isolate a number of nodes at the same time, many other kind of failures, like hardware or software failures local to a single node, are a very notable class of failures that are unlikely to happen at the same time, so it is possible that in your cluster where every master has a replica, the replica is killed at 4am, and the master is killed at 6am. This still will result in a cluster that can no longer operate.
To improve reliability of the system we have the option to add additional replicas to every master, but this is expensive. Replica migration allows to add more replicas to just a few masters. So you have 10 masters with 1 replica each, for a total of 20 instances. However you add, for example, 3 instances more as replicas of some of your masters, so certain masters will have more than a single replica.
With replicas migration what happens is that if a master is left without replicas, a replica from a master that has multiple replicas will migrate to the orphaned master. So after your replica goes down at 4am as in the example we made above, another replica will take its place, and when the master will fail as well at 5am, there is still a replica that can be elected so that the cluster can continue to operate.
So what you should know about replicas migration in short?
- The cluster will try to migrate a replica from the master that has the greatest number of replicas in a given moment.
- To benefit from replica migration you have just to add a few more replicas to a single master in your cluster, it does not matter what master.
- There is a configuration parameter that controls the replica migration feature that is called cluster-migration-barrier: you can read more about it in the example redis.conf file provided with Redis Cluster.
Understanding these details are very important when you are setting up Redis cluster as this will help in deciding right number of nodes and how to have correct notifications with respect of Redis cluster health check.
Good references to refer for more details:
https://cristian.regolo.cc/2015/09/05/life-in-a-redis-cluster.html
