Percona Cluster with lost quorum
Minimum number of servers for Percona/Galera cluster is 3 if quorum should be handled properly. It is ok in case all is working well, but what if you need to do some maintenance on one of the nodes?
This is what happened to us:
We had 3 nodes. One of the nodes had problems, so tried to restart it.
Restart failed and cluster (for unknown reason) lost quorum - last 2 nodes in the cluster were maybe disconnected for short moment.
So we ended up with 2 running servers, each of them was...