Percona Cluster with lost quorum

Minimum number of servers for Percona/Galera cluster is 3 if quorum should be handled properly.
It is ok in case all is working well, but what if you need to do some maintenance on one of the nodes? 
 
This is what happened to us:
 
We had 3 nodes. One of the nodes had problems, so tried to restart it.
Restart failed and cluster (for unknown reason) lost quorum - last 2 nodes in the cluster were maybe disconnected for short moment.
 
So we ended up with 2 running servers, each of them was in the status 'Initialized', but not Primary, none server in the status 'Synced'.
 
Solution:
 
1. stop all mysql servers except one - you have to be sure there is running just one server, which should become Primary
2. open mysql console and execute following command on the last running node: 
SET GLOBAL wsrep_provider_options='pc.bootstrap=true';
 
Now your cluster has one running node with quorum again and you can start adding nodes back to the cluster.
 
 
There are also other possible scenarios, check for more details following blog post:
×