Recently one of our customers DRBD cluster has detected split-brain:
node2 kernel: block drbd1: Split-Brain detected, dropping connection! node2 kernel: block drbd1: helper command: /sbin/drbdadm split-brain minor-1 node2 kernel: block drbd1: helper command: /sbin/drbdadm split-brain minor-1 exit code 0 (0x0)
This is primary/primary cluster. And to fix this issue we had to perform manual procedures since we don’t use automatic recovery feature.
According to the DRDB official documentation the solution is next.
drbdadm secondary drbdadm connect --discard-my-data
The problem is drbdadm doesn’t have ‘–discard-my-data’ option and exits with error:
[root@node2 ~]# drbdadm connect --discard-my-data wwwdata drbdadm: unrecognized option '--discard-my-data' try 'drbdadm help' [root@node2 ~]#
The issue can be solved in two ways.
1. To perform command this way to pass this option directly to drbdsetup:
drbdadm -- --discard-my-data connect wwwdata
2. To use option –verbose to get low-level drbdsetup output and run it with ‘–discard-my-data’ :
drbdsetup 1 net 10.209.1.1:7789 10.209.1.2:7789 C --set-defaults --create-device --allow-two-primaries  --discard-my-data
After DRBD will sync you can change role of split-brain victim to primary.
drbdadm primary wwwdata
Problem solved.
[root@node2 ~]# cat /proc/drbd version: 8.3.7 (api:88/proto:86-91) GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@node2.example.com, 2010-01-28 08:37:51 1:cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r---- ns:125609294 nr:194649913 dw:320259208 dr:36731102 al:34129 bm:4042 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0 [root@node2 ~]#
Didn’t find the answer to your question? Ask it our administrators to reply we will publish on website.