glusterd を再起動したら失敗するようになった問題→とりあえず復旧させてみた

OpenSSLの脆弱性がでたので、Webサーバ、メールサーバ、ほかSSH開けてるとこなんぞにパッチ当てる→サービス再起動なんていうのをしばらくやっていくうちに、だんだん面倒になってきて、OSごと再起動したものがありました。
そしたら、なぜかglusterdがあがらなくなるという問題に遭遇したので、メモしておきます。ちなみに、OpenSSLのパッチとは無関係です。

●ログ

[ec2-user@ip-10-152-253-193 ~]$ sudo less /var/log/glusterfs/etc-glusterfs-glusterd.vol.log

[2014-04-11 01:04:49.486482] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.4.2 (/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)

[2014-04-11 01:04:49.494405] I [glusterd.c:961:init] 0-management: Using /var/lib/glusterd as working directory

[2014-04-11 01:04:49.497892] I [socket.c:3480:socket_init] 0-socket.management: SSL support is NOT enabled

[2014-04-11 01:04:49.497918] I [socket.c:3495:socket_init] 0-socket.management: using system polling thread

[2014-04-11 01:04:49.498045] E [rpc-transport.c:253:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.4.2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory

[2014-04-11 01:04:49.498071] W [rpc-transport.c:257:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine

[2014-04-11 01:04:49.498084] W [rpcsvc.c:1389:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed

[2014-04-11 01:04:51.021829] I [glusterd-store.c:1339:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2

[2014-04-11 01:04:51.029440] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0

[2014-04-11 01:04:51.029475] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1

[2014-04-11 01:04:51.594617] I [glusterd-handler.c:2818:glusterd_friend_add] 0-management: connect returned 0

[2014-04-11 01:04:51.594755] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600

[2014-04-11 01:04:51.594834] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled

[2014-04-11 01:04:51.594851] I [socket.c:3495:socket_init] 0-management: using system polling thread

[2014-04-11 01:04:51.603866] I [glusterd.c:125:glusterd_uuid_init] 0-management: retrieved UUID: 553eb388-83e6-43bd-86b5-0b70cd1ac716

[2014-04-11 01:04:51.607534] E [glusterd-store.c:2487:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore

[2014-04-11 01:04:51.607567] E [xlator.c:390:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again

[2014-04-11 01:04:51.607588] E [graph.c:292:glusterfs_graph_init] 0-management: initializing translator failed

[2014-04-11 01:04:51.607601] E [graph.c:479:glusterfs_graph_activate] 0-graph: init failed

[2014-04-11 01:04:51.607768] W [glusterfsd.c:1002:cleanup_and_exit] (-->/usr/sbin/glusterd(main+0x5d2) [0x406802] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xb7) [0x4051b7] (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x103) [0x4050c3]))) 0-: received signum (0), shutting down

●まずは、[2014-04-11 01:04:49.498045] E [rpc-transport.c:253:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.4.2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory　に対処。

これっておそらく今回の問題に当たる前から出てたんだと思う。rdma使ってないので、設定を修正。

[ec2-user@ip-10-152-253-193 ~]$ diff -Nru /etc/glusterfs/glusterd.vol{.back,}

--- /etc/glusterfs/glusterd.vol.back 2014-01-03 12:38:22.000000000 +0000

+++ /etc/glusterfs/glusterd.vol 2014-04-11 01:07:15.819054736 +0000

@@ -1,7 +1,7 @@

 volume management

     type mgmt/glusterd

     option working-directory /var/lib/glusterd

-    option transport-type socket,rdma

+    option transport-type socket

     option transport.socket.keepalive-time 10

     option transport.socket.keepalive-interval 2

     option transport.socket.read-fail-log off

そしてもっかい起動してみる。

[ec2-user@ip-10-152-253-193 ~]$ sudo /etc/init.d/glusterd start

                                         [FAILED]

ほう。ログ見ると、rdmaのエラーは消えたけど、、、

[2014-04-11 01:08:05.016191] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.4.2 (/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)

[2014-04-11 01:08:05.024142] I [glusterd.c:961:init] 0-management: Using /var/lib/glusterd as working directory

[2014-04-11 01:08:05.027698] I [socket.c:3480:socket_init] 0-socket.management: SSL support is NOT enabled

[2014-04-11 01:08:05.027726] I [socket.c:3495:socket_init] 0-socket.management: using system polling thread

[2014-04-11 01:08:06.544738] I [glusterd-store.c:1339:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2

[2014-04-11 01:08:06.548780] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0

[2014-04-11 01:08:06.548823] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1

[2014-04-11 01:08:07.119147] I [glusterd-handler.c:2818:glusterd_friend_add] 0-management: connect returned 0

[2014-04-11 01:08:07.119290] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600

[2014-04-11 01:08:07.119375] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled

[2014-04-11 01:08:07.119391] I [socket.c:3495:socket_init] 0-management: using system polling thread

[2014-04-11 01:08:07.128143] I [glusterd.c:125:glusterd_uuid_init] 0-management: retrieved UUID: 553eb388-83e6-43bd-86b5-0b70cd1ac716

[2014-04-11 01:08:07.130647] E [glusterd-store.c:2487:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore

[2014-04-11 01:08:07.130679] E [xlator.c:390:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again

[2014-04-11 01:08:07.130695] E [graph.c:292:glusterfs_graph_init] 0-management: initializing translator failed

[2014-04-11 01:08:07.130706] E [graph.c:479:glusterfs_graph_activate] 0-graph: init failed

[2014-04-11 01:08:07.130880] W [glusterfsd.c:1002:cleanup_and_exit] (-->/usr/sbin/glusterd(main+0x5d2) [0x406802] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xb7) [0x4051b7] (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x103) [0x4050c3]))) 0-: received signum (0), shutting down

●なんぞこれ？

[2014-04-11 01:08:06.548780] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0

[2014-04-11 01:08:06.548823] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1

MLアーカイブでいくつかヒットするんだけど、どうもハズしてそうなのと、未解決なまま残っているのと・・・。

http://www.gluster.org/pipermail/gluster-users/2013-August/036984.html

http://www.gluster.org/pipermail/gluster-users/2013-January/035249.html

http://gluster.org/pipermail/gluster-users/2012-August/011189.html

原因究明のためにAWSでスナップショットとっといて、とりあえず復旧優先で進めます。

●ボリュームを再作成する

[ec2-user@ip-10-152-253-193 ~]$ sudo cp -rp /var/lib/glusterd{,.back}

[ec2-user@ip-10-152-253-193 ~]$ sudo diff -Nru /var/lib/glusterd{.back,}

[ec2-user@ip-10-152-253-193 ~]$ sudo gluster peer probe ec2-****

$ sudo gluster volume heal vol01 info

→　あかん

$ sudo /etc/init.d/nginx stop

$ sudo umount /var/www

$ sudo /etc/init.d/glusterfsd stop

$ sudo gluster volume stop vol01

$ sudo gluster volume delete vol01

$ sudo gluster volume create vol01 replica 2 transport tcp ec2-54-238-57-107.ap-northeast-1.compute.amazonaws.com:/data/brick01 ec2-176-34-56-112.ap-northeast-1.compute.amazonaws.com:/data/brick01

volume create: vol01: failed: /data/brick01 or a prefix of it is already part of a volume

→怒られた

$ sudo rm -rf /data/brick01.glusterfs

$ sudo setfattr -x trusted.glusterfs.volume-id /data/brick01

$ sudo setfattr -x trusted.gfid /data/brick01

[ec2-user@ip-10-152-253-193 ~]$ sudo rm -rf /data/brick01.glusterfs

[ec2-user@ip-10-152-253-193 ~]$ sudo setfattr -x trusted.glusterfs.volume-id /data/brick01

[ec2-user@ip-10-152-253-193 ~]$ sudo setfattr -x trusted.gfid /data/brick01

$ sudo gluster volume create vol01 replica 2 transport tcp ec2-54-238-57-107.ap-northeast-1.compute.amazonaws.com:/data/brick01 ec2-176-34-56-112.ap-northeast-1.compute.amazonaws.com:/data/brick01

volume create: vol01: success: please start the volume to access data

→　成功

[ec2-user@ip-10-126-5-43 ~]$ sudo gluster volume start vol01

volume start: vol01: success

[ec2-user@ip-10-126-5-43 ~]$ sudo gluster volume status

Status of volume: vol01

Gluster process Port Online Pid

------------------------------------------------------------------------------

Brick ec2-54-238-57-107.ap-northeast-1.compute.amazonaw

s.com:/data/brick01 49152 Y 3379

Brick ec2-176-34-56-112.ap-northeast-1.compute.amazonaw

s.com:/data/brick01 49153 Y 3370

NFS Server on localhost 2049 Y 3387

Self-heal Daemon on localhost N/A Y 3382

NFS Server on ec2-54-238-57-107.ap-northeast-1.compute.

amazonaws.com 2049 Y 3395

Self-heal Daemon on ec2-54-238-57-107.ap-northeast-1.co

mpute.amazonaws.com N/A Y 3391

There are no active volume tasks

[ec2-user@ip-10-126-5-43 ~]$ sudo /etc/init.d/glusterfsd start

[ec2-user@ip-10-126-5-43 ~]$ df

Filesystem     1K-blocks    Used Available Use% Mounted on

/dev/xvda1       8256952 1164492   7008576  15% /

tmpfs             304144       0    304144   0% /dev/shm

/dev/xvdb1      20308020  203844  19055940   2% /data

[ec2-user@ip-10-126-5-43 ~]$ sudo /etc/init.d/glusterfsd status

glusterfsd (pid 3370) is running...

[ec2-user@ip-10-126-5-43 ~]$ sudo mount /var/www

[ec2-user@ip-10-126-5-43 ~]$ df

Filesystem                                                    1K-blocks    Used Available Use% Mounted on

/dev/xvda1                                                      8256952 1164516   7008552  15% /

tmpfs                                                            304144       0    304144   0% /dev/shm

/dev/xvdb1                                                     20308020  203844  19055940   2% /data

ec2-176-34-56-112.ap-northeast-1.compute.amazonaws.com:/vol01  20307968  203904  19055872   2% /var/www

[ec2-user@ip-10-126-5-43 ~]$ sudo /etc/init.d/nginx start

Starting nginx:                                            [  OK  ]


ひとまず。