11/01/2012

AWSでGlusterを動かす


AWSでGlusterを動かしてみます。EBSだと高くつくので、Instance Storeで。まずは、Amazon Linux(ami-5a6cd35b)にInstance Store Volumesを2つくっつけて起動します。

●なにはともあれ、yumアップデート。
[[email protected] ~]$ sudo yum update

●Glusterのソースをダウンロード。

[[email protected] ~]$ wget
http://download.gluster.org/pub/gluster/glusterfs/3.3/3.3.1/glusterfs-3.3.1.tar.gz
--2012-10-29 03:05:13--
http://download.gluster.org/pub/gluster/glusterfs/3.3/3.3.1/glusterfs-3.3.1.tar.gz
Resolving download.gluster.org... 50.57.69.89
Connecting to download.gluster.org|50.57.69.89|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2636698 (2.5M) [application/x-gzip]
Saving to: “glusterfs-3.3.1.tar.gz”

100%[======================================>] 2,636,698   1.13M/s   in 2.2s

2012-10-29 03:05:16 (1.13 MB/s) - “glusterfs-3.3.1.tar.gz” saved
[2636698/2636698]

●Glusterのコンパイルに必要なパッケージをインストール。
[[email protected] ~]$ sudo yum install make gcc flex bison
openssl-static readline-static

●Glusterをコンパイルしてインストール
[[email protected] ~]$ tar zxvf glusterfs-3.3.1.tar.gz
[[email protected] ~]$ cd glusterfs-3.3.1
[[email protected] glusterfs-3.3.1]$ ./configure
(略)
GlusterFS configure summary
===========================
FUSE client        : yes
Infiniband verbs   : no
epoll IO multiplex : yes
argp-standalone    : no
fusermount         : no
readline           : yes
georeplication     : yes
Linux-AIO          : no

[[email protected] glusterfs-3.3.1]$ make
(略)
[[email protected] glusterfs-3.3.1]$ sudo make install
(略)

●起動スクリプトをコピーして、使えるか確認。
[[email protected] glusterfs-3.3.1]$ sudo cp
extras/init.d/glusterd-Redhat /etc/init.d/glusterd
[[email protected] glusterfs-3.3.1]$ sudo chmod a+x
/etc/init.d/glusterd
[[email protected] glusterfs-3.3.1]$ sudo /sbin/service glusterd
start
Starting glusterd:                                         [  OK  ]
[[email protected] glusterfs-3.3.1]$ sudo /sbin/service glusterd
restart
Stopping glusterd:                                         [  OK  ]
Starting glusterd:                                         [  OK  ]
[[email protected] glusterfs-3.3.1]$ sudo /sbin/service glusterd stop
Stopping glusterd:                                         [  OK  ]

●ここまでで一旦、AMIを作る
※scpなどでX.509証明書と秘密鍵を転送する必要あり。

[[email protected] ~]$ sudo su -
[[email protected] ~]# ec2-bundle-vol -d /media/ephemeral0 --privatekey
~ec2-user/PrivateKey.pem --cert ~ec2-user/X509Certificate.pem --user
XXXX-XXXX-XXXX --fstab /etc/fstab
Please specify a value for arch [x86_64]:
(略)
[[email protected] ~]# ec2-upload-bundle -r ap-northeast-1 --bucket
My_Bucket/Gluster_Node_Base --manifest /media/ephemeral0/image.manifest.xml
--access-key XXXXXXXXXXXXXXXXXXXX --secret-key
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
(略)
[[email protected] ~]# ec2-register --region ap-northeast-1
--aws-access-key XXXXXXXXXXXXXXXXXXXX --aws-secret-key
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --name gluster_node_base
My_Bucket/Gluster_Node_Base/image.manifest.xml
IMAGE   ami-xxxxxxxx


●作成したAMIからインスタンスを作って3台起動。
1台目(ap-northeast-1a)
ec2-46-51-226-247.ap-northeast-1.compute.amazonaws.com

2台目(ap-northeast-1b)
ec2-176-32-64-146.ap-northeast-1.compute.amazonaws.com

3台目(ap-northeast-1c)
ec2-54-248-159-67.ap-northeast-1.compute.amazonaws.com

1台目と2台目のglusterのuuidを変えておく。
[[email protected] ~]$ cd /var/lib/glusterd/
[[email protected] glusterd]$ sudo cp -p glusterd.info
glusterd.info.back
[[email protected] glusterd]$ sudo vi glusterd.info
UUID=48a21eef-7789-45db-8f59-c5214297fbd2
[[email protected] ~]$ sudo /sbin/service glusterd restart
Stopping glusterd:                                         [  OK  ]
Starting glusterd:                                         [  OK  ]


この時点での各インスタンスのGlusterのUuidは、
1: UUID=48a21eef-7789-45db-8f59-c5214297fbd1
2: UUID=48a21eef-7789-45db-8f59-c5214297fbd2
3: UUID=48a21eef-7789-45db-8f59-c5214297fbd3

にしておいた。

●Glusterにノードを追加(Trusted Storage Poolを作る)
[[email protected] ~]$ sudo /usr/local/sbin/gluster
gluster> peer probe ec2-176-32-64-146.ap-northeast-1.compute.amazonaws.com
Probe successful
State: Peer in Cluster (Connected)
gluster> peer probe ec2-54-248-159-67.ap-northeast-1.compute.amazonaws.com
Probe successful
gluster> peer status
Number of Peers: 2

Hostname: ec2-176-32-64-146.ap-northeast-1.compute.amazonaws.com
Uuid: 48a21eef-7789-45db-8f59-c5214297fbd2
State: Peer in Cluster (Connected)

Hostname: ec2-54-248-159-67.ap-northeast-1.compute.amazonaws.com
Uuid: 48a21eef-7789-45db-8f59-c5214297fbd3
State: Peer in Cluster (Connected)


1台目のノードは、他のノードから追加してあげる。
[[email protected] ~]$ sudo /usr/local/sbin/gluster
gluster> peer probe ec2-46-51-226-247.ap-northeast-1.compute.amazonaws.com
Probe successful


●3つのレプリカを持つクラスタボリュームを作成して起動する。
gluster> volume create test-volume replica 3 transport tcp
ec2-46-51-226-247.ap-northeast-1.compute.amazonaws.com:/media/ephemeral0
ec2-176-32-64-146.ap-northeast-1.compute.amazonaws.com:/media/ephemeral0
ec2-54-248-159-67.ap-northeast-1.compute.amazonaws.com:/media/ephemeral0
Creation of volume test-volume has been successful. Please start the volume
to access data.
gluster> volume start test-volume
Starting volume test-volume has been successful


●マウントしてみる
[[email protected] ~]$ sudo mkdir /media/gluster0
[[email protected] ~]$ sudo mount -t glusterfs
ec2-46-51-226-247.ap-northeast-1.compute.amazonaws.com:/test-volume
/media/gluster0
[[email protected] ~]$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda1            10321208   1349420   8447500  14% /
tmpfs                   848480         0    848480   0% /dev/shm
/dev/xvdb            153899044    192084 145889336   1% /media/ephemeral0
ec2-46-51-226-247.ap-northeast-1.compute.amazonaws.com:/test-volume
                    153899008    192128 145889280   1% /media/gluster0


●テスト
[[email protected] ~]$ sudo cp -rp ./* /media/gluster0/
[[email protected] ~]$ sudo du -a /media/ephemeral0 | wc -l
2346

これで、東京にあるAWSの3つのリージョンをまたぐ分散ファイルシステムの完成。容量は140GB。

ひとまず。