This article is a short howto/guide on installing a 3 node glusterfs replicated storage. In this howto I shall be using 4 virtual machines with two hard disks, the second hard disk is used for glusterfs while the first is used for the os. The fourth virtual machine is to be used as a test client.

Hostname IP Address CPU Memory(Gb) Disk /dev/vda Disk /dev/vdb
glusterfs-1.lab.jnvilo.com172.31.1.31115Gigs5Gigs
glusterfs-2.lab.jnvilo.com172.31.1.32115Gigs5Gigs
glusterfs-3.lab.jnvilo.com172.31.1.33115Gigs5Gigs
glusterfs-client.lab.jnvilo.com172.31.1.34115Gigs5Gigs

Install required repositories

yum -y install epel-release
yum install -y centos-release-gluster
yum -y update

Install glusterfs server

yum install -y glusterfs-server
systemctl start glusterd
systemctl status glusterd
systemctl enable glusterd

Configure firewall

firewall-cmd --zone=public --add-port=24007-24008/tcp --permanent
firewall-cmd --zone=public --add-port=24009/tcp --permanent
firewall-cmd --zone=public --add-service=nfs --add-service=samba --add-service=samba-client --permanent
firewall-cmd --zone=public --add-port=111/tcp --add-port=139/tcp --add-port=445/tcp --add-port=965/tcp --add-port=2049/tcp --add-port=38465-38469/tcp --add-port=631/tcp --add-port=111/udp --add-port=963/udp --add-port=49152-49251/tcp --permanent
firewall-cmd --reload

Configure or Create a filesystem for use.

In order to create a gluster volume we need to provision disk storage. This could be anything as mounted LVM or partitions. In this example, I am using virtual machines on kvm as test nodes and have added a 5Gig second hard disk as vdb and the following steps mounts the disk.

create a partion on vdb1 and format it as ext4

fdisk /dev/vdb
mkfs.ext4 /dev/vdb1

create the mountpoint, mount and also add entry to fstab.

mkdir -p /data/gluster
mount /dev/vdb1 /data/gluster/
echo "/dev/vdb1 /data/gluster ext4 defaults 0 0" | tee --append /etc/fstab

Configure the cluster.

The following commands are to be executed on the first node, in my case gluster-1.lab.jnvilo.com:

gluster peer probe gluster-2.lab.jnvilo.com
gluster peer probe gluster-3.lab.jnvilo.com
gluster peer status
gluster pool list

Create a replica volume called gv0 on our provisioned storage.

Create the directory gv0 on all of the nodes

mkdir -p /data/gluster/gv0

Create the replicated volume. Do this on the first node only..

gluster volume create gv0 replica 3 gluster-1.lab.jnvilo.com:/data/gluster/gv0 gluster-2.lab.jnvilo.com:/data/gluster/gv0 gluster-3.lab.jnvilo.com:/data/gluster/gv0

Start the volume.

gluster volume start gv0

You can see information about the volume by doing:

gluster volume info gv0

Install client

on the client server, we install the glusterfs-client

yum -y install glusterfs-client

create the mountpoint

mkdir /mnt/glusterfs

mount

mount -t glusterfs glusterfs-1.lab.lnxsystems.com:/gv0 /mnt/glusterfs

Testing

Now try creating a file on /mnt/glusterfs and find that it is automagically replicated to /data/gluster/gv0 on all of the glusterfs servers. A few other tests would be to stop networking on 1 or 2 of the servers.