Using GlusterFS With GlusterFS Samba vfs plugin on Fedora

Posted on by

This blog covers the steps and implementation details to use GlusterFS Samba VFS plugin.

Please refer below link, If you are looking for architectural information for GlusterFS Samba VFS plugin,  difference between FUSE mount vs Samba VFS plugin

https://lalatendumohanty.wordpress.com/2014/04/20/glusterfs-vfs-plugin-for-samba/

I have setup  two node GlusterFS cluster with Fedora 20 (minimal install) VMs. Each VM has 3 separate XFS partitions with each partitions 100GB each.
One of the Gluster node is used as a Samba server in this setup.

I had originally tested this with Fedora 20. But this example should work fine with latest Fedoras i.e. F21 and F22

GlusterFS Version: glusterfs-3.4.2-1.fc20.x86_64

Samba version:  samba-4.1.3-2.fc20.x86_64

Post installation “df -h” command looked like below in the VMs
$df -h
Filesystem                            Size  Used Avail Use% Mounted on
/dev/mapper/fedora_dhcp159–242-root   50G  2.2G   45G   5% /
devtmpfs                              2.0G     0  2.0G   0% /dev
tmpfs                                 2.0G     0  2.0G   0% /dev/shm
tmpfs                                 2.0G  432K  2.0G   1% /run
tmpfs                                 2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs                                 2.0G     0  2.0G   0% /tmp
/dev/vda1                             477M  103M  345M  23% /boot
/dev/mapper/fedora_dhcp159–242-home   45G   52M   43G   1% /home
/dev/mapper/gluster_vg1-gluster_lv1           100G  539M  100G   1% /gluster/brick1
/dev/mapper/gluster_vg2-gluster_lv2           100G  406M  100G   1% /gluster/brick2
/dev/mapper/gluster_vg3-gluster_lv3           100G   33M  100G   1% /gluster/brick3

You can use following commands to create xfs partitions
1. pvcreate /dev/vdb
2. vgcreate VG_NAME /dev/vdb
3. lvcreate -n LV_NAME -l 100%PVS VG_NAME /dev/vdb
4. mkfs.xfs -i size=512 LV_PATH

Following are the steps and packages need to be performed/installed on each node (which is Fedora 20 for mine)

#Change SELinux to either “permissive” or “disabled” mode

# To put SELinux in permissive mode
$setenforce 0

#To see the current mode of SELinux

$getenforce

SELinux policy rules for Gluster is present in recent Fedora releases e.g. F21, F22 or later. So SELinux should work fine with Gluster.

#Remove all iptable rules, so that it does not interfare with Gluster

$iptables -F

yum install glusterfs-server
yum install samba-vfs-glusterfs
yum install samba-client

#samba-vfs-glusterfs RPMs for CentOS, RHEL, Fedora19/18 are avialable at http://download.gluster.org/pub/gluster/glusterfs/samba/

#To start glusterd and auto start it after boot
$systemctl start glusterd
$systemctl enable glusterd
$systemctl status glusterd

#To start smb and auto start it after boot
$systemctl start smb
$systemctl enable smb
$systemctl status smb

#Create gluster volume and start it. (Running below commands from Server1_IP)

$gluster peer probe Server2_IP
$gluster peer status
Number of Peers: 1

Hostname: Server2_IP
Port: 24007
Uuid: aa6f71d9-0dfe-4261-a2cd-5f281632aaeb
State: Peer in Cluster (Connected)
$gluster v create testvol Server2_IP:/gluster/brick1/testvol-b1 Server1_IP:/gluster/brick1/testvol-b2
$gluster v start testvol

#Modify smb.conf for Samba share

$vi /etc/samba/smb.conf

#
[testvol]
comment = For samba share of volume testvol
path = /
read only = No
guest ok = Yes
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 7
glusterfs:logfile = /var/log/samba/glusterfs-testvol.log
glusterfs:volume = testvol

#For debug logs you can change the log levels to 10 e.g: “glusterfs:loglevel = 10”

# Do not miss “kernel share modes = No” else you won’t be able to write anything in to the share

#verify that your changes are correctly understood by Samba
$testparm -s
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Processing section “[homes]”
Processing section “[printers]”
Processing section “[testvol]”
Loaded services file OK.
Server role: ROLE_STANDALONE
[global]
workgroup = MYGROUP
server string = Samba Server Version %v
log file = /var/log/samba/log.%m
max log size = 50
idmap config * : backend = tdb
cups options = raw

[homes]
comment = Home Directories
read only = No
browseable = No

[printers]
comment = All Printers
path = /var/spool/samba
printable = Yes
print ok = Yes
browseable = No

[testvol]
comment = For samba share of volume testvol
path = /
read only = No
guest ok = Yes
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 10
glusterfs:logfile = /var/log/samba/glusterfs-testvol.log
glusterfs:volume = testvol

#Restart the Samba service. This not a compulsory step as Samba takes latest smb.conf for new connections. But to make sure it uses the latest smb.conf, restart the service.
$systemctl  restart smb

#Set smbpasswd for root. This will be used for mounting the volume/Samba share on the client
$smbpasswd -a root

#Mount the cifs share using following command and it is ready for use 🙂
mount -t cifs -o username=root,password=<smbpassword> //Server1_IP/testvol /mnt/cifs

GlusterFS volume tuning for volume shared through Samba:

  • Gluster volume needs to have: “gluster volume set volname server.allow-insecure on”
  • /etc/glusterfs/glusterd.vol of each of gluster node
    add “option rpc-auth-allow-insecure on”
  • Restart glusterd of each node.

For setups where Samba server and Gluster nodes need to be on different machines:

# put “glusterfs:volfile_server = <server name/ip>” in the smb.conf settings for the specific  volume

e.g:

[testvol]
comment = For samba share of volume testvol
path = /
read only = No
guest ok = Yes
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 7
glusterfs:logfile = /var/log/samba/glusterfs-testvol.log

glusterfs:volfile_server = <server name/ip>
glusterfs:volume = testvol

#Here are the packages that were installed on the nodes

rpm -qa | grep gluster
glusterfs-libs-3.4.2-1.fc20.x86_64
glusterfs-api-3.4.2-1.fc20.x86_64
glusterfs-3.4.2-1.fc20.x86_64
glusterfs-cli-3.4.2-1.fc20.x86_64
glusterfs-server-3.4.2-1.fc20.x86_64
samba-vfs-glusterfs-4.1.3-2.fc20.x86_64
glusterfs-devel-3.4.2-1.fc20.x86_64
glusterfs-fuse-3.4.2-1.fc20.x86_64
glusterfs-api-devel-3.4.2-1.fc20.x86_64

[root@dhcp159-242 ~]# rpm -qa | grep samba
samba-client-4.1.3-2.fc20.x86_64
samba-4.1.3-2.fc20.x86_64
samba-vfs-glusterfs-4.1.3-2.fc20.x86_64
samba-libs-4.1.3-2.fc20.x86_64
samba-common-4.1.3-2.fc20.x86_64

Note: The same smb.conf entries should work with CentOS6 too.

12 thoughts on “Using GlusterFS With GlusterFS Samba vfs plugin on Fedora”

  1. Pingback: GlusterFS VFS plugin for Samba | Free Software, Open Source And My Life Around That

  2. Thanks for this writeup.
    Say if i wanted to move from a Gluster Fuse mounted samba share, using CTDB, to this. How would I implement the clustering element? In this writeup you use samba-vfs-glusterfs but there is no equivalent for CTDB. is there an alternative method of providing a load-balanced/clustered file share using the vfs option?

    Reply ↓

    • CTDB works fine with Samba VFS plugin. Technically CTDB is a different software and it talks to Samba but AFAIK it does not interfere with how Samba manages its share. So you can use the usual way you use CTDB with Samba. The only change would be that Samba will use vfs plugin to export the Gluster volume. I have used CTDB+Samba (with Gluster VFS plugin)+GlusterFS Volume and it works pretty well.

      Reply ↓

  3. Cool, almost there although the share is showing up I don’t seem to be able to browse it (in either a standalone samba config or a clustered ctdb config)

    Have 2 nodes:
    storenode1 & storenode2 each have an identical samba config and contain the gluster volume “share”
    gluster vol info share

    Volume Name: share
    Type: Replicate
    Volume ID: 06d1eb42-873d-43fe-ae94-562e975cca9a
    Status: Started
    Number of Bricks: 1 x 2 = 2
    Transport-type: tcp
    Bricks:
    Brick1: storenode1:/gluster/bricks/share/brick1
    Brick2: storenode2:/gluster/bricks/share/brick1

    I’ve presented the volume via samba/ctdb as per this page and started ctdb/samba. The share is visible at top level (i.e. \\clusternode\ \\storenode1 and \\storenode2 – clusternode being the vip that ctdb presents) but is inaccessible.

    [share]
    comment = Gluster and CTDB based share
    path = /
    valid users = jon
    read only = No
    guest ok = Yes
    kernel share modes = No
    vfs objects = glusterfs
    glusterfs:volume = share
    glusterfs:logfile = /var/log/samba/glusterfs-testvol.log
    glusterfs:loglevel = 7

    with the result the same with samba and ctdb I’m presuming that neither are talking to gluster correctly.

    Any ideas what I’m missing?

    Thanks

    Reply ↓

  4. Sorry to bombard your comments, but have a little more info. From the samba logs I’m getting:
    [2014/07/17 13:31:56.084620, 0] ../source3/modules/vfs_glusterfs.c:253(vfs_gluster_connect)
    Failed to set volfile_server localhost

    I tried adding the volfile_server option in my samba config and setting it to localhost, IP and storenode1 all resulting in a similar message:
    Failed to set volfile_server to {localhost,IP,storenode1}

    Reply ↓

  7. Question on the VFS plugin.. I noticed that when I created a new volume and start the volume, an entry is automatically added to the smb.conf giving a guest access to the whole volume. Is this being done by the VFS plugin and if so, is there a way to disable this automatic add to the smb.conf?

    Reply ↓

    • Yes, you can disable the automatic changes to smb.conf. These changes are done by couple of hook scripts. The scripts location is as below. You can remove these or change the name of the scripts i.e. s/S/K/. So that it wont be called by glusterd.
      find /var/lib/glusterd -type f | grep S30samba
      /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh
      /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh
      /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh

      Reply ↓

Leave a comment