From Fedora Project Wiki

No edit summary
No edit summary
Line 37: Line 37:
* install the RPMs on all brick nodes and client nodes
* install the RPMs on all brick nodes and client nodes
* make filesystems on the iSCSI LUNs and mount them, e.g. using ext4:
* make filesystems on the iSCSI LUNs and mount them, e.g. using ext4:
        for lun in /dev/sd? ; do sudo mkfs.ext4 $lun; done
** <code>for lun in /dev/sd? ; do sudo mkfs.ext4 $lun; done</code>
        for dev in /dev/sd? ; do sudo mkdir -p /bricks/`basename $dev`; done
** <code>for dev in /dev/sd? ; do sudo mkdir -p /bricks/`basename $dev`; done</code>
        for dev in /dev/sd? ; do sudo mount $dev /bricks/`basename $dev`; done
** <code>for dev in /dev/sd? ; do sudo mount $dev /bricks/`basename $dev`; done</code>
        optional make /etc/fstab entries for the mounts
** optional make /etc/fstab entries for the mounts
        Note: if you use qemu/kvm guests as bricks and use xfs on iSCSI LUNs shared from the qemu/kvm host, the guests will not always probe and initialize the iSCSI LUNs correctly (fast enough) and the guests will usually require manual intervention to boot. Unmount all bricks, run xfs_repair on each iSCSI device (e.g. /dev/sda), mount the bricks, and `exit` to continue to boot.  
** Note: if you use qemu/kvm guests as bricks and use xfs on iSCSI LUNs shared from the qemu/kvm host, the guests will not always probe and initialize the iSCSI LUNs correctly (fast enough) and the guests will usually require manual intervention to boot. Unmount all bricks, run xfs_repair on each iSCSI device (e.g. /dev/sda), mount the bricks, and `exit` to continue to boot.  
* open ports in the firewall using the firewall admin utility:
* open ports in the firewall using the firewall admin utility:
        on all bricks, open port 8080 tcp (Other Ports)
** on all bricks, open port 8080 tcp (Other Ports)
        on all bricks, open ports 24007-24029 tcp (Other Ports)
** on all bricks, open ports 24007-24029 tcp (Other Ports)
* setup gluster on the brick nodes
* setup gluster on the brick nodes
        enable glusterfsd, glusterd, and hekafsd on each brick:
*# enable glusterfsd, glusterd, and hekafsd on each brick:
            chkconfig glusterfsd on
*#* chkconfig glusterfsd on
            chkconfig glusterd on
*#* chkconfig glusterd on
            chkconfig hekafsd on  
*#* chkconfig hekafsd on  
        start glusterd and hekafsd on each brick:
*# start glusterd and hekafsd on each brick:
            service glusterd start
*#* service glusterd start
            service hekafsd start  
*#* service hekafsd start  
        open a browser window to principal node port 8080. we use Google Chrome, Firefox seems not to work well
*# open a browser window to principal node port 8080. we use Google Chrome, Firefox seems not to work well
        configure nodes in your cluster
*# configure nodes in your cluster
            select Manage Servers
*#* select Manage Servers
            the IP address of the first or principal node is already listed
*#* the IP address of the first or principal node is already listed
            enter the IP address or node name and press Add
*#* enter the IP address or node name and press Add
            click 'Back to cluster configuration'
*#* click 'Back to cluster configuration'
            repeat for each node in your cluster
*#* repeat for each node in your cluster
            press Done
*#* press Done
        configure one or more volumes in your cluster
*# configure one or more volumes in your cluster
            select Manage Volumes
*#* select Manage Volumes
            As per above, each node in my cluster has ten volumes: /dev/sda ... /dev/sdj, mounted on /bricks/sda ... /bricks/sdj
*#* As per above, each node in my cluster has ten volumes: /dev/sda ... /dev/sdj, mounted on /bricks/sda ... /bricks/sdj
            tick the checkbox for /bricks/sda on each node
*#* tick the checkbox for /bricks/sda on each node
            leave Volume Type: set to Plain
*#* leave Volume Type: set to Plain
            leave Replica or Strip count unset
*#* leave Replica or Strip count blank (unset)
            enter testsda for Volume ID
*#* enter testsda for Volume ID
            press Provision
*#* press Provision
            add_local(testsda) OK ... is displayed for each node
*#* add_local(testsda) OK ... is displayed for each node
            click 'Back to volume configuration'
*#* click 'Back to volume configuration'
            testsda is now shown in the list of Existing Volumes
*#* testsda is now shown in the list of Existing Volumes
            repeat as desired for additional volumes
*#* repeat as desired for additional volumes
            use the Back button in your browser to return to the Configuration Main menu
*#* use the Back button in your browser to return to the Configuration Main menu
        configure one or more tenants in your cluster
*# configure one or more tenants in your cluster
            select Manage Tenants
*#* select Manage Tenants
            enter bob as the Tenant Name
*#* enter bob as the Tenant Name
            enter carol as the Tenant Password
*#* enter carol as the Tenant Password
            enter 10000 as the Tenant UID Range: Low
*#* enter 10000 as the Tenant UID Range: Low
            enter 10999 as the Tenant GID Range: High
*#* enter 10999 as the Tenant GID Range: High
            enter 10000 as the Tenant GID Range: Low
*#* enter 10000 as the Tenant GID Range: Low
            enter 10999 as the Tenant UID Range: High
*#* enter 10999 as the Tenant UID Range: High
            press Add
*#* press Add
            add_local(bob) OK ... is displayed for each node
*#* add_local(bob) OK ... is displayed for each node
            click 'Back to tenant configuration'
*#* click 'Back to tenant configuration'
            bob is now shown in the list of Existing Tenants
*#* bob is now shown in the list of Existing Tenants
            repeat as desired for additional tenants
*#* repeat as desired for additional tenants
            click 'volumes' in the entry for bob
*#* click 'volumes' in the entry for bob
            testsda is shown in the Volume List
*#* testsda is shown in the Volume List
            tick the Enabled checkbox for testsda
*#* tick the Enabled checkbox for testsda
            press Update
*#* press Update
            Volumes enabled for bob ... is displayed for each node
*#* Volumes enabled for bob ... is displayed for each node
            click 'Back to tenant configuration'
*#* click 'Back to tenant configuration'
        start the volume(s)
*# start the volume(s)
            use the Back button in your browser to return to the Configuration Main menu
*#* use the Back button in your browser to return to the Configuration Main menu
            select Manage Volumes
*#* select Manage Volumes
            click 'start' testsda entry in the list of Existing Volumes
*#* click 'start' testsda entry in the list of Existing Volumes
            start_local(testsda) returned 0 ... is displayed for each node
*#* start_local(testsda) returned 0 ... is displayed for each node
* mount the volume(s) on the client(s)
* mount the volume(s) on the client(s)
        sudo hfs_mount 192.168.122.21 testsda bob carol /mnt/testsda
** sudo hfs_mount 192.168.122.21 testsda bob carol /mnt/testsda

Revision as of 18:37, 8 September 2011

Setting up a simple HekaFS cluster

Hypervisor/Host

8 CPU, 24GB RAM, FC HBA to back-end storage, running F15.

Brick node guests on host

  • F15node01 (192.168.122.21)
  • F15node02 (192.168.122.22)
  • F15node03 (192.168.122.23)
  • F15node04 (192.168.122.24)

Client node guest(s) on host

  • F15node05 (192.168.122.25)

N.B. all guest nodes are running F15.

Back-end storage

40 5G LUNs on qlogic fibrechannel, provisioned as SCSI disks, 10 per brick. N.B. Size and number of LUNs is arbitrary.

Important Links

cloudfs git repo is git://git.fedorahosted.org/CloudFS.git F16 glusterfs RPMs are https://koji.fedoraproject.org/koji/buildinfo?buildID=259916 F16 hekafs RPM is https://koji.fedoraproject.org/koji/buildinfo?buildID=259462 HekaFS wiki page is https://fedoraproject.org/wiki/HekaFS F16 HekaFS Feature wiki page is https://fedoraproject.org/wiki/Features/HekaFS

If you use the hekafs RPM on RHEL, change line 23 of /etc/init.d/hekafsd from python2.7 to python2.6 after installing.

the nitty gritty

  • download the glusterfs, glusterfs-server, glusterfs-fuse, and hekafs RPMs
  • install the RPMs on all brick nodes and client nodes
  • make filesystems on the iSCSI LUNs and mount them, e.g. using ext4:
    • for lun in /dev/sd? ; do sudo mkfs.ext4 $lun; done
    • for dev in /dev/sd? ; do sudo mkdir -p /bricks/basename $dev; done
    • for dev in /dev/sd? ; do sudo mount $dev /bricks/basename $dev; done
    • optional make /etc/fstab entries for the mounts
    • Note: if you use qemu/kvm guests as bricks and use xfs on iSCSI LUNs shared from the qemu/kvm host, the guests will not always probe and initialize the iSCSI LUNs correctly (fast enough) and the guests will usually require manual intervention to boot. Unmount all bricks, run xfs_repair on each iSCSI device (e.g. /dev/sda), mount the bricks, and exit to continue to boot.
  • open ports in the firewall using the firewall admin utility:
    • on all bricks, open port 8080 tcp (Other Ports)
    • on all bricks, open ports 24007-24029 tcp (Other Ports)
  • setup gluster on the brick nodes
    1. enable glusterfsd, glusterd, and hekafsd on each brick:
      • chkconfig glusterfsd on
      • chkconfig glusterd on
      • chkconfig hekafsd on
    2. start glusterd and hekafsd on each brick:
      • service glusterd start
      • service hekafsd start
    3. open a browser window to principal node port 8080. we use Google Chrome, Firefox seems not to work well
    4. configure nodes in your cluster
      • select Manage Servers
      • the IP address of the first or principal node is already listed
      • enter the IP address or node name and press Add
      • click 'Back to cluster configuration'
      • repeat for each node in your cluster
      • press Done
    5. configure one or more volumes in your cluster
      • select Manage Volumes
      • As per above, each node in my cluster has ten volumes: /dev/sda ... /dev/sdj, mounted on /bricks/sda ... /bricks/sdj
      • tick the checkbox for /bricks/sda on each node
      • leave Volume Type: set to Plain
      • leave Replica or Strip count blank (unset)
      • enter testsda for Volume ID
      • press Provision
      • add_local(testsda) OK ... is displayed for each node
      • click 'Back to volume configuration'
      • testsda is now shown in the list of Existing Volumes
      • repeat as desired for additional volumes
      • use the Back button in your browser to return to the Configuration Main menu
    6. configure one or more tenants in your cluster
      • select Manage Tenants
      • enter bob as the Tenant Name
      • enter carol as the Tenant Password
      • enter 10000 as the Tenant UID Range: Low
      • enter 10999 as the Tenant GID Range: High
      • enter 10000 as the Tenant GID Range: Low
      • enter 10999 as the Tenant UID Range: High
      • press Add
      • add_local(bob) OK ... is displayed for each node
      • click 'Back to tenant configuration'
      • bob is now shown in the list of Existing Tenants
      • repeat as desired for additional tenants
      • click 'volumes' in the entry for bob
      • testsda is shown in the Volume List
      • tick the Enabled checkbox for testsda
      • press Update
      • Volumes enabled for bob ... is displayed for each node
      • click 'Back to tenant configuration'
    7. start the volume(s)
      • use the Back button in your browser to return to the Configuration Main menu
      • select Manage Volumes
      • click 'start' testsda entry in the list of Existing Volumes
      • start_local(testsda) returned 0 ... is displayed for each node
  • mount the volume(s) on the client(s)
    • sudo hfs_mount 192.168.122.21 testsda bob carol /mnt/testsda