Switched On Tech Design

Tag: vdev

  • ZFS: Adding a new mirror to an existing ZFS pool

     

    Mirrored vdevs are great for performance and it is quite straight-forward to add a mirrored vdev to an existing pool (presumably one with one or more similar vdevs already):

     

    zpool add [poolname] mirror [device01] [device02] [device03]

     

    If it’s a two-way mirror you will only have two devices in the above. An example for ZFS on Ubuntu with a pool named seleucus and two SSDs could look like:

     

    zpool add seleucus mirror ata-SAMSUNG_SSD_830_Series_S0XYNEAC705640 ata-M4-CT128M4SSD2_000000001221090B7BF9

     

    As always, it’s good practice to use the device name found in /dev/disk/by-id/ rather than the sda, sdb, sdc etc. names as the latter can change – the former do not.

  • Western Digital Green drive resilver rates

     

    western-digital-green-drives

     

    We get asked fairly regularly about resilver rates for ZFS pools – these matter as it impacts on how quickly a vdev with faulty disks can rebuild data onto a fresh disk, as well as how quickly you can swap one disk for another. The longer it takes to rebuild the vdev after a disk has died, the longer your pool is operating with less redundancy – meaning that if you have had one disk fail (raidz1) or two disks fail (raidz2) already then one more failure before it has finished rebuilding will cause the vdev and zpool to fail.

     

    Today we have been tasked with swapping new drives into two 6-disk vdevs, each consisting of a mixture of WD20EARX and WD20EARS drives – Western Digital 2TB green drives. One array contains 8TB of information, the other 5TB. The 5TB array fluctuates around 245MB/s resilver rate, and the 8TB fluctuates around 255MB/s – giving around 6 hours and 9 hours rebuild times respectively.

     

    These figures are what we would consider typical for that size of vdev, given the disks involved. We will post more rebuild rates and add them into a database over time – stay tuned 🙂

  • ZFS: Replacing a drive with a larger drive within a vdev

    One way to expand the capacity of a zpool is to replace each disk with a larger disk; once the last disk is replaced the pool can be expanded (or will auto-expand, depending on your pool settings). To do this we do the following:

     

    zpool replace [poolname] [old drive] [new drive]

     

    e.g.:

     

    zpool replace kepler ata-WDC_WD15EARX-00PASB0_WD-WCAZAA512624 ata-WDC_WD20EARX-00PASB0_WD-WCAZAA637471

    If you then check on the pool’s status via zpool status, we see:

     

    NAME                                            STATE     READ WRITE CKSUM
    kepler                                          ONLINE     0     0     0
    raidz2-0                                      ONLINE     0     0     0
    ata-WDC_WD20EARX-00PASB0_WD-WMAZA7352703    ONLINE       0     0     0
    replacing-1                                 ONLINE     0     0     0
    ata-WDC_WD15EARX-00PASB0_WD-WCAZAA512624  ONLINE      0     0     0
    ata-WDC_WD20EARX-00PASB0_WD-WCAZAA637471  ONLINE       0     0     0  (resilvering)

     

    The pool will resilver and copy the drive contents across. If you have large drives which are reasonably full this resilver process can take quite a few hours. You can do this with multiple drives at once; here’s a zpool

     

            NAME                                            STATE     READ WRITE CKSUM
    kepler                                          ONLINE     0     0     0
    raidz2-0                                      ONLINE     0     0     0
    ata-WDC_WD20EARX-00PASB0_WD-WMAZA7352703    ONLINE       0     0     0
    replacing-1                                 ONLINE     0     0     0
    ata-WDC_WD15EARX-00PASB0_WD-WCAZAA512624  ONLINE      0     0     0
    ata-WDC_WD20EARX-00PASB0_WD-WCAZAA637471  ONLINE       0     0     0  (resilvering)
    replacing-2                                 ONLINE       0     0     0
    ata-ST2000DM001-9YN164_W24009TB           ONLINE       0     0     0
    ata-WDC_WD20EARS-00MVWB0_WD-WCAZAC389099  ONLINE       0     0     0  (resilvering)
    ata-WDC_WD20EFRX-68AX9N0_WD-WMC300005367    ONLINE       0     0     0
    ata-WDC_WD20EARX-00MMMB0_WD-WCAWZ0842974    ONLINE       0     0     0
    ata-WDC_WD20EARX-00PASB0_WD-WMAZA7482198    ONLINE       0     0     0

    Please don’t disconnect the old drive before inserting the new one – this can cause issues with some setups where ZFS complains that it cannot find the old drive to replace.

     

    Once your resilver is complete on the final drive you can expand the vdev by running:

     

    zpool online -e [poolname]

     

    or you can turn on automatic expansion with the following settings:

     

    zfs set autoexpand=on [poolname]

     

    If you are using zpool online -e the pool does not have to be offline for it to work. Now sit back and enjoy your increased space!