Integration of LVM with Hadoop-Cluster & providing Elasticity to Datanode Storage

4 min readNov 6, 2020

What is LVM ( Logical Volume Management) ?

Logical volume manager (LVM) introduces an extra layer between the physical disks and the file system allowing file systems to be :
– resized and moved easily and online without requiring a system-wide outage.
– Using discontinuous space on disk
– meaningful names to volumes, rather than the usual cryptic device names.
– span multiple physical disks

Concepts

LVM comprises of few conceptual layers such as physical volume, logical volume and file systems.

Physical Volume (PV)

Each Physical Volume can be a disk partition, whole disk, meta-device, or a loopback file. Use the command pvcreate to initialize storage for use by LVM. Initializing a block device as physical volume places a label at the start of the device.

Volume Group (VG)

A Volume Group gathers together a collection of Logical Volumes and Physical Volumes into one administrative unit. Volume group is divided into fixed size physical extents. The command vgcreate creates a new volume group using the block special device Physical Volume path previously configured for LVM with pvcreate.
– VGs are made up of PVs, which in turn are made up of physical extents (PEs). The size of PE can differe in different VGs and is defined at the time of creating VG.
– The default size of PE is 4MB, but you can change it to the value you want at the time of VG creation.
– Generally, larger the PE size, better the performance (though less granular control of LV).

Logical Volume (LV)

A Logical Volume is the conceptual equivalent of a disk partition in a non-LVM system. Logical volumes are block devices which are created from the physical extents present in the same volume group. You can use command lvcreate to create a logical volume in an existing volume group.

File system

File systems are built on top of logical volumes. The command mkfs can be used to create file system on top of a logical volume. Once the file system is created we can mount the logical volume as per our need.

Task Description 📄

🔅Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

🔅Increase or Decrease the Size of Static Partition in Linux.

Lets add the Harddisk to our datanode

Here I Added two harddisk

/dev/sdb (10GiB) and

/dev/sdc (20GiB)

To Check its Successfully attach or not run this command

#fdisk -l

Convert these HD’s into Physical Volume (PV)

# pvcreate /dev/sdb(1st HD) /dev/sdc (2nd HD)

To see details about Physical Volume

# pvdisplay /dev/sdb(1st HD) /dev/sdc (2nd HD)

Create Volume Group (VG) with physical volumes

For creating VG use command as

# vgcreate name /dev/sdb /dev/sdc

To see whether the VG is created or not use command

# vgdisplay name

Create partition i.e. Logical Volume (LV) of a volume group of size you want to contribute to namenode

creating a partition (LV)

lvcreate — size 20G — name LV_name VG

Format the partition use command

# mkfs.ext4 /dev/VG_name/LV_name

Mount that partition on datanode folder (/dn) use command

# mount /dev/VG_name/LV_name /dn

On the fly we can increase/decrease the storage to be contributed to namenode without unmounting or stopping any services. We can increase the size upto 30G .