Integration of LVM with Hadoop-Cluster & providing Elasticity to Datanode Storage

What is LVM ( Logical Volume Management) ?

Logical volume manager (LVM) introduces an extra layer between the physical disks and the file system allowing file systems to be :
– resized and moved easily and online without requiring a system-wide outage.
– Using discontinuous space on disk
– meaningful names to volumes, rather than the usual cryptic device names.
– span multiple physical disks

Concepts

LVM comprises of few conceptual layers such as physical volume, logical volume and file systems.

Physical Volume (PV)

Each Physical Volume can be a disk partition, whole disk, meta-device, or a loopback file. Use the command pvcreate to initialize storage for use by LVM. Initializing a block device as physical volume places a label at the start of the device.

Volume Group (VG)

A Volume Group gathers together a collection of Logical Volumes and Physical Volumes into one administrative unit. Volume group is divided into fixed size physical extents. The command vgcreate creates a new volume group using the block special device Physical Volume path previously configured for LVM with pvcreate.
– VGs are made up of PVs, which in turn are made up of physical extents (PEs). The size of PE can differe in different VGs and is defined at the time of creating VG.
– The default size of PE is 4MB, but you can change it to the value you want at the time of VG creation.
– Generally, larger the PE size, better the performance (though less granular control of LV).

Logical Volume (LV)

A Logical Volume is the conceptual equivalent of a disk partition in a non-LVM system. Logical volumes are block devices which are created from the physical extents present in the same volume group. You can use command lvcreate to create a logical volume in an existing volume group.

File system

File systems are built on top of logical volumes. The command mkfs can be used to create file system on top of a logical volume. Once the file system is created we can mount the logical volume as per our need.

Task Description 📄

🔅Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

🔅Increase or Decrease the Size of Static Partition in Linux.

Lets add the Harddisk to our datanode

Here I Added two harddisk

/dev/sdb (10GiB) and

/dev/sdc (20GiB)

To Check its Successfully attach or not run this command

#fdisk -l

Convert these HD’s into Physical Volume (PV)

# pvcreate /dev/sdb(1st HD) /dev/sdc (2nd HD)

To see details about Physical Volume

# pvdisplay /dev/sdb(1st HD) /dev/sdc (2nd HD)

Create Volume Group (VG) with physical volumes

For creating VG use command as

# vgcreate name /dev/sdb /dev/sdc

To see whether the VG is created or not use command

# vgdisplay name

Create partition i.e. Logical Volume (LV) of a volume group of size you want to contribute to namenode

creating a partition (LV)

lvcreate — size 20G — name LV_name VG

Format the partition use command

# mkfs.ext4 /dev/VG_name/LV_name

Mount that partition on datanode folder (/dn) use command

# mount /dev/VG_name/LV_name /dn

On the fly we can increase/decrease the storage to be contributed to namenode without unmounting or stopping any services. We can increase the size upto 30G .

For extending the volume contribution use command

# lvextend — size +2G /dev/VG_name/LV_name

Format the extended part use the command ( creating file system for unmounted new volume)

# resize2fs /dev/VG_name/LV_name

Now check for the storage

TASK COMPLETED👨

AUTOMATE LVM USING PYTHON

--

--

--

★ Aspiring DevOps Engineer ★ Technical Volunteer @LinuxWorld ★ Technical Content Writer @Medium ★ ARTH Learner

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Top 9 IDE & Tools for Flutter Development

A scalable Deep Link approach to navigation in Android

Nym Incentivized Testnet and Reputation System

Build your own recommendation system chatbot with Facebook Messenger and Node.js (Part 1)

Introducing Pinknode’s strategic investor.. Moonrock Capital!

Elasticsearch

Merkle Membership Proofs

Ready…Set..DEPLOY— A short overview of different deployment models

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Vineet Negi

Vineet Negi

★ Aspiring DevOps Engineer ★ Technical Volunteer @LinuxWorld ★ Technical Content Writer @Medium ★ ARTH Learner

More from Medium

Wemanage Journal: Introducing dataFORMAT

The 5-Minute / 10-Question User Guide

Why is zinc plated screws?

The New Mutants (2020, Dir. Josh Boone)