LVM2 Delta Snapshot file Import & Export

Starting from a basic LVM2 deploy, i.e.:

root@target:~# df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                          3.9G     0  3.9G   0% /dev
tmpfs                         790M  9.1M  781M   2% /run
/dev/mapper/target--vg-root  118G   12G  100G  11% /
tmpfs                         3.9G     0  3.9G   0% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                         3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                     236M  117M  108M  53% /boot
tmpfs                         790M     0  790M   0% /run/user/1000
root@target:~#
root@target:~#
root@target:~# fdisk -l /dev/sda
Disk /dev/sda: 139.8 GiB, 150039945216 bytes, 293046768 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x7cee11fa

Device     Boot  Start       End   Sectors   Size Id Type
/dev/sda1  *      2048    499711    497664   243M 83 Linux
/dev/sda2       501758 293046271 292544514 139.5G  5 Extended
/dev/sda5       501760 293046271 292544512 139.5G 8e Linux LVM


root@target:~#
root@target:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda5
  VG Name               target-vg
  PV Size               139.50 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              35710
  Free PE               2965
  Allocated PE          32745
  PV UUID               TsycDe-9il2-Fy1C-2L28-o0g4-RMtJ-iCxpPQ

root@target:~# vgdisplay
  --- Volume group ---
  VG Name               target-vg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  8
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               139.49 GiB
  PE Size               4.00 MiB
  Total PE              35710
  Alloc PE / Size       32745 / 127.91 GiB
  Free  PE / Size       2965 / 11.58 GiB
  VG UUID               LyeTsz-ecd4-Ca2R-Puqe-6tgn-Z3Zp-uvemNH

root@target:~# lvdisplay
  --- Logical volume ---
  LV Path                /dev/target-vg/root
  LV Name                root
  VG Name                target-vg
  LV UUID                chaege-Iq3z-22PL-9X52-5poc-iFOz-i3qP8T
  LV Write Access        read/write
  LV Creation host, time target, 559220-01-17 05:58:44 +0100
  LV Status              available
  # open                 1
  LV Size                120.00 GiB
  Current LE             30720
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0

  --- Logical volume ---
  LV Path                /dev/target-vg/swap_1
  LV Name                swap_1
  VG Name                target-vg
  LV UUID                oRtjRZ-nz71-9U8g-vYrT-MR4A-XYTI-qz6Jwp
  LV Write Access        read/write
  LV Creation host, time target, 559220-01-17 05:58:44 +0100
  LV Status              available
  # open                 2
  LV Size                7.91 GiB
  Current LE             2025
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

root@target:~#

Let’s create a rollback point, i.e. 11G space from 11.58 GiB available in the VG.

root@target:~# lvcreate -L11G -s -n rootsnapshot /dev/target-vg/root
  Using default stripesize 64.00 KiB.
  Logical volume "rootsnapshot" created.
root@target:~#
root@target:~# lvdisplay
[...]
  --- Logical volume ---
  LV Path                /dev/target-vg/rootsnapshot
  LV Name                rootsnapshot
  VG Name                target-vg
  LV UUID                Ivx48R-yzAK-hO2z-ZtM7-m3pj-PMyF-mu4RaO
  LV Write Access        read/write
  LV Creation host, time target, 2018-03-14 10:44:35 +0100
  LV snapshot status     active destination for root
  LV Status              available
  # open                 0
  LV Size                120.00 GiB
  Current LE             30720
  COW-table size         11.00 GiB
  COW-table LE           2816
  Allocated to snapshot  0.00%
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:4
root@target:~#

Let’s mess it up, Seeing is believing

root@target:~# mkdir remove.me
root@target:~# cd remove.me/
root@target:~/remove.me# dd if=/dev/urandom of=./removeme.file
^C870385+0 records in
870385+0 records out
445637120 bytes (446 MB, 425 MiB) copied, 4.2948 s, 104 MB/s

root@target:~/remove.me# du -hs .
425M    .

root@target:~/remove.me#

Was 12 GiB enough?

root@target:~/remove.me# lvdisplay
[...]
--- Logical volume ---
  LV Path                /dev/target-vg/rootsnapshot
  LV Name                rootsnapshot
  VG Name                target-vg
  LV UUID                Ivx48R-yzAK-hO2z-ZtM7-m3pj-PMyF-mu4RaO
  LV Write Access        read/write
  LV Creation host, time target, 2018-03-14 10:44:35 +0100
  LV snapshot status     active destination for root
  LV Status              available
  # open                 0
  LV Size                120.00 GiB
  Current LE             30720
  COW-table size         11.00 GiB
  COW-table LE           2816
  Allocated to snapshot  3.79%
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:4
root@target:~#

3,79%, yeah. BTW,  how was the root filesystem before messing it up?

root@target:~/remove.me# mkdir -p /mnt/rootsnapshot
root@target:~/remove.me# mount /dev/target-vg/rootsnapshot /mnt/rootsnapshot/
root@target:~/remove.me# ls -la /mnt/rootsnapshot/root/ 
total 44
drwx------  5 root root 4096 Mar 14 10:22 .
drwxr-xr-x 24 root root 4096 Mar 13 16:47 ..
-rw-r--r--  1 root root  219 Mar 14 10:23 .bash_history
-rw-r--r--  1 root root  557 Jan 24 10:45 .bashrc
-rw-r--r--  1 root root  148 Aug 17  2015 .profile
-rw-r--r--  1 root root   75 Feb 21 08:26 .selected_editor
drwx------  2 root root 4096 Mar 14 08:58 .ssh
-rw-------  1 root root  899 Mar 14 10:04 .viminfo
root@target:~/remove.me# ls -la /root/ 
total 48
drwx------  6 root root 4096 Mar 14 10:47 .
drwxr-xr-x 24 root root 4096 Mar 13 16:47 ..
-rw-r--r--  1 root root  219 Mar 14 10:23 .bash_history
-rw-r--r--  1 root root  557 Jan 24 10:45 .bashrc
-rw-r--r--  1 root root  148 Aug 17  2015 .profile
drwxr-xr-x  2 root root 4096 Mar 14 10:47 remove.me
-rw-r--r--  1 root root   75 Feb 21 08:26 .selected_editor
drwx------  2 root root 4096 Mar 14 08:58 .ssh
-rw-------  1 root root  899 Mar 14 10:04 .viminfo
root@target:~/remove.me# cd && umount /mnt/rootsnapshot && reboot

OK, will the snapshot be still available after reboot?

kip@target:~$ 
kip@target:~$ sudo su -
root@target:~# lvs
  LV           VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root         target-vg owi-aos--- 120.00g
  rootsnapshot target-vg swi-a-s---  11.00g      root   0.01
  swap_1       target-vg -wi-ao----   7.91g

Yeah, cool. Let’s rollback.

root@target:~#
root@target:~# lvconvert --merge /dev/target-vg/rootsnapshot
  Can't merge until origin volume is closed.
  Merging of snapshot target-vg/rootsnapshot will occur on next activation of target-vg/root.
root@target:~# reboot

So now what?

kip@target:~$ 
kip@target:~$ sudo su -
root@target:~# ls -la
total 44
drwx------  5 root root 4096 Mar 14 10:22 .
drwxr-xr-x 24 root root 4096 Mar 13 16:47 ..
-rw-r--r--  1 root root  219 Mar 14 10:23 .bash_history
-rw-r--r--  1 root root  557 Jan 24 10:45 .bashrc
-rw-r--r--  1 root root  148 Aug 17  2015 .profile
-rw-r--r--  1 root root   75 Feb 21 08:26 .selected_editor
drwx------  2 root root 4096 Mar 14 08:58 .ssh
-rw-------  1 root root  899 Mar 14 10:04 .viminfo
root@target:~#

Cool, same stuff as before messing it up, btw snapshot volume had a vanishing act

kip@target:~$ 
kip@target:~$ sudo su -
root@target:~# lvs
  LV           VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root         target-vg owi-aos--- 120.00g
  swap_1       target-vg -wi-ao----   7.91g

Let’s do it again, same 11G for the snapshot.

root@target:~# lvcreate -L11G -s -n rootsnapshot /dev/target-vg/root
  Using default stripesize 64.00 KiB.
  Logical volume "rootsnapshot" created.
root@target:~#

But now let’s keep the snapshot (delta) into a file, just in case (store/distribute). There is a cool tool to do it (lvmsync), so let’s install it (this time we’ll not mess it up but provide the lvmsync installed in the snapshot delta file).

root@target:~# # Curl and other required packages install
root@target:~# apt-get install -y curl gnupg2 dirmngr
[...]
root@target:~# 
root@target:~# # Import the RVM's PK 
root@target:~# gpg2 --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 3804BB82D39DC0E3: public key "Michal Papis (RVM signing) <mpapis@gmail.com>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
root@target:~# 
root@target:~# 
root@target:~# curl -sSL https://get.rvm.io | bash -s stable
Downloading https://github.com/rvm/rvm/archive/1.29.3.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.3/1.29.3.tar.gz.asc
gpg: Signature made Sun 10 Sep 2017 10:59:21 PM CEST
gpg:                using RSA key E206C29FBF04FF17
gpg: Good signature from "Michal Papis (RVM signing) <mpapis@gmail.com>" [unknown]
gpg:                 aka "Michal Papis <michal.papis@toptal.com>" [unknown]
gpg:                 aka "[jpeg image of size 5015]" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 409B 6B17 96C2 7546 2A17  0311 3804 BB82 D39D C0E3
     Subkey fingerprint: 62C9 E5F4 DA30 0D94 AC36  166B E206 C29F BF04 FF17
GPG verified '/usr/local/rvm/archives/rvm-1.29.3.tgz'
Creating group 'rvm'

Installing RVM to /usr/local/rvm/
Installation of RVM in /usr/local/rvm/ is almost complete:

  * First you need to add all users that will be using rvm to 'rvm' group,
    and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`.

  * To start using RVM you need to run `source /etc/profile.d/rvm.sh`
    in all your open shell windows, in rare cases you need to reopen all shell windows.
root@target:~# 
root@target:~# 
root@target:~# # Load RVM environment variables 
root@target:~# source /etc/profile.d/rvm.sh
root@target:~# 
root@target:~# 
root@target:~# # Install Ruby's dependencies
root@target:~# rvm requirements
Checking requirements for debian.
Installing requirements for debian.
Updating system..
Installing required packages: libyaml-dev, libsqlite3-dev, sqlite3, libgmp-dev, libgdbm-dev, bison, libffi-dev, libgmp-dev, libreadline-dev.........
Requirements installation successful.
root@target:~# 
root@target:~# 
root@target:~# # Check Ruby versions available.
root@target:~# rvm list known
# MRI Rubies
[ruby-]1.8.6[-p420]
[ruby-]1.8.7[-head] # security released on head
[ruby-]1.9.1[-p431]
[ruby-]1.9.2[-p330]
[ruby-]1.9.3[-p551]
[ruby-]2.0.0[-p648]
[ruby-]2.1[.10]
[ruby-]2.2[.7]
[ruby-]2.3[.4]
[ruby-]2.4[.1]
ruby-head
[...]
root@target:~# 
root@target:~# 
root@target:~# # Install the Ruby 2.4.1 version
root@target:~# rvm install 2.4.1
Searching for binary rubies, this might take some time.
No binary rubies available for: debian/9/x86_64/ruby-2.4.1.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for debian.
Requirements installation successful.
Installing Ruby from source to: /usr/local/rvm/rubies/ruby-2.4.1, this may take a while depending on your cpu(s)...
ruby-2.4.1 - #downloading ruby-2.4.1, this may take a while depending on your connection...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.9M  100 11.9M    0     0  4634k      0  0:00:02  0:00:02 --:--:-- 4636k
ruby-2.4.1 - #extracting ruby-2.4.1 to /usr/local/rvm/src/ruby-2.4.1....
ruby-2.4.1 - #applying patch /usr/local/rvm/patches/ruby/2.4.1/random_c_using_NR_prefix.patch.
ruby-2.4.1 - #configuring..................................................................
ruby-2.4.1 - #post-configuration..
ruby-2.4.1 - #compiling................................................................................
ruby-2.4.1 - #installing...........
ruby-2.4.1 - #making binaries executable..
ruby-2.4.1 - #downloading rubygems-2.6.14
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  751k  100  751k    0     0  2294k      0 --:--:-- --:--:-- --:--:-- 2297k
No checksum for downloaded archive, recording checksum in user configuration.
ruby-2.4.1 - #extracting rubygems-2.6.14....
ruby-2.4.1 - #removing old rubygems.........
ruby-2.4.1 - #installing rubygems-2.6.14...........................
ruby-2.4.1 - #gemset created /usr/local/rvm/gems/ruby-2.4.1@global
ruby-2.4.1 - #importing gemset /usr/local/rvm/gemsets/global.gems...............................................
ruby-2.4.1 - #generating global wrappers........
ruby-2.4.1 - #gemset created /usr/local/rvm/gems/ruby-2.4.1
ruby-2.4.1 - #importing gemsetfile /usr/local/rvm/gemsets/default.gems evaluated to empty gem list
ruby-2.4.1 - #generating default wrappers........
ruby-2.4.1 - #adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
Install of ruby-2.4.1 - #complete
Ruby was built without documentation, to build it run: rvm docs generate-ri
root@target:~# 
root@target:~# 
root@target:~# # Set default ruby version to 2.4.1.
root@target:~# rvm use 2.4.1 --default
Using /usr/local/rvm/gems/ruby-2.4.1
root@target:~# 
root@target:~# 
root@target:~# # Install lvmsync & dependencies
root@target:~# gem install lvmsync
Fetching: git-version-bump-0.15.1.gem (100%)
Successfully installed git-version-bump-0.15.1
Fetching: polyglot-0.3.5.gem (100%)
Successfully installed polyglot-0.3.5
Fetching: treetop-1.6.9.gem (100%)
Successfully installed treetop-1.6.9
Fetching: lvmsync-3.3.2.gem (100%)
Successfully installed lvmsync-3.3.2
Parsing documentation for git-version-bump-0.15.1
Installing ri documentation for git-version-bump-0.15.1
Parsing documentation for polyglot-0.3.5
Installing ri documentation for polyglot-0.3.5
Parsing documentation for treetop-1.6.9
Installing ri documentation for treetop-1.6.9
Parsing documentation for lvmsync-3.3.2
Installing ri documentation for lvmsync-3.3.2
Done installing documentation for git-version-bump, polyglot, treetop, lvmsync after 1 seconds
4 gems installed
root@target:~# 
root@target:~# # We'are done 
root@target:~# which lvmsync
/usr/local/rvm/gems/ruby-2.4.1/bin/lvmsync
root@target:~#

Went smooth, let’s check, again…

root@target:~# lvdisplay
  --- Logical volume ---
  LV Path                /dev/target-vg/root
  LV Name                root
  VG Name                target-vg
  LV UUID                chaege-Iq3z-22PL-9X52-5poc-iFOz-i3qP8T
  LV Write Access        read/write
  LV Creation host, time target, 559220-01-17 05:58:44 +0100
  LV snapshot status     source of
                         rootsnapshot [active]
  LV Status              available
  # open                 1
  LV Size                120.00 GiB
  Current LE             30720
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

  --- Logical volume ---
  LV Path                /dev/target-vg/swap_1
  LV Name                swap_1
  VG Name                target-vg
  LV UUID                oRtjRZ-nz71-9U8g-vYrT-MR4A-XYTI-qz6Jwp
  LV Write Access        read/write
  LV Creation host, time target, 559220-01-17 05:58:44 +0100
  LV Status              available
  # open                 2
  LV Size                7.91 GiB
  Current LE             2025
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:4

  --- Logical volume ---
  LV Path                /dev/target-vg/rootsnapshot
  LV Name                rootsnapshot
  VG Name                target-vg
  LV UUID                Hp3zIF-l0fG-lZn5-WR2q-RbkW-uFqO-CbjoJK
  LV Write Access        read/write
  LV Creation host, time target, 2018-03-14 11:13:49 +0100
  LV snapshot status     active destination for root
  LV Status              available
  # open                 0
  LV Size                120.00 GiB
  Current LE             30720
  COW-table size         11.00 GiB
  COW-table LE           2816
  Allocated to snapshot  6.52%
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:3
root@target:~#

6.52% out of 11G for lvmsync, ok. Time to use lvmsync. Do reboot with your prefered live distro and install also lvmsync on it (or use your failback partition with lvmsync installed if you have it). I.o.w, don’t use lvmsync with the VG mounted or you will have to use e2fsck to cleanup a lil afterwards…

rescue@rescue:~$ 
rescue@rescue:~$ sudo su -
root@rescue:~# # Let's assume you already have an NTFS or EXT4 usb external disk
root@rescue:~# mkdir -p /mnt/externalusb &&  mount /dev/sdc1 /mnt/externalusb && cd /mnt/externalusb
root@rescue:/mnt/externalusb#
root@rescue:/mnt/externalusb# # Let's generate now the LVM delta and store it in the external USB volume
root@rescue:/mnt/externalusb# lvmsync --stdout /dev/target-vg/rootsnapshot > ./target-sda-lvm2vg-root-11G-v0.1.snap
Transferred 767098880 bytes in 19.94 seconds
You transferred your changes 167.97x faster than a full dd!
root@rescue:/mnt/externalusb# ls -laFh
total 734M
drwxrwxrwx 1 root root  192 Mar 14 11:31 ./
drwxrwxrwx 1 root root 4.0K Mar 14 11:29 ../
-rwxrwxrwx 1 root root 734M Mar 14 12:59 target-sda-lvm2vg-root-11G-v0.1.snap*
root@rescue:/mnt/externalusb# cd && umount /mnt/externalusb
root@rescue:~#

Notice it is using 734M out of 11G, that is the delta space. Indeed cool tool.

Let’s rollback now, we’ll merge it later the snapshot delta from the file and we want the baseline again to check if it works as expected.

root@rescue:/mnt/externalusb# # Let's merge and reboot
root@rescue:/mnt/externalusb# lvconvert --merge /dev/target-vg/rootsnapshot
  Merging of volume target-vg/rootsnapshot started.
  root: Merged: 99.99%
  root: Merged: 100.00%
root@rescue:/mnt/externalusb#
root@rescue:~# cd && umount /mnt/externalusb && reboot

So now being in the same situation as before installing lvmsync in the root partition. The good thing is we now have the delta snapshot to recover it in a safe place. So again, proceed to boot with the rescue and create the placeholder as you did it before.

rescue@rescue:~$ 
rescue@rescue:~$ sudo su -
root@rescue:~#
root@rescue:~# # Let's create the snapshot placeholder
root@rescue:~# lvcreate -L11G -s -n rootsnapshot /dev/target-vg/root
  Using default stripesize 64.00 KiB.
  Logical volume "rootsnapshot" created.
root@rescue:~#
root@rescue:~# # Let's assume you already have an NTFS or EXT4 usb external disk, please adjust the final path to your liking
root@rescue:~# mkdir -p /mnt/externalusb &&  mount /dev/sdc1 /mnt/externalusb && cd /mnt/externalusb
root@rescue:/mnt/externalusb#
root@rescue:/mnt/externalusb# # Let's flash the current snapshot with the provided one
root@rescue:/mnt/externalusb# lvmsync --apply ./target-sda-lvm2vg-root-11G-v0.1.snap /dev/target-vg/rootsnapshot
root@rescue:/mnt/externalusb#

Now you have the snapshot but this time containing the delta stream imported, not the baseline.

Let’s merge.

root@rescue:/mnt/externalusb# # Let's merge and reboot
root@rescue:/mnt/externalusb# lvconvert --merge /dev/target-vg/rootsnapshot
  Merging of volume target-vg/rootsnapshot started.
  root: Merged: 99.99%
  root: Merged: 100.00%
root@rescue:/mnt/externalusb#
root@rescue:~# cd && umount /mnt/externalusb && reboot

Check, lvmsync should be installed as you did it. That’s it.

Cheers,

JJ