direct update at dump session

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
Post Reply
User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#1 by abelM » 15 Nov 2017, 21:40

Hi all!

I'd like to show you an alternative way of saving files with dump-session as using the changes=EXIT: cheatcode. The idea is to copy files straightforward from aufs to your persistent mount -- avoiding the need of a temporary folder. This saves writes and memory and feels faster. Besides you can now use spaces in your `changes-exit.conf' and there some are other minor tweaks. Wildcards can also be used as before. I'm using it at present and it seems to work ok.

Please note
  • You can add lines like these to your changes-exit.conf file
    /home
    !/home/guest/.cache/moonchild productions
    !/home/guest/.serverauth.*
  • WDIR is not present. No need to copy files to a temporary folder.
  • find outputs files from wanted folders and grep -Ev excludes unwanted.
  • cp -uaf just updates files -- it doesn't make a bulk copy
  • If you create a folder without any file inside it will not be saved. New folders with at least one file will be saved.
In the end you should get less writes -- something I care about flash drives -- and more speed. If you think this is interesting please give me your opinion. Maybe it's useful or funny al least. :)

Thank you!

Code: Select all

#!/bin/bash
# /opt/porteus-scripts/dump-session
# Save live session ... 
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Author: fanthom <fanthom@porteus.org>
# Modification proposed by: abelM - forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

# Variables:
DEST=`cat /mnt/live/tmp/changes-exit`
NAME=`basename $DEST`
MNAME="/mnt/live/memory/images/changes"

# Configuration file
FOLDERS_CONF="/etc/changes-exit.conf"

# Save: build expresion for `find'
FOLDERS="$(grep '^/' $FOLDERS_CONF | sed 's/.//')"

# Exclude: build expresion for `grep -Ev'
XFOLDERS="$(grep '^!/' $FOLDERS_CONF | sed -e 's/../^/' \
  -e 's/\./\\./g' -e 's/\?/./g' -e 's/\*/.*/g' \
  | tr '\n' '|' | sed -e 's/|$//')"

# List of files to copy 
FILES_CP="/tmp/files_cp.lst"

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 
echo "saving changes to $DEST - this may take a while..."

cd /mnt/live/memory/changes

# HANDLE FILENAMES WITH SPACES: set IFS to EOL
IFS="
"

# Handle whiteouts properly: 
for x in $(find $FOLDERS -name ".wh.*"  | sed 's/\.wh\./g/'); do test -e "$MNAME/$x" && rm -rf "$MNAME/$x"; done
for y in $(find $MNAME -name ".wh.*"); do wh="$(echo $y | sed -e 's^$MNAME^^g' -e 's/\.wh\./g/')"; test -e "$wh" && rm "$y"; done

# Merge changes with 'changes=EXIT' destination:
# BUILD LIST OF FILES SKIPPING EXCLUDED FOLDERS
if test -n "$XFOLDERS"; then 
  find $FOLDERS -not -type d | grep -Ev "$XFOLDERS" > "$FILES_CP"
else 
  find $FOLDERS -not -type d > "$FILES_CP"
fi
# UPDATE FILES NOW
cp -uaf --parents `cat "$FILES_CP"` "$MNAME"

# Reset IFS to default 
unset IFS

# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none /

echo "session saved"


User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#2 by abelM » 21 Nov 2017, 01:47

Hi guys,

This post goes a bit further than the previous one. I have gathered different issues about dump-session and cleanup script and done a rewrite of dump-session to work them out. Cleanup runs in a different environment and I did not try it yet. I am testing this code by now and it seems to work ok. You will see numbers in brackets like [1] etc. to point some comments.

Declare variables at first

Code: Select all

#!/bin/bash
# Save live session ... 
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Author: fanthom <fanthom@porteus.org>
# Modifications proposed by: abelM -- forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

# Variables:
DEST=`cat /mnt/live/tmp/changes-exit`
NAME=`basename $DEST`
MNAME=/mnt/live/memory/images/changes

# Images path
INAME=/mnt/live/memory/images

# Configuration file
CHNEXIT_CONF="/etc/changes-exit.conf"

# Save folders: argument for `find'
FOLDERS="$(grep '^/' $CHNEXIT_CONF | sed 's/.//')"

# List of eXcluded whiteouts and folders: argument for `egrep -vf' --fix dots and wildcards
EXCL="/tmp/dump-session.excl.lst"
grep '^!/' $CHNEXIT_CONF | sed -e 's/../^/' \
  -e 's/\./\\./g' -e 's/\?/./g' -e 's/\*/.*/g' > "$EXCL"

# List of files to Save --update really--
SFILES="/tmp/dump-session.sfiles.lst"

Let's start

Code: Select all

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 
echo "saving changes to $DEST - this may take a while..."
cd /mnt/live/memory/changes

# Handle filenames with spaces: set IFS to EOL. [1]
IFS="
"
Handle whiteouts

Code: Select all

# Handling whiteouts properly: [2]
# Remove files deleted in the union system
# Whiteouts not matching any file will be excluded from saving
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's/\.wh\.//g')"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e $INAME/*/"$f" || echo "^$y" | sed 's/\./\\./g' >> "$EXCL"
done
Save changes

Code: Select all

# Merge changes with 'changes=EXIT' destination: [3]
# Save new or modified files from union to destination -- and skip excluded whiteouts and folders
if test -s "$EXCL"; then 
  find $FOLDERS  -not -type d | egrep -vf "$EXCL" > "$SFILES"
else 
  find $FOLDERS  -not -type d > "$SFILES"
fi

cp -uaf --parents $(cat "$SFILES") "$MNAME"
Handle whiteouts

Code: Select all

# Handling whiteouts properly: [4]
# Remove whiteouts in destination if they match a file there
for y in $(find $MNAME -name ".wh.*"); do 
  f="$(echo $y | sed -e 's^$MNAME^^g' -e 's/\.wh\.//g')"; 
  test -e "$f" && rm "$y"; 
done
Close everything

Code: Select all

# Reset IFS to default 
unset IFS

# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none /
echo "session saved"
Notes

[1] Setting bash internal field separator to newline makes the shell treat spaces as any other character. This solves the issue of files whith spaces in their names as processing with cp, rm, find and other commands.

[2] The first part of "Handling whiteouts properly" with a minor tweak: Whiteouts not matching any file are added to an exclusion list.

[3] Quite an important difference with the original dump-session. Only new or modified files will be copied directly to destination without using a temporary folder. It significantly reduces writes to your flash drive and runs faster too. Unwanted folders and not matching whiteouts are excluded from the copy in the same pipe.

[4] This other statement of "Handling whiteouts properly" has been moved after saving files to destination so it keeps destination clean in the same run.

Regards,
Last edited by abelM on 21 Nov 2017, 16:44, edited 5 times in total.

User avatar
brokenman
Site Admin
Site Admin
Posts: 5659
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

direct update at dump session

Post#3 by brokenman » 21 Nov 2017, 08:35

Great work. I will implement the change here and put it to the test.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#4 by abelM » 21 Nov 2017, 16:08

Great! I hope it helps :roll:

Regards,

User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#5 by abelM » 21 Nov 2017, 21:37

Hi brokenman,

There's something I've missed. Skipping non matching whiteouts as updating files is not enough: if they existed before you moved to the testing dump-session they will be kept in the persistence! This can be fixed by hand or --maybe better-- by adding just one command to dump-session. Please note test -e $INAME/*/"$f" ...

Code: Select all

# Handling whiteouts properly: [2]
# Remove files deleted in the union system
# Whiteouts not matching any file will be excluded from saving and removed from persistence
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's/\.wh\.//g')"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e $INAME/*/"$f" \
    || { echo "^$y" | sed 's/\./\\./g' >> "$EXCL"
         test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }
done
BTW [code] understands /* as start comment --like C language. You can cheat it with #*/ although it's not very elegant ;)

Regards!
Last edited by abelM on 27 Nov 2017, 14:39, edited 1 time in total.

User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#6 by abelM » 23 Nov 2017, 18:48

Hi all!

I'm leaving here the code in a single frame in case you want to give it a try. It's keeping my system updated and clean by now -- while it grabs roughly between 1% and 5% of the files and total size it used to. In case you're curious just read /tmp/dump-session.*.tmp files after a run. It's interesting to compare files candidate for saving --as defined in /etc/changes-exit.conf-- with actually updated files.

Please note that # fake */ is a trick to close /* colorizing here. You can safely remove it in your system.

Code: Select all

#!/bin/bash
# Save live session ...
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Author: fanthom <fanthom@porteus.org>
# Modifications proposed by: abelM -- forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

# Variables:
DEST=`cat /mnt/live/tmp/changes-exit`
NAME=`basename $DEST`

# Persistence
MNAME=/mnt/live/memory/images/changes

# Images
INAME=/mnt/live/memory/images

# Configuration file
CHNEXIT_CONF="/etc/changes-exit.conf"

# Save folders: argument for `find'
FOLDERS="$(grep '^/' $CHNEXIT_CONF | sed 's/.//')"

# Excluded whiteouts and folders: argument for `egrep -vf'
EXCL="/tmp/dump-session.excl.tmp"
grep '^!/' $CHNEXIT_CONF | sed -e 's/../^/' -e 's/\./\\./g' -e 's/\?/./g' -e 's/\*/.*/g' > "$EXCL"

# Files to Save
SFILES="/tmp/dump-session.sfiles.tmp"

# Updated files
UPDATE="/tmp/dump-session.update.tmp"

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 
echo "saving changes to $DEST - this may take a while..."
cd /mnt/live/memory/changes

# Handle filenames with spaces.
IFS="
"

# Remove deleted files and non matching whiteouts
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's/\.wh\.//g')"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e "$INAME"/*/"$f" \
    || { echo "^$y" | sed 's/\./\\./g' >> "$EXCL"
         test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }
done
# fake */
# Copy new and modified files -- skip excluded whiteouts and folders
if test -s "$EXCL"; then 
  find $FOLDERS  -not -type d | egrep -vf "$EXCL" > "$SFILES"
else 
  find $FOLDERS  -not -type d > "$SFILES"
fi
cp -uafv --parents $(cat "$SFILES") "$MNAME" > "$UPDATE"

# Remove conflicting whiteouts 
for y in $(find $MNAME -name ".wh.*"); do 
  f="$(echo $y | sed -e 's^$MNAME^^g' -e 's^\.wh\.^^g')"
  test -e "$f" && rm "$y"
done

# Reset IFS to default 
unset IFS

# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none / 

echo "session saved"
This script is running ok in the normal Porteus environment and in a Busybox 1.27 environment too -- so it could perhaps be useful to modify the next release of cleanup script :D

Regards,
Last edited by abelM on 26 Nov 2017, 12:25, edited 1 time in total.

User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#7 by abelM » 24 Nov 2017, 22:12

I've made a modification to the code posted at direct update at dump session (Post by abelM #60431)

Code: Select all

# Remove deleted files and non matching whiteouts
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's/\.wh\.//g')"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e "$INAME"/*/"$f" \
    || { echo "^$y" | sed 's/\./\\./g' >> "$EXCL"
#        test -e "$MNAME/$y" && rm -fr "$MNAME/$y"; }      # BEFORE 
         test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }       # NOW
done
I've removed the recursive flag from rm -fr because I don' see it's really necessary and I'm not sure it's harmless.

Regards,

User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#8 by abelM » 27 Nov 2017, 16:11

Hi all!
I've fixed `+' and `?' in some odd filenames

Code: Select all

#!/bin/bash
# Save live session ...
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Author: fanthom <fanthom@porteus.org>
# Modifications proposed by: abelM -- forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

# Variables:
DEST="$(cat /mnt/live/tmp/changes-exit)"
NAME="$(basename $DEST)"

# Persistence
MNAME="/mnt/live/memory/images/changes"

# Images
INAME="/mnt/live/memory/images"

# Configuration file
CHNEXIT_CONF="/etc/changes-exit.conf"

# Save folders: argument for `find'
FOLDERS="$(grep '^/' $CHNEXIT_CONF | sed 's/.//')"

# Excluded whiteouts and folders: argument for `egrep -vf'
EXCL="/tmp/dump-session.excl.tmp"
grep '^!/' $CHNEXIT_CONF | sed -e 's/../^/' -e 's/\./\\./g' -e 's/\?/./g' -e 's/\*/.*/g' > "$EXCL"

# Files to Save
SFILES="/tmp/dump-session.sfiles.tmp"

# Updated files
UPDATE="/tmp/dump-session.update.tmp"

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 
echo "saving changes to $DEST - this may take a while..."
cd /mnt/live/memory/changes

# Handle filenames with spaces.
IFS="
"
# Remove deleted files and non matching whiteouts
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's/\.wh\.//g')"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e "$INAME/"*"/$f" \
	|| { echo "^$y" | sed -e 's^\.^\\.^g' -e 's^\+^\\+^g' -e 's^\?^\\?^g' >> "$EXCL"
             test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }
done

# Copy new and modified files -- skip excluded whiteouts and folders
if test -s "$EXCL"; then 
  find $FOLDERS  -not -type d | egrep -vf "$EXCL" > "$SFILES"
else 
  find $FOLDERS  -not -type d > "$SFILES"
fi
cp -uafv --parents $(cat "$SFILES") "$MNAME" > "$UPDATE"

# Remove conflicting whiteouts 
for y in $(find $MNAME -name ".wh.*"); do 
  f="$(echo $y | sed -e 's^$MNAME^^g' -e 's^\.wh\.^^g')"
  test -e "$f" && rm "$y"
done

# Reset IFS to default 
unset IFS

# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none / 

echo "session saved"
Regards,

User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#9 by abelM » 29 Nov 2017, 22:43

Hi all!

Cached files often use special characters egrep interprets as regex so I've tried to collect and fix them. A couple of characters shouldn't be used in filenames so I didn't include them but it would be easy to if needed. Besides I've writen the code in a little more organized and compact way. I'm using this all the time and it doesn't seem to be any trouble. Please give me your opinion and tell me if you detect any issue.

Code: Select all

#!/bin/bash
# Save live session ...
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Author: fanthom <fanthom@porteus.org>
# Modifications proposed by: abelM -- forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

## Variables:
DEST="$(cat /mnt/live/tmp/changes-exit)"; NAME="$(basename $DEST)"
MNAME=/mnt/live/memory/images/changes   ; INAME=/mnt/live/memory/images
CHNEXIT_CONF=/etc/changes-exit.conf     ; EXCL=/tmp/save.excl.tmp
SFILES=/tmp/save.sfiles.tmp             ; UPDATE=/tmp/save.update.tmp

# Fix folder names and make `egrep' interpret wildcards
fixdir() { sed -e 's_\._\\._g' -e 's_?_._g' -e 's_*_.*_g' -e 's@..@^@'; }

# Fix whiteout names so `egrep' doesn't interpret special characters as regex
fixwh() { sed -e 's_\._\\._g' -e 's_?_\\?_g' -e 's_+_\\+_g' -e 's_\$_\\$_g' -e 's_\[_\\[_g' -e 's_\&_\\&_g' -e 's@^@^@'; }

# Folders to save: argument for `find'
FOLDERS="$(grep ^/ $CHNEXIT_CONF | sed s/.//)"

# Folders to exclude: argument for `egrep'
grep ^!/ $CHNEXIT_CONF | fixdir > $EXCL 

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 
echo "saving changes to $DEST - this may take a while..."
cd /mnt/live/memory/changes

# Handle filenames with spaces.
IFS="
"

# Remove deleted files and non matching whiteouts
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's@\.wh\.@@g')" 
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e "$INAME/"*"/$f" || { echo "$y" | fixwh >> $EXCL; test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }
done

# Copy new and modified files -- skip excluded whiteouts and folders
{ test -s "$EXCL" && find $FOLDERS -not -type d | egrep -vf $EXCL || find $FOLDERS -not -type d; } > $SFILES
cp -uafv --parents $(cat $SFILES) $MNAME > $UPDATE

# Remove conflicting whiteouts 
for y in $(find $MNAME -name ".wh.*"); do 
	f="$(echo $y | sed -e 's^$MNAME^^g' -e 's^\.wh\.^^g')"; 
	test -e "$f" && rm "$y"; 
done

# Reset IFS to default 
unset IFS
# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none / 
echo "session saved"

Regards!

User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#10 by abelM » 03 Dec 2017, 19:33

Hi all!

I have fixed opening parenthesis in whiteout names. I haven't found stars in filenames -that would be a bad idea- but I think it's better to fix that too. So please change fixwh function in your testing dump-session like this:

Code: Select all

# Fix whiteout names so `egrep' doesn't interpret special characters as regex
fixwh() { sed -e 's_\._\\._g' -e 's_?_\\?_g' -e 's_+_\\+_g' -e '_*_\\*_g' -e 's_\$_\\$_g' -e 's_\[_\\[_g' -e 's_(_\\(_g'  -e 's_\&_\\&_g' -e 's@^@^@'; }
Regards,

User avatar
brokenman
Site Admin
Site Admin
Posts: 5659
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

direct update at dump session

Post#11 by brokenman » 03 Dec 2017, 19:49

Updated in my TREE. Thanks. It seems stable to me after my testing. I have added it for inclusion to the next rc1 version.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#12 by abelM » 04 Dec 2017, 16:30

Great! Please remember this code is compatible with Busybox 1.27.2 so it could be eventually be included in cleanup.

Regards,

User avatar
abelM
Black ninja
Black ninja
Posts: 49
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

direct update at dump session

Post#13 by abelM » 13 Dec 2017, 15:43

Hi all, I've fixed vertical bar and written a more concise expression in fixwh function. It keeps Busybox 1.27.1 compatibility.

Code: Select all

fixwh() { sed -e 's_[.?*+|&^$([]_\\&_g' -e 's@^@^@'; }
Regards!

Post Reply