Page 1 of 1

direct update at dump session

Posted: 15 Nov 2017, 21:40
by abelM
Hi all!

I'd like to show you an alternative way of saving files with dump-session as using the changes=EXIT: cheatcode. The idea is to copy files straightforward from aufs to your persistent mount -- avoiding the need of a temporary folder. This saves writes and memory and feels faster. Besides you can now use spaces in your `changes-exit.conf' and there some are other minor tweaks. Wildcards can also be used as before. I'm using it at present and it seems to work ok.

Please note
  • You can add lines like these to your changes-exit.conf file
    /home
    !/home/guest/.cache/moonchild productions
    !/home/guest/.serverauth.*
  • WDIR is not present. No need to copy files to a temporary folder.
  • find outputs files from wanted folders and grep -Ev excludes unwanted.
  • cp -uaf just updates files -- it doesn't make a bulk copy
  • If you create a folder without any file inside it will not be saved. New folders with at least one file will be saved.
In the end you should get less writes -- something I care about flash drives -- and more speed. If you think this is interesting please give me your opinion. Maybe it's useful or funny al least. :)

Thank you!

Code: Select all

#!/bin/bash
# /opt/porteus-scripts/dump-session
# Save live session ... 
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Author: fanthom <fanthom@porteus.org>
# Modification proposed by: abelM - forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

# Variables:
DEST=`cat /mnt/live/tmp/changes-exit`
NAME=`basename $DEST`
MNAME="/mnt/live/memory/images/changes"

# Configuration file
FOLDERS_CONF="/etc/changes-exit.conf"

# Save: build expresion for `find'
FOLDERS="$(grep '^/' $FOLDERS_CONF | sed 's/.//')"

# Exclude: build expresion for `grep -Ev'
XFOLDERS="$(grep '^!/' $FOLDERS_CONF | sed -e 's/../^/' \
  -e 's/\./\\./g' -e 's/\?/./g' -e 's/\*/.*/g' \
  | tr '\n' '|' | sed -e 's/|$//')"

# List of files to copy 
FILES_CP="/tmp/files_cp.lst"

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 
echo "saving changes to $DEST - this may take a while..."

cd /mnt/live/memory/changes

# HANDLE FILENAMES WITH SPACES: set IFS to EOL
IFS="
"

# Handle whiteouts properly: 
for x in $(find $FOLDERS -name ".wh.*"  | sed 's/\.wh\./g/'); do test -e "$MNAME/$x" && rm -rf "$MNAME/$x"; done
for y in $(find $MNAME -name ".wh.*"); do wh="$(echo $y | sed -e 's^$MNAME^^g' -e 's/\.wh\./g/')"; test -e "$wh" && rm "$y"; done

# Merge changes with 'changes=EXIT' destination:
# BUILD LIST OF FILES SKIPPING EXCLUDED FOLDERS
if test -n "$XFOLDERS"; then 
  find $FOLDERS -not -type d | grep -Ev "$XFOLDERS" > "$FILES_CP"
else 
  find $FOLDERS -not -type d > "$FILES_CP"
fi
# UPDATE FILES NOW
cp -uaf --parents `cat "$FILES_CP"` "$MNAME"

# Reset IFS to default 
unset IFS

# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none /

echo "session saved"


direct update at dump session

Posted: 21 Nov 2017, 01:47
by abelM
Hi guys,

This post goes a bit further than the previous one. I have gathered different issues about dump-session and cleanup script and done a rewrite of dump-session to work them out. Cleanup runs in a different environment and I did not try it yet. I am testing this code by now and it seems to work ok. You will see numbers in brackets like [1] etc. to point some comments.

Declare variables at first

Code: Select all

#!/bin/bash
# Save live session ... 
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Author: fanthom <fanthom@porteus.org>
# Modifications proposed by: abelM -- forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

# Variables:
DEST=`cat /mnt/live/tmp/changes-exit`
NAME=`basename $DEST`
MNAME=/mnt/live/memory/images/changes

# Images path
INAME=/mnt/live/memory/images

# Configuration file
CHNEXIT_CONF="/etc/changes-exit.conf"

# Save folders: argument for `find'
FOLDERS="$(grep '^/' $CHNEXIT_CONF | sed 's/.//')"

# List of eXcluded whiteouts and folders: argument for `egrep -vf' --fix dots and wildcards
EXCL="/tmp/dump-session.excl.lst"
grep '^!/' $CHNEXIT_CONF | sed -e 's/../^/' \
  -e 's/\./\\./g' -e 's/\?/./g' -e 's/\*/.*/g' > "$EXCL"

# List of files to Save --update really--
SFILES="/tmp/dump-session.sfiles.lst"

Let's start

Code: Select all

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 
echo "saving changes to $DEST - this may take a while..."
cd /mnt/live/memory/changes

# Handle filenames with spaces: set IFS to EOL. [1]
IFS="
"
Handle whiteouts

Code: Select all

# Handling whiteouts properly: [2]
# Remove files deleted in the union system
# Whiteouts not matching any file will be excluded from saving
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's/\.wh\.//g')"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e $INAME/*/"$f" || echo "^$y" | sed 's/\./\\./g' >> "$EXCL"
done
Save changes

Code: Select all

# Merge changes with 'changes=EXIT' destination: [3]
# Save new or modified files from union to destination -- and skip excluded whiteouts and folders
if test -s "$EXCL"; then 
  find $FOLDERS  -not -type d | egrep -vf "$EXCL" > "$SFILES"
else 
  find $FOLDERS  -not -type d > "$SFILES"
fi

cp -uaf --parents $(cat "$SFILES") "$MNAME"
Handle whiteouts

Code: Select all

# Handling whiteouts properly: [4]
# Remove whiteouts in destination if they match a file there
for y in $(find $MNAME -name ".wh.*"); do 
  f="$(echo $y | sed -e 's^$MNAME^^g' -e 's/\.wh\.//g')"; 
  test -e "$f" && rm "$y"; 
done
Close everything

Code: Select all

# Reset IFS to default 
unset IFS

# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none /
echo "session saved"
Notes

[1] Setting bash internal field separator to newline makes the shell treat spaces as any other character. This solves the issue of files whith spaces in their names as processing with cp, rm, find and other commands.

[2] The first part of "Handling whiteouts properly" with a minor tweak: Whiteouts not matching any file are added to an exclusion list.

[3] Quite an important difference with the original dump-session. Only new or modified files will be copied directly to destination without using a temporary folder. It significantly reduces writes to your flash drive and runs faster too. Unwanted folders and not matching whiteouts are excluded from the copy in the same pipe.

[4] This other statement of "Handling whiteouts properly" has been moved after saving files to destination so it keeps destination clean in the same run.

Regards,

direct update at dump session

Posted: 21 Nov 2017, 08:35
by brokenman
Great work. I will implement the change here and put it to the test.

direct update at dump session

Posted: 21 Nov 2017, 16:08
by abelM
Great! I hope it helps :roll:

Regards,

direct update at dump session

Posted: 21 Nov 2017, 21:37
by abelM
Hi brokenman,

There's something I've missed. Skipping non matching whiteouts as updating files is not enough: if they existed before you moved to the testing dump-session they will be kept in the persistence! This can be fixed by hand or --maybe better-- by adding just one command to dump-session. Please note test -e $INAME/*/"$f" ...

Code: Select all

# Handling whiteouts properly: [2]
# Remove files deleted in the union system
# Whiteouts not matching any file will be excluded from saving and removed from persistence
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's/\.wh\.//g')"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e $INAME/*/"$f" \
    || { echo "^$y" | sed 's/\./\\./g' >> "$EXCL"
         test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }
done
BTW [code] understands /* as start comment --like C language. You can cheat it with #*/ although it's not very elegant ;)

Regards!

direct update at dump session

Posted: 23 Nov 2017, 18:48
by abelM
Hi all!

I'm leaving here the code in a single frame in case you want to give it a try. It's keeping my system updated and clean by now -- while it grabs roughly between 1% and 5% of the files and total size it used to. In case you're curious just read /tmp/dump-session.*.tmp files after a run. It's interesting to compare files candidate for saving --as defined in /etc/changes-exit.conf-- with actually updated files.

Please note that # fake */ is a trick to close /* colorizing here. You can safely remove it in your system.

Code: Select all

#!/bin/bash
# Save live session ...
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Author: fanthom <fanthom@porteus.org>
# Modifications proposed by: abelM -- forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

# Variables:
DEST=`cat /mnt/live/tmp/changes-exit`
NAME=`basename $DEST`

# Persistence
MNAME=/mnt/live/memory/images/changes

# Images
INAME=/mnt/live/memory/images

# Configuration file
CHNEXIT_CONF="/etc/changes-exit.conf"

# Save folders: argument for `find'
FOLDERS="$(grep '^/' $CHNEXIT_CONF | sed 's/.//')"

# Excluded whiteouts and folders: argument for `egrep -vf'
EXCL="/tmp/dump-session.excl.tmp"
grep '^!/' $CHNEXIT_CONF | sed -e 's/../^/' -e 's/\./\\./g' -e 's/\?/./g' -e 's/\*/.*/g' > "$EXCL"

# Files to Save
SFILES="/tmp/dump-session.sfiles.tmp"

# Updated files
UPDATE="/tmp/dump-session.update.tmp"

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 
echo "saving changes to $DEST - this may take a while..."
cd /mnt/live/memory/changes

# Handle filenames with spaces.
IFS="
"

# Remove deleted files and non matching whiteouts
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's/\.wh\.//g')"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e "$INAME"/*/"$f" \
    || { echo "^$y" | sed 's/\./\\./g' >> "$EXCL"
         test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }
done
# fake */
# Copy new and modified files -- skip excluded whiteouts and folders
if test -s "$EXCL"; then 
  find $FOLDERS  -not -type d | egrep -vf "$EXCL" > "$SFILES"
else 
  find $FOLDERS  -not -type d > "$SFILES"
fi
cp -uafv --parents $(cat "$SFILES") "$MNAME" > "$UPDATE"

# Remove conflicting whiteouts 
for y in $(find $MNAME -name ".wh.*"); do 
  f="$(echo $y | sed -e 's^$MNAME^^g' -e 's^\.wh\.^^g')"
  test -e "$f" && rm "$y"
done

# Reset IFS to default 
unset IFS

# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none / 

echo "session saved"
This script is running ok in the normal Porteus environment and in a Busybox 1.27 environment too -- so it could perhaps be useful to modify the next release of cleanup script :D

Regards,

direct update at dump session

Posted: 24 Nov 2017, 22:12
by abelM
I've made a modification to the code posted at direct update at dump session (Post by abelM #60431)

Code: Select all

# Remove deleted files and non matching whiteouts
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's/\.wh\.//g')"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e "$INAME"/*/"$f" \
    || { echo "^$y" | sed 's/\./\\./g' >> "$EXCL"
#        test -e "$MNAME/$y" && rm -fr "$MNAME/$y"; }      # BEFORE 
         test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }       # NOW
done
I've removed the recursive flag from rm -fr because I don' see it's really necessary and I'm not sure it's harmless.

Regards,

direct update at dump session

Posted: 27 Nov 2017, 16:11
by abelM
Hi all!
I've fixed `+' and `?' in some odd filenames

Code: Select all

#!/bin/bash
# Save live session ...
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Author: fanthom <fanthom@porteus.org>
# Modifications proposed by: abelM -- forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

# Variables:
DEST="$(cat /mnt/live/tmp/changes-exit)"
NAME="$(basename $DEST)"

# Persistence
MNAME="/mnt/live/memory/images/changes"

# Images
INAME="/mnt/live/memory/images"

# Configuration file
CHNEXIT_CONF="/etc/changes-exit.conf"

# Save folders: argument for `find'
FOLDERS="$(grep '^/' $CHNEXIT_CONF | sed 's/.//')"

# Excluded whiteouts and folders: argument for `egrep -vf'
EXCL="/tmp/dump-session.excl.tmp"
grep '^!/' $CHNEXIT_CONF | sed -e 's/../^/' -e 's/\./\\./g' -e 's/\?/./g' -e 's/\*/.*/g' > "$EXCL"

# Files to Save
SFILES="/tmp/dump-session.sfiles.tmp"

# Updated files
UPDATE="/tmp/dump-session.update.tmp"

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 
echo "saving changes to $DEST - this may take a while..."
cd /mnt/live/memory/changes

# Handle filenames with spaces.
IFS="
"
# Remove deleted files and non matching whiteouts
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's/\.wh\.//g')"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e "$INAME/"*"/$f" \
	|| { echo "^$y" | sed -e 's^\.^\\.^g' -e 's^\+^\\+^g' -e 's^\?^\\?^g' >> "$EXCL"
             test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }
done

# Copy new and modified files -- skip excluded whiteouts and folders
if test -s "$EXCL"; then 
  find $FOLDERS  -not -type d | egrep -vf "$EXCL" > "$SFILES"
else 
  find $FOLDERS  -not -type d > "$SFILES"
fi
cp -uafv --parents $(cat "$SFILES") "$MNAME" > "$UPDATE"

# Remove conflicting whiteouts 
for y in $(find $MNAME -name ".wh.*"); do 
  f="$(echo $y | sed -e 's^$MNAME^^g' -e 's^\.wh\.^^g')"
  test -e "$f" && rm "$y"
done

# Reset IFS to default 
unset IFS

# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none / 

echo "session saved"
Regards,

direct update at dump session

Posted: 29 Nov 2017, 22:43
by abelM
Hi all!

Cached files often use special characters egrep interprets as regex so I've tried to collect and fix them. A couple of characters shouldn't be used in filenames so I didn't include them but it would be easy to if needed. Besides I've writen the code in a little more organized and compact way. I'm using this all the time and it doesn't seem to be any trouble. Please give me your opinion and tell me if you detect any issue.

Code: Select all

#!/bin/bash
# Save live session ...
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Author: fanthom <fanthom@porteus.org>
# Modifications proposed by: abelM -- forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

## Variables:
DEST="$(cat /mnt/live/tmp/changes-exit)"; NAME="$(basename $DEST)"
MNAME=/mnt/live/memory/images/changes   ; INAME=/mnt/live/memory/images
CHNEXIT_CONF=/etc/changes-exit.conf     ; EXCL=/tmp/save.excl.tmp
SFILES=/tmp/save.sfiles.tmp             ; UPDATE=/tmp/save.update.tmp

# Fix folder names and make `egrep' interpret wildcards
fixdir() { sed -e 's_\._\\._g' -e 's_?_._g' -e 's_*_.*_g' -e 's@..@^@'; }

# Fix whiteout names so `egrep' doesn't interpret special characters as regex
fixwh() { sed -e 's_\._\\._g' -e 's_?_\\?_g' -e 's_+_\\+_g' -e 's_\$_\\$_g' -e 's_\[_\\[_g' -e 's_\&_\\&_g' -e 's@^@^@'; }

# Folders to save: argument for `find'
FOLDERS="$(grep ^/ $CHNEXIT_CONF | sed s/.//)"

# Folders to exclude: argument for `egrep'
grep ^!/ $CHNEXIT_CONF | fixdir > $EXCL 

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 
echo "saving changes to $DEST - this may take a while..."
cd /mnt/live/memory/changes

# Handle filenames with spaces.
IFS="
"

# Remove deleted files and non matching whiteouts
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="$(echo $y | sed 's@\.wh\.@@g')" 
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e "$INAME/"*"/$f" || { echo "$y" | fixwh >> $EXCL; test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }
done

# Copy new and modified files -- skip excluded whiteouts and folders
{ test -s "$EXCL" && find $FOLDERS -not -type d | egrep -vf $EXCL || find $FOLDERS -not -type d; } > $SFILES
cp -uafv --parents $(cat $SFILES) $MNAME > $UPDATE

# Remove conflicting whiteouts 
for y in $(find $MNAME -name ".wh.*"); do 
	f="$(echo $y | sed -e 's^$MNAME^^g' -e 's^\.wh\.^^g')"; 
	test -e "$f" && rm "$y"; 
done

# Reset IFS to default 
unset IFS
# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none / 
echo "session saved"

Regards!

direct update at dump session

Posted: 03 Dec 2017, 19:33
by abelM
Hi all!

I have fixed opening parenthesis in whiteout names. I haven't found stars in filenames -that would be a bad idea- but I think it's better to fix that too. So please change fixwh function in your testing dump-session like this:

Code: Select all

# Fix whiteout names so `egrep' doesn't interpret special characters as regex
fixwh() { sed -e 's_\._\\._g' -e 's_?_\\?_g' -e 's_+_\\+_g' -e '_*_\\*_g' -e 's_\$_\\$_g' -e 's_\[_\\[_g' -e 's_(_\\(_g'  -e 's_\&_\\&_g' -e 's@^@^@'; }
Regards,

direct update at dump session

Posted: 03 Dec 2017, 19:49
by brokenman
Updated in my TREE. Thanks. It seems stable to me after my testing. I have added it for inclusion to the next rc1 version.

direct update at dump session

Posted: 04 Dec 2017, 16:30
by abelM
Great! Please remember this code is compatible with Busybox 1.27.2 so it could be eventually be included in cleanup.

Regards,

direct update at dump session

Posted: 13 Dec 2017, 15:43
by abelM
Hi all, I've fixed vertical bar and written a more concise expression in fixwh function. It keeps Busybox 1.27.1 compatibility.

Code: Select all

fixwh() { sed -e 's_[.?*+|&^$([]_\\&_g' -e 's@^@^@'; }
Regards!

direct update at dump session

Posted: 19 Dec 2017, 18:33
by abelM
Hi all

I'm leaving here the whole dump-session stuff with minor changes. It works exactly the same -I've just removed an unused variable inherited from the original script, renamed another two, and tried to make the code a bit more compact. It keeps compatibility with Busybox 1.27.x so it could eventually be useful in cleanup. I'm using it daily and haven't detected any issue. Please tell me if you find any!

Code: Select all

#!/bin/bash
# Save live session ...
# Script takes action only when 'changes=EXIT' cheatcode is used.
# /tmp/save.*.tmp files show how sessions are dumped
# Author: fanthom <fanthom@porteus.org>
# Modified by: abelM / forum.porteus.org

# Check if 'changes=EXIT' cheatcode is active:
test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }

## Variables
DEST="$(cat /mnt/live/tmp/changes-exit)"; 
MNAME=/mnt/live/memory/images/changes   ; INAME=/mnt/live/memory/images
CHNEXIT_CONF=/etc/changes-exit.conf     ; EXCLUDE=/tmp/save.exclude.tmp
FILELIST=/tmp/save.filelist.tmp         ; UPDATE=/tmp/save.update.tmp

# Fix folder names and make `egrep' interpret wildcards
fixdir() { sed 's_\._\\._g; s_?_._g; s_*_.*_g; s@..@^@'; }

# Fix whiteout names so `egrep' doesn't interpret special characters as regex
fixwh() { sed 's_[.?*+|&^$([]_\\&_g; s_^_^_'; }

# Folders to save: argument for `find'
FOLDERS="$(grep ^/ $CHNEXIT_CONF | sed s/.//)"

# Folders to exclude: argument for `egrep'
grep ^!/ $CHNEXIT_CONF | fixdir > $EXCLUDE 

## Save session:
echo "syncing what's left in buffers"
sync

# Remount aufs with 'udba=notify' flag: 
mount -o remount,udba=notify / 

echo "saving changes to $DEST - this may take a while..."
cd /mnt/live/memory/changes

# Handle filenames with spaces.
IFS="
"

# Remove deleted files and exclude non matching whiteouts from saving 
for y in $(find $FOLDERS -name ".wh.*"); do 
  f="${y/.wh.}"
  test -e "$MNAME/$f" && rm -rf "$MNAME/$f"; 
  test -e "$INAME/"*"/$f" || { echo "$y" | fixwh >> $EXCLUDE; test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }
done

# Copy new and modified files -- skip excluded whiteouts and folders
{ test -s "$EXCLUDE" && find $FOLDERS -not -type d | egrep -vf $EXCLUDE \
	|| find $FOLDERS -not -type d; } > $FILELIST
cp -uafv --parents $(cat $FILELIST) $MNAME > $UPDATE

# Remove conflicting whiteouts 
for y in $(find $MNAME -name ".wh.*"); do 
	f="$(echo $y | sed 's^$MNAME^^g; s^\.wh\.^^g')"; 
	test -e "$f" && rm "$y"; 
done

# Reset IFS to default 
unset IFS

# Remount aufs with 'udba=none' flag: 
mount -o remount,udba=none / 

echo "session saved"