dump-session: too many error messages

[rych]

including the 'find' ones?

Yes. Also, playing with your modified line, and reducing the conf file to one line:

Code: Select all

root@porteus:~# cat /etc/changes-exit.conf 
/bin

, AND running the dump-session 2nd time, the /tmp/save.sfiles.tmp and /tmp/save.update.tmp exploded and a lot more junk was let through to be saved to /changes. I had to clean it manually.

That's disturbing. When only /bin is specified, I don't expect other folders to be saved -- but they were. Perhaps, the script is not re-entrant?

For now I reverted back to

Code: Select all

FOLDERS="$(grep ^/ $CHNEXIT_CONF | sed s/.//)"

[rych]

a typo in line 20:

Oh, thank you! The "sed: -e expression #4" errors have disappeared. How was it even working before?! Anyway, now it looks like this:

Code: Select all

root@porteus:~# /opt/porteus-scripts/dump-session
syncing what's left in buffers
saving changes to /mnt/sdb3//porteus - this may take a while...
find: ‘bin’: No such file or directory
find: ‘lib’: No such file or directory
find: ‘lib64’: No such file or directory
find: ‘sbin’: No such file or directory
find: ‘bin’: No such file or directory
find: ‘lib’: No such file or directory
find: ‘lib64’: No such file or directory
find: ‘sbin’: No such file or directory
session saved

AND the same output 2nd time, AND no new junk in the /changes! Nice.

Now, adding your earlier fix

change line 23:

further reduces the number of errors:

Code: Select all

root@porteus:~# /opt/porteus-scripts/dump-session
syncing what's left in buffers
saving changes to /mnt/sdb3//porteus - this may take a while...
find: ‘lib’: No such file or directory
find: ‘sbin’: No such file or directory
find: ‘lib’: No such file or directory
find: ‘sbin’: No such file or directory
session saved

(same output on he 2nd run, still no unwanted junk in /changes -- good)

If only we could not see the remaining double "find:" errors. My /memory/changes don't even have those (just like it didn't have /bin etc.):

Code: Select all

root@porteus:~# ls -a /mnt/live/memory/changes
./   .cache/       .wh..wh.orph/  dev/  home/  opt/   run/  usr/
../  .wh..wh.aufs  .wh..wh.plnk/  etc/  mnt/   root/  tmp/  var/

Also, perhaps the new script properly fixed by ncmprhnsbl deserves a name change along with its config file, for example:

dump-session --> commit-changes
changes-exit.conf -- > commit-changes.conf

[Ed_P]

looking again, i see a typo in line 20:

So obvious, now that you have pointed it out.

and did you use the suggested edit

I did not. I assumed the error was file related rather than script related, since it has been around for so long, and the nonexistent folder msgs are superficial and don't impact the desired outcome.

My final version has only 2 line changes:
line 5

Code: Select all

# Modifications proposed by: abelM -- forum.porteus.org & Fixed by ncmprhnsbl -- forum.porteus.org admin 

line 20
your edit to the #4 sed.

Thank you ncmp.

The things that concern me in my changes folder are the .wh. folders. Why do I have 3 deleted folders?

Code: Select all

guest@porteus:~$ ls -a /mnt/live/memory/changes
./    ../    .cache/  .wh..wh.aufs  .wh..wh.orph/  .wh..wh.plnk/  dev/  etc/  home/
mnt/  root/  run/     tmp/          usr/           var/
guest@porteus:~$ 

Will have to run my removewhfiles script and see what happens.

Added in 53 minutes 15 seconds:
It didn't help.

Code: Select all

guest@porteus:~$ ls -a /mnt/live/memory/changes
./    ../    .cache/  .wh..wh.aufs  .wh..wh.orph/  .wh..wh.plnk/  dev/  etc/  home/
mnt/  root/  run/     tmp/          usr/           var/
guest@porteus:~$ 

Added in 16 minutes 20 seconds:
But they are empty, so not going to worry about them too much.

Code: Select all

guest@porteus:~$ ls -a /mnt/live/memory/changes/.wh..wh.*
/mnt/live/memory/changes/.wh..wh.aufs

/bin/ls: cannot open directory '/mnt/live/memory/changes/.wh..wh.orph': Permission denied
/bin/ls: cannot open directory '/mnt/live/memory/changes/.wh..wh.plnk': Permission denied
guest@porteus:~$ su
Password: 
root@porteus:/home/guest# ls -aR /mnt/live/memory/changes/.wh..wh.*
/mnt/live/memory/changes/.wh..wh.aufs

/mnt/live/memory/changes/.wh..wh.orph:
./  ../

/mnt/live/memory/changes/.wh..wh.plnk:
./  ../
root@porteus:/home/guest# 

Added in 2 hours 52 minutes 54 seconds:

Also, perhaps the new script properly fixed by ncmprhnsbl deserves a name change

You can save a copy of the fixed script in your /home/guest/ folder and name it my-changes-commit.

[rych]

You can save a copy of the fixed script in your /home/guest/

Thanks for that insight, about the guest . Seriously though it's a porteus fundamental system script and once it's fixed it'd better be given a better name too. Moreover, the only place its logic is essentially used, is in the initrd>cleanup:

Code: Select all

# Save 'changes=EXIT:' session:
if [ -e /tmp/changes-exit ]; then
    echo -e "Your session will be saved in [1;33m3[0m seconds.\nPress space/enter to start doing it now or any other key to skip."; x=3
    while [ $x -gt 0 ]; do read -s -t1 -n1 ans && break || sleep 1; let x=x-1; done
    if [ "$ans" = "" ]; then
	DEST=`cat /tmp/changes-exit`; NAME=`basename $DEST`; MNAME=/memory/images/changes; FOLDERS=`grep '^/' /union/etc/changes-exit.conf | sed s/^.//g`
	echo "saving changes to $NAME - do not power off the PC"
	cd /memory/changes; rm -rf var/lock/subsys/* var/run/laptop-mode-tools/* `grep '^!' /union/etc/changes-exit.conf | sed s/^..//g | tr "\n" " "`
	for x in `find $FOLDERS -name ".wh.*" 2>/dev/null | sed s/.wh.//g | tr ' ' '@'`; do x=`echo $x | tr '@' ' ' `; test -e $MNAME/"$x" && rm -rf $MNAME/"$x"; done
	for x in `find $MNAME -name ".wh.*" 2>/dev/null | tr ' ' '@'`; do x=`echo $x | tr '@' ' ' `; wh=`echo $x | sed -e s^$MNAME^^g -e s/.wh.//g`; test -e "$wh" && rm "$x"; done
	cp -afu --parents $FOLDERS $MNAME 2>/dev/null
    fi

.. and it'd better be as identical as possible. Thus (duplicated inlined) commit-changes code is part of initrd too, executed at the shutdown event. However, the user can choose to manually commit-changes before shutdown and it should result in virtually same /changes state as at the shutdown.

ncmprhnsbl, do you know why the initrd>cleanup doesn't invoke this script by name, like it does with accessing /union/etc/changes-exit.conf? (Under a proposed new naming and location convention it'd be:

/union/etc/commit-changes
/union/etc/commit-changes.conf

... but that's up to you of course)

[ncmprhnsbl]

ok, i think i had the existence test pointing at the wrong place, this should be correct:

Code: Select all

FOLDERS="$(for x in $(grep ^/ $CHNEXIT_CONF | sed s/.//); do test -e /mnt/live/memory/changes/$x && echo $x; done)"

or maybe even, since the script has cd'd to /mnt/live/memory/changes at the point when $FOLDERS is called:

Code: Select all

FOLDERS="$(for x in $(grep ^/ $CHNEXIT_CONF | sed s/.//); do test -e $x && echo $x; done)"

but i don't suppose the absolute path hurts.

do you know why the initrd>cleanup doesn't invoke this script by name

all before my time.
dump-session seems to do more and differently, with egrep, IFS word-splitting contingency and elaborate sed instead of some tr ...
changes-exit users havn't complained of any problem that i'm aware of, but that may not mean anything..

just a crazy thought bubble: a possibility would be to call it earlier from rc.6
unless there's some particular reason to have it in cleanup, but i don't really see it, since you can clearly 'dump-session' any time you want.

EDIT: did a bit of a forum search and found:
viewtopic.php?f=53&t=7396 and
viewtopic.php?f=53&t=7316&start=45

and it seems that abelM was thinking along these lines.. if you do this, ie add this to rc.local_shutdown

Code: Select all

# /etc/rc.d/rc.local_shutdown
[ -e /mnt/live/tmp/changes-exit ] && /opt/porteus-scripts/dump-session && rm -f /mnt/live/tmp/changes-exit

that will cancel the cleanup changes-exit routine and use the script instead.
there have to be more to it that in the end eg. an option to skip

[Ed_P]

Thinking, wouldn't commenting out the "/bin" line in the changes-exit.conf file remove the "find: 'bin'" errors? Same with the "/lib" line? A simpler solution than the coding changes.

Added in 21 minutes 28 seconds:

changes-exit users havn't complained of any problem that i'm aware of, but that may not mean anything..

It means it's working and has been for years -> users are happy.
Changes might cause concerns/worries/skepticism/ulcers,etc... as you noted here:

viewtopic.php?f=53&t=7316&start=45

Added in 16 minutes 4 seconds:

ok, i think i had the existence test pointing at the wrong place, this should be correct:

Code: Select all

FOLDERS="$(for x in $(grep ^/ $CHNEXIT_CONF | sed s/.//); do test -e /mnt/live/memory/changes/$x && echo $x; done)"

You had:

Code: Select all

FOLDERS="$(for x in $(grep ^/ $CHNEXIT_CONF | sed s/.//); do test -e $MNAME/$x && echo $x; done)"

and on line 12 of dump-session is:

Code: Select all

MNAME=/mnt/live/memory/images/changes   ; INAME=/mnt/live/memory/images

so you had the right name.

[ncmprhnsbl]

so you had the right name.

read it again: /mnt/live/memory/images/changes != /mnt/live/memory/changes

Thinking, wouldn't commenting out the "/bin" line in the changes-exit.conf file remove the "find: 'bin'" errors? Same with the "/lib" line? A simpler solution than the coding changes.

i think the point of this script is: you don't know/need to know where or what changes there are and the script should just find them (within whatever parameters you set in change-exit.conf).
bin and lib are just obvious examples but it could be any specified directory that doesn't happen to have any changes on a particular occasion.
the question here is whether to silence/ignore the find errors(2>/dev/null) since they cause no harm or to filter the list via an existence loop.
one seems cleaner, but whether it equals less processing idk .. probly not, a bunch of test loops vs less error redirects..

[Ed_P]

read it again: /mnt/live/memory/images/changes != /mnt/live/memory/changes

And you wonder why I didn't see the sed's missing "s".

It's also interesting to note:

Code: Select all

guest@porteus:/mnt/live/memory$ ls -l
total 0
drwxr-xr-x 14 root root 300 Jul  6 20:32 changes/
drwxr-xr-x  2 root root  40 Jul  6 16:32 copy2ram/
drwxr-xr-x 13 root root 260 Jul  6 16:32 images/
drwxr-xr-x  2 root root  40 Jul  6 16:32 xino/
guest@porteus:/mnt/live/memory$ 
guest@porteus:/mnt/live/memory$ ls -l images
total 0
drwxr-xr-x  5 root root  61 May 17 11:31 000-kernel.xzm/
drwxr-xr-x 20 root root 305 May 11 01:59 001-core.xzm/
drwxr-xr-x 10 root root 154 May 11 02:04 002-xorg.xzm/
drwxr-xr-x  6 root root  72 May 11 02:06 002-xtra.xzm/
drwxr-xr-x 10 root root 129 Dec 29  2023 003-cinnamon.xzm/
drwxr-xr-x  7 root root  82 Feb  4  2023 07-printing-x86_64-2023-02-04.xzm/
drwxr-xr-x  3 root root  26 Aug 18  2023 Dell6.4.10-wifi.xzm/
drwxr-xr-x  3 root root  26 May 14  2022 Dellsound.xzm/
drwxr-xr-x 11 root root 114 Jul  6 16:32 changes/
drwxr-xr-x  4 root root  37 Jan 27 23:51 firefox-122.0-x86_64-en-US.xzm/
drwxr-xr-x  4 root root  37 Feb  6 11:54 wmctrl-1.07-x86_64-2salix15.0.xzm/
guest@porteus:/mnt/live/memory$ 

both changes/ folders exist.

Added in 55 minutes 6 seconds:

the question here is whether to silence/ignore the find errors(2>/dev/null) since they cause no harm or to filter the list via an existence loop.

Another option could be to reduce the repetition. (2>/dev/null) one find error but not the 2nd one thus a user still knows if a folder is missing.

[rych]

FOLDERS="$(for x in $(grep ^/ $CHNEXIT_CONF | sed s/.//); do test -e /mnt/live/memory/changes/$x && echo $x; done)"

Code: Select all

root@porteus:~# /opt/porteus-scripts/dump-session
syncing what's left in buffers
saving changes to /mnt/sdb3//porteus - this may take a while...
session saved

So, it's all fixed. Now, a few remaining points/questions, more academic so to speak:

1. The English word dump isn't what we're doing from a user's perspective: changes in RAM are being saved, committed and merged into the persistent storage. I still suggest a name change, since no other system script ever calls dump-session by the name except a rare user's manual invocation.

2. "this may take a while..." is about 6 seconds on my current system, and it is exactly the same time the 2nd run. Why does it takes long the 1st run, by the way. More interestingly, why does it takes exactly same time the 2nd run?? We've just saved all the changes, it could just fly through 2nd run?

3. It's good to be able to commit-changes at any time of course. And then another, final commit at shutdown (either in initrd>cleanup or in rc.local_shutdown) is fine. I was just surprised that there are 2 different code blocks for the same task

Thank you ncmprhnsbl for fixing things, and thanks Ed_P for facilitating the discussion!

[ncmprhnsbl]

2. "this may take a while..." is about 6 seconds on my current system, and it is exactly the same time the 2nd run. Why does it takes long the 1st run, by the way. More interestingly, why does it takes exactly same time the 2nd run?? We've just saved all the changes, it could just fly through 2nd run?

there should be a file: /tmp/save.update.tmp that records the the `cp -uafv --parents` output.
the -u denotes "update older" so only newer existing files will copied/overwritten.
my bet would be that most of time is taken by the preliminary tests and so forth, and six seconds is close to the minimum.

so yeah, a name change can happen (changes-commit seems ok, similar vein to changes-time or maybe just changes-save**)
and i'm leaning towards running it from rc.6 for shutdown, with some options for that and possibly moving what's in rc.local_shutdown in as well.
in short: when run at shutdown: if save.dat checks and reports space, offers to skip saving
when run any other time: not those^ (although, the save.dat checks might be useful here too)

** there is also "save-changes" which is a very simple script to save to a module, that could probly benefit from some the whiteout shenanigans and exclude options going on here.

[Ed_P]

(although, the save.dat checks might be useful here too)

And excellent thought. Overfilling the .dat file can result in a nightmare the next time you boot.

Added in 4 minutes 6 seconds:

so yeah, a name change can happen (changes-commit seems ok, similar vein to changes-time or maybe just changes-save**)

How about savedat-update since the changes are updating the save.dat file rather than creating a module or saving files to the /porteus/changes folder.

[ncmprhnsbl]

How about savedat-update since the changes are updating the save.dat file rather than creating a module or saving files to the /porteus/changes folder.

my understanding is that changes=EXIT: works with both a folder and a file.dat

[rych]

(changes-commit seems ok, similar vein to changes-time or maybe just changes-save**)

How about savedat-update since the changes are updating the save.dat file

changes=EXIT: works with both a folder and a file.dat

I've found there are 3 (!) schools of persistent changes in Porteus: save.dat file on a Microsoft file system, an ext4 /changes folder, and a changes.xzm modules! The changes-commit only "saves" (commits) the new changes in RAM, updating the underlying /changes folder or the save.dat file. Then the combined /changes could be further "saved" (backed-up) into a changes<date>.xzm for example, and I think that's what the save-changes does:

** there is also "save-changes" which is a very simple script to save to a module, that could probably benefit from some the whiteout shenanigans and exclude options going on here.

Ideally all those could be unified and use the same "/union/etc/commit-changes.conf" inclusion/exclusion lists. Because the stuff we want/don't want to be committed from RAM is usually the same stuff we ultimately expect to be/not to be in the /changes, save.dat, or changes<date>.xzm.

Sorry I've made this such a big deal long thread. It's because I had run into a problem of my /memory/changes growing big for RAM and /changes folder growing big for disk storage. Even if I have enough disk space (and RAM space) to hold it, looking into it, I realized that most of it is some junk that Porteus can live without or recreate easily on the next boot, or the staff which doesn't belong in changes and should be extracted to live as a module or separate portable app etc. Anyway, it's been lots of careful monitoring the /changes folder and cleaning it there first, then adding more exclusions into /etc/changes-exit.conf so that it doesn't come back. Then optionally backing it up into an economic changes<date>.xzm. Thanks!

[ncmprhnsbl]

Sorry I've made this such a big deal long thread.

not at all, by the looks of it, this is something that was worked on years ago, that's gotten lost in the mix, and certainly not something i would have noticed.
anyway, watch this space and i'll see what i can cobble together for testing.

[ncmprhnsbl]

watch this space and i'll see what i can cobble together for testing.

here is a completely untested cobble: zz-changes-exit-test.xzm
don't use with anything important, please create a special test install.
contents:

Code: Select all

├── etc
│   └── rc.d
│       ├── rc.6       ## calls /opt/porteus-scripts/changes-commit at shutdown with argument "6" 
│       └── rc.local_shutdown  ## is blanked ie. removed the size testing relocated to changes-commit .. so if you have any of your own custom actions here, restore them.
├── opt
│   └── porteus-scripts
│       ├── changes-commit   ## the new script
│       └── dump-session  ## simply echos to use changes-commit
└── usr
    └── bin
        └── changes-commit -> ../../opt/porteus-scripts/changes-commit    ## link for convenience

what's in it:
rc.6 runs at shutdown/reboot:

Code: Select all

# Run changes-commit for changes=EXIT:
[ -f /mnt/live/tmp/changes-exit ] && /opt/porteus-scripts/changes-commit 6

so, won't run if changes=EXIT: cheatcode isn't present and if it is, runs, telling changes-exit it's running at shutdown ($1 = 6)
changes-commit:

Code: Select all

#!/bin/bash
# Commits live session to the active changes folder or .dat file.
# Script takes action only when 'changes=EXIT' cheatcode is used.
# Can be fine tuned by adjusting /etc/changes-exit.conf to in/exclude files and folders.
# Author: fanthom <[email protected]>
# Modifications proposed by: abelM -- forum.porteus.org
# Further adjustments by: ncmprhnsbl -- forum.porteus.org

txtbld=$(tput bold)               # Bold
txtred=${txtbld}$(tput setaf 1)   # Red
rst=$(tput sgr0)                  # Reset

redwarning() {
echo -e $txtred "$1" $rst
}

# Check if 'changes=EXIT' cheatcode is active and not run from rc.6:
if [ "$1" != "6" ]; then
	test -f /mnt/live/tmp/changes-exit || { echo "'changes=EXIT:' cheatcode is not active - exiting..."; exit; }
fi

# Run as root if not run from rc.6:
if [ "$1" != "6" ]; then
	if [ `whoami` != root ]; then
		echo "Please enter root's password below:"
		su -c '$0'
		exit
	fi
fi

## Variables:
DEST="$(cat /mnt/live/tmp/changes-exit)"; NAME="$(basename $DEST)"
MNAME=/mnt/live/memory/images/changes   ; INAME=/mnt/live/memory/images
CHNEXIT_CONF=/etc/changes-exit.conf     ; EXCL=/tmp/save.excl.tmp
SFILES=/tmp/save.sfiles.tmp             ; UPDATE=/tmp/save.update.tmp

if [ $(grep ".dat" /mnt/live/tmp/changes-exit 2>/dev/null) ]; then
   ## wait for elogind/chvt to settle
   sleep 3
   SDspace=$(df -BM --output=size,used,avail,pcent /mnt/live/memory/images/changes | tail -n1)
   read size used avail percent <<< $SDspace
   if [ ${percent%\%} -gt  85 ]; then
	  echo
	  redwarning "Warning! save.dat file $percent full.  Used: $used  Free: $avail"
	  sleep 5
	  echo
   else
	  echo
	  echo $txtbold "save.dat file $percent full.  Used: $used  Free: $avail" $rst
	  echo
   fi
fi



# Fix folder names and make `egrep' interpret wildcards
fixdir() { sed -e 's_\._\\._g' -e 's_?_._g' -e 's_*_.*_g' -e 's@..@^@'; }

# Fix whiteout names so `egrep' doesn't interpret special characters as regex
fixwh() { sed -e 's_\._\\._g' -e 's_?_\\?_g' -e 's_+_\\+_g' -e 's_*_\\*_g' -e 's_\$_\\$_g' -e 's_\[_\\[_g' -e 's_(_\\(_g'  -e 's_\&_\\&_g' -e 's@^@^@'; }

# Folders to save: argument for `find' & check that they exist.
#FOLDERS="$(grep ^/ $CHNEXIT_CONF | sed s/.//)"
#FOLDERS="$(for x in $FOLDERS; do test -e /mnt/live/memory/changes/$x && echo $x; done)"
FOLDERS="$(for x in $(grep ^/ $CHNEXIT_CONF | sed s/.//); do test -e /mnt/live/memory/changes/$x && echo $x; done)"

# Folders to exclude: argument for `egrep'
grep ^!/ $CHNEXIT_CONF | fixdir > $EXCL

echo -e "Your session will be saved in [1;33m3[0m seconds.\nPress space/enter to start doing it now or any other key to skip."; x=3
while [ $x -gt 0 ]; do read -s -t1 -n1 ans && break || sleep 1; let x=x-1; done
if [ "$ans" = "" ]; then
	## Save session:
	echo "syncing what's left in buffers"
	sync

	# Remount aufs with 'udba=notify' flag:
	mount -o remount,udba=notify /
	echo "saving changes to $DEST - this may take a while..."
	cd /mnt/live/memory/changes

	# Handle filenames with spaces.
	IFS="
	"

	# Remove deleted files and non matching whiteouts
	for y in $(find $FOLDERS -name ".wh.*"); do
	  f="$(echo $y | sed 's@\.wh\.@@g')"
	  test -e "$MNAME/$f" && rm -rf "$MNAME/$f";
	  test -e "$INAME/"*"/$f" || { echo "$y" | fixwh >> $EXCL; test -e "$MNAME/$y" && rm -f "$MNAME/$y"; }
	done

	# Copy new and modified files -- skip excluded whiteouts and folders
	{ test -s "$EXCL" && find $FOLDERS -not -type d | egrep -vf $EXCL || find $FOLDERS -not -type d; } > $SFILES
	cp -uafv --parents $(cat $SFILES) $MNAME > $UPDATE

	# Remove conflicting whiteouts
	for y in $(find $MNAME -name ".wh.*"); do
		f="$(echo $y | sed -e 's^$MNAME^^g' -e 's^\.wh\.^^g')";
		test -e "$f" && rm "$y";
	done

	# Reset IFS to default
	unset IFS
	# Remount aufs with 'udba=none' flag:
	mount -o remount,udba=none /
	echo "session saved"

	# Remove /mnt/live/tmp/changes-exit if called from rc.6 (temporary for legacy initrd where cleanup still has changes=EXIT: routine)
	[ "$1" == "6" ] && rm /mnt/live/tmp/changes-exit
fi

summary:
added check for rc.6 operation or not
added root escalation (skipped for rc.6 operation)
added storage size check (only .dat for now)
added timeout skip from initrd/cleanup (could be annoying when an in session commit is done)
added remove rm /mnt/live/tmp/changes-exit when run from rc.6, to prevent initrd/cleanup running it's own routine.