changes smart backup

Post tutorials, HOWTO's and other useful resources here.
rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

changes smart backup

Post#1 by rych » 22 Apr 2022, 08:55

I've recently got a corrupted home directory (and maybe even some system live places) and all my customization settings were lost bothlive in the live tree and in the changes folder. Now every day I run

Code: Select all

root@porteus:/mnt/sda2/porteus# cp -r changes changesLastGood
However the changes folder can be 100MB or more and clearly I don't need all the temporary or cache files (tmp, var, etc.) or easily reproducible stuff such as $HOME/.wine (although within the huge .wine there are valuable Windows registry modifications I'd like to backup).

I only value those lightweight, mostly textual settings and customization, occasional binary patches, etc. I make on top of the mounted modules, which nevertheless take days (!) to reproduce if lost. So, the stuff I'd like to intelligently back up from changes would be a few Mb at most so no need for incremental backup. I'd simply like a script that automatically runs at exit producing "changes_selected_backup_2022-04-22.zip"

Could a Linux/Porteus expert help me please make such a script that would back-up changes according to an inclusion or exclusion selection list of sub-directories and important files inside changes. Or, perhaps it's already available within Porteus (maybe using changes not saved on EXIT or by dump-session)? Or any advice -- thank you.

User avatar
babam
Warlord
Warlord
Posts: 526
Joined: 16 Nov 2016, 10:30
Distribution: Porteus 5.0rc3 Xfce K6.1.1
Location: Rainy city

changes smart backup

Post#2 by babam » 22 Apr 2022, 10:28

I've never had a corrupt "changes" directory, I used the "fsck" and "changes-ro" cheatcodes.
Sorry, my English is bad.

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

changes smart backup

Post#3 by Ed_P » 22 Apr 2022, 20:40

Yes, losing your changes can be a royal pita rych. Been there, done that, and I don't use the /changes folder, I use a save.dat file.

So, a couple of things to take into account for changes. 1. Backup, 2. Use different approaches.

I use a save.dat file with the changes=EXIT cheatcode. In the early days it would unexpectedly overflow and corrupt the save.dat file. :evil: But over the years changes have been made to Porteus that minimize that from occurring. When shutting down a msg is displayed indicating how full the saved changes are. If too full you can abort the save.dat being updated. :good: In addition a method to tweak what files are being saved has been added also. :good:

My current save.dat is 524MB and I am about 42 full.

To save a backup of your save.dat file you can copy it to a new file name like you were thinking, although unless you plan on keeping multiple copies a simple save.bkup.dat is sufficient. In addition to saving it as save.bkup.dat you can save it as save.dat.xzm. :o Yes, .xzm and it can be read in your /modules folder. The advantage of this is you can run Porteus, with your changes, in a safe, non-updatable, mode. :celebrate14: Great if you have visitors who access your pc on occasion.

And yes you can create a script to backup your changes, if you know what has changed when you saved a bookmark to your browser or added a add-on to your browser to block ads, or changed your power setting, etc..... :unknown: I've used that approach also and my mychanges.xzm file is 12MB.

My /etc/changes-exit.conf file.

Code: Select all

# Folders listed in this config file will be saved during reboot and shutdown when 'changes=EXIT:' cheatcode is used.
# Folders starting with '!' are omitted. This is useful if you want to save whole folder except for particular subfolder(s).
# An example is inclued in default config below: Porteus will save whole /var folder except for /var/run and /var/tmp subfolders.
# Other example: "!/home/guest/.mozilla/firefox/c3pp43bg.default/Cache" will skip saving of Firefox caches from guest account.
# Thanks to Rava for suggesting implementation of '!' exceptions in /etc/changes-exit.conf.

/bin
/etc
/home
/lib
/lib64
/opt
/root
/sbin
/usr
/var
!/var/run
!/var/tmp
!/root/.local/share/Trash/
!/root/.cache/thumbnails/normal
!/home/guest/.local/share/Trash/ 
!/home/guest/.cache/thumbnails/normal
!/home/guest/.cache/mozilla/firefox/ple67xm3.default-release/thumbnails/
!/home/guest/.cache/mozilla/firefox/ple67xm3.default-release/cache2/
!/home/guest/.mozilla/firefox/Crash?Reports/pending/
#/home/guest/.mozilla/firefox/ple67xm3.default-release/storage/default/
!/home/guest/.mozilla/firefox/ple67xm3.default-release/datareporting/archived
!/home/guest/.java/deployment/cache/6.0/

My save.dat backup script.

Code: Select all

#!/bin/sh

# https://forum.porteus.org/viewtopic.php?f=81&t=2866&p=20331#p20292

# https://forum.porteus.org/viewtopic.php?f=53&t=3801&start=30#p28472
# awk 'c-->0;/Booting device:/{c=1}' /var/log/porteus-livedbg
BOOTDEV=`grep -A1 "Booting" /var/log/porteus-livedbg|tail -n1|sed 's^//^/^g'`
FOLDER=
GUEST="$BOOTDEV/GUEST"
MODULES="$BOOTDEV/Modules"
CHANGES="$BOOTDEV/porteus/changes"
#MODSAVEDAT="$BOOTDEV/modsavedat"
if [ ! -d $CHANGES ]; then 
   CHANGES="$BOOTDEV/changes"
fi
if [ "$BOOTDEV" == "/mnt/isoloop" ]; then
   BOOTDEV=`grep -A1 "ISO=" /var/log/porteus-livedbg`
   if [ ${BOOTDEV:4:9} == "/mnt/nvme" ]; then
      BOOTDEV=${BOOTDEV:4:14}
   else
      BOOTDEV=${BOOTDEV:4:9}
   fi
   VERSION=$(cat /etc/porteus-version)
   FOLDER=porteus${VERSION:9:3}
   GUEST="$BOOTDEV/$FOLDER/Guest"
   MODULES="$BOOTDEV/$FOLDER/Modules"
   CHANGES="$BOOTDEV/$FOLDER/changes"
#   MODSAVEDAT="$BOOTDEV/$FOLDER/modsavedat"
fi
SAVEDAT=50rc3save

DIR () {
echo "$1"
#  ls -ghlst --full-time $1 | awk '{print $1 " " $2 " "$6 " " $7 " " $8 " " $9}' | more 
ls -lgh --time-style long-iso $1 | awk '{print $5,$6,$4,$7}'
# http://forum.porteus.org/viewtopic.php?f=81&t=3965#p29205
}

if [ -d "/mnt/loop/home" ]; then
  uloop
fi

DIR "$MODULES" 
echo
DIR "$CHANGES/"
echo 
echo -en "Press 1 to copy .dat to .xzm, 2 to backup the .dat file,  Ctrl+C to end.\n"
read answer  
 
if [ "$answer" == "" ]; then
   echo No option selected.  Bye.
   sleep 3
   exit
fi
if [ $answer == 1 ]; then
   echo "Pls wait, copying in progress"
   cp   $CHANGES/$SAVEDAT.dat $MODULES/$SAVEDAT.dat.xzm
   DIR "$MODULES"  
fi

if [ $answer == 2 ]; then
   echo "Pls wait, copying in progress"
   cp   $CHANGES/$SAVEDAT.dat $CHANGES/${SAVEDAT}Bkup.dat
   DIR "$CHANGES"  
fi

sleep 8
exit
Modify to fit your needs.

My script to check my save.dat space usage.

Code: Select all

#!/bin/sh
# Add to /etc/rc.d/rc.local_shutdown.  And make rc.local_shutdown executable.
#                                      sudo chmod +x /etc/rc.d/rc.local_shutdown
# Bogomips help: https://forum.porteus.org/viewtopic.php?f=140&t=6069&p=47591#p47589
#                https://forum.porteus.org/viewtopic.php?f=140&t=6069&p=47614#p47598

# du -BM -d3 /mnt/live/memory/images/changes 

# Color definitions
txtbld=$(tput bold)               # Bold
txtred=${txtbld}$(tput setaf 1)   # Red
rst=$(tput sgr0)                  # Reset

function redwarning() {
   echo -e $txtred "$1" $rst
}

BOOTDEV=`grep -A1 "Booting" /var/log/porteus-livedbg|tail -n1|sed 's^//^/^g'`
if [ "$BOOTDEV" == "/mnt/isoloop" ]; then
   BOOTDEV=`grep -A1 "ISO=" /var/log/porteus-livedbg`
   BOOTDEV=${BOOTDEV:4:9}
fi
if [ "$BOOTDEV" == "/mnt/nvme" ]; then
   BOOTDEV=`grep -A1 "ISO=" /var/log/porteus-livedbg`
   BOOTDEV=${BOOTDEV:4:14}
fi
VERSION=$(cat /etc/porteus-version)
FOLDER=porteus${VERSION:9:3}
Changes=$BOOTDEV/$FOLDER/changes
if [ ! -d $Changes ]; then 
   Changes="$BOOTDEV/changes"
fi
#set -x;
#SAVEDAT=$(ls -g --sort=time $Changes/ | grep -A1 "^total " | tail -n1)
#SaveDat=${SAVEDAT:41}
SAVEDAT=$(ls --time=use $Changes/ | head -n1)
SaveDat=$SAVEDAT    #SaveDat=50rc3save.dat    # porteussave.dat
if [ -n "$1" ]; then
   SaveDat=$1
fi
#set -x;

if [ -a /mnt/live/memory/images/changes ]; then
   SDspace=$(df -BM --output=size,used,avail,pcent /mnt/live/memory/images/changes | tail -n1)
   Changes=/mnt/live/memory/images/changes
else
   if [ ! -f $Changes/$SaveDat ]; then
      echo "No $Changes/$SaveDat file."
      read && exit
   fi
   echo toor | sudo -S -u root mloop $Changes/$SaveDat &>/dev/null        
   SDspace=$(df -BM --output=size,used,avail,pcent /mnt/loop | tail -n1)
   Changes=$Changes/$SaveDat
   echo toor | sudo -S -u root uloop &>/dev/null
fi

# echo $SDspace
read size used avail percent <<< $SDspace
if [ ${percent%\%} -ge  85 ]; then
   /usr/share/porteus/gtkdialog.py -p "Warning!  $SaveDat file $percent full" -s "Used: $used - Free: $avail" -d err
   exit
else
#  https://forum.porteus.org/viewtopic.php?p=78822#p78822 - ncmprhnsbl - '/usr/share/porteus/gtkdialog.py -h' to see the usage..  
   /usr/share/porteus/gtkdialog.py -p "$Changes"    -s "$SaveDat file $percent full - Used: $used - Free: $avail" -d info 
   exit
fi

# if no dialog option.
if [ ${percent%\%} -ge  85 ]; then
   echo 
   redwarning "Warning! save.dat file $percent full.  Used: $used  Free: $avail"
   echo 
else
   echo "save.dat file $percent full.  Used: $used  Free: $avail"
fi
sleep 5
exit
BTW I boot Porteus ISOs not installs. As such my folders are outside the ISO and on my hardrive. Such as /porteus5.0/Modules/ and /porteus5.0/Optional/.

hth
Last edited by Ed_P on 28 Oct 2022, 21:14, edited 2 times in total.
Reason: Edited an error in my changes_exit.conf file
Ed

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

changes smart backup

Post#4 by rych » 16 Jun 2022, 09:34

I could simply run

Code: Select all

rm changes.xzm
/opt/porteus-scripts/save-changes changes.xzm
but the resulting archive contains everything including all the temporary files, caches, and the large .wine folder -- nicely compressed but still a 30MB xzm.

If only the save-changes would use the same changes-exit.conf just how dump-session does it
in direct update at dump session

tome
Contributor
Contributor
Posts: 675
Joined: 26 Jun 2013, 14:03
Distribution: x64 Openbox
Location: Russia is causing the immense damage to humanity
Contact:

changes smart backup

Post#5 by tome » 16 Jun 2022, 20:10

# Start of lines descriptions:

Maybe you are interested something like this changes-exit.conf:
# "/" - file/dir saved during reboot and shutdown when 'changes=EXIT:' cheatcode is used
# "!/" - exclude particular subfolder(s) if parent directory is designed to be saved during shutdown ('changes=EXIT:')
# "@/" - 'path' will be symlink to "LINKDEST"/'path' made during boot (if target exist and other conditions are fulfilled)
# "+/" - if file/dir was changed will be copied to "COPYDEST" during shutdown (or running /etc/rc.d/copyto)
# "++/" - like "+/" but will be created numbered backups of newer files next to copied ones
# "+-/" - like "++" but backups will be moved to (kept in) Trash
# "&" - line will be executed during boot (designed for special symlinks)

# Additional descriptions:
# "+/", "++" and "+-" are designed to keep some (not large) files in rootcopy or another place
# "@" - designed to keep large, important or frequently changed content of directories/files between reboots. If paths are files or non empty directories made by you outside modules, symlinks will not be created - it is better to use it without changes=/ cheatcode. Paths with space characters may be not supported.
# By default LINKSTREE is "/" (LINKSTREE=/) and it should be set or changed only for testing

# Uncomment proper destinations or change them, only one for COPYDEST and one for LINKDEST:

# Below COPYDEST is /data/rootcopies/rootcopy3 on boot device and LINKDEST is /data on boot device
#COPYDEST=grep -A1 "Booting" /var/log/porteus-livedbg|tail -n1|sed 's^//^/^g' | sed 's^$^/data/rootcopies/rootcopy3^'
#LINKDEST=grep -A1 "Booting" /var/log/porteus-livedbg|tail -n1|sed 's^//^/^g' | sed 's^$^/data^'

#COPYDEST=/mnt/sda1/data/rootcopies/rootcopy3
COPYDEST=/mnt/data/rootcopies/rootcopy4

#LINKDEST=/mnt/data
#LINKDEST=/tmp/data
You have mind and feelings. Be wise and clever.

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

changes smart backup

Post#6 by Ed_P » 17 Jun 2022, 14:44

:shock: Where did you find this tome?
Ed

tome
Contributor
Contributor
Posts: 675
Joined: 26 Jun 2013, 14:03
Distribution: x64 Openbox
Location: Russia is causing the immense damage to humanity
Contact:

changes smart backup

Post#7 by tome » 17 Jun 2022, 15:38

I created this and scripts that do it 3 years ago for me. I uploaded this somewhere and other tools, if someone is interested can share it here.
You have mind and feelings. Be wise and clever.

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

changes smart backup

Post#8 by Ed_P » 19 Jun 2022, 01:49

I'm sure there are users that would be interested tome. Please share the links.
Ed

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

changes smart backup

Post#9 by rych » 19 Jun 2022, 11:03

rych wrote:
16 Jun 2022, 09:34
the resulting archive contains everything including all the temporary files, caches, and the large .wine folder -- nicely compressed but still a 30MB xzm.
I guess one way is not to have the changes folder so large in the first place :) So I moved /root/.wine folder into a dotwine.xzm module always activated. Now my changes.xzm is down to 14MB.

Besides valuable settings and customization, the changes.xzm still contains (mostly) some other stuff -- altogether representing an exact snapshot of the system when overlaid on top of modules.

Still, even the /opt/porteus-scripts/save-changes does exclude some obvious directories:

Code: Select all

Line#37: mksquashfs $ch $1 -e dev tmp mnt
We could use instead the option

Code: Select all

−ef exclude_file
where exclude_file is a file containing a list of files / directories to exclude, one per line.

Please share your ideas on which folders and files inside the changes folder don't really need to be backed up :)
exclude_file.txt:
/dev
/tmp
/mnt

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

changes smart backup

Post#10 by Ed_P » 19 Jun 2022, 15:14

These are the folders I have in my changes-exit.conf file.

Code: Select all

/bin
/etc
/home
/lib
/lib64
/opt
/root
/sbin
/usr
/var
!/var/run
!/var/tmp
!/root/.local/share/Trash/
!/root/.cache/thumbnails/normal
!/home/guest/.local/share/Trash/
!/home/guest/.cache/thumbnails/normal
!/home/guest/.cache/mozilla/firefox/ple67xm3.default-release/thumbnails/
!/home/guest/.cache/mozilla/firefox/ple67xm3.default-release/cache2/
!/home/guest/.mozilla/firefox/Crash?Reports/pending/
#/home/guest/.mozilla/firefox/ple67xm3.default-release/storage/default/
!/home/guest/.mozilla/firefox/ple67xm3.default-release/datareporting/archived
!/home/guest/.java/deployment/cache/6.0/

Last edited by Ed_P on 28 Oct 2022, 21:16, edited 1 time in total.
Reason: Edited an error in my changes_exit.conf file
Ed

rych
Warlord
Warlord
Posts: 622
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

changes smart backup

Post#11 by rych » 20 Jun 2022, 09:36

Code: Select all

ch=/mnt/live/memory/changes
chxzm=/tmp/changes.xzm
rm $chxzm
mksquashfs $ch $chxzm -e dev tmp mnt root/.cache
That's what I'll use for now.

Post Reply