a question about whiteouts and persistence

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
Post Reply
User avatar
abelM
White ninja
White ninja
Posts: 28
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#1 by abelM » 15 Nov 2017, 02:37

Hi all,

I use Porteus with the changes=EXIT cheatcode and I've noticed my persistent mount is getting cluttered with whiteouts. Aufs use whiteouts to hide files when they are deleted -- it doesn't remove them really. But do whiteouts have a function in the same filesystem where files are deleted?

If I understand the point, Porteus goes through cycles like this:
- At startup it loads files from write-only modules
- It loads a write-read persistent mount where you save changes
- You write or edit some files
- At logout (or dump session) Porteus copies new or modified files to the persistent mount

When you delete files things go different:
- You delete some files
- If these files exist in modules or in the persistent mount then Aufs puts whiteouts to hide them from your sight.
- At logout (or dump session)
-- Porteus do remove files from the persistent mount if matching whiteouts exist
-- And it saves whiteouts to the persistent mount

Finally in case you name a file as a previously deleted one Porteus will remove the whiteout from the persistent mount -- if not you wouldn't see the file!

In the end the persistent mount keeps more and more whiteouts matching no files at all. So, is it necesary to copy whiteouts of deleted files coming from the persistent mount? If not you could keep the system cleaner and save some writes and space too. I'd like to know what you think about all this.

Regards,

User avatar
Ed_P
Contributor
Contributor
Posts: 3319
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 3.2.2 64-bit ISO
Location: Western NY, USA

a question about whiteouts and persistence

Post#2 by Ed_P » 15 Nov 2017, 06:29

I use changes=EXIT also and have occasionally experienced problems due to whiteouts in the savedat file and I run a script to delete them periodically. The vast majority being files deleted from the Firefox cache. As to whether their presence impacts speed or space significantly I don't think so.

You can purge the wh files from the mounted changes file but to be more effective it's better to clear the wh files from the actual savedat file when you run Porteus in AF mode.

My savedat file is in a folder named changes which is within a folder matching the name of the Porteus version running. Thus for 3.2.2 my savedat is:
/porteus3.2/changes/porteussave.dat. And this is the script I run to clear out the wh files.

Code: Select all

#!/bin/sh 
# https://forum.porteus.org/viewtopic.php?f=81&t=3501&p=25151#p25162  <- USM problem 
# https://forum.porteus.org/viewtopic.php?f=81&t=3501&p=25151#p25178
# https://forum.porteus.org/viewtopic.php?f=81&t=5698#p43823

#  mloop requires changes= cheatcodes not be used (Always Fresh mode)
#  nor modsavedat/porteussave.dat.xzm!!

BOOTDEV=`grep -A1 "Booting" /var/log/porteus-livedbg|tail -n1|sed 's^//^/^g'`
if [ "$BOOTDEV" == "/mnt/isoloop" ]; then
   BOOTDEV=`grep -A1 "ISO=" /var/log/porteus-livedbg`
   BOOTDEV=${BOOTDEV:4:9}
fi
VERSION=$(cat /etc/porteus-version)
FOLDER=porteus${VERSION:9:3}
GUEST="$BOOTDEV/$FOLDER/Guest"
MODULES="$BOOTDEV/$FOLDER/Modules"

# Color definitions
txtbld=$(tput bold)               # Bold
txtred=${txtbld}$(tput setaf 1)   # Red
rst=$(tput sgr0)                  # Reset

function redpswd() {
echo -e "$1" $txtred  
}

if [ `whoami` != "root" ]; then
    redpswd "Enter root's password"
    su -c "sh $0 $1"
    exit
fi
echo $rst

if [ "$1" == "" ]; then
   if [ -a /mnt/live/memory/images/changes ]; then
      SPACEDATDIR=/mnt/live/memory/images/changes 
   else
      echo mloop $BOOTDEV/$FOLDER/changes/porteussave.dat
      mloop $BOOTDEV/$FOLDER/changes/porteussave.dat
      SPACEDATDIR=/mnt/loop
   fi
else
   echo mloop $1
   mloop $1
   SPACEDATDIR=/mnt/loop
fi

if [ ! -d ${SPACEDATDIR}/home ]; then 
   echo " Error!! "
   dmesg | tail > t
   UUIDck=$(grep -c "duplicate UUID" t) 
   if [ ! "$UUIDck" == "0" ]; then
      echo
      echo " * This is NOT Always Fresh mode. * "
      echo
   else 
      echo
      echo "dmesg | tail "
      dmesg | tail  
   fi
   rm t
   uloop
   exit
fi


echo find ${SPACEDATDIR} -name .wh.*
find ${SPACEDATDIR} -name .wh.*
echo
echo -en "Enter 1 to delete, 2 to exit.\n"
select yn in "Yes" "No"; do
   case $yn in
        Yes ) echo "find ${SPACEDATDIR}/home -name .wh.* -delete"
              find ${SPACEDATDIR}/home -name ".wh.*" -delete
              echo
              echo find ${SPACEDATDIR} -name .wh.*
              find ${SPACEDATDIR} -name .wh.*
              break;;
        No )  break;; 
   esac
done

if [ -a /mnt/live/memory/images/changes ]; then
   exit
else
   uloop 
fi
exit
Review and tweak to fit your needs.
Ed

User avatar
abelM
White ninja
White ninja
Posts: 28
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#3 by abelM » 15 Nov 2017, 15:05

Thank you Ed! I'm trying your script.

Regards,

User avatar
abelM
White ninja
White ninja
Posts: 28
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#4 by abelM » 16 Nov 2017, 16:42

Well, besides cleaning my persistent mount using Ed's script, I still have interest to know if whiteouts have a function in the same filesystem -- I'd better say in the same branch -- where matching files have been deleted. In the context of Porteus usage it seems they don't. If so it would be nice to avoid saving them.

I hope to be clear, my English is not so good. :wall:

Regards,

User avatar
Ed_P
Contributor
Contributor
Posts: 3319
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 3.2.2 64-bit ISO
Location: Western NY, USA

a question about whiteouts and persistence

Post#5 by Ed_P » 16 Nov 2017, 17:25

Ed

User avatar
abelM
White ninja
White ninja
Posts: 28
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#6 by abelM » 17 Nov 2017, 00:04

Hi Ed! I've read the posts you've suggested. All interesting. I've taken your script as reference and worked it out in a primitive way
I've booted Porteus in AF mode, opened a terminal and done this

Code: Select all

su
mloop
cd /mnt/loop
find -name "*.wh.*" -delete
cd ..
uloop
exit
I see this is a task I should do from time to time. Anyway I'll try to find the way of avoiding saving whiteouts which have no function in the persistent mount. I'll tell you later.

And thank you very much! :worship:

burdi01
Black ninja
Black ninja
Posts: 63
Joined: 18 Aug 2013, 12:09
Distribution: Slackware, Xubuntu
Location: The Netherlands

a question about whiteouts and persistence

Post#7 by burdi01 » 17 Nov 2017, 11:20

Actually aufs has an option to handle the whiteouts in a "save session" feature.
See http://aufs.sourceforge.net/aufs2/shwh/README.txt for details.
If you have access to a not-too-old PartedMagic release see the /usr/bin/pmagic_save_session.v2 script for my implementation.
:D

User avatar
abelM
White ninja
White ninja
Posts: 28
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#8 by abelM » 17 Nov 2017, 18:41

Hi burdi01,

I've read the document -- interesting indeed but I'm asking something different. Do whiteouts have any function in the same branch where files are deleted? It seems not just because there's nothing to hide there. The case is completely different when whiteouts hide files existing in a different branch.

If I'm understanding the point, as you remove /etc/lynx.cfg you'll get a /etc/.wh.lynx.cfg whiteout to hide a file existing in a different branch -- a read-only module. That's what whiteouts do. But as you remove /home/guest/my_useless_doc you'll get /home/guest/.wh.my_useless_doc in the same branch where the file was -- your persistent mount. It seems innecesary, doesn't it? So I'm thinking of some way to avoid saving this kind of whiteouts. But I'm not sure I'm going in the right direction. I'm far from being an expert as you have noticed. :pardon:

Thanks!

burdi01
Black ninja
Black ninja
Posts: 63
Joined: 18 Aug 2013, 12:09
Distribution: Slackware, Xubuntu
Location: The Netherlands

a question about whiteouts and persistence

Post#9 by burdi01 » 18 Nov 2017, 11:13

On a first impulse I would say too that the whiteout for your /home/guest/my_useless_doc *) is unnecessary. I did a preliminary test and did not see such a whiteout -- when looking at /mnt/live/memory/changes/home/guest/ (which access is outside of the union).
You run with persistence and I do not. To be honest I never looked in detail into how Porteus implements persistence, but for now I fail to see how these whiteouts would end up in the persistence file.
:D

*) By the way: I assumed this to be a file. Please state whether this is a file or a directory.

User avatar
abelM
White ninja
White ninja
Posts: 28
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#10 by abelM » 18 Nov 2017, 15:38

I see we are working in different contexts. I'm saving persistence in a save.dat file at exit -- changes=EXIT:porteussave.dat in the command line. Whiteouts do appear in /mnt/live/memory/changes/home/guest/ instantly as any other modification I do in the union system. On the other hand /mnt/live/memory/images/changes/home/guest/ keeps persistence and changes are copied there just at dump-session or logout.

I'm testing filtering out whiteouts in /home/guest only to avoid messing whith system files. Until now it seems to be ok but I need more time to tell you.

Regards,

User avatar
brokenman
Site Admin
Site Admin
Posts: 5562
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

a question about whiteouts and persistence

Post#11 by brokenman » 18 Nov 2017, 18:48

Do whiteouts have any function in the same branch where files are deleted?
No. It is a function of how whiteouts work, but in the same branch they would serve no real purpose except to hide underlying filesystem duplicates.
How do i become super user?
Wear your underpants on the outside and put on a cape.

Post Reply