Page 1 of 2

a question about whiteouts and persistence

Posted: 15 Nov 2017, 02:37
by abelM
Hi all,

I use Porteus with the changes=EXIT cheatcode and I've noticed my persistent mount is getting cluttered with whiteouts. Aufs use whiteouts to hide files when they are deleted -- it doesn't remove them really. But do whiteouts have a function in the same filesystem where files are deleted?

If I understand the point, Porteus goes through cycles like this:
- At startup it loads files from write-only modules
- It loads a write-read persistent mount where you save changes
- You write or edit some files
- At logout (or dump session) Porteus copies new or modified files to the persistent mount

When you delete files things go different:
- You delete some files
- If these files exist in modules or in the persistent mount then Aufs puts whiteouts to hide them from your sight.
- At logout (or dump session)
-- Porteus do remove files from the persistent mount if matching whiteouts exist
-- And it saves whiteouts to the persistent mount

Finally in case you name a file as a previously deleted one Porteus will remove the whiteout from the persistent mount -- if not you wouldn't see the file!

In the end the persistent mount keeps more and more whiteouts matching no files at all. So, is it necesary to copy whiteouts of deleted files coming from the persistent mount? If not you could keep the system cleaner and save some writes and space too. I'd like to know what you think about all this.

Regards,

a question about whiteouts and persistence

Posted: 15 Nov 2017, 06:29
by Ed_P
I use changes=EXIT also and have occasionally experienced problems due to whiteouts in the savedat file and I run a script to delete them periodically. The vast majority being files deleted from the Firefox cache. As to whether their presence impacts speed or space significantly I don't think so.

You can purge the wh files from the mounted changes file but to be more effective it's better to clear the wh files from the actual savedat file when you run Porteus in AF mode.

My savedat file is in a folder named changes which is within a folder matching the name of the Porteus version running. Thus for 3.2.2 my savedat is:
/porteus3.2/changes/porteussave.dat. And this is the script I run to clear out the wh files.

Code: Select all

#!/bin/sh 
# https://forum.porteus.org/viewtopic.php?f=81&t=3501&p=25151#p25162  <- USM problem 
# https://forum.porteus.org/viewtopic.php?f=81&t=3501&p=25151#p25178
# https://forum.porteus.org/viewtopic.php?f=81&t=5698#p43823

#  mloop requires changes= cheatcodes not be used (Always Fresh mode)
#  nor modsavedat/porteussave.dat.xzm!!

BOOTDEV=`grep -A1 "Booting" /var/log/porteus-livedbg|tail -n1|sed 's^//^/^g'`
if [ "$BOOTDEV" == "/mnt/isoloop" ]; then
   BOOTDEV=`grep -A1 "ISO=" /var/log/porteus-livedbg`
   BOOTDEV=${BOOTDEV:4:9}
fi
VERSION=$(cat /etc/porteus-version)
FOLDER=porteus${VERSION:9:3}
GUEST="$BOOTDEV/$FOLDER/Guest"
MODULES="$BOOTDEV/$FOLDER/Modules"

# Color definitions
txtbld=$(tput bold)               # Bold
txtred=${txtbld}$(tput setaf 1)   # Red
rst=$(tput sgr0)                  # Reset

function redpswd() {
echo -e "$1" $txtred  
}

if [ `whoami` != "root" ]; then
    redpswd "Enter root's password"
    su -c "sh $0 $1"
    exit
fi
echo $rst

if [ "$1" == "" ]; then
   if [ -a /mnt/live/memory/images/changes ]; then
      SPACEDATDIR=/mnt/live/memory/images/changes 
   else
      echo mloop $BOOTDEV/$FOLDER/changes/porteussave.dat
      mloop $BOOTDEV/$FOLDER/changes/porteussave.dat
      SPACEDATDIR=/mnt/loop
   fi
else
   echo mloop $1
   mloop $1
   SPACEDATDIR=/mnt/loop
fi

if [ ! -d ${SPACEDATDIR}/home ]; then 
   echo " Error!! "
   dmesg | tail > t
   UUIDck=$(grep -c "duplicate UUID" t) 
   if [ ! "$UUIDck" == "0" ]; then
      echo
      echo " * This is NOT Always Fresh mode. * "
      echo
   else 
      echo
      echo "dmesg | tail "
      dmesg | tail  
   fi
   rm t
   uloop
   exit
fi


echo find ${SPACEDATDIR} -name .wh.*
find ${SPACEDATDIR} -name .wh.*
echo
echo -en "Enter 1 to delete, 2 to exit.\n"
select yn in "Yes" "No"; do
   case $yn in
        Yes ) echo "find ${SPACEDATDIR}/home -name .wh.* -delete"
              find ${SPACEDATDIR}/home -name ".wh.*" -delete
              echo
              echo find ${SPACEDATDIR} -name .wh.*
              find ${SPACEDATDIR} -name .wh.*
              break;;
        No )  break;; 
   esac
done

if [ -a /mnt/live/memory/images/changes ]; then
   exit
else
   uloop 
fi
exit
Review and tweak to fit your needs.

a question about whiteouts and persistence

Posted: 15 Nov 2017, 15:05
by abelM
Thank you Ed! I'm trying your script.

Regards,

a question about whiteouts and persistence

Posted: 16 Nov 2017, 16:42
by abelM
Well, besides cleaning my persistent mount using Ed's script, I still have interest to know if whiteouts have a function in the same filesystem -- I'd better say in the same branch -- where matching files have been deleted. In the context of Porteus usage it seems they don't. If so it would be nice to avoid saving them.

I hope to be clear, my English is not so good. :wall:

Regards,

a question about whiteouts and persistence

Posted: 16 Nov 2017, 17:25
by Ed_P

a question about whiteouts and persistence

Posted: 17 Nov 2017, 00:04
by abelM
Hi Ed! I've read the posts you've suggested. All interesting. I've taken your script as reference and worked it out in a primitive way
I've booted Porteus in AF mode, opened a terminal and done this

Code: Select all

su
mloop
cd /mnt/loop
find -name "*.wh.*" -delete
cd ..
uloop
exit
I see this is a task I should do from time to time. Anyway I'll try to find the way of avoiding saving whiteouts which have no function in the persistent mount. I'll tell you later.

And thank you very much! :worship:

a question about whiteouts and persistence

Posted: 17 Nov 2017, 11:20
by burdi01
Actually aufs has an option to handle the whiteouts in a "save session" feature.
See http://aufs.sourceforge.net/aufs2/shwh/README.txt for details.
If you have access to a not-too-old PartedMagic release see the /usr/bin/pmagic_save_session.v2 script for my implementation.
:D

a question about whiteouts and persistence

Posted: 17 Nov 2017, 18:41
by abelM
Hi burdi01,

I've read the document -- interesting indeed but I'm asking something different. Do whiteouts have any function in the same branch where files are deleted? It seems not just because there's nothing to hide there. The case is completely different when whiteouts hide files existing in a different branch.

If I'm understanding the point, as you remove /etc/lynx.cfg you'll get a /etc/.wh.lynx.cfg whiteout to hide a file existing in a different branch -- a read-only module. That's what whiteouts do. But as you remove /home/guest/my_useless_doc you'll get /home/guest/.wh.my_useless_doc in the same branch where the file was -- your persistent mount. It seems innecesary, doesn't it? So I'm thinking of some way to avoid saving this kind of whiteouts. But I'm not sure I'm going in the right direction. I'm far from being an expert as you have noticed. :pardon:

Thanks!

a question about whiteouts and persistence

Posted: 18 Nov 2017, 11:13
by burdi01
On a first impulse I would say too that the whiteout for your /home/guest/my_useless_doc *) is unnecessary. I did a preliminary test and did not see such a whiteout -- when looking at /mnt/live/memory/changes/home/guest/ (which access is outside of the union).
You run with persistence and I do not. To be honest I never looked in detail into how Porteus implements persistence, but for now I fail to see how these whiteouts would end up in the persistence file.
:D

*) By the way: I assumed this to be a file. Please state whether this is a file or a directory.

a question about whiteouts and persistence

Posted: 18 Nov 2017, 15:38
by abelM
I see we are working in different contexts. I'm saving persistence in a save.dat file at exit -- changes=EXIT:porteussave.dat in the command line. Whiteouts do appear in /mnt/live/memory/changes/home/guest/ instantly as any other modification I do in the union system. On the other hand /mnt/live/memory/images/changes/home/guest/ keeps persistence and changes are copied there just at dump-session or logout.

I'm testing filtering out whiteouts in /home/guest only to avoid messing whith system files. Until now it seems to be ok but I need more time to tell you.

Regards,

a question about whiteouts and persistence

Posted: 18 Nov 2017, 18:48
by brokenman
Do whiteouts have any function in the same branch where files are deleted?
No. It is a function of how whiteouts work, but in the same branch they would serve no real purpose except to hide underlying filesystem duplicates.

a question about whiteouts and persistence

Posted: 20 Nov 2017, 17:34
by burdi01
I recreated abelM's problem -- or rather symptom because the whiteouts in the persistence file, though not nice, do not pose a problem for as far as I can see -- at logout as well as via dump-session.
As guest I created the "useless.txt" file in home directory. Technically speaking this file is created in the /mnt/live/memory/images/changes/home/guest/ directory. At logout this file is moved to the persistence file.
In a subsequent session I deleted that file. As the persistence file is mounted just like any other module, a whiteout in the whiteout appears in the /mnt/live/memory/images/changes/home/guest/ directory is necessary. At logout that whiteout is moved to the persistence file, while the original useless.txt file is removed.
<to be continued>
:D

a question about whiteouts and persistence

Posted: 21 Nov 2017, 00:21
by abelM
That's it! I also do agree with you whiteouts seem to do no harm although they don't match any file. But you'll find a clutter in your save.dat file if you need to mloop it for some reason. You'll have to eventually remove all whiteouts -- the good and the bad. I see it's not a critical issue but I'm trying to work it out though. BTW you can look at this post if you like direct update at dump session

Regards :)

a question about whiteouts and persistence

Posted: 21 Nov 2017, 08:32
by brokenman
Great to see empirical testing! The results should be expected. Here's a scenario ...

What would happen if you left the whiteout file in place for useless.txt, then unpacked 001 module and placed the same file there, then rebooted.

a question about whiteouts and persistence

Posted: 21 Nov 2017, 11:33
by burdi01
@abelM: I will have a look at that thread.

@brokenman: As I said: < to be continued > ...
The next test I planned was to "delete" a file from a "normal" xzm, e.g. guest's .config/gecko-mediaplayer/gecko-mediaplayer.conf from 003-mate.xzm. The whitout indeed is copied to the persistence file too, and in a subsquent session the .conf file turns out to be "deleted" still.
When I looked at things (2 years ago for PartedMagic) my tests would indicate that whiteouts in non-upper branches were not handled consistently. On top of that I also found the http://aufs.sourceforge.net/aufs2/shwh/README.txt document that seemed to confirm my test results.
So I chose to not copy the whiteouts to the persistence file. Actually I implemented things twice (once with and once without the CONFIG_AUFS_SHWH kernel setting). By implication the "user" files (that only occur in the persistence file) can be deleted, but the "distribution" files (from the "normal" xzm's) cannot. Actually we never saw user complaints about this, so the ability to delete "distribution" files does not seem to be in high-demand.
Obviously you chose to copy the whiteouts to the persistence file. Assuming the whiteouts in non-upper branches are now handled correctly the only thing users could complain about is that there is no easy way to undelete these "distribution" files -- as your scenario shows.
:D

Please regard these remarks a discussion and *not* as a negative comment.