a question about whiteouts and persistence

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
User avatar
abelM
Black ninja
Black ninja
Posts: 50
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#1 by abelM » 15 Nov 2017, 02:37

Hi all,

I use Porteus with the changes=EXIT cheatcode and I've noticed my persistent mount is getting cluttered with whiteouts. Aufs use whiteouts to hide files when they are deleted -- it doesn't remove them really. But do whiteouts have a function in the same filesystem where files are deleted?

If I understand the point, Porteus goes through cycles like this:
- At startup it loads files from write-only modules
- It loads a write-read persistent mount where you save changes
- You write or edit some files
- At logout (or dump session) Porteus copies new or modified files to the persistent mount

When you delete files things go different:
- You delete some files
- If these files exist in modules or in the persistent mount then Aufs puts whiteouts to hide them from your sight.
- At logout (or dump session)
-- Porteus do remove files from the persistent mount if matching whiteouts exist
-- And it saves whiteouts to the persistent mount

Finally in case you name a file as a previously deleted one Porteus will remove the whiteout from the persistent mount -- if not you wouldn't see the file!

In the end the persistent mount keeps more and more whiteouts matching no files at all. So, is it necesary to copy whiteouts of deleted files coming from the persistent mount? If not you could keep the system cleaner and save some writes and space too. I'd like to know what you think about all this.

Regards,

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

a question about whiteouts and persistence

Post#2 by Ed_P » 15 Nov 2017, 06:29

I use changes=EXIT also and have occasionally experienced problems due to whiteouts in the savedat file and I run a script to delete them periodically. The vast majority being files deleted from the Firefox cache. As to whether their presence impacts speed or space significantly I don't think so.

You can purge the wh files from the mounted changes file but to be more effective it's better to clear the wh files from the actual savedat file when you run Porteus in AF mode.

My savedat file is in a folder named changes which is within a folder matching the name of the Porteus version running. Thus for 3.2.2 my savedat is:
/porteus3.2/changes/porteussave.dat. And this is the script I run to clear out the wh files.

Code: Select all

#!/bin/sh 
# https://forum.porteus.org/viewtopic.php?f=81&t=3501&p=25151#p25162  <- USM problem 
# https://forum.porteus.org/viewtopic.php?f=81&t=3501&p=25151#p25178
# https://forum.porteus.org/viewtopic.php?f=81&t=5698#p43823

#  mloop requires changes= cheatcodes not be used (Always Fresh mode)
#  nor modsavedat/porteussave.dat.xzm!!

BOOTDEV=`grep -A1 "Booting" /var/log/porteus-livedbg|tail -n1|sed 's^//^/^g'`
if [ "$BOOTDEV" == "/mnt/isoloop" ]; then
   BOOTDEV=`grep -A1 "ISO=" /var/log/porteus-livedbg`
   BOOTDEV=${BOOTDEV:4:9}
fi
VERSION=$(cat /etc/porteus-version)
FOLDER=porteus${VERSION:9:3}
GUEST="$BOOTDEV/$FOLDER/Guest"
MODULES="$BOOTDEV/$FOLDER/Modules"

# Color definitions
txtbld=$(tput bold)               # Bold
txtred=${txtbld}$(tput setaf 1)   # Red
rst=$(tput sgr0)                  # Reset

function redpswd() {
echo -e "$1" $txtred  
}

if [ `whoami` != "root" ]; then
    redpswd "Enter root's password"
    su -c "sh $0 $1"
    exit
fi
echo $rst

if [ "$1" == "" ]; then
   if [ -a /mnt/live/memory/images/changes ]; then
      SPACEDATDIR=/mnt/live/memory/images/changes 
   else
      echo mloop $BOOTDEV/$FOLDER/changes/porteussave.dat
      mloop $BOOTDEV/$FOLDER/changes/porteussave.dat
      SPACEDATDIR=/mnt/loop
   fi
else
   echo mloop $1
   mloop $1
   SPACEDATDIR=/mnt/loop
fi

if [ ! -d ${SPACEDATDIR}/home ]; then 
   echo " Error!! "
   dmesg | tail > t
   UUIDck=$(grep -c "duplicate UUID" t) 
   if [ ! "$UUIDck" == "0" ]; then
      echo
      echo " * This is NOT Always Fresh mode. * "
      echo
   else 
      echo
      echo "dmesg | tail "
      dmesg | tail  
   fi
   rm t
   uloop
   exit
fi


echo find ${SPACEDATDIR} -name .wh.*
find ${SPACEDATDIR} -name .wh.*
echo
echo -en "Enter 1 to delete, 2 to exit.\n"
select yn in "Yes" "No"; do
   case $yn in
        Yes ) echo "find ${SPACEDATDIR}/home -name .wh.* -delete"
              find ${SPACEDATDIR}/home -name ".wh.*" -delete
              echo
              echo find ${SPACEDATDIR} -name .wh.*
              find ${SPACEDATDIR} -name .wh.*
              break;;
        No )  break;; 
   esac
done

if [ -a /mnt/live/memory/images/changes ]; then
   exit
else
   uloop 
fi
exit
Review and tweak to fit your needs.
Ed

User avatar
abelM
Black ninja
Black ninja
Posts: 50
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#3 by abelM » 15 Nov 2017, 15:05

Thank you Ed! I'm trying your script.

Regards,

User avatar
abelM
Black ninja
Black ninja
Posts: 50
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#4 by abelM » 16 Nov 2017, 16:42

Well, besides cleaning my persistent mount using Ed's script, I still have interest to know if whiteouts have a function in the same filesystem -- I'd better say in the same branch -- where matching files have been deleted. In the context of Porteus usage it seems they don't. If so it would be nice to avoid saving them.

I hope to be clear, my English is not so good. :wall:

Regards,

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

a question about whiteouts and persistence

Post#5 by Ed_P » 16 Nov 2017, 17:25

Ed

User avatar
abelM
Black ninja
Black ninja
Posts: 50
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#6 by abelM » 17 Nov 2017, 00:04

Hi Ed! I've read the posts you've suggested. All interesting. I've taken your script as reference and worked it out in a primitive way
I've booted Porteus in AF mode, opened a terminal and done this

Code: Select all

su
mloop
cd /mnt/loop
find -name "*.wh.*" -delete
cd ..
uloop
exit
I see this is a task I should do from time to time. Anyway I'll try to find the way of avoiding saving whiteouts which have no function in the persistent mount. I'll tell you later.

And thank you very much! :worship:

burdi01
Shogun
Shogun
Posts: 201
Joined: 18 Aug 2013, 12:09
Distribution: Slackware PartedMagic Xubuntu
Location: The Netherlands

a question about whiteouts and persistence

Post#7 by burdi01 » 17 Nov 2017, 11:20

Actually aufs has an option to handle the whiteouts in a "save session" feature.
See http://aufs.sourceforge.net/aufs2/shwh/README.txt for details.
If you have access to a not-too-old PartedMagic release see the /usr/bin/pmagic_save_session.v2 script for my implementation.
:D

User avatar
abelM
Black ninja
Black ninja
Posts: 50
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#8 by abelM » 17 Nov 2017, 18:41

Hi burdi01,

I've read the document -- interesting indeed but I'm asking something different. Do whiteouts have any function in the same branch where files are deleted? It seems not just because there's nothing to hide there. The case is completely different when whiteouts hide files existing in a different branch.

If I'm understanding the point, as you remove /etc/lynx.cfg you'll get a /etc/.wh.lynx.cfg whiteout to hide a file existing in a different branch -- a read-only module. That's what whiteouts do. But as you remove /home/guest/my_useless_doc you'll get /home/guest/.wh.my_useless_doc in the same branch where the file was -- your persistent mount. It seems innecesary, doesn't it? So I'm thinking of some way to avoid saving this kind of whiteouts. But I'm not sure I'm going in the right direction. I'm far from being an expert as you have noticed. :pardon:

Thanks!

burdi01
Shogun
Shogun
Posts: 201
Joined: 18 Aug 2013, 12:09
Distribution: Slackware PartedMagic Xubuntu
Location: The Netherlands

a question about whiteouts and persistence

Post#9 by burdi01 » 18 Nov 2017, 11:13

On a first impulse I would say too that the whiteout for your /home/guest/my_useless_doc *) is unnecessary. I did a preliminary test and did not see such a whiteout -- when looking at /mnt/live/memory/changes/home/guest/ (which access is outside of the union).
You run with persistence and I do not. To be honest I never looked in detail into how Porteus implements persistence, but for now I fail to see how these whiteouts would end up in the persistence file.
:D

*) By the way: I assumed this to be a file. Please state whether this is a file or a directory.

User avatar
abelM
Black ninja
Black ninja
Posts: 50
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#10 by abelM » 18 Nov 2017, 15:38

I see we are working in different contexts. I'm saving persistence in a save.dat file at exit -- changes=EXIT:porteussave.dat in the command line. Whiteouts do appear in /mnt/live/memory/changes/home/guest/ instantly as any other modification I do in the union system. On the other hand /mnt/live/memory/images/changes/home/guest/ keeps persistence and changes are copied there just at dump-session or logout.

I'm testing filtering out whiteouts in /home/guest only to avoid messing whith system files. Until now it seems to be ok but I need more time to tell you.

Regards,

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

a question about whiteouts and persistence

Post#11 by brokenman » 18 Nov 2017, 18:48

Do whiteouts have any function in the same branch where files are deleted?
No. It is a function of how whiteouts work, but in the same branch they would serve no real purpose except to hide underlying filesystem duplicates.
How do i become super user?
Wear your underpants on the outside and put on a cape.

burdi01
Shogun
Shogun
Posts: 201
Joined: 18 Aug 2013, 12:09
Distribution: Slackware PartedMagic Xubuntu
Location: The Netherlands

a question about whiteouts and persistence

Post#12 by burdi01 » 20 Nov 2017, 17:34

I recreated abelM's problem -- or rather symptom because the whiteouts in the persistence file, though not nice, do not pose a problem for as far as I can see -- at logout as well as via dump-session.
As guest I created the "useless.txt" file in home directory. Technically speaking this file is created in the /mnt/live/memory/images/changes/home/guest/ directory. At logout this file is moved to the persistence file.
In a subsequent session I deleted that file. As the persistence file is mounted just like any other module, a whiteout in the whiteout appears in the /mnt/live/memory/images/changes/home/guest/ directory is necessary. At logout that whiteout is moved to the persistence file, while the original useless.txt file is removed.
<to be continued>
:D

User avatar
abelM
Black ninja
Black ninja
Posts: 50
Joined: 22 Oct 2017, 12:35
Distribution: Porteus v-3.2.2 64 bit
Location: Argentina
Contact:

a question about whiteouts and persistence

Post#13 by abelM » 21 Nov 2017, 00:21

That's it! I also do agree with you whiteouts seem to do no harm although they don't match any file. But you'll find a clutter in your save.dat file if you need to mloop it for some reason. You'll have to eventually remove all whiteouts -- the good and the bad. I see it's not a critical issue but I'm trying to work it out though. BTW you can look at this post if you like direct update at dump session

Regards :)
Last edited by abelM on 21 Nov 2017, 16:21, edited 1 time in total.

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

a question about whiteouts and persistence

Post#14 by brokenman » 21 Nov 2017, 08:32

Great to see empirical testing! The results should be expected. Here's a scenario ...

What would happen if you left the whiteout file in place for useless.txt, then unpacked 001 module and placed the same file there, then rebooted.
How do i become super user?
Wear your underpants on the outside and put on a cape.

burdi01
Shogun
Shogun
Posts: 201
Joined: 18 Aug 2013, 12:09
Distribution: Slackware PartedMagic Xubuntu
Location: The Netherlands

a question about whiteouts and persistence

Post#15 by burdi01 » 21 Nov 2017, 11:33

@abelM: I will have a look at that thread.

@brokenman: As I said: < to be continued > ...
The next test I planned was to "delete" a file from a "normal" xzm, e.g. guest's .config/gecko-mediaplayer/gecko-mediaplayer.conf from 003-mate.xzm. The whitout indeed is copied to the persistence file too, and in a subsquent session the .conf file turns out to be "deleted" still.
When I looked at things (2 years ago for PartedMagic) my tests would indicate that whiteouts in non-upper branches were not handled consistently. On top of that I also found the http://aufs.sourceforge.net/aufs2/shwh/README.txt document that seemed to confirm my test results.
So I chose to not copy the whiteouts to the persistence file. Actually I implemented things twice (once with and once without the CONFIG_AUFS_SHWH kernel setting). By implication the "user" files (that only occur in the persistence file) can be deleted, but the "distribution" files (from the "normal" xzm's) cannot. Actually we never saw user complaints about this, so the ability to delete "distribution" files does not seem to be in high-demand.
Obviously you chose to copy the whiteouts to the persistence file. Assuming the whiteouts in non-upper branches are now handled correctly the only thing users could complain about is that there is no easy way to undelete these "distribution" files -- as your scenario shows.
:D

Please regard these remarks a discussion and *not* as a negative comment.

Post Reply