a question about whiteouts and persistence
- abelM
- Black ninja
- Posts: 50
- Joined: 22 Oct 2017, 12:35
- Distribution: Porteus v-3.2.2 64 bit
- Location: Argentina
- Contact:
a question about whiteouts and persistence
Hi all,
I use Porteus with the changes=EXIT cheatcode and I've noticed my persistent mount is getting cluttered with whiteouts. Aufs use whiteouts to hide files when they are deleted -- it doesn't remove them really. But do whiteouts have a function in the same filesystem where files are deleted?
If I understand the point, Porteus goes through cycles like this:
- At startup it loads files from write-only modules
- It loads a write-read persistent mount where you save changes
- You write or edit some files
- At logout (or dump session) Porteus copies new or modified files to the persistent mount
When you delete files things go different:
- You delete some files
- If these files exist in modules or in the persistent mount then Aufs puts whiteouts to hide them from your sight.
- At logout (or dump session)
-- Porteus do remove files from the persistent mount if matching whiteouts exist
-- And it saves whiteouts to the persistent mount
Finally in case you name a file as a previously deleted one Porteus will remove the whiteout from the persistent mount -- if not you wouldn't see the file!
In the end the persistent mount keeps more and more whiteouts matching no files at all. So, is it necesary to copy whiteouts of deleted files coming from the persistent mount? If not you could keep the system cleaner and save some writes and space too. I'd like to know what you think about all this.
Regards,
I use Porteus with the changes=EXIT cheatcode and I've noticed my persistent mount is getting cluttered with whiteouts. Aufs use whiteouts to hide files when they are deleted -- it doesn't remove them really. But do whiteouts have a function in the same filesystem where files are deleted?
If I understand the point, Porteus goes through cycles like this:
- At startup it loads files from write-only modules
- It loads a write-read persistent mount where you save changes
- You write or edit some files
- At logout (or dump session) Porteus copies new or modified files to the persistent mount
When you delete files things go different:
- You delete some files
- If these files exist in modules or in the persistent mount then Aufs puts whiteouts to hide them from your sight.
- At logout (or dump session)
-- Porteus do remove files from the persistent mount if matching whiteouts exist
-- And it saves whiteouts to the persistent mount
Finally in case you name a file as a previously deleted one Porteus will remove the whiteout from the persistent mount -- if not you wouldn't see the file!
In the end the persistent mount keeps more and more whiteouts matching no files at all. So, is it necesary to copy whiteouts of deleted files coming from the persistent mount? If not you could keep the system cleaner and save some writes and space too. I'd like to know what you think about all this.
Regards,
- Ed_P
- Contributor
- Posts: 8360
- Joined: 06 Feb 2013, 22:12
- Distribution: Cinnamon 5.01 ISO
- Location: Western NY, USA
a question about whiteouts and persistence
I use changes=EXIT also and have occasionally experienced problems due to whiteouts in the savedat file and I run a script to delete them periodically. The vast majority being files deleted from the Firefox cache. As to whether their presence impacts speed or space significantly I don't think so.
You can purge the wh files from the mounted changes file but to be more effective it's better to clear the wh files from the actual savedat file when you run Porteus in AF mode.
My savedat file is in a folder named changes which is within a folder matching the name of the Porteus version running. Thus for 3.2.2 my savedat is:
/porteus3.2/changes/porteussave.dat. And this is the script I run to clear out the wh files.
Review and tweak to fit your needs.
You can purge the wh files from the mounted changes file but to be more effective it's better to clear the wh files from the actual savedat file when you run Porteus in AF mode.
My savedat file is in a folder named changes which is within a folder matching the name of the Porteus version running. Thus for 3.2.2 my savedat is:
/porteus3.2/changes/porteussave.dat. And this is the script I run to clear out the wh files.
Code: Select all
#!/bin/sh
# https://forum.porteus.org/viewtopic.php?f=81&t=3501&p=25151#p25162 <- USM problem
# https://forum.porteus.org/viewtopic.php?f=81&t=3501&p=25151#p25178
# https://forum.porteus.org/viewtopic.php?f=81&t=5698#p43823
# mloop requires changes= cheatcodes not be used (Always Fresh mode)
# nor modsavedat/porteussave.dat.xzm!!
BOOTDEV=`grep -A1 "Booting" /var/log/porteus-livedbg|tail -n1|sed 's^//^/^g'`
if [ "$BOOTDEV" == "/mnt/isoloop" ]; then
BOOTDEV=`grep -A1 "ISO=" /var/log/porteus-livedbg`
BOOTDEV=${BOOTDEV:4:9}
fi
VERSION=$(cat /etc/porteus-version)
FOLDER=porteus${VERSION:9:3}
GUEST="$BOOTDEV/$FOLDER/Guest"
MODULES="$BOOTDEV/$FOLDER/Modules"
# Color definitions
txtbld=$(tput bold) # Bold
txtred=${txtbld}$(tput setaf 1) # Red
rst=$(tput sgr0) # Reset
function redpswd() {
echo -e "$1" $txtred
}
if [ `whoami` != "root" ]; then
redpswd "Enter root's password"
su -c "sh $0 $1"
exit
fi
echo $rst
if [ "$1" == "" ]; then
if [ -a /mnt/live/memory/images/changes ]; then
SPACEDATDIR=/mnt/live/memory/images/changes
else
echo mloop $BOOTDEV/$FOLDER/changes/porteussave.dat
mloop $BOOTDEV/$FOLDER/changes/porteussave.dat
SPACEDATDIR=/mnt/loop
fi
else
echo mloop $1
mloop $1
SPACEDATDIR=/mnt/loop
fi
if [ ! -d ${SPACEDATDIR}/home ]; then
echo " Error!! "
dmesg | tail > t
UUIDck=$(grep -c "duplicate UUID" t)
if [ ! "$UUIDck" == "0" ]; then
echo
echo " * This is NOT Always Fresh mode. * "
echo
else
echo
echo "dmesg | tail "
dmesg | tail
fi
rm t
uloop
exit
fi
echo find ${SPACEDATDIR} -name .wh.*
find ${SPACEDATDIR} -name .wh.*
echo
echo -en "Enter 1 to delete, 2 to exit.\n"
select yn in "Yes" "No"; do
case $yn in
Yes ) echo "find ${SPACEDATDIR}/home -name .wh.* -delete"
find ${SPACEDATDIR}/home -name ".wh.*" -delete
echo
echo find ${SPACEDATDIR} -name .wh.*
find ${SPACEDATDIR} -name .wh.*
break;;
No ) break;;
esac
done
if [ -a /mnt/live/memory/images/changes ]; then
exit
else
uloop
fi
exit
Ed
- abelM
- Black ninja
- Posts: 50
- Joined: 22 Oct 2017, 12:35
- Distribution: Porteus v-3.2.2 64 bit
- Location: Argentina
- Contact:
a question about whiteouts and persistence
Thank you Ed! I'm trying your script.
Regards,
Regards,
- abelM
- Black ninja
- Posts: 50
- Joined: 22 Oct 2017, 12:35
- Distribution: Porteus v-3.2.2 64 bit
- Location: Argentina
- Contact:
a question about whiteouts and persistence
Well, besides cleaning my persistent mount using Ed's script, I still have interest to know if whiteouts have a function in the same filesystem -- I'd better say in the same branch -- where matching files have been deleted. In the context of Porteus usage it seems they don't. If so it would be nice to avoid saving them.
I hope to be clear, my English is not so good.
Regards,
I hope to be clear, my English is not so good.
Regards,
- Ed_P
- Contributor
- Posts: 8360
- Joined: 06 Feb 2013, 22:12
- Distribution: Cinnamon 5.01 ISO
- Location: Western NY, USA
a question about whiteouts and persistence
abelM don't forget to check out the links at the top of my script.
Strange save.dat behavior [Solved ]
Strange save.dat behavior [Solved ]
Help!! Filenames with blanks in them
Strange save.dat behavior [Solved ]
Strange save.dat behavior [Solved ]
Help!! Filenames with blanks in them
Ed
- abelM
- Black ninja
- Posts: 50
- Joined: 22 Oct 2017, 12:35
- Distribution: Porteus v-3.2.2 64 bit
- Location: Argentina
- Contact:
a question about whiteouts and persistence
Hi Ed! I've read the posts you've suggested. All interesting. I've taken your script as reference and worked it out in a primitive way
I've booted Porteus in AF mode, opened a terminal and done this
I see this is a task I should do from time to time. Anyway I'll try to find the way of avoiding saving whiteouts which have no function in the persistent mount. I'll tell you later.
And thank you very much!
I've booted Porteus in AF mode, opened a terminal and done this
Code: Select all
su
mloop
cd /mnt/loop
find -name "*.wh.*" -delete
cd ..
uloop
exit
And thank you very much!
-
- Shogun
- Posts: 205
- Joined: 18 Aug 2013, 12:09
- Distribution: Slackware PartedMagic Xubuntu
- Location: The Netherlands
a question about whiteouts and persistence
Actually aufs has an option to handle the whiteouts in a "save session" feature.
See http://aufs.sourceforge.net/aufs2/shwh/README.txt for details.
If you have access to a not-too-old PartedMagic release see the /usr/bin/pmagic_save_session.v2 script for my implementation.
See http://aufs.sourceforge.net/aufs2/shwh/README.txt for details.
If you have access to a not-too-old PartedMagic release see the /usr/bin/pmagic_save_session.v2 script for my implementation.
- abelM
- Black ninja
- Posts: 50
- Joined: 22 Oct 2017, 12:35
- Distribution: Porteus v-3.2.2 64 bit
- Location: Argentina
- Contact:
a question about whiteouts and persistence
Hi burdi01,
I've read the document -- interesting indeed but I'm asking something different. Do whiteouts have any function in the same branch where files are deleted? It seems not just because there's nothing to hide there. The case is completely different when whiteouts hide files existing in a different branch.
If I'm understanding the point, as you remove /etc/lynx.cfg you'll get a /etc/.wh.lynx.cfg whiteout to hide a file existing in a different branch -- a read-only module. That's what whiteouts do. But as you remove /home/guest/my_useless_doc you'll get /home/guest/.wh.my_useless_doc in the same branch where the file was -- your persistent mount. It seems innecesary, doesn't it? So I'm thinking of some way to avoid saving this kind of whiteouts. But I'm not sure I'm going in the right direction. I'm far from being an expert as you have noticed.
Thanks!
I've read the document -- interesting indeed but I'm asking something different. Do whiteouts have any function in the same branch where files are deleted? It seems not just because there's nothing to hide there. The case is completely different when whiteouts hide files existing in a different branch.
If I'm understanding the point, as you remove /etc/lynx.cfg you'll get a /etc/.wh.lynx.cfg whiteout to hide a file existing in a different branch -- a read-only module. That's what whiteouts do. But as you remove /home/guest/my_useless_doc you'll get /home/guest/.wh.my_useless_doc in the same branch where the file was -- your persistent mount. It seems innecesary, doesn't it? So I'm thinking of some way to avoid saving this kind of whiteouts. But I'm not sure I'm going in the right direction. I'm far from being an expert as you have noticed.
Thanks!
-
- Shogun
- Posts: 205
- Joined: 18 Aug 2013, 12:09
- Distribution: Slackware PartedMagic Xubuntu
- Location: The Netherlands
a question about whiteouts and persistence
On a first impulse I would say too that the whiteout for your /home/guest/my_useless_doc *) is unnecessary. I did a preliminary test and did not see such a whiteout -- when looking at /mnt/live/memory/changes/home/guest/ (which access is outside of the union).
You run with persistence and I do not. To be honest I never looked in detail into how Porteus implements persistence, but for now I fail to see how these whiteouts would end up in the persistence file.
*) By the way: I assumed this to be a file. Please state whether this is a file or a directory.
You run with persistence and I do not. To be honest I never looked in detail into how Porteus implements persistence, but for now I fail to see how these whiteouts would end up in the persistence file.
*) By the way: I assumed this to be a file. Please state whether this is a file or a directory.
- abelM
- Black ninja
- Posts: 50
- Joined: 22 Oct 2017, 12:35
- Distribution: Porteus v-3.2.2 64 bit
- Location: Argentina
- Contact:
a question about whiteouts and persistence
I see we are working in different contexts. I'm saving persistence in a save.dat file at exit -- changes=EXIT:porteussave.dat in the command line. Whiteouts do appear in /mnt/live/memory/changes/home/guest/ instantly as any other modification I do in the union system. On the other hand /mnt/live/memory/images/changes/home/guest/ keeps persistence and changes are copied there just at dump-session or logout.
I'm testing filtering out whiteouts in /home/guest only to avoid messing whith system files. Until now it seems to be ok but I need more time to tell you.
Regards,
I'm testing filtering out whiteouts in /home/guest only to avoid messing whith system files. Until now it seems to be ok but I need more time to tell you.
Regards,
- brokenman
- Site Admin
- Posts: 6105
- Joined: 27 Dec 2010, 03:50
- Distribution: Porteus v4 all desktops
- Location: Brazil
a question about whiteouts and persistence
No. It is a function of how whiteouts work, but in the same branch they would serve no real purpose except to hide underlying filesystem duplicates.Do whiteouts have any function in the same branch where files are deleted?
How do i become super user?
Wear your underpants on the outside and put on a cape.
Wear your underpants on the outside and put on a cape.
-
- Shogun
- Posts: 205
- Joined: 18 Aug 2013, 12:09
- Distribution: Slackware PartedMagic Xubuntu
- Location: The Netherlands
a question about whiteouts and persistence
I recreated abelM's problem -- or rather symptom because the whiteouts in the persistence file, though not nice, do not pose a problem for as far as I can see -- at logout as well as via dump-session.
As guest I created the "useless.txt" file in home directory. Technically speaking this file is created in the /mnt/live/memory/images/changes/home/guest/ directory. At logout this file is moved to the persistence file.
In a subsequent session I deleted that file. As the persistence file is mounted just like any other module, a whiteout in the whiteout appears in the /mnt/live/memory/images/changes/home/guest/ directory is necessary. At logout that whiteout is moved to the persistence file, while the original useless.txt file is removed.
<to be continued>
As guest I created the "useless.txt" file in home directory. Technically speaking this file is created in the /mnt/live/memory/images/changes/home/guest/ directory. At logout this file is moved to the persistence file.
In a subsequent session I deleted that file. As the persistence file is mounted just like any other module, a whiteout in the whiteout appears in the /mnt/live/memory/images/changes/home/guest/ directory is necessary. At logout that whiteout is moved to the persistence file, while the original useless.txt file is removed.
<to be continued>
- abelM
- Black ninja
- Posts: 50
- Joined: 22 Oct 2017, 12:35
- Distribution: Porteus v-3.2.2 64 bit
- Location: Argentina
- Contact:
a question about whiteouts and persistence
That's it! I also do agree with you whiteouts seem to do no harm although they don't match any file. But you'll find a clutter in your save.dat file if you need to mloop it for some reason. You'll have to eventually remove all whiteouts -- the good and the bad. I see it's not a critical issue but I'm trying to work it out though. BTW you can look at this post if you like direct update at dump session
Regards
Regards
Last edited by abelM on 21 Nov 2017, 16:21, edited 1 time in total.
- brokenman
- Site Admin
- Posts: 6105
- Joined: 27 Dec 2010, 03:50
- Distribution: Porteus v4 all desktops
- Location: Brazil
a question about whiteouts and persistence
Great to see empirical testing! The results should be expected. Here's a scenario ...
What would happen if you left the whiteout file in place for useless.txt, then unpacked 001 module and placed the same file there, then rebooted.
What would happen if you left the whiteout file in place for useless.txt, then unpacked 001 module and placed the same file there, then rebooted.
How do i become super user?
Wear your underpants on the outside and put on a cape.
Wear your underpants on the outside and put on a cape.
-
- Shogun
- Posts: 205
- Joined: 18 Aug 2013, 12:09
- Distribution: Slackware PartedMagic Xubuntu
- Location: The Netherlands
a question about whiteouts and persistence
@abelM: I will have a look at that thread.
@brokenman: As I said: < to be continued > ...
The next test I planned was to "delete" a file from a "normal" xzm, e.g. guest's .config/gecko-mediaplayer/gecko-mediaplayer.conf from 003-mate.xzm. The whitout indeed is copied to the persistence file too, and in a subsquent session the .conf file turns out to be "deleted" still.
When I looked at things (2 years ago for PartedMagic) my tests would indicate that whiteouts in non-upper branches were not handled consistently. On top of that I also found the http://aufs.sourceforge.net/aufs2/shwh/README.txt document that seemed to confirm my test results.
So I chose to not copy the whiteouts to the persistence file. Actually I implemented things twice (once with and once without the CONFIG_AUFS_SHWH kernel setting). By implication the "user" files (that only occur in the persistence file) can be deleted, but the "distribution" files (from the "normal" xzm's) cannot. Actually we never saw user complaints about this, so the ability to delete "distribution" files does not seem to be in high-demand.
Obviously you chose to copy the whiteouts to the persistence file. Assuming the whiteouts in non-upper branches are now handled correctly the only thing users could complain about is that there is no easy way to undelete these "distribution" files -- as your scenario shows.
Please regard these remarks a discussion and *not* as a negative comment.
@brokenman: As I said: < to be continued > ...
The next test I planned was to "delete" a file from a "normal" xzm, e.g. guest's .config/gecko-mediaplayer/gecko-mediaplayer.conf from 003-mate.xzm. The whitout indeed is copied to the persistence file too, and in a subsquent session the .conf file turns out to be "deleted" still.
When I looked at things (2 years ago for PartedMagic) my tests would indicate that whiteouts in non-upper branches were not handled consistently. On top of that I also found the http://aufs.sourceforge.net/aufs2/shwh/README.txt document that seemed to confirm my test results.
So I chose to not copy the whiteouts to the persistence file. Actually I implemented things twice (once with and once without the CONFIG_AUFS_SHWH kernel setting). By implication the "user" files (that only occur in the persistence file) can be deleted, but the "distribution" files (from the "normal" xzm's) cannot. Actually we never saw user complaints about this, so the ability to delete "distribution" files does not seem to be in high-demand.
Obviously you chose to copy the whiteouts to the persistence file. Assuming the whiteouts in non-upper branches are now handled correctly the only thing users could complain about is that there is no easy way to undelete these "distribution" files -- as your scenario shows.
Please regard these remarks a discussion and *not* as a negative comment.