Hi, I'm just trying to figure out a memory usage issue I'm having. It seems writing to the file system (aufs?) shows up as "cached" memory in top. It is confusing because cached memory is usually freed as needed, but this cannot be freed on demand? So the result is I'm unable to tell how much actual free memory is left.
Any pointers? or is there a way to have it not show up as cached memory? maybe just kernel memory, or a processes memory?
In a maybe or maybe not related topic, I custom built a 3.3.3 kernel, just added a few extra features, but I can never get swap space to be used. The system crashes, out of memory, before it will swap. Swap just always shows as 0k used. And yes it is definently on
Cheers, Ryan
Porteus file system memory shows up as cache memory?
- fanthom
- Moderator Team
- Posts: 5667
- Joined: 28 Dec 2010, 02:42
- Distribution: Porteus Kiosk
- Location: Poland
- Contact:
Re: Porteus file system memory shows up as cache memory?
hello Ryan,
1) "It seems writing to the file system (aufs?) shows up as "cached" memory in top."
yes - and this is normal for all linuxes out there, not only porteus. every copying operation leaves it's "shadow" in RAM (a cache) just in case you want to repeat copying the same object to another place.
2) "It is confusing because cached memory is usually freed as needed"
yes - it's designed to work like that.
3) "but this cannot be freed on demand?"
please run 'echo 3 > /proc/sys/vm/drop_caches' to clear your cache completely (but dropping the cache makes no sense to me as per second point)
4) "So the result is I'm unable to tell how much actual free memory is left"
a) if you are using 'changes=' cheatcode then aufs writable branch is set on a real filesystem (not in RAM) so it's enough to run:
and subtract the result (value is present in megabytes) from your total RAM amount.
b) if you are not using 'changes=' cheatcode then you must add amount of RAM taken by applications to amount of RAM taken by aufs, here is the code:
and subtract the result (value is present in megabytes) from your total RAM amount.
no need to drop the caches at any point to tell how much of free RAM you currently have
5) "In a maybe or maybe not related topic, I custom built a 3.3.3 kernel, just added a few extra features, but I can never get swap space to be used. The system crashes, out of memory, before it will swap. Swap just always shows as 0k used. And yes it is definently on"
i think you are talking about a situation when you are not using a 'changes=' cheat and aufs is filled up completely.
at this point you could get "out of space" error message and system "could crash" as being not able to write anything to /tmp directory (this applies to every other linux as well).
if 'changes=' cheatcode is not used then at the boot time aufs is reserving 60% of RAM (this can be tune-able with ramsize= cheatcode, check /boot/docs/cheatcodes.txt for details) for the "/" (root) partition. this partition has a static size which cannot be expandable with a swap (it acts as every other normal partition).
swapping is done only when necessary so seems that you have filled up "/" (root) partition but system still had enough free memory (40% or RAM) to continue working without swapping.
i have done a simple test on my PC: filled up aufs completely, opened as many apps as i could in kde-4 and ran 'free -m' command just to find that only 46MB were used on the swap partition.
if your hardware is ok then it's really hard to take porteus down
1) "It seems writing to the file system (aufs?) shows up as "cached" memory in top."
yes - and this is normal for all linuxes out there, not only porteus. every copying operation leaves it's "shadow" in RAM (a cache) just in case you want to repeat copying the same object to another place.
2) "It is confusing because cached memory is usually freed as needed"
yes - it's designed to work like that.
3) "but this cannot be freed on demand?"
please run 'echo 3 > /proc/sys/vm/drop_caches' to clear your cache completely (but dropping the cache makes no sense to me as per second point)
4) "So the result is I'm unable to tell how much actual free memory is left"
a) if you are using 'changes=' cheatcode then aufs writable branch is set on a real filesystem (not in RAM) so it's enough to run:
Code: Select all
free -m | grep buffers/cache | awk '{print $3}'
b) if you are not using 'changes=' cheatcode then you must add amount of RAM taken by applications to amount of RAM taken by aufs, here is the code:
Code: Select all
expr $(expr `free | grep buffers/cache | awk '{print $3}'` + `df | grep aufs | awk '{print $3}'`) / 1024
no need to drop the caches at any point to tell how much of free RAM you currently have
5) "In a maybe or maybe not related topic, I custom built a 3.3.3 kernel, just added a few extra features, but I can never get swap space to be used. The system crashes, out of memory, before it will swap. Swap just always shows as 0k used. And yes it is definently on"
i think you are talking about a situation when you are not using a 'changes=' cheat and aufs is filled up completely.
at this point you could get "out of space" error message and system "could crash" as being not able to write anything to /tmp directory (this applies to every other linux as well).
if 'changes=' cheatcode is not used then at the boot time aufs is reserving 60% of RAM (this can be tune-able with ramsize= cheatcode, check /boot/docs/cheatcodes.txt for details) for the "/" (root) partition. this partition has a static size which cannot be expandable with a swap (it acts as every other normal partition).
swapping is done only when necessary so seems that you have filled up "/" (root) partition but system still had enough free memory (40% or RAM) to continue working without swapping.
i have done a simple test on my PC: filled up aufs completely, opened as many apps as i could in kde-4 and ran 'free -m' command just to find that only 46MB were used on the swap partition.
if your hardware is ok then it's really hard to take porteus down
Please add [Solved] to your thread title if the solution was found.
Re: Porteus file system memory shows up as cache memory?
Hi, Thanks for the reply!
Sorry, I Should have said I'm running in always fresh mode, so I guess that means not with the changes= cheatcode.
So it is normal behaviour for file system space in this instance showing up as "cached" memory, even though it isn't actually a cache?
So I think the "free" memory is actually how much is free. Where normally it would be "free" + "cached" which is usable by applications. Is that right?
So 4GB - 2678 = approx 1.3G
So that works out
So it seems it dropped 500MB, which would have been actual fs cache, but the other 2.2 is from aufs?
Theres the 2.2
Sorry I hope that makes sense . It's ok if this is how it is meant to work, it just seems a bit confusing to me? Because I'm used to treating cache as free memory... I think it would be more appropriately taked into account in "used" memory?
Posted after 30 seconds:
Please Note: I didn't have any swap enabled there
Sorry, I Should have said I'm running in always fresh mode, so I guess that means not with the changes= cheatcode.
So it is normal behaviour for file system space in this instance showing up as "cached" memory, even though it isn't actually a cache?
Code: Select all
total used free shared buffers cached
Mem: 4052456 1342016 2710440 0 99872 735440
-/+ buffers/cache: 506704 3545752
Swap: 0 0 0
Code: Select all
dd if=/dev/zero of=/tmp/blah bs=1M count=2000
Code: Select all
total used free shared buffers cached
Mem: 4052456 3406128 646328 0 99872 2783460
-/+ buffers/cache: 522796 3529660
Swap: 0 0 0
Code: Select all
expr $(expr `free | grep buffers/cache | awk '{print $3}'` + `df | grep aufs | awk '{print $3}'`) / 1024
Code: Select all
2678
Code: Select all
echo 3 > /proc/sys/vm/drop_caches
Code: Select all
total used free shared buffers cached
Mem: 4052456 2753200 1299256 0 388 2248532
-/+ buffers/cache: 504280 3548176
Swap: 0 0 0
So it seems it dropped 500MB, which would have been actual fs cache, but the other 2.2 is from aufs?
Code: Select all
df -h
Filesystem Size Used Avail Use% Mounted on
aufs 2.4G 2.2G 220M 91% /
Sorry I hope that makes sense . It's ok if this is how it is meant to work, it just seems a bit confusing to me? Because I'm used to treating cache as free memory... I think it would be more appropriately taked into account in "used" memory?
Posted after 30 seconds:
Please Note: I didn't have any swap enabled there
- fanthom
- Moderator Team
- Posts: 5667
- Joined: 28 Dec 2010, 02:42
- Distribution: Porteus Kiosk
- Location: Poland
- Contact:
Re: Porteus file system memory shows up as cache memory?
you are right, files from aufs are treated as a cache (data which can be dropped when its necessary) while in fact they can't be removed from RAM.
i have tested this behavior also on my gentoo box:
a) mounted tmpfs in /tmp folder
b) copied porteus.iso to /tmp
c) ran 'echo 3 > /proc/sys/vm/drop_caches'
and porteus.iso size was still counted as a cache.
i guess that's a bug in the 'free' utility which i wasn't aware of. thanks for sharing
to answer questions from your first post directly:
which adds amount of RAM currently addressed by the applications to data held in aufs rw branch (which is set on tmpfs when changes= is not present/or valid).
Cheers
i have tested this behavior also on my gentoo box:
a) mounted tmpfs in /tmp folder
b) copied porteus.iso to /tmp
c) ran 'echo 3 > /proc/sys/vm/drop_caches'
and porteus.iso size was still counted as a cache.
i guess that's a bug in the 'free' utility which i wasn't aware of. thanks for sharing
to answer questions from your first post directly:
true - as per our findings in this thread.It seems writing to the file system (aufs?) shows up as "cached" memory in top. It is confusing because cached memory is usually freed as needed, but this cannot be freed on demand?
you can. in case of running in 'Always Fresh' mode the only reliable way of telling how much of free RAM you have is to run:So the result is I'm unable to tell how much actual free memory is left.
Code: Select all
expr $(expr `free | grep buffers/cache | awk '{print $3}'` + `df | grep aufs | awk '{print $3}'`) / 1024
Cheers
Please add [Solved] to your thread title if the solution was found.