Porteus file system memory shows up as cache memory?

[Kartweel]

Hi, I'm just trying to figure out a memory usage issue I'm having. It seems writing to the file system (aufs?) shows up as "cached" memory in top. It is confusing because cached memory is usually freed as needed, but this cannot be freed on demand? So the result is I'm unable to tell how much actual free memory is left.

Any pointers? or is there a way to have it not show up as cached memory? maybe just kernel memory, or a processes memory?

In a maybe or maybe not related topic, I custom built a 3.3.3 kernel, just added a few extra features, but I can never get swap space to be used. The system crashes, out of memory, before it will swap. Swap just always shows as 0k used. And yes it is definently on

Cheers, Ryan

[fanthom]

hello Ryan,

1) "It seems writing to the file system (aufs?) shows up as "cached" memory in top."
yes - and this is normal for all linuxes out there, not only porteus. every copying operation leaves it's "shadow" in RAM (a cache) just in case you want to repeat copying the same object to another place.

2) "It is confusing because cached memory is usually freed as needed"
yes - it's designed to work like that.

3) "but this cannot be freed on demand?"
please run 'echo 3 > /proc/sys/vm/drop_caches' to clear your cache completely (but dropping the cache makes no sense to me as per second point)

4) "So the result is I'm unable to tell how much actual free memory is left"
a) if you are using 'changes=' cheatcode then aufs writable branch is set on a real filesystem (not in RAM) so it's enough to run:

Code: Select all

free -m | grep buffers/cache | awk '{print $3}'

and subtract the result (value is present in megabytes) from your total RAM amount.
b) if you are not using 'changes=' cheatcode then you must add amount of RAM taken by applications to amount of RAM taken by aufs, here is the code:

Code: Select all

expr $(expr `free | grep buffers/cache | awk '{print $3}'` + `df | grep aufs | awk '{print $3}'`) / 1024

and subtract the result (value is present in megabytes) from your total RAM amount.

no need to drop the caches at any point to tell how much of free RAM you currently have

5) "In a maybe or maybe not related topic, I custom built a 3.3.3 kernel, just added a few extra features, but I can never get swap space to be used. The system crashes, out of memory, before it will swap. Swap just always shows as 0k used. And yes it is definently on"
i think you are talking about a situation when you are not using a 'changes=' cheat and aufs is filled up completely.
at this point you could get "out of space" error message and system "could crash" as being not able to write anything to /tmp directory (this applies to every other linux as well).

if 'changes=' cheatcode is not used then at the boot time aufs is reserving 60% of RAM (this can be tune-able with ramsize= cheatcode, check /boot/docs/cheatcodes.txt for details) for the "/" (root) partition. this partition has a static size which cannot be expandable with a swap (it acts as every other normal partition).

swapping is done only when necessary so seems that you have filled up "/" (root) partition but system still had enough free memory (40% or RAM) to continue working without swapping.

i have done a simple test on my PC: filled up aufs completely, opened as many apps as i could in kde-4 and ran 'free -m' command just to find that only 46MB were used on the swap partition.

if your hardware is ok then it's really hard to take porteus down

[Kartweel]

Hi, Thanks for the reply!

Sorry, I Should have said I'm running in always fresh mode, so I guess that means not with the changes= cheatcode.

So it is normal behaviour for file system space in this instance showing up as "cached" memory, even though it isn't actually a cache?

Code: Select all

             total       used       free     shared    buffers     cached
Mem:       4052456    1342016    2710440          0      99872     735440
-/+ buffers/cache:     506704    3545752
Swap:            0          0          0

Code: Select all

dd if=/dev/zero of=/tmp/blah bs=1M count=2000

Code: Select all

             total       used       free     shared    buffers     cached
Mem:       4052456    3406128     646328          0      99872    2783460
-/+ buffers/cache:     522796    3529660
Swap:            0          0          0

So I think the "free" memory is actually how much is free. Where normally it would be "free" + "cached" which is usable by applications. Is that right?

Code: Select all

expr $(expr `free | grep buffers/cache | awk '{print $3}'` + `df | grep aufs | awk '{print $3}'`) / 1024

Code: Select all

2678

So 4GB - 2678 = approx 1.3G

Code: Select all

echo 3 > /proc/sys/vm/drop_caches

Code: Select all

             total       used       free     shared    buffers     cached
Mem:       4052456    2753200    1299256          0        388    2248532
-/+ buffers/cache:     504280    3548176
Swap:            0          0          0

So that works out

So it seems it dropped 500MB, which would have been actual fs cache, but the other 2.2 is from aufs?

Code: Select all

 df -h
Filesystem        Size  Used Avail Use% Mounted on
aufs              2.4G  2.2G  220M  91% /

Theres the 2.2

Sorry I hope that makes sense . It's ok if this is how it is meant to work, it just seems a bit confusing to me? Because I'm used to treating cache as free memory... I think it would be more appropriately taked into account in "used" memory?

Posted after 30 seconds:
Please Note: I didn't have any swap enabled there

[fanthom]

you are right, files from aufs are treated as a cache (data which can be dropped when its necessary) while in fact they can't be removed from RAM.

i have tested this behavior also on my gentoo box:
a) mounted tmpfs in /tmp folder
b) copied porteus.iso to /tmp
c) ran 'echo 3 > /proc/sys/vm/drop_caches'
and porteus.iso size was still counted as a cache.
i guess that's a bug in the 'free' utility which i wasn't aware of. thanks for sharing

to answer questions from your first post directly:

It seems writing to the file system (aufs?) shows up as "cached" memory in top. It is confusing because cached memory is usually freed as needed, but this cannot be freed on demand?

true - as per our findings in this thread.

So the result is I'm unable to tell how much actual free memory is left.

you can. in case of running in 'Always Fresh' mode the only reliable way of telling how much of free RAM you have is to run:

Code: Select all

expr $(expr `free | grep buffers/cache | awk '{print $3}'` + `df | grep aufs | awk '{print $3}'`) / 1024

which adds amount of RAM currently addressed by the applications to data held in aufs rw branch (which is set on tmpfs when changes= is not present/or valid).

Cheers