SAVE.DAT, XZM & MLOOP

Technical issues/questions of an intermediate or advanced nature.
Bogomips
Full of knowledge
Full of knowledge
Posts: 2564
Joined: 25 Jun 2014, 15:21
Distribution: 3.2.2 Cinnamon & KDE5
Location: London

rc4 Savefile manager

Post#31 by Bogomips » 28 Jul 2016, 16:03

Not able to enter /Path/o/Folder/savefile.dat. Having to browse. picked a folder out, but not allowed as in Porteus live filesystem, then a vfat partition not mouinted and get same message. Lastly usb stick not mounted and available from browse dialog, but to no aviail :wall: also supposed to be in live filesystem. :unknown:

Maybe one has to be running Porteus from FAT/NTFS filesystem in order to be able to create a savefile, could be the reason.
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

Re: SAVE.DAT, XZM & MLOOP

Post#32 by Ed_P » 29 Jul 2016, 02:25

Bogomips wrote:[*]Size

Code: Select all

guest@porteus:~$ ls --block-size=M -l p10/tmp/tst/Eds.xzm
-rw-r--r-- 1 guest users 16M Jul 26 22:24 p10/tmp/tst/Eds.xzm
guest@porteus:~$ z=$(ls --block-size=M -l p10/tmp/tst/Eds.xzm)
-rw-r--r-- 1 guest users 16M Jul 26 22:24 p10/tmp/tst/Eds.xzm
guest@porteus:~$ z=${z%%M*}; echo $z
-rw-r--r-- 1 guest users 16
guest@porteus:~$ z=${z##* }; echo $z
16
Tried to use some of your %% and ## commands and ended up with this version.

Code: Select all

#!/bin/sh
# Add to /etc/rc.d/rc.local_shutdown.  And make rc.local_shutdown executable.

# Color definitions
txtbld=$(tput bold)               # Bold
txtred=${txtbld}$(tput setaf 1)   # Red
rst=$(tput sgr0)                  # Reset

function redwarning() {
echo -e $txtred "$1" $rst
}

if [ -a /mnt/live/memory/images/changes ]; then
   SDspace=$(df -h /mnt/live/memory/images/changes)
   SDspace=${SDspace%% /*}
   SDspace=${SDspace:50}
   if [ "${SDspace:33:-1}" -gt "85" ]; then
      echo 
      redwarning "Warning! save.dat space: Used/Free - ${SDspace:20:16}"
      echo 
   else
      echo "save.dat space: Used/Free - ${SDspace:20:16}"
   fi
fi
Hopefully it will work with larger save.dat files.
Ed

Bogomips
Full of knowledge
Full of knowledge
Posts: 2564
Joined: 25 Jun 2014, 15:21
Distribution: 3.2.2 Cinnamon & KDE5
Location: London

Re: SAVE.DAT, XZM & MLOOP

Post#33 by Bogomips » 29 Jul 2016, 20:17

@Ed_P
Can't make any comments, as can't see output of

Code: Select all

df -h /mnt/live/memory/images/changes
Myself not having this directory.

Also in rc4 are you able to make save file, using Savefile manager?

P.S. After bit of research, trying analogous solution :wink:

Code: Select all

guest@porteus:~$ df -BM /mnt/live/memory/images/
Filesystem     1M-blocks  Used Available Use% Mounted on
tmpfs               437M  284M      153M  66% /mnt/live
guest@porteus:~$ df -BM /mnt/live/memory/images/ | tail -n1
tmpfs               437M  284M      153M  66% /mnt/live

guest@porteus:~$ df -BM --output=size,used,avail /mnt/live/memory/images
1M-blocks  Used Avail
     437M  284M  153M
guest@porteus:~$ t=$(df -BM --output=size,used,avail /mnt/live/memory/images|tail -n1); echo $t
437M 284M 153M
guest@porteus:~$ read s u a <<< $t
guest@porteus:~$ echo ${s%M}
437
guest@porteus:~$ echo ${u%M}
284
guest@porteus:~$ echo ${a%M}
153
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

Re: SAVE.DAT, XZM & MLOOP

Post#34 by Ed_P » 29 Jul 2016, 22:37

You my friend are absolutely unbelievable!!. :shock:

Code: Select all

guest@porteus:~$ df -h --output=size,used,avail,pcent /mnt/live/memory/images/changes | tail -n1
 508M  252M  257M  50%

:Yahoo!:
Bogomips wrote:Also in rc4 are you able to make save file, using Savefile manager?
Yes.

savedatspaceck.sh

Code: Select all

#!/bin/sh
# Add to /etc/rc.d/rc.local_shutdown.  And make rc.local_shutdown executable.
# Bogomips help: http://forum.porteus.org/viewtopic.php?f=140&t=6069&p=47591#p47589
# Color definitions
txtbld=$(tput bold)               # Bold
txtred=${txtbld}$(tput setaf 1)   # Red
rst=$(tput sgr0)                  # Reset

function redwarning() {
echo -e $txtred "$1" $rst
}

if [ -a /mnt/live/memory/images/changes ]; then
   SDspace=$(df -h --output=size,used,avail,pcent /mnt/live/memory/images/changes | tail -n1)
   read size used avail percent <<< $SDspace
   if [ "${percent::2}" -gt "85" ]; then
      echo 
      redwarning "Warning! save.dat space: Used/Free - $used/$avail $percent"
      echo 
   else
      echo "save.dat space: Used/Free - $used/$avail $percent"
   fi
fi
Ed

Bogomips
Full of knowledge
Full of knowledge
Posts: 2564
Joined: 25 Jun 2014, 15:21
Distribution: 3.2.2 Cinnamon & KDE5
Location: London

Re: SAVE.DAT, XZM & MLOOP

Post#35 by Bogomips » 29 Jul 2016, 23:28

:oops: Don't use -h, Use -BM. Otherwise you can't do > 999M, as it will change to G at some point thereafter.
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

Re: SAVE.DAT, XZM & MLOOP

Post#36 by Ed_P » 30 Jul 2016, 02:38

Bogomips wrote::oops: Don't use -h, Use -BM. Otherwise you can't do > 999M, as it will change to G at some point thereafter.
OK. But at this point with the 4 fields separated into separate values I'm not sure it matters. The key field, pcent, is now a separate value that can be tested without knowing how far into the df display it's displacement is.

savedatspaceck.sh

Code: Select all

#!/bin/sh
# Add to /etc/rc.d/rc.local_shutdown.  And make rc.local_shutdown executable.
# Bogomips help: http://forum.porteus.org/viewtopic.php?f=140&t=6069&p=47591#p47589
# Color definitions
txtbld=$(tput bold)               # Bold
txtred=${txtbld}$(tput setaf 1)   # Red
rst=$(tput sgr0)                  # Reset

function redwarning() {
echo -e $txtred "$1" $rst
}

if [ -a /mnt/live/memory/images/changes ]; then
   SDspace=$(df -BM --output=size,used,avail,pcent /mnt/live/memory/images/changes | tail -n1)
   read size used avail percent <<< $SDspace
   if [ "$percent" > "85%" ]; then
      echo 
      redwarning "Warning! save.dat space: Used/Free - $used/$avail $percent"
      echo 
   else
      echo "save.dat space: Used/Free - $used/$avail $percent"
   fi
fi
Ed

Bogomips
Full of knowledge
Full of knowledge
Posts: 2564
Joined: 25 Jun 2014, 15:21
Distribution: 3.2.2 Cinnamon & KDE5
Location: London

Re: SAVE.DAT, XZM & MLOOP

Post#37 by Bogomips » 30 Jul 2016, 11:11

Code: Select all

guest@porteus:~$ [[ 100% < 85% ]] && echo false
false
:)

Better

Code: Select all

if [[ ${percent%\%} -gt  85 ]]; then
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

Re: SAVE.DAT, XZM & MLOOP

Post#38 by Ed_P » 30 Jul 2016, 15:52

Bogomips wrote:

Code: Select all

guest@porteus:~$ [[ 100% < 85% ]] && echo false
false
:ROFL:
I actually wondered about that scenario but figured the odds were so small....
Better

Code: Select all

if [[ ${percent%\%} -gt  85 ]]; then
Thank you again oh coding master. :)

BTW What does one call the %\% and %%M and ## characters you use? I tried searching Google for them but since I didn't know what they were called didn't find anything helpful.
Ed

Bogomips
Full of knowledge
Full of knowledge
Posts: 2564
Joined: 25 Jun 2014, 15:21
Distribution: 3.2.2 Cinnamon & KDE5
Location: London

Re: SAVE.DAT, XZM & MLOOP

Post#39 by Bogomips » 30 Jul 2016, 19:22

Ed_P wrote:BTW What does one call the %\% and %%M and ## characters you use? I tried searching Google for them but since I didn't know what they were called didn't find anything helpful.
From my notes file:

Code: Select all

       ${parameter#word}
       ${parameter##word}
              Remove matching prefix pattern.  The word is expanded to produce
              a pattern just as in pathname expansion.  If the pattern matches
              the  beginning of the value of parameter, then the result of the
              expansion is the expanded value of parameter with  the  shortest
              matching  pattern  (the  ``#''  case)  or  the  longest matching
              pattern (the ``##'' case) deleted.  If parameter is @ or *,  the
              pattern   removal   operation  is  applied  to  each  positional
              parameter in turn, and the expansion is the resultant list.   If
              parameter  is  an  array  variable  subscripted with @ or *, the
              pattern removal operation is applied to each member of the array
              in turn, and the expansion is the resultant list.

       ${parameter%word}
       ${parameter%%word}
              Remove matching suffix pattern.  The word is expanded to produce
              a pattern just as in pathname expansion.  If the pattern matches
              a  trailing portion of the expanded value of parameter, then the
              result of the expansion is the expanded value of parameter  with
              the  shortest  matching  pattern (the ``%'' case) or the longest
              matching pattern (the ``%%'' case) deleted.  If parameter  is  @
              or   *,  the  pattern  removal  operation  is  applied  to  each
              positional parameter in turn, and the expansion is the resultant
              list.   If  parameter is an array variable subscripted with @ or
              *, the pattern removal operation is applied to  each  member  of
              the array in turn, and the expansion is the resultant list.
 
Or normally in browser man.cx/bash, then search for ## or %%.
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

User avatar
Ed_P
Contributor
Contributor
Posts: 8341
Joined: 06 Feb 2013, 22:12
Distribution: Cinnamon 5.01 ISO
Location: Western NY, USA

SAVE.DAT, XZM & MLOOP

Post#40 by Ed_P » 31 Jul 2016, 03:13

Bogomips wrote:From my notes file:

Code: Select all

       ${parameter#word}
       ${parameter##word}
  :
  :
 etc
Or normally in browser man.cx/bash, then search for ## or %%.
Wow!! Thank you Bogomips. :friends:

IMO This has been a great project. We developed something useful for save.dat users and I learned a ton of new stuff. :good:

:beer:

savedatspaceck.sh

Code: Select all

#!/bin/sh
# Add to /etc/rc.d/rc.local_shutdown.  And make rc.local_shutdown executable.
# Bogomips help: http://forum.porteus.org/viewtopic.php?f=140&t=6069&p=47591#p47589
#                http://forum.porteus.org/viewtopic.php?f=140&t=6069&p=47614#p47598
# Color definitions
txtbld=$(tput bold)               # Bold
txtred=${txtbld}$(tput setaf 1)   # Red
rst=$(tput sgr0)                  # Reset

function redwarning() {
echo -e $txtred "$1" $rst
}

if [ -a /mnt/live/memory/images/changes ]; then
   SDspace=$(df -BM --output=size,used,avail,pcent /mnt/live/memory/images/changes | tail -n1)
   read size used avail percent <<< $SDspace
   if [ ${percent%\%} -gt  85 ]; then
      echo 
      redwarning "Warning! save.dat file $percent full.  Used: $used  Free: $avail"
      echo 
      sleep 4
   else
      echo 
      echo "save.dat file $percent full.  Used: $used  Free: $avail"
   fi
fi
Ed

Bogomips
Full of knowledge
Full of knowledge
Posts: 2564
Joined: 25 Jun 2014, 15:21
Distribution: 3.2.2 Cinnamon & KDE5
Location: London

Re: SAVE.DAT, XZM & MLOOP

Post#41 by Bogomips » 31 Jul 2016, 21:12

Ed_P wrote:We developed something useful for save.dat users
Bite the bullet, Open a topic for something like short scripts in Tutorials, pretty up the script with helpful comments, and place it there. I also would like to put out a short script for flash, when it's been properly tried. Then it's always there when you need it, without having to rummage through one's files.
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

Bogomips
Full of knowledge
Full of knowledge
Posts: 2564
Joined: 25 Jun 2014, 15:21
Distribution: 3.2.2 Cinnamon & KDE5
Location: London

DIY save.dat

Post#42 by Bogomips » 07 Aug 2016, 02:03

Ed_P wrote:I'm not seeing what you're seeing with these commands.

Code: Select all

guest@porteus:~$ ls --block-size=M -s /mnt/sda5/porteus3.0/changes/porteussave.dat
512M /mnt/sda5/porteus3.0/changes/porteussave.dat*
guest@porteus:~$ ls -l --block-size=M /mnt/sda5/porteus3.0/changes/porteussave.dat
-rwxrwxrwx 2 root root 512M Jul 25 15:19 /mnt/sda5/porteus3.0/changes/porteussave.dat*
ls -s and ls -l are both showing the file's allocated size not used size.
Ed_P wrote:

Code: Select all

guest@porteus:~$ file  /mnt/sda5/porteus3.0/changes/porteussave.dat
/mnt/sda5/porteus3.0/changes/porteussave.dat: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs)
  • Create Container File in NTFS and set up XFS within:

    Code: Select all

    /dev/sdc2: UUID="2F2F3C037EAEA623" TYPE="ntfs" PARTUUID="33373bed-02"
    /dev/sdc2 on /mnt/sdc2 type fuseblk (rw,noatime,allow_other,blksize=4096)
    
    guest@porteus:~$ dd if=/dev/null of=/mnt/sdc2/xfi bs=1M seek=16
    0+0 records in
    0+0 records out
    0 bytes copied, 0.000297699 s, 0.0 kB/s
    
    guest@porteus:~$ ls -lh /mnt/sdc2/xfi
    -rwxrwxrwx 1 root root 16M Aug  5 23:22 /mnt/sdc2/xfi*
    
    guest@porteus:~$ /sbin/mkfs.xfs -b size=1024 -l internal /mnt/sdc2/xfi
    meta-data=/mnt/sdc2/xfi                 isize=512    agcount=1, agsize=16384 blks
             =                       sectsz=512   attr=2, projid32bit=1
             =                       crc=1        finobt=1, sparse=0
    data     =                       bsize=1024   blocks=16384, imaxpct=25
             =                       sunit=0      swidth=0 blks
    naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
    log      =internal log           bsize=1024   blocks=2573, version=2
             =                       sectsz=512   sunit=0 blks, lazy-count=1
    realtime =none                   extsz=4096   blocks=0, rtextents=0
    
    guest@porteus:~$ ls -sh /mnt/sdc2/xfi
    2.7M /mnt/sdc2/xfi*
    guest@porteus:~$ file /mnt/sdc2/xfi
    /mnt/sdc2/xfi: SGI XFS filesystem data (blksz 1024, inosz 512, v2 dirs)
    
    guest@porteus:~$ mkdir x; sudo mount /mnt/sdc2/xfi x
    guest@porteus:~$ du -sh x
    0	x
    guest@porteus:~$ df -h x
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop12      14M  865K   13M   7% /home/guest/x
    
  • Fake Root

    Code: Select all

    guest@porteus:~$ : Rootcopy
    guest@porteus:~$ sudo cp -a /mnt/sda4/cnrc64rc3/* x
    guest@porteus:~$ tree -nd x
    x
    ├── etc
    │   └── rc.d
    └── home
        └── guest
            └── Prolog
    5 directories
    
    guest@porteus:~$ du -sh x
    60K	x
    
    guest@porteus:~$ df -h x
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop12      14M  925K   13M   7% /home/guest/x
    
  • Details

    Code: Select all

    guest@porteus:~$ : Unmounted
    guest@porteus:~$ sudo umount x
    guest@porteus:~$ ls -lh /mnt/sdc2/xfi
    -rwxrwxrwx 1 root root 16M Aug  6 00:03 /mnt/sdc2/xfi*
    
    guest@porteus:~$ ls -sh /mnt/sdc2/xfi
    2.8M /mnt/sdc2/xfi*
    guest@porteus:~$ du -h /mnt/sdc2/xfi
    2.8M	/mnt/sdc2/xfi
    
  • Add Large File

    Code: Select all

    guest@porteus:~$ : Add Large File
    guest@porteus:~$ sudo mount /mnt/sdc2/xfi x
    
    guest@porteus:~$ is=/mnt/sda1/tmp/iso
    guest@porteus:~$ ls -lh $is/lms_9.avi
    -rw-r--r-- 1 guest users 11M Apr  3 20:28 /mnt/sda1/tmp/iso/lms_9.avi
    guest@porteus:~$ cp -p $is/lms_9.avi  x/home/guest
    
    guest@porteus:~$ du -sh x
    11M	x
    guest@porteus:~$ df -h x
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop12      14M   12M  1.9M  87% /home/guest/x
    
    guest@porteus:~$ sudo umount x
    guest@porteus:~$ ls -sh /mnt/sdc2/xfi
    14M /mnt/sdc2/xfi*
    guest@porteus:~$ du -h /mnt/sdc2/xfi
    14M	/mnt/sdc2/xfi
    
  • Add smaller file

    Code: Select all

    guest@porteus:~$ sudo mount /mnt/sdc2/xfi x
    guest@porteus:~$ du -sh x
    11M	x
    guest@porteus:~$ df -h x
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop12      14M   12M  1.9M  87% /home/guest/x
    
    guest@porteus:~$ : Add smaller file
    guest@porteus:~$ ls -lh /mnt/sda1/tmp/2048px-Rembrandt_Harmensz._van_Rijn_137.jpg
    -rw-r--r-- 1 vboxadd vboxsf 1.2M Mar  8  2014 /mnt/sda1/tmp/2048px-Rembrandt_Harmensz._van_Rijn_137.jpg
    guest@porteus:~$ cp -p /mnt/sda1/tmp/2048px-Rembrandt_Harmensz._van_Rijn_137.jpg  x/home/guest
    
    guest@porteus:~$ du -sh x
    12M	x
    guest@porteus:~$ df -h x
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop12      14M   13M  773K  95% /home/guest/x
    
    guest@porteus:~$ sudo umount x
    guest@porteus:~$ ls -sh /mnt/sdc2/xfi
    15M /mnt/sdc2/xfi*
    guest@porteus:~$ du -h /mnt/sdc2/xfi
    15M	/mnt/sdc2/xfi
    
    guest@porteus:~$ ls -lh /mnt/sdc2/xfi
    -rwxrwxrwx 1 root root 16M Aug  6 03:33 /mnt/sdc2/xfi*
    
In this instance it can be seen that file usage and available space can be ascertained thru the commands shown.

Update

Missing Free Space:

Code: Select all

/dev/sdc1 on /mnt/sdc1 type vfat (rw,noatime,nodiratime,umask=0,check=s,utf8)
/dev/sdc2 on /run/media/guest/2F2F3C037EAEA623 type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
guest@porteus:~$ sudo mount --bind /run/media/guest/2F2F3C037EAEA623  /mnt/sdc2

guest@porteus:~$ df -h /mnt/sdc2
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc2       3.5G   33M  3.4G   1% /home/guest/c2
guest@porteus:~$ df -h /mnt/sdc2/xfi
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc2       3.5G   33M  3.4G   1% /home/guest/c2

guest@porteus:~$ ls -a /mnt/sdc2
./  ../  xfi*
guest@porteus:~$ du -sh /mnt/sdc2
15M	c2

guest@porteus:~$ ls -lh /mnt/sdc2/xfi
-rwxrwxrwx 1 guest users 16M Aug  6 18:09 /mnt/sdc2/xfi*
guest@porteus:~$ ls -sh /mnt/sdc2/xfi
15M /mnt/sdc2/xfi*

guest@porteus:~$ du -h /mnt/sdc2/xfi
15M	c2/xfi
Last edited by Bogomips on 14 Aug 2016, 22:04, edited 2 times in total.
Reason: Missing Free Space
Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

Bogomips
Full of knowledge
Full of knowledge
Posts: 2564
Joined: 25 Jun 2014, 15:21
Distribution: 3.2.2 Cinnamon & KDE5
Location: London

A great big savefile for all does away with resize issue.1/3

Post#43 by Bogomips » 28 Aug 2016, 00:22

Ran mksavfile.sh http://forum.porteus.org/viewtopic.php? ... =45#p47971 with all defaults, and inadvertently created 512M container file in 262M filesystem:

Code: Select all

guest@porteus:~$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
aufs            262M  6.2M  256M   3% /
Create of 512M Container file:

Code: Select all

guest@porteus:~$ sh p10/Por/mksavfile.sh save
0+0 records in
0+0 records out
0 bytes copied, 0.000201849 s, 0.0 kB/s

ls -lh save
-rw-r--r-- 1 guest users 512M Aug 12 13:14 save

meta-data=save                   isize=512    agcount=4, agsize=131072 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0
data     =                       bsize=1024   blocks=524288, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=1024   blocks=2573, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

file save
save: SGI XFS filesystem data (blksz 1024, inosz 512, v2 dirs)

ls -sh save
2.8M save
du -h save
2.8M	save

sudo mount save f2439

du -sh f2439
0	f2439

df -h f2439
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop11     510M  8.1M  502M   2% /home/guest/f2439
Extrapolating :wink:
  • FAILED: 1TB SAVEFILE

    Code: Select all

    guest@porteus:~$ sh p10/Por/mksavfile.sh savtb 1048576 4096
    0+0 records in
    0+0 records out
    0 bytes copied, 0.000175394 s, 0.0 kB/s
    
    ls -lh savtb
    -rw-r--r-- 1 guest users 1.0T Aug 12 23:36 savtb
    
    meta-data=savtb                  isize=512    agcount=4, agsize=67108864 blks
             =                       sectsz=512   attr=2, projid32bit=1
             =                       crc=1        finobt=1, sparse=0
    data     =                       bsize=4096   blocks=268435456, imaxpct=5
             =                       sunit=0      swidth=0 blks
    naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
    log      =internal log           bsize=4096   blocks=131072, version=2
             =                       sectsz=512   sunit=0 blks, lazy-count=1
    realtime =none                   extsz=4096   blocks=0, rtextents=0
    mkfs.xfs: libxfs_device_zero write failed: No space left on device
    
    file savtb
    savtb: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
    
    ls -sh savtb
    258M savtb
    du -h savtb
    258M	savtb
    
    sudo mount savtb f2976
    NTFS signature is missing.
    Failed to mount '/dev/loop11': Invalid argument
    The device '/dev/loop11' doesn't seem to have a valid NTFS.
    Maybe the wrong device is used? Or the whole disk instead of a
    partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?
    
    du -sh f2976
    0	f2976
    
    df -h f2976
    Filesystem      Size  Used Avail Use% Mounted on
    aufs            265M  265M     0 100% /
    umount: f2976: not mounted
    
    guest@porteus:~$ rm savtb
    guest@porteus:~$ df -h .
    Filesystem      Size  Used Avail Use% Mounted on
    aufs            265M  6.1M  258M   3% /
    
  • SUCCESS: 100GB SAVEFILE

    Code: Select all

    guest@porteus:~$ df -h .
    Filesystem      Size  Used Avail Use% Mounted on
    aufs            262M  6.2M  256M   3% /
    
    guest@porteus:~$ sh p10/Por/mksavfile.sh sav100gb 102400 4096
    0+0 records in
    0+0 records out
    0 bytes copied, 0.000328088 s, 0.0 kB/s
    
    ls -lh sav100gb
    -rw-r--r-- 1 guest users 100G Aug 13 00:32 sav100gb
    
    meta-data=sav100gb               isize=512    agcount=4, agsize=6553600 blks
             =                       sectsz=512   attr=2, projid32bit=1
             =                       crc=1        finobt=1, sparse=0
    data     =                       bsize=4096   blocks=26214400, imaxpct=25
             =                       sunit=0      swidth=0 blks
    naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
    log      =internal log           bsize=4096   blocks=12800, version=2
             =                       sectsz=512   sunit=0 blks, lazy-count=1
    realtime =none                   extsz=4096   blocks=0, rtextents=0
    
    file sav100gb
    sav100gb: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
    
    ls -sh sav100gb
    51M sav100gb
    du -h sav100gb
    51M	sav100gb
    
    sudo mount sav100gb f2091
    
    du -sh f2091
    0	f2091
    
    df -h f2091
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop14     100G   33M  100G   1% /home/guest/f2091
    
    guest@porteus:~$ df -h .
    Filesystem      Size  Used Avail Use% Mounted on
    aufs            262M   58M  205M  22% /
    
    guest@porteus:~$ rm sav100gb
    guest@porteus:~$ df -h .
    Filesystem      Size  Used Avail Use% Mounted on
    aufs            262M  6.2M  256M   3% /
    
Container file of this sort can be made, and seems to work on aufs, ext2, ext4, reiserfs(not so well), not at all on vfat, but seems to be viable on ntfs:
  • Create file

    Code: Select all

    /dev/sdb2: UUID="2F2F3C037EAEA623" TYPE="ntfs" PARTUUID="33373bed-02"
    guest@porteus:~$ sh p10/Por/mksavfile.sh /mnt/sdb2/sav100gb 102400 4096
    0+0 records in
    0+0 records out
    0 bytes copied, 0.000261632 s, 0.0 kB/s
    
    ls -lh /mnt/sdb2/sav100gb
    -rwxrwxrwx 1 root root 100G Aug 15 02:24 /mnt/sdb2/sav100gb
    
    meta-data=/mnt/sdb2/sav100gb     isize=512    agcount=4, agsize=6553600 blks
             =                       sectsz=512   attr=2, projid32bit=1
             =                       crc=1        finobt=1, sparse=0
    data     =                       bsize=4096   blocks=26214400, imaxpct=25
             =                       sunit=0      swidth=0 blks
    naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
    log      =internal log           bsize=4096   blocks=12800, version=2
             =                       sectsz=512   sunit=0 blks, lazy-count=1
    realtime =none                   extsz=4096   blocks=0, rtextents=0
    
  • Creation Stats

    Code: Select all

    file /mnt/sdb2/sav100gb
    /mnt/sdb2/sav100gb: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
    
    ls -sh /mnt/sdb2/sav100gb
    51M /mnt/sdb2/sav100gb
    du -h /mnt/sdb2/sav100gb
    51M	/mnt/sdb2/sav100gb
    
    sudo mount /mnt/sdb2/sav100gb f5323
    
    du -sh f5323
    0	f5323
    
    df -h f5323
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop16     100G   33M  100G   1% /home/guest/f5323
    
  • Interpretation
    File is empty. Overhead less than 1 MB for Container File this Size. Complete size of savefile indicated as 51 MB, so deduce overhead of creating xfs for this size container to be 50 MB.
    • Size Verification in terms of amount of space taken from underlying FS, by deleting savefile.

      Code: Select all

      guest@porteus:~$ /sbin/blkid | grep ntfs
      /dev/sdb2: UUID="2F2F3C037EAEA623" TYPE="ntfs" PARTUUID="33373bed-02"
      guest@porteus:~$ sudo mount /dev/sdb2  b2
      Mount is denied because the NTFS volume is already exclusively opened.
      
      guest@porteus:~$ ls /mnt/sdb2
      aufs100gb*  sav100gb*  xfi*
      guest@porteus:~$ du -sh /mnt/sdb2
      67M	/mnt/sdb2
      guest@porteus:~$ df -h /mnt/sdb2
      Filesystem      Size  Used Avail Use% Mounted on
      /dev/sdb2       3.5G   86M  3.4G   3% /mnt/sdb2
      
      guest@porteus:~$ df -BM /mnt/sdb2
      Filesystem     1M-blocks  Used Available Use% Mounted on
      /dev/sdb2          3504M   86M     3419M   3% /mnt/sdb2
      guest@porteus:~$ rm -i /mnt/sdb2/sav100gb
      rm: remove regular file '/mnt/sdb2/sav100gb'? y
      
      guest@porteus:~$ df -BM /mnt/sdb2
      Filesystem     1M-blocks  Used Available Use% Mounted on
      /dev/sdb2          3504M   35M     3469M   1% /mnt/sdb2
      guest@porteus:~$ du -sh /mnt/sdb2
      17M	/mnt/sdb2
      
      Savefile took up 67-17 = 50 MB of Space.
    Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
    NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

    Bogomips
    Full of knowledge
    Full of knowledge
    Posts: 2564
    Joined: 25 Jun 2014, 15:21
    Distribution: 3.2.2 Cinnamon & KDE5
    Location: London

    Viability of Great Big Savefile Post 2/3

    Post#44 by Bogomips » 28 Aug 2016, 00:27

    Using unused(dedicated) ext2 Partition allows Collecting Statistics and Ascribing meaning to them.

    Test Run
    • Dedicated ext2 Partition

      Code: Select all

      /dev/sda6: UUID="f554b881-2089-4c0f-8de2-70900bdc4e14" TYPE="ext2" 
      
      guest@porteus:~$ df -h /mnt/sda6 
      Filesystem      Size  Used Avail Use% Mounted on
      /dev/sda6       1.9G  581M  1.2G  33% /home/guest//mnt/sda6 
      
      guest@porteus:~$ sudo du -sh /mnt/sda6 
      578M	/mnt/sda6 
      guest@porteus:~$ df -BM /mnt/sda6 
      Filesystem     1M-blocks  Used Available Use% Mounted on
      /dev/sda6          1889M  581M     1213M  33% /mnt/sda6 
      
    • Savefile Creation

      Code: Select all

      guest@porteus:~$ sh p10/Por/mksavfile.sh /mnt/sda6/sav/savgbg.dat 102400 4096
      0+0 records in
      0+0 records out
      0 bytes copied, 0.000178608 s, 0.0 kB/s
      
      ls -lh /mnt/sda6 /sav/savgbg.dat
      -rw-r--r-- 1 guest users 100G Aug 16 19:23 /mnt/sda6 /sav/savgbg.dat
      
      meta-data=/mnt/sda6 /sav/savgbg.dat      isize=512    agcount=4, agsize=6553600 blks
               =                       sectsz=512   attr=2, projid32bit=1
               =                       crc=1        finobt=1, sparse=0
      data     =                       bsize=4096   blocks=26214400, imaxpct=25
               =                       sunit=0      swidth=0 blks
      naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
      log      =internal log           bsize=4096   blocks=12800, version=2
               =                       sectsz=512   sunit=0 blks, lazy-count=1
      realtime =none                   extsz=4096   blocks=0, rtextents=0
      
      file /mnt/sda6 /sav/savgbg.dat
      /mnt/sda6 /sav/savgbg.dat: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
      
      ls -sh /mnt/sda6 /sav/savgbg.dat
      51M /mnt/sda6 /sav/savgbg.dat
      du -h /mnt/sda6 /sav/savgbg.dat
      51M	/mnt/sda6 /sav/savgbg.dat
      
    • Creation and Underlying FS Statistics
      • Creation Stats

        Code: Select all

        sudo mount /mnt/sda6/sav/savgbg.dat f2391
        
        du -sh f2391
        0	f2391
        
        df -h f2391
        Filesystem      Size  Used Avail Use% Mounted on
        /dev/loop14     100G   33M  100G   1% /home/guest/f2391
        
        guest@porteus:~$ df -BM /mnt/sda6
        Filesystem     1M-blocks  Used Available Use% Mounted on
        /dev/sda6          1889M  631M     1163M  36% /mnt/sda6
        guest@porteus:~$ sudo du -sh /mnt/sda6
        628M	/mnt/sda6
        
        guest@porteus:~$ df -h /mnt/sda6
        Filesystem      Size  Used Avail Use% Mounted on
        /dev/sda6       1.9G  631M  1.2G  36% /mnt/sda6
        
      • Deduction from Stats:
        Size of Data = 0 MB. Complete size of savefile = 631 - 581 = 50 MB. Overhead of creating 100 GB container file < 1 MB. Container File stats indicate 50 MB xfs overhead.
    • Sampling (Just first samples and last samples would be of interest. Left in rest for reference.)
        • Code: Select all

          root@porteus:/home/guest# usm -i
          ++++++++++++++++++++++++++++++
          usm application information
          
          USM version:  3.2.0
          DB version:  160320
          Current storage: /home/guest/p10/Por/tmp/usm
          Current config: /etc/usm/usm.conf
          Current platform: Porteus-v3.2rc5
          Current arch: i686
          Current Slackware 14.2
          Current kernel 4.6.3-porteus
          Current distros: slackware slackwarepatches slacky salix alien ponce slackonly
          quiet from=/dev/sda8/rc5 copy2ram ramsize=30% noauto rootcopy=/dev/sda4/cin_rcp guiexec=Prolog/pmx.sh cliexec=mount~--bind~/home/guest/p9/tmp~/tmp extramod=/dev/sda10/tmp/iso/rc4/tst/mods changes=/dev/sda6/sav/savgbg.dat
          
          guest@porteus:~$ cat /var/log/porteus-livedbg
          # Recognized devices:
          ...
          /dev/sda6: UUID="f554b881-2089-4c0f-8de2-70900bdc4e14" TYPE="ext2"
          ...
          /mnt/sda6/sav/savgbg.dat: UUID="b001ac5e-6039-4c05-a1af-00d144ae6adc" TYPE="xfs"
          
          # Booting device:
          /mnt/sda8
          
          # Porteus data found in:
          /mnt/sda8/rc5/porteus
          
          # Changes are stored in:
          /dev/sda6/sav/savgbg.dat
          
          # Non standard /rootcopy dir:
          /dev/sda4/cin_rcp
          
          # Modules activated during boot time:
          /memory/copy2ram/000-kernel.xzm
          /memory/copy2ram/001-core.xzm
          /memory/copy2ram/002-xorg.xzm
          /memory/copy2ram/003-cinnamon.xzm
          /memory/copy2ram/009-caches.xzm
          
        • Code: Select all

          guest@porteus:~$ date
          Wed Aug 17 00:43:11 BST 2016
          guest@porteus:~$ ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 17 00:28 /mnt/sda6/sav/savgbg.dat
          
          guest@porteus:~$ ls -sh /mnt/sda6/sav/savgbg.dat
          58M /mnt/sda6/sav/savgbg.dat
          guest@porteus:~$ du -h /mnt/sda6/sav/savgbg.dat
          58M	/mnt/sda6/sav/savgbg.dat
          
          guest@porteus:~$ sudo du -sh /mnt/live/memory/changes
          6.3M	/mnt/live/memory/changes
          guest@porteus:~$ df -h  /mnt/live/memory/changes
          Filesystem      Size  Used Avail Use% Mounted on
          /dev/loop1      100G   40M  100G   1% /mnt/live/memory/changes
          
          guest@porteus:~$ df -BM /mnt/sda6
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  638M     1156M  36% /mnt/sda6
          
        • Summary:
          Data Size = 6.3 MB
          Disk Space taken up by Savefile = 638 - 581 = 57 MB
        • Code: Select all

          root@porteus:/home/guest# usm -i
          ++++++++++++++++++++++++++++++
          usm application information
          
          USM version:  3.2.0
          DB version:  160320
          Current storage: /home/guest/p10/Por/tmp/usm
          Current config: /etc/usm/usm.conf
          Current platform: Porteus-v3.2rc5
          Current arch: i686
          Current Slackware 14.2
          Current kernel 4.6.3-porteus
          Current distros: slackware slackwarepatches slacky salix alien ponce slackonly
          quiet from=/dev/sda8/rc5 copy2ram ramsize=30% noauto extramod=/dev/sda10/tmp/iso/rc4/tst/mods changes=/dev/sda6/sav/savgbg.dat
          
          guest@porteus:~$ cat /var/log/porteus-livedbg
          # Recognized devices:
          ...
          /dev/sda10: LABEL="p_10" UUID="ed3f74cd-26bb-47bd-86b0-5e690663dd55" TYPE="ext2"
          /dev/sda11: UUID="6a0167f6-5cf7-412d-b85f-0076ca590371" TYPE="swap"
          /mnt/sda6/sav/savgbg.dat: UUID="b001ac5e-6039-4c05-a1af-00d144ae6adc" TYPE="xfs"
          
          # Booting device:
          /mnt/sda8
          
          # Porteus data found in:
          /mnt/sda8/rc5/porteus
          
          # Changes are stored in:
          /dev/sda6/sav/savgbg.dat
          
          # Non standard /rootcopy dir:
          none
          
          # Modules activated during boot time:
          /memory/copy2ram/000-kernel.xzm
          /memory/copy2ram/001-core.xzm
          /memory/copy2ram/002-xorg.xzm
          /memory/copy2ram/003-cinnamon.xzm
          /memory/copy2ram/009-caches.xzm
          
          guest@porteus:~$ df -h .
          Filesystem      Size  Used Avail Use% Mounted on
          aufs            100G   41M  100G   1% /
          
        • Code: Select all

          17.08.2016 11:29:37
          guest@porteus:~$ chinfo
          ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 17 11:22 /mnt/sda6/sav/savgbg.dat
          
          ls -s --block-size=M /mnt/sda6/sav/savgbg.dat
          60M /mnt/sda6/sav/savgbg.dat
          du -BM /mnt/sda6/sav/savgbg.dat
          60M	 /mnt/sda6/sav/savgbg.dat
          
          df -BM /mnt/sda6
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  640M     1153M  36% /mnt/sda6
          
          LIVE CHANGES
          
          guest@porteus:~$ sudo du -sBM /mnt/live/memory/changes/
          8M	/mnt/live/memory/changes/
          guest@porteus:~$ df -BM /mnt/live/memory/changes/
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/loop1       102350M   41M   102310M   1% /mnt/live/memory/changes
          
          guest@porteus:~$ df -h /mnt/live/memory/changes/
          Filesystem      Size  Used Avail Use% Mounted on
          /dev/loop1      100G   41M  100G   1% /mnt/live/memory/changes
          
        • Code: Select all

          guest@porteus:~$ . p10/Por/chinfo.sh
          guest@porteus:~$ chinfo
          ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 17 11:30 /mnt/sda6/sav/savgbg.dat
          
          ls -s --block-size=M /mnt/sda6/sav/savgbg.dat
          60M /mnt/sda6/sav/savgbg.dat
          du -BM /mnt/sda6/sav/savgbg.dat
          60M	/mnt/sda6/sav/savgbg.dat
          
          df -BM /mnt/sda6
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  641M     1153M  36% /mnt/sda6
          
          XFS
          
          guest@porteus:~$ mkdir f; sudo mount /mnt/sda6/sav/savgbg.dat f
          guest@porteus:~$ sudo du -sBM f
          8M	f
          guest@porteus:~$ df -h f
          Filesystem      Size  Used Avail Use% Mounted on
          /dev/loop11     100G   41M  100G   1% /home/guest/f
          
          guest@porteus:~$ sudo umount f
          
        • Code: Select all

          18.08.2016 00:17:43
          guest@porteus:~$ . p10/Por/chinfo.sh
          guest@porteus:~$ chinfo
          ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 17 15:24 /mnt/sda6/sav/savgbg.dat
          
          ls -s --block-size=M /mnt/sda6/sav/savgbg.dat
          60M /mnt/sda6/sav/savgbg.dat
          du -BM /mnt/sda6/sav/savgbg.dat
          60M	/mnt/sda6/sav/savgbg.dat
          
          df -BM /mnt/sda6
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  641M     1153M  36% /mnt/sda6
          
        • Summary
          Data Size (xfs stats) = 41 - 33 = 8 MB
          Disk Space taken up by Savefile = 641 - 581 = 60 MB
        • Code: Select all

          root@porteus:/home/guest# usm -i
          ++++++++++++++++++++++++++++++
          usm application information
          
          USM version:  3.2.0
          DB version:  160320
          Current storage: /home/guest/p10/Por/tmp/usm
          Current config: /etc/usm/usm.conf
          Current platform: Porteus-v3.2rc5
          Current arch: i686
          Current Slackware 14.2
          Current kernel 4.6.3-porteus
          Current distros: slackware slackwarepatches slacky salix alien ponce slackonly
          quiet from=/dev/sda8/rc5 copy2ram ramsize=30% noauto guiexec=Prolog/pmx.sh cliexec=mount~--bind~/home/guest/p9/tmp~/tmp;chown~guest:users~/home/guest extramod=/dev/sda10/tmp/iso/rc4/tst/mods changes=EXIT:/dev/sda6/sav/savgbg.dat
          
          guest@porteus:~$ cat /var/log/porteus-livedbg
          # Recognized devices:
          ...
          /dev/sda11: UUID="6a0167f6-5cf7-412d-b85f-0076ca590371" TYPE="swap"
          /mnt/sda6/sav/savgbg.dat: UUID="b001ac5e-6039-4c05-a1af-00d144ae6adc" TYPE="xfs"
          
          # Booting device:
          /mnt/sda8
          
          # Porteus data found in:
          /mnt/sda8/rc5/porteus
          
          # Changes are stored in:
          memory
          
          # Non standard /rootcopy dir:
          none
          
          # Modules activated during boot time:
          /memory/copy2ram/000-kernel.xzm
          /memory/copy2ram/001-core.xzm
          /memory/copy2ram/002-xorg.xzm
          /memory/copy2ram/003-cinnamon.xzm
          /memory/copy2ram/009-caches.xzm
          /mnt/sda6/sav/savgbg.dat/changes
          
        • Code: Select all

          18.08.2016 21:33:44
          guest@porteus:~$ chinfo
          ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 18 14:00 /mnt/sda6/sav/savgbg.dat
          
          ls -s --block-size=M /mnt/sda6/sav/savgbg.dat
          65M /mnt/sda6/sav/savgbg.dat
          du -BM /mnt/sda6/sav/savgbg.dat
          65M	p6/sav/savgbg.dat
          
          df -BM /mnt/sda6
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  645M     1149M  36% /mnt/sda6
          
          XFS INACCESSIBLE
          
          guest@porteus:~$ sudo du -sBM /mnt/live/memory/changes
          5M	/mnt/live/memory/changes
          
          guest@porteus:~$ df -BM /mnt/live/memory/changes
          Filesystem     1M-blocks  Used Available Use% Mounted on
          tmpfs               265M    5M      260M   2% /mnt/live/memory/changes
          
        • Summary
          Disk Space taken up by Savefile = 645 - 581 = 64 MB
          Increase in Size = 4 MB
          Deduce Data Size ~ 12 MB
      • State of Play prior to Adding Large File
        • Code: Select all

          root@porteus:/home/guest# usm -i
          ++++++++++++++++++++++++++++++
          usm application information
          
          USM version:  3.2.0
          DB version:  160320
          Current storage: /home/guest/p10/Por/tmp/usm
          Current config: /etc/usm/usm.conf
          Current platform: Porteus-v3.2rc5
          Current arch: i686
          Current Slackware 14.2
          Current kernel 4.6.3-porteus
          Current distros: slackware slackwarepatches slacky salix alien ponce slackonly
          quiet from=/dev/sda8/rc5 copy2ram ramsize=30% noauto guiexec=Prolog/pmx.sh cliexec=mount~--bind~/home/guest/p9/tmp~/tmp;chown~guest:users~/home/guest extramod=/dev/sda10/tmp/iso/rc4/tst/mods changes=EXIT:/dev/sda6/sav/savgbg.dat
          
          guest@porteus:~$ cat /var/log/porteus-livedbg
          # Recognized devices:
          ...
          /dev/sda11: UUID="6a0167f6-5cf7-412d-b85f-0076ca590371" TYPE="swap"
          /mnt/sda6/sav/savgbg.dat: UUID="b001ac5e-6039-4c05-a1af-00d144ae6adc" TYPE="xfs"
          
          # Booting device:
          /mnt/sda8
          
          # Porteus data found in:
          /mnt/sda8/rc5/porteus
          
          # Changes are stored in:
          memory
          
          # Non standard /rootcopy dir:
          none
          
          # Modules activated during boot time:
          /memory/copy2ram/000-kernel.xzm
          /memory/copy2ram/001-core.xzm
          /memory/copy2ram/002-xorg.xzm
          /memory/copy2ram/003-cinnamon.xzm
          /memory/copy2ram/009-caches.xzm
          /mnt/sda6/sav/savgbg.dat/changes
          
          guest@porteus:~$ df -h .
          Filesystem      Size  Used Avail Use% Mounted on
          aufs            265M  4.2M  260M   2% /
          
        • Code: Select all

          guest@porteus:~$ chinfo
          ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 20 01:39 /mnt/sda6/sav/savgbg.dat
          
          ls -s --block-size=M /mnt/sda6/sav/savgbg.dat
          68M /mnt/sda6/sav/savgbg.dat
          
          du -BM /mnt/sda6/sav/savgbg.dat
          68M	p6/sav/savgbg.dat
          
          df -BM /mnt/sda6
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  648M     1146M  37% /mnt/sda6
          ______________________________________________________________________________
          guest@porteus:~$ chfi /mnt/sda6/sav/savgbg.dat
          file /mnt/sda6/sav/savgbg.dat
          /mnt/sda6/sav/savgbg.dat: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
          
          Latest Update
          ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 20 01:36 /mnt/sda6/sav/savgbg.dat
          
          Underlying FS
          df -BM /mnt/sda6/sav/savgbg.dat
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  648M     1146M  37% /mnt/sda6
          
          Actual Size
          ls -s --block-size=M /mnt/sda6/sav/savgbg.dat
          68M /mnt/sda6/sav/savgbg.dat
          du -BM /mnt/sda6/sav/savgbg.dat
          68M	/mnt/sda6/sav/savgbg.dat
          
          Data Size
          sudo mount /mnt/sda6/sav/savgbg.dat c2172;
          sudo du -sBM c2172
          9M	c2172
          
          Container Capacity
          df -BM c2172
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/loop11      102350M   42M   102309M   1% /home/guest/c2172
          
        • Summary
          Occupancy underlying FS at the moment = 648 MB
          Occupancy underlying FS prior making Savefile = 581 MB
          Savefile Size = 67 MB
          Data Size = Occupancy of XFS - 33(const) = 42 - 33 = 9 MB
      • Large File Added
        • Container Deatails

          Code: Select all

          Downloads  <-  Unknown.mkv
          
          guest@porteus:~$ ls -lh p10/tor/Flm/Unknown.mkv
          -rw-r--r-- 1 guest 1000 18M Feb  7  2016 p10/tor/Flm/Unknown.mkv
          
          guest@porteus:~$ chinfo
          ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 20 11:03 /mnt/sda6/sav/savgbg.dat
          
          Used up Space:
          ls -s --block-size=M /mnt/sda6/sav/savgbg.dat
          85M /mnt/sda6/sav/savgbg.dat
          du -BM /mnt/sda6/sav/savgbg.dat
          85M	p6/sav/savgbg.dat
          
          Underlying FS Details:
          df -BM /mnt/sda6
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  666M     1128M  38% /mnt/sda6
          
        • XFS Details

          Code: Select all

          guest@porteus:~$ chfi /mnt/sda6/sav/savgbg.dat
          file /mnt/sda6/sav/savgbg.dat
          p6/sav/savgbg.dat: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
          
          Latest Update:
          ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 20 13:55 /mnt/sda6/sav/savgbg.dat
          
          Underlying FS:
          df -BM /mnt/sda6/sav/savgbg.dat
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  666M     1128M  38% /mnt/sda6
          
          Actual Size:
          ls -s --block-size=M /mnt/sda6/sav/savgbg.dat
          85M /mnt/sda6/sav/savgbg.dat
          du -BM /mnt/sda6/sav/savgbg.dat
          85M	p6/sav/savgbg.dat
          
          Data Size:
          sudo mount /mnt/sda6/sav/savgbg.dat c2000;
          sudo du -sBM c2000
          26M	c2000
          
          Container Capacity:
          df -BM c2000
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/loop12      102350M   59M   102292M   1% /home/guest/c2000
          
        • Summary
          Occupancy underlying FS at the moment = 666 MB
          Occupancy underlying FS prior making Savefile = 581 MB
          Savefile Size = 85 MB
          Data Size = Occupancy of XFS - 33(const) = 59 - 33 = 26MB

          Savefile Increase in Size = File Size Now - Prior File Size = 85 - 67 = 18 MB
          Data Increase in Size= Data Size Now - Prior Data Size = 26 - 9 = 17 MB

          Differently Calculated Increases in Size correspond roughly to Size of Added Large File (=18.2 MB)
    Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
    NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

    Bogomips
    Full of knowledge
    Full of knowledge
    Posts: 2564
    Joined: 25 Jun 2014, 15:21
    Distribution: 3.2.2 Cinnamon & KDE5
    Location: London

    Great Big Savefile Conclusion Post 3/3

    Post#45 by Bogomips » 28 Aug 2016, 00:29

    Conclusion
    1. Changes Differences
      • changes=EXIT:/Path/to/Savefile
        • Setup

          Code: Select all

          root@porteus:/home/guest# usm -i
          ++++++++++++++++++++++++++++++
          usm application information
          
          USM version:  3.2.0
          DB version:  160320
          Current storage: /home/guest/p10/Por/tmp/usm
          Current config: /etc/usm/usm.conf
          Current platform: Porteus-v3.2rc5
          Current arch: i686
          Current Slackware 14.2
          Current kernel 4.6.3-porteus
          Current distros: slackware slackwarepatches slacky salix alien ponce slackonly
          quiet from=/dev/sda8/rc5 copy2ram ramsize=30% noauto guiexec=Prolog/pmx.sh cliexec=mount~--bind~/home/guest/p9/tmp~/tmp;chown~guest:users~/home/guest extramod=/dev/sda10/tmp/iso/rc4/tst/mods changes=EXIT:/dev/sda6/sav/savgbg.dat
          
          guest@porteus:~$ cat /var/log/porteus-livedbg
          # Recognized devices:
          ...
          /dev/sda11: UUID="6a0167f6-5cf7-412d-b85f-0076ca590371" TYPE="swap"
          /mnt/sda6/sav/savgbg.dat: UUID="b001ac5e-6039-4c05-a1af-00d144ae6adc" TYPE="xfs"
          
          # Booting device:
          /mnt/sda8
          
          # Porteus data found in:
          /mnt/sda8/rc5/porteus
          
          # Changes are stored in:
          memory
          
          # Non standard /rootcopy dir:
          none
          
          # Modules activated during boot time:
          /memory/copy2ram/000-kernel.xzm
          /memory/copy2ram/001-core.xzm
          /memory/copy2ram/002-xorg.xzm
          /memory/copy2ram/003-cinnamon.xzm
          /memory/copy2ram/009-caches.xzm
          /mnt/sda6/sav/savgbg.dat/changes
          
          guest@porteus:~$ df -h .
          Filesystem      Size  Used Avail Use% Mounted on
          aufs            265M  4.2M  260M   2% /
          
        • Running (before exiting)

          Code: Select all

          guest@porteus:~$ c=$(cat /proc/cmdline); c=${c##*changes=}; echo changes=${c%% *}
          changes=EXIT:/dev/sda6/sav/savgbg.dat
          
          guest@porteus:~$ sudo du -sBM /mnt/live/memory/changes/
          10M	/mnt/live/memory/changes/
          guest@porteus:~$ df -h .
          Filesystem      Size  Used Avail Use% Mounted on
          aufs            265M  9.3M  255M   4% /
          
          guest@porteus:~$ df -BM /mnt/live/memory/changes/
          Filesystem     1M-blocks  Used Available Use% Mounted on
          tmpfs               265M   10M      255M   4% /mnt/live/memory/changes
          
          guest@porteus:~$ df -BM /mnt/live/memory/images
          Filesystem     1M-blocks  Used Available Use% Mounted on
          tmpfs               440M  258M      183M  59% /mnt/live
          
        The aufs and the tmpfs appear to be almost identical. The used amounts in /mnt/live/memory/changes are almost the same as that in aufs. This is expected as changes in aufs have to go into the changes savefile.
      • changes=/Path/to/Savefile
        • Savefile in AF mode

          Code: Select all

          guest@porteus:~$ chinfo
          ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 17 11:30 /mnt/sda6/sav/savgbg.dat
          
          ls -s --block-size=M /mnt/sda6/sav/savgbg.dat
          60M /mnt/sda6/sav/savgbg.dat
          du -BM /mnt/sda6/sav/savgbg.dat
          60M	/mnt/sda6/sav/savgbg.dat
          
          df -BM /mnt/sda6
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  641M     1153M  36% /mnt/sda6
          
          guest@porteus:~$ mkdir f; sudo mount /mnt/sda6/sav/savgbg.dat f
          guest@porteus:~$ sudo du -sBM f
          8M	f
          guest@porteus:~$ df -h f
          Filesystem      Size  Used Avail Use% Mounted on
          /dev/loop11     100G   41M  100G   1% /home/guest/f
          guest@porteus:~$ sudo umount f
          
        • Setup

          Code: Select all

          root@porteus:/home/guest# usm -i
          ++++++++++++++++++++++++++++++
          usm application information
          
          USM version:  3.2.0
          DB version:  160320
          Current storage: /home/guest/p10/Por/tmp/usm
          Current config: /etc/usm/usm.conf
          Current platform: Porteus-v3.2rc5
          Current arch: i686
          Current Slackware 14.2
          Current kernel 4.6.3-porteus
          Current distros: slackware slackwarepatches slacky salix alien ponce slackonly
          quiet from=/dev/sda8/rc5 copy2ram ramsize=30% noauto extramod=/dev/sda10/tmp/iso/rc4/tst/mods changes=/dev/sda6/sav/savgbg.dat
          
          guest@porteus:~$ cat /var/log/porteus-livedbg
          # Recognized devices:
          ...
          /dev/sda10: LABEL="p_10" UUID="ed3f74cd-26bb-47bd-86b0-5e690663dd55" TYPE="ext2"
          /dev/sda11: UUID="6a0167f6-5cf7-412d-b85f-0076ca590371" TYPE="swap"
          /mnt/sda6/sav/savgbg.dat: UUID="b001ac5e-6039-4c05-a1af-00d144ae6adc" TYPE="xfs"
          
          # Booting device:
          /mnt/sda8
          
          # Porteus data found in:
          /mnt/sda8/rc5/porteus
          
          # Changes are stored in:
          /dev/sda6/sav/savgbg.dat
          
          # Non standard /rootcopy dir:
          none
          
          # Modules activated during boot time:
          /memory/copy2ram/000-kernel.xzm
          /memory/copy2ram/001-core.xzm
          /memory/copy2ram/002-xorg.xzm
          /memory/copy2ram/003-cinnamon.xzm
          /memory/copy2ram/009-caches.xzm
          
        • Savefile being used for Changes

          Code: Select all

          guest@porteus:~$ chinfo
          ls -lh /mnt/sda6/sav/savgbg.dat
          -rw-r--r-- 1 guest users 100G Aug 17 11:22 /mnt/sda6/sav/savgbg.dat
          
          ls -s --block-size=M /mnt/sda6/sav/savgbg.dat
          60M /mnt/sda6/sav/savgbg.dat
          du -BM /mnt/sda6/sav/savgbg.dat
          60M	/mnt/sda6/sav/savgbg.dat
          
          df -BM /mnt/sda6
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/sda6          1889M  640M     1153M  36% /mnt/sda6
          
          guest@porteus:~$ df -h .
          Filesystem      Size  Used Avail Use% Mounted on
          aufs            100G   41M  100G   1% /
          guest@porteus:~$ pwd
          /home/guest
          
          guest@porteus:~$ sudo du -sBM /mnt/live/memory/changes/
          8M	/mnt/live/memory/changes/
          
          guest@porteus:~$ df -BM /mnt/live/memory/changes/
          Filesystem     1M-blocks  Used Available Use% Mounted on
          /dev/loop1       102350M   41M   102310M   1% /mnt/live/memory/changes
          guest@porteus:~$ df -h /mnt/live/memory/changes/
          Filesystem      Size  Used Avail Use% Mounted on
          /dev/loop1      100G   41M  100G   1% /mnt/live/memory/changes
          guest@porteus:~$ lsblk -f | grep xfs
          loop1   xfs            95697672-145f-45b6-b190-c02b3ae82906 
          
          1. Container file's xfs is mounted on /mnt/live/memory/changes.
          2. /home/guest magicked on to the xfs fake root. Consequently even though copy2ram chosen, manipulation of large data amounts would have in core speed hampered by disk access waits.
    2. This approach does not work with vfat; works to certain extent with reiserfs, but seems to work with aufs, ext2, ext4 and ntfs. Underlying FS overhead for 100 GB Container file is negligible (< 1 MB) compared to XFS overhead of 50 MB. And 50 MB is nothing nowadays, just the requirement of a 20 minute video.
        • Ascertaining Size of Savefile (Amount of Disk Space taken up by File)
          • Code: Select all

            ls  -s  <savefile>
            and
            du  <savefile>
            Both so far provide the same value for this.
        • Ascertaining Amount of Data Amassed
          • Code: Select all

            mount  <savefile> <mountpoint>
            du  -s  <mountpoint>
            and
            df  -BM <mountpoint>
            
            Both provide approximations to the amount of data stored. The latter's (df) used value provides data size in MB after subtraction of 33 (a constant).
          Given a large value for the container file like 100 GB, the issue of savefile corruption should disappear, putting onus on user to ensure savefile does not eventually swallow up the underlying FS.
          • Scripts
              • mksavfile.sh Target has to be on a real filesystem.
                Script has to be made executable and run. Cannot run in subshell, due to subshell bug.

                Code: Select all

                #!/bin/bash
                # Make executable. Not to be run in subshell.			25.08.2016 23:13:59
                # 12.08.2016
                # Updated with stats of underlying FS				24.08.2016 17:34:25
                # 25.08.2016 23:13:59	Subshell Bug Update
                # mksavfile.sh	<[/Path/to/]savefile>	[ <Size of file in MB>  [ Block size for XFS ] ]
                # mksavfile.sh	savefile.dat
                # mksavfile.sh	MyPort/mysave.dat	256
                # mksavfile.sh	savechanges			400		2048
                # ls -s --block-size=M
                # u=${1%/*}; [[ ${u:=/} == $1 ]] && u=.;
                # echo $1; eval $1
                ecmd()
                {
                	echo $1; $1
                }
                title()			# 2015-04-23 21:09:32
                {
                  local t; local l; local u;
                  [[ ! $eqs ]] &&
                  eqs="============================================================"
                #  l=${#1}; u=${eqs:0:l}; t="$'\n'$u$'\n'${1:0:l}$'\n'$u$'\n'"
                  l=${#1}; u=${eqs:0:l}; t="$u$'\n'${1:0:l}$'\n'$u"
                  eval echo $t
                }
                savefile()
                {
                	# Usage
                	[[ $1 ]] || { echo -e "Usage:\t${0##*/}\t[/Path/to/]Savefile [<Size of file in MB>[Block size for XFS]]"; return 1; }
                	# Prevent corruption of existing file
                	[[ -e $1 ]] && { echo $1 already exists!; return 2; }
                	# Container File
                	title "Container File"
                	echo Underlying FS stats before Container File Creation:
                	u=${1%/*}; [[ ${u:=/} == $1 ]] && u=.;
                	ecmd "df -BM $u"
                	# Need read to get rid of leading spaces
                	read used < <(df -BM --output=used  $u|tail -n1); usd0=${used%M};
                	echo; ecmd "dd if=/dev/null of=$1 bs=1M seek=${2:-102400}";        # Size in MB
                
                	echo; ecmd "ls -lh $1"; echo
                	ecmd "du -BM $1"
                	ecmd "ls -s --block-size=M $1"
                	echo -e '\n'Underlying FS stats following Container File Creation:
                	ecmd "df -BM $1"
                	# XFS
                	echo; title "XFS Creation"
                	/sbin/mkfs.xfs -b size=${3:-4096} -l internal $1;  # Block Size to Use
                	echo; ecmd "file  $1"; echo
                	ecmd "ls -s --block-size=M $1";
                	ecmd "du -BM $1"
                	# Data Details
                	echo; title "Data in XFS"
                	[[ ! -d f$$ ]] && mkdir f$$;
                	ecmd "sudo mount $1 f$$"; echo
                	ecmd "du -sBM  f$$"
                	echo; ecmd "df  -BM  f$$"
                	sudo umount f$$
                	rmdir f$$
                	echo -e '\n'Underlying FS stats after XFS formatting of Container File:
                	ecmd "df -BM $1"
                	# Need read to get rid of leading spaces
                	read used < <(df -BM --output=used $1|tail -n1); usd1=${used%M};
                	let as=usd1-usd0;
                	echo -e "\nActual Size of Savefile  =  Amount of FS used up upon Creation - Amount of FS prior to Creation\n=  $usd1 - $usd0  =  $as MB";
                }
                savefile $*
                
              • chinfo.sh Changes file info. Will show state of file even when changes in operation.

                Code: Select all

                #!/bin/bash
                # 25.08.2016	Usage added.
                # 17.08.2016
                # 24.08.2016	Generalised.
                # sh chinfo.sh  [/Path/to/]Savefile
                # ls -s --block-size=M
                # du -BM
                chinfo()
                {
                	ecmd() { echo $1; $1; }
                	[[ $1 ]] || { echo -e "Usage:\tsh\t${0##*/}\t[/Path/to/]Savefile"; return 1; }
                	ecmd "ls -lh $1"; echo
                	echo Used up Space:
                	ecmd "ls -s --block-size=M $1";
                	ecmd "du -BM $1"; echo
                	echo Underlying FS Details:
                	ecmd "df -BM $1"
                }
                chinfo $*
                
              • chfi.sh Provides info as well on Data Size when changes file offline.

                Code: Select all

                #!/bin/bash
                # 25.08.2016	Usage added. Plus Changes File online Check.
                # 19.08.2016
                # sh chfi.sh
                # ls -s --block-size=M
                # set -x
                chfi()
                {
                	[[ $1 ]] || { echo -e "Usage:\tsh\t${0##*/}\t[/Path/to/]Savefile"; return 1; }
                	ecmd() { echo $1; eval $1 2>/dev/null; }  # eval required for sudo or rather for c$$.
                	ecmd "file  $1";
                	# Date the File
                	echo -e "\nLatest Update:"; ecmd "ls  -lh   $1"
                	# Do Underlying FS
                #	[[ ${1:0:1} != / ]] && u=. || u=${1%/*}; [[ ! $u ]] && u=/;
                	echo -e "\nUnderlying FS:"; ecmd "df -BM $1";
                	# Actual Size
                	echo -e "\nActual Size:"; ecmd "ls  -s --block-size=M  $1"
                	ecmd "du -BM  $1"
                	[[ ! -e c$$ ]] && mkdir c$$;
                	mountpoint -q c$$ && sudo umount -f c$$;
                	# Data Size
                	echo -e "\nData Size:"; ecmd "sudo mount $1 c$$;" || { echo -e '\n'XFS not Accessible: Data Size not Computable. Savefile probably in Use.; return 2; }
                	ecmd "sudo du  -sBM  c$$"
                	echo -e "\nContainer Capacity:"; ecmd "df  -BM  c$$"
                	sudo umount c$$; rmdir c$$
                }
                chfi $*
                
            Last edited by Bogomips on 29 Aug 2016, 15:31, edited 2 times in total.
            Reason: Setup omitted
            Linux porteus 4.4.0-porteus #3 SMP PREEMPT Sat Jan 23 07:01:55 UTC 2016 i686 AMD Sempron(tm) 140 Processor AuthenticAMD GNU/Linux
            NVIDIA Corporation C61 [GeForce 6150SE nForce 430] (rev a2) MemTotal: 901760 kB MemFree: 66752 kB

            Post Reply