Ravas coding goodies

[Rava]

I see the backup_loop.py code on the page.

I see that as well.
have a screenshot with the popup in it and the relevant parts marked in red by yours truly:

[Ed_P]

That "secret" applies to this link also: Ravas coding goodies

[Rava]

Tee (command)

In computing, tee is a command in command-line interpreters (shells) using standard streams which reads standard input and writes it to both standard output and one or more files, effectively duplicating its input. It is primarily used in conjunction with pipes and filters. The command is named after the T-splitter used in plumbing.

The tee command might be unfamiliar to some but tee which comes in very handy at times, especially when you have a live system and want to capture outputs (e.g. from /var/log/messages ) even when the system would crash unexpectedly meaning any shutdown / reboot saving-current-settings into a module would fail even when the saving-current-settings script would incorporate saving of /var/log/messages.

Have some examples as well:
https://linuxize.com/post/linux-tee-command/

https://phoenixnap.com/kb/linux-tee


https://unix.stackexchange.com/question ... r-commands

The last line of my script that captures /var/log/messages uses this line:

Code: Select all

tail -fn 10 /var/log/messages|tee -a $myfile

and is meant to be run from a virtual terminal not from your DE so that a mere exit of XWindows would still not interrupt the writing (redirecting) of /var/log/messages into $myfile

Added in 3 minutes 27 seconds:
Of course when the system crashed there would be no writing into $myfile after the crash (duh! that's what a crash is) but at least you would see what was last reported into /var/log/messages - meaning you can examine the reasons of the crash. Unless the system shuts down due to CPU or GPU overheating issues, since this is something the hardware does and it gives no notice whatsoever to your OS.
Unless you write e.g. once per minute the relevant info into /var/log/messages e.g. via the command logger - but that would bloat up your /var/log/messages quite a lot.

Added in 6 minutes 44 seconds:

Code: Select all

sensors|grep Core

gives me the temperature of my CPU cores. I add a line with date/time stamp so that I can compare the recent info with older ones; created a script called mysensors.sh :

Code: Select all

guest@rava:/usr/local/bin$ cat /usr/local/bin/mysensors.sh 
#!/bin/sh
#V0.2
function tz () { 
    echo $(date +%d.%m.%Y\ %H:%M:%S) ____________________________________________________________
}

tz
sensors|grep Core
guest@rava:/usr/local/bin$ mysensors.sh 
13.09.2023 06:36:51 ____________________________________________________________
Core 0:       +63.0°C  (high = +84.0°C, crit = +100.0°C)
Core 1:       +62.0°C  (high = +84.0°C, crit = +100.0°C)
Core 2:       +63.0°C  (high = +84.0°C, crit = +100.0°C)
Core 3:       +62.0°C  (high = +84.0°C, crit = +100.0°C)

[Rava]

updating my daueralarm.sh script

Aside from what I wrote above already, I realized having two sound files chose from (my usual lowered in volume by -15 dB an the non-volume-changed original) can be useful at times, especially when I plan on watching some video (or maybe listening to audio) during the wait time and I want he dauerlalarm sound to be more loud and more annoying )

That means when I wt he script not using the usual approach of traditional syntax and determining what parameter means by the position of the parameter only - I have to put more thought into how to set up that idea.
I think for now i will make a dummy script hat only lists what would be done - interpreting the given parameter (if any) and what result in behaviour the script would think it should have to see if my unorthodox approach of parameter handling is workable at all…

Added in 16 hours 28 minutes 57 seconds:
Ideas for parameter-specific-less calling of daueralarm.sh to turn it into an alarm clock.

no parameter
● standard pause between playing the sound (2 seconds)
● standard mp3 file (-15dB)
● no pause prior executing the sound-loop
when useful? e.g.when you have a longer program from the CLI to run (e.g. new encoding of a longer video) to be alerted by audio as soon as it is finished; you can add a visual alert via

Code: Select all

notify-send -i info -t 0 "Encoding" "finished"

one parameter
● parameter sets the pause in seconds between playing the sound
● standard mp3 file (-15dB)
● no pause prior executing the sound-loop
same reason as above: to be alerted when a longer process is finished.

two parameters:
● 1st parameter sets the pause in seconds between playing the sound
● 2nd parameter : s = standard mp3 file (-15dB); l = louder mp3 file (0dB)
● no pause prior executing the sound-loop


three parameters:
● 1st parameter sets the pause in seconds between playing the sound
● 2nd parameter : s = standard mp3 file (-15dB); l = louder mp3 file (0dB)
● 3rd parameter: pause prior executing the sound-loop (in the way pause accepts the pause-parameter: 70 = 70 seconds; 2m = 2 minutes; 2.1m = 2 minutes 6 seconds etcetera.
When you want to use it as self-coded alarm clock (e.g. to set it for 9m to get the alarm when the pasta is cooked)

[Rava]

Lazy me quoting beny and myself:

Hi roro manual login and startx Is Better,well on my system try it

[Porteus supports the "3" kernel cheat code, add it to our kernel append parameter and you boot into init 3 (Slackware standard, Virtual Text terminals, multiuser, network)]

I prefer it that way as well, it takes a mere seconds to manually log in as guest and then type

Code: Select all

startx

but gives you much more control, also you can scroll up in the 1st terminal and review all boot messages even when they are already scrolled outside of the visible screen, something you cannot do when you boot into your XWindows and your chosen DE.

Added in 3 minutes 59 seconds:
Cave! The scrolling up only works when you did not change to a different virtual terminal ("VT"). When you stay in one virtual terminal you can scroll up to see much more output of current program or from boot, but when you change the VT you can only access what is currently seen, all that is scrolled up is gone.

for those who do not know, scrolling up or scrolling down in a VT is done via

Code: Select all

Shift + PageUp

or

Code: Select all

Shift + PageDown

Thanks beny.

Added in 2 hours 25 minutes 44 seconds:
And one more quoting, this time Ed_P and me. It's about how to efficiently use mc navigating around the file system using uniquely named symlinks.

But I don't see the /tmp folder in it.

⬤ When using mc you can either navigate through the folders like usual (double-click or press return on a folder, move up by doing that on the ".." entry)

⬤ or just type in the command prompt below the command

Code: Select all

cd /tmp

⬤ or start mc with the two folders you want to copy files in between.
Let's assume your module is in /tmp/ and you have a symlink that links to your main Porteus modules folder called /Porteus_modules
then just type

Code: Select all

mc  /tmp/ /Porteus_modules

Usually it would be enough to type

Code: Select all

mc  /tmp/ /P[tab]

since by default there exists no folder or file in / that starts with a upper-case P like your link /Porteus_modules does.
The [tab] then expands it to the only existing name -> /Porteus_modules

Yes, in some many cases using the terminal (and that includes mc) can let you do things quicker than a GUI equivalent especially when you use symlinks named in an easily remembered and unique way (as in: the first character of your symlink doesn't exist in that very folder aside from your symlink)

[Rava]

How To set up a small logging script that - like syslog - keeps logging for all eternity - only stops when you reboot, the system crashes or you manually zero the target file (for when the old logged info is no longer needed and you want to conserve space on that drive) or you exit the script forcefully via kill or Ctrl+C

You have to use a physical drive for all of it to make sense. You could run it in your aufs and hope your system will never crash and your auto-save-module on exit will also save your log file. But that is not how you should do things. When you set up any kind of log you will have a reason for doing so (e.g. your system crashed because of several possible reasons, and you alter my below script to monitor the possible reasons)

My example is quite easy.

You create a script, best in an ext[234] partition, in a folder, not in the root of that partition.

Let's presume that folder is /mnt/sda3/log/ and your script is called mysensorslog.sh and sits in /mnt/sda3/log/mysensorslog.sh

Here the code:

Code: Select all

#!/bin/sh
#based on mysensors.sh V0.2 and tail+tee_VLM.sh V0.2
function tz () { 
    echo $(date +%d.%m.%Y\ %H:%M:%S) ____________________________________________________________
}
if [ $UID -ne 0 ]; then
    echo "$0: ERROR - start me as user root. Aborting."
    exit 1
fi
mydate=$(date +%Y-%m-%d)
echo mydate suffix I am using: $mydate
myfile=sensors_$mydate
echo myfile I am using: $myfile - WILL OVERWRITE EXISTING ENTRIES
echo $(date +%d.%m.%Y\ %H:%M:sensors_%S) started: $0 >$myfile
while true; do { tz|tee -a $myfile; sensors|grep Core|tee -a $myfile ; sleep 1m ; } done

What does it do?
It sets up its file name based on the current date - sensors_YYYY--MM-DD

Here is when I run it as example:

Code: Select all

guest@rava:/mnt/sda3/log$ ls -o sensors_2023-09-20 
-rw-r--r-- 1 root 18618 2023-09-20 02:20 sensors_2023-09-20

This is what the script tells you when first started:

Code: Select all

root@rava:/mnt/sda3/log# ./mysensorslog.sh 
mydate suffix I am using: 2023-09-20
myfile I am using: sensors_2023-09-20 - WILL OVERWRITE EXISTING ENTRIES
20.09.2023 01:20:56 ____________________________________________________________
Core 0:       +56.0°C  (high = +84.0°C, crit = +100.0°C)
Core 1:       +55.0°C  (high = +84.0°C, crit = +100.0°C)
Core 2:       +57.0°C  (high = +84.0°C, crit = +100.0°C)
Core 3:       +55.0°C  (high = +84.0°C, crit = +100.0°C)

Every minute a new set of log info will be displayed on the terminal that runs the script and also written into its log file (in my example called sensors_2023-09-20 )

Code: Select all

20.09.2023 01:20:56 ____________________________________________________________
Core 0:       +56.0°C  (high = +84.0°C, crit = +100.0°C)
Core 1:       +55.0°C  (high = +84.0°C, crit = +100.0°C)
Core 2:       +57.0°C  (high = +84.0°C, crit = +100.0°C)
Core 3:       +55.0°C  (high = +84.0°C, crit = +100.0°C)
20.09.2023 01:21:56 ____________________________________________________________
Core 0:       +55.0°C  (high = +84.0°C, crit = +100.0°C)
Core 1:       +52.0°C  (high = +84.0°C, crit = +100.0°C)
Core 2:       +56.0°C  (high = +84.0°C, crit = +100.0°C)
Core 3:       +52.0°C  (high = +84.0°C, crit = +100.0°C)
20.09.2023 01:22:56 ____________________________________________________________
Core 0:       +55.0°C  (high = +84.0°C, crit = +100.0°C)
Core 1:       +52.0°C  (high = +84.0°C, crit = +100.0°C)
Core 2:       +55.0°C  (high = +84.0°C, crit = +100.0°C)
Core 3:       +52.0°C  (high = +84.0°C, crit = +100.0°C)

You can change the pause between displaying and writing info to the log file to any period you want, e.g. every 30 seconds, every minute (my example), every 5 minutes, every 10 minutes, every hour…


And my example being a mere log about the CPU core temperatures is also just an easy example. Use any command you want to be logged, just start with this part

Code: Select all

tz|tee -a $myfile;

since a log without info when something you logged started going off the rails (in this example: the temperature logged is approaching the critical temperature) is of not much help afterwards when your system did fail and you examine the log file (examining sensors_2023-09-20 in my example)

When you plan on leaving the machine on its own and you made sure the error you are monitoring for did not happen up to the current time you can erase the contents of the log file from another terminal.
Open a new tab of your terminal program and log in as root and do this:

Code: Select all

echo -n >/mnt/sda3/log/sensors_2023-09-20

When you would monitor the log file via a

Code: Select all

tail -f /mnt/sda3/log/sensors_2023-09-20

that tail will report this:

Code: Select all

tail: /mnt/sda3/log/sensors_2023-09-20: file truncated

But since the terminal that runs the script does display what the program reports and writes the output into the log file, you do not need to monitor the log via tail -f /mnt/sda3/log/sensors_2023-09-20


When you want to be extra secure do this

Append "3" to your Porteus kernel line at boot time

Your DE will not start, instead you will see the Virtual Terminal ("VT")
Log into the 1st VT as root, move to the correct folder and execute the script like so:

Code: Select all

root@rava:~# cd  /mnt/sda3/log/
root@rava:/mnt/sda3/log/# ./mysensorslog.sh 

Then switch to another VT, log in as guest and start X via

Code: Select all

startx

Your DE will start, but in your 1st VT your root script will continue to log .

And here it can be handy to monitor the log in your DE since you do not want to switch from your Xwindows session to the VT and back to Xwindows to just see the last entries of the log. Therefore a

Code: Select all

tail -f /mnt/sda3/log/sensors_2023-09-20

started from a Xwindows terminal Emulator would be a good idea.

I hope I was able to explain all details good enough so you understand the workings and can alter my script to fit your needs.

Added in 13 minutes 16 seconds:
Cave! Do not alter the script without any valid sleep command at the end of the while true loop.
I recommend not going any lower than logging every 10 seconds since every log strains your system, and remember: you have your DE and all your usual programs running as well, you do not want a mere log to go berserk in the background (filling up your partition that has the log file on it and also using too much IO / write cycles - e.g. when you only have a mere 1 second pause and log much more than my 4 lines of info and the one line with the date/time)

Please understand what your script does prior changing any script.

[Rava]

Only with the help of the wonderful people at https://stackoverflow.com my inept self was able to get a working code for creating a script that would execute make-ffplay-script for each folder directly in the current folder thus creating for each FOLDER a FOLDER.sh that will use ffplay for playing multimedia files that sit in FOLDER/ or in FOLDER/ and FOLDER/SUBFOLDER/

Cave! Even when the current version is 3.50, my make-ffplay-script is still too stupid to check if there are actual multimedia files in the folder it was given as parameter. It will create is FOLDERNAME.sh regardless and executing that script will do… nothing.
(You find my make-ffplay-script somewhere above)

Special thanks to user pjh @ stackoverflow

My preferred approach uses printf:

Code: Select all

for d in */ ; do printf 'make-ffplay-script %q\n' "${d%/}"; done > myfindloop_pjh.sh
chmod -v a+x myfindloop_pjh.sh

But echo with added " also works:

Code: Select all

for d in */ ; do echo make-ffplay-script \""$d"\"; done > myfindloop.sh
chmod -v a+x myfindloop.sh

[Rava]

Got tipped towards this https://mywiki.wooledge.org/BashFAQ/020 and it is just so good I have to share it. All of https://mywiki.wooledge.org/ is a read worthwhile when you are interested in the working of the shell and about script coding.

How can I find and safely handle file names containing newlines, spaces or both?

First and foremost, to understand why you're having trouble, read Arguments to get a grasp on how the shell understands the statements you give it. It is vital that you grasp this matter well if you're going to be doing anything with the shell.

The preferred method to deal with arbitrary filenames is still to use find(1):

Code: Select all

find ... -exec command {} \;

or, if you need to handle filenames en masse:

Code: Select all

find ... -exec command {} +

xargs is rarely ever more useful than the above, but if you really insist, remember to use -0 (-0 is not in the POSIX standard, but is implemented by GNU and BSD systems):

Code: Select all

# Requires GNU/BSD find and xargs
find ... -print0 | xargs -r0 command

# Never use xargs without -0 or similar extensions!

Use one of these unless you really can't.

Another way to deal with files with spaces in their names is to use the shell's filename expansion (globbing). This has the disadvantage of not working recursively (except with zsh's extensions or bash 4's globstar), and it normally does not include hidden files (filenames beginning with "."). But if you just need to process all the files in a single directory, and omitting hidden files is okay, it works fantastically well.

For example, this code renames all the *.mp3 files in the current directory to use underscores in their names instead of spaces (this uses the bash/ksh extension allowing "/" in parameter expansion):

Code: Select all

# Bash/ksh
for file in ./*\ *.mp3; do
  if [ -e "$file" ] ; then  # Make sure it isn't an empty match
    mv "$file" "${file// /_}"
  fi
done

You can omit the "if..." and "fi" lines if you're certain that at least one path will match the glob. The problem is that if the glob doesn't match, instead of looping 0 times (as you might expect), the loop will execute once with the unexpanded pattern (which is usually not what you want). You can also use the bash extension "shopt -s nullglob" to make empty globs expand to nothing, and then again you can omit the if and fi.

For more examples of renaming files, see FAQ #30.

Remember, you need to quote all your Parameter Expansions using double quotes. If you don't, the expansion will undergo WordSplitting (see also argument splitting and BashPitfalls). Also, always prefix globs with "/" or "./"; otherwise, if there's a file with "-" as the first character, the expansions might be misinterpreted as options.

Another way to handle filenames recursively involves using the -print0 option of find (a GNU/BSD extension), together with bash's -d extended option for read:

Code: Select all

# Bash
unset a i
while IFS= read -r -d $'\0' file; do
  a[i++]="$file"        # or however you want to process each file
done < <(find /tmp -type f -print0)

The preceding example reads all the files under /tmp (recursively) into an array, even if they have newlines or other whitespace in their names, by forcing read to use the NUL byte (\0) as its line delimiter. Since NUL is not a valid byte in Unix filenames, this is the safest approach besides using find -exec. IFS= is required to avoid trimming leading/trailing whitespace, and -r is needed to avoid backslash processing. In fact, $'\0' is actually the empty string (bash doesn't support passing NUL bytes to commands even built-in ones) so we could also write it like this:

Code: Select all

# Bash
unset a i
while IFS= read -r -d '' file; do
  a[i++]="$file"
done < <(find /tmp -type f -print0)

So, why doesn't this work?

Code: Select all

# DOES NOT WORK
unset a i
find /tmp -type f -print0 | while IFS= read -r -d '' file; do
  a[i++]="$file"
done

Because of the pipeline, the entire while loop is executed in a SubShell and therefore the array assignments will be lost after the loop terminates. (For more details about this, see FAQ #24.)

For a longer discussion about handling filenames in shell, see Filenames and Pathnames in Shell: How to do it Correctly.

CategoryShell

Other essential reads:
https://mywiki.wooledge.org/BashProgramming
https://mywiki.wooledge.org/BashPitfalls

[Rava]

Also from another thread on here, I just leave out the quoting myself part and copy what I wrote verbatim:

About self-discipline when it comes to script debugging (you can apply that to any programming language as well if possible, though the ' discard all program's output by directing it to "/dev/null" ' is a script thingy and will not apply verbatim like so in coding in e.g. C or C++ or python)
───────────────────────────────────────────────────


Remove all and any discard all program's output by directing it to /dev/null code

It might be even better if you create a copy of your script and use that copy to remove all /dev/null code and even add some "markers" like so:

#!/bin/bash
code
more code
echo DEBUG - 1
even more code
echo DEBUG - 2
even more code 2
even more code 3
echo DEBUG - 3
even more code 4
# etcetera pp

Highlighting to show you what I meant with 'add some "markers"' - if you just number the markers like in my example or give them unique descriptive names is up to you, usually I go by the more simple

Code: Select all

echo DEBUG1
[…]
echo DEBUG2

way of doing things. Also possible when you have a variable that gets changed is doing this:

Code: Select all

echo DEBUG1:
echo 'variable1$ - ' "variable1$"
echo 'variable2$ - ' "variable2$"
[…]
echo DEBUG2:
echo 'variable1$ - ' "variable1$"
echo 'variable2$ - ' "variable2$"

Just make sure when you do work on a copy of the script that you are always aware if you currently work on the copy or on the original of the script. Confusing both can make debugging even more confusing.

Work for the time being only on the copy with all >/dev/null removed - and when you get that script to work without any issues, either manually insert all >/dev/null into the copy, or insert the lesson or lessons learned from the copied-and-now-bugfree-script into your original.


Use whatever of these two technique floats your personal boat better, just not during debugging confuse one version with the other, like editing and saving one version but testing / running the other (=unedited one when it comes to the very last edit / debugging attempt) and being confused why your "changes" did make no difference.

Believe me, happened to me in the past more than once. And that created unnecessary headaches and confusion.
A good example of why slow and thorough will get you quicker (and better) results than quick and dirty and confusing.

[Rava]

Once more, the importance of knowing how to use the special parameter of "--" in the wild:

I know it's obvious to some but what am I missing??

Code: Select all

guest@porteus:/mnt/sda6$ grep -i "-2023 " testdisk.log
grep: invalid option -- ' '
Usage: grep [OPTION]... PATTERNS [FILE]...
Try 'grep --help' for more information.
guest@porteus:/mnt/sda6$ grep -i -2023  testdisk.log


^C
guest@porteus:/mnt/sda6$ cat testdisk.log | grep -i -2023 >> testdisk2023.txt
Usage: grep [OPTION]... PATTERNS [FILE]...
Try 'grep --help' for more information.
guest@porteus:/mnt/sda6$ cat testdisk.log | grep -i '-2023 ' >> testdisk2023.txt
grep: invalid option -- ' '
Usage: grep [OPTION]... PATTERNS [FILE]...
Try 'grep --help' for more information.
guest@porteus:/mnt/sda6$ cat testdisk.log | grep -i "-2023 " >> testdisk2023.txt
grep: invalid option -- ' '
Usage: grep [OPTION]... PATTERNS [FILE]...
Try 'grep --help' for more information.
guest@porteus:/mnt/sda6$ cat testdisk.log | grep -F "-2023 " >> testdisk2023.txt
grep: invalid option -- ' '
Usage: grep [OPTION]... PATTERNS [FILE]...
Try 'grep --help' for more information.
guest@porteus:/mnt/sda6$ cat testdisk.log | grep -F '-2023 ' >> testdisk2023.txt
grep: invalid option -- ' '
Usage: grep [OPTION]... PATTERNS [FILE]...
Try 'grep --help' for more information.
guest@porteus:/mnt/sda6$ 
 

I know the "-2023" exists in the file.

Code: Select all

   73575 -r--r--r--     0     0     10759 16-Sep-2023 00:23 My Utilities/FirefoxPortable/Data/profile/cache2/entries/5A84833AC677669F92C22170A71045DD3B224619

The obvious is you not thinking of using "--"

When something starting with a "-" is getting you trouble, use it as the last given parameter together with a prior "--" - that tells all programs it's the end of all options.

Created a test file "dummy", here its contents, and one grep run with -- and one without:

Code: Select all

guest@rava:/tmp$ cat dummy 
blabla
Kafkaesque
-2023 blubbs
Whatever
Kafkaesque-2023 bla
blablubb
guest@rava:/tmp$ grep -i -- "-2023 " dummy 
-2023 blubbs
Kafkaesque-2023 bla
guest@rava:/tmp$ grep -i "-2023 " dummy 
/bin/grep: invalid option -- ' '
Usage: grep [OPTION]... PATTERNS [FILE]...
Try 'grep --help' for more information.
guest@rava:/tmp$ 

Does that help to get your train back on the correct track?

Also, you not need the "-i" parameter here. Unless you are able to show me upper case numbers and lower case numbers. And no, "2" vs "²" and "3" vs "³" do not count.
Is a mere "2" upper case or lower case? And if is is one, what is the other case?

Answer: Arabic numerals do not have upper or lower cases.

Roman numerals on the other hand can have these:
IV vs iv and XXI vs xxi and such, though usually Roman numerals are written as upper case only.

Added in 34 minutes 55 seconds:
Fun fact: I looked up for some light and fun readings the wikipedia entries on "Arabic numerals" and "Roman numerals"
While Arabic numerals states

The ten Arabic numerals 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9 are the most commonly used symbols for writing numbers.

(highlighting by me)

Still, even when Arabic numerals are the most commonly used symbols for writing numbers - its wikipedia article Arabic numerals exists in 70 languages as of this writing, while Roman numerals exist in tremendous 104 languages.

Added in 1 hour 3 minutes 23 seconds:
And the above even continues. Like so:

I see less lines but I don't see an .sh. And grep -- '-2023 ' testdisk.log | grep -i .. ".sh "didn't work either.

Could it be you defeated yourself by using Dr. Copy and Mr. Paste?
Why is there a whitespace at the end of grep -i .. ".sh " ?
And what is the ".." meant to be?
Could it be you meant to use a simple | grep ".sh"

Though, be aware that for grep executed likes so the "." in the ".sh" statement is meant as "any character", you have to escape the "." to make grep search for a literal ".".

──────────────────────────────────────────────────────────

Now that this step is resolved do you have any suggestions for my wanting to use multiple selections when reading the log file? I want the -2023 lines plus .sh lines and .jpg lines.

What do you mean by "use multiple selections when reading the log file"

You can add a grep onto a grep:

Code: Select all

grep "Azur Lane" index.txt 

would find all and any entries of "Azur Lane"
Let's presume "index.txt" is created using find and thus has entries like so:

Code: Select all

/Random Folder/whatever.file
/sound/Azur Lane/fan-music.mp3
/video/Azur Lane/Azur Lane 01.mp4
/video/Kantai Collection/Kantai Collection 01.mp4

When you only want to find all entries for "Azur Lane" and "/video/" then you use | and another grep like so:

Code: Select all

grep "Azur Lane" index.txt | grep "/video/" 

When you want to find all "Azur Lane" hits but the "/video/" ones you use -v (Cave! In grep -v is not the same as -V and --version) like so:

Code: Select all

grep "Azur Lane" index.txt | grep -v "/video/" 

And when you want to combine several search patterns use -E like so to find all entries named "Azur Lane" and "Kantai Collection":

Code: Select all

grep -E "Azur Lane|Kantai Collection" index.txt 

Examples:

Code: Select all

guest@rava:/tmp$ cat index.txt 
/Random Folder/whatever.file
/sound/Azur Lane/fan-music.mp3
/video/Azur Lane/Azur Lane 01.mp4
/video/Kantai Collection/Kantai Collection 01.mp4
guest@rava:/tmp$ grep -E "Azur Lane|Kantai Collection" index.txt
/sound/Azur Lane/fan-music.mp3
/video/Azur Lane/Azur Lane 01.mp4
/video/Kantai Collection/Kantai Collection 01.mp4
guest@rava:/tmp$ grep "Azur Lane" index.txt | grep "/video/"
/video/Azur Lane/Azur Lane 01.mp4
guest@rava:/tmp$ grep "Azur Lane" index.txt | grep -v "/video/"
/sound/Azur Lane/fan-music.mp3
guest@rava:/tmp$ 

If none of this is what you meant, please explain what you meant by "use multiple selections when reading the log file".

[Rava]

About escaping - and the program less.

[…]

I'll look up how to escape a character.

Here snippet of how I escape the "." of filename extentions in my make-ffplay-script :

Code: Select all

grep -E "\.avi$|\.flv$|\.mp[34g]$|\.mkv$|\.ogv$|\.[tT][sS]$|\.webm$|\.divx$|\.[vV][oO][bB]$|\.ogg$|\.aac$"

To do it like so, you have to use both " and \ like in my code above.

Just searched through a gzipped text file via zless (that is just a wrapper for less) and was looking for a literal "0.6" and searching just for that gave me results for regex "0.6" as in its "." being regarded as "any character except newline"… e.g. this "4096" would be a hit, since it is "0"+"any character except newline"+"6"
Then instead of using the search term of "0.6" (without the brackets, of course) I used this:

Code: Select all

0\.6

et voilà! - it only displayed literal hits for "0.6" - this time the "." in the search parameter meaning only a literal "."

[Ed_P]

If viewing a file with fields of variable length that align to the left how does one get a field to align to the right? Is there a special character for that?

Code: Select all

Money field:
  100.00
  90.00
  1.50
How to get them to display as:
  100.00
   90.00
    1.50

The same problem for date fields:
  12 Aug 2023
  2 Oct 2023

[Rava]

^
viewing as in: using the terminal program "less"?

Added in 2 minutes 21 seconds:

Code: Select all

The same problem for date fields:
  12 Aug 2023
   2 Oct 2023

Looks pretty aligned to the right for me already?

[Ed_P]

I've never used "less". I'm working with awk and print.

Added in 38 seconds:

Looks pretty aligned to the right for me already?

Check again.

Added in 1 minute 33 seconds:
Time for bed.

[Rava]

Check again.

Seems you edited your article after I quoted it.

I've never used "less". I'm working with awk and print.

awk and print. awk can do that, and if by print you mean the printf function, that can do so as well. Alas! I hardly use these so I do not know the needed code necessary for accomplishing your goals off the top of my head. (I am more of a less and sed kind of guy)
──────────────────────────────────────────────────────
I mentioned my lsfind suite in the past several times. Broken down it identifies all my internal and external drives via empty files that sit in the root of the drive:

Code: Select all

guest@rava:~$ ls --color=auto -d /mnt/sd[a-z][1-9]/ext[HU][DS]* /mnt/sd[a-z]/ext[HU][DS]* /mnt/sd[a-z][1-9]/intHD* /mnt/sd[a-z][1-9]/extSD* 2>/dev/null
/mnt/sda1/intHDD1_1
/mnt/sda1/intHDD1_1_Toshiba_Satellite_500GB_intern
/mnt/sda2/intHDD1_2
/mnt/sda3/intHDD1_2
/mnt/sda4/intHDD1_4
/mnt/sda5/intHDD1_5
/mnt/sda7/intHDD1_7
/mnt/sda8/intHDD1_8
/mnt/sdb1/extHD11
/mnt/sdb1/extHD11_1
/mnt/sdb1/extHD11_1_2TB_ex_SynologyNAS
/mnt/sdb3/extHD11_3
/mnt/sdb3/extHD11_3_2TB_ex_SynologyNAS
/mnt/sdc2/extHD13_2
/mnt/sdc2/extHD13_Toshiba_2,5"_1TB_USB3
/mnt/sdc3/extHD13_3
/mnt/sdc4/extHD13_4
/mnt/sdd1/extSD_2GB
/mnt/sdd1/extSD_2GB_SDcard

intHD means an internal drive.
extHD means an external harddrive.
extSD means an external SD card.
extUSB would mean an external USB thumbdrive.

Now my lsfind suite creates two files for each partition it is indexing.
One done via

Code: Select all

cd /into/root-of-partition/
ls -loaR --time-style="+%Y-%m-%d %H:%M" >>"${TMPDIR}${base}.ls"

and the other via a simple

Code: Select all

cd /into/root-of-partition/
find . >>"${TMPDIR}${base}.find"

and doing a gz on "${TMPDIR}${base}.ls" and "${TMPDIR}${base}.find"

Then I can search these indexes to my hearts content. My lsfind approach of indexing all my drives have been especially valuable when my internal sm-Witless-7 "C:" partition crashed, it was the largest of my partitions by far on that 500 GB internal harddrive. The very same you see as intHDD1_* above.
But since I had a quite recently created and now renamed intHDD1_2-preserved.find.gz and intHDD1_2-preserved.ls.gz especially the ls.gz index has been very helpful in restoring files when using the then most recent version of testdisk 7.2WIP (2020 version) - seems the one Ed_P is currently using or has been using

Code: Select all

TestDisk 7.2-WIP, Data Recovery Utility, August 2023

is much improved compared to the one back then.

Anyhow, this is how a simple search can go: I have an ext2 container file that holds all my lsfind indexes and the scripts for indexing my drives:

Code: Select all

root@rava:/Lsfind# file /welt/lsfind/lsfind.ext2
/welt/lsfind/lsfind.ext2: Linux rev 1.0 ext2 filesystem data (mounted or unclean), UUID=WhAt-EvEr, volume name "LSFIND" (large files)
root@rava:/Lsfind# ls -o /welt/lsfind/lsfind.ext2
-rw-r--r-- 1 root 80740352 2023-10-17 08:52 /welt/lsfind/lsfind.ext2

By only adding a new 2 GB external harddrive to my fold and to the drives being indexed, I had to increase my lsfind.ext2 from 70.00 MB to 77.00 MM.

Cave! These are the external sizes, due to the need of the filesystem to have data on its own, the internal available capacity is significantly smaller - the currently mounted on /Lsfind has this

Code: Select all

root@rava:/Lsfind# /bin/df -Tm /Lsfind/
Filesystem     Type 1M-blocks  Used Available Use% Mounted on
/dev/loop19    ext2        71    68         4  96% /Lsfind

You see, and external ext2 container file of 77MB only hold 71MB of data internally.

And of course, lazy but coding-able as I am, I created a help.* file for just creating such container files. Its part of my help.dd.random+cp script:

Code: Select all

root@rava:/Lsfind# help.dd.random+cp 
help.dd.random+cp.sh V0.9
dd if=/dev/urandom of=random.data count=1024c bs=1 # prefered, more secure
dd if=/dev/random of=random.data count=1024c bs=1  # quicker? less secure!
dd if=/dev/hda of=cd.iso bs=2048		# copies nonprotected CD to file
dd if=/dev/sda1 of=sda1.mbr count=512c bs=1	# backups MBR of sda1 into file
dd if=/dev/zero of=16MB.img bs=1k count=16384	# creates an empty 16 MB file
dd if=/dev/zero of=1GB.file bs=1M count=1024	# creates an empty 1 GB file
  -> then use mkfs.ext3 -F 1GB.file to create an ext3 fs
  or for ext2 / lsfind:
  --> dd if=/dev/zero of=lsfind.ext2 bs=1M count=60 # create 60 MB container
  --> mkfs.ext2 -F lsfind.ext2 -L LSFIND # format
  --> tune2fs -r 50 lsfind.ext2          # tune
"myNrg2iso" :P
dd skip=307200 if=image.nrg of=image.iso

Now finally all these extra explanations out of the way, back to what I wanted to show you all along, a search on all of my indexes in /Lsfind:

Code: Select all

 root@rava:/Lsfind# zgrep leafpad *gz|grep xzm$|grep -v leafpad-0.8.18.1
root@rava:/Lsfind# 

Is an empty search result that intriguing?
In this case, yes, it was for me.
The original search - using wc to summarise:

Code: Select all

root@rava:/Lsfind# zgrep leafpad *gz|grep xzm$|wc
     78     334    8207

78 lines to go through if there is a version other than leafpad-0.8.18.1 ? There is a much better way accomplishing this by excluding just "leafpad-0.8.18.1".
Therefore I used grep -v leafpad-0.8.18.1 to exclude that phrase from the previous results (done via zgrep leafpad *gz|grep xzm$ ) and the empty result tells me: all previous hits have been only about leafpad-0.8.18.1

zgrep leafpad *gz of course gives all hits for any kind of "leafpad":

Code: Select all

root@rava:/Lsfind# zgrep leafpad *gz|wc
    515    1983   48127

515 lines.

And reducing that to what I am interested in, I used the special character of "$" indicating "end of line" since any *.xzm file would end in just that phrase.

I could have used and escaped the "." in ".xzm" as well if I wanted:

Code: Select all

root@rava:/Lsfind# zgrep leafpad *gz|grep ".\xzm$"|wc
     78     334    8207
root@rava:/Lsfind# zgrep leafpad *gz|grep xzm$|wc
     78     334    8207

but as you see, for this search it would have made no difference.

Though, it can be.

Like when you search for the more generic file ending of "sh" - most shell scripts end like so, but unlike "xzm" "sh" can easily be ending part of folder of file names as well.

Code: Select all

guest@rava:/myvideo$ ls -d *sh/
Grimgar\ of\ Fantasy\ and\ Ash/

and thus, a simple search like so

Code: Select all

zgrep whatever *gz|grep sh$

will result in many false positive hits when this search

Code: Select all

zgrep whatever *gz|grep "\.sh$"

would reduce these false positive hits.