HOWTO search multiple modules

Technical issues/questions of an intermediate or advanced nature.
User avatar
Rava
Contributor
Contributor
Posts: 2480
Joined: 11 Jan 2011, 02:46
Distribution: Porteus 5.0rc1 x86-64 XFCe
Location: Forests of Germany

HOWTO search multiple modules

Post#1 by Rava » 21 Dec 2018, 17:21

Has anyone ever attempted to create a search script that is able to temporary "mount" several Port modules and search for a file pattern in these modules? (Like a local search could be done e.g. via "find [path] | grep [pattern]"

The script should be able to do so without having to use lots of RAM. B)

Currently I cannot think of an elegant way to code it by myself. :(

Update
I also asked on https://stackoverflow.com and posted a working result in #9: search multiple modules
Cheers!
Yours Rava

User avatar
brokenman
Site Admin
Site Admin
Posts: 6104
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil
Contact:

Re: search multiple modules

Post#2 by brokenman » 21 Dec 2018, 19:00

I'm not on linux right now but a simple loop should be able to list the content modules and then grep for what you need.

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    lsxzm $module | grep "$2"
done
You would call this script with 2 arguments.

1. Path to a bunch of modules
2. Argument to grep for.

./path/to/script.sh /path/to/mymodules docs
This would search for modules at /path/to/mymodules and show results containing the word docs.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Rava
Contributor
Contributor
Posts: 2480
Joined: 11 Jan 2011, 02:46
Distribution: Porteus 5.0rc1 x86-64 XFCe
Location: Forests of Germany

Re: search multiple modules

Post#3 by Rava » 21 Dec 2018, 20:47

Currently I simply try to find where the 2 missing dependencies for mtPaint are for the most recent 5.0 testversion with Openbox DE.
The missing libs are

Code: Select all

ldd /usr/bin/mtpaint|grep not
	libtiff.so.3 => not found
	libjpeg.so.8 => not found
Put your code into a text executable named /usr/local/bin/lsxzmgrep . The result looks like so:

Code: Select all

root@porteus:/mnt/sda1/Porteus_4.0/porteus/base# lsxzmgrep . libtiff.so.3
/usr/lib64/libtiff.so.3
/usr/lib64/libtiff.so.3.9.7
So, it lists the files, but one crucial info is missing: the name of the module(s) that contain the found files. All we know that they are in one of the modules in /mnt/sda1/Porteus_4.0/porteus/base

The for do loop could list all modules it is searching, but best would be to only list the ones that get results with the scripts grep pattern "$2"...

I only could think one one way achieving it: buffering the grep results instead of printing it right away, and if the grep found something, then first print the module name, and then the results, if grep finds nothing continue with the next module just like the loop would have done anyway.

Or even better, do it like grep would do it when searching multiple files, print the module name with added ": " in front of the "grep result" list:

Code: Select all

root@porteus:/mnt/sda1/Porteus_4.0/porteus/base# lsxzmgrep . libtiff.so.3
./example1.xzm: /usr/lib64/libtiff.so.3
./example1.xzm: /usr/lib64/libtiff.so.3.9.7
Maybe there is even a trick I am not aware of, so that we can archive the above without the need of buffering the grep results and put together the lines like
echo ${module}:{grep_result(n)}
echo ${module}:{grep_result(n+1)}
...
Cheers!
Yours Rava

User avatar
Ed_P
Contributor
Contributor
Posts: 5501
Joined: 06 Feb 2013, 22:12
Distribution: 4.0 Cinnamon 64-bit ISO
Location: Western NY, USA

Re: search multiple modules

Post#4 by Ed_P » 21 Dec 2018, 21:43

:%) If the "missing" libs are in the Porteus base then the app, mtpaint, will run. They don't need to be in the module also.
Ed

User avatar
Rava
Contributor
Contributor
Posts: 2480
Joined: 11 Jan 2011, 02:46
Distribution: Porteus 5.0rc1 x86-64 XFCe
Location: Forests of Germany

Re: search multiple modules

Post#5 by Rava » 21 Dec 2018, 21:47

Ed_P wrote:
21 Dec 2018, 21:43
:%) If the "missing" libs are in the Porteus base then the app, mtpaint, will run. They don't need to be in the module also.
No, it won't.

I currently run 5.0 from a different base folder. And quite obviously, 5.0 lacks the 2 libs as listed above.
This is the reason why I got the mtPaint error /when running 5.0 and trying to start mtpaint) with the listed missing libs as quoted above. See here for the quote search multiple modules - post #3 by Rava
When booting 4.0 it sure works okay. But this is not the question of the thread anyway. *points to thread title*
Cheers!
Yours Rava

User avatar
Ed_P
Contributor
Contributor
Posts: 5501
Joined: 06 Feb 2013, 22:12
Distribution: 4.0 Cinnamon 64-bit ISO
Location: Western NY, USA

Re: search multiple modules

Post#6 by Ed_P » 22 Dec 2018, 00:24

Rava wrote:
21 Dec 2018, 20:47
Put your code into a text executable named /usr/local/bin/lsxzmgrep . The result looks like so:

Code: Select all

root@porteus:/mnt/sda1/Porteus_4.0/porteus/base# lsxzmgrep . libtiff.so.3
/usr/lib64/libtiff.so.3
/usr/lib64/libtiff.so.3.9.7
Ok, I missed the part that this was done with Porteus 4.0's base where mtpaint works.

As for brokenman's code this might yield something close to what you're looking for.

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    echo " * " $module
    lsxzm $module | grep "$2"
done
Yes, it will display all module names and the name proceeding the grep hits is the module name you want.
Last edited by Ed_P on 22 Dec 2018, 06:28, edited 1 time in total.
Reason: Tweaked the echo.
Ed

User avatar
Rava
Contributor
Contributor
Posts: 2480
Joined: 11 Jan 2011, 02:46
Distribution: Porteus 5.0rc1 x86-64 XFCe
Location: Forests of Germany

Re: search multiple modules

Post#7 by Rava » 22 Dec 2018, 01:55

Ed_P wrote:
22 Dec 2018, 00:24
As for brokenman's code this might yield something close to what you're looking for.

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    echo $module
    lsxzm $module | grep "$2"
done
Yes, it will display all module names and the name proceeding the grep hits is the module name you want.
Indeed it is not ideal, but for now it works okay, since the module name is important, or else the result is more or less meaningless. "Yes, the files are in one of the 4 or 8 modules... but dunno in which ones" is sure not a result anyone wants.

Code: Select all

root@porteus:/mnt/sda1/Porteus_4.0/porteus/base# lsxzmgrep . libtiff.so.3
./000-kernel.xzm
./001-core.xzm
/usr/lib64/libtiff.so.3
/usr/lib64/libtiff.so.3.9.7
./001-de-core_locales.xzm
./002-xorg.xzm
./003-xfce.xzm
./010-nvidia-304.137-k.4.16.3-x86_64-don.xzm
./991-usr_local_binV2018-06-25.xzm
Not very pretty, but it gives the results we wanted: now we know that one of the libs are in ./001-core.xzm :Yahoo!:

Update
And after adding a small module, mtpaint works like a charm. :)

Code: Select all

root 454656 2018-12-22 03:03 023-mtpaint-libs.xzm

Code: Select all

root@porteus:/# 1024calc 454656
444 KB
444.00 KB
root@porteus:/# cat /usr/local/bin/1024calc 
#!/usr/bin/awk -f

BEGIN{	x = ARGV[1]

        split("B KB MB GB TB PB",type)

	for(i=5;y < 1;i--)
	    y = x / (2**(10*i))

	print y " " type[i+2]
	printf("%.2f %s\n",y,type[i+2])
}
(with the large blocksize of standard xzm modules, no wonder the size calculated as KB ends with "X.00" )

And I even inserted more than the libs mtpaint originally complained as quoted above, I also added higher numbered / never versions of the libs, just to make sure it won't fail.
Cheers!
Yours Rava

User avatar
Rava
Contributor
Contributor
Posts: 2480
Joined: 11 Jan 2011, 02:46
Distribution: Porteus 5.0rc1 x86-64 XFCe
Location: Forests of Germany

Re: search multiple modules

Post#8 by Rava » 25 Dec 2018, 05:30

I asked about it on https://stackoverflow.com/questions/539 ... ve-results and got this code snipped as solution:

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    lsxzm ${module} | grep "$2" | sed -r "s/.+/${module}\n&/"
done
But sadly that won`t work:

Code: Select all

/porteus/base$ lsxzmgrep . libtiff.so.3
sed: -e expression #1, char 10: number option to `s' command may not be zero
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
Cheers!
Yours Rava

User avatar
Rava
Contributor
Contributor
Posts: 2480
Joined: 11 Jan 2011, 02:46
Distribution: Porteus 5.0rc1 x86-64 XFCe
Location: Forests of Germany

Re: search multiple modules

Post#9 by Rava » 25 Dec 2018, 07:10

And a bit later someone posted a working code:

Code: Select all

#!/bin/bash
for module in $(find $1 -type f -name "*.xzm"); do
    result=$(lsxzm $module | grep "$2")
    if [ $? -eq 0 ]; then
        # grep returned OK, write out results
        echo $module
        echo "$result"
    fi
done
I was hoping there being some neat bash script coding-fu that gets the trick done without the need of storing grep's result, but seems there is no such trick. Or the ones who know of such trick not reply.
Cheers!
Yours Rava

User avatar
Ed_P
Contributor
Contributor
Posts: 5501
Joined: 06 Feb 2013, 22:12
Distribution: 4.0 Cinnamon 64-bit ISO
Location: Western NY, USA

Re: HOWTO search multiple modules

Post#10 by Ed_P » 25 Dec 2018, 17:28

I was thinking about playing with an "if [ $? -eq 0 ]; then" option but after you showed the simple script showed only 2 libs I figured why bother.
https://stackoverflow.com/questions/53918069/live-linux-grep-script-only-print-positive-results wrote:I try to code a bash 4 (or sh, if that's possible) script that does the following: 1st parameter: the path to the compressed modules to search, could be "." 2nd the search pattern to look for
"I" :lol:
Ed

User avatar
brokenman
Site Admin
Site Admin
Posts: 6104
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil
Contact:

Re: HOWTO search multiple modules

Post#11 by brokenman » 26 Dec 2018, 13:47

Or even better, do it like grep would do it when searching multiple files, print the module name with added ": " in front of the "grep result" list:

Code: Select all

grep -Hn
This will show you the filename and the line number.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Rava
Contributor
Contributor
Posts: 2480
Joined: 11 Jan 2011, 02:46
Distribution: Porteus 5.0rc1 x86-64 XFCe
Location: Forests of Germany

Re: HOWTO search multiple modules

Post#12 by Rava » 26 Dec 2018, 14:00

brokenman wrote:
26 Dec 2018, 13:47

Code: Select all

grep -Hn
This will show you the filename and the line number.
Hmm I got an update on stackoverflow:

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    lsxzm ${module} | grep "$2" | sed -r "s#.+#${module}\n&#"
done
That results in:

Code: Select all

/porteus/base# /tmp/lsxzmgrep2 . output_dummy
./002-xorg.xzm
/usr/lib64/mpg123/output_dummy.so
And it works without saving the grep result in a variable, or running grep two times. So quite neat solution for now.

When I combine that with your approach

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    lsxzm ${module} | grep -Hn "$2" | sed -r "s#.+#${module}\n&#"
done
I get this:

Code: Select all

/porteus/base# /tmp/lsxzmgrep2 . output_dummy
./002-xorg.xzm
(standard input):2562:/usr/lib64/mpg123/output_dummy.so
I think that is not the result you wanted it to have.
Cheers!
Yours Rava

User avatar
Rava
Contributor
Contributor
Posts: 2480
Joined: 11 Jan 2011, 02:46
Distribution: Porteus 5.0rc1 x86-64 XFCe
Location: Forests of Germany

Re: HOWTO search multiple modules

Post#13 by Rava » 14 Jan 2019, 05:40

So, to conclude the HOWTO, for now it seems this script code

Code: Select all

#!/bin/bash

for module in $(find $1 -type f -name "*.xzm"); do
    lsxzm ${module} | grep "$2" | sed -r "s#.+#${module}\n&#"
done
does create the best results. :) :punk:

Thanks to all that helped in creating the code!

____________________________________________

(I should add a bit of debugging, e.g. when lsxzmgrep is just started like so its output is less helpful to the user, to say the least:

Code: Select all

root@porteus:/tmp/64bit-kernel4.20.2# lsxzmgrep initrd
find: ‘initrd’: No such file or directory
root@porteus:/tmp/64bit-kernel4.20.2# lsxzmgrep . initrd
./06-crippled_sources-4.20.2-64bit.xzm
/usr/src/linux-4.20.2/include/config/blk/dev/initrd.h
./06-crippled_sources-4.20.2-64bit.xzm
/usr/src/linux-4.20.2/include/linux/initrd.h
./06-crippled_sources-4.20.2-64bit.xzm
/usr/src/linux-4.20.2/init/.do_mounts_initrd.o.cmd
./06-crippled_sources-4.20.2-64bit.xzm
/usr/src/linux-4.20.2/init/do_mounts_initrd.c
./06-crippled_sources-4.20.2-64bit.xzm
/usr/src/linux-4.20.2/tools/testing/selftests/rcutorture/doc/initrd.txt
How should the user know that the correct syntax for lsxzmgrep is "lsxzmgrep [path] [grep-pattern]")
Cheers!
Yours Rava

User avatar
Ed_P
Contributor
Contributor
Posts: 5501
Joined: 06 Feb 2013, 22:12
Distribution: 4.0 Cinnamon 64-bit ISO
Location: Western NY, USA

Re: HOWTO search multiple modules

Post#14 by Ed_P » 14 Jan 2019, 05:58

Or that the name of the script code is lsxzmgrep? :D
Ed

User avatar
Rava
Contributor
Contributor
Posts: 2480
Joined: 11 Jan 2011, 02:46
Distribution: Porteus 5.0rc1 x86-64 XFCe
Location: Forests of Germany

Re: HOWTO search multiple modules

Post#15 by Rava » 14 Jan 2019, 06:00

Ed_P wrote:
14 Jan 2019, 05:58
Or that the name of the script code is lsxzmgrep? :D
How can a user give a command the wrong syntax without calling the command? Please do explain how this is possible.

Your reply makes no sense, like, at all.
Cheers!
Yours Rava

Post Reply