HOWTO search multiple modules

[Rava]

Has anyone ever attempted to create a search script that is able to temporary "mount" several Port modules and search for a file pattern in these modules? (Like a local search could be done e.g. via "find [path] | grep [pattern]"

The script should be able to do so without having to use lots of RAM.

Currently I cannot think of an elegant way to code it by myself.

Update
I also asked on https://stackoverflow.com and posted a working result in #9: search multiple modules

[brokenman]

I'm not on linux right now but a simple loop should be able to list the content modules and then grep for what you need.

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    lsxzm $module | grep "$2"
done

You would call this script with 2 arguments.

1. Path to a bunch of modules
2. Argument to grep for.

./path/to/script.sh /path/to/mymodules docs
This would search for modules at /path/to/mymodules and show results containing the word docs.

[Rava]

Currently I simply try to find where the 2 missing dependencies for mtPaint are for the most recent 5.0 testversion with Openbox DE.
The missing libs are

Code: Select all

ldd /usr/bin/mtpaint|grep not
	libtiff.so.3 => not found
	libjpeg.so.8 => not found

Put your code into a text executable named /usr/local/bin/lsxzmgrep . The result looks like so:

Code: Select all

root@porteus:/mnt/sda1/Porteus_4.0/porteus/base# lsxzmgrep . libtiff.so.3
/usr/lib64/libtiff.so.3
/usr/lib64/libtiff.so.3.9.7

So, it lists the files, but one crucial info is missing: the name of the module(s) that contain the found files. All we know that they are in one of the modules in /mnt/sda1/Porteus_4.0/porteus/base

The for do loop could list all modules it is searching, but best would be to only list the ones that get results with the scripts grep pattern "$2"...

I only could think one one way achieving it: buffering the grep results instead of printing it right away, and if the grep found something, then first print the module name, and then the results, if grep finds nothing continue with the next module just like the loop would have done anyway.

Or even better, do it like grep would do it when searching multiple files, print the module name with added ": " in front of the "grep result" list:

Code: Select all

root@porteus:/mnt/sda1/Porteus_4.0/porteus/base# lsxzmgrep . libtiff.so.3
./example1.xzm: /usr/lib64/libtiff.so.3
./example1.xzm: /usr/lib64/libtiff.so.3.9.7

Maybe there is even a trick I am not aware of, so that we can archive the above without the need of buffering the grep results and put together the lines like
echo ${module}:{grep_result(n)}
echo ${module}:{grep_result(n+1)}
...

[Ed_P]

If the "missing" libs are in the Porteus base then the app, mtpaint, will run. They don't need to be in the module also.

[Rava]

If the "missing" libs are in the Porteus base then the app, mtpaint, will run. They don't need to be in the module also.

No, it won't.

I currently run 5.0 from a different base folder. And quite obviously, 5.0 lacks the 2 libs as listed above.
This is the reason why I got the mtPaint error /when running 5.0 and trying to start mtpaint) with the listed missing libs as quoted above. See here for the quote search multiple modules - post #3 by Rava
When booting 4.0 it sure works okay. But this is not the question of the thread anyway. *points to thread title*

[Ed_P]

Put your code into a text executable named /usr/local/bin/lsxzmgrep . The result looks like so:

Code: Select all

root@porteus:/mnt/sda1/Porteus_4.0/porteus/base# lsxzmgrep . libtiff.so.3
/usr/lib64/libtiff.so.3
/usr/lib64/libtiff.so.3.9.7

Ok, I missed the part that this was done with Porteus 4.0's base where mtpaint works.

As for brokenman's code this might yield something close to what you're looking for.

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    echo " * " $module
    lsxzm $module | grep "$2"
done

Yes, it will display all module names and the name proceeding the grep hits is the module name you want.

[Rava]

As for brokenman's code this might yield something close to what you're looking for.

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    echo $module
    lsxzm $module | grep "$2"
done

Yes, it will display all module names and the name proceeding the grep hits is the module name you want.

Indeed it is not ideal, but for now it works okay, since the module name is important, or else the result is more or less meaningless. "Yes, the files are in one of the 4 or 8 modules... but dunno in which ones" is sure not a result anyone wants.

Code: Select all

root@porteus:/mnt/sda1/Porteus_4.0/porteus/base# lsxzmgrep . libtiff.so.3
./000-kernel.xzm
./001-core.xzm
/usr/lib64/libtiff.so.3
/usr/lib64/libtiff.so.3.9.7
./001-de-core_locales.xzm
./002-xorg.xzm
./003-xfce.xzm
./010-nvidia-304.137-k.4.16.3-x86_64-don.xzm
./991-usr_local_binV2018-06-25.xzm

Not very pretty, but it gives the results we wanted: now we know that one of the libs are in ./001-core.xzm

Update
And after adding a small module, mtpaint works like a charm.

Code: Select all

root 454656 2018-12-22 03:03 023-mtpaint-libs.xzm

Code: Select all

root@porteus:/# 1024calc 454656
444 KB
444.00 KB
root@porteus:/# cat /usr/local/bin/1024calc 
#!/usr/bin/awk -f

BEGIN{	x = ARGV[1]

        split("B KB MB GB TB PB",type)

	for(i=5;y < 1;i--)
	    y = x / (2**(10*i))

	print y " " type[i+2]
	printf("%.2f %s\n",y,type[i+2])
}

(with the large blocksize of standard xzm modules, no wonder the size calculated as KB ends with "X.00" )

And I even inserted more than the libs mtpaint originally complained as quoted above, I also added higher numbered / never versions of the libs, just to make sure it won't fail.

[Rava]

I asked about it on https://stackoverflow.com/questions/539 ... ve-results and got this code snipped as solution:

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    lsxzm ${module} | grep "$2" | sed -r "s/.+/${module}\n&/"
done

But sadly that won`t work:

Code: Select all

/porteus/base$ lsxzmgrep . libtiff.so.3
sed: -e expression #1, char 10: number option to `s' command may not be zero
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'
sed: -e expression #1, char 11: unknown option to `s'

[Rava]

And a bit later someone posted a working code:

Code: Select all

#!/bin/bash
for module in $(find $1 -type f -name "*.xzm"); do
    result=$(lsxzm $module | grep "$2")
    if [ $? -eq 0 ]; then
        # grep returned OK, write out results
        echo $module
        echo "$result"
    fi
done

I was hoping there being some neat bash script coding-fu that gets the trick done without the need of storing grep's result, but seems there is no such trick. Or the ones who know of such trick not reply.

[Ed_P]

I was thinking about playing with an "if [ $? -eq 0 ]; then" option but after you showed the simple script showed only 2 libs I figured why bother.

I try to code a bash 4 (or sh, if that's possible) script that does the following: 1st parameter: the path to the compressed modules to search, could be "." 2nd the search pattern to look for

"I"

[brokenman]

Or even better, do it like grep would do it when searching multiple files, print the module name with added ": " in front of the "grep result" list:

Code: Select all

grep -Hn

This will show you the filename and the line number.

[Rava]

Code: Select all

grep -Hn

This will show you the filename and the line number.

Hmm I got an update on stackoverflow:

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    lsxzm ${module} | grep "$2" | sed -r "s#.+#${module}\n&#"
done

That results in:

Code: Select all

/porteus/base# /tmp/lsxzmgrep2 . output_dummy
./002-xorg.xzm
/usr/lib64/mpg123/output_dummy.so

And it works without saving the grep result in a variable, or running grep two times. So quite neat solution for now.

When I combine that with your approach

Code: Select all

for module in $(find $1 -type f -name "*.xzm"); do
    lsxzm ${module} | grep -Hn "$2" | sed -r "s#.+#${module}\n&#"
done

I get this:

Code: Select all

/porteus/base# /tmp/lsxzmgrep2 . output_dummy
./002-xorg.xzm
(standard input):2562:/usr/lib64/mpg123/output_dummy.so

I think that is not the result you wanted it to have.

[Rava]

So, to conclude the HOWTO, for now it seems this script code

Code: Select all

#!/bin/bash

for module in $(find $1 -type f -name "*.xzm"); do
    lsxzm ${module} | grep "$2" | sed -r "s#.+#${module}\n&#"
done

does create the best results.

Thanks to all that helped in creating the code!

____________________________________________

(I should add a bit of debugging, e.g. when lsxzmgrep is just started like so its output is less helpful to the user, to say the least:

Code: Select all

root@porteus:/tmp/64bit-kernel4.20.2# lsxzmgrep initrd
find: ‘initrd’: No such file or directory
root@porteus:/tmp/64bit-kernel4.20.2# lsxzmgrep . initrd
./06-crippled_sources-4.20.2-64bit.xzm
/usr/src/linux-4.20.2/include/config/blk/dev/initrd.h
./06-crippled_sources-4.20.2-64bit.xzm
/usr/src/linux-4.20.2/include/linux/initrd.h
./06-crippled_sources-4.20.2-64bit.xzm
/usr/src/linux-4.20.2/init/.do_mounts_initrd.o.cmd
./06-crippled_sources-4.20.2-64bit.xzm
/usr/src/linux-4.20.2/init/do_mounts_initrd.c
./06-crippled_sources-4.20.2-64bit.xzm
/usr/src/linux-4.20.2/tools/testing/selftests/rcutorture/doc/initrd.txt

How should the user know that the correct syntax for lsxzmgrep is "lsxzmgrep [path] [grep-pattern]")

[Ed_P]

Or that the name of the script code is lsxzmgrep?

[Rava]

Or that the name of the script code is lsxzmgrep?

How can a user give a command the wrong syntax without calling the command? Please do explain how this is possible.

Your reply makes no sense, like, at all.