Page 1 of 1
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 21 Nov 2022, 02:54
by Rava
Code: Select all
guest@porteus:/tmp/ls-test$ touch a.sh "a and b.sh" "a and b and c.sh"
guest@porteus:/tmp/ls-test$ chmod a+x *.sh
guest@porteus:/tmp/ls-test$ ls -otr ./*.sh
-rwxr-xr-x 1 guest 0 2022-11-21 03:41 ./a.sh
-rwxr-xr-x 1 guest 0 2022-11-21 03:41 ./a\ and\ b.sh
-rwxr-xr-x 1 guest 0 2022-11-21 03:41 ./a\ and\ b\ and\ c.sh
guest@porteus:/tmp/ls-test$ ls -otr ./*.sh |tail -n 25
-rwxr-xr-x 1 guest 0 2022-11-21 03:41 ./a.sh
-rwxr-xr-x 1 guest 0 2022-11-21 03:41 ./a and b.sh
-rwxr-xr-x 1 guest 0 2022-11-21 03:41 ./a and b and c.sh
Now, when the local folder has many scripts (like what happens in my audio or video folders with the .sh scripts as created by my make-ffplay-script) - I would like to append a tail -n 25 to only see the most recent scripts.
And I would like the output keep its escaping a whitespace via "\ " ... so that I can copy and paste that directly with the mouse, for when i want to execute one of these scripts via terminal.
But
unfortunately ls changes its output when it detects that it will be piped to another program as you can see above.
Unfortunately² there is nothing in
man ls explaining that or ways to control this behaviour, I looked for "pipe" and "white" as in whitespace, but to no avail (no hit for any of these keywords).
Anyone any idea how one gets ls to use the notion of escaping white spaces with "\ " even when its output is piped?
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 21 Nov 2022, 23:25
by ncmprhnsbl
Rava wrote: ↑21 Nov 2022, 02:54
But unfortunately ls changes its output when it detects that it will be piped to another program
i don't have a solution, but, it seems to me that ls would have no way to "detects that it will be piped to another program"
i would have thought that ls just dumps it's output to STDOUT, and that's it. ..and that this is more to do with how tail reads that..
and that it's removing escapes in it's output..
here's something not entirely unrelated, though possibly not helpful in your use case:
https://unix.stackexchange.com/question ... removing-n
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 22 Nov 2022, 05:52
by Rava
ncmprhnsbl wrote: ↑21 Nov 2022, 23:25
i don't have a solution, but, it seems to me that ls would have no way to "detects that it will be piped to another program"
at least it seems to know when it will be directed into a file.
Look for yourself, I use [
quote
] because then I can use colours to highlight what I mean:
guest@porteus:/tmp$ mkdir test
guest@porteus:/tmp$ cd test/
guest@porteus:/tmp/test$ touch test.txt test.sh
guest@porteus:/tmp/test$ chmod a+x test.sh
guest@porteus:/tmp/test$ /bin/ls -o *
-rwxr-xr-x 1 guest 0 2022-11-22 06:42 test.sh
-rw-r--r-- 1 guest 0 2022-11-22 06:42 test.txt
guest@porteus:/tmp/test$ /bin/ls -o --color=auto *
-rwxr-xr-x 1 guest 0 2022-11-22 06:42 test.sh
-rw-r--r-- 1 guest 0 2022-11-22 06:42 test.txt
guest@porteus:/tmp/test$ /bin/ls -o --color=auto * >tmp
guest@porteus:/tmp/test$ cat tmp
-rwxr-xr-x 1 guest 0 2022-11-22 06:42 test.sh
-rw-r--r-- 1 guest 0 2022-11-22 06:42 test.txt
Please do replicate the above on your own system to see for yourself.
cat sure is always outputting the contents of a file verbatim. Still, its output of "tmp" file lacks any colours.
The ls output of
uses colours for test.sh - then why does the redirection into a file lacks colours?
It cannot be tail or any other program this time altering the output of ls, only
ls is responsible for the contents of the file "tmp".
EDIT
when I use --color=always instead of --color=auto the tmp file contains the colour codes and thus cat displays the one file in colours:
guest@porteus:/tmp/test$ /bin/ls -o --color=always *
-rwxr-xr-x 1 guest 0 2022-11-22 06:42 test.sh
-rw-r--r-- 1 guest 0 2022-11-22 06:42 test.txt
-rw-r--r-- 1 guest 93 2022-11-22 06:45 tmp
guest@porteus:/tmp/test$ /bin/ls -o --color=always * >tmp
guest@porteus:/tmp/test$ cat tmp
-rwxr-xr-x 1 guest 0 2022-11-22 06:42 test.sh
-rw-r--r-- 1 guest 0 2022-11-22 06:42 test.txt
-rw-r--r-- 1 guest 0 2022-11-22 06:58 tmp
guest@porteus:/tmp/test$ ls -o tmp
-rw-r--r-- 1 guest 169 2022-11-22 06:58 tmp
guest@porteus:/tmp/test$ /bin/ls -o --color=auto * >tmp
guest@porteus:/tmp/test$ ls -o tmp
-rw-r--r-- 1 guest 135 2022-11-22 07:01 tmp
I presume the difference between 135 bytes for the "tmp" file and 169 bytes for it is mainly due to the escape sequence responsible for the colour output.
Here I use hexdump -C to illustrate the difference between the 135 bytes version of "tmp" and its 169 bytes version:
Code: Select all
guest@porteus:/tmp/test$ ls -o tmp
-rw-r--r-- 1 guest 135 2022-11-22 07:01 tmp
guest@porteus:/tmp/test$ hexdump -C tmp
00000000 2d 72 77 78 72 2d 78 72 2d 78 20 31 20 67 75 65 |-rwxr-xr-x 1 gue|
00000010 73 74 20 30 20 32 30 32 32 2d 31 31 2d 32 32 20 |st 0 2022-11-22 |
00000020 30 36 3a 34 32 20 74 65 73 74 2e 73 68 0a 2d 72 |06:42 test.sh.-r|
00000030 77 2d 72 2d 2d 72 2d 2d 20 31 20 67 75 65 73 74 |w-r--r-- 1 guest|
00000040 20 30 20 32 30 32 32 2d 31 31 2d 32 32 20 30 36 | 0 2022-11-22 06|
00000050 3a 34 32 20 74 65 73 74 2e 74 78 74 0a 2d 72 77 |:42 test.txt.-rw|
00000060 2d 72 2d 2d 72 2d 2d 20 31 20 67 75 65 73 74 20 |-r--r-- 1 guest |
00000070 30 20 32 30 32 32 2d 31 31 2d 32 32 20 30 37 3a |0 2022-11-22 07:|
00000080 30 31 20 74 6d 70 0a |01 tmp.|
00000087
guest@porteus:/tmp/test$ /bin/ls -o --color=always * >tmp
guest@porteus:/tmp/test$ ls -o tmp
-rw-r--r-- 1 guest 169 2022-11-22 07:05 tmp
guest@porteus:/tmp/test$ hexdump -C tmp
00000000 2d 72 77 78 72 2d 78 72 2d 78 20 31 20 67 75 65 |-rwxr-xr-x 1 gue|
00000010 73 74 20 30 20 32 30 32 32 2d 31 31 2d 32 32 20 |st 0 2022-11-22 |
00000020 30 36 3a 34 32 20 1b 5b 30 6d 1b 5b 30 31 3b 33 |06:42 .[0m.[01;3|
00000030 32 6d 74 65 73 74 2e 73 68 1b 5b 30 6d 0a 2d 72 |2mtest.sh.[0m.-r|
00000040 77 2d 72 2d 2d 72 2d 2d 20 31 20 67 75 65 73 74 |w-r--r-- 1 guest|
00000050 20 30 20 32 30 32 32 2d 31 31 2d 32 32 20 30 36 | 0 2022-11-22 06|
00000060 3a 34 32 20 1b 5b 30 30 6d 74 65 73 74 2e 74 78 |:42 .[00mtest.tx|
00000070 74 1b 5b 30 6d 0a 2d 72 77 2d 72 2d 2d 72 2d 2d |t.[0m.-rw-r--r--|
00000080 20 31 20 67 75 65 73 74 20 30 20 32 30 32 32 2d | 1 guest 0 2022-|
00000090 31 31 2d 32 32 20 30 37 3a 30 35 20 1b 5b 30 30 |11-22 07:05 .[00|
000000a0 6d 74 6d 70 1b 5b 30 6d 0a |mtmp.[0m.|
000000a9
Thanks for the find on unix.stackexchange.com
I looked into
https://unix.stackexchange.com/question ... removing-n … and most probably I will post my own question about the very topic on unix.stackexchange.com in the future.
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 22 Nov 2022, 06:52
by ncmprhnsbl
Rava wrote: ↑22 Nov 2022, 05:52
at least it seems to know when it will be directed into a file.
hmm, something like that.. from man ls:
Code: Select all
With --color=auto, ls emits color codes only when standard output is connected to a terminal.
so, at least, it can tell if stdout is or not connected to a terminal ..
Code: Select all
-b, --escape
print C-style escapes for nongraphic characters
not on porteus at the moment, using coreutils 9.1:
Code: Select all
ls -otbr ./*.sh | tail -n 25
-rwxr-xr-x 1 user 0 Nov 22 16:40 ./a.sh*
-rwxr-xr-x 1 user 0 Nov 22 16:40 ./a\ and\ b.sh*
-rwxr-xr-x 1 user 0 Nov 22 16:40 ./a\ and\ b\ and\ c.sh*
EDIT: same result with coreutils-9.0 (porteus-5.0)
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 22 Nov 2022, 08:46
by itrukrakso
Hi,
I had to deal with the variable IFS some time ago.
In my archive I found this link.
https://mywiki.wooledge.org/BashPitfalls
The first paragraph may be helpful.
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 22 Nov 2022, 11:24
by Rava
Thanks ncmprhnsbl and itrukrakso
will look into all in detail and report back.
Added in 8 hours 58 minutes 53 seconds:
itrukrakso, I looked into
https://mywiki.wooledge.org/BashPitfalls and will see what I can make out of it.
But this
Code: Select all
ls -otbr ./*.sh | cut -c 19- |tail -n 25
for now also does the trick.
The solutions in
https://mywiki.wooledge.org/BashPitfalls sure would be more robust and overall less error prone, but it needs time to work them out, and time is rare for the next several weeks at least, so the tip by ncmprhnsbl will be what I go with.
Of course many one liners or bash scripts would fail when they encounter a file name that contains more than one line (=using the newline character) but most people do not have such file names.
Added in 1 hour 2 minutes 40 seconds:
And just as I seem to be content with the ls approach I so happen to stumble upon a file name that shows why the ls approach is flawed - a filename that has "&" in it.
Code: Select all
guest@porteus:/tmp/test$ rm *
guest@porteus:/tmp/test$ touch "a and b.sh" "c & d.sh"
guest@porteus:/tmp/test$ ls -l
total 0
-rw-r--r-- 1 guest users 0 2022-11-22 22:16 a\ and\ b.sh
-rw-r--r-- 1 guest users 0 2022-11-22 22:16 c\ &\ d.sh
guest@porteus:/tmp/test$ ls -otbr ./*.sh | cut -c 19- |tail -n 25
0 2022-11-22 22:16 ./c\ &\ d.sh
0 2022-11-22 22:16 ./a\ and\ b.sh
guest@porteus:/tmp/test$ ./c\ &\ d.sh
[1] 21606
bash: ./c : No such file or directory
bash: d.sh: command not found
[1]+ Exit 127 ./c\
Of course the correct way of quoting that file is not
but
:
Code: Select all
guest@porteus:/tmp/test$ echo 'echo "$0"-test' >./c\ \&\ d.sh
guest@porteus:/tmp/test$ chmod a+x c\ \&\ d.sh
guest@porteus:/tmp/test$ ./c\ \&\ d.sh
./c & d.sh-test
guest@porteus:/tmp/test$
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 22 Nov 2022, 22:37
by ncmprhnsbl
wouldn't you be better off using 'quoting' rather than \escapes?
ie.
Code: Select all
ls -otr --quoting-style=shell
-rwxr-xr-x 1 guest 0 Nov 23 08:09 'c & d.sh'
-rwxr-xr-x 1 guest 0 Nov 23 08:09 'a and b.sh'
./'c & d.sh'
./c & d.sh-test
interestingly, in arch, ls defaults to outputting quoted split named entries, where in slackware, it doesn't..
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 23 Nov 2022, 00:16
by Rava
Code: Select all
guest@porteus:/tmp/test$ ls -otr --quoting-style=shell ./*sh
-rw-r--r-- 1 guest 0 2022-11-22 22:16 './a and b.sh'
-rwxr-xr-x 1 guest 15 2022-11-22 22:23 './c & d.sh'
guest@porteus:/tmp/test$ './c & d.sh'
./c & d.sh-test
Indeed that works.
And I use "./" as starting point of my search to have that included into the result because I need it or else would get just this:
Code: Select all
guest@porteus:/tmp/test$ 'c & d.sh'
bash: c & d.sh: command not found
because some random PATH where some video or sound files containing folders lie that I processed via my make-ffplay-script would hardly ever be in my $PATH for the shell.
ncmprhnsbl wrote: ↑22 Nov 2022, 22:37
interestingly, in arch, ls defaults to outputting quoted split named entries, where in slackware, it doesn't..
Can you please post arch's $LS_OPTIONS?
And its $QUOTING_STYLE when it exists?
Porteus' $LS_OPTIONS are:
Code: Select all
guest@porteus:/tmp/test$ echo $LS_OPTIONS
-F -b -T 0 --color=auto
man ls says
Code: Select all
--quoting-style=WORD
use quoting style WORD for entry names: literal, locale, shell,
shell-always, shell-escape, shell-escape-always, c, escape
(overrides QUOTING_STYLE environment variable)
but at least in Port there is no $QUOTING_STYLE:
Code: Select all
guest@porteus:/tmp/test$ echo $QUOTING_STYLE
guest@porteus:/tmp/test$
Added in 8 minutes 51 seconds:
At least in Slackware (Porteus) having QUOTING_STYLE as extra environment variable doesn't work:
Code: Select all
guest@porteus:/tmp/test$ QUOTING_STYLE=shell
guest@porteus:/tmp/test$ ls -otr ./*sh
-rw-r--r-- 1 guest 0 2022-11-22 22:16 ./a\ and\ b.sh
-rwxr-xr-x 1 guest 15 2022-11-22 22:23 ./c\ &\ d.sh
guest@porteus:/tmp/test$ echo $QUOTING_STYLE
shell
guest@porteus:/tmp/test$ unset QUOTING_STYLE
but adding QUOTING_STYLE=shell to the LS_OPTIONS environment variable works:
Code: Select all
guest@porteus:/tmp/test$ LS_OPTIONS='-F -b -T 0 --color=auto QUOTING_STYLE=shell'
guest@porteus:/tmp/test$ ls
a\ and\ b.sh c\ &\ d.sh
guest@porteus:/tmp/test$ ls -otr ./*sh
-rw-r--r-- 1 guest 0 2022-11-22 22:16 ./a\ and\ b.sh
-rwxr-xr-x 1 guest 15 2022-11-22 22:23 ./c\ &\ d.sh
guest@porteus:/tmp/test$ echo $LS_OPTIONS
-F -b -T 0 --color=auto QUOTING_STYLE=shell
guest@porteus:/tmp/test$
I presume one would have to export the new LS_OPTIONS globally (e.c. via rc.local at startup) to have it globally available.
Still when putting it into a script and therefore relying on its existence is risky, then I would prefer using your manual approach of
Code: Select all
/bin/ls -otr --quoting-style=shell "$@"
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 23 Nov 2022, 01:01
by Rava
For now I changed my global "lcrt" function from this
Code: Select all
lcrt ()
{
/bin/ls -otr --time-style=long-iso "$@" | tail -n 26 | cut -c 20-$(expr 19 + $(tput cols))
}
to this
Code: Select all
lcrt ()
{
/bin/ls -otr --time-style=long-iso --quoting-style=shell "$@" | tail -n 26 | cut -c 20-$(expr 19 + $(tput cols))
}
Of course my /usr/local/bin/aliasset uses a more compact syntax:
Code: Select all
function lcrt () { /bin/ls -otr --time-style=long-iso --quoting-style=shell "$@" |tail -n 26|cut -c 20-$(expr 19 + $(tput cols)); }
Added in 5 minutes 32 seconds:
lcrt stands for
ls
cut
reverse (sorted by date)
tail
and there also exists a lcr (=sans the tail) and an lc (=sans tail and sans reverse (sorted by date))
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 23 Nov 2022, 02:26
by ncmprhnsbl
so.. looking at
http://ftp.slackware.com/pub/slackware/ ... coreutils/
there's a patch: no_ls_quoting.patch.gz
that looks like:
Code: Select all
--- ./src/ls.c.orig 2021-09-24 06:31:05.000000000 -0500
+++ ./src/ls.c 2021-09-24 19:47:32.230001419 -0500
@@ -2342,7 +2342,7 @@
qs = getenv_quoting_style ();
if (qs < 0)
qs = (ls_mode == LS_LS
- ? (stdout_isatty () ? shell_escape_quoting_style : -1)
+ ? (stdout_isatty () ? escape_quoting_style : -1)
: escape_quoting_style);
if (0 <= qs)
set_quoting_style (NULL, qs);
...in the slackbuild the explanation for which is:
Code: Select all
# Revert change to ls quoting style introduced in coreutils-8.25:
presumably pat has a good reason for it... arch doesn't do this.
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 23 Nov 2022, 04:50
by Rava
I don't see the reason for doing so
, but a least using
Code: Select all
-F -b -T 0 --color=auto QUOTING_STYLE=shell
as your $LS_OPTIONS also works in Slackware.
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 23 Nov 2022, 21:07
by Ed_P
Code: Select all
guest@porteus:~$ ls -F -b -T 0 --color=auto QUOTING_STYLE=shell
/bin/ls: cannot access 'QUOTING_STYLE=shell': No such file or directory
guest@porteus:~$
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 24 Nov 2022, 01:54
by Rava
Ed_P wrote: ↑23 Nov 2022, 21:07
Code: Select all
guest@porteus:~$ ls -F -b -T 0 --color=auto QUOTING_STYLE=shell
/bin/ls: cannot access 'QUOTING_STYLE=shell': No such file or directory
guest@porteus:~$
Do not confuse $LS_OPTIONS and calling ls directly.
Look at my function as quoted above
Rava wrote: ↑23 Nov 2022, 01:07
lcrt ()
{
/bin/ls -otr --time-style=long-iso --quoting-style=shell "$@" | tail -n 26 | cut -c 20-$(expr 19 + $(tput cols))
}
Try
Code: Select all
ls -F -b -T 0 --color=auto --quoting-style=shell
instead.
And know that your approach mixes two things (for whatever reason you choose doing so
, in my book it makes not that much sense, but that's only my opinion on that - you might have a good reason why you mix up the options like so), therefore having a quite confusing result of e.g.
'.moonchild productions/'/ - the inner part of '.moonchild productions/' is because of --quoting-style=shell - the extra / at the end is because of one of your options. I let you and man ls figure out which one that is and therefore which one to keep away when adding "--quoting-style=shell".
Code: Select all
guest@porteus:~$ ls -F -b -T 0 --color=auto --quoting-style=shell .*/ -d
..// .cache// .local// .nv//
.// .config// '.moonchild productions/'/
.FBReader// .dbus// .mozilla//
'.binary outcast/'/ .netsurf//
and also using [
quote
] so that you see the partially colours in its output:
guest@porteus:~$ ls -F -b -T 0 --color=auto --quoting-style=shell .*/ -d
..// .cache// .local// .nv//
.// .config// '.moonchild productions/'/
.FBReader// .dbus// .mozilla//
'.binary outcast/'/ .netsurf//
How to control ls's listing (handling of whitespaces) when piping its output to another program?
Posted: 04 Dec 2022, 07:02
by Rava
I realized going the lazy and somewhat also insecure way of using ls has its downfalls.
Often my above described function works lcrt ()
Code: Select all
{
/bin/ls -otr --time-style=long-iso --quoting-style=shell "$@" | tail -n 26 | cut -c 20-$(expr 19 + $(tput cols))
}
but only when files are owned by guest or by root and guest.
When a user has files whose name is longer than "guest" it messes up the listing somewhat.
When only files owned by root exist my function cuts of the 1st character of the file size.
Look for yourself, both guest and root use the same function since it is defined in my global file /usr/local/bin/aliasset and activated for root or guest via this hack in their respective ~/.bashrc:
Code: Select all
if [ -f /usr/local/bin/aliasset ]; then
. /usr/local/bin/aliasset
fi
Of course /usr/local/bin/aliasset needs to be bug free as including it so in every started terminal or virtual terminal session can cause issues when it has some bugs.
When a selection of files contains both files by user root and user guest my function works as it should.
But when the selection of files
only contains files by user root it cuts off the 1st character of the size.
Creating mixed owned dummy files:
Code: Select all
root@porteus:/tmp# mkdir temp
root@porteus:/tmp# cd temp/
root@porteus:/tmp/temp# touch test1.txt test2.txt test3.txt Test1.txt Test2.txt Test3.txt
root@porteus:/tmp/temp# chown guest.users Test*
root@porteus:/tmp/temp# ls -o
total 0
-rw-r--r-- 1 guest 0 2022-12-04 07:57 Test1.txt
-rw-r--r-- 1 guest 0 2022-12-04 07:57 Test2.txt
-rw-r--r-- 1 guest 0 2022-12-04 07:57 Test3.txt
-rw-r--r-- 1 root 0 2022-12-04 07:57 test1.txt
-rw-r--r-- 1 root 0 2022-12-04 07:57 test2.txt
-rw-r--r-- 1 root 0 2022-12-04 07:57 test3.txt
Checking
all works for lcrt
Code: Select all
root@porteus:/tmp/temp# lcrt
0 2022-12-04 07:57 test2.txt
0 2022-12-04 07:57 test1.txt
0 2022-12-04 07:57 test3.txt
0 2022-12-04 07:57 Test3.txt
0 2022-12-04 07:57 Test2.txt
0 2022-12-04 07:57 Test1.txt
Checking only the files owned by root fails:
Code: Select all
root@porteus:/tmp/temp# lcrt test*
2022-12-04 07:57 test2.txt
2022-12-04 07:57 test1.txt
2022-12-04 07:57 test3.txt