USB-stick keeps stalling my whole system!

Post here if you are a new Porteus member and you're looking for some help.
JustME
White ninja
White ninja
Posts: 8
Joined: 01 Sep 2014, 12:35
Distribution: Porteus
Location: Europe

USB-stick keeps stalling my whole system!

Post#1 by JustME » 01 Sep 2014, 12:56

Hi boys, girls and others,

imagine this: I have a new 64GB USB-stick with Porteus installed on it.
But when I use it, Porteus more or less randomly just stalls while the
read/write-LED on the stick is flashing - for somewhat 30-60 seconds.

I've tried different filesystems, different desktops (KDE/LXDX/RazorQT),
using computers with more RAM, deactivating swap completely, removing the
wireless-drivers, setting the journaling to writeback+no barriers, and over
and over again trying to dump what the kernel is doing to the logs... AND
deactivating the logs, even deleting them... but to no use.

The only thing logged when I do a "echo l > /proc/sysrq-trigger" is this:

Aug 29 15:34:26 porteus kernel: [ 3062.646196] sending NMI to all CPUs:
Aug 29 15:34:27 porteus kernel: [ 3063.318139] SysRq : Show backtrace of all active CPUs


Many lines of it, and as far as I understand it, it's just saying I triggered
a backtrace of what the kernel was doing to be logged. And I already knew
that damn it! I didn't want to know what I was doing, I wanted to know what
program or kernel-module was occupying the usb-stick.

No matter what I do, over and over again Porteus keeps almost freezing up
for up to over a minute before it continues as if nothing happened.
And during this stall, iotop is not reacting, I can move the mouse and click
on icons, but the commands will only be executed once the USB-traffic stops.

And, the network-traffic usually also stalls more or less completely.
Mostly completely, but sometimes a few KBytes will pass through downstream.
Which is really a nerving since I'm often using Porteus in internet-cafes
where it's especially important to have as much stable network-traffic and
as little problems as possible.

These "stalls" occur much more often when I'm doing some hardcore downloading.
Which first made me suspect the filesystem, but after much tweaking and
benchmarking - I've found nothing wrong with it.

The lowest sustained writerate is 5,7MBytes/sec. The fastest 12,5MBytes/sec.
Read: constantly around 30MBytes/sec.
(it's a USB 3.0 -memory, but I currently only have access to USB 2.0-ports)

So... while I don't know what the read-write-activity is, I can still
estimate that the data-volume it has to do with is roundabout 300-600MBytes.
Which leaves me even more clueless since I just yesterday observed this
problem occurring even if I wasn't not doing anything at all. (!)

And I do seem to remember having more or less the same problem while using
Slax. So, could this have something to do with me stubbornly trying to run
Linux on a USB-stick? I don't see the problem with it. Because 90% of the
time Porteus etc works just fine. More than fine, I simply love it. It's just
these seconds of intense USB-activity that really sucks the fun out of it.

Does anyone know if Linux does some kind of "scrubbing" of flash-units?
Because with windows this never happens. But I have on the other hand never
tried to run windows off a usb-stick.

And yes, I've also read about linux "dirty" data and tried setting those
intervals higher, but - you guessed it - to no use. Problem still occurs.

My currently best guess is that the filesystem backlogs something
which it more or less in panic dumps onto the stick once and a while.
Never the less, I feel I've gotten stuck right now.

Therefore: anyone of you got any bright throughts on the matter?

JustME

User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5666
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

Re: USB-stick keeps stalling my whole system!

Post#2 by fanthom » 01 Sep 2014, 13:56

anybody can confirm this?

@JustME
never seen such behavior. are you saving changes? if yes then maybe system dumps caches on the stick but it's definitely too much of data. try 'mopt=user,noatime,nodiratime,suid,dev,exec,sync' cheatcode (mind sync flag instread of async) which should save everything on the stick in real time. this way you should be able to avoid stalls.
or you could go with 'changes=EXIT:' eventually 'changes-ro' cheatcodes which should resolve problem of unnecessary writes completely.
Please add [Solved] to your thread title if the solution was found.

User avatar
francois
Contributor
Contributor
Posts: 6434
Joined: 28 Dec 2010, 14:25
Distribution: xfce plank porteus nemesis
Location: Le printemps, le printemps, le printemps... ... l'hiver s'essoufle.

Re: USB-stick keeps stalling my whole system!

Post#3 by francois » 02 Sep 2014, 02:04

Working with 8 gig and 16 gig sticks (sony, sandisk, verbatim), I never had that type of problems.

However, some sticks would reject linux systems. This is the case of dt101 from lg (8 gig and 16 gig).
Prendre son temps, profiter de celui qui passe.

JustME
White ninja
White ninja
Posts: 8
Joined: 01 Sep 2014, 12:35
Distribution: Porteus
Location: Europe

Re: USB-stick keeps stalling my whole system!

Post#4 by JustME » 03 Sep 2014, 11:00

Alrighty, I might have a solution for this... (remains to be seen, but I'm hopeful)

First of all, changes-ro makes the system load with lightspeed, but it does NOT
make the stalls go away. I'm right now experiencing stalls while the changes-ro -switch is active.

But while googling for "linux usb problem", I stumbled onto this text in german:
http://de.comp.os.unix.linux.misc.narki ... t4-problem

- It's about a guy who experiences read/write-activity on his 1TB usb-harddrive right after
he's mkfs.ext4 it. And another reader mentions the ext4 "Lazy"-feature.

Which makes me remember that I have never never never formatted a usb-stick
while deactivating the "lazy"-feature. Which absolutely explains why I've experienced
read/write-activities on a completely empty device.

So, later today I'm gonna take a complete backup of the contents of the usb-stick,
mkfs.ext4 with the lazy-feature OFF, restore the data from my backup, and see how it works.

Because that is the only thing that makes any sense at all. Well, either that or this just weeks old
usb-stick has so many bad cells on it that it's firmware almost constantly is doing some kind
of "scrubbing" of it.

But to my mind, the more realistic solution is that when I'm doing "heavy downloading",
the ext4-filesystem over and over again needs to prepare the stick with new... eh...
inodes or whatever... it does in any case keep the computer quite busy since it's a usb-stick.
- which over and over again brings it to a stall.

Hopefully that's it. I'll write more as soon as I've tried it out.

cttan
Shogun
Shogun
Posts: 333
Joined: 26 Jan 2011, 16:15
Distribution: Porteus 3.2 and 4.0 64bit KDE
Location: Malaysia

Re: USB-stick keeps stalling my whole system!

Post#5 by cttan » 03 Sep 2014, 15:29

Try this in your boot line:-
APPEND initrd=initrd.xz copy2ram splash=silent vga=791 quiet noauto login=root ramsize=75% zram=25% volume=5%

It is modified from copy2ram. Basically it load everything in the RAM and do not mount the USB automatically(no auto).
Set a side some RAM for tmpfs(ramsize=75%) and make a swap using zram(zram=25%).
I have 4GB RAM laptop running Porteus 3.0.1 KDE x84_64.

Code: Select all

root@a10b23c45d67:~# df
Filesystem     1K-blocks     Used Available Use% Mounted on
aufs             2804288    81096   2723192   3% /
devtmpfs         1868764       12   1868752   1% /dev
/mnt/live/run    1869524   789552   1079972  43% /run
/dev/sdb1       30650436 28290520    779884  98% /mnt/sdb1
/dev/sda1      488384508 49552860 438831648  11% /mnt/sda1
You have new mail in /var/mail/root
root@a10b23c45d67:~# free
             total       used       free     shared    buffers     cached
Mem:       3739048    3518964     220084          0      98072    1288904
-/+ buffers/cache:    2131988    1607060
Swap:       934760       2208     932552
root@a10b23c45d67:~# swapon -s
Filename                                Type            Size    Used    Priority
/dev/zram0                              partition       934760  2208    100
root@a10b23c45d67:~# 
I have been running Porteus using copy2ram mode without changes ever since the beginning.
No problem on the USB drive from 2GB, 4GB, 8GB, 16GB up to 32GB now.

Hope this help.

JustME
White ninja
White ninja
Posts: 8
Joined: 01 Sep 2014, 12:35
Distribution: Porteus
Location: Europe

Re: USB-stick keeps stalling my whole system!

Post#6 by JustME » 04 Sep 2014, 14:21

Update: disabling the lazy-Ext4-feature didn't help at all.
It might just be me, but "changes-ro" makes the system stall
a bit less, but it doesn't eliminate the problem.

Copy2ram doesn't do much more than make the startup-sequence
a lot slower, thereafter I'm neither experiencing less "stalls" nor any
performance-improvement. :cry:

But big thanks for mentioning the "noauto"-switch, I have found it
irritating to have to remount several devices from /mnt/something
to where I want them (under /media/)

But, even with this:
APPEND initrd=initrd.xz copy2ram changes=/porteus changes-ro splash=silent fsck vga=791 quiet noauto login=root ramsize=75% zram=25% volume=100% guiexec=xterm mopt=user,noatime,nodiratime,suid,dev,exec,sync

- the problem STILL occurs!

And it's slowly but surely driving me mad!

And if I'm the only one who's experiencing this kind of very irritating thing,
then I don't see any other possibility than the usb-stick itself causing this.

It's a Lexar S33 64GB. A cheap one, but except for these "stalls" I've not
had any real issues with it at all. Under windows it's not making any fuzz
whatsoever.

I have one little thing left which I wish to try before clearly announcing this
usb-stick as "not Linux-compatible"... and that is ext2 and ext3
without journaling. I don't see how the journal could be the problem
since I have tried to activate the writeback-mode - to no use.
But what the heck, it's worth a try.

Well, that and converting my "changes"-directory into a module.
Think I'll try both alternatives.

BTW - are there any usb-stick-brands out there that are guaranteed to
work with Linux to 100%? Linux-certified so to speak?

User avatar
francois
Contributor
Contributor
Posts: 6434
Joined: 28 Dec 2010, 14:25
Distribution: xfce plank porteus nemesis
Location: Le printemps, le printemps, le printemps... ... l'hiver s'essoufle.

Re: USB-stick keeps stalling my whole system!

Post#7 by francois » 04 Sep 2014, 14:49

sony, sandisk, verbatim), I never had that type of problems.

Some of the keys have the little pingouin logo or linux compatible mention. However, non-linux microsoft approved usb-keys are becoming more the fad.

Searching for 64 gig usb keys linux compatible, kingston is said to be, see specs:
http://www.kingston.com/us/usb/personal ... ss#dt101g2
Prendre son temps, profiter de celui qui passe.

donald
Full of knowledge
Full of knowledge
Posts: 2064
Joined: 17 Jun 2013, 13:17
Distribution: Porteus 3.2.2 XFCE 32bit
Location: Germany

Re: USB-stick keeps stalling my whole system!

Post#8 by donald » 04 Sep 2014, 15:40

@JustME
You can test your usb-stick (on a windows box) with this Tool:
http://www.heise.de/download/h2testw.html

I never had any problems with sandisk or samsung usb-flash.From my personal
experience I can not recommend the kingston/verbatim brand (sorry francois)
because they were often not bootable.

beny
Full of knowledge
Full of knowledge
Posts: 2086
Joined: 02 Jan 2011, 11:33
Location: italy

Re: USB-stick keeps stalling my whole system!

Post#9 by beny » 04 Sep 2014, 16:22

have you tried this: tune2fs -O ^has_journal /dev/sdx this remove the default journaling in ext4 i don't know if porteus have done this when install itself
also you have enough ram on board? the usb key have a good score in usb3, 64GB in purple (up to 100MB/s read, 55MB/s write)

JustME
White ninja
White ninja
Posts: 8
Joined: 01 Sep 2014, 12:35
Distribution: Porteus
Location: Europe

Re: USB-stick keeps stalling my whole system!

Post#10 by JustME » 05 Sep 2014, 12:50

Result from h2testw:

Test finished without errors.
You can now delete the test files *.h2w or verify them again.
Writing speed: 20.8 MByte/s
Reading speed: 30.5 MByte/s
H2testw v1.4


Unfortunately I formatted the stick with exFAT to get Windows to
communicate with it in an meaningful way. That was most likely
the reason why it this morning refused to create either a ext2 or
a ext3 filesystem. Mkfs went through just fine, but when I then
tried to restore the data onto the stick - it suddenly just stopped.
Had to low-level format the stick with the HDD low level format
tool (http://www.hddguru.com) in order to get it to respond
properly again. Or, at least I thought so, because back in Linux it
behaved in exactly the same way as before: refusing to create
either ext4 or ext3.

I can only assume it had something to do with the default
clustersize for exFAT (128kB). I don't know why that should
matter, or what FS the stick has on it, but mkfs just doesn't
go through with the initialization. AND the lowlevelformat-tool
kinda stopped at some sectors also, but at least it continued.

I'm feeling quite pissed right now. Because I still don't know
what's going on. But ok, I'll do another complete lowlevelformat,
put fat32 on the stick (smallest clustersize possible), fill it with
data, and THEN try to both format with ext4 + remove the Journal,
IF the mkfs.ext4-process completes, or format with ext3...

Or maybe I should try dd-ing the whole device with /dev/zero first?

So very annoying, because this stick DID work. It booted Porteus
just fine. It was fast, and it was stable if you just disregard the
stalling-problem.

Which by the way kind of seemed to get worse the more data I put
on the stick. Could of course also have to do with the amount of
network-traffic, but since a reboot didn't make the problem go
away, I kind of want to disregard everything but either the hardware
or the filesystem.

Oh well, I'll give it another shot later today.

beny
Full of knowledge
Full of knowledge
Posts: 2086
Joined: 02 Jan 2011, 11:33
Location: italy

Re: USB-stick keeps stalling my whole system!

Post#11 by beny » 05 Sep 2014, 15:15

if you have a windows computer there is a good software to format again your usb key that i have used to recover usb fat 32 filesystem : HP usb disk storage format ,run very well or have to use gparted if have only linux,but fdisk -l in console how see the key.

donald
Full of knowledge
Full of knowledge
Posts: 2064
Joined: 17 Jun 2013, 13:17
Distribution: Porteus 3.2.2 XFCE 32bit
Location: Germany

Re: USB-stick keeps stalling my whole system!

Post#12 by donald » 05 Sep 2014, 17:17

Well,I don't know what this "low level format" tool did.Hopefully it is just the modern use of the term Low Level Format,which means zero-filling.
http://www.lowlevelformat.info/history.aspx
However,what I would do is:
dd if=/dev/zero of=/dev/sdX bs=100M count=10
((replace sdX with your drive (fdisk -l))
then open gparted, create a new partition table (ms-dos) and format the drive.

beny
Full of knowledge
Full of knowledge
Posts: 2086
Joined: 02 Jan 2011, 11:33
Location: italy

Re: USB-stick keeps stalling my whole system!

Post#13 by beny » 06 Sep 2014, 16:20

exfat in linux you have to build a package named fuse-exfat if you want to mount this file system into linux and gparted see this file system,so the hp format tool isn't necessary to format usb,not fat32

JustME
White ninja
White ninja
Posts: 8
Joined: 01 Sep 2014, 12:35
Distribution: Porteus
Location: Europe

Re: USB-stick keeps stalling my whole system!

Post#14 by JustME » 07 Sep 2014, 09:20

My god, we are really loosing focus here aren't we?

I do NOT want exFAT on my usb-stick! :bad:
What on earth did you get that from? I only formatted it
with it to be able to use the h2test-tool under WINDOWS.

Anyway, trying to low-level-format the stick under windows
took ages (after about an hour the progress had still not
reached 0.1%), so I aborted that thing and did the
dd if=/dev/zero -thing instead. Which worked like a
charm. Finished after around two hours and after that I
had no problem what so ever making mkfs.ext4 create a
ext4-fs without a journal.

Which is what I'm running at the moment. And I'm still
having these stalls.

But, there is one little thing that's different today...
- I'm now absolutely sure I know what the problem is.

And the winner iiiiiis:
http://en.wikipedia.org/wiki/Wear_leveling

Because it's the only thing that makes sense at all.
Think about it. What makes a usb-stick stall while not
telling the linux-kernel about what it's doing?
And of course my network-traffic stalls - the storage-unit
to which the network-data will be written doesn't respond,
so the system has to wait for the storage-device to finish
whatever it's doing before it can continue.

So, either Lexar has somehow optimized it's
flash-controlling-algorithm for windows and osx-traffic only,
and thereby effectively somehow created a kind of "bug"
which only gets visible on Linux-systems...
- or this problem occurs under windows also, but since
windows implements other ways of caching data, the
problem doesn't "stall" the system like it does on Porteus.

In any case, I'm on the lookout for a new usb-stick.
Because even if this one works beautifully under the right
conditions, it's still not living up to my expectations.
- Which among other things includes that it should be enough
compatible to be used under any operating-system which
implements industry-standardized storage-protocols, damn it!

But thank ya all for your ideas and support.

And, I know I might be wrong about the wear-leveling-thing,
so if anyone has a better suggestion for what could be the
cause behind these "stalls", feel free to tell me about it.

beny
Full of knowledge
Full of knowledge
Posts: 2086
Joined: 02 Jan 2011, 11:33
Location: italy

Re: USB-stick keeps stalling my whole system!

Post#15 by beny » 07 Sep 2014, 13:08

well this filesystem exfat if you check it via fdisk -l is unknow so if you want that porteus check or make or erase or change the file system, you have to install fuse-exfat and exfat -util, after this you can do all you want also make ext4 format,i have a samsung 64 giga microsd formatted in exfat in this evening i change file system,ext4 and report here i have luck with.

Post Reply