[SOLVED] data lost when copying

[imam]

Porteus is extraordinary fast in terms of copying the files, but I'd like to report some errors when copying dukumen. Some of the events that I often experienced:

When moving files from local hdd (NTFS) to flashdisk (I use Kingston with FAT32 file system). File a copy of the results is often lost when opened on another computer, so I had to repeat copy for 2 or 3 times.

Sometimes files are not always lost, sometimes for mp3 file data will be there, but can not be played (it would look like with a file size of 0KB). And the solution I use now is to create a folder within a folder on the flashdisk and copy files to it (often can not).

The same thing happens when copying files mp3 data into MMC card and lost (again) when they want to read in the other media (I use a Nokia mobile phone).

In this case I've been trying to do the procedure should be. I copy the file, wait for it to finish and do the 'Safely Remove' media.

I had this problem since v1.1 until now I use v1.2. Is this a bug or something else I need to know?

Thank you for your hard work .. and expect a lot of improvements for the upcoming Porteus v2

[fanthom]

hello imam,

please update to latest kernel as per latest announcement:
http://porteus.org/forum/viewtopic.php?f=35&t=1472

Code: Select all

I copy the file, wait for it to finish and do the 'Safely Remove' media.

yes - this is the correct procedure. data should never be lost when you do it like that.
simple 'umount' command from console should be enough as all date would be synced before ejecting the media.

the only scenario which could cause data lost is when you copy large file on a usb stick and then unplug it manually straight after that.

be aware that files are cached in the RAM and then copied on the stick with a little delay.

you may also try to switch default I/O scheduler from 'CFQ' to 'NOOP' which should write data immediately.
please add following command to your rc.local file:

Code: Select all

echo "noop" > /sys/block/{DEVICE-NAME}/queue/scheduler

or boot with "elevator=noop" cheatcode (will affect all drives) and check if that helps.

Cheers

[imam]

Thank you @ fanthom ..

Code: Select all

imam@porteus:~$ uname -a
Linux porteus 3.4.9-porteus #1 SMP PREEMPT Thu Aug 16 21:33:09 UTC 2012 x86_64 AMD E-450 APU with Radeon(tm) HD Graphics AuthenticAMD GNU/Linux

With the latest update does not seem to work. I try again with my Creative MuVo audio player I get the following result:

My first test move all mp3 files from the MuVo into the local hard drive, then copy it back into the MuVo. The process of copying 110 mp3 files (491 MB total) running very fast, I think less than 1 minute (compared to about 3 minutes or more if I use win7). When finished I try to use the preview to make sure that the dolphin mp3 file can be played, and finally 'Safely Remove' media. And the result MuVo displays a 'no media' and when I return plug into Porteus, and no one files stored in Muvo.

The second test, I did the same procedure on the MuVo and I get the results but only 21 files mp3 (89.8 MB) and I can confirm that all files can be played normally. But where are the rest? : D

I guess Porteus never had a problem when working with large files. But often experienced in those files smaller than text files.

To boot with "elevator = noop" cheatcode I will try tomorrow with a few other flashdisk. I will definitely report back

[fanthom]

The process of copying 110 mp3 files (491 MB total) running very fast, I think less than 1 minute (compared to about 3 minutes or more if I use win7).

Win 7 is more accurate and shows real time which is needed to copy these files while KDE closes copying window after files are cached in the RAM (but still not stored on the removable media yet).

When finished I try to use the preview to make sure that the dolphin mp3 file can be played, and finally 'Safely Remove' media. And the result MuVo displays a 'no media' and when I return plug into Porteus, and no one files stored in Muvo.

this could be a bug in KDE's 'Safely Remove' feature.
if 'elevator=noop' wont help then please do another test:
a) copy 110 mp3 files as before
b) open konsole any run 'sync' command
c) unmount the media with 'Safely Remove'

please let me know about the result.

Posted after 20 minutes 12 seconds:
i have done this test myself and seems that KDE4 is incorrectly showing that pendrive is unmounted while the data are still being copied.
there is a notification window at the end which says that 'it's safe to remove device now' but some users could have disable notifications completely.

it's a bit confusing indeed.
hopefully will be fixed upstream in the next KDE4 release.

possible solutions:
a) copy the data -> run 'remove safely' -> wait till kde4 notification window appears
b) copy the data -> run 'sync' command in terminal -> run 'remove safely'

let me know if the tips above resolved your problem.

[imam]

OK, this time I have 4 items to test:

1. MP3 player Creative MuVo N200 512MB (default FAT)
2. V-Gen 1GB MMC mobile (default FAT)
3. DT101G2 Flashdisk Kingston 8GB (default FAT32)
4. V-Gen 16GB microSDHC Class10 (default FAT32)

Previously I have added a cheatcode elevator = noop

I'll plug them into the USB to be emptied (just delete the contents, without reformatting). Then I copy to each of the 110 files totaling 491.2 MB mp3.

After all filled in, from each medium I take at least 3 to preview files before starting the test.

The first test, after the 'safely remove' my media unplug each medium. Then I reboot the computer to get into windows. When ready, I plug back all my media to check the contents. And the result: a device to 3 and 4 appear normal while trying to play in WMP. But the device to 1 (Creative MuVo) give different results. All files are copied, but some can not play and some jump into the middle of the song at the start of play. While the file size has not changed. While the device to 2 (V-Gen MMC) plug it into my Nokia phone to see it and it disappeared, none of the files are stored. How did this happen?

The second test, I focus on the two devices. I repeat the process and give them additional time approximately 10 minutes before finally 'safely remove' media. And it works well.

So, my conclusion is the addition cheatcode while elevator = noop helpful enough, and need to take a few moments to make sure all the files copied. But I really do not understand how this works cheatcode?

[fanthom]

But I really do not understand how this works cheatcode?

this cheatcode switches default I/O scheduler from CFQ to NOOP which is more appropriate for flash drives and supposed to result in immediate writes to the media.

anyway - the main point is to either:

Code: Select all

a) copy the data -> run 'remove safely' -> wait till kde4 notification window appears
b) copy the data -> run 'sync' command in terminal -> run 'remove safely'

as per my updated post above.

Cheers

[imam]

Results below may be different for some computers, but may represent a comparison of the time each process. And this time I use USB3 ports to test.

a) All data is OK!
Creative MuVo : copy the data (about 4s) -> run 'safely remove' -> wait till Appears kde4 notification window (about 2 minutes)
V-Gen MMC Mobile : copy the data (about 4s) -> run 'safely remove' -> wait till Appears kde4 notification window (about 2 minutes)
Kingston DT101G2 : copy the data (about 4s) -> run 'safely remove' -> wait till Appears kde4 notification window (about 1 minutes)
V-Gen SDHC : copy the data (about 5s) -> run 'safely remove' -> wait till Appears kde4 notification window (about 1 minutes)


b) All data is OK!
Creative MuVo : copy the data (about 4s) -> run 'sync' (about 2 minutes) -> run 'safely remove' (about 4s)
V-Gen MMC Mobile : copy the data (about 4s) -> run 'sync' (less than 3 minutes) -> run 'safely remove' (less than 3s)
Kingston DT101G2 : copy the data (about 4s) -> run 'sync' (less than 2 minutes) -> run 'safely remove' (about 3s)
V-Gen SDHC : copy the data (about 4s) -> run 'sync' (less than 2 minutes) -> run 'safely remove' (about 3s)

- For Creative MuVo & V-Gen 1GB MMC Mobile when I plug in again, Dolphin failed to show the contents, until I press the 'F5'.
- The command 'sync' could be combined on each key 'paste' and 'ctrl+V' button?

[fanthom]

Dolphin failed to show the contents, until I press the 'F5'

yes - this is an old known bug in KDE

The command 'sync' could be combined on each key 'paste' and 'ctrl+V' button?

that requires hacking on qt sources and i dont have a skills for that.

KDE dev's assume that everyone waits for notification window.
as i wrote above - that window may never appear in case when user disabled notifications completely.
definitely a KDE bug.
will check if something got changed in that matter in kde-4.9.x series.

[imam]

thank fhanthom... I'll look forward too