Mounting USB Storage devices with "sync" option ?

[Abyss Ether]

I created a simple PoC udev rule to mount USB Storage devices with the "sync option.
Available here :
https://github.com/larina3315/personal-stuff/blob/main/linux/10-usb-stora...

Currently, USB Storage devices are mounted without the "sync" option, causing
their writes to be cached.
This causes an issue, especially with GUI file managers like GNOME Files, where after a
file copy operation, it would notify the user that the operation has been completed. If
the user then tries to unmount the USB Storage device, they get notified that data is
still being written to disk and that they should not remove the USB Storage device from
their PC/Laptop/device.

This can take a more severe form if the user is using a CLI/GUI utility that DOES NOT
inform the user that data is still being written, like the 'cp' CLI utility, the
user might be misled into thinking that all writes have finished and plug the USB drive
out, at which point data corruption due to unfinished writes occur.

This is why, I think functionality should be included in Fedora Linux (or atleast in
Fedora Workstation and other desktop spins) such that USB Storage devices are mounted with
the "sync" options by default.

[Zbigniew Jędrzejewski-Szmek]

On Wed, Jan 31, 2024 at 06:43:00AM -0000, Abyss Ether via devel wrote:
>  I created a simple PoC udev rule to mount USB Storage devices with
> the "sync option.
>  Available here :
> https://github.com/larina3315/personal-stuff/blob/main/linux/10-usb-stora...
>  
>  Currently, USB Storage devices are mounted without the "sync" option, causing
> their writes to be cached.
>  This causes an issue, especially with GUI file managers like GNOME Files, where after a
> file copy operation, it would notify the user that the operation has been completed. If
> the user then tries to unmount the USB Storage device, they get notified that data is
> still being written to disk and that they should not remove the USB Storage device from
> their PC/Laptop/device. 
'sync' has some strong downsides though: various operations become
painfully slow (this depends a lot on the hardware and its age, and
the history of previous writes, etc.), write operations block read
operations, and the total number of writes may be increased, leading
to more wear&tear on the hardware.

We approach this problem from a different angle: the user is supposed
to sync the filesystem before removing. Graphical environments have
an "eject" button, and for non-graphical environments, the user
just needs to do a sync manually.

>  This can take a more severe form if the user is using a CLI/GUI
>  utility that DOES NOT inform the user that data is still being
>  written, like the 'cp' CLI utility, the user might be misled into
>  thinking that all writes have finished and plug the USB drive out,
>  at which point data corruption due to unfinished writes occur. 
The same is true for "normal" writes to a harddrive. Operations are
generally async, and the user needs to do a shutdown, during which
we sync and unmount filesystems. If you "yank" the cord, this may (*)
result in lost data and file system inconsistency.

Zbyszek


(*) There are extensive mechanisms to avoid loss of data, so
    "may" does not mean it's likely.

[Larina Loriasel]

>  'sync' has some strong downsides though: various operations
> become
>  painfully slow (this depends a lot on the hardware and its age, and
>  the history of previous writes, etc.), write operations block read
>  operations, and the total number of writes may be increased, leading
>  to more wear&tear on the hardware. 
The udev rule only applies to USB Storage devices, and I think the total number of writes
increasing only happens if the data to be written is changed constantly (e.g. writes that
overlap on same region/block of a device)
Most USB Storage devices are used for storing simple, consecutive streams of data, such as
Music, Pictures and other simple files and are not usually modified much, thus, I think
enabling sync makes sense in this regard.
In my opinion, the benefits of mounting USB Storage devices with sync outweigh the
detriments.
Writes blocking reads is concerning.
 
>  We approach this problem from a different angle: the user is
> supposed
>  to sync the filesystem before removing. Graphical environments have
>  an "eject" button, and for non-graphical environments, the user
>  just needs to do a sync manually. 
I am aware of that, but it leads to a poor user experience, being notified that a copying
operation has been completed, while in reality, it has not.

>  The same is true for "normal" writes to a harddrive.
> Operations are
>  generally async, and the user needs to do a shutdown, during which
>  we sync and unmount filesystems. If you "yank" the cord, this may (*)
>  result in lost data and file system inconsistency. 
My proposal was exclusively for USB Storage devices, not internal ones, as I can
understand the benefits of async outweighing the detriments in the case of
internal/network etc storage devices.

[Leon Fauster]

Am 31.01.24 um 09:57 schrieb Larina Loriasel via devel:
> > 'sync' has some strong downsides though: various
> operations become
> > painfully slow (this depends a lot on the hardware and its age, and
> > the history of previous writes, etc.), write operations block read
> > operations, and the total number of writes may be increased, leading
> > to more wear&tear on the hardware.
>  
>  The udev rule only applies to USB Storage devices, and I think the total number of writes
> increasing only happens if the data to be written is changed constantly (e.g. writes that
> overlap on same region/block of a device)
>  Most USB Storage devices are used for storing simple, consecutive streams of data, such
> as Music, Pictures and other simple files and are not usually modified much, thus, I think
> enabling sync makes sense in this regard.
>  In my opinion, the benefits of mounting USB Storage devices with sync outweigh the
> detriments.
>  Writes blocking reads is concerning. 


Thats a to narrow view, as the USB interface is used in a ton of 
scenarios (backups etc.) where maximum data flow is an expectation.



> > We approach this problem from a different angle: the user is
> supposed
> > to sync the filesystem before removing. Graphical environments have
> > an "eject" button, and for non-graphical environments, the user
> > just needs to do a sync manually.
>  
>  I am aware of that, but it leads to a poor user experience, being notified that a copying
> operation has been completed, while in reality, it has not. 

CLI: Doesn't that trigger the umount command already ... so no need to 
think about it (as user). As soon the umount completes the device can be
unplugged.



> > The same is true for "normal" writes to a harddrive.
> Operations are
> > generally async, and the user needs to do a shutdown, during which
> > we sync and unmount filesystems. If you "yank" the cord, this may (*)
> > result in lost data and file system inconsistency.
>  
>  My proposal was exclusively for USB Storage devices, not internal ones, as I can
> understand the benefits of async outweighing the detriments in the case of
> internal/network etc storage devices. 

I never had a big issue in the GUI (long time/waits), as it clearly 
states to wait until removal is allowed (also stated via notify).
Normally (here) this is just a few seconds (single digit).

Maybe the idea can be implemented more transparently via vm.dirty_* 
sysctl per device if possible or is it a global threshold only?


--
Leon

[Justin Forbes]

On Wed, Jan 31, 2024 at 6:36 AM Leon Fauster via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  Am 31.01.24 um 09:57 schrieb Larina Loriasel via devel:
>  >> 'sync' has some strong downsides though: various operations become
>  >> painfully slow (this depends a lot on the hardware and its age, and
>  >> the history of previous writes, etc.), write operations block read
>  >> operations, and the total number of writes may be increased, leading
>  >> to more wear&tear on the hardware.
>  >
>  > The udev rule only applies to USB Storage devices, and I think the total number of
> writes increasing only happens if the data to be written is changed constantly (e.g.
> writes that overlap on same region/block of a device)
>  > Most USB Storage devices are used for storing simple, consecutive streams of data,
> such as Music, Pictures and other simple files and are not usually modified much, thus, I
> think enabling sync makes sense in this regard.
>  > In my opinion, the benefits of mounting USB Storage devices with sync outweigh the
> detriments.
>  > Writes blocking reads is concerning.
>
>
>
>  Thats a to narrow view, as the USB interface is used in a ton of
>  scenarios (backups etc.) where maximum data flow is an expectation. 
This is a huge point.  For someone writing something quickly to a USB
stick, the behavior characteristic of sync is not entirely horrible.
For people with large USB drives frequently or always mounted, the
expectations are completely different.  As USB gets faster, and
laptops/nucs/etc get smaller, we see more and more people using USB
SSDs to expand storage.

Justin

>
> >> We approach this problem from a different angle: the user is supposed
> >> to sync the filesystem before removing. Graphical environments have
> >> an "eject" button, and for non-graphical environments, the user
> >> just needs to do a sync manually.
> >
> > I am aware of that, but it leads to a poor user experience, being notified that
a copying operation has been completed, while in reality, it has not.
>
>
> CLI: Doesn't that trigger the umount command already ... so no need to
> think about it (as user). As soon the umount completes the device can be
> unplugged.
>
>
>
> >> The same is true for "normal" writes to a harddrive. Operations
are
> >> generally async, and the user needs to do a shutdown, during which
> >> we sync and unmount filesystems. If you "yank" the cord, this may
(*)
> >> result in lost data and file system inconsistency.
> >
> > My proposal was exclusively for USB Storage devices, not internal ones, as I can
understand the benefits of async outweighing the detriments in the case of
internal/network etc storage devices.
>
>
> I never had a big issue in the GUI (long time/waits), as it clearly
> states to wait until removal is allowed (also stated via notify).
> Normally (here) this is just a few seconds (single digit).
>
> Maybe the idea can be implemented more transparently via vm.dirty_*
> sysctl per device if possible or is it a global threshold only?
>
>
> --
> Leon
>
>
> --
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Samuel Sieb]

On 1/31/24 00:57, Larina Loriasel via devel wrote:
> > We approach this problem from a different angle: the user is
> supposed
> > to sync the filesystem before removing. Graphical environments have
> > an "eject" button, and for non-graphical environments, the user
> > just needs to do a sync manually.
>  
>  I am aware of that, but it leads to a poor user experience, being notified that a copying
> operation has been completed, while in reality, it has not. 
As far as I know, every operating system does it the same way, so it's 
not as if users should be unaccustomed to dealing with this correctly. 
I've told windows users since windows existed that you need to "eject" 
the usb drive before taking it out.  I don't know if windows syncs more 
aggressively though.  I haven't actually tested how long it takes to 
eject after copying large amounts of data.  I think that's something 
that can be tuned in Linux.

[Florian Weimer]

* Zbigniew Jędrzejewski-Szmek:

>  On Wed, Jan 31, 2024 at 06:43:00AM -0000, Abyss Ether via devel
> wrote:
> > I created a simple PoC udev rule to mount USB Storage devices with the "sync
> option.
> > Available here :
> https://github.com/larina3315/personal-stuff/blob/main/linux/10-usb-stora...
> > 
> > Currently, USB Storage devices are mounted without the "sync" option,
> causing their writes to be cached.
> > This causes an issue, especially with GUI file managers like GNOME Files, where after
> a file copy operation, it would notify the user that the operation has been completed. If
> the user then tries to unmount the USB Storage device, they get notified that data is
> still being written to disk and that they should not remove the USB Storage device from
> their PC/Laptop/device.
>
>  'sync' has some strong downsides though: various operations become
>  painfully slow (this depends a lot on the hardware and its age, and
>  the history of previous writes, etc.), write operations block read
>  operations, and the total number of writes may be increased, leading
>  to more wear&tear on the hardware. 
I think this is somewhat counteracted by Linux treating USB mass storage
devices as not having write caches (according to dmesg at least).
Doesn't this mean that those costly barriers won't be used?

Of course it also means that “sync” is not effective as we would like.

Thanks,
Florian

[Barry]

>  On 31 Jan 2024, at 11:41, Florian Weimer <fweimer(a)redhat.com&gt;
> wrote:
>  
>  I think this is somewhat counteracted by Linux treating USB mass storage
>  devices as not having write caches (according to dmesg at least).
>  Doesn't this mean that those costly barriers won't be used? 
Isn’t that a reference to caches within the drive and not the buffer cache in the kernel?
It is the kernel buffer cache flushing that eject/umount triggers getting written out.

Barry

[Florian Weimer]

> > On 31 Jan 2024, at 11:41, Florian Weimer
> <fweimer(a)redhat.com&gt; wrote:
> > 
> > I think this is somewhat counteracted by Linux treating USB mass storage
> > devices as not having write caches (according to dmesg at least).
> > Doesn't this mean that those costly barriers won't be used?
>
>  Isn’t that a reference to caches within the drive and not the buffer
>  cache in the kernel? 
Yes, the kernel assumes that there are no such caches, but I think in
practice there are.  I think this means that journaling file systems are
not working correctly, in the sense that you do not get just user data
loss if the device is unplugged prematurely, but also metadata
corruption.

Thanks,
Florian

[Barry Scott]

>  On 2 Feb 2024, at 11:30, Florian Weimer <fweimer(a)redhat.com&gt;
> wrote:
>  
>  Yes, the kernel assumes that there are no such caches, but I think in
>  practice there are.  I think this means that journaling file systems are
>  not working correctly, in the sense that you do not get just user data
>  loss if the device is unplugged prematurely, but also metadata
>  corruption. 
As I understand it the kernel will request that writes are not cached. Which means
that journaling file systems do in fact work well.

As an aside a few years ago (10+ years?) Microsoft found that HDDs where using caching
to spoof benchmark results.

But this was leading to support hell for Microsoft with user reporting corrupt file
systems to them.

What Microsoft did was refuse to issue WHQL for any disk that did not have a write
through cache. We in Linux land benefit from this as spoofing hardware, I believe, is
not a common occurrence any more.

Barry

[Florian Weimer]

* Barry Scott:

>  As I understand it the kernel will request that writes are not
>  cached. Which means that journaling file systems do in fact work well. 
The kernel messages I get look like this:

kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
kernel: sd 0:0:0:0: [sda] 15814656 512-byte logical blocks: (8.10 GB/7.54 GiB)
kernel: sd 0:0:0:0: [sda] Write Protect is off
kernel: sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
kernel: sd 0:0:0:0: [sda] No Caching mode page found
kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
kernel: sd 0:0:0:0: [sda] Attached SCSI removable disk

kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
kernel: sd 0:0:0:0: [sda] Spinning up disk...
kernel: sd 0:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
kernel: sd 0:0:0:0: [sda] 7814035456 512-byte logical blocks: (4.00 TB/3.64 TiB)
kernel: sd 0:0:0:0: [sda] 4096-byte physical blocks
kernel: sd 0:0:0:0: [sda] Write Protect is off
kernel: sd 0:0:0:0: [sda] Mode Sense: 53 00 10 08
kernel: sd 0:0:0:0: [sda] No Caching mode page found
kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
kernel: sd 0:0:0:0: [sda] Attached SCSI disk

I thought that ‘Assuming drive cache: write through’ means that no
barriers are used.

The second one is a standard SATA drive in an USB enclosure, and those
have write-reordering caches, as far as I understand it.

Thanks,
Florian

[Barry Scott]

>  On 2 Feb 2024, at 17:58, Florian Weimer <fweimer(a)redhat.com&gt;
> wrote:
>  
>  The second one is a standard SATA drive in an USB enclosure, and those
>  have write-reordering caches, as far as I understand it. 
We need a kernel storage expert to tell us the definitive truth on this stuff.
I may be out of date on this stuff.

What I understand is that the drive will be told via the appropriate SCSI(?) command
that it must not reorder the writes. Failure to implement that command means the
drive will not have WHQL from Microsoft. Without WHQL its very hard to sell a drive
in the market place.

Barry

[Dominique Martinet]

Barry Scott wrote on Fri, Feb 02, 2024 at 08:07:46PM +0000:
>  > On 2 Feb 2024, at 17:58, Florian Weimer
> <fweimer(a)redhat.com&gt; wrote:
>  > The second one is a standard SATA drive in an USB enclosure, and those
>  > have write-reordering caches, as far as I understand it.
>  
>  We need a kernel storage expert to tell us the definitive truth on this stuff.
>  I may be out of date on this stuff. 
This isn't really my turf so I just "read the code" (as of 6.7), but...

>  What I understand is that the drive will be told via the appropriate
> SCSI(?) command
>  that it must not reorder the writes. Failure to implement that command means the
>  drive will not have WHQL from Microsoft. Without WHQL its very hard to sell a drive
>  in the market place. 
What seems to happen is the other way around:
- the scsi layer sends a SENSE command to query what the drive supports,
but that fails ("No Caching mode page found")
- it then goes on to pick whatever default value is configured
(litteraly assuming the drive works as per the default, hence the
"Assuming drive cache: write through" message)
- In this write through mode the 'WCE' flag is set to 0, which'll in
turn configure the blk queue (linux side) to clear "QUEUE_FLAG_WC" and
"QUEUE_FLAG_HW_WC" flags
- Things are starting to get a little harder to follow from here, but it
looks like that flush ops when QUEUE_FLAG_WC isn't set will clear
REQ_PREFLUSH | REQ_FUA from the request, and if there was no data to
write straight out skip the op...  (submit_bio_noacct around the "Filter
flush bio's early so that bio based drivers without flush support don't
have to worry about them." comment - I definitely could be
misunderstanding the code below)
- I also don't see anything that'd tell the disk about our assumption --
there's a "cache_type_store" function that'll expose a cache_type sysfs
knob for userspace to override, but at least kernel doesn't look like
it'll send a scsi command to set it by default.
Also, since we couldn't read the cache mode, there's no guarantee it'd
be settable in the first place.



So I think Florian is correct in that barriers won't be issued on
these disks, and if they internally have such a cache it'd probably be
unsafe...

Now does the disk itself know that it's in such an enclosure and
properly behaves as write through?
I think we'd have a lot more corruptions on our hands if it was
incorrect here, btrfs in particular is very sensitive to disks that
lie with barriers but I'm not sure how much it's used on such drivers.


I've had a quick look but didn't find any 'disk barrier sanity tool'
that'd issue a succession of write + flush ops in an order that'd be
easy to reorder (e.g. 13_______2) for one to unplug the disk and then
check there was no hole after plugging it back in; if someone is aware
of one that'd be interesting to test on such an enclosed HDD.
(Of course, getting safe order back is no guarantee that the disk is
always consistent, but it's probably possible to come up with a few
patterns that often fail when manually misconfiguring a disk)

-- 
Dominique Martinet | Asmadeus

[Simon Farnsworth]

On Friday, 2 February 2024 21:47:00 GMT Dominique Martinet wrote:
>  So I think Florian is correct in that barriers won't be issued
> on
>  these disks, and if they internally have such a cache it'd probably be
>  unsafe...
>  
>  Now does the disk itself know that it's in such an enclosure and
>  properly behaves as write through?
>  I think we'd have a lot more corruptions on our hands if it was
>  incorrect here, btrfs in particular is very sensitive to disks that
>  lie with barriers but I'm not sure how much it's used on such drivers.
>   
As far as I can tell, having owned such an enclosure in the past, it doesn't 
tell the drive anything - it relies on the fact that SATA drives write back 
fairly aggressively when the command interface is idle, and thus the cache is 
fully written back to the medium when it's idle for a few seconds. If you go 
through the sequence "remove safely", then wait for UI confirmation, then 
unplug the USB port, it'll usually have written back the cache before you 
physically move your hands to the USB cable.

As these enclosures aren't particularly fast (for example, commands must 
complete in issue order, rather than being able to complete out of order as in 
TCQ/NCQ), I suspect most users of them don't use btrfs or xfs on them (I'd 
expect ext4 if Linux-only, FAT32, exFAT or NTFS if shared between OSes); 
they're basically intended to let you have a huge "thumb drive" for
machine-
to-machine transfer, and they're fine in that use case.

On the other hand, turning on the "sync" mount option doesn't help with
these 
enclosures; if anything, it'll make things worse by increasing the number of 
commands in flight which have to be handled before the drive switches to 
cleaning its cache.
-- 
Simon Farnsworth

[Lennart Poettering]

On Mi, 31.01.24 06:43, Fedora Development ML (devel(a)lists.fedoraproject.org) wrote:

>  I created a simple PoC udev rule to mount USB Storage devices with
> the "sync option.
>  Available here :
> https://github.com/larina3315/personal-stuff/blob/main/linux/10-usb-stora...
>
>  Currently, USB Storage devices are mounted without the "sync"
>  option, causing their writes to be cached.  This causes an issue,
>  especially with GUI file managers like GNOME Files, where after a
>  file copy operation, it would notify the user that the operation has
>  been completed. If the user then tries to unmount the USB Storage
>  device, they get notified that data is still being written to disk
>  and that they should not remove the USB Storage device from their
>  PC/Laptop/device.
>
>  This can take a more severe form if the user is using a CLI/GUI
>  utility that DOES NOT inform the user that data is still being
>  written, like the 'cp' CLI utility, the user might be misled into
>  thinking that all writes have finished and plug the USB drive out,
>  at which point data corruption due to unfinished writes occur.
>
>  This is why, I think functionality should be included in Fedora
>  Linux (or atleast in Fedora Workstation and other desktop spins)
>  such that USB Storage devices are mounted with the "sync" options by
>  default. 
This tanks performance when writing to the device though. There's a
much better approach however: use an automount in between with a very
short timeout (2s or so). This means the mount appears continously
available from application PoV but the backing fs is only mounted for
a brief time around accesses. This allows caching and asynchronous
behaviour to work, but after 2s everything is forced out to disk
anyway and it is guaranteed the superblock of the disk is put back
into a clean state.

systemd supports this natively, for example with a simple
"systemd-mount -A /dev/sda1".

I wished the desktop UIs would use this, or that udisks would learn
such a mode too, and in fact default to it.

Lennart

--
Lennart Poettering, Berlin

[Larina Loriasel]

Such a way to dynamically mount and perform reads/writes to the device only when needed is
truly an ideal solution, looking forward to udisks implementing such a feature.

[Roberto Ragusa]

On 1/31/24 09:41, Lennart Poettering wrote:

>  This tanks performance when writing to the device though. There's
> a
>  much better approach however: use an automount in between with a very
>  short timeout (2s or so). This means the mount appears continously
>  available from application PoV but the backing fs is only mounted for
>  a brief time around accesses. This allows caching and asynchronous
>  behaviour to work, but after 2s everything is forced out to disk
>  anyway and it is guaranteed the superblock of the disk is put back
>  into a clean state.
>  
>  systemd supports this natively, for example with a simple
>  "systemd-mount -A /dev/sda1".
>   
Another possible approach: run "sync -f" every 3 seconds.

The filesystem will still be dirty, but data should be there
for careless users too. But performance will not be ideal.

On the other hand it avoids a problem that your continuous mount/umount
has: read cache is lost as soon as the device is not continuously active,
which is not optimal.

Regards.

-- 
    Roberto Ragusa    mail at robertoragusa.it

[Lennart Poettering]

On Fr, 02.02.24 10:10, Roberto Ragusa (mail(a)robertoragusa.it) wrote:

>  On 1/31/24 09:41, Lennart Poettering wrote:
>
>  > This tanks performance when writing to the device though. There's a
>  > much better approach however: use an automount in between with a very
>  > short timeout (2s or so). This means the mount appears continously
>  > available from application PoV but the backing fs is only mounted for
>  > a brief time around accesses. This allows caching and asynchronous
>  > behaviour to work, but after 2s everything is forced out to disk
>  > anyway and it is guaranteed the superblock of the disk is put back
>  > into a clean state.
>  >
>  > systemd supports this natively, for example with a simple
>  > "systemd-mount -A /dev/sda1".
>  >
>
>  Another possible approach: run "sync -f" every 3 seconds. 
While that should make sure the unwritten data hits the disk it doesn't
put the superblock in order to mark it as "this fs has been cleanly
unmounted". That's quite limiting.

Lennart

--
Lennart Poettering, Berlin

[Roberto Ragusa]

On 2/2/24 10:25, Lennart Poettering wrote:

> > Another possible approach: run "sync -f" every 3
> seconds.
>  
>  While that should make sure the unwritten data hits the disk it doesn't
>  put the superblock in order to mark it as "this fs has been cleanly
>  unmounted". That's quite limiting.
>   
Sure.

Sounds like a possible improvement at kernel level:
- half-unmount: sync the data, write a clean superblock, keep the memory
cache alive, when a new write happens make the superblock dirty again

Sort of "suspend/wakeup" for filesystems.
Should be a cousin of FIFREEZE / FITHAW ioctl for quiescing.

Regards.
-- 
    Roberto Ragusa    mail at robertoragusa.it

[Roberto Ragusa]

On 1/31/24 07:43, Abyss Ether via devel wrote:
>  I created a simple PoC udev rule to mount USB Storage devices with
> the "sync option.
>  Available here :
> https://github.com/larina3315/personal-stuff/blob/main/linux/10-usb-stora...
>  
>  Currently, USB Storage devices are mounted without the "sync" option, causing
> their writes to be cached.
>  This causes an issue, especially with GUI file managers like GNOME Files, where after a
> file copy operation, it would notify the user that the operation has been completed. If
> the user then tries to unmount the USB Storage device, they get notified that data is
> still being written to disk and that they should not remove the USB Storage device from
> their PC/Laptop/device. 
Users should have learned to use "eject", this should be known since Windows
98,
it is 25 years ago.
Why should I be forced to work around terrible speed by default on my backup drive
because of these clueless users?

Regards.

-- 
    Roberto Ragusa    mail at robertoragusa.it

[Simon Farnsworth]

On Wednesday, 31 January 2024 06:43:00 GMT Abyss Ether via devel wrote:
>  I created a simple PoC udev rule to mount USB Storage devices with
> the "sync
>  option. Available here :
>  https://github.com/larina3315/personal-stuff/blob/main/linux/10-usb-storage
>  .rules 
>  Currently, USB Storage devices are mounted without the "sync" option,
>  causing their writes to be cached. This causes an issue, especially with
>  GUI file managers like GNOME Files, where after a file copy operation, it
>  would notify the user that the operation has been completed. If the user
>  then tries to unmount the USB Storage device, they get notified that data
>  is still being written to disk and that they should not remove the USB
>  Storage device from their PC/Laptop/device. 
>  This can take a more severe form if the user is using a CLI/GUI utility that
>  DOES NOT inform the user that data is still being written, like the 'cp'
>  CLI utility, the user might be misled into thinking that all writes have
>  finished and plug the USB drive out, at which point data corruption due to
>  unfinished writes occur. 
 
>  This is why, I think functionality should be included in Fedora Linux
> (or
>  atleast in Fedora Workstation and other desktop spins) such that USB
>  Storage devices are mounted with the "sync" options by default. -- 
Coming at the problem in a different direction; the kernel has options to limit 
how much data is in flight to any given device, but the Fedora system doesn't 
set these. See

https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-class-bdi

for documentation. As per the experiment documented below, if we set per-
device BDI options "sensibly" based on the device's performance, we'd be
able 
to reduce the window of surprise for users, without forcing all writes to be 
unbuffered via the "sync" option; instead, we could reduce the buffer to under a

second for removable devices, and thus ensure that the user isn't surprised by 
massive data loss when they remove a device.

As a nice side-effect, these changes would also speed up unmount operations 
from a GUI, since it limits the amount of dirty data to be written back before 
the unmount completes.

The experiment follows:

First, there's global settings:

[sfarnsworth@ USB]$ sysctl vm | grep dirty
sysctl: permission denied on key 'vm.mmap_rnd_bits'
sysctl: permission denied on key 'vm.mmap_rnd_compat_bits'
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 1500
vm.dirtytime_expire_seconds = 43200
sysctl: permission denied on key 'vm.stat_refresh'

This tells me that on my system, we use up to 10% of my total RAM (64 GiB) as 
buffering for writes without even accessing the device, and up to 20% as 
buffering before the kernel will force a process to wait while it writes data 
out to the device.

I just plugged a 4 GiB USB 2.0 stick into my machine, and I get the following 
values:

[sfarnsworth@ /sys/class/bdi/8:0]$ grep . *
max_bytes:7245996032
max_ratio:100
max_ratio_fine:1000000
min_bytes:0
min_ratio:0
min_ratio_fine:0
grep: power: Is a directory
read_ahead_kb:128
stable_pages_required:0
strict_limit:0

This tells the kernel that it can buffer up to 7 GiB of data (max_bytes), and 
it doesn't need to consider the per-device limit until it's exhausted a global 
dirty limit (strict_limit). The default kernel settings allow you to complete 
your write immediately if you've used not more than 10% of the total buffer.

If I tell dd to write a gigabyte to the device with the dsync flag set to 
ensure that it writes data to the device before returning (although not 
metadata), I can see that the kernel is permitted to buffer a huge amount of 
data in terms of time taken to write everything out:

[sfarnsworth@ USB]$ dd if=/dev/zero of=test bs=$((1024 * 1024)) count=1024 
oflag=dsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 174.692 s, 6.1 MB/s

I change the mount options to remove "flush", since this is a FAT32 stick, and 
the "flush" option causes the kernel to block on file close until the data is 
fully written:

[sfarnsworth@USB]$ dd if=/dev/zero bs=$((1024 * 1024)) count=1024 of=test
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.664286 s, 1.6 GB/s

This means we've buffered (quite happily) 3 minutes worth of writeback, which 
I'd have to wait for before removing the stick - but dd has already exited, so 
it's not "obvious" to me that I need to wait a long time.

I now change the per-device BDI options for the USB stick, based on what I've 
found out above:

[root@ 8:0]# echo $((10 * 1024 * 1024)) > max_bytes
[root@ 8:0]# echo 1 > strict_limit 
[root@8:0]# pwd && grep . *
/sys/devices/virtual/bdi/8:0
max_bytes:10475956
max_ratio:0
max_ratio_fine:1485
min_bytes:0
min_ratio:0
min_ratio_fine:0
grep: power: Is a directory
read_ahead_kb:128
stable_pages_required:0
strict_limit:1
grep: subsystem: Is a directory

This time, when I repeat the "fast" dd, it doesn't exit until almost 
everything is written out:
[sfarnsworth@ USB]$ dd if=/dev/zero bs=$((1024 * 1024)) count=1024 of=test
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 163.928 s, 6.6 MB/s