Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. For my deployments, I would like to use ext4 or xfs, so my question are: 1) Is it possible to change the root file system at deploy time? 2) If not, how could one change the cloud images to use a different root file system?
Thanks and regards, Tadej Janež
[1] https://fedoraproject.org/wiki/Changes/FedoraCloudBtrfsByDefault
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež tadej.j@nez.si wrote:
Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. For my deployments, I would like to use ext4 or xfs, so my question are:
- Is it possible to change the root file system at deploy time?
- If not, how could one change the cloud images to use a different
root file system?
It is not possible to change at deployment time, you would need to build your own custom images.
Our images are now defined here: https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud....
If you want to use something else, you'd want to have your own version of the definitions and modify that file to use the filesystem of your choice.
Is there a particular reason you want to use ext4 or xfs for your rootfs? Typically the pattern we see is that people attach a secondary volume or use S3 and put their data on that instead of the rootfs.
On 4/9/24 10:00, Neal Gompa wrote:
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež tadej.j@nez.si wrote:
Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. For my deployments, I would like to use ext4 or xfs, so my question are:
- Is it possible to change the root file system at deploy time?
- If not, how could one change the cloud images to use a different
root file system?
It is not possible to change at deployment time, you would need to build your own custom images.
We do have Fedora CoreOS, which may or may not fit your use case (i.e. run software in containers, automatic updates, rpm-ostree as the update mechanism), which can re-provision the root filesystem on first boot and does provide images for various clouds:
- https://fedoraproject.org/coreos/download?stream=stable - https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfiguring_t...
This may or may not be an option for you, but figured I'd mention.
Our images are now defined here: https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud....
If you want to use something else, you'd want to have your own version of the definitions and modify that file to use the filesystem of your choice.
Is there a particular reason you want to use ext4 or xfs for your rootfs? Typically the pattern we see is that people attach a secondary volume or use S3 and put their data on that instead of the rootfs.
On Tue, 2024-04-09 at 11:04 -0400, Dusty Mabe wrote:
On 4/9/24 10:00, Neal Gompa wrote:
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež tadej.j@nez.si wrote:
Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. For my deployments, I would like to use ext4 or xfs, so my question are:
- Is it possible to change the root file system at deploy time?
- If not, how could one change the cloud images to use a
different root file system?
It is not possible to change at deployment time, you would need to build your own custom images.
We do have Fedora CoreOS, which may or may not fit your use case (i.e. run software in containers, automatic updates, rpm-ostree as the update mechanism), which can re-provision the root filesystem on first boot and does provide images for various clouds:
https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfiguring_t...
This may or may not be an option for you, but figured I'd mention.
Thanks for mentioning this, this looks very nice!
Does it copy the whole rootfs to RAM, erases and reformats it with a new file system and copies the contents from RAM back to it?
On the subject if this fits my use case, the answer is no. The systems don't run applications in containers. They are set up to run services via classical systemd units.
On Thu, Apr 11, 2024 at 5:25 AM Tadej Janež tadej.j@nez.si wrote:
On Tue, 2024-04-09 at 11:04 -0400, Dusty Mabe wrote:
On 4/9/24 10:00, Neal Gompa wrote:
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež tadej.j@nez.si wrote:
Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. For my deployments, I would like to use ext4 or xfs, so my question are:
- Is it possible to change the root file system at deploy time?
- If not, how could one change the cloud images to use a
different root file system?
It is not possible to change at deployment time, you would need to build your own custom images.
We do have Fedora CoreOS, which may or may not fit your use case (i.e. run software in containers, automatic updates, rpm-ostree as the update mechanism), which can re-provision the root filesystem on first boot and does provide images for various clouds:
https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfiguring_t...
This may or may not be an option for you, but figured I'd mention.
Thanks for mentioning this, this looks very nice!
Does it copy the whole rootfs to RAM, erases and reformats it with a new file system and copies the contents from RAM back to it?
This is basically what ignition does, yes.
On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote:
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež tadej.j@nez.si wrote:
Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. For my deployments, I would like to use ext4 or xfs, so my question are:
- Is it possible to change the root file system at deploy time?
- If not, how could one change the cloud images to use a different
root file system?
It is not possible to change at deployment time, you would need to build your own custom images.
Ok, thanks for clarifying!
Our images are now defined here: https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud....
If you want to use something else, you'd want to have your own version of the definitions and modify that file to use the filesystem of your choice.
I see, one would need to modify the "filesystem" key of the selected Cloud-Base-<provider> image.
Are there instructions on how to build a custom cloud image?
However, even if I can easily build a custom cloud image, the overhead of maintaining custom cloud images is very high.
For every Fedora release and cloud provider, I would need to build the image and then with IaC (e.g. Terraform), handle uploading the custom image and using it...
Is there a particular reason you want to use ext4 or xfs for your rootfs? Typically the pattern we see is that people attach a secondary volume or use S3 and put their data on that instead of the rootfs.
Yes, I use that pattern as well. Usually, /srv or /var would be on a separate block storage and formatted with the file system of choice.
The reasons why I would want to use, e.g. ext4 for the rootfs, would be:
1. Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS underneath) for the last 2 decades. I know the tools and I know what "care" such filesystems need.
2. Maturity. Ext2/3/4 have been round for quite longer than btrfs and there are very little "unknowns" or "surprises" with it.
3. Simplicity. When provisioning machines with a cloud provider, I actually don't need the LVM+ext4 combination because the cloud provider would typically handle the things LVM would handle for a non-cloud machine, e.g. increasing the block storage size, snapshotting, ...
Please, don't read this as a critique against btrfs, just me trying to explain why I would find it nicer to just use ext4 for the rootfs as well.
Regards, Tadej
On Wed, Apr 10, 2024 at 10:54 AM Tadej Janež tadej.j@nez.si wrote:
On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote:
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež tadej.j@nez.si wrote:
Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. For my deployments, I would like to use ext4 or xfs, so my question are:
- Is it possible to change the root file system at deploy time?
- If not, how could one change the cloud images to use a different
root file system?
It is not possible to change at deployment time, you would need to build your own custom images.
Ok, thanks for clarifying!
Our images are now defined here: https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud....
If you want to use something else, you'd want to have your own version of the definitions and modify that file to use the filesystem of your choice.
I see, one would need to modify the "filesystem" key of the selected Cloud-Base-<provider> image.
Are there instructions on how to build a custom cloud image?
The repository README includes a quickstart section for building your own images.
However, even if I can easily build a custom cloud image, the overhead of maintaining custom cloud images is very high.
For every Fedora release and cloud provider, I would need to build the image and then with IaC (e.g. Terraform), handle uploading the custom image and using it...
Yeah, it is more overhead, for sure.
Is there a particular reason you want to use ext4 or xfs for your rootfs? Typically the pattern we see is that people attach a secondary volume or use S3 and put their data on that instead of the rootfs.
Yes, I use that pattern as well. Usually, /srv or /var would be on a separate block storage and formatted with the file system of choice.
The reasons why I would want to use, e.g. ext4 for the rootfs, would be:
- Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS
underneath) for the last 2 decades. I know the tools and I know what "care" such filesystems need.
- Maturity. Ext2/3/4 have been round for quite longer than btrfs and
there are very little "unknowns" or "surprises" with it.
- Simplicity. When provisioning machines with a cloud provider, I
actually don't need the LVM+ext4 combination because the cloud provider would typically handle the things LVM would handle for a non-cloud machine, e.g. increasing the block storage size, snapshotting, ...
Please, don't read this as a critique against btrfs, just me trying to explain why I would find it nicer to just use ext4 for the rootfs as well.
That's totally fair. I will point out that one of the big reasons we use Btrfs is actually to enable space efficiency (which can lead to cost savings across the board). The used storage is roughly 40% less than on ext4 due to leveraging both transparent compression and reflinks.
The other reason is to make it easy to do replication for backups and other purposes.
You may also find it to be beneficial for easily avoiding certain kinds of failure cases that can happen on ext4 (such as inode exhaustion with containers), since btrfs dynamically allocates inodes per subvolume.
You can also grow (and shrink!) the volume live while the system is mounted and online safely.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, 2024-04-10 at 11:22 -0400, Neal Gompa wrote:
On Wed, Apr 10, 2024 at 10:54 AM Tadej Janež tadej.j@nez.si wrote:
On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote:
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež tadej.j@nez.si wrote:
Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. For my deployments, I would like to use ext4 or xfs, so my question are:
- Is it possible to change the root file system at deploy
time? 2) If not, how could one change the cloud images to use a different root file system?
It is not possible to change at deployment time, you would need to build your own custom images.
Ok, thanks for clarifying!
Our images are now defined here: https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud....
If you want to use something else, you'd want to have your own version of the definitions and modify that file to use the filesystem of your choice.
I see, one would need to modify the "filesystem" key of the selected Cloud-Base-<provider> image.
Are there instructions on how to build a custom cloud image?
The repository README includes a quickstart section for building your own images.
However, even if I can easily build a custom cloud image, the overhead of maintaining custom cloud images is very high.
For every Fedora release and cloud provider, I would need to build the image and then with IaC (e.g. Terraform), handle uploading the custom image and using it...
Yeah, it is more overhead, for sure.
Thinking out loud... the modification in my case would not be a custom tailored image for my use case but more similar to how Fedora project builds different spins if people want to use a non-GNOME desktop environment or ALT images for people using alternative architectures.
Is there a possibility to reuse the existing image building infrastructure and add tasks for building images with e.g. ext4 as the rootfs?
Is there a particular reason you want to use ext4 or xfs for your rootfs? Typically the pattern we see is that people attach a secondary volume or use S3 and put their data on that instead of the rootfs.
Yes, I use that pattern as well. Usually, /srv or /var would be on a separate block storage and formatted with the file system of choice.
The reasons why I would want to use, e.g. ext4 for the rootfs, would be:
- Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS
underneath) for the last 2 decades. I know the tools and I know what "care" such filesystems need.
- Maturity. Ext2/3/4 have been round for quite longer than btrfs
and there are very little "unknowns" or "surprises" with it.
- Simplicity. When provisioning machines with a cloud provider, I
actually don't need the LVM+ext4 combination because the cloud provider would typically handle the things LVM would handle for a non-cloud machine, e.g. increasing the block storage size, snapshotting, ...
Please, don't read this as a critique against btrfs, just me trying to explain why I would find it nicer to just use ext4 for the rootfs as well.
That's totally fair. I will point out that one of the big reasons we use Btrfs is actually to enable space efficiency (which can lead to cost savings across the board). The used storage is roughly 40% less than on ext4 due to leveraging both transparent compression and reflinks.
The other reason is to make it easy to do replication for backups and other purposes.
You may also find it to be beneficial for easily avoiding certain kinds of failure cases that can happen on ext4 (such as inode exhaustion with containers), since btrfs dynamically allocates inodes per subvolume.
You can also grow (and shrink!) the volume live while the system is mounted and online safely.
Thanks for enumerating these benefits. And I agree that these benefits alone justify the reason for switching to btrfs by default.
I might be tempted to check out the btrfs status page again and start playing with it.
On Thu, Apr 11, 2024 at 5:58 AM Tadej Janež tadej.j@nez.si wrote:
On Wed, 2024-04-10 at 11:22 -0400, Neal Gompa wrote:
On Wed, Apr 10, 2024 at 10:54 AM Tadej Janež tadej.j@nez.si wrote:
On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote:
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež tadej.j@nez.si wrote:
Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1]. For my deployments, I would like to use ext4 or xfs, so my question are:
- Is it possible to change the root file system at deploy
time? 2) If not, how could one change the cloud images to use a different root file system?
It is not possible to change at deployment time, you would need to build your own custom images.
Ok, thanks for clarifying!
Our images are now defined here: https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/cloud....
If you want to use something else, you'd want to have your own version of the definitions and modify that file to use the filesystem of your choice.
I see, one would need to modify the "filesystem" key of the selected Cloud-Base-<provider> image.
Are there instructions on how to build a custom cloud image?
The repository README includes a quickstart section for building your own images.
However, even if I can easily build a custom cloud image, the overhead of maintaining custom cloud images is very high.
For every Fedora release and cloud provider, I would need to build the image and then with IaC (e.g. Terraform), handle uploading the custom image and using it...
Yeah, it is more overhead, for sure.
Thinking out loud... the modification in my case would not be a custom tailored image for my use case but more similar to how Fedora project builds different spins if people want to use a non-GNOME desktop environment or ALT images for people using alternative architectures.
Is there a possibility to reuse the existing image building infrastructure and add tasks for building images with e.g. ext4 as the rootfs?
I think we could consider adding some overlay samples that show different ways to leverage the descriptions for building custom stuff on top of ours in the pagure.io/fedora-cloud namespace. I know David and I were already talking about doing that after we got done with the F40 cycle.
Is there a particular reason you want to use ext4 or xfs for your rootfs? Typically the pattern we see is that people attach a secondary volume or use S3 and put their data on that instead of the rootfs.
Yes, I use that pattern as well. Usually, /srv or /var would be on a separate block storage and formatted with the file system of choice.
The reasons why I would want to use, e.g. ext4 for the rootfs, would be:
- Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS
underneath) for the last 2 decades. I know the tools and I know what "care" such filesystems need.
- Maturity. Ext2/3/4 have been round for quite longer than btrfs
and there are very little "unknowns" or "surprises" with it.
- Simplicity. When provisioning machines with a cloud provider, I
actually don't need the LVM+ext4 combination because the cloud provider would typically handle the things LVM would handle for a non-cloud machine, e.g. increasing the block storage size, snapshotting, ...
Please, don't read this as a critique against btrfs, just me trying to explain why I would find it nicer to just use ext4 for the rootfs as well.
That's totally fair. I will point out that one of the big reasons we use Btrfs is actually to enable space efficiency (which can lead to cost savings across the board). The used storage is roughly 40% less than on ext4 due to leveraging both transparent compression and reflinks.
The other reason is to make it easy to do replication for backups and other purposes.
You may also find it to be beneficial for easily avoiding certain kinds of failure cases that can happen on ext4 (such as inode exhaustion with containers), since btrfs dynamically allocates inodes per subvolume.
You can also grow (and shrink!) the volume live while the system is mounted and online safely.
Thanks for enumerating these benefits. And I agree that these benefits alone justify the reason for switching to btrfs by default.
I might be tempted to check out the btrfs status page again and start playing with it.
Give it a shot, I think you'll be pleasantly surprised! :)
-
Dusty Mabe -
Neal Gompa -
Tadej Janež