[atomic-wg] Issue #281: Figure out comprehensive strategy for atomic host container storage
dustymabe reported a new issue against the project: `atomic-wg` that you are following: `` We have system containers storing things in /var/lib/containers/storage. We have docker/moby containers storing things in /var/lib/docker. We might move to overlay2 on a single big root partition in the future, we might not.
We have discussed these topics in a few maililng list threads [[1](https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2017-Apri...)] [[2]()] and also ran out of storage on atomic host before [[3](https://bugzilla.redhat.com/show_bug.cgi?id=1391725)].
Let's put some thought into this and try to figure out a proper strategy going forward. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
walters added a new comment to an issue you are following: `` Another thread on list:
https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/... ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
dustymabe added a new comment to an issue you are following: `` Following from that thread the proposal from colin is that we move to one big / with overlayfs by default for our cloud/vagrant images with all space being taken by the root filesystme by default . This is something we had talked about in the past, but we have not formally made this proposal until now. Pending negative feedback in the f26 timeframe, I'm +1 to this.
The other part of this puzzle is making sure that all container storage *can* be mounted on another block device/filesystem easily.
So proposal is:
- 1 large filesystem by default - Ability to easily move container storage to another filesystem on system setup.
``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
walters added a new comment to an issue you are following: `` I'm going to do some work on this; though we're blocked a bit in testing by [rawhide issues](https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...). ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
walters added a new comment to an issue you are following: `` I tested the `20170615.n.0` FAH iso; it looks like we'll need this:
``` diff --git a/docker.spec b/docker.spec index b654082..b170151 100644 --- a/docker.spec +++ b/docker.spec @@ -569,10 +569,14 @@ echo 'STORAGE_DRIVER=overlay2' >> %{repo}-storage-setup-workstation ln -s %{repo}-storage-setup-workstation %{repo}-storage-setup-cloud # create server override config ln -s %{repo}-storage-setup-workstation %{repo}-storage-setup-server -# create atomic override config +# create atomic override config; see https://pagure.io/atomic-wg/issue/281 +%if 0%{?fedora} >= 27 +ln -s %{repo}-storage-setup-workstation %{repo}-storage-setup-atomichost +%else cp %{repo}-storage-setup-server %{repo}-storage-setup-atomichost echo 'CONTAINER_ROOT_LV_NAME=docker-root-lv' >> %{repo}-storage-setup-atomichost echo 'CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker' >> %{repo}-storage-setup-atomichost +%endif #atomichost %endif # custom_storage popd
```
Basically all the editions become the same thing. Seem sane?
``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
dustymabe added a new comment to an issue you are following: `` seems sane to me - i asked @dwalsh and @vgoyal to weigh in - also i'll bring this up in the next meeting we have. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
jberkus added a new comment to an issue you are following: `` @dustymabe seems sensible ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
vgoyal added a new comment to an issue you are following: `` So this relies on that fact that atomic users will not use devicemapper graph driver. If they really need to use devicemapper, they will need to add additional disk. (Which requires extra effort and planning).
So has this been established that overlayfs is working well for container workload and most of the users don't have the need to go back to devicemapper.
In F26, we wanted to retain capability to easily go back to devicemapper if things don't work out well with overlayfs. Given F26 is not GA, I guess we don't even have that data to make a decision? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
walters added a new comment to an issue you are following: `` @vgoyal: Not quite; the AH storage is the same as Server, we only use up to 15G. For real baremetal machines, they'll have larger disks, and hence there will be reserved space in the VG.
For cloud images, it's up to the operator; they can use cloud functionality to expand the root disk, or they can allocate a separate disk.
``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
vgoyal added a new comment to an issue you are following: `` I think I am missing something. So rootfs which used to be 3G by default will now be 15G by default.
Ok, so for cloud images we will continue to have volume group and when image is grown, additional space will go in volume group and one can carve out space for thin pool. Yep, that will work. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
walters added a new comment to an issue you are following: `` Thinking of this change as "3G → 15G by default" indeed is a good and simple way to phrase it. It's not *entirely* accurate since we also need to factor in the default size of the cloud images and the vagrant box. But it's a good first approximation.
But also we're no longer using a separate partition by default for AH.
(Incidentally, I have a pending proposal to also use this same partitioning for "Atomic Workstation", though for somewhat different reasons: https://pagure.io/pungi-fedora/pull-request/257 )
For reference, here's the first commit: http://pkgs.fedoraproject.org/cgit/rpms/fedora-productimg-atomic.git/commit/...
``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
walters added a new comment to an issue you are following: `` Docker patch [committed](http://pkgs.fedoraproject.org/cgit/rpms/docker.git/commit/?id=78b777a086500b...) and [built](https://koji.fedoraproject.org/koji/buildinfo?buildID=910722). ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
dustymabe added a new comment to an issue you are following: `` Discussed in the atomic working group meeting today:
*changing fedora 27 and beyond to default to overlayfs on the root partition will help us simplify our storage setup and align better with other fedora variants. This seems like a reasonable change to make assuming that overlayfs proves stable for the f26 cycle.* ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
walters added a new comment to an issue you are following: `` I swear we were defaulting to XFS before, but I may have been confused by the fact that it's hardcoded in the cloud kickstart.
Any objections to backporting this to f26?
``` diff --git a/installclass_atomic.py b/installclass_atomic.py index cce1470..7363360 100644 --- a/installclass_atomic.py +++ b/installclass_atomic.py @@ -36,6 +36,7 @@ class AtomicInstallClass(FedoraBaseInstallClass): name = "Atomic Host" sortPriority = 11000 hidden = False + defaultFS = "xfs"
def setDefaultPartitioning(self, storage): # 3GB is obviously arbitrary, but we have to pick some default. ```
(We could also consider backporting unified storage) ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
dustymabe added a new comment to an issue you are following: ``
Any objections to backporting this to f26?
I think if we make a good case for why XFS vs ext4 then backporting it to f26 would be a reasonable thing to do. I'd hate to do so 'just because', though. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
walters added a new comment to an issue you are following: `` We already discussed the rationale as an aside in the larger partitioning discussion earlier. F27 AH defaults to XFS already; that's what Fedora Server does, which also matches the downstream RHEL Atomic Host. The dynamic inodes in particular are nice for overlayfs. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
walters added a new comment to an issue you are following: `` https://koji.fedoraproject.org/koji/taskinfo?taskID=20464792 ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
mattdm added a new comment to an issue you are following: `` For what it's worth, I was talking with a Red Hat filesystems engineer and he strongly recommended moving to XFS across all of Fedora. Pretty much the only downside is lack of filesystem shrinking, and I don't think that's a big issue with AH. Not that we have to do it because RH says so -- I mean that that's an opinion I have some faith in. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
mattdm added a new comment to an issue you are following: `` I mean, not that I don't also have faith in you guys. :) ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
dustymabe added a new comment to an issue you are following: ``
Pretty much the only downside is lack of filesystem shrinking,
yeah, which i think sucks since we had that for ext4 and I actually used it a decent amount. That's just a personal gripe, not one I've really had to deal with too much in real life, though.
``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
jberkus added a new comment to an issue you are following: `` Issue #197 has been folded into this issue. Please do look at that issue, as it has some specific use-cases we want to solve. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
jberkus added a new comment to an issue you are following: `` Specifically, we want to be sure that the full disk is allocate on AWS instances. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
chrismurphy added a new comment to an issue you are following: `` All the partitioning, sizing, and resizing concerns mentioned in this issue vanish with a certain other filesystem, which does all resizes (grow, shrink, add and remove devices) online and atomically and typically in a single command. Whether scripted or user issued, the commands are shorter, easier to understand, complete faster and are safer.
Gotcha though is I haven't used it with overlayfs. A cursory search yields no hits. But it seems sane to allow Docker to continue to use overlayfs for the shared page cache benefit, and even snapshotting (if Docker supports that overlayfs feature now?).
But the main pro is that you can have separate fstrees read-only or read-write mounted, but they share the same storage pool, without hard barriers between them.
*shrug* ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
jberkus added a new comment to an issue you are following: `` "certain other filesystem"? Can you please be less opaque? ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
chrismurphy added a new comment to an issue you are following: `` Ahh sorry, I kinda figured realistically there are only three options: ext4, XFS, and Btrfs, and the only one not mentioned so far is Btrfs.
There's some hits of people using it in AWS contexts, but I have not yet run across Btrfs + overlayfs. So, I started a thread on linux-btrfs@vger.kernel.org to see if anyone's using containers with Btrfs + overlayfs. Insofar as I'm aware it's not a pathological combination, I'm just gonna guess to the vast majority it seems redundant, but they each bring different things to the table. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
jberkus added a new comment to an issue you are following: `` @chrismurphy ZFS is still out there, even if we can't touch it. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
chrismurphy added a new comment to an issue you are following: `` Yeah I wasn't considering anything we don't have in anaconda, but then also anything not already in the Fedora kernel for some time now.
Plus ZFS lacks fs shrink, and so you can't remove block devices arbitrarily, it also lacks online replication and seeding. So I even if it weren't for licensing it wouldn't be the direction I'd go in. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
jberkus added a new comment to an issue you are following: `` Anyway, I refuse to weigh into another XFS vs. BtrFS debate, and starting one here won't result in us having a storage plan. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
chrismurphy added a new comment to an issue you are following: `` I read a list of problems already with negative arguments, not supplied by me. And I've presented something that obviates literally all of them. I see it as advice, not debate. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
jberkus added a new comment to an issue you are following: `` I'm saying that the XFS vs. BtrFS decision was made elsewhere for Fedora, and we can't change it *in this issue*. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
dustymabe added a new comment to an issue you are following: ``
@jberkus Specifically, we want to be sure that the full disk is allocate on AWS instances.
I just tried this with a [rawhide qcow](https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20170715.n...) (not on aws but with kvm/qemu and giving it a bigger disk). I don't end up with a large disk, but just with the 3G and docker using the root filesystem + overlayfs.
``` [root@cloudhost sysconfig]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 40G 0 disk ├─sda1 8:1 0 300M 0 part /boot └─sda2 8:2 0 39.7G 0 part └─atomicos-root 253:0 0 3G 0 lvm /sysroot ```
There is some more configuration that needs to be done to make this happen. ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
walters added a new comment to an issue you are following: `` Oh right, duh: https://pagure.io/fedora-kickstarts/pull-request/266 ``
To reply, visit the link below or just reply to this email https://pagure.io/atomic-wg/issue/281
-
Chris Murphy -
Colin Walters
-
Dusty Mabe
-
Josh Berkus
-
Matthew Miller
-
Vivek Goyal