I am testing migration of CoreOS with Ignition 2.2 (named here current env) to to preview FedoraCoreOS with Ignition 3.0 (named here new env) Tests are performed on OpenStack Rocky on prem.
In current env we mount /dev/vdb to a folder /dockerdata and /dev/vdc to /var/lib/docker which works fine using Ignition 2.0.
I have transferred config to Ignition 3 and test now in new env. Mount to /var/lib/docker is fine, mount to /dockerdata fails :
.. [ 9.931065] ignition[696]: "filesystems": [ [ 9.931075] ignition[696]: { [ 9.931087] ignition[696]: "device": "/dev/vdc", [ 9.931097] ignition[696]: "format": "ext4", [ 9.931107] ignition[696]: "label": "docker", [ 9.932644] ignition[696]: "path": "/var/lib/docker", [ 9.932660] ignition[696]: "wipeFilesystem": true [ 9.932670] ignition[696]: }, [ 9.932680] ignition[696]: { [ 9.932689] ignition[696]: "device": "/dev/vdb", [ 9.932699] ignition[696]: "format": "ext4", [ 9.932714] ignition[696]: "label": "dockerdata", [ 9.932724] ignition[696]: "path": "/dockerdata", [ 9.932734] ignition[696]: "wipeFilesystem": true [ 9.932743] ignition[696]: } [ 9.932753] ignition[696]: ] .. .. [ 9.935087] ignition[696]: }CRITICAL : Ignition failed: mkdir /sysroot/dockerdata: operation not permitted [ 9.936383] umount[704]: umount: /sysroot/var: not mounted. [ 9.936548] systemd[1]: ignition-mount.service: Main process exited, code=exited, status=1/FAILURE [ 9.936696] systemd[1]: ignition-mount.service: Failed with result 'exit-code'. [ 9.936730] systemd[1]: Failed to start Ignition (mount).
Also tried to explicitly add a create directory /dockerdata using Ignition "directories .. /dockerdata" but same error. [ 9.537684] ignition[696]: }CRITICAL : Ignition failed: mkdir /sysroot/dockerdata: operation not permitted
Did I do something in the past that is not ok but was "tolerated" and now fails due to more strict checking?
Thanks for any feedback cheers Heiko
On 9/8/19 5:07 AM, Heiko Onnebrink wrote:
Hi Heiko!
I am testing migration of CoreOS with Ignition 2.2 (named here current env) to to preview FedoraCoreOS with Ignition 3.0 (named here new env) Tests are performed on OpenStack Rocky on prem.
Thanks for trying out Fedora CoreOS!
In current env we mount /dev/vdb to a folder /dockerdata and /dev/vdc to /var/lib/docker which works fine using Ignition 2.0.
I have transferred config to Ignition 3 and test now in new env. Mount to /var/lib/docker is fine, mount to /dockerdata fails :
.. [ 9.931065] ignition[696]: "filesystems": [ [ 9.931075] ignition[696]: { [ 9.931087] ignition[696]: "device": "/dev/vdc", [ 9.931097] ignition[696]: "format": "ext4", [ 9.931107] ignition[696]: "label": "docker", [ 9.932644] ignition[696]: "path": "/var/lib/docker", [ 9.932660] ignition[696]: "wipeFilesystem": true [ 9.932670] ignition[696]: }, [ 9.932680] ignition[696]: { [ 9.932689] ignition[696]: "device": "/dev/vdb", [ 9.932699] ignition[696]: "format": "ext4", [ 9.932714] ignition[696]: "label": "dockerdata", [ 9.932724] ignition[696]: "path": "/dockerdata", [ 9.932734] ignition[696]: "wipeFilesystem": true [ 9.932743] ignition[696]: } [ 9.932753] ignition[696]: ] .. .. [ 9.935087] ignition[696]: }CRITICAL : Ignition failed: mkdir /sysroot/dockerdata: operation not permitted [ 9.936383] umount[704]: umount: /sysroot/var: not mounted. [ 9.936548] systemd[1]: ignition-mount.service: Main process exited, code=exited, status=1/FAILURE [ 9.936696] systemd[1]: ignition-mount.service: Failed with result 'exit-code'. [ 9.936730] systemd[1]: Failed to start Ignition (mount).
Also tried to explicitly add a create directory /dockerdata using Ignition "directories .. /dockerdata" but same error. [ 9.537684] ignition[696]: }CRITICAL : Ignition failed: mkdir /sysroot/dockerdata: operation not permitted
Did I do something in the past that is not ok but was "tolerated" and now fails due to more strict checking?
I believe you are hitting this problem because of the filesystem structure/layout of OSTree (the underlying technology behind Fedora CoreOS). With OSTree there are certain filesystems that are read-write and certain ones that are read-only. It just so happens that the root of the filesystem tree is marked as immutable, which means you can't create files or directories under `/`. You can see that by running `lsattr -d /` on a booted system:
``` [root@localhost ~]# lsattr -d / ----i-------------- / ```
There is an open issue discussing this upstream as well: https://github.com/projectatomic/rpm-ostree/issues/337
Is it possible for you to use a directory under a writable filesystem (like `/var/lib/dockerdata`) instead?
Thanks for fast reply and guiding me to the discussion link,, I posted my comment there.
To answer your question:
If I would be on a green-field I would just go for /var/lib/dockerdata..
But unfortunately I am on a brown field and have to lock how to stay backward compatible. I would prefer an immutable directory creation option in the ignition config that allows me to create a folder under root (with immutable = true) that is just used as mount dir..
On 9/9/19 8:58 AM, Heiko Onnebrink wrote:
Thanks for fast reply and guiding me to the discussion link,, I posted my comment there.
To answer your question:
If I would be on a green-field I would just go for /var/lib/dockerdata..
But unfortunately I am on a brown field and have to lock how to stay backward compatible.
I understand. You might be able to get creative with symlinks, but yeah, not ideal.
I would prefer an immutable directory creation option in the ignition config that allows me to create a folder under root (with immutable = true) that is just used as mount dir..
Yeah I would say maybe open an issue at github.com/coreos/fedora-coreos-tracker where we can have a discussion on the feasibility of that.
Thanks for the feedback! Dusty
Opened issue here https://github.com/coreos/fedora-coreos-tracker/issues/270
Hope for a fruitful discussion ;-)
coreos@lists.fedoraproject.org