AVC denied for docker while trying to set labels for tmpfs mounts

[Sujithra P]

Hi SELinux Experts,

The following issue is described in the below post as well.
https://github.com/containers/container-selinux/issues/141

Occasionally running into the following selinux denials for docker

type=AVC msg=audit(1626732057.636:4583): avc:  denied  { associate }
for  pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
scontext=system_u:object_r:container_file_t:s0:c263,c914
tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0

type=AVC msg=audit(1626812823.170:9434): avc:  denied  { associate }
for  pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
scontext=system_u:object_r:container_file_t:s0:c578,c672
tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0

level=error msg="Handler for POST
/v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
returned error: error setting label on mount source
'/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
failed to set file label on
/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
permission denied"

Docker is not able to set labels for these tmpfs mounts because they
end up having wrong labels when they are created (sometimes
"locale_t", sometimes "lib_t" which of course is not the
default/correct context for tmpfs fs).
Apparently semodule -R and deleting these tmps files or reboot of the
node fixes the problem.
Not sure what is causing the tmpfs mounts to get wrong labels in the
first place.

Everything seems to be fine to begin with, but as the system keeps
scheduling pods on the node, this behavior is observed sometimes (not
always consistent).

OS Details:

NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"

Docker Version:
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:36 2020
OS/Arch: linux/amd64
Experimental: false

Kubernetes Version*
v1.20.8-gke.1500

Any help on how to debug this issue  would be greatly appreciated.

Thanks
Sujithra.

[Ondrej Mosnacek]

On Thu, Jul 22, 2021 at 12:23 AM Sujithra P  wrote:
> Hi SELinux Experts,
>
> The following issue is described in the below post as well.
> https://github.com/containers/container-selinux/issues/141
>
> Occasionally running into the following selinux denials for docker
>
> type=AVC msg=audit(1626732057.636:4583): avc:  denied  { associate }
> for  pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
> scontext=system_u:object_r:container_file_t:s0:c263,c914
> tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
>
> type=AVC msg=audit(1626812823.170:9434): avc:  denied  { associate }
> for  pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
> scontext=system_u:object_r:container_file_t:s0:c578,c672
> tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
>
> level=error msg="Handler for POST
> /v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
> returned error: error setting label on mount source
> '/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
> failed to set file label on
> /var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
> permission denied"
>
> Docker is not able to set labels for these tmpfs mounts because they
> end up having wrong labels when they are created (sometimes
> "locale_t", sometimes "lib_t" which of course is not the
> default/correct context for tmpfs fs).
> Apparently semodule -R and deleting these tmps files or reboot of the
> node fixes the problem.
> Not sure what is causing the tmpfs mounts to get wrong labels in the
> first place.
>
> Everything seems to be fine to begin with, but as the system keeps
> scheduling pods on the node, this behavior is observed sometimes (not
> always consistent).
>
> OS Details:
>
> NAME="CentOS Linux"
> VERSION="8 (Core)"
> ID="centos"
> ID_LIKE="rhel fedora"
> VERSION_ID="8"
> PLATFORM_ID="platform:el8"
> PRETTY_NAME="CentOS Linux 8 (Core)"
>
> Docker Version:
> Client: Docker Engine - Community
> Version: 19.03.13
> API version: 1.40
> Go version: go1.13.15
> Git commit: 4484c46d9d
> Built: Wed Sep 16 17:02:36 2020
> OS/Arch: linux/amd64
> Experimental: false
>
> Kubernetes Version*
> v1.20.8-gke.1500
>
> Any help on how to debug this issue  would be greatly appreciated.

This looks really weird. Could be some subtle kernel bug. What is your
kernel version (run `uname -r`)? Are there any unusual errors/warnings
in the kernel log (`dmesg`) when this happens?

[Sujithra P]

Thanks Ondrej.

Kernel version:  Linux #2 SMP Fri Apr 23 09:05:57 PDT 2021 x86_64
x86_64 x86_64 GNU/Linux

yum list installed | grep selinux
container-selinux.noarch
2:2.162.0-1.module+el8.4.0+20195+0a4a4953      @ol8_appstream
libselinux.x86_64                                 2.9-5.el8
                          @ol8_baseos_latest
libselinux-utils.x86_64                           2.9-5.el8
                          @ol8_baseos_latest
python3-libselinux.x86_64                         2.9-5.el8
                          @ol8_baseos_latest
rpm-plugin-selinux.x86_64                         4.14.3-13.el8
                          @ol8_baseos_latest
selinux-policy.noarch                             3.14.3-67.0.1.el8
                          @ol8_baseos_latest
selinux-policy-targeted.noarch                    3.14.3-67.0.1.el8
                          @ol8_baseos_latest

No unusual errors/warnings in dmesg except

SELinux: mount invalid. Same superblock, different security settings
for (dev mqueue, type mqueue)

but I believe this is not harmful?

Thanks.

On Thu, Jul 22, 2021 at 3:07 AM Ondrej Mosnacek  wrote:
> On Thu, Jul 22, 2021 at 12:23 AM Sujithra P  wrote:
> > Hi SELinux Experts,
> >
> > The following issue is described in the below post as well.
> > https://github.com/containers/container-selinux/issues/141
> >
> > Occasionally running into the following selinux denials for docker
> >
> > type=AVC msg=audit(1626732057.636:4583): avc:  denied  { associate }
> > for  pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
> > scontext=system_u:object_r:container_file_t:s0:c263,c914
> > tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
> >
> > type=AVC msg=audit(1626812823.170:9434): avc:  denied  { associate }
> > for  pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
> > scontext=system_u:object_r:container_file_t:s0:c578,c672
> > tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
> >
> > level=error msg="Handler for POST
> > /v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
> > returned error: error setting label on mount source
> > '/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
> > failed to set file label on
> > /var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
> > permission denied"
> >
> > Docker is not able to set labels for these tmpfs mounts because they
> > end up having wrong labels when they are created (sometimes
> > "locale_t", sometimes "lib_t" which of course is not the
> > default/correct context for tmpfs fs).
> > Apparently semodule -R and deleting these tmps files or reboot of the
> > node fixes the problem.
> > Not sure what is causing the tmpfs mounts to get wrong labels in the
> > first place.
> >
> > Everything seems to be fine to begin with, but as the system keeps
> > scheduling pods on the node, this behavior is observed sometimes (not
> > always consistent).
> >
> > OS Details:
> >
> > NAME="CentOS Linux"
> > VERSION="8 (Core)"
> > ID="centos"
> > ID_LIKE="rhel fedora"
> > VERSION_ID="8"
> > PLATFORM_ID="platform:el8"
> > PRETTY_NAME="CentOS Linux 8 (Core)"
> >
> > Docker Version:
> > Client: Docker Engine - Community
> > Version: 19.03.13
> > API version: 1.40
> > Go version: go1.13.15
> > Git commit: 4484c46d9d
> > Built: Wed Sep 16 17:02:36 2020
> > OS/Arch: linux/amd64
> > Experimental: false
> >
> > Kubernetes Version*
> > v1.20.8-gke.1500
> >
> > Any help on how to debug this issue  would be greatly appreciated.
>
> This looks really weird. Could be some subtle kernel bug. What is your
> kernel version (run `uname -r`)? Are there any unusual errors/warnings
> in the kernel log (`dmesg`) when this happens?
>
> --
> Ondrej Mosnacek
> Software Engineer, Linux Security - SELinux kernel
> Red Hat, Inc.

[Ondrej Mosnacek]

On Thu, Jul 22, 2021 at 9:25 PM Sujithra P  wrote:
> Thanks Ondrej.
>
> Kernel version:  Linux #2 SMP Fri Apr 23 09:05:57 PDT 2021 x86_64
> x86_64 x86_64 GNU/Linux

Somehow that string doesn't contain the actual version :) uname -r
should return the right version string (something like
"4.18.0-305.el8.x86_64").
> yum list installed | grep selinux
> container-selinux.noarch
> 2:2.162.0-1.module+el8.4.0+20195+0a4a4953      @ol8_appstream
> libselinux.x86_64                                 2.9-5.el8
>                           @ol8_baseos_latest
> libselinux-utils.x86_64                           2.9-5.el8
>                           @ol8_baseos_latest
> python3-libselinux.x86_64                         2.9-5.el8
>                           @ol8_baseos_latest
> rpm-plugin-selinux.x86_64                         4.14.3-13.el8
>                           @ol8_baseos_latest
> selinux-policy.noarch                             3.14.3-67.0.1.el8
>                           @ol8_baseos_latest
> selinux-policy-targeted.noarch                    3.14.3-67.0.1.el8
>                           @ol8_baseos_latest
>
> No unusual errors/warnings in dmesg except
>
> SELinux: mount invalid. Same superblock, different security settings
> for (dev mqueue, type mqueue)
>
> but I believe this is not harmful?

Yes, that should be unrelated and (probably) harmless.

FWIW, I did find a data race bug in the related code, but at this
point I don't see how it would cause the behavior you're seeing. I
also haven't been able to reproduce the issue by trying to trigger the
race condition. So it may be just a red herring or I'm just missing
some key element that makes it happen. I'll keep digging...
> Thanks.
>
> On Thu, Jul 22, 2021 at 3:07 AM Ondrej Mosnacek  wrote:
> > On Thu, Jul 22, 2021 at 12:23 AM Sujithra P  wrote:
> > > Hi SELinux Experts,
> > >
> > > The following issue is described in the below post as well.
> > > https://github.com/containers/container-selinux/issues/141
> > >
> > > Occasionally running into the following selinux denials for docker
> > >
> > > type=AVC msg=audit(1626732057.636:4583): avc:  denied  { associate }
> > > for  pid=57450 comm="dockerd" name="/" dev="tmpfs" ino=150014
> > > scontext=system_u:object_r:container_file_t:s0:c263,c914
> > > tcontext=system_u:object_r:lib_t:s0 tclass=filesystem permissive=0
> > >
> > > type=AVC msg=audit(1626812823.170:9434): avc:  denied  { associate }
> > > for  pid=20027 comm="dockerd" name="/" dev="tmpfs" ino=198147
> > > scontext=system_u:object_r:container_file_t:s0:c578,c672
> > > tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0
> > >
> > > level=error msg="Handler for POST
> > > /v1.40/containers/a3a875e7896384e3bff53b8317e91ed4301a13957f42187eb227f28e09bd877c/start
> > > returned error: error setting label on mount source
> > > '/var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret':
> > > failed to set file label on
> > > /var/lib/kubelet/pods/f7cee5b2-bcd9-4aa1-9d67-c75b677ba2a1/volumes/kubernetes.io~secret/secret:
> > > permission denied"
> > >
> > > Docker is not able to set labels for these tmpfs mounts because they
> > > end up having wrong labels when they are created (sometimes
> > > "locale_t", sometimes "lib_t" which of course is not the
> > > default/correct context for tmpfs fs).
> > > Apparently semodule -R and deleting these tmps files or reboot of the
> > > node fixes the problem.
> > > Not sure what is causing the tmpfs mounts to get wrong labels in the
> > > first place.
> > >
> > > Everything seems to be fine to begin with, but as the system keeps
> > > scheduling pods on the node, this behavior is observed sometimes (not
> > > always consistent).
> > >
> > > OS Details:
> > >
> > > NAME="CentOS Linux"
> > > VERSION="8 (Core)"
> > > ID="centos"
> > > ID_LIKE="rhel fedora"
> > > VERSION_ID="8"
> > > PLATFORM_ID="platform:el8"
> > > PRETTY_NAME="CentOS Linux 8 (Core)"
> > >
> > > Docker Version:
> > > Client: Docker Engine - Community
> > > Version: 19.03.13
> > > API version: 1.40
> > > Go version: go1.13.15
> > > Git commit: 4484c46d9d
> > > Built: Wed Sep 16 17:02:36 2020
> > > OS/Arch: linux/amd64
> > > Experimental: false
> > >
> > > Kubernetes Version*
> > > v1.20.8-gke.1500
> > >
> > > Any help on how to debug this issue  would be greatly appreciated.
> >
> > This looks really weird. Could be some subtle kernel bug. What is your
> > kernel version (run `uname -r`)? Are there any unusual errors/warnings
> > in the kernel log (`dmesg`) when this happens?
> >
> > --
> > Ondrej Mosnacek
> > Software Engineer, Linux Security - SELinux kernel
> > Red Hat, Inc.

--
Ondrej Mosnacek
Software Engineer, Linux Security - SELinux kernel
Red Hat, Inc.