error adding seccomp filter rule for syscall bdflush: permission denied

[cevich]

#573 Causes an error on Ubuntu when running any container using cri-o-runc 1.0.0~rc95.1. It also happens (not too unexpectedly) from buildah bud (first identified here). The problem does not occur when using the prior version of containers-common (which doesn't contain the #573 changes).

# podman --runtime=runc run -it --rm quay.io/libpod/alpine:latest echo hello
Error: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied: OCI permission denied

Conveniently, I have two VM Images built 3-days apart which bisect the problem versions. See test PR containers/podman#10709

• c4635821094469632 built on the 13th does not reproduce the problem (containers-common 1_15)
• c4805484248039424 built on the 16th does reproduce the problem (containers-common 1_16)

Hint: If you check out commit 9efcf62 from that PR, you can use hack/get_ci_vm.sh int podman ubuntu-2010 root host to see it break. Commit f4c50f6 will get you the prior version VM (without the problem).

[cevich]

@giuseppe broke it 😀

Context: This isn't a problem released into the wild yet. Only seeing it in CI when testing out new VM images.

[giuseppe]

@cevich I've seen the error "container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied: OCI permission denied" with versions of runc older than 1.0-rc95.

Could we first double check the runc version?

[ryester19]

@cevich I've seen the error "container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied: OCI permission denied" with versions of runc older than 1.0-rc95.

Could we first double check the runc version?

Having the same problem with containers-common-0.40.0-1, and my Arch Linux setup has runc 1.0.0rc95-1

[giuseppe]

@ryester27 is it on x86_64?

Could you try with a custom seccomp profile where you drop bdflush from it?

[ryester19]

@ryester27 is it on x86_64?

Could you try with a custom seccomp profile where you drop bdflush from it?

x86_64, yes

I removed bdflush from /etc/containers/seccomp.json. Tried running another container, and then it complained about clone3. Removed that too, then complained about io_pgetevents next. And so on....

Let me know if that wasn't the right way to test this

[cevich]

Thanks @giuseppe

with versions of runc older than 1.0-rc95

I just looked, the problem images have cri-o-runc 1.0.0~rc95.1 from Kubik, but maybe it's right on the line. Is there a specific commit we can check to see if it's present or not?

Also, if we're building new packages anyway, should we also bump libseccomp (2.5.1) or anything else?

[cevich]

Tried running another container, and then it complained about clone3.

FWIW: I also tried this, and had the exact same result. It just fails on the next one in the list 😢

[cevich]

Idea: Since it should be easy to check, let me try my reproducer after rebuilding/installing from upstream runc...

[cevich]

...okay, this failure is reproducing even with the latest runc from upstream source:

# podman --runtime=/usr/local/sbin/runc run -it --rm quay.io/libpod/alpine:latest
 echo hello
Trying to pull quay.io/libpod/alpine:latest...
Getting image source signatures
Copying blob 9d16cba9fb96 done
Copying config 9617696764 done
Writing manifest to image destination
Storing signatures
Error: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied: OCI permission denied

# type -P runc
/usr/local/sbin/runc

# /usr/local/sbin/runc --version
runc version 1.0.0-rc95+dev
commit: v1.0.0-rc95-124-g9f937782
spec: 1.0.2-dev
go: go1.15.11
libseccomp: 2.5.1

[giuseppe]

I see what is happening. We also need an updated Podman that is using #573, otherwise some of the settings in #573 are not honored by Podman

[kris7t]

@ryester27 I'm also having this issue on Arch Linux with containers-common 0.40.0, podman 3.2.1, and runc 1.0.0rc95. However, replacing /etc/containers/seccomp.json with the file from containers-common 0.39.0 is a workaround and lets containers starts properly, albeit at the expense of less strict seccomp filters, I guess.

[rammanokar]

Issue resolved after rolling back to containers-common 0.39.0

sudo pacman -U https://archive.archlinux.org/packages/c/containers-common/containers-common-0.39.0-4-any.pkg.tar.zst

[cevich]

I see what is happening.

Fantastic news @giuseppe

Please let me know if/how I can help.

[cevich]

We also need an updated Podman that is using #573

I'm hoping 'updated Podman' means code changes then package updates. Remember, in addition to the pre-packaged podman, I also reproducing this on Ubuntu + Runc in podman CI, where we build from source. Though that was 6-days ago, so maybe the situation has changed?

[cevich]

Update: I re-ran my podman test PR and it passed, so the issue must be resolved in upstream podman. So we're just waiting for updated podman packages to go out to Kubik testing?

Assuming so, that's great news.

[rhatdan]

Can we close this issue?

[dtchanpura]

I tried with following versions and everything seems to be working as expected.

I believe runc version has been updated on 23rd June

~ ❯ pacman -Q containers-common podman runc
containers-common 0.40.1-1
podman 3.2.1-1
runc 1.0.0-1
~ ❯ podman run -it --rm ubuntu:20.04
root@af8fd5340d0b:/#

[cevich]

Can we close this issue?

I haven't checked...did @lsm5 build new podman packages in Kubik testing?

[cevich]

I'll catch up with this early next week, since new VM images need to be made again anyway. I can re-open this if it's still a problem.

[cevich]

Ref, new runc package in kubik testing: https://build.opensuse.org/package/rdiff/devel:kubic:libcontainers:testing/cri-o-runc?linkrev=base&rev=17

[darrellenns]

Contrary to @dtchanpura, this is still broken for me.

> pacman -Q containers-common podman runc                                                                                                                                        
containers-common 0.40.1-1
podman 3.2.2-1
runc 1.0.0-1
> podman run -it --rm ubuntu:20.04
Error: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied: OCI permission denied

It works if I downgrade to containers-common-0.39.0-4.

> pacman -Q containers-common podman runc        
containers-common 0.39.0-4
podman 3.2.2-1
runc 1.0.0-1
> podman run -it --rm ubuntu:20.04
root@c06728581153:/#

If I remove bdflush from seccomp.json, it just fails with a different error:

Error: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall io_pgetevents: permission denied: OCI permission denied

[darrellenns]

@dtchanpura did you have crun installed when you ran that test? I have found that (at least on arch linux) containers-common will default to crun if it's installed, which is not affected by this issue. So an effective workaround for arch users with the default configuration is to just install crun. Others may need to set runtime="crun" in contianers.conf.

Arch bug report here: https://bugs.archlinux.org/task/71397

[dtchanpura]

@darrellenns Sorry about that, I did not see other dependencies, but yes, I do have crun installed as a dependency for buildah and it coincided with the runc update so couldn't distinguish if that was because of runc or crun.

~ ❯ pacman -Q crun runc podman containers-common
crun 0.20.1-1
runc 1.0.0-1
podman 3.2.2-1
containers-common 0.40.1-1

Edit: I could reproduce the issue removing buildah and crun.

[cyphar]

Was there ever a runc bug report opened about this issue? I can't seem to find it.

[giuseppe]

Was there ever a runc bug report opened about this issue? I can't seem to find it.

AFAIK there is no issue in upstream runc. The problem existed with the old runc version we've used that didn't know about defaultErrno

[cyphar]

The way I understood the error message above is that you get the error when generating the seccomp program, not during execution (so it shouldn't be related to defaultErrnoRet?). I ask because we have a bug open for this issue since it seems there's still a bug in runc but everyone has "worked around it" by switching to crun rather than reporting the bug?

[giuseppe]

from what I saw and I could reproduce locally, the defaultErrno set to ENOSYS is ignored by old runc versions thus adding any other rule that fails with EPERM would cause that error, because it is already the default errno.

I could not reproduce the issue with newer runc versions

[cevich]

This was a complex issue to fix, there may be some parts still unaddressed in the older VM images. In case it helps: #742 makes them current with what's in podman CI.

[Rwarcards762]

FWIW, if anyone ends up here for podman issues regarding this error on RHEL, RHEL 8.4 has a borked version of runc whilst updating to RHEL 8.5's bundled runc fixes the error. 🚀

[Qix-]

I'm on Debian Bullseye facing this issue after building Podman from source. There's no clear fix given that Debian doesn't use Pacman here.

What's the solution? Bullseye is currently the latest version that gets security updates, and current stable podman seems to entirely ignore .containerignore (a separate issue I'm ultimately trying to evaluate). Hence building from source.

EDIT: Turns out crun wasn't installed. Installing it fixed the issue. Some better error messages are really needed here.

[sajith]

I'm on Debian Bullseye facing this issue after building Podman from source. There's no clear fix given that Debian doesn't use Pacman here.

Thank you for leaving a hint here, @Qix-!

I'm on Debian Bookworm (11.3) and I installed Podman from Debian. I too had to install crun, although runc was already installed. I don't understand why runc didn't work for me. I agree that the error message should be better.

[rhatdan]

Most likely runc is out of date.

[Kapkiai]

If anyone ends up here today and is using redhat 8.4, update runc (dnf update runc) fixes this