Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large output of processes with TTY gets occasionally truncated #3738

Closed
danail-branekov opened this issue Oct 9, 2019 · 1 comment · Fixed by #3743
Closed

Large output of processes with TTY gets occasionally truncated #3738

danail-branekov opened this issue Oct 9, 2019 · 1 comment · Fixed by #3743
Labels
kind/bug

Comments

@danail-branekov
Copy link
Contributor

Description

Hi containerd,

We (CF Garden) are experiencing occasional container process output being cut out in our CI environment. We managed to reproduce the issue via a unit test

The test loops 100 times exec'ing a process with TTY enabled in a container which prints numbers from 1 to 1000000 to its stdout. The test checks the final numbers are printed. Depending on the runtime we use to run the test with we get the following results:

  • When running with V1 (io.containerd.runtime.v1.linux) the test would either fail with a process output being cut out, or the process being executed (i.e. seq) just blocks writing to stdout. In the cases when the test fails we get the following error:
--- FAIL: TestContainerExecLargeOutputWithTTY (49.19s)
    container_test.go:320: process output does not end with "999999 1000000" at iteration 66, here are the last 20 characters of the output:

         "999931 999932 999933"
FAIL
  • When running with V2 (io.containerd.runc.v2) the test would always hang with the seq process getting stuck writing to stdout.

We verified that the issue is available on both master and 1.3.0

cc @kieron-pivotal
@crosbymichael
Copy link
Member

I'll take a look at this one and see if I can find anything

@crosbymichael crosbymichael mentioned this issue Oct 11, 2019
Merged
crosbymichael added a commit to crosbymichael/containerd that referenced this issue Oct 11, 2019
@crosbymichael
Handle large output in v2 shim with TTY
f8cca26 
Reized the I/O buffers to align with the size of the kernel buffers with fifos
and move the close aspect of the console to key off of the stdin closing.

Fixes containerd#3738

Signed-off-by: Michael Crosby <[email protected]>
estesp pushed a commit to estesp/containerd that referenced this issue Oct 14, 2019
@crosbymichael @estesp
Handle large output in v2 shim with TTY
0f6aab1 
Reized the I/O buffers to align with the size of the kernel buffers with fifos
and move the close aspect of the console to key off of the stdin closing.

Fixes containerd#3738

Signed-off-by: Michael Crosby <[email protected]>
Signed-off-by: Phil Estes <[email protected]>
erikwilson pushed a commit to erikwilson/rancher-containerd that referenced this issue Oct 23, 2019
@crosbymichael @erikwilson
Handle large output in v2 shim with TTY
850959a 
Reized the I/O buffers to align with the size of the kernel buffers with fifos
and move the close aspect of the console to key off of the stdin closing.

Fixes containerd#3738

Signed-off-by: Michael Crosby <[email protected]>
Signed-off-by: Phil Estes <[email protected]>
@thaJeztah thaJeztah mentioned this issue Nov 8, 2019
Merged
dmcgowan pushed a commit to thaJeztah/containerd that referenced this issue Nov 19, 2019
@thaJeztah @dmcgowan
Prepare v1.3.1 release
d512382 
* Update the runc vendor to v1.0.0-rc9 which includes an additional mitigation for [CVE-2019-16884](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-16884).
    - More details on the runc CVE in [opencontainers/runc#2128](opencontainers/runc#2128), and the additional mitigations in [opencontainers/runc#2130](opencontainers/runc#2130).
* Add local-fs.target to service file to fix corrupt image after unexpected host reboot. Reported in [containerd#3671](containerd#3671), and fixed by [containerd#3745](containerd#3745).
* Fix large output of processes with TTY getting occasionally truncated. Reported in [containerd#3738](containerd#3738) and fixed by [containerd#3754](containerd#3754).
* Fix direct unpack when running in user namespace. Reported in [containerd#3762](containerd#3762), and fixed by [containerd#3779](containerd#3779).
* Update Golang runtime to 1.12.13, which includes security fixes to the `crypto/dsa` package made in Go 1.12.11 ([CVE-2019-17596](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17596)), and fixes to the go command, `runtime`, `syscall` and `net` packages (Go 1.12.12).
* Add Windows process shim installer [containerd#3792](containerd#3792)

* CRI fixes:
    - Fix shim delete error code to avoid unnecessary retries in the CRI plugin. Discovered in [containerd/cri#1309](containerd/cri#1309), and fixed by [containerd#3733](containerd#3733) and [containerd#3740](containerd#3740).

Signed-off-by: Sebastiaan van Stijn <[email protected]>
@danail-branekov danail-branekov mentioned this issue Mar 13, 2020
Closed
tussennet pushed a commit to tussennet/containerd that referenced this issue Sep 11, 2020
@crosbymichael @87654testtesttesttesttesttest
Handle large output in v2 shim with TTY
3b16524 
Reized the I/O buffers to align with the size of the kernel buffers with fifos
and move the close aspect of the console to key off of the stdin closing.

Fixes containerd#3738

Signed-off-by: Michael Crosby <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Handle large output in v2 shim with TTY crosbymichael/containerd
2 participants
@crosbymichael @danail-branekov