Skip to content

Intermittent tty timeouts on AIX #1820

Closed
Closed
Intermittent tty timeouts on AIX#1820
Labels
platform:aix

Description

@refack
refack
opened on Jun 4, 2019

We have a different but intermittent problem on AIX:

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/23617/console

08:32:35 ok 2550 pseudo-tty/test-async-wrap-getasyncid-tty
08:32:35   ---
08:32:35   duration_ms: 0.166
08:32:35   ...
08:42:35 Build timed out (after 10 minutes). Marking the build as failed.
08:42:35 gmake[1]: *** [test-ci] Terminated

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/23618/console

09:35:15 ok 2550 pseudo-tty/test-async-wrap-getasyncid-tty
09:35:15   ---
09:35:15   duration_ms: 0.176
09:35:15   ...
09:45:15 Build timed out (after 10 minutes). Marking the build as failed.
09:45:15 gmake[1]: *** [test-ci] Terminated

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/23622/console

11:38:22 ok 2539 message/vm_dont_display_syntax_error
11:38:22   ---
11:38:22   duration_ms: 0.110
11:38:22   ...
11:48:22 Build timed out (after 10 minutes). Marking the build as failed.
11:48:23 gmake[1]: *** [test-ci] Terminated

easy to reproduce:

> su iojs
> cd /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
> NODE_NAME=test-osuosl-aix61-ppc64_be-3 . build/jenkins/scripts/select-compiler.sh
> python tools/test.py pseudo-tty/console-dumb-tty

or

python -m trace -t tools/test.py --report pseudo-tty

to see a trace of all lines. Seems like there's an issue with sub-process spawning and ptty, since the process stalls on:

test.py(648):   if faketty:
test.py(649):     os.close(rest['stdout'])

where rest['stdout'] is fd_out created by (out_master, fd_out) = pty.openpty()

Originally posted by @refack in #1815 (comment)

Activity

  1. refack
    added
    platform:aix
    on Jun 4, 2019

  2. richardlau commented on Jun 4, 2019

    @richardlau
    richardlau
    on Jun 4, 2019
    Member

    Possibly related to nodejs/node#9728? cc @gireeshpunathil

  3. mhdawson commented on Jun 5, 2019

    @mhdawson
    mhdawson
    on Jun 5, 2019
    Member

    I think @sam-github has been looking at this one.

  4. sam-github commented on Jun 5, 2019

    @sam-github
    sam-github
    on Jun 5, 2019
    Contributor

    Nope, first I've heard of it.

    I seem to recall that years ago, during the first ports to AIX, that there was a problem with python involving ptys. @gireeshpunathil or @richardlau, you were around then, do you remember this?

  5. richardlau commented on Jun 5, 2019

    @richardlau
    richardlau
    on Jun 5, 2019
    Member

    Nope, first I've heard of it.

    I seem to recall that years ago, during the first ports to AIX, that there was a problem with python involving ptys. @gireeshpunathil or @richardlau, you were around then, do you remember this?

    I think it was the nodejs/node#9728 I referenced earlier.

  6. sam-github commented on Jun 5, 2019

    @sam-github
    sam-github
    on Jun 5, 2019
    Contributor

    @refack just to be clear, python tools/test.py --report pseudo-tty is how to repro the issue, but its not what we are actually doing in CI? I ask, because if we were doing that, a possible workaround would be to just stop using --report pseudo-tty.

  7. refack commented on Jun 5, 2019

    @refack
    refack
    on Jun 5, 2019
    ContributorAuthor

    @sam-github, yes.
    --report pseudo-tty are two independent args --report is just informative, pseudo-tty is a positional arg telling the harness which suite to run.

    I updated the recipe to:

    > su iojs
> cd /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
> NODE_NAME=test-osuosl-aix61-ppc64_be-3 . build/jenkins/scripts/select-compiler.sh
> python tools/test.py pseudo-tty/console-dumb-tty

  8. refack commented on Jun 5, 2019

    @refack
    refack
    on Jun 5, 2019 · Hidden as off-topic
    Author

  9. sam-github commented on Jun 6, 2019

    @sam-github
    sam-github
    on Jun 6, 2019 · Hidden as off-topic

  10. richardlau commented on Jun 6, 2019

    @richardlau
    richardlau
    on Jun 6, 2019
    Member

    https://ci.nodejs.org/computer/test-osuosl-aix61-ppc64_be-2/ (the problematic host) is currently offline (disconnected by @joyeecheung since this morning).

  11. mhdawson commented on Jun 7, 2019

    @mhdawson
    mhdawson
    on Jun 7, 2019
    Member

    @sam-github I guess its needs another look.

  12. bnoordhuis commented on Jun 21, 2019

    @bnoordhuis
    bnoordhuis
    on Jun 21, 2019
    Member

    I'll take this one.

  13. sam-github commented on Jun 21, 2019

    @sam-github
    sam-github
    on Jun 21, 2019
    Contributor

    I'm not sure how to read the following failures, they have the same timed out after 10 minutes message, but they did NOT timeout during a tools/test.py run... So, does that mean the problem is not related to tools/test.py and python and PTYs? Or does it mean there are two problems?

    ppc64/out/Release/obj.target/tools/v8_gypfiles/libv8_libsampler.a /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/tools/v8_gypfiles/libv8_compiler.a /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/tools/v8_gypfiles/libv8_initializers.a -latomic -ldl -lrt -lm -lperfstat
02:10:44 Build timed out (after 10 minutes). Marking the build as failed.
02:10:44 Build was aborted

    ppc64/out/Release/obj.target/tools/v8_gypfiles/libv8_compiler.a /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/tools/v8_gypfiles/libv8_initializers.a -latomic -ldl -lrt -lm -lperfstat
09:15:14 Build timed out (after 10 minutes). Marking the build as failed.
09:15:14 Build was aborted

  14. sam-github commented on Jun 21, 2019

    @sam-github
    sam-github
    on Jun 21, 2019
    Contributor

    I reviewed all the failures on https://ci.nodejs.org/job/node-test-commit-aix/ from 23960 to 23988 and categorized by failure, and by node number (the : # refers to ....-be-#).

    during cc test compiles: Build timed out (after 10 minutes). Marking the build as failed.

    • 23988: 3,
    • 23987: 1
    • 23986: 3
    • 23985: 3
    • 23984: 3
    • 23981: 1
    • 23978: 3

    during js tests: Build timed out (after 10 minutes). Marking the build as failed.

    • 23983: 1
    • 23964: 3
    • 23960: 3

    Those last are possibly related to PTY issues, and I'm leaving them here in this issue.

    Found also

  15. sam-github commented on Jun 24, 2019

    @sam-github
    sam-github
    on Jun 24, 2019
    Contributor

    @bnoordhuis This bug is easily reproduceable, as @refack says, you can do it via ssh:

    ssh test-osuosl-aix61-ppc64_be-1
su - iojs
cd /home/iojs/build/workspace/node-stress-single-test/nodes/aix61-ppc64
export NODE_NAME=test-osuosl-aix61-ppc64_be-1
. ./build/jenkins/scripts/select-compiler.sh
time ./out/Release/node test/pseudo-tty/console-dumb-tty.js
# instantly completes
python tools/test.py -p tap --logfile testx.tap --mode=release pseudo-tty/console-dumb-tty
# hangs indefinitely

    I added your key to the machine, see #1848, and I suspect the build above will sit around for a while (until the next time a node-stress-single-test is run on that machine, which doesn't happen by default).

  16. 32 remaining items

  17. sam-github
    added a commit that references this issue on Jul 25, 2019

    test: specialize OOM check for AIX

    a187f04
  18. Trott
    added a commit that references this issue on Jul 27, 2019

    test: specialize OOM check for AIX

    41fa553
  19. targos
    added a commit that references this issue on Aug 2, 2019

    test: specialize OOM check for AIX

    f6051f9
  20. vikashk1
    added a commit that references this issue on Aug 13, 2019

    Please ignore this msg (#1)

    9af17ad
  21. BethGriggs
    added 3 commits that reference this issue on Sep 19, 2019

    test: skip tests related to CI failures on AIX

    930647d

    test: fix pty test hangs on aix

    7aca9cb

    test: specialize OOM check for AIX

    65e9b0f
  22. sjarmak
    added a commit that references this issue on Jul 27, 2026

    test: fix pty test hangs on aix

    5bed327
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    platform:aix

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions