Skip to content

Frequent ARM failures #1738

Closed
Closed
Frequent ARM failures#1738
Labels
ci-publicincident

Description

@BridgeAR
BridgeAR
opened on Mar 22, 2019

Recently there have been a lot of ARM failures such as: https://ci.nodejs.org/job/node-test-commit-arm/23076/nodes=ubuntu1604-arm64/console

Activity

  1. refack commented on Mar 22, 2019

    @refack
    refack
    on Mar 22, 2019
    Contributor

    It's related to manual testing being done on the machine. We're trying to setup better procedures WRT such usage that should minimize interference.

  2. richardlau commented on Mar 22, 2019

    @richardlau
    richardlau
    on Mar 22, 2019
    Member

    The problem is the following with the clean leftover processes code (even though it's not supposed to error -- possibly because the processes belong to a different user than being used to run the Jenkins job?).

    01:05:17 Clean up any leftover processes but don't error if found.
01:05:17 ps awwx | grep Release/node | grep -v grep | cat
01:05:18 41622 ?        Rl     0:10 out/Release/node --expose-internals /var/tmp/shigeki/node_v8_71/test/parallel/test-heapdump-inspector.js
01:05:18 41643 ?        Rl     0:10 out/Release/node --expose-internals /var/tmp/shigeki/node_v8_71/test/parallel/test-heapdump-zlib.js
01:05:18 41863 ?        Sl     0:09 out/Release/node --expose-internals /var/tmp/shigeki/node_v8_71/test/parallel/test-heapdump-dns.js
01:05:18 41880 ?        Sl     0:06 out/Release/node --expose-internals /var/tmp/shigeki/node_v8_71/test/parallel/test-heapdump-http2.js
01:05:18 41881 ?        Rl     0:08 out/Release/node --expose-internals /var/tmp/shigeki/node_v8_71/test/parallel/test-heapdump-fs-promise.js
01:05:18 41909 ?        Rl     0:07 out/Release/node --expose-internals /var/tmp/shigeki/node_v8_71/test/parallel/test-heapdump-tls.js
01:05:18 41920 ?        Sl     0:08 out/Release/node --expose-internals --experimental-worker /var/tmp/shigeki/node_v8_71/test/parallel/test-heapdump-worker.js
01:05:18 42827 ?        Sl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-client-timeout-option-with-agent.js
01:05:18 43261 ?        Rl     0:01 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-pipeline-flood.js
01:05:18 43289 ?        Rl     0:02 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-pipeline-requests-connection-leak.js
01:05:18 43524 ?        Sl     0:00 /var/tmp/shigeki/node_v8_71/out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-pipeline-flood.js child 44623
01:05:18 43647 ?        Sl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-server-keep-alive-timeout.js
01:05:18 43819 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-response-splitting.js
01:05:18 43830 ?        Sl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-set-timeout-server.js
01:05:18 43840 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-should-keep-alive.js
01:05:18 43851 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-response-statuscode.js
01:05:18 43855 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-status-code.js
01:05:18 43874 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-timeout-overflow.js
01:05:18 43889 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-unix-socket-keep-alive.js
01:05:18 43904 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-outgoing-settimeout.js
01:05:18 43920 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-outgoing-proto.js
01:05:18 43925 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-server-multiheaders2.js
01:05:18 43931 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-upgrade-binary.js
01:05:18 43943 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-upgrade-client.js
01:05:18 43963 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-server-options-server-response.js
01:05:18 43976 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-upgrade-server.js
01:05:18 43995 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-url.parse-auth.js
01:05:18 44005 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-server-reject-cr-no-lf.js
01:05:18 44008 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-url.parse-basic.js
01:05:18 44022 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-server-unconsume-consume.js
01:05:18 44024 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-url.parse-path.js
01:05:18 44029 ?        Rl     0:00 out/Release/node /var/tmp/shigeki/node_v8_71/test/parallel/test-http-server-write-after-end.js
01:05:18 Makefile:448: recipe for target 'clear-stalled' failed
01:05:18 make[1]: *** [clear-stalled] Error 123
01:05:18 Makefile:532: recipe for target 'run-ci' failed

    cc @nodejs/build

  3. refack
    added
    incident
    ci-public
    on Mar 22, 2019

  4. refack commented on Mar 22, 2019

    @refack
    refack
    on Mar 22, 2019
    Contributor

    possibly because the processes belong to a different user

    Yes, those tests are being run right now (under the root user). I guess it's better to fail the cleanup then to succeed, invalidate the manual testing, and probably generated other unexplained flakes...

  5. Trott commented on Mar 22, 2019

    @Trott
    Trott
    on Mar 22, 2019
    Member

    I marked the node offline. When someone is doing manual testing like that, that's probably the thing to do?

  6. refack commented on Mar 22, 2019

    @refack
    refack
    on Mar 22, 2019
    Contributor

    I marked the node offline. When someone is doing manual testing like that, that's probably the thing to do?

    For those who are given access to machines, but don't have "Jenkins/disable" privileges, the thing to do is kill the jenkins daemon.

  7. Trott commented on Mar 22, 2019

    @Trott
    Trott
    on Mar 22, 2019
    Member

    Might want to contact everyone on our list of folks who have shell access and ask that they hop into #node-build on Freenode IRC and get the machine taken offline before doing a bunch of stuff?

  8. refack commented on Mar 22, 2019

    @refack
    refack
    on Mar 22, 2019
    Contributor

    We ask... people forget (me included).

    BTW: list of access requests is radiated by https://github.com/nodejs/build/projects/2

  9. Trott commented on Mar 22, 2019

    @Trott
    Trott
    on Mar 22, 2019
    Member

    list of access requests is radiated

    Nice!

    Since ARM failures seem to have stopped, should we close this?

    There's 228 issues open in this repo. 😱 Might go through and close a few old and inactive ones now....

  10. refack commented on Mar 22, 2019

    @refack
    refack
    on Mar 22, 2019
    Contributor

    I say keep this open till the end of the day, just for visibility, in case someone sees this failure and want to see if it was already reported...

    There's 228 issues open in this repo.

    You can definatly see my influence
    image
    (hay I have a lot of thing I want to look at $LATER)

  11. refack commented on Mar 23, 2019

    @refack
    refack
    on Mar 23, 2019
    Contributor

    Ahhh
    image

  12. refack
    closed this as completedon Mar 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ci-publicincident

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions