PPC: test-osuosl-ubuntu1404-ppc64_le-2 has strange linker issues #1815

Activity

1. refack

   added

   incident

   platform:ppc

   on May 30, 2019
2. refack commented on May 30, 2019

   

   refack

   on May 30, 2019

   ContributorAuthor

   More actions

   /CC @mhdawson @sam-github
3. mhdawson commented on Jun 4, 2019

   

   mhdawson

   on Jun 4, 2019

   Member

   More actions

   I've pinged @sam-github to make sure this is on his radar. We were both out last week at the Collab summit.
4. sam-github commented on Jun 4, 2019

   

   sam-github

   on Jun 4, 2019

   Contributor

   More actions

   @miladfarca, you looked at this a bit? Did you have any ideas, or even manage to reproduce this?
5. refack commented on Jun 4, 2019

   

   refack

   on Jun 4, 2019 · Hidden as off-topic

   Author

   show commentMore actions
6. richardlau commented on Jun 4, 2019

   

   richardlau

   on Jun 4, 2019 · Hidden as off-topic

   show commentMore actions
7. miladfarca commented on Jun 4, 2019

   

   miladfarca

   on Jun 4, 2019

   Contributor

   More actions

   @sam-github this is a separate issue I assume, the one I was looking at was related to the /test-worker-prof.js test case.
8. refack commented on Jun 4, 2019

   

   refack

   on Jun 4, 2019 · Hidden as off-topic

   Author

   show commentMore actions
9. sam-github commented on Jun 5, 2019

   

   sam-github

   on Jun 5, 2019

   Contributor

   More actions

   The history here is a bit confusing, because multiple issues are discussed. I hid @refack and @richardlau comments above that are related to #1820, just to make this easier to read.

   @miladfarca, we have discussed this linker error before, in slack:

   16:06:17   g++ -o /home/iojs/build/workspace/node-test-commit-plinux/nodes/centos7-ppcle/out/Release/obj.target/v8_base/deps/v8/src/extensions/statistics-extension.o ../deps/v8/src/extensions/statistics-extension.cc '-DV8_GYP_BUILD' '-DV8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64' '-DOPENSSL_THREADS' '-DV8_TARGET_ARCH_PPC' '-DV8_TARGET_ARCH_PPC64' '-
   Milad Farazmand  12 days ago
   The build that's failing isn't compiling this file, hence a missing symbol error, now look into why this file is not included when you are compiling your code (edited)
   SAM ROBERTS  12 days ago
   interesting. and odd :disappointed:. thanks for looking.
   Milad Farazmand  12 days ago
   no issues,  might be helpful to look into tools/v8_gypfiles/v8.gyp, every machine specific file that needs to be compiled for v8 is listed there
   

   Luckily (?) the linker problems stopped over the weekend, and my build passed and was merged (by someone else): nodejs/node#27834

   But, there seems to be a real issue.

   Reviewing https://ci.nodejs.org/computer/test-osuosl-ubuntu1404-ppc64_le-2/builds, many failed with linker errors (though some with pty timeouts, as in #1820).

   • https://ci.nodejs.org/job/node-test-commit-plinux/nodes=ppcle-ubuntu1404/26123/console: undefined reference to `vtable for v8::internal::StatisticsExtension'
   • https://ci.nodejs.org/job/node-test-commit-plinux/nodes=ppcle-ubuntu1404/26122/console: ditto
   • https://ci.nodejs.org/job/node-test-commit-plinux/nodes=ppcle-ubuntu1404/26119/console
   • ... etc

   I'll go look at the .gyp file that Milad mentions in the workspace, and compare it to a clean build. Since the workers is offline, I should be able to redo the build in the workspace from ssh, and see if it fails to link for me, too.
10. miladfarca commented on Jun 18, 2019

    

    miladfarca

    on Jun 18, 2019

    Last edited by miladfarca

    Contributor

    More actions

    v8::internal::StatisticsExtension::kSource is defined in statistics-extension.cc under V8. Looking at the latest job under this machine (hash cfbb5bb) on May 30, the statistics-extension object file is not compiled properly, this is the output of nm:

    $ nm out/Release/obj.target/v8_base/deps/v8/src/extensions/statistics-extension.o
    
    nm: out/Release/obj.target/v8_base/deps/v8/src/extensions/statistics-extension.o: File truncated
    

    I copied the directory, re configured and compiled it from scratch and it worked fine. Seems like jenkins was interrupted mid compilation and the truncated statistics-extension.o was used in the later test jobs and commits?

    Are we clearing cache/workspace properly before/after each build?
    Is it possible to clear the workspace manually and activate the machine and see if it happens again.
11. sam-github commented on Jun 18, 2019

    

    sam-github

    on Jun 18, 2019

    Last edited by sam-github

    Contributor

    More actions

    Its possible the workspace isn't cleared, but its also possible that a bad build got ccached, which would cause the issue to come back over and over.

    I removed the .o file, cleared the ccache, and reenabled the machine.

    iojs@test-osuosl-ubuntu1404-ppc64--le-2:~/build/workspace$ rm ./node-test-commit-plinux/nodes/ppcle-ubuntu1404/out/Release/obj.target/v8_base/deps/v8/src/extensions/statistics-extension.o
    

    EDIT: I'll leave this open until some builds pass on this machine.
12. sam-github commented on Jun 18, 2019

    

    sam-github

    on Jun 18, 2019

    Contributor

    More actions

    I just got a green build from the daily: https://ci.nodejs.org/computer/test-osuosl-ubuntu1404-ppc64_le-2/builds

    Perhaps this issue is resolved?
13. sam-github commented on Jun 20, 2019

    

    sam-github

    on Jun 20, 2019

    Contributor

    More actions

    https://ci.nodejs.org/computer/test-osuosl-ubuntu1404-ppc64_le-2/builds looks resolved, closing this.
14. sam-github

    closed this as completedon Jun 20, 2019