Repository navigation
What is the cause of "Build timed out"? #1829
Description
An aix61-ppc64 nightly failed, with no XML test results. Can someone point me to the timeout? I'm a bit confused, I know tools/test.py will timeout a test if it takes too long, but that cause a tap entry indicating the test that failed, what is this extra timeout that occurs and causes no test results to be reported? And what causes it, is it a bug in the test.py tool? Is it a too low jenkins timeout?
See https://ci.nodejs.org/job/node-test-commit-aix/23557/nodes=aix61-ppc64/:
00:12:31 ok 2535 message/vm_dont_display_syntax_error
00:12:31 ---
00:12:31 duration_ms: 0.108
00:12:31 ...
00:22:31 Build timed out (after 10 minutes). Marking the build as failed.
00:22:31 gmake[1]: *** [test-ci] Terminated
00:22:31 gmake[1]: Leaving directory `/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64'
00:22:31 gmake: *** [run-ci] Error 2
00:22:31 Build was aborted
Activity
refack commentedon Jun 7, 2019
This is the symptom of #1820. The test harness stalls while trying to setup the tested process. The timeout is triggered by Jenkins which is configured to kill jobs that show no activity for a certain time. For AIX it's 600s - https://ci.nodejs.org/job/node-test-commit-aix/configure under "Build Environment".
sam-github commentedon Jun 7, 2019
#1820 could be one cause of this, but its not unique to AIX, it just happened to me on Windows. Its entirely possible there are multiple things happening, depending on platform, ending in the same symptom.
refack commentedon Jun 7, 2019
Indeed, I've seen that before, on Windows AFAICT it has a different cause. One of the tested procs spawns a child that does not exit, and thus blocking the python harness (I seem to have misplaced the
cmdof that process that I wrote down, but AFAIR it was not obvious to which test it is related)sam-github commentedon Jun 7, 2019
@refack have you ever seen it locally on Windows, or just CI? I'm wondering where someone interested in fixing this on Windows (if I can find a volunteer) would start?
refack commentedon Jun 7, 2019
Only on CI, and only on Windows2008R2 which BTW is going out of support at the end of the year (also only platform we see nodejs/node#20750). I think that's why those issues are hard to get priority for...
sam-github commentedon Jun 8, 2019
Hard to fix, but pretty frequent, https://ci.nodejs.org/job/node-test-binary-windows-2/1327/COMPILED_BY=vs2017,RUNNER=win2008r2-vs2017,RUN_SUBSET=1/console failed on this, too.
sam-github commentedon Jun 8, 2019
Hm, this happens on lots of platforms, here's a linux one, but this happened during the build phase:
joaocgreis commentedon Jun 18, 2019
On Windows (at least on recent runs) this only happened in https://ci.nodejs.org/computer/test-rackspace-win2008r2-x64-3/. For now I've took it offline, will try to understand what was happening there or recreate that machine.
This seems indeed to be a problem with node processes left running.
parallel\test-child-process-exec-timeoutleaves the child process running,parallel/test-child-process-execsync-maxbufandparallel/test-child-process-execfileare somehow blocked by it (complete when it is killed - but have no node.exe running). Needs more investigation.I've improved the diagnostics script in
node-test-binary-windows-2to show this. If this happens again somewhere else, please log into the machine and backupc:\jenkins_diagnostics.txt.joaocgreis commentedon Jul 16, 2019
This happens in other machines as well. Once nodejs/node#28723 lands I'll bring
test-rackspace-win2008r2-x64-3back online and see if it happens again on Windows.test,win: cleanup exec-timeout processes
test,win: cleanup exec-timeout processes
sam-github commentedon Aug 9, 2019
@nodejs/platform-windows Anyone interested in this?
I originally reported it on AIX, but I think it was a symptom of other failures that are now fixed, like #1847.
The conversation above seems to indicate that there are actual Windows problems, though.
Please ignore this msg (#1)
joaocgreis commentedon Aug 14, 2019
I haven't seen failures on Windows after nodejs/node#28723 landed.
There might still be an issue with the test runner detecting child processes running, but unless someone wants to dig deep into
test.pywe can close this issue. Please reopen if there's something else to do here.test,win: cleanup exec-timeout processes
test,win: cleanup exec-timeout processes