Repository navigation
OS X CI strategy roadmap #2188
Description
There are a number of activities going on in parallel with OS X in CI, but its hard to get a sense of the overall strategy. Some decisions have been made, and open issues track the progress on them, but some more decisions are upcoming. @AshCripps and I will keep this up to date.
Notes below are in rough chronological order, some parallelization is posssible:
- 10.15+notarization runing at nearform - this unblocks Catalina issues (ci setup) (notarization) (bug report)
- 10.15 test (and release+notarization maybe?) running in permanent orka hosts at macstadium (some initial steps taken, paused pending Ash completing above)(assumes Orka will donate permanent hosts to us) (tracking issue)
^-- At this point we will have to choose whether to use an Orka or Nearform host for https://ci-release.nodejs.org, because either would work but we build releases only once, on a single machine.
-
Benchmark: at this point, we can see relative build speed of Orka vs Nearform, this information may guide decisions about where we put effort in next (though it is possible "ethereal" hosts will benchmark differently than permanent)
-
POC of Orka Jenkins plugin that runs workers on "ethereal" hosts
- Macstadium provides the plugin, so in theory we just enable it on the job, point it at the images we want it to use for the build job, and it will spin up a container, run the build, and spin down the container. This sounds very promising.
- Q: does it work for us?
- Q: is it all-or-nothing, ie. can we have node-test-commit-osx run some jobs on nearform, others on Orka?
- Q: can we run older OS X (10.12 perhaps) as ethereal build jobs (just for test, not release)?
^--- If its one or the other, we may have to choose, and might even stand down the Nearform machines. Its not clear whether using a mixture of near-duplicate build infrastructure makes us more robust to failure, or just gives us more things to manage that have their own independent failure modes.
^--- At this point, we can consider (if all went well) removing OS X 10.10 and 10.11 test hosts from the older macstadium to Orka ethereal jobs. We will have to consider what we want to move --- what selection of older OS X versions do we want to run tests on? Ccurrently, we run only on 10.10 and 10.11. We have 2 instances of 10.12 test hosts, but no builds run on them, and it doesn't appear OS X of those ancient vintages are very popular stats.
Activity
slohiya commentedon Feb 25, 2020
This is blocking our release. We cannot release the mac version of our app. Any update on when this will be addressed is appreciated.
rvagg commentedon Feb 26, 2020
current, more granular, progress on where we're at right now can probably be followed in nodejs/node#31459
AshCripps commentedon Mar 18, 2020
Tracking issue for setting up ORKA - #2236
BethGriggs commentedon Mar 25, 2020
When manually testing the binary of v10.20.0 on my MacOS Catalina I noticed that I am now unable to run the binary that is extracted from the tar.gz. It looks like Apple may have made the notarization checks more strict? 😨
I downloaded https://nodejs.org/download/rc/v10.20.0-rc.1/node-v10.20.0-rc.1-darwin-x64.tar.gz, extracted the file, and tried to run
./bin/node./cc @rvagg @AshCripps
rvagg commentedon Mar 25, 2020
Yikes, they were going to get more "strict" at the beginning of this month but I didn't know it would mean this. I still don't have Catalina running on my laptop (because of chaos like this!) so I can't check right now, but if you go to Security & Privacy in your settings do you have "App Store and identified developers" checked?
I wonder if you did a
curl|tarstyle install if it would be fine. If not then Homebrew must be having trouble with this, surely. This must be because it's been tagged as coming from the web via Chrome.The plan was to get notarization out in this next 13.x (it's queued, I don't think released yet), then in the next 10.x and 12.x straight after that once we've tested it in a 13.x and made sure there aren't any major screams from users. I think we should probably stick with that but it might mean we should bring forward the next 10.x and 12.x to deal specifically with this in a more timely manner, a bit like pushing out a security release. @nodejs/releasers?
BethGriggs commentedon Mar 25, 2020
App Store and identified developersis checked. This is a new laptop that I haven't particularly played with any settings yet, so quite close to a fresh install.nvmworks fine, so I triedwgetand it doesn't show the warning that way either. The issue only seems to show when I click to download the.tar.gz, odd. I'll keep digging. Another colleague on the same version of MacOS Catalina is seeing different results.AshCripps commentedon Mar 25, 2020
@rvagg do we notorize the tar files aswell or just the pkd? just incase this is going to be the new norm for apple to require every zipped folder to come from a "trusted source"
BethGriggs commentedon Mar 25, 2020
I'm guessing this shouldn't hold up nodejs/node#31984, as we are already aware of notarization issues and there are various work arounds.
Timing of upcoming releases is due to be discussed in the Release WG meeting tomorrow (nodejs/Release#554), I'll bring getting notarized binaries out into that discussion.
richardlau commentedon Mar 25, 2020
@BethGriggs What we could do is put a note in the release post that it is a known issue that will be addressed in a future release?
slohiya commentedon Mar 25, 2020
We also use the tar.gz package of nodejs in our application that gets installed on mac. Before this proposed notarization change we were not able to notarize as we were getting errors during notarization on the nodejs files. I am hoping that after this change we will be able to notarize our app that contains tar.gz. Please confirm. Thanks for all the help.
rvagg commentedon Mar 26, 2020
No, we don't notarize the .tar.gz, and can't. But I would assume that people downloading with their browser are going to be choosing the .pkg, not the .tar.?z file. But that doesn't help in this instance since the .pkg isn't notarized and won't even install.
@slohiya check with a recent nightly build to see if that works: https://nodejs.org/download/nightly/v14.0.0-nightly20200325f4153c2fe7/, our signing on
masterfor those nightly builds includes the necessary entitlements that may be causing your hiccup. The additionalterm-sizebinary that ships with npm is in there too if you're bundling npm with your application as well.v13.12.0 is due to go out any day now, see nodejs/node#32376, and it has all of this stuff in it. When that's released you could put that through your bundling process to see if it's resolved.
slohiya commentedon Mar 26, 2020
@rvagg : I understand tar.gz will not be notarized. As you suspect, we were running into issues notarizing our app that includes content of tar.gz probably because of missing entitlements and term-size issue. I will try the nightly build or 13.X if released and see if that works for me. Thanks for quick response and help on this.
slohiya commentedon Mar 26, 2020
@rvagg : My preliminary testing using the above mentioned nightly build shows that including the contents of the nodejs tar.gz in our app does not give any notarization issue. So I think its looking good. I will just have to wait till this fix is back ported to 12.X (LTS). We will perform some more testing on this. Thank you and others again for your help on this.
slohiya commentedon Mar 27, 2020
@rvagg: Repeated our testing on the 13.12.0 release and notarization looks good. Do you know when this notarization fix back porting to 12.X (LTS) is scheduled? Or where can I track the progress of the back port to 12.X?
rvagg commentedon Mar 30, 2020
Backports for notarization:
I'd expect these to land easily, so far I'm not seeing feedback about the 13.x notarization impacting anything but I'm not well tuned in to where people would report such problems. Barring any problems, the next 10.x and 12.x should be when we see releases:
Both have the 21st of April listed as their next dates. I believe the release team has been discussing timing in light of notarization but I don't know if this will change anything.
Any of the release folks on this thread have anything to add?
richardlau commentedon Jun 27, 2020
The notarization changes happened and we're also using Orka in the CI. @AshCripps Is there anything remaining to do for this issue?
AshCripps commentedon Jun 27, 2020
whats left is removal of the old hosting solution and possibly trialing the jenkins plugin but those can be/are tracked in other issues so closing