cmd/go,cmd/link: TestScript/build_issue48319 and TestScript/build_plugin_reproducible failing on LUCI gotip-darwin-amd64-longtest builder due to non-reproducible LC_UUID #64947
Comments
prattmic
added
Builders
|
Workaround to get Xcode on a gomote: Note: Depending on which machine you get, the |
|
Might have something to do with code-signing? (But then why aren't those tests failing on the |
|
With Xcode installed, this (thankfully) does reproduce (no pun intended): |
|
Complete recipe: Note: Depending on which machine you get, the |
|
The only differences between a.so and b.so are something near the beginning of the file (still investigating) and the Go Build ID: |
|
Based on the otool output, it looks like this other component is the I don't know MachO very well, but it seems that this is just another build ID... |
|
Huh. See also https://bugs.chromium.org/p/chromium/issues/detail?id=1068970. 😵💫 |
|
The output of |
|
Yeah, looks like the I'm not sure what |
|
Thanks for the reference! Looking at the Running the link step (first line) multiple times, even without changing the path, yields a Diffing the binary shows that the UUID is the only difference (Go Build ID is identical presumably because I'm not running the buildid command). |
|
It doesn't seem to be related to the file paths. cmd/link invokes clang like so: The Even with identical paths each time we get different output. |
|
Maybe we can just set the |
|
Perhaps, but I'd like to better understand what is happening. Plus it seems like some users may want the UUID, as Chrome did. FWIW, the ( Edit: I'm not 100% certain about dsymutil being at fault here, as I can't seem to reproduce the non-reproducibility when running clang + dsymutil manually. |
|
cc @thanm see #64947 (comment) for reproducer instructions |
bcmills
changed the title
bcmills
added
the
compiler/runtime
bcmills
changed the title
|
I spent a little while looking at this. What's weird is that the actual DWARF in the two So basically what seems to be happening is that dsymutil is generating a different uuid each time and embedding it into the go.dwarf file, in spite of the fact that the dwarf is the same, hmm. I will spend a little time digging into the dsymutil source code, maybe I can find out more. |
|
FWIW, the version of Xcode we're installing is 15.0.0. I peeked at the release notes for 15.0.1 and 15.1 and nothing stood out as a fix for this kind of issue, but I'll see if we can get a different version to test. |
|
OK (duh) in fact dsymutil is just faithfully copying the uuid from its input, so the problem here is that clang is generating a different uuid. I'll look into the clang source code instead. |
|
FWIW, it looks like there are more versions of Xcode available to try out, though I haven't tested them:
|
|
OK, I think I am making some progress here. For a while I thought this might be an Running the link with Note the "-o" output, which incorporates the go build dir and the external linker is almost certainly going to hash this section when creating the build ID. Not sure what the best approach is to fix this. Also not sure why we aren't seeing similar problems with the older gomotes (I will spin one up and compare). |
|
@thanm, that sounds very similar to an existing reproducibility workaround here: Perhaps we need to extend that workaround to more build modes, or take a similar approach when running other commands? |
|
Well phooey, I am afraid I've had a Homer Simpson moment here. My gomote expired, and I created a new one, but when I started using the new one I didn't update the PATH setting in my script, so it wasn't picking up the correct version of Go. It looks like with LUCI gomotes the location of GOROOT is slightly different each time: Oh well, a learning experience I suppose. That explains why I was not picking up Cherry's fix (https://go-review.googlesource.com/c/go/+/478196, which extends the workaround that you mention Bryan). Now I'm back to seeing only a difference in the UUID. |
|
One more important bit of info: problem goes away if I build with Looking at the setup we have on our old-style gomotes I see: e.g. command line tools, not a complete Xcode installation. For the new LUCI gomotes we are obviously a full Xcode install, and we're using version 15, which defaults to ld-prime. |
I just tested the most recent one (15.2) and it appears to have the same problem. Hmph. |
|
OK, one more update. I can reproduce the problem with just the C compiler, and what I think must be going on is that the name of the output file is being incorporated into the UUID. If I do: then I see a difference, whereas if I instead do The UUIDs are the same (the only thing different in the second example is that both builds target How would we feel about changing the test in question to target the same filename? Or does the current ld-prime behavior not really meet our criteria for reproducible builds? |
|
Should we add skips for these tests while we figure out how to work around this? |
Probably a good idea. I sent a CL (565376). |
|
Change https://go.dev/cl/565376 mentions this issue: |
Skip two build reproducibility tests (build_issue48319 and build_plugin_reproducible) on Darwin if GO_BUILDER_NAME is set until issue 64947 can be resolved; on the LUCI darwin longtest builder the more contemporary version of Xcode is doing things that are unfriendly to Go's build reproducibility. For #64947. Change-Id: Iebd433ad6dfeb84b6504ae9355231d897d8ae174 Reviewed-on: https://go-review.googlesource.com/c/go/+/565376 Reviewed-by: Cherry Mui <[email protected]> Reviewed-by: Michael Pratt <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]>
|
See previously: |
|
Change https://go.dev/cl/584937 mentions this issue: |
|
Please do. Given there isn't a better fix available, let's do that to get GOOS=darwin builders passing on release branches. Thanks. |
|
@gopherbot please create a backport issue for Go 1.22 |
|
Backport issue(s) opened: #67314 (for 1.22). Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases. |
|
Comment #64947 (comment) suggested trying to set So |
|
Change https://go.dev/cl/584238 mentions this issue: |
It's broken with the latest XCode versions, and is also already disabled on darwin builders. Disable the test to get go test cmd/go working on local builds again. For #64947 Change-Id: I5a4b46cf23cbe887df4903f90b54cd2225f51233 Reviewed-on: https://go-review.googlesource.com/c/go/+/584937 Reviewed-by: Dmitri Shuralyov <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>
…selected buildrepro tests Skip two build reproducibility tests (build_issue48319 and build_plugin_reproducible) on Darwin if GO_BUILDER_NAME is set until issue 64947 can be resolved; on the LUCI darwin longtest builder the more contemporary version of Xcode is doing things that are unfriendly to Go's build reproducibility. For #64947. Fixes #67314 Change-Id: Iebd433ad6dfeb84b6504ae9355231d897d8ae174 Reviewed-on: https://go-review.googlesource.com/c/go/+/565376 Reviewed-by: Cherry Mui <[email protected]> Reviewed-by: Michael Pratt <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]> (cherry picked from commit 53708d8) Reviewed-on: https://go-review.googlesource.com/c/go/+/584238 Reviewed-by: Than McIntosh <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]> TryBot-Bypass: Dmitri Shuralyov <[email protected]>
|
Change https://go.dev/cl/585356 mentions this issue: |
|
Change https://go.dev/cl/585355 mentions this issue: |
This patch changes the Go linker to "clean" (reset to Unix epoch) the timestamps on object files copied to the tmpdir that is presented to the external linker or archive tool. The intent is to improve build reproducibility on Darwin, where later versions of xcode seem to want to incorporate object file timestamps into the hash used for the final build ID (which precludes the possibility of having reproducible Go builds). Credit for this idea goes to Cherry (see #64947 (comment)). Updates #64947. Change-Id: I2eb7dddff538e247122b04fdcf8a57c923f61201 Reviewed-on: https://go-review.googlesource.com/c/go/+/585355 Reviewed-by: Cherry Mui <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]>
|
OK, latest go-round: following CL 585355 if I run this sequence of commands (essentially the guts of the plugin repro test) on the I can execute this 50, 100, 200 times and it works every time. If I remove the So: working theory: external linker seems to sometimes (not always, who knows why) take into account the path to the object files being fed it when it generates the UUID. At this point I am now leaning towards falling back on Cherry's second suggestions: run the external linker and then just stomp on the uuid in the generated binary. |
|
Change https://go.dev/cl/586079 mentions this issue: |
When building Go binaries using external linking, rewrite the LC_UUID Macho load command to replace the content placed there by the external linker, so as to ensure that we get reproducible builds. Updates #64947. Change-Id: I263a89d1a067807404febbc801d4dade33bc3288 Reviewed-on: https://go-review.googlesource.com/c/go/+/586079 LUCI-TryBot-Result: Go LUCI <[email protected]> Reviewed-by: Cherry Mui <[email protected]>
github-project-automation
bot
moved this from In Progress
to Done
in Go Compiler / Runtime
dmitshur
added
NeedsFix
(It is unclear to me if this is an issue with the test, cmd/go, the compiler/linker, or the builder itself)
Example failure: https://ci.chromium.org/ui/inv/build-8759926960216361809/test-results?sortby=&groupby=
Both tests are failing because they aren't getting a reproducible build.
I haven't yet been able to reproduce on a gomote because the LUCI gomote setup doesn't currently set up Xcode properly, so cgo doesn't work (which these tests require).
cc @bcmills @dmitshur @mknyszek @cagedmantis
The text was updated successfully, but these errors were encountered: