[TEALIUM SWIFT BUG]: xcodebuild timed out while trying to read ConsentManagerDemo.xcodeproj

[surpher]

Your Environment
Fill out as appropriate.
iOS Version(s) [e.g. 12.1]: 10...13.x
Xcode version [Xcode> About Xcode e.g. 10.1 (10B61)]: 13.1 (11C505)
Swift version [e.g. 4.2]: 5
Tealium Swift library version [e.g. 1.6.7]: 1.9.1
Dependency manager [None/Carthage/CocoaPods]: Carthage
Dependency manager version: 0.33.0

Describe the bug
When running builds on the CI machine the builds consistently fail at building the dependencies due to xcodebuild timed out while trying to read ConsentManagerDemo.xcodeproj.
Even though we are using carthage bootstrap ... --cache-builds, the cache got invalidated and when trying to rebuild, we are hitting this issue.

Does a workaround exist?
Not that we have found one. Also we do not want to check-in the dependencies.

Update: using SPM for Tealium dependency. (We're running two dependency managers for our project. Far from ideal.)

To Reproduce
Steps to reproduce the behavior. If applicable, please attach sample code showing how to reproduce the bug. Remember to keep your Tealium account details private, and only post generic code. Additional code or crash logs may be shared with us privately on [email protected].

1. Setup CI with AWS, RxSwift and Tealium as dependencies
2. Run on Buildkite with latest top spec Mac Mini as an agent
3. Run carthage bootstrap ... (see the log below)
4. Fail at building dependencies

Expected behavior
Dependencies along with Tealium to build without timing out.

Screenshots

FastlaneCore::Interface::FastlaneShellError: [!] Exit status of command 'carthage bootstrap --platform iOS --configuration Release --cache-builds' was 1 instead of 0.
--
  | *** Checking out aws-sdk-ios at "2.9.9"
  | *** Checking out tealium-swift at "1.9.1"
  | *** Checking out Nimble at "v8.0.5"
  | *** Checking out pact-consumer-swift at "c4d86cfaaab53ed23b0946a8106ab23953accc97"
  | *** Checking out RxSwift at "5.0.1"
  | *** Checking out RxDataSources at "4.0.1"
  | *** Checking out TrustKit at "1.6.1"
  | *** No cache found for aws-sdk-ios, building with all downstream dependencies
  | *** No cache found for Nimble, building with all downstream dependencies
  | *** No cache found for RxSwift, building with all downstream dependencies
  | *** No cache found for tealium-carthage-plcrashreporter, building with all downstream dependencies
  | *** No cache found for TrustKit, building with all downstream dependencies
  | *** xcodebuild output can be found in /var/folders/7w/r8bq_3797d921fkclb3k0nhm0000gn/T/carthage-xcodebuild.2MbWJ1.log
  | *** Downloading aws-sdk-ios.framework binary at "AWS SDK for iOS 2.9.9"
  | *** Downloading binary-only framework tealium-carthage-plcrashreporter at "https://tags.tiqcdn.com/dle/tealiummobile/tealium-ios-carthage/tealium-carthage-plcrashreporter.json"
  | *** Downloading RxSwift.framework binary at "ShaiTheBravest"
  | ***  Skipped installing RxSwift.framework binary due to the error:
  | "Incompatible Swift version - framework was built with 5.0.1 (swiftlang-1001.0.82.4 clang-1001.0.46.5) and the local version is 5.1.3 (swiftlang-1100.0.282.1 clang-1100.0.33.15)."
  |  
  | Falling back to building from the source
  | ***  Skipped installing aws-sdk-ios.framework binary due to the error:
  | "Incompatible Swift version - framework was built with 4.2.1 (swiftlang-1000.11.42 clang-1000.11.45.1) and the local version is 5.1.3 (swiftlang-1100.0.282.1 clang-1100.0.33.15)."
  |  
  | Falling back to building from the source
  | *** Building scheme "AWSCloudWatch" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSAPIGateway" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSAutoScaling" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSCognitoIdentityProvider" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSCognito" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSCognitoAuth" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSCognitoIdentityProviderASF" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSComprehend" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSCore" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSDynamoDB" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSEC2" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSIoT" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSElasticLoadBalancing" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSKinesis" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSKinesisVideo" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSKMS" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSLambda" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSKinesisVideoArchivedMedia" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSLex" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSLogs" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSMachineLearning" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSMobileAnalytics" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSPinpoint" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSPolly" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSRekognition" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSS3" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSSES" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSSimpleDB" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSSNS" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSSQS" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSTranscribe" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSTranslate" in AWSiOSSDKv2.xcodeproj
  | *** Building scheme "AWSAuthCore" in AWSAuthSDK.xcodeproj
  | *** Building scheme "AWSAuthUI" in AWSAuthSDK.xcodeproj
  | *** Building scheme "AWSFacebookSignIn" in AWSAuthSDK.xcodeproj
  | *** Building scheme "AWSGoogleSignIn" in AWSAuthSDK.xcodeproj
  | *** Building scheme "AWSMobileClient" in AWSAuthSDK.xcodeproj
  | *** Building scheme "AWSUserPoolsSignIn" in AWSAuthSDK.xcodeproj
  | *** Building scheme "Nimble-iOS" in Nimble.xcodeproj
  | *** Building scheme "PactConsumerSwift iOS" in PactConsumerSwift.xcodeproj
  | *** Building scheme "RxBlocking" in Rx.xcworkspace
  | *** Building scheme "RxRelay" in Rx.xcworkspace
  | *** Building scheme "RxCocoa" in Rx.xcworkspace
  | *** Building scheme "RxSwift" in Rx.xcworkspace
  | *** Building scheme "RxTest" in Rx.xcworkspace
  | *** Building scheme "RxDataSources" in RxDataSources.xcodeproj
  | *** Building scheme "Differentiator" in RxDataSources.xcodeproj
  | xcodebuild timed out while trying to read ConsentManagerDemo.xcodeproj 😭

Additional context
Builds just fine on developer machines running 15" MacBook Pro.

[surpher]

Temporary workaround:

Have opened the ConsentManagerDemo.xcodeproj using Xcode. Xcode complained it needed to be unlocked. Have unlocked the file, closed Xcode, moved to CLI again and ran carthage .... Seems to be progressing now, but that means we will always need to sit next to the build machine on each new branch pipeline, unlock the file in order for the subsequent builds to progress?

UPDATE: Any subsequent clean builds still fail to pass carthage builds.

UPDATE No.2: Using SPM for Tealium seems to be reaping more success than using Carthage.

[craigrouse]

@surpher just confirming that we've seen this and I've also spoken to the support team who are dealing with your ticket. Looking into it and will aim to get you an update soon. Glad you were able to get a temporary workaround with SPM. Carthage should not be building that project, since that's just one of the sample apps... not sure why that's happening in your CICD env. Thanks for all the info you've provided.

[craigrouse]

@surpher Looking into this, I noticed something slightly strange about your environment, which I haven't got to the bottom of - wonder if you can help:

My Xcode version is also 11.3.1, but build 11C504, whereas yours is 11C505. Your build number isn't on the Apple releases page: https://developer.apple.com/news/releases/

This exact issue appears to have happened previously for other SDKs using Carthage, and it seemed it was traced back to an xcode bug. See:

Carthage/Carthage#2104
Carthage/Carthage#2119

I also have a sneaking suspicion (may well be incorrect) that this may be related to a Carthage bug we found recently with binary framework dependencies (we had to use a binary framework dependency from our own repo to deal with some issues in PLCrashReporter... it's a shame Carthage doesn't allow you to exclude schemes to build, as this is an optional module, but Carthage forces you to build all schemes in a project). See: Carthage/Carthage#2940

Thanks for your patience; hopefully we'll get to the bottom of this soon.

[craigrouse]

@surpher I believe we've resolved this now - Carthage didn't like the "SDKROOT" build setting in our project. Could you please confirm if version 1.9.3 resolves your issue? Thanks.

[surpher]

Hi there.

Tried and got the following:

[11:58:09]: Exit status of command 'carthage bootstrap --platform iOS --configuration Debug --cache-builds' was 1 instead of 0.
--
  | *** Cloning tealium-swift
  | *** Checking out RxDataSources at "4.0.1"
  | *** Checking out Nimble at "v8.0.5"
  | *** Checking out tealium-swift at "1.9.3"
  | *** Checking out RxSwift at "5.0.1"
  | *** Checking out TrustKit at "1.6.1"
  | *** Checking out aws-sdk-ios at "2.9.9"
  | *** Checking out pact-consumer-swift at "c4d86cfaaab53ed23b0946a8106ab23953accc97"
  | *** xcodebuild output can be found in /var/folders/d6/kjh3jhh90_sb5q550q9_ldh40000gp/T/carthage-xcodebuild.4ajZrq.log
  | The dependency graph contained a cycle:
  | tealium-swift: plcrashreporter
  | RxDataSources: RxSwift
  | TrustKit:
  | pact-consumer-swift: Nimble
  | Nimble:
  | aws-sdk-ios:
  | RxSwift:
  | tealium-carthage-plcrashreporter:

Edit: I’ll run it again with a bit more focus on what I’m running. Maybe it was still baked in through SPM. Not sure, so I’ll double check.

[surpher]

Not sure about the Xcode build ID:

[craigrouse]

@surpher it definitely builds ok for us. I'm curious about the log message tealium-carthage-plcrashreporter:. This isn't something I recognise. Let us know if you're still having trouble.

[surpher]

Yeah, I'm struggling to find a time to test this properly. It might have been that Tealium was still in the project through SPM? Not sure. Need some more time to figure out how to confirm this properly.

[craigrouse]

No problem @surpher. Whenever you get time. It's possible it was SPM interfering, but hard to say.

[surpher]

Sorry guys but I have had no luck with v1.9.3 using Carthage.

I have even created a brand spanking new (and blank) Xcode iOS project (Xcode 11.3.1), copied the Cartfile.resolved into the new folder and ran carthage bootstrap ....

Running the following carthage command (to avoid unnecessary 30 minute builds of the ridiculously big aws-sdk-ios):

carthage bootstrap --configuration Debug --platform iOS --cache-builds --no-use-binaries

It kept failing due to dependency graph containing a cycle even when I removed each and every other dependency one by one:

> carthage bootstrap --configuration Debug --platform iOS --cache-builds --no-use-binaries                                                                                                                                                                                                                                                                                                                                                5.9s > Tue 31 Mar 00:32:26 2020
*** Checking out tealium-swift at "1.9.3"
*** xcodebuild output can be found in /var/folders/__/g0dqv2n918xcx59w_bdns7th0000gn/T/carthage-xcodebuild.w2YXBp.log
The dependency graph contained a cycle:
tealium-swift: plcrashreporter

Tried just a simple carthage bootstrap, got the same dependency graph cycle error.

Carthage version 0.34.0

In the original project, I have also noticed that even when getting rid of the SPM package (tealium-swift in this case), it still persisted in the Source Control Navigator.

[craigrouse]

@surpher I believe the failing Carthage build is the same problem as on this thread: #128

Should have a fix for this very soon.

[craigrouse]

@surpher Could you possibly try editing your cartfile for me, and add the branch name "1.9.4"?

github "tealium/tealium-swift" "1.9.4"

I was unable to reproduce your issue, running the exact same commands as you, but I believe switching to the binary dependency for the crash reporter module should solve the problem. Once you confirm it fixes your build issue, we'll publish this to master. Thanks for your patience.

[surpher]

Seems to be a-ok now using 1.9.4 🎉

💯

[craigrouse]

@surpher that's great. Sincere apologies that you've had this issue, but hopefully it's now closed once and for all, and we've added some steps to our Carthage testing process to hopefully catch these issues in future. Thanks for bearing with us. We'll get this put up as a full release today.