Fix Xcode 4.3 w/o CLT and CLT w/o Xcode

[samueljohn]

This request is now about both

• the CLT without Xcode and
• Xcode without the CLT.

Having Xcode 4.3 without the command line tools, it is currently not possible to build anything.
Sadly, downloading the CLT requires a Developer ID (free, but still).
Since everything needed is basically within /Applications/Xcode.app, it would be cool, if homebrew could just use that. And further it is now possible to _only have the CLTs installed without Xcode to save quite some megs.

I made an attempt to set the correct paths. Therefore, I moved the xcrun method to MacOS.xcrun for the indirection. Further, I coded MacOS.macosxsdk to setup all the needed CFLAGS, CPPFLAGS, LDFLAGS, and so on using xcodebuild to query the path to the sdk. This is set up in ENV.setup_build_environment.

I took extra care not to break things on machines that have not the Xcode 4.3.

Most formulae that use ./configure && make work out of the box with this code. Simple Makefiles with hard-coded CFLAGS are somewhat problematic because they don't pick up the new locations of the SDKs.

Still ToDo:

• Update https://github.com/mxcl/homebrew/wiki/Xcode

[mxcl]

This bit should probably go in ENV.setup_build_whatsit. Since that is called at the right time.

[mxcl]

Only call MacOS.macosxsdk to add the paths if the CLT are missing. How to detect that? Missing /usr/bin/cc?

I'd say yes, this is a good enough check for now.

Also you should add an -isystem to CFLAGS so that the Xcode include directory gets treated as a system include directory. This prevents a variety of bugs.

Also not sure if it's there but adding to LDFLAGS is probably necessary.

Great work on this! Very happy to see some progress.

[MikeMcQuaid]

If you can get this working, great!

[samueljohn]

Glad, you are positive about this. I hope we can get this stable quickly because many tiny changes are made which will make it difficult to merge if the main repro diverges too far.

Updates:

• @mxcl added -isystem to CFLAGS
• @mxcl moved the modifications of env variables to ENV.setup_build_environment
• @mxcl Improved the doctor checks as suggested.
• only change variables if CLTs are NOT installed and only on lion. Hoping this ensures other systems are unaffected.
• better handling and fallback for macosx_sdk_path

I'll try to build more and more formulae with this setup and when it fails I build on my newly lion-ized MacMini where I have only the CLT package but no Xcode. So I can tell if a build fails because of my changes.
I hope there are not too many Makefiles with hardcoded "/usr/bin/something" :-(

[mxcl]

Great, give it some more testing (sadly I cannot help, need another mac). If it seems to be working we'll merge (after I review a little more carefully) and add a brew doctor warning stating that this setup is not that well tested and badness may occur.

Even once we have more user-testing for it we will probably leave the warning (less severe) because I bet a lot of build systems out there will be very surprised that build tools aren't in /usr/something. Though maybe time will prove me wrong on this.

[2bits]

What is the general purpose of this? I'm sorry to ask that simple question, but my confusion is this:
Is it to disassociate Homebrew from xcode commands, or is it to use Homebrew with XCode-4.3 but no CLT?

[samueljohn]

@mxcl brew doctor tells that this is "Experimental" in commit cf3893d, if that is ok for you. Suggestions welcome. Give me a bit more time to test this.

@2bits Your question is valid. Let me answer it.

• If it is possible to build with Xcode only, why should we not make use of it?
• Installing Xcode is easy, but having to tell people to go to the prefs, click there, then click through the Developer ID setup etc. is much more tedious for people just brewing a beer once in a while.
• A Developer ID ist not just an Apple ID. Of course, you can turn an Apple ID into a Developer ID for free. I just went through that process with a fresh Apple ID. I don't say it's complicated. You have to answer a few questions and click through five webpages, then click on the link in a mail, log in again and navigate to the "downloads" section look through the lists of downloads, pick the CLT and you're done.
• I need Xcode but want to spare space on my SSD drive and basically everything in the CLT is duplicated (I checked). I hope Apple will make the CLT available as a normal download in the future (once they get more feedback?) or integrate it into the basic system. I don't know how the chances are.

[samueljohn]

@2bits hey, you changed your question while I was answering :-)
Yes, it's an attempt to make Xcode 4.3+ work without the CTLs. Don't get me wrong. I love the CLTs and have them installed on my second Mac! It's a good move of Apple. They had to make Xcode a normal App Store citizen.

[2bits]

I often think omg i did not just post that. So now I understand more. It sounds fun, like a treasure hunt.

[samueljohn]

Status update:

• what builds successfully (I have not tested all in homebrew just a small subset for my own working env):

apiextractor        hdf5            portaudio spidermonkey  xz
bash-completion libtiff         python     sqlite              zeromq
cmake               libusb       qt            ssh-copy-id
gdbm                 nspr         readline   szip
git                     pkg-config rlog         wget

• what fails right now, but can be fixed with simple patches in the formulae, is often something like missing ENV.x11 (e.g. pixman) or hardcoded paths to SDKs like for wxmac: ENV.append_to_cflags '-isysroot /Developer/SDKs/MacOSX10.6.sdk. These things can be fixed quite easily.
• Some formulae have bad Makefiles. For example multimarkdown, sip/pyqt (I'll have to look into).

Overall it looks pretty promising I would say.
I have a second machine setup only with the CLTs but no Xcode, so I can test two set ups: Xcode without CLT and CLT without Xcode. However, someone will have to test my stuff on Xcode 4.2 and on 10.6, too. I cannot do that :-(

[2bits]

graphviz and valgrind are oddballs you might like to test with. Great news, your success so far.

[samueljohn]

@2bits hmm http://trac.macports.org seems down right now. Cannot test graphviz because patches are not loading.

[samueljohn]

@2bits graphviz and valgrind both fail even on my test system that has got the CLT installed. (E.g. there seem to be hardcoded /Developer paths in there, which are not there any longer on 10.7.3 with the CLTs.

for valgrind I filed an (upstream) issue #10591.

[samueljohn]

So I am pretty happy with the functionality right now. It works for both Xcode 4.3 without CLTs and for CLTs without Xcode 4.3. The --config and doctor can handle both cases and prints useful information about the experimental state.

I hope I did'nt break anything for older Xcode and the default /Developer dir.

Formulae that did not build are mostly due to hardcoded paths in the formula or build scripts and we cannot do much about this besides patching or tweaking the specific formula.

Is there anybody with an older system who could test brew doctor, brew --config, and brewing some things in general?

After that, I think this stuff is ready for squash and code-review by the core devs.
In the meantime I am testing more and more formulae.

[samueljohn]

updated title and description to reflect the fact that this now also handles the case of CLTs w/o Xcode.

[samueljohn]

ok, what fails right now:

• brew install cmake --fresh because cmake ignores SDKROOT and all other std. ENV vars.
• brew python because python links against /usr/lib/libpython. However, brew install python --framework is fine!

[samueljohn]

haha, on a fresh Lion with Xcode (no CLT) /usr/X11/lib/pkgconfig/freetype2.pc reports the includes to be /usr/X11/include but in fact there are no includes (only bitmaps in there). I'll have to teach the tools (e.g. cmake) to pick up #{sdk}/usr/X11/include/X11 ... hmmm I guess I could need help on this one :-/

[2bits]

Those X11 includes can be merciless. Here is what I see with the CLT only.

$ ls -l /usr/X11/include
total 32
drwxr-xr-x  24 root  wheel   816 Feb 17 03:53 GL
drwxr-xr-x   6 root  wheel   204 Feb 17 03:53 VG
drwxr-xr-x  87 root  wheel  2958 Feb 17 03:53 X11
drwxr-xr-x  16 root  wheel   544 Feb 17 03:53 cairo
drwxr-xr-x   5 root  wheel   170 Feb 17 03:53 fontconfig
drwxr-xr-x   3 root  wheel   102 May 25  2011 freetype2
-rw-r--r--   1 root  wheel  3890 Dec  5 14:46 ft2build.h
drwxr-xr-x   5 root  wheel   170 Feb 17 03:53 libpng15
drwxr-xr-x   4 root  wheel   136 Feb 17 03:53 pixman-1
lrwxr-xr-x   1 root  wheel    14 Feb 17 03:53 png.h -> libpng15/png.h
lrwxr-xr-x   1 root  wheel    18 Feb 17 03:53 pngconf.h -> libpng15/pngconf.h
lrwxr-xr-x   1 root  wheel    21 Feb 17 03:53 pnglibconf.h -> libpng15/pnglibconf.h
drwxr-xr-x  41 root  wheel  1394 Feb 17 03:53 xcb

I was interested when you found a brewed python linked to system /usr/lib/libpython. I'm going to brew one and see what otool says for curiosity sake.

[samueljohn]

For the python thing, I'd rather go for --framework always.

I have to rework ENV.x11, too.

[samueljohn]

@2bits Thanks for the list! I have two Macs one with Xcode but no CLTs and one with the CLTs but no Xcode. (both freshly installed with Lion). I modified ENV.x11 to include freetype2 to the cflags. Yet, I am unsure if the others (you listed) should be put there, too. Perhaps it's best to wait and see if something fails because it cannot find , e.g. "fontconfig". Then we explicitly add it.

The problem with python (only non-framework build):

I only have it when I build with this proposed Xcode 4.3 w/o CLT modification (this pull request here). It seems that python without the --framework somehow picks the systems lib and/or config. At least the systems distribute is found and then fails because it's not allowed to write in /Sytem... :

From brew install python

/usr/local/Cellar/python/2.7.2/bin/python setup.py install
Before install bootstrap.
Scanning installed packages
Setuptools installation detected at /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python

Argh! That must be why the gods warned us about dupes!
However, the --framework build of python seems to be "self-contained" and does not have this err. I don't think there is any point in not doing a --framework install as default.

Update on cmake:

I am stuck at the point where ./bootstrap of CMAKE fails to find ZLIB. I tend to blame cmake for not finding it because the other system libs are found (even if they are in the SDK). At least the bottle of cmake works for now (to install other cmake-dependent formula)

[mxcl]

CMake is shit, though @MikeMcQuaid would marry it for some reason.

[MikeMcQuaid]

CMake needs fixed. There's discussion going on here Kitware/CMake#15

I like CMake because it's a cross-platform buildsystem that works pretty much everywhere.

[samueljohn]

@MikeMcQuaid I don't know much about cmake :-/
Perhaps you have an idea how to tell the ./bootstrap to find the zlib?
Using the bottle of cmake seems to work fine.

[2bits]

For finding zlib, did you try ./bootstrap --help ?

[samueljohn]

I tried, yes @2bits but there are no options other than using built-in zlib.

[2bits]

Also you can adjust the FindZLIB.cmake to add some printfs. So you can change the ZLIB_ROOT part to read:

# Search ZLIB_ROOT first if it is set.
IF(ZLIB_ROOT)
  MESSAGE(STATUS "I found your ZROOT man!")
  SET(_ZLIB_SEARCH_ROOT PATHS ${ZLIB_ROOT} NO_DEFAULT_PATH)
  LIST(APPEND _ZLIB_SEARCHES _ZLIB_SEARCH_ROOT)
ENDIF()

That combined with ./bootstrap --system-libs or something should trigger that message.

[samueljohn]

A question of style: Do you prefer:

• MacOS.macosx_sdk_path right now as in this pull request
• MacOS.sdk_path

I had the extra "macosx" in there because theoretically you could have MacOS.ios_sdk_path. One day when homebrew is ported to iOS :-P

[mistydemeo]

MacOS implies macosx. ;) Let's use the simpler name and create a method for iOS if it really comes to that.

[samueljohn]

Then let me do that tomorrow. Don't pull this today :-)

[samueljohn]

btw. this pull request might solve #12893 and #12380 en passant because on default I add MACOSX_DEPLOYMENT_TARGET to ENV.

[samueljohn]

@mistydemeo I pushed a new (sqaushed) commit with the shorter MacOS.sdk_path.

I added -L#{HOMEBREW_PREFIX}/lib to the LDFLAGS. Always, even if CLT's are installed.
(see #12962 (comment))

[samueljohn]

last (squashed) commit fixes some minor style issues pointed out by @adamv. Thanks.

[samueljohn]

What will change for people with Xcode and CLT (so the classic setup) with my changes is:

• -mmacosx-version-min=#{v} will always be in the CFLAGS. (Apple does this too, as we have seen)
• MACOSX_DEPLOYMENT_TARGET is set to the current VERSION. Unless you explicitly call ENV.macosxsdk 10.6 or something.
• CPPFLAGS will have -isystem #{HOMEBREW_PREFIX}/include in prefer homebrew's stuff over other (duplicates like sqlite).
• LDFLAGS gets prepended "-L#{HOMEBREW_PREFIX}/lib" for the same reason as above.

For the doctor.rb: I'd like to make the Error about the "experimental status" into a warning but that seems not to be possible with the doctor.rb, currently. The question is if we are bold enough to remove these warnings? At least the Xcode-only setup should be sufficient to build everything. The CLT-only version cannot build some xcodebuild-based projects and some frameworks are missing (the headers).

[MikeMcQuaid]

Let's leave it a few weeks and see if it causes any problems before removing the current error/warning. Seems good to me otherwise.

[samueljohn]

So shall it be.

[adamv]

Is there any other testing we want to do before pulling this?

[MikeMcQuaid]

I think it's been tested enough and we need to just pull the lever. As long as we can revert it easily: let's do it.

[samueljohn]

Yay! Thanks all.
And we have tag for xcode-only issues in the tracker.

[xgregious]

Sweet! Outstanding work and many thanks!!!

[MikeMcQuaid]

Congratulations @samueljohn and thanks for all the hard work and patience.

[mistydemeo]

Thanks again for your awesome work, @samueljohn!