Ground rules for when compiling applications from source

Mattias Geniar, Wednesday, November 30, 2011 - last modified: Monday, December 29, 2014

Today I got thinking about a few rules that I should keep when compiling applications from source. It's a small list, I'm looking for any feedback you may have. There's some good ground for discussion here, and depending on which Linux courses you look at, opinions will vary.

For this, I'm also assuming you won't be making RPM packages out of those source compiles to place in a personal repository (which is overkill for 99% of the cases).

Don't do it if it's not necessary

Compiling from source breaks the dependency-lists in yum and aptitude. If possible, don’t try to compile from source but use a package manager to get the packages you need.

Source location & symlinks

  • If there’s already a yum/apt-get package installed of the software you’re trying to install from source, make sure to remove that package first! (unless you have a -very- good reason to keep both versions, you are only making it more confusing for yourself)
  • Place an exclude in /etc/yum.conf to prevent that package from ever being installed via a package manager.
  • Place all the source code in /usr/local/src
  • Extract the software in a directory containing its specific version (see next point)
  • Symlink that specific version to the actual package name to point to the currently active one

For instance:

# ls -alh /usr/local/src/
lrwxrwxrwx  1 root root   10 Nov 30 13:13 php -> php-5.2.14
drwxr-xr-x 18 1002 1002 4.0K Jun 10  2010 php-5.2.13
drwxr-xr-x 19 1005 1005 4.0K Nov 15 09:15 php-5.2.14

That way, you can have all the source code ready in case you need to switch back to a previous version, you use the symlink to point the directory that has now been used to create the current binaries.

  • Always try to install the binaries in /usr/local/bin or /usr/local/sbin, try never to install in /usr/bin or /usr/sbin. That way, it should be more obvious when a specific binary has been installed from yum/apt-get (when it’s in /usr/bin or /usr/sbin) or when it comes from source (everything in /usr/local/*). If necessary, symlink your /usr/local/bin/my-app to /usr/bin/my-app.
  • Place a note in the /etc/motd that this specific piece of software has been self-compiled with a location to the source
  • Always create a "configure_custom" file (chmod +x that) with your configure parameters in your source dir. Even when it’s just a "./configure", place that in a "configure_custom" file so we know that there are no parameters needed.

As a bonus, when you start to run make, consider the concurrency options in make to cut down in your compile times.

I'm curious about any other pointers that are worth keeping when compiling (some of) your applications from source.



Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek, public speaker and podcaster. Currently working on DNS Spy. Follow me on Twitter as @mattiasgeniar.

I respect your privacy and you won't get spam. Ever.
Just a weekly newsletter about Linux and open source.

SysCast podcast

In the SysCast podcast I talk about Linux & open source projects, interview sysadmins or developers and discuss web-related technologies. A show by and for geeks!

cron.weekly newsletter

A weekly newsletter - delivered every Sunday - for Linux sysadmins and open source users. It helps keeps you informed about open source projects, Linux guides & tutorials and the latest news.

Share this post

Did you like this post? Will you help me share it on social media? Thanks!

Comments

Kris Buytaert Saturday, December 3, 2011 at 11:09 (permalink)

You pointed out the most important one .. don’t do it ..
but you missed the second important one

Package your own build ! .. :)

With tools like fpm around there is no excuse to not package your own builds anymore.
If you want to recompile an existing package with different options , modify the spec/rules and rebuild.

Put those builds in your own repository for other machines to be reused.

And throw everything you need to rebuild the package under version control !

Ideally you do these builds on a secondary machine because there are no good excuses to install compilers and stuff on production hosts.

Reply


TomDV Saturday, December 3, 2011 at 20:27 (permalink)

Couldn’t agree more with what Kris already said.
I even seem to recall a few discussions we’ve had on this subject back in the days. ;-)

Packages, packages, packages!

Reply


Matti Sunday, December 4, 2011 at 23:34 (permalink)

Package your own build ! .. :)

There are advantages, but the extra work needed for creating those packages seems like big overkill for most systems. If it’s something as simple as a Varnish/Nginx that needs to be compiled, I see it as a lot of hassle to create RPM packages every time.

It pays of as soon as you’re doing deploying the same package on 2+ systems, but for a locally-needed binary, I don’t do it.

With tools like fpm around there is no excuse to not package your own builds anymore.
If you want to recompile an existing package with different options , modify the spec/rules and rebuild.

Haven’t tried those yet, but working with the default rpmbuild stuff seriously slows things down. Messing with those .spec files makes it even more of a crap-job. :-)

And throw everything you need to rebuild the package under version control !

Amen!

I even seem to recall a few discussions we’ve had on this subject back in the days. ;-)

True. :-)

Reply


Wouter Verhelst Monday, December 5, 2011 at 12:30 (permalink)

Couldn’t agree more with the ‘package it!’ mantra. Once you know how, packaging really isn’t all that hard — and it’s far easier to figure out ‘how did I build that again back in the day’ when all you need to do is ‘dpkg-buildpackage’ or ‘rpmbuild’ with relevant options. Even if considering your ‘configure_custom’ script, which really is a “poor man’s spec or debian/rules file”.

This also makes uninstalling easier later on; it also will make your life a lot easier once the inevitable moment arrives where you have to install another host with the same software at the same version; either because the current machine can’t keep up with the load (and you need a second to do load balancing or whatever), or because you need to replace an old and aging machine with a new one.

Compiling software on production servers is what the old unix dinosaurs do, but it’s silly for everyone else.

Reply


VIncent Van der Kussen Tuesday, December 6, 2011 at 21:49 (permalink)

+1 on the package your stuff statement.

Once you have your spec (Debian seems to call it rules) file, the next time you need to do this (because of an updated version) it will take almost no effort. And you can add the package to your own repository which is also an advantage.
It will also save a lot of time when you need it to be deployed on more than one machine (what about 100 machines ?)

Or maybe you just like the XKCD comic http://xkcd.com/303/

Happy compiling!

Reply


hykosphy Wednesday, December 7, 2011 at 17:15 (permalink)

Why not use Gentoo? It’s made for this!

Reply


J G Miller Sunday, May 24, 2015 at 19:18 (permalink)

My preference is that only software that is built by the distribution should be packaged and installed under /usr. If one does an in-place distribution upgrade, then the versions of self-made packages are likely to cause the upgrade to file because of incompatible version numbers. Even in a non upgrade situation, installing new packages may be broken because of a version change from the distribution package number to your own installed version because of dependencies and checking by the package manager.

For good management of locally installed software under /usr/local, using “stow” (or possible alternatives) is essential. Even managing local data files eg things like wallpapers or additional themes for display managers, desktops, window manager etc, or configuration files for such, is greatly simplified with the use of stow, which with being bundled together in a directory tree, is easy to copy/sync with other machines. And with using stow, removal, or trying out different versions (last stable release vs git/hg/svn version) is so simple.

And with your software installed in a tree of categories with each category or sub-category being a stow directory, it is easy to see at a glance what one has installed locally.

stow works on such a simple mechanism (symbolic links) which some will deride for its lack of coolness or sophistication but it is so easy to use and makes system management of local software including executables, libraries, configuration, and data remarkable easier.

Reply


Leave a Reply

Your email address will not be published. Required fields are marked *