This document describes the new Makefile build system for Unreal 4.0, intended to replace the one used by InspIRCd.
This replaces only the Makefile in src/ (not (currently) src/Modules). The current src/Makefile is pretty much statically generated in configure, and it's a pain in the butt to add new source files in to src/.
This document refers to the new makefile as src/Makefile.new. This is where the makefile is actually being generated (as of svn revision 190) until it has been tested enough to be put to live use. After that it will be src/Makefile. This document will be appropriately updated at that time.
This is a Makefile based off the principles used for src/modules/Makefile. Essentially we scan all the .cpp files in src/ and generate rules that will either generate a .so for that file or ultimately make it part of the unrealircd binary.
The generation process works by reading all the .cpp files. We actually look for any file that ends with '.cpp', '.cxx', '.C', or '.cc' – the generally standard C++ extensions.
When a .cpp file is excluded from the build for any reason, an explanatory note is left src/Makefile.new. Those adding new source files to src/ should check the output in src/Makefile.new to ensure their file is excluded when it should be and not excluded when it shouldn't be. The following are reasons for exclusion:
We first scan the cpp files line by line, looking for lines like this:
/* $<Directive>: <Value> */
Despite the use of C-style multiline comment, this directive must be contained in one line. Probably // should've been used to avoid confusion.
<Directive> must consist only of word characters (letters and underscores, no digits allowed). <Value> is an arbitrary string, but whitespace surrounding <Value> will not be included. Any amount of whitespace (spaces and tabs) are allowed before the $, between the directive and the :, after the : (but before the value), and after the value. Whitespace must not appear between the $ and <Directive>. The following are valid:
/*$Directive:Value*/ /* $Directive : Value */ /* $Directive : Value */
This is not:
/* $ Directive : Value */
Invalid directives are silently ignored, as otherwise perfectly good C++ comments would mess up the build system.
The following directives are recognized. Unknown directives are ignored. (Of particular note to any who make enhancements to the relevant portions of configure, the directive 'Id' must not be handled, due to its interaction with CVS/SVN.)
CompilerFlags: Each CompilerFlags directive adds a set of switches to the command line used to compile the .cpp file into a .o file. All .cpp files are compiled to .o, so this directive is useful in any .cpp file.
LinkerFlags: Each LinkerFlags directive adds a set of switches to the linker used to build the unrealircd binary.
LinkerEndFlags: Each LinkerEndFlags directive adds a set of switches to the linker, at the end of the command line. This is needed for, eg, adding in static libraries.
SharedObject: Only one of SharedObject or Archive may be used. If both are used the file is excluded (see above). The value specifies the name of the shared object (.so) file to generate. This file will not be directly linked to the binary, but instead added as a dependancy of 'all'. More than one .cpp file can specify the same SharedObject value - their respective .o files will be linked into the same .so.
Archive: Only one of SharedObject or Archive may be used. If both are used the file is excluded (see above). The value specifies the name of the static archive (.a) file to generate. This archive file will be directly linked into the binary, and added as a dependecy of the binary. More than one .cpp file can specify the same Archive value - their respective .o files will be added to the same .a.
If neither SharedObject nor Archive are used, the .o file will be staticly linked directly to the unrealircd binary.
TargetFlags: This option adds command line flags to add to the options passed to the tool used to generate a shared object or archive. This is not meaningful if neither of SharedObject or Archive are used. The meaning is slightly different for SharedObjects and Archives.
MakeDir: This option specified a directory, relative to src/, in which we run $(MAKE) prior to gcc when compiling the source file. Note: This needs to be better supported.
IncludeIf: Includes, in a single line, a perl expression. This is evaluated in scope of the makefile generator in ./configure, so such things as %config are available (as is $se, set to the name of the cpp file representing the selected socket engine). Not only must this be in one line, but you must avoid using a */, as this will terminate the value (equally as much as it would terminate the C comment enclosing it). If this expression raises a perl expression or evaluates to anything perl considers false (undef, 0, ””, “0”, the empty list ()), the file is excluded with the expression (and error message, if any) given. Also: remember perl's paradigm of asking nicely not to mess with things, it is generally best not to use the assignment operator anywhere, and remember to use the system function (and not exec) to run external programs!
Each source file's dependencies are calculated by using the preprocessor, namely gcc -MM. If gcc fails, the file is excluded with error output given.
The resulting makefile should look something like this:
all: unrealircd sampleso.so
unrealircd: samplear.a sampleobj.o
$(CC) -pipe -omg_flags <LinkerFlagsGoHere> -o $@ $+ <LinkerEndFlagsGoHere>
sampleso.so: sampleso1.o sampleso2.o
$(CC) -pipe -omg_flags <TargetFlagsGoHere> -o $@ $+
samplear.a: samplear1.o samplear2.o
ar cr $@ $+ <TargetFlagsGoHere>
sampleso1.o: sampleso1.cpp headers.h # <-- It's important that the .cpp comes first, as that's what $< expands to.
$(CC) -pipe -omg_flags <CompilerFlagsGoHere> -o $@ $<
sampleso2.o sampleso2.cpp headers.h
$(CC) -pipe -a_different_flag <CompilerFlagsGoHere> -o $@ $<
samplear1.o: samplear1.cpp headers.h
$(CC) -pipe <CompilerFlagsGoHere> -o $@ $<
samplear2.o samplear2.cpp headers.h
$(CC) -pipe <CompilerFlagsGoHere> -o $@ $<
sampleobj.o: sampleobj.cpp headers.h anotherheader.h
$(CC) -pipe <CompilerFlagsGoHere> -o $@ $<
# The file unsupported.cpp was excluded by request of a false IncludeIf test.
# The test is: $config{SUPPORT_UNSUPPORT} eq "yes";
# The file trytousenonexistantheader.cpp was excluded because it did not pass preprocessing by gcc.
# GCC exited with exit 1, signal 0
# The following is the error output from gcc:
# cc1: nonexistant.h: No such file or directory
# The file invaliddirectives.cpp was excluded because it has both a SharedObject and Archive directive.
# It may only have at most one of either directive. It may not have both.
# The file includedie.cpp was excluded because it has an IncludeIf test that failed in evaluation.
# The failing expression is: die "Sample";
# The following is the error message from perl:
# Sample at includedie.cpp line 21.
The simplest way to do an IncludeIf test based on the result of a program execution is to simply !system(“command line”); although POSIX::WEXITSTATUS(system(“command line”)) == 0 is better for portability. Note that the leading “POSIX::” is required in this case.
As of SVN revision 208, the build system should now be functional (ie, it is possible to build a binary that will successfully start). It is also now accessible from the root Makefile. Take note of the following:
./configure -update needs to be run anytime something in the build system changes. This includes:make clean.make all install, as before. Building under the new system uses the targets all-new, install-new, and for debug builds, debug-new. Thus, make all-new install-new will build and install under the new build system.make clean and ./configure -update. Then use the old targets (make, make debug, or make install - the *-new targets have been removed)../configure -update if and only if the cache file is both readable and has the same modified-time as the source file (not enough to be newer - but when generated the modified-time is set to the source file's time). Note that no cache is left if gcc preprocessing fails, so the next time ./configure -update runs it'll try again. Source files that don't work on certain platforms ought to have $IncludeIf tests (this should avoid that problem).One problem that seems to have been reported in #unreal-devel is:
<nowiki>14:54:39 <@owine> g++ -export-dynamic -o unrealircd -lstdc++ 14:54:39 <@owine> /usr/lib/crt1.o(.text+0x72): In function `_start': 14:54:40 <@owine> : undefined reference to `main' 14:54:41 <@owine> *** Error code 1</nowiki>
This is one possible symptom of an incomplete build - that is, the build system simply didn't include all the proper .cpp files. Note the lack of any .cpp files on the command line, but the real indicator is the 'undefined reference' error from the linker (it doesn't have to be 'main' or the reference from crt1.o, any symbol that comes up is grounds for this possibility). I've also seen the undefined reference appear when using make -jX with X > 1 as a result of Make for some reason trying to link before the last object was compiled. It could also happen that an incomplete, outdated, or otherwise bad object file was left in src but because somehow the modified time is newer Make didn't rebuild it. You could fix that with make clean, or simpler: rm -f src/*.o (the latter should allow you to avoid rebuilding all your modules).
Another possible symptom is that you have no support for some function that should be present, for example:
Before doing anything, make sure you can reproduce this on a clean copy the latest svn. To verify sanity in this regard: svn update && svn revert -R. (If you made any changes you can save them with svn diff.) Alternatively, checkout into a seperate directory.
If you are using make -jX (or your make has -jX by default) where X is anything but “1”, then try to reproduce it using different values of X (especially try -j1, and generally don't exceed the number of CPUs you have + 1 (especially don't try -j with no value unless you want to make your box have a seizure). Be sure you use make clean on each try to have it build from the beginning (like it would on a fresh install). Even if you find one that makes it work, continue onwards, as it's probably worth a bug report anyway.
If you've gotten this far, then at this point you will need to look in src/Makefile for the problem. Look for lines like this:
# The file <somethinghere>.cpp has been excluded ...
Eg, run less src/Makefile and then use the command: /^# The file \w+\.cpp has been excluded. Look above for explanations of why files are excluded. Particularly look at any thing that says it was a result of failing preprocess, as the GCC error output will be included (so if you're just missing a system header, you'll see it and can figure out what you need to install from there).
If you verified that your build environment is correct (especially if you could build prior to SVN 235), file a Bug Report and attach the information detailed below.
If there are no contents of src/Makefile apart from the unrealircd: rule which has no dependencies and the install: rule (such as in the example quoted above), or the only files excluded are the proper socketengine_<thing>.cpp files (eg all the socketengines your OS doesn't support, plus either _select if you used your OS's socket engine or whatever your OS would've supported if you chose to use the select engine), then this is almost certainly a bug and you should report it.
When reporting a buildsystem bug, include the following in your report:
Attach your src/Makefile. If you tried different make -j options and some or all had different results, attach each Makefile (and name it on the bugtracker something like: Makefile-j1, Makefile-j2, according to what you passed to -j) and also describe what effects (apart from what's inside the Makefile) you observed from such changes (were the errors different, from different files, did they go away/multiply?).
Run these shell commands. Replace the '> bug-report… stuff' with '| tee bug-report…' if you want to watch what it's doing.
whereever/unreal4 $ touch bug-report.{config,uname,gcc-ver,cpp-ver,perl-config,unreal4-defines,svn-info,source-list,deps-list,perl-ls}.txt
whereever/unreal4 $ chmod 600 bug-report.{config,uname,gcc-ver,cpp-ver,perl-config,unreal4-defines,svn-info,source-list,deps-list,perl-ls}.txt
whereever/unreal4 $ uname -a 2>&1 > bug-report.uname.txt
whereever/unreal4 $ gcc -v 2>&1 > bug-report.gcc-ver.txt
whereever/unreal4 $ cpp -v /dev/null 2>&1 > bug-report.cpp-ver.txt
whereever/unreal4 $ perl -V 2>&1 > bug-report.perl-config.txt
whereever/unreal4 $ cpp -x c++ $(for FILE in include/*.h; do echo -n "-imacros $FILE "; done; echo) -dM /dev/null 2>&1 > bug-report.unreal4-defines.txt
whereever/unreal4 $ svn info 2>&1 > bug-report.svn-info.txt
whereever/unreal4 $ ls -l src/*.{cpp,cxx,cc,C} 2>&1 > bug-report.source-list.txt
whereever/unreal4 $ ls -l src/*.{cpp,cxx,cc,C}.dep 2>&1 > bug-report.deps-list.txt
whereever/unreal4 $ perl -e 'opendir DD, $ARGV[0]; while (defined($_ = readdir(DD))) { print "$ARGV[0]$_\n"; } closedir DD' src/ 2>&1 > bug-report.perl-ls.txt
whereever/unreal4 $ cp -v .config.cache bug-report.config.txt
whereever/unreal4 $ tar czvf bug-report-files.tar.gz .bug-report.{config,uname,gcc-ver,cpp-ver,perl-config,unreal4-defines,svn-info,source-list,deps-list,perl-ls}.txt
whereever/unreal4 $ rm -v bug-report.{config,uname,gcc-ver,cpp-ver,perl-config,unreal4-defines,svn-info,source-list,deps-list,perl-ls}.txt
Attach bug-report-files.tar.gz to the bug report along with your Makefile(s). Mark the bug private if you're worried about private information.