Having checked out the source code (to “src”), you can configure and build the JIT library like this:
mkdir build
mkdir install
PREFIX=$(pwd)/install
cd build
../src/configure \
--enable-host-shared \
--enable-languages=jit,c++ \
--disable-bootstrap \
--enable-checking=release \
--prefix=$PREFIX
nice make -j4 # altering the "4" to however many cores you have
This should build a libgccjit.so within jit/build/gcc:
[build] $ file gcc/libgccjit.so*
gcc/libgccjit.so: symbolic link to `libgccjit.so.0'
gcc/libgccjit.so.0: symbolic link to `libgccjit.so.0.0.1'
gcc/libgccjit.so.0.0.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
Here’s what those configuration options mean:
Configuring with this option means that the compiler is built as position-independent code, which incurs a slight performance hit, but it necessary for a shared library.
This specifies which frontends to build. The JIT library looks like a frontend to the rest of the code.
The C++ portion of the JIT test suite requires the C++ frontend to be enabled at configure-time, or you may see errors like this when running the test suite:
xgcc: error: /home/david/jit/src/gcc/testsuite/jit.dg/test-quadratic.cc: C++ compiler not installed on this system
c++: error trying to exec 'cc1plus': execvp: No such file or directory
For hacking on the “jit” subdirectory, performing a full bootstrap can be overkill, since it’s unused by a bootstrap. However, when submitting patches, you should remove this option, to ensure that the compiler can still bootstrap itself.
The compile can perform extensive self-checking as it runs, useful when debugging, but slowing things down.
For maximum speed, configure with --enable-checking=release to disable this self-checking.
[build] $ cd gcc
[gcc] $ make check-jit RUNTESTFLAGS="-v -v -v"
A summary of the tests can then be seen in:
jit/build/gcc/testsuite/jit/jit.sum
and detailed logs in:
jit/build/gcc/testsuite/jit/jit.log
The test executables can be seen as:
jit/build/gcc/testsuite/jit/*.exe
which can be run independently.
You can compile and run individual tests by passing “jit.exp=TESTNAME” to RUNTESTFLAGS e.g.:
[gcc] $ make check-jit RUNTESTFLAGS="-v -v -v jit.exp=test-factorial.c"
and once a test has been compiled, you can debug it directly:
[gcc] $ PATH=.:$PATH \
LD_LIBRARY_PATH=. \
LIBRARY_PATH=. \
gdb --args \
testsuite/jit/test-factorial.c.exe
The jit testsuite detects if RUN_UNDER_VALGRIND is present in the environment (with any value). If it is present, it runs the test client code under valgrind, specifcally, the default memcheck tool with –leak-check=full.
It automatically parses the output from valgrind, injecting XFAIL results if any issues are found, or PASS results if the output is clean. The output is saved to TESTNAME.exe.valgrind.txt.
For example, the following invocation verbosely runs the testcase test-sum-of-squares.c under valgrind, showing an issue:
$ RUN_UNDER_VALGRIND= \
make check-jit \
RUNTESTFLAGS="-v -v -v jit.exp=test-sum-of-squares.c"
(...verbose log contains detailed valgrind errors, if any...)
=== jit Summary ===
# of expected passes 28
# of expected failures 2
$ less testsuite/jit/jit.sum
(...other results...)
XFAIL: jit.dg/test-sum-of-squares.c: test-sum-of-squares.c.exe.valgrind.txt: definitely lost: 8 bytes in 1 blocks
XFAIL: jit.dg/test-sum-of-squares.c: test-sum-of-squares.c.exe.valgrind.txt: unsuppressed errors: 1
(...other results...)
$ less testsuite/jit/test-sum-of-squares.c.exe.valgrind.txt
(...shows full valgrind report for this test case...)
When running under valgrind, it’s best to have configured gcc with --enable-valgrind-annotations, which automatically suppresses various known false positives.
When running client code against a locally-built libgccjit, three environment variables need to be set up:
libgccjit.so is dynamically linked into client code, so if running against a locally-built library, LD_LIBRARY_PATH needs to be set up appropriately. The library can be found within the “gcc” subdirectory of the build tree:
$ file libgccjit.so*
libgccjit.so: symbolic link to `libgccjit.so.0'
libgccjit.so.0: symbolic link to `libgccjit.so.0.0.1'
libgccjit.so.0.0.1: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped
The library uses a driver executable for converting from .s assembler files to .so shared libraries. Specifically, it looks for a name expanded from ${target_noncanonical}-gcc-${gcc_BASEVER}${exeext} such as x86_64-unknown-linux-gnu-gcc-5.0.0.
Hence PATH needs to include a directory where the library can locate this executable.
The executable is normally installed to the installation bindir (e.g. /usr/bin), but a copy is also created within the “gcc” subdirectory of the build tree for running the testsuite, and for ease of development.
The driver executable invokes the linker, and the latter needs to locate support libraries needed by the generated code, or you will see errors like:
ld: cannot find crtbeginS.o: No such file or directory
ld: cannot find -lgcc
ld: cannot find -lgcc_s
Hence if running directly from a locally-built copy (without installing), LIBRARY_PATH needs to contain the “gcc” subdirectory of the build tree.
For example, to run a binary that uses the library against a non-installed build of the library in LIBGCCJIT_BUILD_DIR you need an invocation of the client code like this, to preprend the dir to each of the environment variables:
$ LD_LIBRARY_PATH=$(LIBGCCJIT_BUILD_DIR):$(LD_LIBRARY_PATH) \
PATH=$(LIBGCCJIT_BUILD_DIR):$(PATH) \
LIBRARY_PATH=$(LIBGCCJIT_BUILD_DIR):$(LIBRARY_PATH) \
./jit-hello-world
hello world
libgccjit.c implements the API entrypoints. It performs error checking, then calls into classes of the gcc::jit::recording namespace within jit-recording.c and jit-recording.h.
The gcc::jit::recording classes (within jit-recording.c and jit-recording.h) record the API calls that are made:
/* Indentation indicates inheritance: */ class context; class memento; class string; class location; class type; class function_type; class compound_type; class struct_; class union_; class field; class fields; class function; class block; class rvalue; class lvalue; class local; class global; class param; class statement;
When the context is compiled, the gcc::jit::playback classes (within jit-playback.c and jit-playback.h) replay the API calls within langhook:parse_file:
/* Indentation indicates inheritance: */ class context; class wrapper; class type; class compound_type; class field; class function; class block; class rvalue; class lvalue; class param; class source_file; class source_line; class location;Client Code . Generated . libgccjit.so . code . . . JIT API . JIT "Frontend". (libbackend.a) .................................................................................... │ . . . . ──────────────────────────> . . . . │ . . . . V . . . . ──> libgccjit.c . . . │ (error-checking). . . │ . . . ──> jit-recording.c . . (record API calls) . . <─────── . . . │ . . <─────────────────────────── . . │ . . . . │ . . . . V . . gcc_jit_context_compile . ──────────────────────────> . . . . │ start of recording::context::compile () . . │ . . . . │ start of playback::context::compile () . . │ (create tempdir) . . . │ . . . . │ ACQUIRE MUTEX . . . │ . . . . V───────────────────────> toplev::main (for now) . . . . │ . . . . (various code) . . . . │ . . . . V . . . <───────────────── langhook:parse_file . . . │ . . . . │ (jit_langhook_parse_file) . . . │ . ..........................................│..................VVVVVVVVVVVVV... . . . │ . No GC in here . . . │ jit-playback.c . . . │ (playback of API calls) . . . ───────────────> creation of functions, . . . . types, expression trees . . . <──────────────── etc . . . │(handle_locations: add locations to . . . │ linemap and associate them with trees) . . . │ . . . . │ . No GC in here ..........................................│..................AAAAAAAAAAAAA... . . . │ for each function . . . ──> postprocess . . . │ . . . . ────────────> cgraph_finalize_function . . . <──────────── . . . <── . . . . │ . . . . ──────────────────> (end of . . . . │ langhook_parse_file) . . . . │ . . . . (various code) . . . . │ . . . . ↓ . . . <───────────────── langhook:write_globals . . . │ . . . . │ (jit_langhook_write_globals) . . . │ . . . . │ . . . . ──────────────────> finalize_compilation_unit . . . . │ . . . . (the middle─end and backend) . . . . ↓ . . <───────────────────────────── end of toplev::main . . │ . . . . V───────────────────────> toplev::finalize . . . . │ (purge internal state) . . <──────────────────────── end of toplev::finalize . . │ . . . . V─> playback::context::postprocess: . . │ . . . . │ (assuming an in-memory compile): . . │ . . . . │ . Convert assembler to DSO ("fake.so") . . │ . . . . │ . Load DSO (dlopen "fake.so") . . │ . . . . │ . Bundle it up in a jit::result . . <── . . . . │ . . . . │ RELEASE MUTEX . . . │ . . . . │ end of playback::context::compile () . . │ . . . . │ playback::context dtor . . ──> . . . . │ Normally we cleanup the tempdir here: . . │ ("fake.so" is unlinked from the . . │ filesystem at this point) . . │ If the client code requested debuginfo, the . . │ cleanup happens later (in gcc_jit_result_release) . . │ to make it easier on the debugger (see PR jit/64206) . . <── . . . . │ . . . . │ end of recording::context::compile () <─────────────────────────── . . │ . . . . V . . gcc_jit_result_get_code . ──────────────────────────> . . . . │ dlsym () within loaded DSO <─────────────────────────── . . Get (void*). . . . │ . . . . │ Call it . . . . ───────────────> . . . . │ . . . . │ . . . <─────────────── . . . │ . . . . etc│ . . . . │ . . . . V . . gcc_jit_result_release . ──────────────────────────> . . . . │ dlclose () the loaded DSO . . │ (code becomes uncallable) . . │ . . . . │ If the client code requested debuginfo, then . . │ cleanup of the tempdir was delayed. . . │ If that was the case, clean it up now. <─────────────────────────── . . │ . . . .
Here is a high-level summary from jit-common.h:
In order to allow jit objects to be usable outside of a compile whilst working with the existing structure of GCC’s code the C API is implemented in terms of a gcc::jit::recording::context, which records the calls made to it.
When a gcc_jit_context is compiled, the recording context creates a playback context. The playback context invokes the bulk of the GCC code, and within the “frontend” parsing hook, plays back the recorded API calls, creating GCC tree objects.
So there are two parallel families of classes: those relating to recording, and those relating to playback:
- Visibility: recording objects are exposed back to client code, whereas playback objects are internal to the library.
- Lifetime: recording objects have a lifetime equal to that of the recording context that created them, whereas playback objects only exist within the frontend hook.
- Memory allocation: recording objects are allocated by the recording context, and automatically freed by it when the context is released, whereas playback objects are allocated within the GC heap, and garbage-collected; they can own GC-references.
- Integration with rest of GCC: recording objects are unrelated to the rest of GCC, whereas playback objects are wrappers around “tree” instances. Hence you can’t ask a recording rvalue or lvalue what its type is, whereas you can for a playback rvalue of lvalue (since it can work with the underlying GCC tree nodes).
- Instancing: There can be multiple recording contexts “alive” at once (albeit it only one compiling at once), whereas there can only be one playback context alive at one time (since it interacts with the GC).
Ultimately if GCC could support multiple GC heaps and contexts, and finer-grained initialization, then this recording vs playback distinction could be eliminated.
During a playback, we associate objects from the recording with their counterparts during this playback. For simplicity, we store this within the recording objects, as void *m_playback_obj, casting it to the appropriate playback object subclass. For these casts to make sense, the two class hierarchies need to have the same structure.
Note that the playback objects that m_playback_obj points to are GC-allocated, but the recording objects don’t own references: these associations only exist within a part of the code where the GC doesn’t collect, and are set back to NULL before the GC can run.
Another way to understand the structure of the code is to enable logging, via gcc_jit_context_set_logfile(). Here is an example of a log generated via this call:
JIT: libgccjit (GCC) version 5.0.0 20150123 (experimental) (x86_64-unknown-linux-gnu)
JIT: compiled by GNU C version 4.8.3 20140911 (Red Hat 4.8.3-7), GMP version 5.1.2, MPFR version 3.1.2, MPC version 1.0.1
JIT: entering: gcc_jit_context_set_str_option
JIT: GCC_JIT_STR_OPTION_PROGNAME: "./test-hello-world.c.exe"
JIT: exiting: gcc_jit_context_set_str_option
JIT: entering: gcc_jit_context_set_int_option
JIT: GCC_JIT_INT_OPTION_OPTIMIZATION_LEVEL: 3
JIT: exiting: gcc_jit_context_set_int_option
JIT: entering: gcc_jit_context_set_bool_option
JIT: GCC_JIT_BOOL_OPTION_DEBUGINFO: true
JIT: exiting: gcc_jit_context_set_bool_option
JIT: entering: gcc_jit_context_set_bool_option
JIT: GCC_JIT_BOOL_OPTION_DUMP_INITIAL_TREE: false
JIT: exiting: gcc_jit_context_set_bool_option
JIT: entering: gcc_jit_context_set_bool_option
JIT: GCC_JIT_BOOL_OPTION_DUMP_INITIAL_GIMPLE: false
JIT: exiting: gcc_jit_context_set_bool_option
JIT: entering: gcc_jit_context_set_bool_option
JIT: GCC_JIT_BOOL_OPTION_SELFCHECK_GC: true
JIT: exiting: gcc_jit_context_set_bool_option
JIT: entering: gcc_jit_context_set_bool_option
JIT: GCC_JIT_BOOL_OPTION_DUMP_SUMMARY: false
JIT: exiting: gcc_jit_context_set_bool_option
JIT: entering: gcc_jit_context_get_type
JIT: exiting: gcc_jit_context_get_type
JIT: entering: gcc_jit_context_get_type
JIT: exiting: gcc_jit_context_get_type
JIT: entering: gcc_jit_context_new_param
JIT: exiting: gcc_jit_context_new_param
JIT: entering: gcc_jit_context_new_function
JIT: exiting: gcc_jit_context_new_function
JIT: entering: gcc_jit_context_new_param
JIT: exiting: gcc_jit_context_new_param
JIT: entering: gcc_jit_context_get_type
JIT: exiting: gcc_jit_context_get_type
JIT: entering: gcc_jit_context_new_function
JIT: exiting: gcc_jit_context_new_function
JIT: entering: gcc_jit_context_new_string_literal
JIT: exiting: gcc_jit_context_new_string_literal
JIT: entering: gcc_jit_function_new_block
JIT: exiting: gcc_jit_function_new_block
JIT: entering: gcc_jit_block_add_comment
JIT: exiting: gcc_jit_block_add_comment
JIT: entering: gcc_jit_context_new_call
JIT: exiting: gcc_jit_context_new_call
JIT: entering: gcc_jit_block_add_eval
JIT: exiting: gcc_jit_block_add_eval
JIT: entering: gcc_jit_block_end_with_void_return
JIT: exiting: gcc_jit_block_end_with_void_return
JIT: entering: gcc_jit_context_dump_reproducer_to_file
JIT: entering: void gcc::jit::recording::context::dump_reproducer_to_file(const char*)
JIT: exiting: void gcc::jit::recording::context::dump_reproducer_to_file(const char*)
JIT: exiting: gcc_jit_context_dump_reproducer_to_file
JIT: entering: gcc_jit_context_compile
JIT: in-memory compile of ctxt: 0x1283e20
JIT: entering: gcc::jit::result* gcc::jit::recording::context::compile()
JIT: GCC_JIT_STR_OPTION_PROGNAME: "./test-hello-world.c.exe"
JIT: GCC_JIT_INT_OPTION_OPTIMIZATION_LEVEL: 3
JIT: GCC_JIT_BOOL_OPTION_DEBUGINFO: true
JIT: GCC_JIT_BOOL_OPTION_DUMP_INITIAL_TREE: false
JIT: GCC_JIT_BOOL_OPTION_DUMP_INITIAL_GIMPLE: false
JIT: GCC_JIT_BOOL_OPTION_DUMP_GENERATED_CODE: false
JIT: GCC_JIT_BOOL_OPTION_DUMP_SUMMARY: false
JIT: GCC_JIT_BOOL_OPTION_DUMP_EVERYTHING: false
JIT: GCC_JIT_BOOL_OPTION_SELFCHECK_GC: true
JIT: GCC_JIT_BOOL_OPTION_KEEP_INTERMEDIATES: false
JIT: entering: void gcc::jit::recording::context::validate()
JIT: exiting: void gcc::jit::recording::context::validate()
JIT: entering: gcc::jit::playback::context::context(gcc::jit::recording::context*)
JIT: exiting: gcc::jit::playback::context::context(gcc::jit::recording::context*)
JIT: entering: gcc::jit::playback::compile_to_memory::compile_to_memory(gcc::jit::recording::context*)
JIT: exiting: gcc::jit::playback::compile_to_memory::compile_to_memory(gcc::jit::recording::context*)
JIT: entering: void gcc::jit::playback::context::compile()
JIT: entering: gcc::jit::tempdir::tempdir(gcc::jit::logger*, int)
JIT: exiting: gcc::jit::tempdir::tempdir(gcc::jit::logger*, int)
JIT: entering: bool gcc::jit::tempdir::create()
JIT: m_path_template: /tmp/libgccjit-XXXXXX
JIT: m_path_tempdir: /tmp/libgccjit-CKq1M9
JIT: exiting: bool gcc::jit::tempdir::create()
JIT: entering: void gcc::jit::playback::context::acquire_mutex()
JIT: exiting: void gcc::jit::playback::context::acquire_mutex()
JIT: entering: void gcc::jit::playback::context::make_fake_args(vec<char*>*, const char*, vec<gcc::jit::recording::requested_dump>*)
JIT: reusing cached configure-time options
JIT: configure_time_options[0]: -mtune=generic
JIT: configure_time_options[1]: -march=x86-64
JIT: exiting: void gcc::jit::playback::context::make_fake_args(vec<char*>*, const char*, vec<gcc::jit::recording::requested_dump>*)
JIT: entering: toplev::main
JIT: argv[0]: ./test-hello-world.c.exe
JIT: argv[1]: /tmp/libgccjit-CKq1M9/fake.c
JIT: argv[2]: -fPIC
JIT: argv[3]: -O3
JIT: argv[4]: -g
JIT: argv[5]: -quiet
JIT: argv[6]: --param
JIT: argv[7]: ggc-min-expand=0
JIT: argv[8]: --param
JIT: argv[9]: ggc-min-heapsize=0
JIT: argv[10]: -mtune=generic
JIT: argv[11]: -march=x86-64
JIT: entering: bool jit_langhook_init()
JIT: exiting: bool jit_langhook_init()
JIT: entering: void gcc::jit::playback::context::replay()
JIT: entering: void gcc::jit::recording::context::replay_into(gcc::jit::replayer*)
JIT: exiting: void gcc::jit::recording::context::replay_into(gcc::jit::replayer*)
JIT: entering: void gcc::jit::recording::context::disassociate_from_playback()
JIT: exiting: void gcc::jit::recording::context::disassociate_from_playback()
JIT: entering: void gcc::jit::playback::context::handle_locations()
JIT: exiting: void gcc::jit::playback::context::handle_locations()
JIT: entering: void gcc::jit::playback::function::build_stmt_list()
JIT: exiting: void gcc::jit::playback::function::build_stmt_list()
JIT: entering: void gcc::jit::playback::function::build_stmt_list()
JIT: exiting: void gcc::jit::playback::function::build_stmt_list()
JIT: entering: void gcc::jit::playback::function::postprocess()
JIT: exiting: void gcc::jit::playback::function::postprocess()
JIT: entering: void gcc::jit::playback::function::postprocess()
JIT: exiting: void gcc::jit::playback::function::postprocess()
JIT: exiting: void gcc::jit::playback::context::replay()
JIT: entering: void jit_langhook_write_globals()
JIT: entering: void gcc::jit::playback::context::write_global_decls_1()
JIT: exiting: void gcc::jit::playback::context::write_global_decls_1()
JIT: entering: void gcc::jit::playback::context::write_global_decls_2()
JIT: exiting: void gcc::jit::playback::context::write_global_decls_2()
JIT: exiting: void jit_langhook_write_globals()
JIT: exiting: toplev::main
JIT: entering: void gcc::jit::playback::context::extract_any_requested_dumps(vec<gcc::jit::recording::requested_dump>*)
JIT: exiting: void gcc::jit::playback::context::extract_any_requested_dumps(vec<gcc::jit::recording::requested_dump>*)
JIT: entering: toplev::finalize
JIT: exiting: toplev::finalize
JIT: entering: virtual void gcc::jit::playback::compile_to_memory::postprocess(const char*)
JIT: entering: void gcc::jit::playback::context::convert_to_dso(const char*)
JIT: entering: void gcc::jit::playback::context::invoke_driver(const char*, const char*, const char*, timevar_id_t, bool, bool)
JIT: argv[0]: x86_64-unknown-linux-gnu-gcc-5.0.0
JIT: argv[1]: -shared
JIT: argv[2]: /tmp/libgccjit-CKq1M9/fake.s
JIT: argv[3]: -o
JIT: argv[4]: /tmp/libgccjit-CKq1M9/fake.so
JIT: argv[5]: -fno-use-linker-plugin
JIT: argv[6]: (null)
JIT: exiting: void gcc::jit::playback::context::invoke_driver(const char*, const char*, const char*, timevar_id_t, bool, bool)
JIT: exiting: void gcc::jit::playback::context::convert_to_dso(const char*)
JIT: entering: gcc::jit::result* gcc::jit::playback::context::dlopen_built_dso()
JIT: GCC_JIT_BOOL_OPTION_DEBUGINFO was set: handing over tempdir to jit::result
JIT: entering: gcc::jit::result::result(gcc::jit::logger*, void*, gcc::jit::tempdir*)
JIT: exiting: gcc::jit::result::result(gcc::jit::logger*, void*, gcc::jit::tempdir*)
JIT: exiting: gcc::jit::result* gcc::jit::playback::context::dlopen_built_dso()
JIT: exiting: virtual void gcc::jit::playback::compile_to_memory::postprocess(const char*)
JIT: entering: void gcc::jit::playback::context::release_mutex()
JIT: exiting: void gcc::jit::playback::context::release_mutex()
JIT: exiting: void gcc::jit::playback::context::compile()
JIT: entering: gcc::jit::playback::context::~context()
JIT: exiting: gcc::jit::playback::context::~context()
JIT: exiting: gcc::jit::result* gcc::jit::recording::context::compile()
JIT: gcc_jit_context_compile: returning (gcc_jit_result *)0x12f75d0
JIT: exiting: gcc_jit_context_compile
JIT: entering: gcc_jit_result_get_code
JIT: locating fnname: hello_world
JIT: entering: void* gcc::jit::result::get_code(const char*)
JIT: exiting: void* gcc::jit::result::get_code(const char*)
JIT: gcc_jit_result_get_code: returning (void *)0x7ff6b8cd87f0
JIT: exiting: gcc_jit_result_get_code
JIT: entering: gcc_jit_context_release
JIT: deleting ctxt: 0x1283e20
JIT: entering: gcc::jit::recording::context::~context()
JIT: exiting: gcc::jit::recording::context::~context()
JIT: exiting: gcc_jit_context_release
JIT: entering: gcc_jit_result_release
JIT: deleting result: 0x12f75d0
JIT: entering: virtual gcc::jit::result::~result()
JIT: entering: gcc::jit::tempdir::~tempdir()
JIT: unlinking .s file: /tmp/libgccjit-CKq1M9/fake.s
JIT: unlinking .so file: /tmp/libgccjit-CKq1M9/fake.so
JIT: removing tempdir: /tmp/libgccjit-CKq1M9
JIT: exiting: gcc::jit::tempdir::~tempdir()
JIT: exiting: virtual gcc::jit::result::~result()
JIT: exiting: gcc_jit_result_release
JIT: gcc::jit::logger::~logger()
It should not be possible for client code to cause an internal compiler error. If this does happen, the root cause should be isolated (perhaps using gcc_jit_context_dump_reproducer_to_file()) and the cause should be rejected via additional checking. The checking ideally should be within the libgccjit API entrypoints in libgccjit.c, since this is as close as possible to the error; failing that, a good place is within recording::context::validate () in jit-recording.c.