Reverse-engineering J2CL–Bazel integration
Tags: j2cl, gwt, bazel, java
Published: 2020-07-05 18:20:24

---
title: Reverse-engineering J2CL–Bazel integration
canonical_url: https://blog.ltgt.net/reverse-engineering-j2cl-bazel-integration/
series: gradle-j2cl-plugin
published: true
tags: j2cl,gwt,bazel,java
---

<p>J2CL is a tool by Google that transpiles Java code to Closure-compatible JavaScript.
It’s been started in 2015 to eventually replace GWT at Google,
for various reasons explained in the <a href="https://github.com/google/j2cl#j2cl-vs-gwt">project README</a>,
and will be used as the basis for GWT 3.
As a Google project, it’s deeply integrated with their build tool: <a href="https://bazel.build">Bazel</a>.
We’ll look at how it all works,
with the goal to create an equivalent <a href="https://gradle.org">Gradle</a> plugin.</p>

<p>This reverse-engineering work has already been done several times,
I had done it a couple times a few years ago,
but I don’t think it’s ever be documented though,
so here it is.</p>

<h2 id="starting-points">Starting points</h2>

<p>From a user point of view,
the main starting points are the Bazel rules <code class="language-plaintext highlighter-rouge">j2cl_library</code> and <code class="language-plaintext highlighter-rouge">j2cl_application</code>.</p>

<p>The <code class="language-plaintext highlighter-rouge">j2cl_application</code> rule is actually just a macro around the <a href="https://github.com/bazelbuild/rules_closure"><code class="language-plaintext highlighter-rouge">rules_closure</code></a> rules.
It takes as input <code class="language-plaintext highlighter-rouge">closure_js_library</code> and <code class="language-plaintext highlighter-rouge">j2cl_library</code> dependencies,
and entry point Closure namespaces (that must be present in the dependencies),
and produces optimized JavaScript through a <code class="language-plaintext highlighter-rouge">closure_js_binary</code> rule,
with a handful configuration options.
It also generates a <code class="language-plaintext highlighter-rouge">web_library</code> rule for running your code during development,
we’ll come back to it later.</p>

<p>As we just saw, a <code class="language-plaintext highlighter-rouge">j2cl_application</code> doesn’t directly have sources,
but takes them as dependencies.
All the code of your application would thus actually be in a <code class="language-plaintext highlighter-rouge">j2cl_library</code>.</p>

<p>A <code class="language-plaintext highlighter-rouge">j2cl_library</code> rule is what actually translates your Java source to JS.
It takes Java source files as input
and produces a JAR of compiled Java classes,
and a ZIP of the transpiled JS (known as a JSZip).</p>

<p>Finally, there are rules for tests
(we’ll get to them later on),
and for importing third-party libraries,
including from Maven repositories.</p>

<h2 id="j2cl_library"><code class="language-plaintext highlighter-rouge">j2cl_library</code></h2>

<p>Let’s look closer at what a <code class="language-plaintext highlighter-rouge">j2cl_library</code> does.</p>

<p>A <code class="language-plaintext highlighter-rouge">j2cl_library</code> is somehow both a <code class="language-plaintext highlighter-rouge">java_library</code> and a <code class="language-plaintext highlighter-rouge">closure_js_library</code>.
It takes as input a set of Java and JS source files,
along with dependencies that can be either <code class="language-plaintext highlighter-rouge">closure_js_library</code> or other <code class="language-plaintext highlighter-rouge">j2cl_library</code> rules.
Note that in Bazel, Java code that’s also used in the server for example will <strong>also</strong> be used by a <code class="language-plaintext highlighter-rouge">java_library</code>.</p>

<p>A <code class="language-plaintext highlighter-rouge">j2cl_library</code> will start by removing all the code from the Java source files that’s annotated with <code class="language-plaintext highlighter-rouge">@GwtIncompatible</code>
(technically, it replaces that code with spaces, such that line numbers are preserved).
This stripped code will then be compiled with <code class="language-plaintext highlighter-rouge">javac</code>;
this step could also produce new Java sources through annotation processing
(technically, it could even generate JS files).
The Java and JS source files, and the ones generated during compilation,
will then be passed to J2CL to generate a Closure JS library.</p>

<p>To transpile Java code,
J2CL needs the dependencies as compiled Java classes to resolve the Java APIs
(specifically the method overloads, implicit casts, or even type inference through Java 10’s <code class="language-plaintext highlighter-rouge">var</code> keyword);
those need to be the <em>stripped</em> variants of the dependencies,
which is why <code class="language-plaintext highlighter-rouge">j2cl_library</code> rules depend on other <code class="language-plaintext highlighter-rouge">j2cl_library</code> rules,
and not on <em>standard</em> <code class="language-plaintext highlighter-rouge">java_library</code> rules.
J2CL also resolves <code class="language-plaintext highlighter-rouge">.native.js</code> files sibling to the <code class="language-plaintext highlighter-rouge">.java</code> files,
that generally contains JS code implementing <code class="language-plaintext highlighter-rouge">native</code> methods (somehow equivalent to GWT’s JSNI),
and concatenates their content into the generated <code class="language-plaintext highlighter-rouge">.js</code> files.
It should be noted that the <code class="language-plaintext highlighter-rouge">javac</code> step replaces the bootstrap classpath
with the Java Runtime Emulation library
(which is almost-100% shared with GWT).</p>

<p>So, a <code class="language-plaintext highlighter-rouge">j2cl_library</code> outputs both a JAR like a <code class="language-plaintext highlighter-rouge">java_library</code>,
except the sources have been stripped of any <code class="language-plaintext highlighter-rouge">@GwtIncompatible</code> code first,
and a ZIP of JS files like a <code class="language-plaintext highlighter-rouge">closure_js_library</code>.</p>

<p>Technically, in Bazel, a <code class="language-plaintext highlighter-rouge">j2cl_library</code>, just like a <code class="language-plaintext highlighter-rouge">closure_js_library</code>,
also type-checks the JS (this is probably a bit redundant in J2CL’s case, though thereare the <code class="language-plaintext highlighter-rouge">.native.js</code> files too),
and outputs metadata files about the library
to help speed up downstream Closure JS libraries
(by not rebuilding / re-type-checking them when their upstream API hasn’t changed;
this is similar to how Bazel, and Gradle, extract an API-only JAR/info from Java classes)
and pass diagnostic suppressions down to the Closure compilation.
This is quite specific to Bazel and the <code class="language-plaintext highlighter-rouge">rules_closure</code>,
though could possibly be implemented in Gradle as well
(the way Gradle extracts and checks the API-only info from Java classes
is entirely different from how Bazel does it though:
Gradle apparently fingerprinting a whole classpath of a Java compilation,
whereas Bazel generates an iJAR for each <code class="language-plaintext highlighter-rouge">java_library</code> and then compares those files checksums as any other input).</p>

<p>As we saw, a <code class="language-plaintext highlighter-rouge">j2cl_library</code> leverages three tools:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">GwtIncompatibleStripper</code></li>
  <li><code class="language-plaintext highlighter-rouge">javac</code></li>
  <li>and finally <code class="language-plaintext highlighter-rouge">J2clTranspiler</code></li>
</ul>

<p>And last, but not least, those tools are all run in persistent worker processes,
so you don’t have to spawn a JVM every time
(which with Bazel’s design of having rules almost for each Java package,
would mean a <strong>lot</strong> of times).</p>

<h2 id="importing-jars">Importing JARs</h2>

<p>There are three Bazel rules for importing JARs in J2CL.</p>

<p>The <code class="language-plaintext highlighter-rouge">j2cl_import</code> rule is a simple <em>bridge</em> for when you need a <code class="language-plaintext highlighter-rouge">j2cl_library</code> to depend on a <code class="language-plaintext highlighter-rouge">java_library</code>;
this should only be used for annotation-only JARs though
(J2CL doesn’t need the sources for annotations, and they don’t generate JS code;
they’re only useful to trigger annotation processors,
or configure static analysis tools, such as <a href="https://errorprone.info">ErrorProne</a>).</p>

<p>The <code class="language-plaintext highlighter-rouge">j2cl_import_external</code> rule takes a set of alternative URLs to a JAR (and its SHA256 checksum),
and generates either a <code class="language-plaintext highlighter-rouge">j2cl_import</code> for annotation-only JARs,
or a <code class="language-plaintext highlighter-rouge">j2cl_library</code>.
In the latter case, the JAR should then contain Java sources.
It is actually expected to be GWT library,
with super-sources in <code class="language-plaintext highlighter-rouge">super</code> subpackages.
The <code class="language-plaintext highlighter-rouge">j2cl_library</code> will then use the super-sources
(and ignore the equivalent super-sourced files, that is then GWT-incompatible),
and also ignore any <code class="language-plaintext highlighter-rouge">*_CustomFieldSerializer*</code> file
(as Google doesn’t use GWT-RPC anymore, and thus didn’t port it to J2CL).</p>

<p>Finally, the <code class="language-plaintext highlighter-rouge">j2cl_maven_import_external</code> rule is a wrapper around <code class="language-plaintext highlighter-rouge">j2cl_import_external</code>
simply generating the URLs from Maven coordinates and a set of Maven repository URLs.
It should be noted that this macro uses the sources JAR,
i.e. it replaces the classifier and packaging with <code class="language-plaintext highlighter-rouge">sources</code> and <code class="language-plaintext highlighter-rouge">jar</code> respectively.</p>

<p>Yes, you understand it right:
those last two rules will only consider source files in the JARs
(unless they are annotation-only JARs)
and will therefore strip their <code class="language-plaintext highlighter-rouge">@GwtIncompatible</code> code
and (re)compile them with <code class="language-plaintext highlighter-rouge">javac</code>,
before finally translating them to JS.
It is thus expected that the JARs either include all the source code
(most importantly including the sources generated by annotation processing),
or that, possibly, their dependencies are configured in such a way that
the <code class="language-plaintext highlighter-rouge">javac</code> step will be able to process the annotations and generate the missing files.
GWT libraries would fall in the former bucket,
but their sources JAR as deployed in Maven repositories might not,
so when reproducing this outside Bazel,
we’d possibly make different choices.</p>

<p>Actually, if you look at the J2CL repository,
you’ll see that it has to use a different technique when importing the jbox2d library,
as that one puts super-sources in a <code class="language-plaintext highlighter-rouge">gwtemul</code> subpackage
(and includes GWT-incompatible code in another package).
In this specific case, it grabs the sources from GitHub,
filters out the super-sourced or J2CL-incompatible files,
and then declare a <code class="language-plaintext highlighter-rouge">j2cl_library</code> for those source files
(therefore really rebuilding the library from sources).</p>

<h2 id="j2cl_application"><code class="language-plaintext highlighter-rouge">j2cl_application</code></h2>

<p>As we briefly saw above, this macro is actually only a (rather simple) <em>helper</em> to generate Closure rules,
and you could actually just use the Closure rules directly.
This means that this rule is actually not concerned at all with J2CL,
except for the few configuration options it passes to the Closure compiler.</p>

<p>The inputs to the Closure compiler are the transitive closure of all the <code class="language-plaintext highlighter-rouge">closure_js_library</code>
and the JS output of the <code class="language-plaintext highlighter-rouge">j2cl_library</code> dependencies.
This is the only place where the JS output of the <code class="language-plaintext highlighter-rouge">j2cl_library</code> rules are used;
when a <code class="language-plaintext highlighter-rouge">j2cl_library</code> depends on another <code class="language-plaintext highlighter-rouge">j2cl_library</code>, as we saw above,
it only uses its JAR output.</p>

<p>It’s interesting to look at how users woud run and debug their code though:
the <code class="language-plaintext highlighter-rouge">j2cl_application</code> will generate a JS non-optimized version
(mostly as a <code class="language-plaintext highlighter-rouge">closure_js_binary</code> with different configuration),
and an HTML file that to load it,
and will launch a development server to serve them all.
The HTML page and dev server are also <code class="language-plaintext highlighter-rouge">ibazel</code> and livereload aware,
such that if run through <code class="language-plaintext highlighter-rouge">ibazel</code> the page will <em>livereload</em> whenever a source file is changed.</p>

<p>There are notes in the code about applications with <em>custom dev servers</em>,
one could imagine servers that guard the page behind authentication,
or need to somehow inject dynamic things into the page.
This is left undocumented for now though.
In any case, for those who know <a href="/how-does-gwts-super-dev-mode-work/">how GWT’s super dev mode works</a>,
this is much different, much more like <code class="language-plaintext highlighter-rouge">parcel serve</code> than <code class="language-plaintext highlighter-rouge">parcel watch</code> for those who know JS development with <a href="https://parceljs.org/">Parcel</a>, and without hot module replacement (HMR);
though we do not really know how GWT development works at Google with Bazel,
the <a href="https://github.com/bazelbuild/rules_gwt"><code class="language-plaintext highlighter-rouge">rules_gwt</code></a> being developped and maintained by a Googler whose not in the GWT team.</p>

<h2 id="tests">Tests</h2>

<p>Tests are defined by <code class="language-plaintext highlighter-rouge">j2cl_test</code> rules,
whose implementation has not actually been open sourced (<a href="https://github.com/google/j2cl/issues/93">yet</a>).
Those rules can be generated through a <code class="language-plaintext highlighter-rouge">gen_j2cl_tests</code> rule,
where we can learn a bit more about it,
though we’ll actually learn more by looking at how it’s used in J2CL’s own tests.</p>

<p>Each <code class="language-plaintext highlighter-rouge">j2cl_test</code> rule corresponds to only one test class,
which can be test case or a test suite.
It should be noted that J2CL supports both JUnit 3 and JUnit 4 tests cases,
but only supports JUnit 4 test suites.</p>

<p>Tests can be run in two flavors: <code class="language-plaintext highlighter-rouge">compiled</code> or not;
I <em>suppose</em> it means whether to use optimized or unoptimized Closure output.
And they can be run in several specific browsers,
or, I suppose, against a set of globally-defined ones.</p>

<p>In the J2CL Git repository, we can also find an unused <code class="language-plaintext highlighter-rouge">j2cl_generate_jsunit_suite</code> macro,
which is probably used internally at Google by the <code class="language-plaintext highlighter-rouge">j2cl_test</code> rule.
It takes as input a test class name,
generates a dummy Java file that references it in a <code class="language-plaintext highlighter-rouge">@J2clTestInput</code> annotation,
and compiles it with an annotation processor that will generate support files.
So, the dummy Java file is <strong>only</strong> used to trigger the annotation processor,
but is otherwise completely useless.
The processor will generate
a <code class="language-plaintext highlighter-rouge">test_summary.json</code> file describing the tests,
a JS file for each test case,
and a Java file that needs to be processed by J2CL and is used by the JS files.
Compilation is done through a <code class="language-plaintext highlighter-rouge">j2cl_library</code>,
so the JS files actually have a <code class="language-plaintext highlighter-rouge">.testsuite</code> extension
such that they’re not picked up by J2CL.
Technically, the dummy Java file could be processed using <code class="language-plaintext highlighter-rouge">javac -proc:only</code>,
and then the generate Java file processed by J2CL without needing to be compiled to a Java class first.
In the Bazel macro, the output of the <code class="language-plaintext highlighter-rouge">j2cl_library</code> is then processed
to generate a ZIP containing only JS (with <code class="language-plaintext highlighter-rouge">.testsuite</code> renamed to <code class="language-plaintext highlighter-rouge">.js</code>) and the <code class="language-plaintext highlighter-rouge">test_summary.json</code>.
The <code class="language-plaintext highlighter-rouge">jsunit_test</code> referenced in the macro
is probably similar to the <code class="language-plaintext highlighter-rouge">closure_js_test</code> rule from <code class="language-plaintext highlighter-rouge">rules_closure</code>,
but we can’t really know.</p>

<p>To actually learn more about how testing works with J2CL,
we have to look at the ongoing <a href="https://github.com/Vertispan/j2clmavenplugin"><code class="language-plaintext highlighter-rouge">j2cl-maven-plugin</code></a> effort,
whose dev team asked Google how to actually do it.</p>

<p>In the Maven plugin,
test cases (or test suites) are expected to be annotated with <code class="language-plaintext highlighter-rouge">@J2clTestInput</code>
directly referencing the annotated class
(whereas in Bazel, the annotation is actually entirely kept as an implementation detail).
The plugin will itself compile the classes
(despite them already having been compiled by the <code class="language-plaintext highlighter-rouge">maven-compiler-plugin</code>),
because is will also, as Bazel does, first strip <code class="language-plaintext highlighter-rouge">@GwtIncompatible</code> code,
and add the annotation processor to the compilation process.
It will then read the generated <code class="language-plaintext highlighter-rouge">test_summary.json</code> and, for each test,
copy the generated <code class="language-plaintext highlighter-rouge">.testsuite</code> file to <code class="language-plaintext highlighter-rouge">.js</code>,
run the Closure compiler on it,
similar to a <code class="language-plaintext highlighter-rouge">j2cl_application</code> with that script as the entry point,
generate a simple HTML file loading the resulting script,
and finally load that page in a browser to actually run the tests.
To detect that the tests have finished running,
the plugin will poll the page for specific JS state;
this is actually the same as the <code class="language-plaintext highlighter-rouge">phantomjs_test</code> plumbing in <code class="language-plaintext highlighter-rouge">rules_closure</code>.
AFAICT, the main difference from <code class="language-plaintext highlighter-rouge">rules_closure</code> is that
the HTML page is loaded as a <code class="language-plaintext highlighter-rouge">file://</code> URL in the Maven plugin,
whereas it’s actually served through HTTP in the <code class="language-plaintext highlighter-rouge">rules_closure</code> test harness,
and this can actually have real consequences when dealing with cookies, resources, or HTTP requests.</p>

<h2 id="closing-thoughts">Closing thoughts</h2>

<p>Gradle should have all that’s needed to make it performant to build
a similar tooling as the J2CL–Bazel integration described here:
built-in up-to-date checks for performant incremental builds,
variants (JS vs Java classes for a J2CL library),
worker processes for best performance,
artifact transforms for J2CL’ing external dependencies
(with built-in caching),
continuous builds for rerunning the whole build pipeline on file change,
etc.</p>

<p>I’ll try to design such a Gradle plugin in a following post. Stay tuned!</p>