April 15, 2016

What’s in a namespace?

Every well-behaved clojure source file starts with a namespace declaration. The ns macro, as we all know, is responsible for declaring the namespace to which the definitions in the rest of the file belong, and generally also includes some requirements and imports and whatnot. But today (at Clojure/West, shoutout!) Stuart Sierra made a passing reference to the internals of ns during his talk that got me interested.

But what is a namespace really?

Some things in Clojure are implemented as Java classes of terrifying scope and, for reasons known and important only to the Man himself, using a perplexing and obscure formatting style. But, namespace is not one of those things, which is nice because otherwise the rest of this post would be Java source code and nobody wants to see that. (I’m perfectly comfortable with normal Java code, by the way, it’s just that clojure’s, for probably reasonable reasons, is not that).

No, you came to see some down and dirty Clojure internals, and that’s what you’re going to get. It turns out that ns is just a regular old macro that expends to a bunch of regular old functions that just happen to be how Clojure happens to load code. In other words, we can use our good friend macroexpand to peer (partway) down this particular rabbit hole, which as you might have guessed is what is about to happen. Luckily, clojure.core’s code is about as transparent as the underlying Java is opaque, so everything went better than expected for the purposes of this post.

The result of (macroexpand '(ns namespaces.test)) is the following:

(do
  (clojure.core/in-ns (quote namespaces.test))
  (clojure.core/with-loading-context (clojure.core/refer (quote clojure.core)))
  (if (.equals
        (quote namespaces.test)
        (quote clojure.core))
    nil
    (do (clojure.core/dosync
          (clojure.core/commute
            (clojure.core/deref
              (var clojure.core/*loaded-libs*))
            clojure.core/conj
            (quote namespaces.test))) nil)))

Hey, that’s not so bad! Inside that do, there are 3 things happening:

  • in-ns is called, setting the current value of the ns var in the clojure.core namespace. Stateful! Shame! But, there’s a rich tradition of lisps in general working this way behind it, and it saves adding an extra indentation level to every source file by including it in some with-ns macro, so I think we can let this slide. (Also, judging by the Java code, someone has a vendetta against excess indentation).
  • Inside the with-loading-context macro, we refer the clojure.core namespace. This is good, because we always want the functions in clojure.core to be handy. Also, refer turns out to be the root of all code-loading in clojure; more on this later.
  • Finally, if the namespace in question is not clojure.core, we append its name (as a symbol) to the *loaded-libs* ref, using commute in a dosync transaction as one does when working with refs.

“I hope he’s not about to go into excruciating detail about each of those steps,” I hear you psychically mutter. Too bad for you!

From the top: in-ns

The in-ns is one of those things that Clojure defines in Java. If I wasn’t clear before, I have no intention of delving beyond that barrier, but from context it’s clear that in-ns does what ns would do if ns wasn’t dedicated to encapsulating the half-dozen different things involved in initializing said namespace.

In other words, in-ns is what you would be using if you were also using the require and use functions in your code, instead of the :require and :use (p.s. don’t use use or :use) in ns. In code form, this…

(in-ns 'namespaces.test)
(require 'clojure.string)

… is about equivalent to:

(ns namespaces.test
  (:require clojure.string))

Well, except for all the other stuff that ns does. Incidentally, while sternly recommending against it, the clojure.core internals use in-ns extensively, sometimes spreading a namespace across files. But who are we to judge?

refer madness

The next thing that happens is that the clojure.core namespace is refer’d into the context of the namespace that was just declared the current namespace in in-ns above. This is important because we probably want access to the functions in clojure.core. Ultimately, refer ends up calling the reference method on the *ns* (current namespace) var, which is an instance of clojure.lang.Namespace, which does a voodoo dance that results in the relevant symbols being made available to us.

I didn’t explore too deeply into what the undocumented with-loading-context macro does, accomplishes, because Java, but perhaps it has something to do with how we’re able to use functions defined in clojure.core to load clojure.core. Or perhaps not?

Getting loaded

Finally, the rigamarole surrounding the *loaded-libs* refs is just there so that load-lib doesn’t reload a loaded lib (er, namespace) unless specifically asked to. More about load-lib right now:

What about require and use?

Popping a :require (or :use or :import) into your ns declaration just adds a matching call to require (or use or import) to the with-loading-context part of the namespace macro expansion.

Require and use turn out to be different flavors of the same thing:

(defn require "...docs..." [& args] (apply load-libs :require args))
(defn use "...docs..." [& args] (apply load-libs :require :use args))

load-libs is a collection of checks and whatnot wrapping a bunch of calls to load-lib, which we met above. In turn, load-lib translates the namespace into a path and does some other stuff and then calls load on the path, while load invokes the Java methd clojure.lang.RT/load on it. The eldritch magic contained therein in turn grabs the file, compiles it, and puts all the top-level symbols it found into the current namespace.

The import macro is a bit different, deferring to import*, another magic clojure symbol that’s only defined in Java. The details aren’t clear to me, but I think it’s safe to assume that ultimately this just uses Java reflection to load the class.

Conclusion

So what can we take away from this mutually enlightening experience?

  1. The ns call is actually not so special, until you get to the special parts.
  2. We could manually implement just about everything with in-ns and require directly, but we shouldn’t.
  3. I should talk to someone about my underlying Java phobia.

That’s all for tonight!