Learning Clojure

Tuesday, September 14, 2010

Clojure Macro Tutorial (Part III: Syntax Quote)

I am asked to point out that this is part of a series, following the form of the Ring Cycle, although not directly inspired by it. Syntax Quote plays the role of Siegfried in part III.

http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-i-getting.html http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-ii-compiler.html http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-ii-syntax.html

Suggestions for GĂ¶tterdĂ¤mmerung will be most welcome.
```;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; Clojure Macro Tutorial Part III: Syntax Quote
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Let's finally look at how we'd go about writing forloop the correct way,
;; using the built in syntax-quote method.

;; The problem is:

;; We want:
'(forloop [i end]
code)

;; to turn into:
'(let [finish end]
(loop [i 0]
(when (< i finish)
(print i)
(recur (inc i)))))

;; First Step

;; Just cut and paste the desired code, and backquote it:
;; This gives us a function which will always return the same code.

(defn forloop-fn-1 []
`(let [finish end]
(loop [i 0]
(when (< i finish)
(print i)
(recur (inc i))))))

;; What does forloop-fn give us?

(forloop-fn-1)
;;evaluates to:
'(clojure.core/let [user/finish user/end]
(clojure.core/loop [user/i 0]
(clojure.core/when (clojure.core/< user/i user/finish)
(clojure.core/print user/i)
(recur (clojure.core/inc user/i)))))

;; This has done all the name resolution (the really ugly bit) for us! Otherwise it's just like quote.

;; But if we try evaluating this code, we'll get an error:
;; Can't let qualified name: user/finish

;; The problem is that user/finish isn't a thing that you can put in a let.
;; finish is the local variable in the expanded code that we wanted to use a
;; gensym for. user/finish is a namespaced variable, which can't be let.

;; So we use the auto-gensym feature:
;; We add # to all occurrences of finish, which tells the compiler to use a gensym here.

(defn forloop-fn-2 []
`(let [finish# end]
(loop [i 0]
(when (< i finish#)
(print i)
(recur (inc i))))))

;; Again we'll evaluate:
(forloop-fn-2)
;; to get:
'(clojure.core/let [finish__2254__auto__ user/end]
(clojure.core/loop [user/i 0]
(clojure.core/when (clojure.core/< user/i finish__2254__auto__)
(clojure.core/print user/i)
(recur (clojure.core/inc user/i)))))

;; So now all the occurrences of finish# have been replaced in the generated code with gensym values,
;; in this case finish__2254__auto__

;; But this code still isn't executable.
;; This first problem with the code generated by forloop-fn-2 is that it expects a variable user/end to be defined.

;; But actually, end is one of the things that we want to vary in the generated
;; code.  We'll make it an argument of the macro, and use the unquote operator ~
;; to tell the function that whenever it sees ~end, it should replace it with
;; the argument.

(defn forloop-fn-3 [end]
`(let [finish# ~end]
(loop [i 0]
(when (< i finish#)
(print i)
(recur (inc i))))))

;; Now let's evaluate:
(forloop-fn-3 10)
;; to get:
'(clojure.core/let [finish__2276__auto__ 10]
(clojure.core/loop [user/i 0]
(clojure.core/when (clojure.core/< user/i finish__2276__auto__)
(clojure.core/print user/i)
(recur (clojure.core/inc user/i)))))

;; Looking good so far! If we try to evaluate this code, though, it objects to
;; the fact that user/i doesn't exist.  We can fix that in the same manner as we
;; fixed the problem with end, because the loop variable is, again, one of the
;; things which we want to vary.

(defn forloop-fn-4 [i end]
`(let [finish# ~end]
(loop [~i 0]
(when (< ~i finish#)
(print ~i)
(recur (inc ~i))))))

(forloop-fn-4 'j 10)
;; ->
(clojure.core/let [finish__2298__auto__ 10]
(clojure.core/loop [j 0]
(clojure.core/when (clojure.core/< j finish__2298__auto__)
(clojure.core/print j)
(recur (clojure.core/inc j)))))

;; (Notice that we have to quote j, because forloop-4-fn is a function, so its
;; arguments get evaluated before it is called)

;; And this code is actually executable! Try evaluating it directly, or use eval
;; to evaluate the return value of the function:

(eval (forloop-fn-4 'j 10))
;; 0123456789nil
(eval (forloop-fn-4 'different-loop-variable 15))
;; 01234567891011121314nil

;; So we've got a function that will give us code that will print out different
;; length runs of numbers, using the loop variable of our choice.

;; Of course to make it useful, we've got to make the (print i) bit a variable as well.
;; We could use the unquoting mechanism here too:
(defn forloop-fn-5 [i end code]
`(let [finish# ~end]
(loop [~i 0]
(when (< ~i finish#)
~code
(recur (inc ~i))))))

;;Evaluate:
(forloop-fn-5 'i 10 '(print (* i i)))
;;To get:
(clojure.core/let [finish__2335__auto__ 10]
(clojure.core/loop [i 0]
(clojure.core/when (clojure.core/< i finish__2335__auto__)
(print (* i i))
(recur (clojure.core/inc i)))))
;; Evaluate that:
(eval (forloop-fn-5 'i 10 '(print (* i i))))
;; 0149162536496481nil

;; Can you not sense imminent success?

;; But in fact, forloop would be much more useful if we were allowed an
;; unlimited number of expressions in our loop code, So we make the function
;; accept a variable number of arguments.  Because the 3rd, 4th, 5th etc
;; arguments are now made into the list "code", we need to use ~@, the
;; unquote-splicing operator to insert them without their outer brackets.
(defn forloop-fn-6 [i end & code]
`(let [finish# ~end]
(loop [~i 0]
(when (< ~i finish#)
~@code
(recur (inc ~i))))))

(eval (forloop-fn-6 'i 10 '(print i) '(print (* i i))))
;;00112439416525636749864981nil

;; This would be make a perfectly good macro, but we remember that we wanted
;; (forloop [i 10] code) rather than (forloop i 10 code), so we use the
;; destructuring notation for the first two arguments:
(defn forloop-fn-7 [[i end] & code]
`(let [finish# ~end]
(loop [~i 0]
(when (< ~i finish#)
~@code
(recur (inc ~i))))))

(eval (forloop-fn-7 '[i 10] '(print i) '(print (* i i))))
;;00112439416525636749864981nil

;; And finally, we change defn to defmacro, so that the compiler knows that when
;; it evaluates forloop-fn-7 it should pass in the arguments unevaluated, and
;; then treat the return value as code and compile it.

;; This allows us to dispense with the quotes on the arguments and the eval

(defmacro forloop [[i end] & code]
`(let [finish# ~end]
(loop [~i 0]
(when (< ~i finish#)
~@code
(recur (inc ~i))))))

(forloop [i 10]
(print i)
(print (* i i)))

;;00112439416525636749864981nil

;; And we're done.

;; Let's look at the code our macro makes for us:

(macroexpand-1 '(forloop [i 10]
(print i)
(print (* i i))))

(clojure.core/let [finish__2442__auto__ 10]
(clojure.core/loop [i 0]
(clojure.core/when (clojure.core/< i finish__2442__auto__)
(print i)
(print (* i i))
(recur (clojure.core/inc i)))))

;; All names resolved to the namespaces that they would have resolved to at the
;; time the macro was defined.

;; Gensyms done automatically, with readable silly names.

;; Bombproof, surprisingly straightforward to write, and the finished macro
;; looks awfully like the code it's trying to generate.

;; And that's how you really write a macro in clojure. Hacking it together by
;; hand is error prone for the reasons given above, and much harder.
```

1. That's a clear and nice explanation, thank you.

2. Thank you! I thought I'd made a bit of a botch of it, to be honest.

I'm now trying to find a way of explaining this without having to wade through part II, since in fact you don't really need to know all that stuff to *use* syntax-quote, just to understand *why* it does what it does.

3. This is a great series on Clojure macros, syntax quote and the machinery behind the scenes- thank you for taking the time to write it. I especially appreciate your deconstructing of macros to a simple function(with quoted args)+syntax-quote+eval. It removes a lot of the mystery behind them, and great way to debug as you're developing a macro. I'll be using that approach in the future.

I disagree with your statement about getting rid of part II - it may take a little more time to wade through it but I think it provides some great background on syntax quote.

Keep on posting- there is some great stuff on your blog!

4. Superb! I completed your three part tutorial on macros (it was nerve-wrecking at the start!) and now I finally got to write my first (small but useful) macro.
Hats off, John!

5. Thank you anonymous. It is this sort of feedback that keeps a chap blogging!

6. Thanks very much, John. The three parts tutorial do not only give me great knowledge about Clojure's macros but also a great inspiration.

7. Thanks! This was very helpful and clear.

8. Excellent introduction, thank you! It could be worth mentioning in part 1 the fact that x# and ~x get explained in part 3?

9. I touched clojure > 2 years ago, now i finally find a post on its macro system which is clear and beautiful. I LIKE it too much!

Thank you for your contrib.

10. Good stuff. If you want to see how your hand coded version in Part II stacks up against the syntax-quote version in Part III, wrap the syntax-quoted expression into a string and send it off to read-string and pprint the results.

11. Thanks so much for this tutorial, and I personally found the most important part the second with the gnarly deconstruction of exactly what syntax-quote is up to, and why.

Probably the best resource on macros I have seen.

Thanks again, it must have been a lot of work.

12. thanks, finally I've understood macros in clojure. I started to read Part I around half a year ago. Now I've finished part III :)