Blog Archive

Search This Blog

Tuesday, September 14, 2010

Clojure Macro Tutorial (Part III: Syntax Quote)

I am asked to point out that this is part of a series, following the form of the Ring Cycle, although not directly inspired by it. Syntax Quote plays the role of Siegfried in part III.

http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-i-getting.html http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-ii-compiler.html http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-ii-syntax.html

Suggestions for Götterdämmerung will be most welcome.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; Clojure Macro Tutorial Part III: Syntax Quote
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Let's finally look at how we'd go about writing forloop the correct way,
;; using the built in syntax-quote method.

;; The problem is:

;; We want:
'(forloop [i end]
          code)

;; to turn into:
'(let [finish end]
   (loop [i 0]
     (when (< i finish)
       (print i)
       (recur (inc i)))))

;; First Step

;; Just cut and paste the desired code, and backquote it:
;; This gives us a function which will always return the same code.

(defn forloop-fn-1 []
  `(let [finish end]
     (loop [i 0]
       (when (< i finish)
         (print i)
         (recur (inc i))))))

;; What does forloop-fn give us?

(forloop-fn-1)
;;evaluates to:
'(clojure.core/let [user/finish user/end]
                   (clojure.core/loop [user/i 0]
                                      (clojure.core/when (clojure.core/< user/i user/finish)
                                                         (clojure.core/print user/i)
                                                         (recur (clojure.core/inc user/i)))))

;; This has done all the name resolution (the really ugly bit) for us! Otherwise it's just like quote.

;; But if we try evaluating this code, we'll get an error:
;; Can't let qualified name: user/finish

;; The problem is that user/finish isn't a thing that you can put in a let.
;; finish is the local variable in the expanded code that we wanted to use a
;; gensym for. user/finish is a namespaced variable, which can't be let.

;; So we use the auto-gensym feature:
;; We add # to all occurrences of finish, which tells the compiler to use a gensym here.

(defn forloop-fn-2 []
  `(let [finish# end]
     (loop [i 0]
       (when (< i finish#)
         (print i)
         (recur (inc i))))))

;; Again we'll evaluate:
(forloop-fn-2)
;; to get:
'(clojure.core/let [finish__2254__auto__ user/end]
                   (clojure.core/loop [user/i 0]
                                      (clojure.core/when (clojure.core/< user/i finish__2254__auto__)
                                                         (clojure.core/print user/i)
                                                         (recur (clojure.core/inc user/i)))))

;; So now all the occurrences of finish# have been replaced in the generated code with gensym values,
;; in this case finish__2254__auto__

;; But this code still isn't executable.
;; This first problem with the code generated by forloop-fn-2 is that it expects a variable user/end to be defined.

;; But actually, end is one of the things that we want to vary in the generated
;; code.  We'll make it an argument of the macro, and use the unquote operator ~
;; to tell the function that whenever it sees ~end, it should replace it with
;; the argument.

(defn forloop-fn-3 [end]
  `(let [finish# ~end]
     (loop [i 0]
       (when (< i finish#)
         (print i)
         (recur (inc i))))))

;; Now let's evaluate:
(forloop-fn-3 10)
;; to get:
'(clojure.core/let [finish__2276__auto__ 10]
                   (clojure.core/loop [user/i 0]
                                      (clojure.core/when (clojure.core/< user/i finish__2276__auto__)
                                                         (clojure.core/print user/i)
                                                         (recur (clojure.core/inc user/i)))))

;; Looking good so far! If we try to evaluate this code, though, it objects to
;; the fact that user/i doesn't exist.  We can fix that in the same manner as we
;; fixed the problem with end, because the loop variable is, again, one of the
;; things which we want to vary.

(defn forloop-fn-4 [i end]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         (print ~i)
         (recur (inc ~i))))))

(forloop-fn-4 'j 10) 
;; ->
(clojure.core/let [finish__2298__auto__ 10]
                  (clojure.core/loop [j 0]
                                     (clojure.core/when (clojure.core/< j finish__2298__auto__)
                                                        (clojure.core/print j)
                                                        (recur (clojure.core/inc j)))))

;; (Notice that we have to quote j, because forloop-4-fn is a function, so its
;; arguments get evaluated before it is called)

;; And this code is actually executable! Try evaluating it directly, or use eval
;; to evaluate the return value of the function:

(eval (forloop-fn-4 'j 10))
;; 0123456789nil
(eval (forloop-fn-4 'different-loop-variable 15))
;; 01234567891011121314nil

;; So we've got a function that will give us code that will print out different
;; length runs of numbers, using the loop variable of our choice.

;; Of course to make it useful, we've got to make the (print i) bit a variable as well.
;; We could use the unquoting mechanism here too:
(defn forloop-fn-5 [i end code]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         ~code
         (recur (inc ~i))))))

;;Evaluate:
(forloop-fn-5 'i 10 '(print (* i i)))
;;To get:
(clojure.core/let [finish__2335__auto__ 10]
                  (clojure.core/loop [i 0]
                                     (clojure.core/when (clojure.core/< i finish__2335__auto__)
                                                        (print (* i i))
                                                        (recur (clojure.core/inc i)))))
;; Evaluate that:
(eval (forloop-fn-5 'i 10 '(print (* i i))))
;; 0149162536496481nil

;; Can you not sense imminent success?


;; But in fact, forloop would be much more useful if we were allowed an
;; unlimited number of expressions in our loop code, So we make the function
;; accept a variable number of arguments.  Because the 3rd, 4th, 5th etc
;; arguments are now made into the list "code", we need to use ~@, the
;; unquote-splicing operator to insert them without their outer brackets.
(defn forloop-fn-6 [i end & code]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         ~@code
         (recur (inc ~i))))))


(eval (forloop-fn-6 'i 10 '(print i) '(print (* i i))))
;;00112439416525636749864981nil


;; This would be make a perfectly good macro, but we remember that we wanted
;; (forloop [i 10] code) rather than (forloop i 10 code), so we use the
;; destructuring notation for the first two arguments:
(defn forloop-fn-7 [[i end] & code]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         ~@code
         (recur (inc ~i))))))

(eval (forloop-fn-7 '[i 10] '(print i) '(print (* i i))))
;;00112439416525636749864981nil

;; And finally, we change defn to defmacro, so that the compiler knows that when
;; it evaluates forloop-fn-7 it should pass in the arguments unevaluated, and
;; then treat the return value as code and compile it.

;; This allows us to dispense with the quotes on the arguments and the eval

(defmacro forloop [[i end] & code]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         ~@code
         (recur (inc ~i))))))

(forloop [i 10]
         (print i)
         (print (* i i)))

;;00112439416525636749864981nil

;; And we're done.

;; Let's look at the code our macro makes for us:

(macroexpand-1 '(forloop [i 10]
         (print i)
         (print (* i i))))

(clojure.core/let [finish__2442__auto__ 10]
                  (clojure.core/loop [i 0]
                                     (clojure.core/when (clojure.core/< i finish__2442__auto__)
                                                        (print i)
                                                        (print (* i i))
                                                        (recur (clojure.core/inc i)))))

;; All names resolved to the namespaces that they would have resolved to at the
;; time the macro was defined.

;; Gensyms done automatically, with readable silly names.

;; Bombproof, surprisingly straightforward to write, and the finished macro
;; looks awfully like the code it's trying to generate.

;; And that's how you really write a macro in clojure. Hacking it together by
;; hand is error prone for the reasons given above, and much harder.

12 comments:

  1. That's a clear and nice explanation, thank you.

    ReplyDelete
  2. Thank you! I thought I'd made a bit of a botch of it, to be honest.

    I'm now trying to find a way of explaining this without having to wade through part II, since in fact you don't really need to know all that stuff to *use* syntax-quote, just to understand *why* it does what it does.

    ReplyDelete
  3. This is a great series on Clojure macros, syntax quote and the machinery behind the scenes- thank you for taking the time to write it. I especially appreciate your deconstructing of macros to a simple function(with quoted args)+syntax-quote+eval. It removes a lot of the mystery behind them, and great way to debug as you're developing a macro. I'll be using that approach in the future.

    I disagree with your statement about getting rid of part II - it may take a little more time to wade through it but I think it provides some great background on syntax quote.

    Keep on posting- there is some great stuff on your blog!

    ReplyDelete
  4. Superb! I completed your three part tutorial on macros (it was nerve-wrecking at the start!) and now I finally got to write my first (small but useful) macro.
    Hats off, John!

    ReplyDelete
  5. Thank you anonymous. It is this sort of feedback that keeps a chap blogging!

    ReplyDelete
  6. Thanks very much, John. The three parts tutorial do not only give me great knowledge about Clojure's macros but also a great inspiration.

    ReplyDelete
  7. Thanks! This was very helpful and clear.

    ReplyDelete
  8. Excellent introduction, thank you! It could be worth mentioning in part 1 the fact that x# and ~x get explained in part 3?

    ReplyDelete
  9. I touched clojure > 2 years ago, now i finally find a post on its macro system which is clear and beautiful. I LIKE it too much!

    Thank you for your contrib.

    ReplyDelete
  10. Good stuff. If you want to see how your hand coded version in Part II stacks up against the syntax-quote version in Part III, wrap the syntax-quoted expression into a string and send it off to read-string and pprint the results.

    ReplyDelete
  11. Thanks so much for this tutorial, and I personally found the most important part the second with the gnarly deconstruction of exactly what syntax-quote is up to, and why.

    Probably the best resource on macros I have seen.

    Thanks again, it must have been a lot of work.

    ReplyDelete
  12. thanks, finally I've understood macros in clojure. I started to read Part I around half a year ago. Now I've finished part III :)

    ReplyDelete

Followers