An Astonishing Macro of Narayan Singhal

Forgive me, for I have stolen a macro from someone else's blog: Here is the original:

 http://clojure101.blogspot.com/2009/04/destructuring-binding-support-in-def.html.

It was so interesting that I had to write an essay in order to understand it, and then I thought, oh what the hell...

;; On Narayan Singhal's blog, I found a truly astonishing macro:

;; Here's a link to Narayan's original post:
;; http://clojure101.blogspot.com/2009/04/destructuring-binding-support-in-def.html

;; The idea of it is that let statements are often difficult to debug.

;; Suppose you are constructing a calculation bit by bit.

;; You will, I hope, forgive the arbitrary and trivial nature of the following
;; example.  I needed a complicated example that is nevertheless easy to
;; understand so that it won't distract from the genius of the idea.

;; You might originally say, take the range from 0 to 9 and map the function
;; x -> 10x over it, to produce a lookup table:
(zipmap (range 10) (map #(* 10 %) (range 10)))

;; And then do the same for the functions x -> 5x^2 and x -> x^3
(zipmap (range 10) (map #(* 5 % %) (range 10)))
(zipmap (range 10) (map #(* % % %) (range 10)))

;; Now say, to construct a map from (range 10) to the minimum of these three functions
(merge-with min
            (zipmap (range 10) (map #(* 10 %) (range 10)))
            (zipmap (range 10) (map #(* 5 % %) (range 10)))
            (zipmap (range 10) (map #(* % % %) (range 10))))

;; And for the maximum, you might say:
(merge-with max
            (zipmap (range 10) (map #(* 10 %) (range 10)))
            (zipmap (range 10) (map #(* 5 % %) (range 10)))
            (zipmap (range 10) (map #(* % % %) (range 10))))

;; Now the thing is, that expressions such as these are very easy to debug.  In
;; emacs, for instance, you'd just put your cursor after a subexpression and use
;; C-x e to evaluate it.

;; Since the subexpressions are all valid code, you can just watch the result
;; building up by looking at the various subexpressions.

(range 10) ;; (0 1 2 3 4 5 6 7 8 9)
(map #(* 5 % %) (range 0 9)) ;;(0 5 20 45 80 125 180 245 320)
(zipmap (range 0 9) (map #(* 5 % %) (range 0 9)))
;; {8 320, 7 245, 6 180, 5 125, 4 80, 3 45, 2 20, 1 5, 0 0}
(map #(* % % %) (range 0 9)) ;; (0 1 8 27 64 125 216 343 512)
;; and so on, all the way to one of the final results:

(merge-with min
            (zipmap (range 10) (map #(* 10 %)  (range 10)))
            (zipmap (range 10) (map #(* 5 % %) (range 10)))
            (zipmap (range 10) (map #(* % % %) (range 10))))
;; {0 0, 1 1, 2 8, 3 27, 4 40, 5 50, 6 60, 7 70, 8 80, 9 90}

;; If you have a lisp-aware editor that can evaluate arbitrary sub-expressions
;; in a file, then such code is very easy to read and understand.

;; But of course, although you might well construct the final list in exactly
;; that way while experimenting at the REPL or writing the program in the first
;; place, you wouldn't use such expressions in your final program.

;; I think I'd feel a certain icy contempt if I came upon this expression in a
;; finished program:

(list
 (merge-with min
             (zipmap (range 10) (map #(* 10 %)  (range 10)))
             (zipmap (range 10) (map #(* 5 % %) (range 10)))
             (zipmap (range 10) (map #(* % % %) (range 10))))

 (merge-with max
             (zipmap (range 10) (map #(* 10 %)  (range 10)))
             (zipmap (range 10) (map #(* 5 % %) (range 10)))
             (zipmap (range 10) (map #(* % % %) (range 10)))))



;; Easy to work through bit by bit though it is, there's just an awful lot of
;; redundancy.  Imagine trying to find a bug caused by an 11 having sneaked in
;; there somehow. There are twelve different places where (range 10) is used.

;; Any programmer who would write such code is neglecting the cardinal virtue of
;; laziness, or, as it is sometimes known, the principle of not repeating
;; yourself twice .

;; Much more idiomatic, easier to understand just by reading, less redundant,
;; more easily modifiable, and altogether better style would be:

(let [r (range 10)
      make-map   (fn [f] (zipmap r (map f r)))
      linears    (make-map #(* 10 %))
      squares    (make-map #(* 5 % %))
      cubes      (make-map #(* % % %))
      mins       (merge-with min squares cubes linears)
      maxs       (merge-with max squares cubes linears)]
  (list mins maxs))

;; This is my preferred version. It seems clear but not excessively redundant.

;; However, if you were very keen on terseness, you might say:

(let [r (range 10)
      make-map   (fn [f] (zipmap r (map f r)))
      [linears, squares, cubes] (map make-map (list  #(* 10 %) , #(* 5 % %) ,  #(* % % %)))
      merge      (fn[f] (merge-with f linears squares cubes))
      mins       (merge min)
      maxs       (merge max)]
  (list mins maxs))

;; or perhaps even:

(let [r (range 10)
      make-map   (fn [f] (zipmap r (map f r)))
      fns        (map make-map (list  #(* 10 %) #(* 5 % %) #(* % % %)))
      merge-fns  (fn[f] (apply merge-with f fns))]
  (map merge-fns (list min max)))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; So far so good, but what about when you come to read your code 6 months
;; later?

;; You might be able to work out what the fourth version does in your head, but
;; it would be work.

;; I tend to find myself working out how complex let statements work by doing
;; this sort of thing in a little corner of the file:

(def r         (range 10))
(def make-map  (fn [f] (zipmap r (map f r))))
(def fns       (map make-map (list  #(* % % %) #(* 5 % %) #(* 10 %))))
(def merge-fns (fn[f] (apply merge-with f fns)))
(map merge-fns (list min max))

;; Now I am back to being able to evaluate the intermediate results.  I can just
;; move down the list of top level expressions evaluating interesting-looking
;; bits with C-x e and defining them as globals with M-C-x until I can see how
;; the whole thing fits together.

;; Narayan obviously does the same sort of thing from time to time, and his
;; insight is that this is a rather mechanical process.

;; Surely there's some way in which your environment or compiler can help?

(defmacro def-let
  "like let, but binds the expressions globally."
  [bindings & more]
  (let [let-expr (macroexpand `(let ~bindings))
        names-values (partition 2 (second let-expr))
        defs   (map #(cons 'def %) names-values)]
    (concat (list 'do) defs more)))

;; This is actually not Narayan's original macro.  I have modified it (notably
;; so that if there's an error somewhere in the let, it will get some of the way
;; down so that you can see why it broke) and simplified it a little, but the
;; idea is his, and any errors I have introduced are my fault.

;; Here's our let statement:

(let [r (range 10)
      make-map   (fn [f] (zipmap r (map f r)))
      fns        (map make-map (list  #(* % % %) #(* 5 % %) #(* 10 %)))
      merge-fns  (fn[f] (apply merge-with f fns))]
  (map merge-fns (list min max)))

;; Change let to def-let:

(def-let [r (range 10)
      make-map   (fn [f] (zipmap r (map f r)))
      fns        (map make-map (list  #(* % % %) #(* 5 % %) #(* 10 %)))
      merge-fns  (fn[f] (apply merge-with f fns))]
  (map merge-fns (list min max)))

;; Evaluate it. The actual result is the same, but the intermediate expressions
;; have been defined as globals.

(list r make-map fns merge-fns)
((0 1 2 3 4 5 6 7 8 9)
 #<user$make_map user$make_map@1a15b82>
 ({0 0, 1 1, 2 8, 3 27, 4 64, 5 125, 6 216, 7 343, 8 512, 9 729}
  {0 0, 1 5, 2 20, 3 45, 4 80, 5 125, 6 180, 7 245, 8 320, 9 405}
  {0 0, 1 10, 2 20, 3 30, 4 40, 5 50, 6 60, 7 70, 8 80, 9 90})
 #<user$merge_fns user$merge_fns@26920c>)

;; This not only means that you can look at them, it means that you can now take
;; interesting sub-expressions in the let statement and evaluate them, and this
;; will work, because the missing values are provided by the globals!!

;; e.g.

(make-map #(* % % %))
;;{0 0, 1 1, 2 8, 3 27, 4 64, 5 125, 6 216, 7 343, 8 512, 9 729}

;; Now of course this has the same disadvantage as the scratchpad method, in
;; that it sprays values all over your namespace.

;; But we're only debugging here. Style doesn't count, and I was doing the
;; scratchpad thing anyway. Now I can screw up faster!

;; So I think I'm going to be using def-let a lot.

;; Even better, it can deal with destructuring in the same manner as a let
;; statement can:

;; Here's a broken let statement. It won't execute at all. How would you debug
;; it if it wasn't so trivial?

(let [a 1
      b 2
      [c d] [(+ a b) (- a b)]
      e (c)]
  (list a b c d e))

;; Let's try def-let:

(def-let [a 1
          b 2
          [c d] [(+ a b) (- a b)]
          e (c)]
  (list a b c d e))

;; Same error, but it's done what it could:
a ;1
b ;2
c ;3
d ;-1
e ; var user/e is unbound.

;; So we can see that the problem was in the last bit, the assignment to e,
;; where we're trying to call c, even though it's a number not a function.

;; In fact def-let created more variables than that:

(macroexpand '(def-let [a 1
                        b 2
                        [c d] [(+ a b) (- a b)]
                        e (c)]
                (list a b c d e)))

(do (def a 1)
    (def b 2)
    (def vec__11853 [(+ a b) (- a b)])
    (def c (clojure.core/nth vec__11853 0 nil))
    (def d (clojure.core/nth vec__11853 1 nil))
    (def e (c)) (list a b c d e))

;; So in fact we can see how the destructuring binds really work.  This isn't in
;; fact that useful as far as I can tell, because the gensym when we macroexpand
;; the expression isn't the same one as when we evaluate it, but it might lead
;; to some sort of insight into what's going on.

;; And it works for all the destructuring forms that let does.

(let [a {:a 1 :b 2}
      {:keys [a b]} a]
  [a b])

;;[1 2]

(macroexpand '(def-let [a {:a 1 :b 2}
                        {:keys [a b]} a]
                [a b]))

(do (def a {:a 1, :b 2})
    (def map__11926 a)
    (def map__11926 (if (clojure.core/seq? map__11926)
                      (clojure.core/apply clojure.core/hash-map map__11926)
                      map__11926))
    (def b (clojure.core/get map__11926 :b))
    (def a (clojure.core/get map__11926 :a))
    [a b])

;; I've never seen anything like this before, and yet it strikes me as a useful
;; debugging technique, and it's been staring us in the face for years. If ever
;; you find yourself doing something repetitive and mechanical, macros can make
;; it go away.

;; With a little extra work, it would be possible to get def-let to produce an
;; extra definition of a function, say remove-crapspray or something, to undo
;; the damage that it's done once you've finished.

;; Obviously this is just crazy talk. But I'm going to do it anyway. If anyone's
;; interested in the final version, let me know.

Clojure Swank Server is Insecure by Default : Use a Firewall

As far as the maven clojure plugin goes, which is what I personally use, this issue is fixed in version 1.3.4, which is now publicly available. 

I'm told that modern leiningen setups open local ports, and that Phil has patched swank-clojure so that a local port is the default, so as long as you use latest versions of things this issue should be dead.

If you start your swank servers from old pom.xmls or project.cljs, you should probably still be careful just in case they pull in old versions of swank-clojure and clojure-maven-plugin. Update them if you can.

Well done to Phil and Mark, who fixed this within hours of it being noticed. Thanks guys!

A warning to anyone using SLIME/SWANK with clojure.

clojure-swank appears to open its server on the public port 4005 by default.
Note, not the loopback port 4005, the real one, visible to everyone.

If you don't have a firewall set up, then you're not only giving everyone on your local network the ability to execute arbitrary code on your machine, you're giving them a very nice interface with which to do it.

This appears to be the default behaviour of:

(swank.swank/start-server "/tmp/swank-port")
Connection opened on local port  34633
#<ServerSocket ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=34633]>

If you want a loopback port, then you have to say:

(swank.swank/start-server "/tmp/swank-port" :host "localhost")
Connection opened on local port  47233
#<ServerSocket ServerSocket[addr=localhost/127.0.0.1,port=0,localport=47233]>

And this behaviour appears to be inherited by the maven-clojure-plugin, and by leiningen using certain project files.

It does appear that a newly-created leiningen project opens the loopback port, but that's a property of project.clj, not of the leiningen tool itself.

If you're using swank and clojure, run a firewall, because you really don't want to open this port by accident on a hostile network:

Under Ubuntu, that's a simple as:

$ sudo ufw enable

To block everything, and:

$ sudo ufw allow 22

To reopen port 22 to allow ssh connections.

You can still use a local emacs to interact with a remote machine using this command to tunnel the remote port 4005 to your local port 4005 before using M-x slime-connect.


$ ssh -2 -N -f -L 4005:localhost:4005 user@remotebox

Clojure is Fast

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Clojure is Fast!

;; Paul Graham said that Common Lisp was two languages, one for writing programs
;; fast, and one for writing fast programs.

;; I've never tried to find Clojure's fast bits before, but I thought I'd give
;; it a try, using a simple example of a numerical algorithm that C and FORTRAN
;; would be very good for.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Let's try to integrate a differential equation.

;; Don't be scared! That means that we've got a number that needs to change over
;; time, and a function that tells us how much it needs to change by.

;; You've got a variable y, say it's 0 (feet). We start at time 0 (seconds).

;; We calculate f(0,0), lets say that's 0. (feet/second)

;; Then y has to change by 0 feet per second. So after a tenth of a second we
;; calculate that t should be 0.1 seconds, y should still be about 0 feet, and
;; that lets us work out roughly what f is now.

;; Say f is 0.1 : then y needs to change by 0.1 feet/second. So after another
;; tenth of a second, t is 0.2, y is roughly 0.01, and we can work out f again.

;; And repeat, for as many steps as you're interested in.

;; And that's how you find an approximate numerical solution to the differential
;; equation:

;; dy/dt = f(t,y) where f(t, y) = t-y and y=0 when t=0.

;; using a step size of one-tenth of a second.

;; This challenging procedure is known as Euler's Method, or sometimes as
;; first-order Runge-Kutta.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; A Test Case

;; As it happens, we can work out by devious mathematical trickery that the
;; exact solution (which is what happens if you make the steps so small that you
;; can't tell they're steps any more, and everything is nice and smooth) to this
;; equation is y=e^(-t)+t-1

;; So if we write our program correctly then when t is 1,
;; y should be close to (Math/exp -1) = 0.36787944117144233
;; And it should get closer if we make our steps smaller.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; So that's the scene set. Here is the program in which I am interested:

(defn f [t y] (- t y))

(defn solveit [t0 y0 h its]
  (if (> its 0) 
    (let [t1 (+ t0 h)
          y1 (+ y0 (* h (f t0 y0)))]
      (recur t1 y1 h (dec its)))
    [t0 y0 h its]))


;; And here's an invocation: start from 0.0 at time 0.0, step size is 0.1, run for 10 iterations

(solveit 0.0 0.0 0.1 10)
;; [0.9999999999999999 0.34867844010000004 0.1 0]

;; The answer tells us that after 10 steps t is 0.999..., or 1 as it's
;; traditionally known, and y is 0.348678....  The other two numbers are the
;; step size and the remaining iteration count, now down to 0 because the
;; process has done its ten steps.

;; In the exact answer, when t is 1, y should be e^-1, or 0.36787944117144233.
;; So the answer's right to within 0.02, which is a good indicator that the
;; process works.

;; Let's have a look at the answers with different numbers of steps:

(let [steps '(1 10 100 1000 10000 100000)
      results (map #(second (solveit 0.0 0.0 (/ 1.0 %) %)) steps )
      errors  (map #(- (Math/exp -1) %) results)]
  (partition 3 (interleave steps results errors)))

;; steps result              error
((1      0.0                 0.36787944117144233)
 (10     0.34867844010000004 0.019201001071442292)
 (100    0.3660323412732297  0.001847099898212634)
 (1000   0.36769542477096434 1.8401640047799317E-4)
 (10000  0.367861046432899   1.8394738543314748E-5)
 (100000 0.3678776017662642  1.8394051781167597E-6))

;; Ten times more iterations leads to a ten times better result, which we'd
;; expect from theory. That's why it's called a first order method.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; For this program, I care how fast the iteration is.
;; What gets measured gets improved:

(def *cpuspeed* 2.399) ;; My computer runs at 2.399 GHz

;; We can define a microbenchmarking macro which takes an expression
;; and the number of iterations that its calculation represents, and
;; tell us how many cpu cycles went into every iteration.

(defmacro cyclesperit [expr its]
  `(let [start# (. System (nanoTime))
         ret# ( ~@expr (/ 1.0 ~its) ~its )
         finish# (. System (nanoTime))]
     (int (/ (* *cpuspeed* (- finish# start#)) ~its))))

;; So here's an expression which times the loop over 100000 iterations.

(cyclesperit (solveit 0.0 1.0) 1000000)

;; What are we expecting? Well, if modern computers work the same way as the
;; computers I used to write assembly language for, then we can estimate thus:

;; Here's the program again:
(defn f [t y] (- t y))

(defn solveit [t0 y0 h its]
  (if (> its 0) 
    (let [t1 (+ t0 h)
          y1 (+ y0 (* h (f t0 y0)))]
      (recur t1 y1 h (dec its)))
    [t0 y0 h its]))

;; For every go round the loop we have to:
;; compare its with 0,
;; branch depending on the result,
;; add t0 to h,
;; call f with t0 and y0,
;; multiply h and the result,
;; add that to y0,
;; jump.

;; So if this was an assembly language program that worked the way you'd expect,
;; each loop would take 7 cycles.

;; This estimate turns out to have been a little optimistic.

;; On my desktop machine, the results of the timing expression
(cyclesperit (solveit 0.0 1.0) 1000000)
;; over four trial runs are:
2382
2290
2278
2317

;; So we're looking at a slowdown of about 300 times over what we could probably
;; achieve coding in assembler or in C with a good optimizing compiler (and of
;; course I'm assuming that floating point operations take one cycle each)

;; This is about the sort of speed that you'd expect from a dynamic language
;; without any optimization or type hinting.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; So how do we make it faster?

;; There's a fairly significant speed-up to be had from killing off function
;; calls. I think this is because primitives don't make it through function
;; boundaries They need to be boxed and unboxed.

;; There is something a bit odd about a functional language where function calls
;; are inefficient, but I understand that great men are working on the problem,
;; so it will probably not be a problem for clojure 1.3

;; In the meantime however, we'll inline f by hand and we'll create an internal
;; target for recur, using casts on the initial values to make sure that inside
;; the loop/recur, only the java primitives int and double are seen:

(defn solveit-2 [t0 y0 h its]
  (loop [t0 (double t0), y0 (double y0), h (double h), its (int its)]
    (if (> its 0) 
      (let [t1 (+ t0 h)
            y1 (+ y0 (* h (- t0 y0)))]
        (recur t1 y1 h (dec its)))
      [t0 y0 h its])))

;; Let's time that and see how it goes:
(cyclesperit (solveit-2 0.0 1.0) 10000000)
488
506
486

;; That's much better. The slowdown is now about 70 times compared with the
;; program and CPU in my head.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; The first law of optimizing things is that you need a profiler to find out
;; where the slow bits are.

;; At this point, we'll bring in jvisualvm, an excellent piece of software that
;; can be installed on Ubuntu with:

;; # sudo apt-get visualvm

;; and probably with something similar on any other system where Java will run.

;; Just run it. How it works should be fairly obvious. I'm sure there are
;; docs and stuff. I haven't looked.

;; When using jvisualvm, you should be careful to use the most stripped-down
;; clojure image possible.

;; I usually 'require' all of contrib on startup, and
;; this means that the poor profiler has to instrument something like 10000
;; classes.  This takes ages.

;; If you start with a clean image (it's ok to have everything on the classpath,
;; just don't load it if you don't need it), then it's only about 1000 classes,
;; and everything happens 10 times faster.  You still need to wait about 10
;; seconds while turning profiling on or off, but that's bearable.

;; Attach jvisualvm to your clojure, and then run

(cyclesperit (solveit-2 0.0 1.0) 1000000)

;; The profiling slows everything down to treacle, even the REPL, so remember to
;; de-attach it before trying to do anything that might take noticeable time.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Results of profiling:

;; There are four innocent looking calls to add, minus, and multi, all with
;; signature (double, double).  There's one to dec(int). But there's also one to
;; gt(int, Object). That only takes 20% of the time, apparently, but under it
;; there's a whole tree of other calls. Something is getting resolved at run
;; time, which is usually bad for speed.

;; The profiler suggest that function overload resolutions are being done every
;; time round the loop. Weirdly, it suggests that they're not very expensive
;; compared to add(double,double).  I am suspicious, so I'm going to try
;; changing (> its 0) to (> its (int 0)). That should allow the compiler to work
;; out the type of the call to > at compile time, rather than every time round.

(defn solveit-3 [t0 y0 h its]
  (loop [t0 (double t0), y0 (double y0), h (double h), its (int its)]
    (if (> its (int 0)) 
      (let [t1 (+ t0 h)
            y1 (+ y0 (* h (- t0 y0)))]
        (recur t1 y1 h (dec its)))
      [t0 y0 h its])))

;; Let's time that:
;; Remember to detach the profiler! If you don't you'll get cycle counts in the 100000s

(cyclesperit (solveit-3 0.0 1.0) 1000000)
79
79
63

;; Wow! That's made a vast difference. I don't understand why.

;; Apparently the literal 0 was being treated as a generic object. I can see why
;; that would be slow, but the profiler said that it was only 20% of the running
;; cost.  It seems more likely that removing it has somehow decontaminated the
;; other calls.  Maybe it's allowing the variables to stay in registers where
;; before they were being pushed out back onto the heap, or something?

;; I wonder if there's a way to examine the code that clojure generates for a
;; function?

;; At any rate, the loop is now about six times faster than it was.

;; Let's have another look with the profiler:
;; Attach it and run:

(cyclesperit (solveit-3 0.0 1.0) 1000000)

;; Again, the profiling looks about what you'd expect, except that a method
;; called RT.intCast is being called just as often as the multiplies, minuses,
;; and decs that I'm expecting to see. The profiler claims that it's not taking
;; up much time, but let's try to get rid of it by making an explicit local
;; variable for zero. For some reason this reminds me of ZX81 BASIC.
(defn solveit-4 [t0 y0 h its]
  (let [zero (int 0)]
    (loop [t0 (double t0) y0 (double y0) h (double h) its (int its)]
      (if (> its zero) 
        (let [t1 (+ t0 h)
              y1 (+ y0 (* h (- t0 y0)))]
          (recur t1 y1 h (dec its)))
      [t0 y0 h its]))))

;; Remove the profiler and re-time:
(cyclesperit (solveit-4 0.0  1.0) 100000000)
23
23
23


;; Doing the (int 0) outside the loop again seems to have tripled the speed of
;; the loop again.

;; The profiler is now telling me that there are: 2 adds(double,double), 1
;; gt(int,int), 1 minus(double, double), 1 dec(int) and 1 multiply(double,
;; double) in every loop, which is what I'd expect if I was writing C or Java to
;; do this, but I'm suspicious that it can tell! Presumably there's still some
;; dispatching going on? These should be single assembler instructions, and
;; invisible to a profiler working at function level.

;; With 4 floating point, 1 gt, 1 dec, and 1 conditional branch I'd imagine that
;; 7 cycles/loop would be as fast as this loop could be made to run without being clever.

;; So it appears that there's now only around a factor of 3 between this loop as
;; written, and what I'd expect from a C, Java or assembler program.

;; In absolute terms:

"Elapsed time: 1019.442664 msecs"
[1.0000000022898672 0.7357588790870762 1.0E-8 0]
(time (solveit-4 0.0 1.0 (/ 1.0 100000000) 100000000))


;; 1 second to do 100 000 000 iterations on my desktop, at about 23 cycles/loop

;; I'm pretty happy with that, especially given that the loop is still readable!
;; It's only slightly more complicated than the original. Optimizing Common Lisp
;; tends to make it look horrible.

;; Does anyone have any ideas how to squeeze a few more cycles out of the loop?

;; One more thing. We can make it go pretty fast. Does it still work?

;; Remember y(1) should approximate e^-1 0.36787944117144233, and our vast
;; speedup means that it's now not unreasonable to throw 1 000 000 000
;; iterations at the problem.

(let [steps '(1 10 100 1000 10000 100000 1000000 10000000 100000000 1000000000)
      results (map #(second (solveit-4 0.0 0.0 (/ 1.0 %) %)) steps )
      errors  (map #(- (Math/exp -1) %) results)]
  (partition 3 (interleave steps results errors)))

((1          0.0                 0.36787944117144233)
 (10         0.34867844010000004 0.019201001071442292)
 (100        0.3660323412732297  0.001847099898212634)
 (1000       0.36769542477096434 1.8401640047799317E-4)
 (10000      0.367861046432899   1.8394738543314748E-5)
 (100000     0.3678776017662642  1.8394051781167597E-6)
 (1000000    0.3678792572317447  1.8393969763996765E-7)
 (10000000   0.3678794227282174  1.8443224947262138E-8)
 (100000000  0.3678794397549051  1.4165372208552185E-9)
 (1000000000 0.3678794410553999  1.1604245342411446E-10))

;; Cool! Accuracy improves as predicted, and with 10^9 steps we get nine
;; significant figures in about ten seconds.

(time (solveit-4 0.0 1.0 (/ 1.0 1000000000) 1000000000))

;; Note:
;;
;; Just in order to keep my credibility as a numerical analyst intact, I ought
;; to point out that if I really was trying to solve a smooth ODE (instead of
;; investigating how fast I could make some simple numeric code in Clojure), I
;; wouldn't be using Euler's method. Optimize your algorithm before you optimize
;; your code.

;; No differential equations were harmed in the making of this blogpost.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Conclusion
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Clojure is a fast language, if you write so that your intentions are clear to
;; the compiler.  Something tells me that as clojure gets older, it will be
;; getting better at working out what your intentions are.

;; It would not surprise me in the slightest if very soon, the code as originally
;; written runs as fast or faster than my speeded up version.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

Clojure Macro Tutorial (Part III: Syntax Quote)

I am asked to point out that this is part of a series, following the form of the Ring Cycle, although not directly inspired by it. Syntax Quote plays the role of Siegfried in part III.

http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-i-getting.html http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-ii-compiler.html http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-ii-syntax.html

Suggestions for Götterdämmerung will be most welcome.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; Clojure Macro Tutorial Part III: Syntax Quote
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Let's finally look at how we'd go about writing forloop the correct way,
;; using the built in syntax-quote method.

;; The problem is:

;; We want:
'(forloop [i end]
          code)

;; to turn into:
'(let [finish end]
   (loop [i 0]
     (when (< i finish)
       (print i)
       (recur (inc i)))))

;; First Step

;; Just cut and paste the desired code, and backquote it:
;; This gives us a function which will always return the same code.

(defn forloop-fn-1 []
  `(let [finish end]
     (loop [i 0]
       (when (< i finish)
         (print i)
         (recur (inc i))))))

;; What does forloop-fn give us?

(forloop-fn-1)
;;evaluates to:
'(clojure.core/let [user/finish user/end]
                   (clojure.core/loop [user/i 0]
                                      (clojure.core/when (clojure.core/< user/i user/finish)
                                                         (clojure.core/print user/i)
                                                         (recur (clojure.core/inc user/i)))))

;; This has done all the name resolution (the really ugly bit) for us! Otherwise it's just like quote.

;; But if we try evaluating this code, we'll get an error:
;; Can't let qualified name: user/finish

;; The problem is that user/finish isn't a thing that you can put in a let.
;; finish is the local variable in the expanded code that we wanted to use a
;; gensym for. user/finish is a namespaced variable, which can't be let.

;; So we use the auto-gensym feature:
;; We add # to all occurrences of finish, which tells the compiler to use a gensym here.

(defn forloop-fn-2 []
  `(let [finish# end]
     (loop [i 0]
       (when (< i finish#)
         (print i)
         (recur (inc i))))))

;; Again we'll evaluate:
(forloop-fn-2)
;; to get:
'(clojure.core/let [finish__2254__auto__ user/end]
                   (clojure.core/loop [user/i 0]
                                      (clojure.core/when (clojure.core/< user/i finish__2254__auto__)
                                                         (clojure.core/print user/i)
                                                         (recur (clojure.core/inc user/i)))))

;; So now all the occurrences of finish# have been replaced in the generated code with gensym values,
;; in this case finish__2254__auto__

;; But this code still isn't executable.
;; This first problem with the code generated by forloop-fn-2 is that it expects a variable user/end to be defined.

;; But actually, end is one of the things that we want to vary in the generated
;; code.  We'll make it an argument of the macro, and use the unquote operator ~
;; to tell the function that whenever it sees ~end, it should replace it with
;; the argument.

(defn forloop-fn-3 [end]
  `(let [finish# ~end]
     (loop [i 0]
       (when (< i finish#)
         (print i)
         (recur (inc i))))))

;; Now let's evaluate:
(forloop-fn-3 10)
;; to get:
'(clojure.core/let [finish__2276__auto__ 10]
                   (clojure.core/loop [user/i 0]
                                      (clojure.core/when (clojure.core/< user/i finish__2276__auto__)
                                                         (clojure.core/print user/i)
                                                         (recur (clojure.core/inc user/i)))))

;; Looking good so far! If we try to evaluate this code, though, it objects to
;; the fact that user/i doesn't exist.  We can fix that in the same manner as we
;; fixed the problem with end, because the loop variable is, again, one of the
;; things which we want to vary.

(defn forloop-fn-4 [i end]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         (print ~i)
         (recur (inc ~i))))))

(forloop-fn-4 'j 10) 
;; ->
(clojure.core/let [finish__2298__auto__ 10]
                  (clojure.core/loop [j 0]
                                     (clojure.core/when (clojure.core/< j finish__2298__auto__)
                                                        (clojure.core/print j)
                                                        (recur (clojure.core/inc j)))))

;; (Notice that we have to quote j, because forloop-4-fn is a function, so its
;; arguments get evaluated before it is called)

;; And this code is actually executable! Try evaluating it directly, or use eval
;; to evaluate the return value of the function:

(eval (forloop-fn-4 'j 10))
;; 0123456789nil
(eval (forloop-fn-4 'different-loop-variable 15))
;; 01234567891011121314nil

;; So we've got a function that will give us code that will print out different
;; length runs of numbers, using the loop variable of our choice.

;; Of course to make it useful, we've got to make the (print i) bit a variable as well.
;; We could use the unquoting mechanism here too:
(defn forloop-fn-5 [i end code]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         ~code
         (recur (inc ~i))))))

;;Evaluate:
(forloop-fn-5 'i 10 '(print (* i i)))
;;To get:
(clojure.core/let [finish__2335__auto__ 10]
                  (clojure.core/loop [i 0]
                                     (clojure.core/when (clojure.core/< i finish__2335__auto__)
                                                        (print (* i i))
                                                        (recur (clojure.core/inc i)))))
;; Evaluate that:
(eval (forloop-fn-5 'i 10 '(print (* i i))))
;; 0149162536496481nil

;; Can you not sense imminent success?


;; But in fact, forloop would be much more useful if we were allowed an
;; unlimited number of expressions in our loop code, So we make the function
;; accept a variable number of arguments.  Because the 3rd, 4th, 5th etc
;; arguments are now made into the list "code", we need to use ~@, the
;; unquote-splicing operator to insert them without their outer brackets.
(defn forloop-fn-6 [i end & code]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         ~@code
         (recur (inc ~i))))))


(eval (forloop-fn-6 'i 10 '(print i) '(print (* i i))))
;;00112439416525636749864981nil


;; This would be make a perfectly good macro, but we remember that we wanted
;; (forloop [i 10] code) rather than (forloop i 10 code), so we use the
;; destructuring notation for the first two arguments:
(defn forloop-fn-7 [[i end] & code]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         ~@code
         (recur (inc ~i))))))

(eval (forloop-fn-7 '[i 10] '(print i) '(print (* i i))))
;;00112439416525636749864981nil

;; And finally, we change defn to defmacro, so that the compiler knows that when
;; it evaluates forloop-fn-7 it should pass in the arguments unevaluated, and
;; then treat the return value as code and compile it.

;; This allows us to dispense with the quotes on the arguments and the eval

(defmacro forloop [[i end] & code]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         ~@code
         (recur (inc ~i))))))

(forloop [i 10]
         (print i)
         (print (* i i)))

;;00112439416525636749864981nil

;; And we're done.

;; Let's look at the code our macro makes for us:

(macroexpand-1 '(forloop [i 10]
         (print i)
         (print (* i i))))

(clojure.core/let [finish__2442__auto__ 10]
                  (clojure.core/loop [i 0]
                                     (clojure.core/when (clojure.core/< i finish__2442__auto__)
                                                        (print i)
                                                        (print (* i i))
                                                        (recur (clojure.core/inc i)))))

;; All names resolved to the namespaces that they would have resolved to at the
;; time the macro was defined.

;; Gensyms done automatically, with readable silly names.

;; Bombproof, surprisingly straightforward to write, and the finished macro
;; looks awfully like the code it's trying to generate.

;; And that's how you really write a macro in clojure. Hacking it together by
;; hand is error prone for the reasons given above, and much harder.

Clojure Macro Tutorial (Part II: The Compiler Strikes Back)

I am asked to point out that this is part of a series, following the form of the Ring Cycle, although not directly inspired by it. Syntax Quote plays the role of Siegfried in part III.

http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-i-getting.html http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-ii-compiler.html http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-ii-syntax.html

Suggestions for Götterdämmerung will be most welcome.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; Clojure Macro Tutorial Part II: The Compiler Strikes Back
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; WARNING!

;; The first bit of this tutorial was about the idea of using the compiler
;; itself to write code.  This joyful idea has a slightly chequered history, and
;; turned out to be every bit as hard to get right as the idea of a subroutine
;; was.

;; In order to explain exactly how clojure's macro system works, I have felt it
;; necessary to go into the gory details of what goes wrong when you take the
;; naive approach to code generation.

;; As a result, this post is long, difficult, and ends on a depressing note.

;; I've tried (very hard) to make it only as long, and no more difficult than it
;; needs to be.

;; If I were you, I would skip directly to Part III, where clojure gloriously
;; resolves all these problems, and at the same time makes macros very easy to
;; write. Reading that may give you the courage to come back and face the dark
;; code beneath.

;; I am aware that this is really only a first cut at this way of explaining
;; things, and I'd very much welcome suggestions for how I could make it better.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; So far, we have been considering the dbg macro:

(defmacro dbg[x] `(let [x# ~x] (println '~x "->" x#) x#))

;; And we have got as far as being able to approximate it by:

(defmacro dbg-1 [s]
  (list 'let ['a s] (list 'println (list 'quote s) "->" 'a) 'a))

;; We have by this point understood the essence of macros, but there are a
;; couple of loose ends to tidy up.

;; We need to learn to use the syntax-quote notation `(let [x# .....) , partly
;; because it is easier to write macros when the notation looks like the code to
;; be produced, and partly because it helps us avoid certain difficulties which
;; have historically been a problem for macro writers.

;; And we need to understand what those difficulties are, so that we can
;; understand what syntax-quote is doing for us and why.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; A second problem to solve with macros: C-style for loops
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Before I try to explain syntax-quote, we should look at another simple
;; macro, which is nevertheless complex enough to run into the traditional
;; difficulties of macro-writing.

;; Suppose we find ourselves writing many imperative loops. The sort of thing
;; which C expresses as
;; for(i=0; i<10; i++)
;; {
;;     print "%d" i;
;; }

;;In clojure, we can equivalently write:
(loop [i 0]
  (when (< i 10)
    (print i)
    (recur (inc i))))

;; 012345678910
;; nil

;; Let us see if we can construct a simple macro to take the labour out of
;; reading and writing these little loops using the primitive macro construction
;; methods that we already know.

;; (In effect, we are trying to write a simple version of the doseq macro.)

(comment
;; We would like to be able to write
  (forloop [i 10] (print i))

;; and have it turn into:
  (loop [i 0]
    (when (< i 10)
      (print i)
      (recur (inc i)))))

;; Let us first of all define a code-generating function:
(defn forloop-f [[var finish] & code]
  (list 'loop [var 0]
        (concat (list 'when)
                (list (list '< var finish))
                code
                (list (list 'recur (list 'inc var))))))

;; A quick test
(forloop-f '[i 10] '(print i))
;;evaluates to:
(loop [i 0]
  (when (< i 10)
    (print i)
    (recur (inc i))))

;; which is what we want.

;; So let's make it a macro:
(defmacro forloop-bugs-1 [[var finish] & code]
  (list 'loop [var 0]
        (concat (list 'when)
                (list (list '< var finish))
                code
                (list (list 'recur (list 'inc var))))))


;; And try it out:

(forloop-bugs-1 [i 10] (print i))
;;0123456789
;;nil

(forloop-bugs-1 [j 10] (dbg j))
;; j -> 0
;; j -> 1
;; j -> 2
;; j -> 3
;; j -> 4
;; j -> 5
;; j -> 6
;; j -> 7
;; j -> 8
;; j -> 9
;; nil

;; It seems to have worked! Bingo?

;; It has worked. But it's not called forloop-bugs-1 for nothing.  We will have
;; to work a little to make the bugs show, but they are there.
 
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; There are some problems that one runs into when constructing macros in this
;; way, and they are problems that all lisps have to find ways of solving if
;; they want macros.

;; I don't think many people realize quite how clever clojure's namespace and
;; backquote system are. They make what were serious problems with nasty
;; solutions in traditional lisps into minor difficulties in clojure.

;; We should understand the problems, in order to understand the answer, and use
;; clojure's macros with full confidence, rather than thinking of them as some
;; sort of magic.

;; Let's have a closer look at our naively-written loop macro:
(defmacro forloop-bugs-1 [[var finish] & code]
  (list 'loop [var 0]
        (concat (list 'when)
                (list (list '< var finish))
                code
                (list (list 'recur (list 'inc var))))))


;; This macro, simple though it is, is sufficiently complex that it runs
;; into all the traditional difficulties of macros:

;; Once we've ploughed through the difficulties, we'll be able to see what
;; syntax-quote is for, and better appreciate what it's doing.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; Controlling the evaluation of the arguments
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; The first problem is not so much a problem, as it is the very thing we are
;; trying to do: control when the arguments to our macro get evaluated.

;; Consider
(forloop-bugs-1 [i (rand 10)]
                (print i))

;; The intent is to go round the loop a random number of times.
;; Here are some sample evaluations:
;; 0123nil
;; 01234nil
;; 0nil

;; So by all appearances, the macro is doing what it should.

;; But let's use our debugging macro to find out how many times (rand 10) is getting called:
(forloop-bugs-1 [i (dbg (rand 10))]
                (print i))

;; (rand 10) -> 0.9753528272119372
;; 0(rand 10) -> 2.407051391491234
;; 1(rand 10) -> 8.511366087571314
;; 2(rand 10) -> 6.795055112530893
;; 3(rand 10) -> 1.6571396363426516
;; nil

;; Was that what you expected to happen?

;; It seems that (rand 10) is getting called each time we go round the loop.

;; Let's ask the compiler what
(forloop-bugs-1 [i (dbg (rand 10))]
                (print i))
;; expands to, which we can do using the function macroexpand-1:
(macroexpand-1 '(forloop-bugs-1 [i (dbg (rand 10))]
                                (print i)))

;; The generated code turns out to be:
(loop [i 0]
  (when (< i (dbg (rand 10)))
    (print i)
    (recur (inc i))))

;; That's what we thought we wanted, but looking at it now, it's pretty obvious
;; that there's a problem.

;;The code that we should probably have written might look something like this:
(let [finish (dbg (rand 10))]
  (loop [i 0]
    (when (< i finish)
      (print i)
      (recur (inc i)))))

;; Here are some test evaluations of this new code:

;; (rand 10) -> 9.333250125032992
;; 0123456789nil

;; (rand 10) -> 4.260732182937476
;; 01234nil

;; (rand 10) -> 1.6344563853179461
;; 01nil

;; Which is probably what we want, and almost certainly what someone using the
;; macro will expect.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; The First Law
;;
;; If you take control of when your arguments are evaluated, you have to take
;; control of when your arguments are evaluated.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; So let's modify our macro in the obvious way to do what we now realize that we want:

(defmacro forloop-bugs-2 [[var end] & code]
  (list 'let ['finish end]
        (list 'loop [var 0]
              (concat (list 'when)
                      (list (list '< var 'finish))
                      code
                      (list (list 'recur (list 'inc var)))))))


(forloop-bugs-2 [i (dbg (rand 10))]
                (print i))
;;(rand 10) -> 5.427029108032794
;;012345nil

;; It all seems to be working.
;; Let's have a look at the generated code:

(macroexpand-1 '(forloop-bugs-2 [i (dbg (rand 10))]
                                (print i)))

(let [finish (dbg (rand 10))]
  (loop [i 0]
    (when (< i finish)
      (print i)
      (recur (inc i)))))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; Name Collision
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; The remaining problems are all to do with the way that the names in the
;; expansion collide with the names in the environment, and in any code which is
;; passed in to the macro.

;; You will notice that in the code generated by forloop-bugs-2, there is always
;; a variable called finish

;; Let's see if we can get that fact to cause problems:

;; Imagine that we want the loop to go all the way to ten, but for some reason
;; we don't want the printing to happen after a cutoff point:

;; We might say:
(let [cutoff 4]
  (forloop-bugs-2 [i 10]
                  (when (< i cutoff) (print i))))
;; 0123nil

;; Now let's change the name of the cutoff variable:
(let [finish 4]
  (forloop-bugs-2 [i 10]
                  (when (< i finish) (print i))))
;; 0123456789nil

;; Ooops.

;; If this doesn't violate the principle of least surprise for the user of the
;; macro, I don't know what would!

;; Again, We should look the macro expansion of the clause (forloop-bugs-2 ....)
(macroexpand-1 '(forloop-bugs-2 [i 10]
                                (when (< i finish) (print i))))
;; which is:
(let [finish 10]
  (loop [i 0] (when (< i finish)
                (when (< i finish) (print i))
                (recur (inc i)))))

;; This makes it pretty clear why this is happening, and why it didn't happen
;; when the variable was called cutoff.

;; One solution to this problem is just to choose names for the variables
;; generated by the macro which are unlikely to collide.

(defmacro forloop-bugs-3 [[var end] & code]
  (list 'let ['forloop-bugs-3-finish end]
        (list 'loop [var 0]
              (concat (list 'when)
                      (list (list '< var 'forloop-bugs-3-finish))
                      code
                      (list (list 'recur (list 'inc var)))))))


;; But this is a pretty poor solution:

;; Firstly, we don't want to write code that will only go wrong very occasionally.
;; That's the worst sort of bug to track down when it finally happens.

;; Secondly, million to one chances come up nine times out of ten. There's a
;; compiler involved.  Who knows what crazy names it will decide to come up
;; with, or what will happen when one macro expands into another macro?

;; So clojure gives us a special mechanism for making silly names which are
;; different each time.

(gensym) ;; G__11330 
(gensym) ;; G__11335

;; And we use that. Needless to say, avoid using names of the form G__????? in
;; your own code. The compiler will too.

;; How shall we use gensym?

(defmacro forloop-bugs-4 [[var end] & code]
  (let [finish (gensym)]
    (list 'let [finish end]
          (list 'loop [var 0]
                (concat (list 'when)
                        (list (list '< var finish))
                        code
                        (list (list 'recur (list 'inc var))))))))


;; Now let's look at the code that is generated by forloop-bugs-4
(macroexpand-1 '(forloop-bugs-4 [i 10]
                                (when (< i finish) (print i))))
'(let [G__11353 10]
   (loop [i 0]
     (when (< i G__11353)
       (when (< i finish)
         (print i))
       (recur (inc i)))))

;; Notice that the finish variable in the macro is now called G__11353, whereas
;; the finish variable in the code block has been preserved.

;; That's how you do macros by hand.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Except for one more little detail... 

;; It can happen that your macro is imported into a namespace where one of the
;; functions that was present where it was defined is either not present, or
;; defined differently.

;; So if we really and truly want to bulletproof our macros, which we do if we
;; are going to rely on them, then actually we should write:

(defmacro forloop [[var end] & code]
  (let [finish (clojure.core/gensym)]
    (clojure.core/list 'clojure.core/let [finish end]
          (clojure.core/list 'clojure.core/loop [var 0]
                (clojure.core/concat (clojure.core/list 'when)
                        (clojure.core/list (clojure.core/list '< var finish))
                        code
                        (clojure.core/list (clojure.core/list 'recur (clojure.core/list 'clojure.core/inc var))))))))


(forloop [i 10] (print i))
;;0123456789nil


;; Now obviously this is going to be a gigantic pain to write.

;; It would be repetitive, mechanical and error prone. Exactly the sort of thing
;; we were trying to avoid by using macros in the first place.

;; I'm actually slightly amazed that I got the abortion above to work at all,
;; and I wouldn't be at all surprised if bugs were found in it. Not conceptual
;; bugs. I think the concept is fine. Just stupid little finger-trouble errors.

;; As I said, exactly the sort of thing we use macros to avoid having to do ourselves.

;; And a macro could be written which would automate the tricky bits of macro writing. 

;; But clojure doesn't use an ordinary macro. It considers that macro-writing is
;; so important that there is special syntax built into the reader.

;; This is what forloop should really look like. It's much easier to write like this,
;; and it looks very like the code we're trying to generate.

(defmacro forloop [[i end] & code]
  `(let [finish# ~end]
     (loop [~i 0]
       (when (< ~i finish#)
         ~@code
         (recur (inc ~i))))))

;;  How to write macros like this is explained in part III.