How the Clojure Dojo worked in London

I ended up giving my clojure dojo http://learnclojure.blogspot.com/2010/01/clojure-dojo-method-of-heron-of.html twice at dev8d in London, after two dry runs in Cambridge. The results were as follows:

I In Cambridge to the BioInformatics Group of the Chemistry Department (8 people present for 1 1/2 hours)

Worked well, got as far as a general square root solver. Everyone stayed involved and I think everyone enjoyed it.

II In Cambridge to a man randomly encountered in a pub ( 1 person for 1 hour )

Worked a treat, got as far as a general square root solver. That will teach people to ask me what I'm doing when I look busy.

III In London at dev8d in an extra session organised in advance of the main event (5 people for 2 1/2 hours)

There was a non-native English speaker there, and the dojo format quickly proved its effectiveness. When he sat down at the keyboard it was obvious that he hadn't understood much of my introductory talk, but when he was forced to try to construct expressions at the REPL he quickly got the hang of it, and turned out to be very good.

We got through the square root finder very quickly, and then the group wanted to know about macros. I thought this was perhaps a bit much for people who'd only learned the language an hour ago, and instead led them through the symbolic differentiator, as an example of how LISP can process its own code. This is after all the basis of macros!

One girl out of the five had only the sketchiest notion of what a derivative was, so we made her do most of the typing in this second half.

Afterwards, when I pointed out that a program which takes code and gives back other code is usually known as a compiler, and that she'd just written one, her face lit up with pride. This was the most rewarding moment of the week for me.


IV In London at dev8d in the planned session (27 people for 2 1/2 hours)


With so many people it was very hard to keep everyone properly involved, and so we broke the rules and had people who already had a clojure setup working on their own laptops separately from the communal effort.

After an hour and a half or so we made our square root finder, and there was a big round of applause.


I should really have canned it there, but people were still keen and wanted to know more, so we pressed on to do the general Newton-Raphson method solver.

At that point it went a bit wrong:


I'd been expecting Newton's method to be common knowledge amongst programmers (I'm sure it was on my maths A-level), and so to be a nice simple example of how powerful the general iterative-improve strategy is.

But in fact the method seemed to confuse people, and they got distracted from the programming by trying to understand the algorithm. There were too many people in the room for me to make sure that everyone stayed on board.


Nevertheless, they got the thing done with not too much help from me. I think by the end I had lost some of my audience, and I was a bit disappointed.

But afterwards lots of people came up and told me how much they'd enjoyed it, and apparently there were a lot of positive comments made afterwards (I had spies planted to find out what people really thought.).

I suspect that some people loved it, and some people lost the plot and were bored. Sorry to those people! As they say, if someone isn't learning, someone isn't teaching....


I think next time I either need to find a less scary example (it wasn't meant to be scary!!) or find a better way of explaining the strategy without using the fear-inducing word "derivative"! That may in fact not be too hard. Something like 'try moving the guess a bit, and see how much difference that makes, ok how much does that mean we should change our guess by?'.

I'll also see if I can keep the numbers down to below ten for the next one. Then you can rescue people who are floundering rather than ignoring the fact so that the group can progress.

There will be a next time. Various friends have expressed interest, and I enjoyed doing it, so we'll organise some more sessions, and for the advanced students, I'll try and think of some suitable examples that will illustrate what's particularly good about Clojure, once people have got the hang of why LISP itself is so cool.

Conditioning the REPL

Clojure's REPL is fairly minimal on startup.

I find that I always want to load clojure.contrib.repl-utils, amongst other libraries, and I have some handy debugging and exploring functions which I like to have loaded when interacting with my programs.

Also, the otherwise excellent 'apropos' function find-doc only looks in namespaces which have been loaded, so it's no good for finding things when you don't know which namespace they're in.

For this reason, it's useful to require every namespace that you can find.

The number of batteries included in clojure is getting quite phenomenal, and find-doc, doc, and source and javadoc are good tools for exploring.

I also often want to know what the classpath and current directory are, and I sometimes find it frustrating to have to remember whether various functions take a string, a symbol or a quoted symbol.

Recently I've found myself putting all these little tweaks in a file which I can load first thing on starting a REPL. On my netbook requiring everything takes about 30 seconds.

 

;; LOAD ME WITH
;; (load-file "/home/john/hobby-code/require-all-snippet.clj")

;; This file conditions a repl in various ways that I do all the time.

;; Firstly we want to 'require' all the namespaces on the classpath
;; This ensures that find-doc and the like will work
(require 'clojure.contrib.find-namespaces)

;; Some namespaces may fail to load, so catch any exceptions thrown
(defn- require-may-fail [ns]
  (try

   (print "Attempting to require " ns ": ")
   (require ns)
   (println "success")
   (catch Exception e (println "couldn't require " ns "\nException\n" e "\n\n"))))


;; Generally we'd want clojure.*, clojure.contrib.*, and any project-specific namespaces
(defn require-all-namespaces-starting-with [strng]
  (doall (map require-may-fail 
              (filter #(. (str %) startsWith strng) 
                      (clojure.contrib.find-namespaces/find-namespaces-on-classpath)))))


;; The functions in these namespaces are so useful at the REPL that I want them 'use'd.
;; I.e. I want to be able to type 'source' rather than 'clojure.contrib.repl-utils/source'
(use 'clojure.contrib.repl-utils)
(use 'clojure.inspector)
(use 'clojure.contrib.pprint)
(use 'clojure.contrib.repl-utils)
(use 'clojure.contrib.trace)

;; It drives me up the wall that it's (doc re-pattern) but (find-doc "re-pattern").
;; Can use macros so that (fd re-pattern) (fd "re-pattern") and (fd 're-pattern) all mean the same thing
(defn- stringify [x]
  (println "stringify given" (str x))
  (let [s  (cond (string? x) x
                 (symbol? x) (str x)
                 (and (list? x) (= (first x) 'quote)) (str (second x))
                 :else (str x)) ]
    (println (str "translating to: \"" s "\""))
    s))



;; Sometimes I like to ask which public functions a namespace provides.
(defn- ns-publics-list [ns] (#(list (ns-name %) (map first (ns-publics %))) ns))
;; And occasionally which functions it pulls in (with refer or use)
(defn- ns-refers-list  [ns] (#(list (ns-name %) (map first (ns-refers %))) ns))


;; Nice pretty-printed versions of these functions, accepting strings, symbols or quoted symbol
(defmacro list-publics     
  ([]   `(pprint (ns-publics-list *ns*)))
  ([symbol-or-string] `(pprint (ns-publics-list (find-ns (symbol (stringify '~symbol-or-string)))))))

(defmacro list-refers
  ([]   `(pprint (ns-refers-list *ns*)))
  ([symbol-or-string] `(pprint (ns-refers-list (find-ns (symbol (stringify '~symbol-or-string)))))))

;; List all the namespaces
(defn list-all-ns [] (pprint (map ns-name (all-ns))))

;; List all public functions in all namespaces!
(defn list-publics-all-ns [] (pprint (map #(list (ns-name %) (map first (ns-publics %))) (all-ns))))

;; With all the namespaces loaded, find-doc can be overwhelming.
;; This is like find-doc, but just gives the associated names.

(defn- find-doc-names
  "Prints the name of any var whose documentation or name contains a match for re-string-or-pattern"
  [re-string-or-pattern]
    (let [re  (re-pattern re-string-or-pattern)]
      (doseq [ns (all-ns)
              v (sort-by (comp :name meta) (vals (ns-interns ns)))
              :when (and (:doc ^v)
                         (or (re-find (re-matcher re (:doc ^v)))
                             (re-find (re-matcher re (str (:name ^v))))))]
               (print v "\n"))))





;;find symbol or string in docs 
(defmacro fd [symbol-or-string] `(find-doc (stringify '~symbol-or-string)))

(defmacro fdn [symbol-or-string] `(find-doc-names (stringify '~symbol-or-string)))



;;debugging macro                                try: (* 2 (dbg (* 3 4)))
(defmacro dbg [x] `(let [x# ~x] (do (println '~x "->" x#) x#))) 

;;and pretty-printing version 
(defmacro ppdbg [x]`(let [x# ~x] (do (println "--")(pprint '~x)(println "->")(pprint x#) (println "--") x#))) 


;;and one for running tests 
(defmacro run-test [fn] `(test (resolve '~fn)))


;; Sometimes it's nice to check the classpath
(defn- get-classpath []
   (sort (map (memfn getPath) 
              (seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))))))

(defn print-classpath []
  (clojure.contrib.pprint/pprint (get-classpath)))

(defn get-current-directory []
  (. (java.io.File. ".") getCanonicalPath))


;;require everything from clojure and clojure.contrib, so that find-doc can find it
(require-all-namespaces-starting-with "clojure")

;;print the classpath
(println "Classpath:")
(print-classpath)

(println "Current Directory" (get-current-directory))

;;print the public functions in the current namespace
(println "Current Namespace")
(list-publics)


;;hint on how to require project specific namespaces
(println "to require all namespaces starting with example:")
(println "(require-all-namespaces-starting-with \"example\")")

Watching a macro as it expands

;; macros are code transformers
;; in a lisp, that means tree transformers
;; let's look at the -> macro, built into clojure.

(use 'clojure.contrib.repl-utils)
(source ->)

;; (defmacro ->
;;   "Threads the expr through the forms. Inserts x as the
;;   second item in the first form, making a list of it if it is not a
;;   list already. If there are more forms, inserts the first form as the
;;   second item in second form, etc."
;;   ([x] x)
;;   ([x form] (if (seq? form)
;;               (with-meta `(~(first form) ~x ~@(next form)) (meta form))
;;               (list form x)))
;;   ([x form & more] `(-> (-> ~x ~form) ~@more)))



;; What's 20C in Fahrenheit?
;; Multiply by nine, divide by 5, add 32
(-> 20 (* 9) (/ 5) (+ 32))

;; let's look at the tree for that
(use 'clojure.inspector)
(inspect-tree '(-> 20 (* 9) (/ 5) (+ 32)))

;; What does that look like after macroexpansion?
(use 'clojure.walk)
(macroexpand-all '(-> 20 (* 9) (/ 5) (+ 32)))
(inspect-tree (macroexpand-all '(-> 20 (* 9) (/ 5) (+ 32))))


;; How does the threading macro work?
(macroexpand-1 '(-> 20 (* 9) (/ 5) (+ 32)))

;; We can follow the recursion by using macroexpand-1 on each subexpression in turn.

;;slightly tidied up, stages are:

(macroexpand-1 '(-> 20 (* 9) (/ 5) (+ 32)))

(macroexpand-1
'(-> 
  (-> 20 (* 9))
  (/ 5) (+ 32)))

(macroexpand-1
 '(-> 
   (-> 
    (-> 20 (* 9)) 
    (/ 5)) 
   (+ 32)))

(macroexpand-1
 '(+ 
   (-> 
    (-> 20 (* 9)) 
    (/ 5))
   32))

`(+ 
  ~(macroexpand-1 '
    (-> 
     (-> 20 (* 9)) 
     (/ 5)))
  32)

`(+ 
  (/ 
   ~(macroexpand-1 '
     (-> 20 (* 9)))
   5)
  32)

(+ 
 (/ 
  (* 20 9) 
  5) 
 32)

;;This is a bit long-winded, how would we do this automatically?

(defn slow-tree-transformer [fn tree]
  (cond (not (seq? tree)) (fn tree)
        (empty? tree) '()
        :else (let [nxt (fn tree)]
                (if (not (= nxt tree)) nxt
                    (let [nxt (fn (first tree))]
                      (if (= nxt (first tree))
                        (let [nxt (slow-tree-transformer fn (first tree))]
                          (if (= nxt (first tree))
                            (cons (first tree) (slow-tree-transformer fn (rest tree)))
                            (cons nxt (rest tree))))
                        (cons nxt (rest tree))))))))


(def f (fn[x] (slow-tree-transformer macroexpand-1 x)))

(defn iterate-to-stable [fn start]
  (let [next (fn start)]
    (if (= next start) (list start)
        (cons start (iterate-to-stable fn next)))))

(use 'clojure.contrib.pprint)
(println "-----------------------------------")
(doall (map (fn [x] (pprint x) (println)) (iterate-to-stable f '(-> 20 (* 9) (/ 5) (+ 32)))))
(println "-----------------------------------")

Jihad Scanner

There's a very promising looking new lisp book out, Doug Hoyte's "Let over Lambda" http://letoverlambda.com/

My copy arrived this morning, and to keep me honest while reading it, I wondered how much of it could be translated into clojure. Here's the first example I found interesting:
 

;; The jihad scanner from let over lambda. Watch a communications channel for a word.

(defn make-scanner [ watchword ]
  (let [ curr (atom watchword) ]
    (fn [ chunk ]
      (doseq [c chunk]
        (if @curr
          (swap! curr (fn [cv] (cond (= (first cv) c)        (next cv) 
                                     (= (first watchword) c) (next watchword)
                                     :else                    watchword)))))
      (nil? @curr))))


;; I've added a clause so that jjihad still fires the detector. I think the original version doesn't catch jjihad
;; but does catch qjihad, which seems inconsistent.

;; Compared to the Common Lisp version, the requirement to wrap mutable state in an atom has cost us
;; a couple of spurious @s, and required "(swap! curr (fn [cv]" instead of "(setq" without really 
;; buying us anything. 

;; On the other hand, the fact that strings are just another form of seq has simplified the function
;; and generalised it to an arbitrary subsequence detector.


(let [a (make-scanner "jihad")]
  (a "We will start")
  (a "the jih")
  (a "ad tomorrow"))


(let [a (make-scanner "jihad")]
  (a "The jijihad starts tomorrow"))

(let [a (make-scanner "jihad")]
  (a "The jij ihad starts tomorrow"))

(let [a (make-scanner "jihad")]
  (a "Don't mention the j-word"))

(let [a (make-scanner '(97 98 99))]
  (for [x (partition 3 (range 110))] (a x)))

(let [a (make-scanner '[a b c])]
  (for [ x '((a b) (c d)) ] (a x)))

(let [a (make-scanner '[a b c])]
  (for [ x '((a b) [c d]) ] (a x)))

Requiring all possible namespaces

It often irritates me that I can't use clojure's built in documentation unless I've already loaded the library I need.

Sometimes it turns out that the generic-looking function I need already exists, but it can be hard to find it. Sometimes I can remember the function I want, but not its precise name or library namespace.

It's also often the case that the best way to understand a function is to read the source.

The other day it occurred to me that a function to find all the namespaces available on the classpath might help quite a bit, and I started to write one.

And at that point I had the obvious recursive thought.

   

;; Seek and ye shall find. Thank you Stuart Sierra.
(use 'clojure.contrib.find-namespaces)

;; First let's see what our classpath is:
(println "Classpath")
(doseq [p (seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader)))] (println (str p)))


;; In my case, that's the clojure.jar and clojure.contrib.jar files, and also the directory
;; containing the files for swank-clojure, the library which allows it to talk to emacs.
(println "Swank namespaces")
(println (for [s (for [n (all-ns)] (name (ns-name n))) :when (. s contains "swank")] s))


;; Because I'm still a beginner, and not brave enough to modify swank, I'm more interested in 
;; the namespaces provided by clojure and clojure.contrib

;; So far, the following have managed to get themselves loaded:
(println "None-swank namespaces")
(use 'clojure.contrib.pprint)
(pprint (sort (for [s (for [n (all-ns)] (name (ns-name n))) :when (not (. s contains "swank"))] s)))


;; But the following are in fact available
(pprint (find-namespaces-on-classpath))

;; Again, let's ignore the swank-related ones.
(pprint (filter #(not (. (str %) startsWith "swank")) (find-namespaces-on-classpath)))


;;For me there are only namespaces starting with either swank or clojure, because I don't have much
;;of a classpath. But it's easy enough to check that.
(defn find-namespaces-starting-with [strng]
  (filter #(. (str %) startsWith strng) (find-namespaces-on-classpath)))

(map count (map find-namespaces-starting-with '("clojure" "swank" ""))) ; should add up


;; Some of the clojure.contrib libraries fail to load for me, so we have to bullet-proof require
(defn require-may-fail [ns]
  (try
   (print "Attempting to require " ns ": ")
   (require ns)
   (println "success")
   (catch Exception e (println "couldn't require " ns "\nException\n" e "\n\n"))))


;; Now we can load everything
(doall (map require-may-fail (filter #(. (str %) startsWith "clojure") (find-namespaces-on-classpath))))


;; Now we have lots and lots of namespaces available: The number of 'batteries included' is impressive.
(println "None-swank namespaces")
(pprint (sort (for [s (for [n (all-ns)] (name (ns-name n))) :when (not (. s contains "swank"))] s)))


;; And the find-doc, doc, and source commands become really useful
;; Of course we not only have to find source, we actually can, now.
(find-doc "source code")

;; Also note that all this requiring hasn't brought any new functions into play yet.
;; To use something without qualification, you need to refer as well.
(refer 'clojure.contrib.repl-utils)
(doc source)
(source source)
(source get-source)
(:file (meta (resolve 'source)))

Other sources of enjoyment are the find and grep commands, and browsing clojure.contrib.jar and clojure.jar . These work particularly nicely under emacs. A good arrangement of find and grep is (assuming that your clojure sources are under ~/opt):

find ~/opt/ -name "*.clj" -and -type f -print0 | xargs -0 -e grep -nH -e "repl-util"

Using the trace function in clojure.contrib

There's a fine trace macro in clojure.contrib.trace, which really can help when looking at recursive functions.
Here's a very simple example with our dear friend 'naive fibonacci'.

   
user> (defn fib [n]
        (if (< n 2) n
            (+ (fib (- n 1)) (fib (- n 2)))))
#'user/fib
user> (use 'clojure.contrib.trace)
nil
user> (dotrace (fib) (fib 3))

TRACE t1880: (fib 3)
TRACE t1881: |    (fib 2)
TRACE t1882: |    |    (fib 1)
TRACE t1882: |    |    => 1
TRACE t1883: |    |    (fib 0)
TRACE t1883: |    |    => 0
TRACE t1881: |    => 1
TRACE t1884: |    (fib 1)
TRACE t1884: |    => 1
TRACE t1880: => 2
2
user> 

How the Clojure dojo actually worked in practice.

We decided that I'd try out (the first half of) my Clojure dojo on the Chemistry Department.

The audience was the non-lispers in our group, so there were about eight very bright Java guys, one of whom had done some ML before, and one of whom had been through this very exercise with me in a pub some months before as his first contact with LISP since he met McCarthy in the sixties(!).

The dojo format worked very well, and I mainly sat back and let them get on with it.

I didn't say a great deal after posing the problem in the terms outlined at the start. When they completed a subproblem I gave them a new problem and maybe a hint or two. I was surprised how much they managed to infer from limited information.

At first everything went absolutely swimmingly, and the syntax didn't faze them at all. Questions like 'What's the sum of the squares of 3 and 4' were answered without breaking stride. They understood Heron's method immediately. When I suggested that they turned it into a function to improve guesses, they spontaneously put in a target variable instead of hard-coding 10, and started talking about recursion.

They couldn't figure out how to do it, so I suggested trying to write factorial. This they managed to work out for themselves.

There was then much excitement trying to turn that into a solver, but they couldn't do it.

I suggested trying to write good-enough?

They realised the need for an absolute value function, and to my utter amazement, guessed the syntax for calling Math/abs, and then for Math/sqrt!

They declared the problem solved.

So I asked them to generalize the solution to make a general root-finder, which got them back to trying to get Heron's method to work.

This was the last time I said anything helpful. Much more thinking, and they got on the right track. They defined an okguess function which took the tolerance as a parameter, and were well on the way to solving the problem, when the ML guy and the guy who'd done it before took the chairs.

I said they were only allowed a couple of minutes, but they had the thing done almost immediately.

I still wanted to go back and show them iterate and map, so I asked them to make a function that returned a function. Can we make a function that takes a number and returns a function which multiplies things by it?

I hinted that defn was a combination of def and fn, and that was enough

That solved, I asked them to make a function which took a target and gave back the relevant improver.

Then I told them to try (doc iterate), and see if they could figure out what it was for. And then I told them to look up map and take. They were playing happily with infinite sequences when our hour and a half ran out.