group-by

;; Done it again!

;; Have you ever wanted to filter a sequence?

(filter odd? '(1 2 3 4 5))  ;-> (1 3 5)

;; But as well as keeping the bits that satisfy the predicate, you want the bits that don't as well?

(filter (complement odd?) '(1 2 3 4 5)) ; -> (2 4)

;; I do this so often that I wrote a little function to do it for me:


(defn split-filter [predicate sequence]
  "split a sequence into two sequences by predicate"
  [(filter predicate sequence) (filter (complement predicate) sequence)])


(split-filter odd? '(1 2 3 4 5)) ; ->  [(1 3 5) (2 4)]

;; Handy huh?, but soon after I wanted to split a sequence three ways

(filter #(= 1 (count %)) '((1) (1 2) (3 4) (3 4 5) (9) (8) (7 6))) ; -> ((1) (9) (8))

(filter #(= 2 (count %)) '((1) (1 2) (3 4) (3 4 5) (9) (8) (7 6))) ; -> ((1 2) (3 4) (7 6))

(filter #(= 3 (count %)) '((1) (1 2) (3 4) (3 4 5) (9) (8) (7 6))) ; -> ((3 4 5))

;; And of course that leads to a new function, to split a sequence by an arbitrary function

(defn hash-filter [pred coll]
    "splits sequence by predicate into a hash of predicate results to (reversed) subsequence"
    (reduce (fn [m x] (assoc m (pred x) (cons x (get m (pred x) '())))) {} coll))

(hash-filter count '((1) (1 2) (3 4) (3 4 5) (9) (8) (7 6))) ; ->
                                        ;{  3 ((3 4 5)),
                                        ;   2 ((7 6) (3 4) (1 2)),
                                        ;   1 ((8) (9) (1)) }


;; And so at this point I am feeling nice and smug, because I have invented a nice new abstraction and
;; evidently it is very useful (in fact I used it several times in my program).

;; And I am wary of that feeling, because I have had it before.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; clojure.core is very good. It is beautifully documented on a single page here:

;; http://clojure.org/cheatsheet

;; There is even a nicely coloured A4 version on a pdf, which fits on one double sided sheet.

;; I have now printed out that single A4 page, and have it next to my computer.

;; Looking at it has led me to:

(defn group-by 
  "Returns a map of the elements of coll keyed by the result of
  f on each element. The value at each key will be a vector of the
  corresponding elements, in the order they appeared in coll."
  {:added "1.2"}
  [f coll]  
  (persistent!
   (reduce
    (fn [ret x]
      (let [k (f x)]
        (assoc! ret k (conj (get ret k []) x))))
    (transient {}) coll)))

;; Note the nice use of persistent and transient together with vector conj so that you don't need to reverse the sequences
;; if you want them to stay in the same order.

;; Note the fact that group-by only evaluates the predicate once for each member.

;; I hadn't even noticed that I'd typed that twice.

Gis A Job! (And I'll give you £500)

Decided that I'd offer a £500 reward to anyone who can provide a useful introduction. Details here: http://johnlawrenceaspden.blogspot.com/2010/10/job-hunting-500-reward.html

---

Anyone in Cambridge need a programmer? Obviously Clojure is a speciality, and my current obsession, but I'm also pretty good with C (especially the embedded variety), microcontrollers, and Python, and I have a particular facility with mathematical concepts and algorithms of all kinds. My obsessions can be pretty quickly changed when I need them to be.

I have a reputation for being able to produce heavily optimised but nevertheless bug-free and readable code, but I also know how to hack together sloppy, bug-ridden prototypes, and I know which style is appropriate when, and how to slide along the continuum between them.

I've worked in telecoms, commercial research, banking, university research, a chip design company, server virtualization, a couple of startups, and occasionally completely alone.

I've worked on many sizes of machine. I've written programs for tiny 8-bit microcontrollers, and once upon a time every IBM machine in one building in Imperial College was running my partial differential equation solvers in parallel in the background.

I'm smart and I get things done. I'm confident enough in my own abilities that if I can't do something I admit it and find someone who can.

I also have various ancient and rusty skills with things like Java, C++, R, Common LISP, Scheme, ML, FORTRAN and Pascal which can be brushed up if necessary. Like all lispers, I occasionally write toy interpreters for made-up languages for fun.

If you're a local company using Java, who might be interested in giving Clojure a try (motivation here, in Paul Graham's classic Beating the Averages), I'd love to try to show you what all the fuss is about.

CV here if you're interested: http://www.aspden.com

I've never used a CV before, having always found work through word of mouth. So I expect that it can be improved. If anyone's got any suggestions as to how it could be better written, do please leave comments or e-mail cv@aspden.com.

EMACS Clojure Colour Scheme

Someone rather kindly said in a private e-mail that they like the syntax highlighting on this site, and requested the relevant .emacs

It's actually a standard colour scheme. Under Ubuntu/Debian you install the emacs-goodies-el package:

$ sudo apt-get install emacs-goodies-el

And then the elisp you need is:

(require 'color-theme)

(color-theme-billw)

I don't know who billw is, but thank you. I like it too. 

To make html files from clojure files, use M-x htmlize-file

Generating XML to make SVG vector graphics files

Clojure makes it easy to generate and parse XML. SVG is a form of XML. And therefore clojure is good for doing vector graphics drawings.

This year's Turner Prize Winner

;; The other day, I wanted to make some graph paper.

;; While experimenting with inkscape, I noticed that svg is actually an xml file format.

;; Since clojure is good with xml, that means that it's actually easier to make such a drawing with a program:

;; Each element of the drawing is represented as a map

(defn make-rect [i j squaresize]
  {:tag :rect
   :attrs {:x (str (* squaresize i))
           :y (str (* squaresize j))
           :width  (str squaresize)
           :height (str squaresize)
           :style "fill:white;stroke:black;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"}})

;; The whole file is represented as a map containing those maps

(defn make-svg [gridsize squaresize]
  {:tag :svg
   :attrs {:width "100%"
           :height "100%"
           :version "1.1"
           :xmlns "http://www.w3.org/2000/svg"}
   :content (for [i (range gridsize) j (range gridsize)] (make-rect i j squaresize))})

;; The library clojure.contrib.lazy-xml will turn the nested map into xml:

(require 'clojure.contrib.lazy-xml)

;; We can use with-out-str to capture the output, which is unaccountably printed
;; rather than given back as a string, and spit to write it to a file.

(spit "squares.svg" (with-out-str (clojure.contrib.lazy-xml/emit (make-svg 10 80))))

;; The nice thing about this is that you can then use inkscape to modify the
;; file, and then diff to work out how to take the modifications back into the
;; program. Does anybody know how to make the emit function format the xml so
;; that the output file is nicely readable?

Latest Collection of Filthy Hacks for REPL Conditioning

I have this set as the maven-clojure-plugin replscript, so it loads before the swank server starts.

I got bored waiting for my REPLs to require everything before I could use them, and so I'm using an agent to require all the namespaces in the background after the swank server is started.

I know that this is a filthy hack, but it hasn't bitten me yet, and Lord, the Roman hyacinths are blooming in bowls and

;; LOAD ME WITH
;; (load-file "/home/john/hobby-code/require-all-snippet.clj")

;; This file conditions a repl in various ways that I do all the time.

;; Firstly we want to 'require' all the namespaces on the classpath
;; This ensures that find-doc and the like will work
(require 'clojure.contrib.find-namespaces)

;; Some namespaces may fail to load, so catch any exceptions thrown
(defn- require-may-fail [ns]
  (try
   (print "Attempting to require " ns ": ")
   (require ns)
   (println "success")
   (catch Throwable e (println "couldn't require " ns "\nException\n" e "\n\n"))))

;; Generally we'd want clojure.*, clojure.contrib.*, and any project-specific namespaces
(defn require-all-namespaces-starting-with [strng]
  (doall (map require-may-fail 
              (filter #(. (str %) startsWith strng) 
                      (clojure.contrib.find-namespaces/find-namespaces-on-classpath)))))

;; Some of clojure's extra namespaces are so useful at the REPL that I want them
;; to have shorter names, i.e. I want to be able to type 'r/source' rather than
;; 'clojure.repl/source'.
;; This also means that emacs tab completion can find them with e.g. r/<TAB>

(require '(clojure [test :as t]
                   [inspector :as i]
                   [repl :as r]
                   [pprint :as pp])) 
(require '(clojure.contrib
           [trace :as cct]
           [repl-utils :as ccr]))


;; It drives me up the wall that it's (doc re-pattern) but (find-doc "re-pattern").
;; Can use macros so that (fd re-pattern) (fd "re-pattern") and (fd 're-pattern) all mean the same thing
(defn stringify [x]
  (println "stringify given" (str x))
  (let [s  (cond (string? x) x
                 (symbol? x) (str x)
                 (and (list? x) (= (first x) 'quote)) (str (second x))
                 :else (str x)) ]
    (println (str "translating to: \"" s "\""))
    s))


;; Sometimes I like to ask which public functions a namespace provides.
(defn ns-publics-list [ns] (#(list (ns-name %) (map first (ns-publics %))) ns))
;; And occasionally which functions it pulls in (with refer or use)
(defn ns-refers-list  [ns] (#(list (ns-name %) (map first (ns-refers %))) ns))

;; Nice pretty-printed versions of these functions, accepting strings, symbols or quoted symbol
(defmacro list-publics     
  ([]   `(clojure.pprint/pprint (ns-publics-list *ns*)))
  ([symbol-or-string] `(clojure.pprint/pprint (ns-publics-list (find-ns (symbol (stringify '~symbol-or-string)))))))

(defmacro list-refers
  ([]   `(clojure.pprint/pprint (ns-refers-list *ns*)))
  ([symbol-or-string] `(clojure.pprint/pprint (ns-refers-list (find-ns (symbol (stringify '~symbol-or-string)))))))

;; List all the namespaces
(defn list-all-ns [] (clojure.pprint/pprint (sort (map ns-name (all-ns)))))
;; List all public functions in all namespaces!
(defn list-publics-all-ns [] (clojure.pprint/pprint (map #(list (ns-name %) (map first (ns-publics %))) (all-ns))))

;; With all the namespaces loaded, find-doc can be overwhelming.
;; This is like find-doc, but just gives the associated names.
(defn find-doc-names
  "Prints the name of any var whose documentation or name contains a match for re-string-or-pattern"
  [re-string-or-pattern]
    (let [re  (re-pattern re-string-or-pattern)]
      (doseq [ns (all-ns)
              v (sort-by (comp :name meta) (vals (ns-interns ns)))
              :when (and (:doc (meta v))
                         (or (re-find (re-matcher re (:doc (meta v))))
                             (re-find (re-matcher re (str (:name (meta v)))))))]
               (print v "\n"))))




;;find symbol or string in docs 
(defmacro fd [symbol-or-string] `(find-doc (stringify '~symbol-or-string)))

(defmacro fdn [symbol-or-string] `(find-doc-names (stringify '~symbol-or-string)))

;; find the source file which defines a thing:
(defn source-file* [symbol] (:file (meta (resolve symbol))))

(defmacro source-file [symbol-or-string] `(source-file* (symbol (stringify '~symbol-or-string))))

;;get the methods of a java object
(defn meths [x] (println (apply str (interpose "\n" (map str (.getMethods (if (class? x) x (class x))))))))
;;get just the names of the methods
(defn meth-names[x] (map #(.getName %) (.getMethods (if (class? x) x (class x)))))


;;debugging macro                                try: (* 2 (dbg (* 3 4)))
(defmacro dbg [x] `(let [x# ~x] (do (println '~x "->" x#) x#))) 
;;and pretty-printing version 
(defmacro ppdbg [x]`(let [x# ~x]
                      (do (println "--")
                          (clojure.pprint/pprint '~x)
                          (println "->")
                          (clojure.pprint/pprint x#)
                          (println "--") x#))) 

;;and one for running tests 
(defmacro run-test [fn] `(test (resolve '~fn)))

;; def-let as in blogpost
(defmacro def-let
  "like let, but binds the expressions globally."
  [bindings & more]
  (let [let-expr (macroexpand `(let ~bindings))
        names-values (partition 2 (second let-expr))
        defs   (map #(cons 'def %) names-values)]
    (concat (list 'do) defs more)))


;; Sometimes it's nice to check the classpath
(defn- get-classpath []
   (sort (map (memfn getPath) 
              (seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))))))

(defn print-classpath []
  (clojure.pprint/pprint (get-classpath)))

(defn get-current-directory []
  (. (java.io.File. ".") getCanonicalPath))

;;print the classpath
(println "Classpath:")
(print-classpath)

(println "Current Directory" (get-current-directory))

;;print the public functions in the current namespace
(println "Current Namespace")
(list-publics)

;;hint on how to require project specific namespaces
(println "to require all namespaces starting with example:")
(println "(require-all-namespaces-starting-with \"example\")")

;; see http://blog.n01se.net/?p=85
(println "setting *print-length* to 103, *print-level* to 13 to deal with infinities")
(println "you have to do this before starting the swank server if you want it to work in emacs evaluations")
(set! *print-length* 103)
(set! *print-level* 13)

;; but we don't need this bit, 
;; (require 'clojure.contrib.repl-utils)
;; (clojure.contrib.repl-utils/add-break-thread!)
;; because swank repl threads already have a break handler set.
;; might come in useful for command line repls though.


;;require everything from clojure and clojure.contrib, so that find-doc can find it. Do it in an agent so it doesn't block repl startup. Jesus, am I really allowed to do this?
(def require-all-agent (agent "not done"))
(send-off require-all-agent (fn[agent] (with-out-str (require-all-namespaces-starting-with "clojure"))))

Cambridge Clojure Meetup (Tuesday 5th October in the Punter)

We thought we'd try and revive the defunct Cambridge Clojure meetings.


Who fancies a pint on Tuesday evening at 19:30 in the Punter?






View Larger Map

Polyglot Maven with Clojure and Emacs : My Edge is Bleeding

After a couple of hours hacking, I admit defeat. Google is not my friend any more. I am inches from my goal, and yet so far ....

I decided to try out polyglot maven, which promises the joy of maven without the xml of maven.

It's still pre-release, so it can't be criticized for the fact that its instructions don't work, or for the fact that build from source is broken.

However, it is almost possible to figure out what to do:

Download the prebuild 0.8:
$ wget http://polyglot.sonatype.org/site-nexus/download/pmaven-0.8-SNAPSHOT-bin.zip


Unzip it:

$ unzip pmaven-0.8-SNAPSHOT-bin.zip

Run it, using the pom.clj below as the project file, and ask it to open a swank server:

$ pmaven-0.8-SNAPSHOT/bin/mvn -f pom.clj clojure:swank

Connection opened on local port  4005

#<ServerSocket ServerSocket[addr=localhost/127.0.0.1,port=0,localport=4005]>

All is well. This works on my machine, running Ubuntu 10.04 in a guest account.

But I have cheated. Crucial jars live in the clojars repository.
I cannot figure out how to add the clojars repository to my pom.clj.

However, if you run maven on a corresponding pom.xml, with the same version and that repository listed, then it will download the relevant jars and put them in the local repository, which is good enough. After that maven will find them.

Polyglot maven understands the pom.xml file below as well. Just put it in your current directory and run:

$ pmaven-0.8-SNAPSHOT/bin/mvn -f pom.xml clojure:swank

And that will bring the necessary jars into your local repository. After that the pom.clj file is enough.

I am not claiming that this is in any way an improvement yet, but it looks very promising.

If someone can tell me what to add to pom.clj to add clojars, we are done.

I have grown rather fond of maven's pom.xml files. Soon they will be gone. I will actually miss them.

But brevity is the soul of not cocking things up. Here are the two files:

pom.clj

(defproject main "com.aspden:polyglot-maven-test:1.0-SNAPSHOT"
    :dependencies [["org.clojure:clojure:1.3.0-alpha1"]
                   ["org.clojure:clojure-contrib:1.2.0-SNAPSHOT"]
                   ["swank-clojure:swank-clojure:1.3.0-SNAPSHOT"]
                   ["jline:jline:0.9.94"]]
    :plugins [["com.theoryinpractise:clojure-maven-plugin:1.3.4"]])
 

pom.xml

<project>

  <modelVersion>4.0.0</modelVersion>
  <groupId>com.aspden</groupId>
  <artifactId>polyglot-maven-test</artifactId>
  <version>1.0-SNAPSHOT</version>
  <name>polyglot-maven-test</name>
  <description>polyglot-maven-test</description>


  <repositories>
    <repository>
      <id>central</id>
      <url>http://repo1.maven.org/maven2</url>
    </repository>
    <repository>
      <id>clojure</id>
      <url>http://build.clojure.org/releases</url>
    </repository>
    <repository>
      <id>clojars</id>
      <url>http://clojars.org/repo/</url>
    </repository>
  </repositories>

  <dependencies>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure</artifactId>
      <version>1.3.0-alpha1</version>
    </dependency>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure-contrib</artifactId>
      <version>1.2.0-SNAPSHOT</version>
    </dependency>
    <dependency>
      <groupId>jline</groupId>
      <artifactId>jline</artifactId>
      <version>0.9.94</version>
    </dependency>
    <dependency>
      <groupId>swank-clojure</groupId>
      <artifactId>swank-clojure</artifactId>
      <version>1.3.0-SNAPSHOT</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>com.theoryinpractise</groupId>
        <artifactId>clojure-maven-plugin</artifactId>
        <version>1.3.4</version>
      </plugin>
    </plugins>
  </build>
  
</project>