Someone asked me about the path that lead me to Clojure. On the off chance that he's not the only one curious, I thought I'd post the answer here.
Sometime after college I started hunting for the "best language", not with any formal definition but as a personal quest for what I liked the best. A primary criteria was something I called "expressiveness". I don't know if I can define that accurately, but it has something to do with being able to provide APIs that concisely model whatever specific domain you're dealing with. A common example is how much easier it is to work with text in perl (because of strongly integrated regex support) than it is in C.
My "best language" before Clojure was Ruby, because of its clean object system and iterators, but I hadn't experienced any functional or immutable languages yet.
I had heard a lot about how amazing LISP was, but I didn't get it. I couldn't figure out why anyone liked LISP's annoying syntax, or why anyone (who wasn't deeply into Artificial Intellegence) would want to write programs that write programs. So one big turning point was reading "On Lisp" by Paul Graham (free download). By the time I was done with that book, I was convinced that any "best language" had to support macros.
So I tried to use Common Lisp, but just couldn't make it "stick". I had learned plenty of other languages, and usually enjoyed the process, but Common Lisp's lack of libraries really drove me nuts. I could do network, file or text stuff in 4 or 5 lines of ruby that took a page of code in Common Lisp. So I shelved Common Lisp for a while.
I do a lot of C++ and JavaScript at work, and was learning to really like JavaScript -- clean syntax, closures, direct support for hash-maps and regex, a builtin "gui" system. So it was easy for Yegge to lure me into trying JavaScript on the JVM, especially with the promise of optional static typing, which I was pretty sure I wanted.
Around the same time I learned about Project Euler, and thought it would be a great way to get used to using JavaScript for general purpose programming rather than just browser stuff. I did the first five PE problems in JavaScript before giving up. It was terrible. Not only was there no static typing yet, but things that had been minor annoyances in the browser now seemed to be giant obstacles to solving the problem at hand. Re-reading some of Yegge's cheerleading brought me to the conclusion that he's actually endorsing some private language based roughly on JavaScript. None of the other languages that Google lets him use are as malleable, and Rhino lets him code up whatever new features he wants in Java and morph the language to his will. This is great for him, I'm sure, but was lousy for me.
I hadn't used Java in any capacity for years, but now thanks to Yegge I had it installed and ready to go. This made me more willing to try Scala. I don't remember how I heard about it, but I picked it up to work on the rest of the PE problems, and this is when I started to learn about immutability. Scala encourages the use of immutable locals, and has a few immutable collections. I expended a lot of energy bending my problem-solving thought processes to this new context. Scala also has lazy streams, which was another big hurdle. Both of these were challenging to learn, but once I got it, they were very rewarding tools to have at my fingertips. Project Euler was my primary teaching tool, forcing me to learn new concepts a little at a time. The folks on the IRC channel helped me when I got stuck, though in exchange, I frequently had to endure condescension from some of them.
So this was going along okay, until I thought I had enough Scala under my belt to take a crack at a tricky little API I'd had on my mind. I'd been wanting a way of picking data out of HTML documents, something with the expressive power of XPath, but that embedded cleanly in the host language. I wanted to be able to provide new functions, filters, etc. (have you tried to work with dates in XPath and XSLT??), and for the return values to be host language data structures, not more XML documents. I'd actually built a beastly little project using xlstproc and ruby, but that's another story altogether.
Anyway, I began sculpting this API in Scala. I spent a day or two experimenting with syntax, the right combinations of traits, implicits, etc. to create the kind of XPath-like expressions I wanted. I finally got the beginnings of something that I thought would work, and compiled a minimal test. Well, tried to compile it, but the types weren't quite right. I then spent the another day or two trying to get the type declarations correct enough to even compile, and finally gave up in frustration. I'm sure it would have been possible to make it work, but I wanted to spend my time solving actual problems, not solving problems the compiler was making up for me. Maybe I didn't want static typing after all.
This is when I started learning Clojure. Paul Graham had convinced me I wanted a LISP. Yegge's hype about JavaScript had prepared me for the JVM. Scala had introduced me to immutability and lazy streams. Clojure let me use the best properties of each of these, so it was a very natural fit right from the beginning. I've since learned more -- how to use more immutable collections, software transaction memory, etc., and the discussion group and IRC channel have always been friendly and encouraging.
That's how I got to where I am. I don't think I'd recommend the JavaScript or Scala tangents to anyone else, but "On Lisp" may still be worth reading, at least until Programming Clojure comes out. Since bailing on Scala, I've done another 25 or so Project Euler problems in Clojure, and had so much fun I even went back and did the first 14 all over again. I also took another stab at XPath-inspired document querying, resulting in a zip-filter lib that I'll write about some other time.
Clojure's definitely closer to the "best language" than any other I've tried. If you haven't tried it yet, you should!
Hey, thanks for taking the time to write up this entry. I’ve been working with Common Lisp for a few weeks after spending quite a bit of time with scheme and have been dreaming of making the switch to Clojure for a few days now. After being pampered by Python’s standard library it’s incredibly easy to be frustrated by the state of Common Lisp and the libraries available. I think I’ll finally give Clojure a try tomorrow… Sweet!
I’ve read parts of both On Lisp, and Programming Clojure (in beta), and I’d say the latter is no replacement for the first. Programming Clojure introduces the language and its features (immutability, STM, concurrency model, etc), but doesn’t have near the amount of “why” that On Lisp does.
Dan: I resisted the urge to insert a smiley, but perhaps I shouldn’t have. Anyway, I absolutely agree — Programming Clojure is not trying to accomplish the same thing as On Lisp. In fact, the author of Programming Clojure among others are working porting examples from On Lisp to Clojure:
http://blog.thinkrelevance.com/2008/12/12/on-lisp-clojure
http://blog.fogus.me/2008/09/26/on-lisp-clojure-chapter-2/
Were there any particular problems on project Euler that Clojure was a big advantage for? I worked 58 problems using mostly Ruby, and came back and did 3 or 4 of them using Clojure but I ran out of steam. I’d like to know what I was missing out on.
> I’m sure it would have been possible to make it work, but I wanted to spend my time solving actual problems, not solving problems the compiler was making up for me. Maybe I didn’t want static typing after all.
It’s not the compiler making up problems for you if you can’t get your types correct. Static typing shows you your mistakes at compile time. Dynamic typing shows you your mistakes at runtime. Dynamic typing will not magically fix your incorrect types. But, if you want to buy some magic beans, send me an email. I’ve got a 2 for 1 special now.
Nathan: no problems in particular, I see PE more as a useful tool to explore the language, than that the language is likely to make an enormous difference in solving the problem. I think a language has to have pretty glaring deficiencies to be as bad at solving PE problems as JavaScript was. I did several in ruby as well, which was also quite pleasant. Learning to do them with immutable locals and collections was a useful stretch for me though, and I blogged earlier about one particular problem that I solved a few different ways to try out some of Clojure multi-thread features.
oopjosh: If there is a real error in my logic, this can indeed end up being expressed as an error in type declarations that a compiler can flag. In that case, I’d much rather have the error caught at compile time than at runtime. However, it’s also possible to have an error only in the type declarations, which is what I believe I was dealing with in that particular attempt at using Scala. This belief is strengthened by the fact that I was able to solve essentially the same problem in essentially the same way in a dynamically typed language — if the error had been in my logic I would have seen it at runtime, just as you said. For kind of exploratory API-building I was doing, Scala’s static type system was definitely causing friction.
I used XSLT in a medium/large Java project back in 2001-2003 and it did the trick, though we had to write quite a few extensions in Java. I was considering using XSLT again in a similar project last year and I found it awful! But I can strongly recommend XQuery (we’re using Saxon). It’s much nicer language IMO, though not without problems.
Now, I’m definitely interested in seeing this zip-filter lib! Perhaps soon I’ll recommend something different than XQuery?
Chouser,
Thanks for the write-up. I took a Scheme class back in college, but somehow it didn’t mesh with how my mind worked back then. I spent a number of years doing Java and then C#.
Last year, I learned Python in order to be able to use the Google App Engine, which lead to me doing a bunch of stuff with Django. I’ve found that I really prefer dynamically-typed languages (i.e. languages that associated type information with values, not with variables). I come from a mathematics/logic background, and I find that dynamic typing is actually much more like mathematical logic than static typing is.
However, I recently started to get frustrated by the lack of method overloading (not to be confused with method overriding) in Python. I wanted duck-typing, but I also wanted method overloading, and somehow my brain told me I should take another look at Lisp.
I checked in on Arc, but it didn’t seem like it was going to be ready anytime soon. However, on the Arc website I found a link to Clojure (which I had heard of but never explored). So now I’m trying out Clojure to see if it can do the things I’ve been missing in Python.
-John