#clojure log - Sep 03 2008

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

0:08 Chouser: great!

0:09 jamii: resolveSlot sends the field type off to c++ to find out the number of args the signal has. So the field type has to be exact - there will have to be 10 seperate signalholder classes. But at least it works now

0:10 Chouser: ok, good enough.

0:23 jamii: Chouser: Was it you who was interested in using qt? Do you want the code for this stuff and the qt-repl?

0:31 Chouser: If you want to share it, you should post it to the forum or put it in clojure-contrib or something.

0:32 I'm a general fan of Qt if you have to use C++ -- beyond that I have only vague interest.

0:38 jamii: fair enough

0:39 Chouser: I do wonder if the broad adoption of webkit is going to help push Qt into more places.

9:13 Myoma: hello

9:13 what impact can invokedynamic have on clojure ?

9:15 rhickey: Myoma: possibly not much at all - it is mostly a feature for languages that do by-name single dispatch and thus can leverage per-call-site caching when one call site is typically only used for one 'type'

9:16 Clojure's multimethods are already separated by namespace, and thus most call sites will be genuinely polymorphic

9:17 Clojure caches the signature-to-method lookup results on the multimethods themselves

9:17 Myoma: oh!

9:18 * rhickey would much rather have TCO and tagged numbers than invokedynamic

9:18 Myoma: What is tagged numbers?

9:19 rhickey: Allows an Object reference to be a number (not on the heap)

9:19 using 'tag' bits to distinguish numbers from references

9:19 old Lisp/Smalltalk trick

9:21 Chouser: would that be to reduce memory management overhead? cheaper "new"s?

9:23 rhickey: Lots of benefits to not having each number be an individual object on the heap - no allocation associated with a number, no memory reference associated with using its value - a tagged number is a value

9:23 after the tag bits are masked off

9:24 way faster - CLs and Smalltalks trounce dynamic JVM languages that use boxed numbers

9:24 Myoma: but JVM does have unboxed numbers?

9:25 rhickey: Right now in Clojure you can get locally-unboxed math, which is just as fast as Java, but as soon as you want to pass or return a number it must be boxed

9:25 Myoma: ah I see, that is what I was going to ask .

9:25 Chouser: oh, my understanding was backwards. So this would be storing a number where you'd usually expect to find a heap reference, in order to represent the number itself.

9:26 rhickey: Chouser: right, usually a couple of bits of reference addresses are always unused, due to alignment

9:26 so you put tag bit there saying 'this is really a number'

9:26 Chouser: ah, sure. seems like I've heard of other clever uses for those extra couple bits.

9:27 rhickey: some cost in numeric range - 28-bit fixnums are common in 32-bit architectures

9:37 Chouser: well, we can just write up a patch to Java and submit it, right?

9:37 rhickey: heh

9:38 blackdog: i'm sure i saw john rrose talking about that on the dynlang group

9:38 rhickey: I know these things are on their radar, and now tagged numbers have become important to JRuby...

9:39 blackdog: yes John Rose has an excellent blog entry on it: http://blogs.sun.com/jrose/entry/fixnums_in_the_vm

9:41 blackdog: i always wonder why squeak seems to me to be much more efficient in memory usage than java with an entire envirnment loaded, is that part of the recipe?

9:42 rhickey: blackdog: probably not, as there aren't too many numbers kept around

9:42 and typical Java code does not use boxed numbers at all

9:43 dynamic langs are forced into it because they want unified signatures, which must be Object based

9:44 John Rose on TCO: http://blogs.sun.com/jrose/entry/tail_calls_in_the_vm

9:47 the problem is - when do these things get into JVMs you can use for production work?

9:47 blackdog: well the advent of openjdk will speed things significantly

9:48 rhickey: It will speed up the development, whether it speeds up the integration is TBD

9:48 blackdog: sure

9:49 rhickey: Clojure accepts the compromises associated with targeting today's JVM

9:50 Chouser: ...and isn't dramatically hampered by that, IMHO.

9:52 for example "recur" has merits nearly sufficient to stand on its own, even if it wasn't required for performance.

9:58 rhickey: Long live Chouser - defender of recur! :)

9:58 Chouser: heh

9:58 I guess I'm a bit vocal on recur because I've traveled so far. I thought it was a pretty ugly hack at first.

10:00 cemerick: yeah, recur is very pleasant -- makes tail-recursive algorithms (more) explicit and obvious when you're reading the code

10:00 rhickey: It's funny because I get beat up so often for it

10:00 cemerick: don't let 'em get you down :-)

10:01 rhickey: are you planning on changing delay's equality semantics eventually, so that (= (delay 5) 5) => true?

10:01 * hoeck likes recur too

10:01 * cemerick has a delay fetish :-P

10:02 Chouser: I think it contributes to the general feeling in clojure that it's simple, with very little magic

10:02 hm, with the exception of STM I suppose.

10:03 rhickey: STM is simple to use, no? Compared to locks

10:03 Chouser: But a function call is a function call -- no type lookups, no parameter overloading, no inheritence. If you want any of those types of features, you can bring them in as needed -- explicitly and succinctly.

10:03 rhickey: cemerick: but a delay of 5 isn't (yet) 5

10:05 Chouser: I'm not suggesting we shouldn't have STM, but I can't say it doesn't feel magical. :-)

10:06 rhickey: Chouser: you're onto what I think is underappreciated about OO - there is a lot of accidental complexity

10:06 cemerick: rhickey: yeah, I know, it's a separate value; however, I continue to aim for it to be as transparent as possible. This came up for me when doing some tests, where a fn returns a map whose values might be delayed, and I want to compare it to a known-good map whose values are always strict.

10:06 rhickey: cemerick: the way to treat mixed scenarios is to always force

10:07 Chouser: locks are simple to use in the small. it's only in the aggregate that locks crush all productivity.

10:07 rhickey: Chouser: dosync is as easy as synchronized

10:07 Chouser: rhickey: yeah, Clojure's taught me that. I'd never noticed before.

10:08 rhickey: it's as easy to use, but I feel like I understand all of the very small amount of work that synchronized is doing. I have no such illusion about dosync.

10:09 rhickey: fair enough

10:09 cemerick: rhickey: well, I'd have to cons up a new map (or other collection) in order to use force; I'm looking to do (= some-map other-map), not (= (:key some-map) (:key other-map))

10:09 (which would have to become (= (force (:key some-map)) (force (:key other-map))))

10:09 Chouser: perhaps if I use dosync enough to grok it more deeply it will feel less magical. we'll see. :-)

10:10 rhickey: cemerick: I guess the problem is, if (= (delay 5) 5) works why wouldn't (+ (delay 5) 5) ?

10:11 in a statically typed system one could do that auto-forcing without cost to non-delayed values, but not in a dynamic system

10:11 cemerick: ideally, the latter would, although I understand that that would be impractical

10:15 wlr: am i alone in getting: (ns-publics clojure) => Unable to resolve symbol: clojure in this context

10:16 cemerick: FWIW, my (seemingly constant) mumblings about delays come from my wanting to shoehorn scala's lazy keyword into clojure; certainly a mismatch, but you can call me Sisyphus :-)

10:16 wlr: in fact (ns-* any-ns) behaves similarly

10:16 rhickey: wlr: (find-ns 'clojure)

10:17 cemerick: I imagine that Scala feature leverages its type system

10:18 wlr: ahh, thanks. sorry for the noise.

10:18 cemerick: rhickey: Indeed it does, so that its equivalent of (+ (delay 5) 5) has no cost beyond (+ 5 5)

10:19 rhickey: wlr: no problem. I'd like to make all of the ns-* functions take a symbol as well as a ns object at some point

10:20 cemerick: the key being the lack of cost for non-delayed values. I'd have to add a force around all value expressions

10:20 cemerick: rhickey: sure, and I'd never suggest that -- that's why I said it'd be impractical above

10:21 rhickey: cemerick: maybe some auto-forcing wrapper on the data structure?

10:23 cemerick: rhickey: that would still require a rebuilding of any collections involved, which could be nontrivial

10:23 Chouser: I'm sure people who have made their own runtime systems and assumed they had to have TCO and spent lots of effort getting it right might feel a little snubbed that Clojure doesn't deem that feature necessary. :-)

10:25 cemerick: rhickey: I should say, that an auto-forcing wrapper would be very handy, although not suitable for my current context.

10:25 rhickey: Chouser: one need only to compare the speed of SISC Scheme to Clojure to see the tradeoffs

10:25 cemerick: what are you delaying, if I may ask?

10:26 cemerick: rhickey: in this particular case, a search of a very large rtree for regions that intersect and/or contain another region

10:30 rhickey: cemerick: I see. I certainly thought a lot about transparent delays when designing Clojure, but couldn't figure out a way to avoid the overhead

10:32 cemerick: rhickey: yeah, I understand the difficulty in a dynamic environment. In this particular case, if I were forced to use an auto-forcing wrapper on the collection, that would result in a reindexing of both rtrees (which implement IPC) -- obviously a no-go.

10:32 I mentioned putting together a Delay subclass last week in connection with the delay-as-fn discussion, but wedging auto-forcing equality semantics into the mix is a much more compelling reason to go ahead with that. Those semantics would break things in general use, but they're just right in this case, I think.

10:36 Chouser: FWIW, I benchmarked clojure against SISC before I decided to start using the former professionally -- the results were quite stunning in clojure's favor. Rich's decision to forego reified environments was a good one.

10:39 Chouser: SISC is an unapologetic interpreter, isn't it? Is that how JRuby and Jython are designed as well?

10:42 cemerick: Yeah, they're all interpreted. JRuby was on its way to delivering an AOT compiler, last I looked.

10:42 ...or, just a compiler, I suppose :-)

10:42 rhickey: Chouser: SISC implements Scheme 'purely' on the JVM, and does what it needed to do in order to implement TCO

10:43 JRuby and Jython do a lot of compilation to bytecode

10:44 Chouser: ok, but function calls still can't be pure Java calls like in Clojure because of differences in lookup and inheritence semantics, I'd assume.

10:45 rhickey: you mentioned a few months ago that AOT compilation for Clojure was near the top of your list. Is it still there?

10:45 I have basically zero personal interest in AOT, but the question comes up.

10:45 rhickey: Chouser: function calls can't be real function calls due to TCO

10:47 Chouser: My point is that implementing a pure idea required an interpreter, with lots of consequences. Bigloo & Kawa compile to bytecode but forego TCO as well

10:47 * Chouser googles bigloo

10:48 rhickey: Chouser: the key to AOT is less the concept than the purpose. Is it startup speed? Packaging ease? Running on Android and other platforms with no dynamic classloading?

10:49 Bigloo paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=

10:49 blackdog: does clojure run on dalvik?

10:50 Chouser: yeah, I think a lot of purposes get dumped in there. The most common case I hear for AOT compilation is actually just obfuscation.

10:50 rhickey: blackdog: no, because there is no Java bytecode once you get on the VM, it must all be translated ahead of time

10:50 blackdog: ah ok

10:50 rhickey: Chouser: forgot obfuscation

10:52 I'd like to see some contributions for packaging - the "how do I make a jar?" question comes up often, and is independent of AOT compilation

10:53 cemerick: rhickey: I think many people persistently forget that clojure == java for issues like that.

10:53 rhickey: cemerick: right, but still could be made easy

10:54 cemerick: I suppose -- but short of pointing someone at clojure.lang.Script, what other ground needs to be covered that isn't a prerequisite for packaging a "regular" Java app?

10:55 rhickey: If they want a Java entry point they have to deal with genclass

10:55 cemerick: Oh, I see what you mean.

10:56 Chouser: Speaking of genclass, did you see the incompatibility between it and the new ns macro?

10:57 cemerick: A script or ant task could be put together that would handle all of the genclass stuff in order to bootstrap a "main" namespace...

10:57 rhickey: Chouser: yes, still thinking about that. I realize the not-yet-defined criteria is too precious.

10:57 cemerick: (FWIW, while clojure does work well via ikvm on .NET right now, being able to eliminate dynamic classloading via AOT would make me a lot more comfortable with that deployment path -- appdomains and other fiddly .NET things can sometimes make dynamic classloading dicey)

10:57 Chouser: Perhaps instead of automatically skipping the (refer clojure) if the namespace exists, there could be :dont-refer-clojure flag

10:57 but with a better name.

10:58 rhickey: Chouser: right, that's one of the options, but more likely they want only some of clojure, so maybe :refer-clojure ... with the args of refer, if not present refer all

10:59 Chouser: hm, not bad. maybe allow ":refer-clojure :none" or something, just in case?

10:59 :refer-clojure nil

11:00 rhickey: there is :only already

11:00 I can't imagine having no clojure as a real scenario

11:00 Chouser: good enough

11:00 yeah

11:01 rhickey: cemerick: yeah, I'm amazed by IKVM running Clojure

11:02 cemerick: rhickey: yup, ikvm is a pretty amazing piece of work. Clojure working well on .NET through ikvm is a must-have for us in certain contexts

11:03 of course, I'm keeping a pretty close eye on that, to make sure that things don't go off the tracks in that regard :-)

11:04 rhickey: cemerick: I imagine your experience there would be quite interesting to many, if you got time to write it up as a Wiki entry

11:07 cemerick: rhickey: so far, there's not much experience to tell -- our jar (which includes clojure, a huge pile of Java code, and some clojure we've been adding around the edges) gets dropped into ikvmc, and a DLL pops out the other side that gets backtested against the jar's results across a test suite. We've used the same build and test process with ikvmc for years now, but I suppose that some may not find it so obvious.

11:08 rhickey: do you call .Net from Clojure?

11:09 cemerick: No, we don't call .NET anywhere, and we don't return any instances of .NET classes or types, either.

13:00 rhickey: anyone want to comment on Clojure robustness?: http://groups.google.com/group/clojure/msg/60c7a9aa770dc406

13:18 Chouser: not really. you want a :refer-clojure patch for ns?

14:31 It's seems pretty common to alias clojure.zip as zip, clojure.xml as xml, etc. Any sane way to roll that into the ns macro, or are they better as standalone alias calls?

14:32 oh, nm -- you can use (ns foo (:require [clojure.xml :as xml]))

14:37 rhickey: Chouser: :refer-clojure patch welcome. Do you have robustness issues?

14:39 abrooks: rhickey: Is the individual purely asking about robustness? I suspect he may be asking something of a question of interface stability / consistency as part of asking about using it in production.

14:39 Maybe not. Maybe he is just asking about runtime stability.

14:40 rhickey: abrooks: I don't know, only that it needed answering by someone other than myself :)

14:40 abrooks: Right. :)

14:41 Chouser: not at all, just don't feel credible on the topic. My largest piece of Clojure that has ever had to run with any kind of reliability was about 350 lines. small potatoes

14:41 abrooks: Unfortunately the last several months have only afforded brief moments to poke at Clojure so I'm not in a position to comment.

14:41 Chouser: cemerick answered. I believe him.

14:42 abrooks: Chouser: Oh, I hadn't checked back to see if anyone commented.

14:43 rhickey: I'm interested in any and all usage stories, since I'm frequently asked "what are people doing with Clojure?" and my only answer is "reporting bugs and requesting features" :)

14:43 Chouser: hehe

14:44 cemerick: rhickey: I'm afraid that many people who really, really like clojure will see it as a tremendous competitive advantage, as well as a potential PR liability with those that control I.T. budgets.

14:45 Chouser: I've only ever finished 2 "real" projects in clojure. clojure-log and a tool I wrote to help track house sales listing changes.

14:45 abrooks: Chouser: What about your ICFP08 submission? ;-D

14:46 rhickey: cemerick: I totally understand. I fully expect many early adopters to do so in stealth mode

14:46 Chouser: The first is 190 LOC, the second is the 350 LOC I mentioned earlier.

14:46 abrooks: Chouser: Didn't you rewrite the Movie browser in Clojure at one point?

14:46 Chouser: abrooks: oh, I forgot that! 180 lines.

14:47 abrooks: nope, the movies page is one of many incomplete clojure projects that I intend to finish some day. :-)

14:47 abrooks: :)

14:47 Chouser: I guess I can reply with all this on that thread, for whatever little value it may have.

14:48 rhickey: Chouser: I'm going to do another "what are you doing/what features do you want" message soon. If you don't feel your experience is telling about robustness, no worries

14:50 DrewR: rhickey: I'm curious what you use Clojure for. Do you ever get to actually write Clojure aside from the language itself?

14:50 rhickey: Chouser: if you do :refer-clojure do you want to do :load-resources as well? http://groups.google.com/group/clojure/msg/934d131d1f3b5bfc (might be nothing other than documentation change)

14:51 Chouser: rhickey: sure, I'll look at it.

14:52 rhickey: DrewR: some. My main research work is in trying to make my computer hear (machine listening). Moving that work from C++/Mathematica to Clojure.

14:54 abrooks: rhickey: From Mathematica to Clojure? Interesting. I've never heard complaints about Mathematica (though most of the Mathematica users I've known were scientists and not software engineers). What shortcomings do you see in Mathematica that you'd like to improve on with Clojure?

14:56 rhickey: I think Mathematica is extremely cool, but it is not fast enough to do the bulk audio processing, so I mostly use it for algorithm design and data visualization, with C++ doing the heavy lifting

14:58 the problem is I have had to do a lot of experimentation in the C++ code, which is extremely painful. Using Clojure there gives me something fast _and_ flexible - I expect to use Mathematica mostly for visualization in the end

14:59 DrewR: Are you a PhD candidate or is this corporate research?

14:59 Chouser: any hope of visualization libraries that would be comfortable enough to use directly from a Clojure repl that you'd be able to walk away from Mathematica?

14:59 rhickey: DrewR: no and no. I'm a professional developer and this is self-funded

15:00 DrewR: Ah.

15:00 rhickey: Chouser: definitely - I really haven't looked at the libraries available for Java, but it would be an exciting prospect

15:01 Mathematica's visualization tools are really good though

15:01 Chouser: My brother told me this weekend that his primary language right now is SciLab.

15:02 He likes it, especially hot-loading of code, but that it's painfully slow for heavy computation.

15:02 I don't know much about SciLab, but it sounds like a similar problem space.

15:03 rhickey: Chouser: that's the problem, great for creating the models, but once you need to run 1000 filters at 44kHz...

15:04 especially with any user-written functions - most of the built-ins are very fast C behind the scenes

15:04 abrooks: rhickey: Have you found Clojure able to handle that sort of throughput?

15:05 rhickey: abrooks: It wasn't real-time even in C++, but the difference between seconds and days to get a result I needed

15:06 but yes, Clojure is fast enough, and scales well from experimenting to crunching

15:08 * DrewR rebuilding clojure for first time in ages

15:09 jgracin: If I have a string containing clojure expression, how do I evaluate it from Java? I'm not necessarily interested in the return value.

15:10 rhickey: jgracin: you need to read it, yielding a data structure, then eval that

15:15 jgracin: rhickey, is it with LispReader.read()?

15:15 rhickey, seems a bit low-level

15:17 oh, now I see how repl does it. it seems fine.

15:20 Chouser: It'd be easy to wrap that all up in a thing that eval's strings, except that in general that approach is Discouraged.

15:22 jgracin: I'm writing a maven plugin which will evaluate user-provided expressions.

15:22 Chouser: ah

15:41 jgracin: I'm getting java.lang.ExceptionInInitializerError at clojure.lang.Namespace.<init> if LispReader.read(...) is the first thing I do. Putting RT.init() before makes it work, but warns me that I don't have to use it.

15:46 Chouser: rhickey: currently the references in "ns" are evaluated in order, and each may be listed multiple times. Should I keep that behavior?

15:59 rhickey: Chouser: yes, please

16:02 jgracin: could you paste your failing example?

16:06 lisppaste8: jgracin pasted "No RT.init exception" at http://paste.lisp.org/display/66262

16:24 rhickey: jgracin: fixed (rev 1015) - thanks for the report!

16:24 jgracin: rhickey, cool. thanks.

16:34 ozzilee: Can anyone point me towards something explaining the current state of the art in clojure libraries, with use and require etc?

16:35 Ah, wow the Wiki has expanded since I last looked at it :-)

16:38 * abrooks takes a look at the wiki agian

16:38 abrooks: again

16:39 ozzilee: Is there a standard procedure for packaging libraries that require jar files?

16:40 Specifically I want to write a library that wraps the Apache Commons HTTP lib.

17:01 Chouser: I think the answer would be the same as it would be for Java.

17:02 Could you use maven to name prerequisites for your package?

17:03 Or maybe it's best just to stuff everything you need into a single giant jar.

17:03 Clearly I don't really know. :-)

17:18 ozzilee: Chouser: Right now I've got a bunch of jars in /usr/local/share/java/ that I require with add-classpath, which makes things a little more sane for scripts and exploratory programming and such. Maven is quite a bit heavier than what I'd like.

17:59 cemerick: ozzilee: we always ship a single jar that contains all of the contents of the dependencies we require (obviously we only depend on libraries that have permissive licenses, but that may not apply to your work)

18:01 ozzilee: cemerick: I'm not so much concerned about shipping as I am with writing stuff for myself to use. I'd like to be able to say (require 'mylib) in a fresh repl and have my lib pulled in along with all of the jars it needs, without having to dick about with the classpath.

18:02 That said, I'd probably got with one big jar for shipping as well.

18:03 cemerick: ozzilee: I agree wholeheartedly. FWIW, we use ivy to manage our dependencies within our ant build, and it's been very pleasant to work with (and a *lot* easier than maven, IMO)

18:06 ozzilee: Well, time to get out of here, thanks for the help.

18:31 sirk390: hi, I'm just starting clojure. Does anyone understand why (load (new java.io.StringReader "hello")) throws an Exception "java.lang.Exception: Unable to resolve symbol: hello in this context" ?

18:32 shoover: Hi. Clojure is seeing a symbol hello, not a string "hello". Are you hoping for a string? Try embedding quotes inside the string, like "\"hello\""

18:33 sirk390: yes I was hoping for a string ^^

18:34 Ok, that work much better now, thanks

18:35 shoover: No problem

19:00 rhickey: ozzilee: I think you can put jars in a dir and specify it with -Djava.ext.dir= and not have to put each on classpath individually nor use add-classpath

20:38 Chouser: rhickey: how do you suggest keeping track of whether clojure's been loaded into a namespace?

23:12 arohner: (doc contains?) says that contains? takes a map, but it appears to work on vectors. Am I missing something, or are the docs incorrect?

23:15 ah, it's because vectors are functions of their indices

23:18 Chouser: right, but it won't tell you if a particular value is in a vector.

23:19 arohner: I was using [0 1 2 3] as my test case

23:19 so it worked exactly because the values equaled their indices

23:20 with [:a :b :c] it's a lot more obvious that it didn't work

23:20 Chouser: ah, yep.

23:58 Myoma: how does one read to understand clojure compiler source code?

23:59 I just tried to skim it but that does seem a bit futile

23:59 arohner: are you trying to understand the whole thing or something in particular?

23:59 Myoma: well I would like to roughly understand it all

Logging service provided by n01se.net