#clojure log - Apr 23 2008

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

2:23 patchwork: Hi guys. So I'm starting an application in clojure, and I'm wondering how deep to design the granularity of the relationships. In particular, it would be nice to have an agent for each sliver of the behavior, but I am wondering how reasonable having large numbers of agents is. Is there some guidelines as to about how many agents I should have at any given time?

12:41 jteo: why isn't there tail call optimization?

12:42 rhickey: because the jvm doesn't support it natively, and Clojure uses native JVM calling

12:43 jteo: ah.

14:42 rhickey: Chouser: any progress on the destructuring?

15:11 Chouser: rhickey: yeah, I have my most recent suggestion working.

15:11 I just haven't had time to write it up.

15:11 rhickey: great

15:12 Chouser: It's not quite as succinct as the #{} format was, but it feels "cleaner" and is definitely more complete.

15:12 rhickey: I agree

15:14 Chouser: one issue I ran into was defaults when destructuring a set

15:14 rhickey: ?

15:15 Chouser: 'get' takes an optional 3rd arg, but using a map or set as a function does not.

15:15 http://n01se.net/paste/0FZ -- Extra features for map destructuring

15:16 so if you provide defaults (which don't make sense for sets anyway), my patch uses 'get', otherwise it uses the map or set as the function.

15:20 rhickey: ok

15:20 Chouser: here are some test cases that are working: http://n01se.net/paste/WNk

15:21 wabash: Does clojure code compile to native code via the Java HotSpot server thingy?

15:22 rhickey: yup

15:22 wabash: sweet.

15:22 rhickey: You mentioned a wiki for contributing. Were you referring to the wikibook, the wikipedia entry, or a wiki on the Clojure site?

15:23 rhickey: the wikibook

15:23 wabash: ok. thanks.

15:24 rhickey: http://en.wikibooks.org/wiki/Clojure_Programming

16:10 vincenz: rhickey: I have a questions about macros. How do you make sure that macro-code runs at 'compile'time?

16:15 rhickey: macros always run at compile time, but the code they produce runs at runtime

16:25 Chouser: rhickey: is there anything else you want from me for the destructuring patch?

16:31 rhickey: just looking at it - looks fine, although the original was written before variadic assoc/dissoc, this could be cleaner with them (e.g. apply merge ... apply hash-map...

16:32 * rhickey should make conj variadic too

16:32 Chouser: ok. I can look at that tonight if you want.

16:33 or you can do it; I don't care.

16:34 rhickey: I appreciate what you've done, I can take it from here unless you want the exercise :)

16:34 Chouser: I've got other things I can do.

16:35 rhickey: ok :)

16:35 Chouser: On the other hand if you want to do something I *can't*, that'd be fine too.

16:35 rhickey: thanks

16:35 Chouser: hey, thanks for letting me help. :-)

16:35 rhickey: I'm working on Java-less Java integration

16:36 No RT.init

16:36 Defining named classes right into bytecode

16:36 aceess to protected members and super

16:36 classes can have agent or ref state

16:36 runtime redefinition of methods

16:37 auto-load of user.clj from classpath

16:37 Chouser: see, you shoudl do that and let me monkey around with refactoring apply hash-maps

16:38 would all that help someone derive from PersistentHash in order to, for example, provide a default value that would be returned instead of nil?

16:38 rhickey: that's on the todo list anyway, but probably

16:39 right now there are a lot of cases in which you have to write a little Java

16:39 for main()

16:39 servlets

16:39 working in a framework like Netbeans or Eclipse

16:39 Chouser: Banishing that last 20% is probably hard.

16:40 rhickey: I think I've got it figured out, and half-implemented

16:40 for instance RT.init() is no longer needed

16:40 and Clojure loads user.clj from the classpath at boot time

16:41 much of the proxy stuff will be reused

16:43 Chouser: "classes can have agent or ref state"! cool.

16:44 rhickey: Basically they'll have final state, so if you want mutation you'll have to do it the Clojure way

16:44 vincenz: rhickey: how do you make sure that compiletime code doesn't do "runtimey" things?

16:44 rhickey: vincenz: I don't

16:44 Chouser: vincenz: it *can* do runtimey things. That's what's so cool.

16:45 rhickey: but it does require care and self-discipline

16:45 you can make a mess with macros

16:45 Chouser: I guess you don't want a blocking network call during macro expansion.

16:45 vincenz: rhickey: what about a macro requiring code from a file that has usage of macros/

16:45 rhickey: Chouser: :)

16:46 vincenz: Clojure generally has a define-before-use policy, same applies to macros

16:46 Chouser: anyway, re: destructure reduce/apply/merge/dissoc/apply/hash-map/mapcat can become reduce/apply/assoc/dissoc/map - remember you can assoc pairs onto a map

16:48 Chouser: ok

19:18 leafw: hi all

19:19 rhickey: welcome

19:19 vincenz: rhickey: what did you use as paper for your stm implementation?

19:19 rhickey: I read a _lot_ of papers

19:19 leafw: question: is there an equivalent class to jython's PythonInterpreter, to which one can handle code as String objects and evaluate them within the context of a specific interpreter instance?

19:19 if there is, it's not evident to me ...

19:20 rhickey: leafw: Clojure is compiled, so there isn't exactly a notion of interpreter instance

19:20 leafw: !

19:20 and the REPL.java class?

19:21 rhickey: But Compiler.eval() is a full compiler/evaluator that takes Clojure data structures

19:21 you can get from Strings to Clojure data structures using LispReader.read()

19:21 leafw: thank you

19:21 rhickey: or build the data structures directly using the Clojure Java library

19:23 vincenz: Clojure's STM is not exactly like any one I read about

19:23 leafw: thanks for the tips. I have somewhere to start now.

19:23 rhickey: It has no global lock

19:23 vincenz: rhickey: I think haskell is lockfree as wel

19:23 rhickey: but is not lock-free either

19:23 vincenz: oh

19:23 distributed locking?

19:24 rhickey: I don't believe that lock-free is efficient, there are good papers that indicate it's not

19:24 vincenz: IIRC, lock-free only works well for optimistic lazy stm

19:24 rhickey: Some locking is better than spinning optimistic retries in my book

19:25 vincenz: do you do eager or lazy?

19:25 rhickey: Locks are eager, but there is barging

19:25 also timeouts

19:25 * vincenz is afraid he's not familiar with that term

19:26 rhickey: waking up from a lock before released

19:26 based upon a time

19:26 vincenz: Trying to find out what an easy system would be

19:26 rhickey: STM is not easy

19:26 vincenz: well everything is an axis

19:26 there's gradations

19:27 rhickey: The easiest is to number your resources, work optimistically, then grab the locks in order to commit

19:27 vincenz: so that would be lazy then

19:28 rhickey: right

19:28 but there are so many variations and tradeoffs

19:28 * vincenz nods

19:28 vincenz: I wonder if you could work with a distributed clock

19:29 rhickey: ?

19:29 vincenz: e.g. if you have a generational GC, use one clockc per generation

19:29 rhickey: I don't see the interaction with GC

19:29 vincenz: well you have more clocks

19:30 so less conflicts on those clocks

19:30 rhickey: what's a clock?

19:30 vincenz: I thought lock-free worked with a clock

19:30 you check the data you're writing to after a lazy eager system is still the same clock time for the values you are modifying

19:30 rhickey: lock-free uses any CAS-based mechanism

19:31 but there are many different policies for timestamping the world

19:31 vincenz: in essence you're trading locks for clocks?

19:32 rhickey: is there a reason you chose not to support orElse/

19:32 rhickey: no, they are 2 different things - you can use CAS to toggle flags that effectively 'lock' a resource for one transaction without using a blocking lock - you have to distinguish lock-free from non-

19:32 blocking

19:33 i.e. soft locks don;t

19:33 block threads

19:33 vincenz: no orElse because I don't think it's a good idea

19:34 vincenz: rhickey: why is that?

19:34 (out of curiousity)

19:34 rhickey: well, it requires read-tracking for one

19:34 I don't like mixing the coordination with the transaction

19:34 vincenz: btw, just a side question

19:35 (let ((a @ref)) (sync (update ref ...))

19:35 and

19:35 (sync (let ((a @ref)) (update ref ...))

19:35 are different, right?

19:35 the second guarantees that it occurs in the same timeslice/

19:35 rhickey: yes, the first is a wild-read

19:37 vincenz: alright, thanks

19:39 ooxwo: Hi everybody. I'm starting on a clojure application and I am wondering how fine-grained to make the abstraction. In particular, for my domain it would be nice to make an agent for each atomic element, but I am wondering how feasible it is to use large numbers of agents. Is there a guideline as to about how many agents I should have running at any given time? Is there a way to unite several agents on a single thread? Thanks for your

19:39 help.

19:40 rhickey: what are large numbers?

19:40 ooxwo: like 10-100 thousand

19:41 rhickey: no problem. each agent doesn't get its own thread

19:41 ooxwo: or more possibly, depending

19:41 Really? Ah okay

19:41 rhickey: the actions run in a thread pool, idle agents use no cpu at all

19:41 ooxwo: That is good news, in my trial runs there seemed to be a one to one mapping between threads spawned and the number of agents I had created

19:43 Have you stress tested the limit of number of agents? Just curious if there is a general ceiling.

19:43 rhickey: you should be limited only by memory, but make sure you use send and not send-off or you will run into thread consumption problems

19:45 ooxwo: Ah okay thanks for the tip. I was using send-off so maybe that is why there was a thread for each agent.

21:41 Chouser: http://n01se.net/paste/IiF -- Tighter implementation of map destructuring features.

21:41 rhickey: I don't think that's exactly what you had in mind, but I think it's good.

21:42 Do what you want: take it, advise me, or do it your own way.

22:26 rhickey: Chouser: that looks great - thanks!

22:34 Chouser: np, thanks for letting me help.

23:29 (get #{:a} :a :dflt) ==> :dflt

23:30 hm. maybe that does make sense.

23:30 oh, yeah, ok. I'm fine now.

23:31 landonf: Lisp's propensity for mind-boggling density still throws me for a loop.

23:31 Feels like there a character drought and we're all conserving. Save a tree, type fewer characters.

23:32 Chouser: heh

Logging service provided by n01se.net