#clojure log - Mar 27 2008

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

9:52 Chouser: rhickey: if you don't mind sharing, is that corporate or academic work?

9:53 rhickey: I work for myself

10:05 Chouser: you've found customers for your work, or you have sufficient capital you don't need customers yet?

10:06 I don't mean to pry, I'm just very curious.

10:06 rhickey: that's something I can't discuss

10:06 Chouser: ok

10:07 well, I'm sure glad you've taken the time to work on Clojure. :-)

10:07 cgrand: Maybe it's old news to you but I just noticed this morning that the Eclipse debugger is able to step into clojure code (highlighting current line etc.).

10:08 rhickey: me too - I finally get to develop software the way that I want to, and have huge libraries at hand

10:08 cgrand: can you set breakpoints? last time I tried I couldn't

10:09 Clojure emits the right stuff for debugging, local variable names etc

10:10 Chouser: rhickey: that's how the stack traces from exceptions work, right?

10:11 rhickey: that's how the line numbers get in there, but there's more to it, you need a SourceDebugExtension to map to the source files, which Clojure also emits, and locals entries in the bytecode, also there

10:12 cgrand: Yeah I saw that you emit smap strings. With a rudimentary plugin (never played with eclipse dbugger api) I can set breakpoints

10:12 rhickey: cool

10:13 cgrand: which source files? Clojure emits java source files?

10:14 rhickey: no the smap is a mapping from Clojure to non-existent Java files, which is the bridge

10:14 Full Clojure plugin for Netbeans is coming, if you haven't seen this: http://groups.google.com/group/clojure/msg/4dfb4566c2187704

10:14 cgrand: I saw

10:15 rhickey: they had to do some work to get Netbeans to debug Clojure

10:17 I'd love to see Eclipse support too

10:19 cgrand: The editor framework is a bit complex...

10:19 rhickey: I guess Cusp would provide good editing support, but native debugging is preferable to custom protocol

10:20 does Cusp do debugging anyway?

10:22 cgrand: Not sur but I think so

12:01 sanity: anyone here familiar with SISC? http://sisc-scheme.org/

12:02 Chouser: someone recommended it to me when I was looking for a JVM lisp. Of course I ended up using Clojure instead, so I know nothing useful.

12:04 sanity: Chouser: i only ask because a friend of mine created it

12:05 Chouser: although i think, unlikely Clojure, it is interpreted, rather than compiled

12:05 s/unlikely/unlike

12:05 Chouser: interesting.

14:27 wabash: If I have a multicore cpu, like 8 cores, how would I parallelize a clojure app? Would I start a separate JVM on each core? And, would threads on separate JVMs be able to communicate via some sore of message passing?

14:27 like Erlang?

14:28 rhickey: A single JVM can address multiple cores

14:28 Clojure agents will get distributed among them

14:28 wabash: rhickey: wow, sweet!

14:29 I'm sorry to ask such basic questions. I just found out about clojure like, last week. The more I find out, the more interested I am.

14:29 rhickey: Clojure fns are Callable and Runnable, and will work with the java.util.concurrent.Executors framework as well

14:30 wabash: What is the clojure way of thread communication? And, are threads expensive like Java or are they cheap like Erlang? is msg passing expensive like sockets or cheap like shared mem?

14:30 rhickey: Are all Java libs and classes basically available?

14:30 rhickey: Erlang processes are not threads

14:31 they may or may not map to threads depending on how you have Erlang SMP set up

14:31 wabash: rhickey: Sorry, I got semantically lazy.

14:31 rhickey: Erlang w/o SMP mode runs on one core

14:31 but does its own scheduling

14:32 wabash: Right, I just misused a term.

14:32 rhickey: Clojure agents use a fixed thread pool on message sends

14:32 the pool size is related to the number of cores

14:32 wabash: linearly related?

14:32 rhickey: Agent state can be read directly w/o messages

14:33 agent state gets changes via actions, functions of their state that get executed asynchronously

14:33 it's a different model than Erlang's

14:33 wabash: How do you read stat directly? Is it some sort of polling? is an agent like a process?

14:33 Oh, I see.

14:33 It's a very different way of thinking about hings.

14:33 is the state reading quick between cores?

14:34 rhickey: Erlang's can be transparently distributed, but getting at state requires send/ack & blocking

14:34 wabash: Can clojure be transparently distributed?

14:34 rhickey: Clojure is a single memory space, reading agent state is like any other object access

14:35 No, I don't believe in transparent distribution, but may add distributed actors at some point

14:35 wabash: Is the single mem space a problem for concurrency then? How is that abstracted? Erlang seems so elegant......

14:36 rhickey: Clojure agent states are immutable

14:36 wabash: I get it now.

14:36 Thank you.

14:36 rhickey: immutable objects are no problem for multicore

14:37 wabash: rhickey: I appreciate your time. Learning something new, I'm always faced with the paradox of what to ask when I don't know enough to formulate questions.

14:37 rhickey: I don't think send/ack message pairs to read state is elegant at all, just a necessary evil of distribution

14:37 vs RPC

14:38 wabash: RPC is more elegant than message pairs?

14:38 rhickey: no, message pairs is less evil

14:38 distribution is simply very hard

14:39 why incur the complexity unless you really are distributed?

14:39 wabash: premature scalability, I suppose.

14:40 rhickey: but the constant factors are different - you can make it look the same, but the same payloads and chatter rate that worked locally can totally fall down when distributed

14:40 wabash: Yes, I understand and agree.

14:41 That's kind of what I was getting at with spreading over cores; will the constants be high for communicating between (threads/processes) on different cores.

14:41 rhickey: http://research.sun.com/techrep/1994/smli_tr-94-29.pdf

14:41 wabash: In Clojure, then, you think in terms of "agents"?

14:41 thank you for the link. I will read it after lunch.

14:42 rhickey: agents are one of several tools for MT programming. There are also refs and STM

14:43 wabash: I see. If I may ask an over-general question, if I run Clojure on an 8 core machine, will the communications overhead be small enough that the distribution will be even? Will I have to worry about which core is closer to which?

14:43 Or if I have a small cluster of separate machines, will I have to adjust my algorithms to match the architechture?

14:44 rhickey: that totally depends on the granularity of your work units. Overhead/work is what matters

14:44 wabash: I see.

14:44 rhickey: separate machines is a totally different architecture in my book, and where Erlang's model makes the most sense

14:44 wabash: Right.

14:45 rhickey: I'm thining about doing distribution for Clojure using Shoal: https://shoal.dev.java.net/

14:45 wabash: You seem to have a lot of experience with different languages/models. Is your work research oriented? Or are you lucky enough to have a corporate job doing all these things?

14:46 rhickey: I'm just old

14:46 :)

14:46 But lots of MT in my experience

14:46 wabash: MT?

14:47 rhickey: multithreading

14:47 wabash: Got it.

14:47 Thanks for the discussion. It is useful, significant to me.

14:47 rhickey: sure

14:48 wabash: I went to an average school, got wrapped up in some programming jobs, got ejected, dejected, discouraged, but now have discovered higher level languages, and FP, and think that programming is fun again, a bit.

14:48 well, must have lunch. thanks for sharing your knowledge and experience.

16:14 jgracin: hi all! this place is getting more crowded with each day. :-)

16:14 rhickey: :)

16:15 * mbishop shoves jgracin out the door

17:24 Chouser: I'm not sure this is the right question, but if I wanted to make something that acts exactly like a hash-map, but also generated some Actor dispatches, how would I do that? "implements"? multi-methods?

17:26 ok, that's not quite right. I want something that acts like ref. You can call set-ref on it, and it does that but also it dispatches and Actor function.

17:26 If this were some other language, I'd sub-class Ref.

17:28 rhickey: why not just use a map as the state of an agent?

17:28 or of a ref?

17:29 agent actions anf ref alter/commute functions can be multimethods

17:35 Chouser: I'm right at the edge of my understand here, so bear with me... I've got a function that calls (ref-set db (assoc ...)). It currently expects db be to a ref to a map, but I'd like to also allow db to be one of these new things I'm making. Are you suggesting that I change my set-ref to be (alter db foo) and then make foo a multi-method that can handle either standard ref-maps, or my new thnigs?

17:36 rhickey: right, in your first case, (ref-set db (assoc... you could have used (alter db assoc...

17:36 now swap foo in for assoc...

17:36 make foo a multimethod that can distinguish the things you might out in db

17:37 put

17:37 are your new things going in refs?

17:37 Chouser: ok, I think I follow.

17:38 I don't *think* so. I think they will act like refs.

17:38 rhickey: hmm...

17:38 Chouser: Maybe I didn't understand the question.

17:38 rhickey: why can't they be refs?

17:39 Chouser: I'm thinking I'll make a thing that acts like a ref, but whenever it gets set it will ask an actor to write its new value out to disk.

17:40 rhickey: I see, I've thought about connecting on-commit and validate fns to refs as a general solution for that kind of thing

17:41 but refs are special, don't try to derive...

17:41 you can have your mutating fns send an agent action, but not automatically... could use a macro to define the mutating fns

17:43 Chouser: but the solution you outlined above should still work, right? I'll use (alter db db-assoc ...), and db-assoc will either just assoc, or assoc and dispatch.

17:44 This will be my first agent.

17:44 And my first multi-method. ;-)

17:44 rhickey: oh boy

17:45 yes, if db-assoc sends an action, that will only happen if the transaction commits

17:47 Chouser: whew, good point. I hadn't thought about the assoc getting rolled back. Ok, I'll try to do this.

17:48 rhickey: that's the beauty of the agent/transaction coordination, there's no other way to get run-once-only side-effects in a transaction

22:14 Chouser: Ok, I don't think I really need a multimethod.

22:14 Is there any way to refer to an agent action, or do I have to wrap the call to it in a function.

22:15 rhickey: ?

22:15 agent action are functions, or are you talking about the agent's state?

22:17 Chouser: hm, no I was misunderstanding how they work. Let me read that docs page again...

22:18 rhickey: you can make one-off agents just to execute side-effects:

22:19 (send (agent nil) (fn [_] (side-effect)))

22:20 Chouser: send is the same as ! ?

22:20 huh. ok, I think that's exactly what I want.

22:21 rhickey: yes, send is the new !, there is also send-off for potentially blocking actions

22:21 Chouser: oh, is ! going away?

22:21 rhickey: yes ! is going away

22:22 try it

22:22 it's gone

22:22 Chouser: heh, ok.

22:45 should I avoid multiple deref in a transaction?

22:46 I've got @foo all over the place, but I could put it in a local symbol instead.

23:16 rhickey: multiple deref of refs is no problem, should be same result throughout unless transaction itself changes ref

Logging service provided by n01se.net