#clojure log - Sep 26 2008

The Joy of Clojure
Main Clojure site
Google Group
IRC
List of all logged dates

7:20 leafw: is zipmap lazy?

7:33 fyuryu: leafw: after looking at the definition, I don't think so

7:33 leafw: it returns a map built in a loop/recur

7:34 leafw: thanks.

7:39 "No matching ctor found" means "you can't instantiate an abstract class". Could use some makeup ...

7:40 well, actually, it's true that the abstract class didn;t have that constructor.

7:48 Chouser: You also get that error if you provide wrong argument types to the constructor of a concrete class.

7:48 as you might expect.

9:57 drewr: Is there a way to tell how much memory an object is using?

10:10 fyuryu: drewr: if you mean something like sizeof() in C: not really, almost everything is a reference

10:11 drewr: I didn't know if Object had some clues to where I could do some investigation.

10:12 I'd really like to find out how much memory a collection is using so I can determine the optimum number of rows to return in a db query.

10:15 ozzilee: drewr: http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Runtime.html Does that help?

10:16 I don't know if anything in there will do much good.

10:16 drewr: It's not object-specific, but it's helpful. Thanks.

10:17 ozzilee: Just calling the freeMemory method from Clojure seems to allocate memory, so you'd have to account for that, and who knows what else.

10:17 Probably there's a better way.

11:08 drewr: Is there something wrong with conj'ing a map in a commute?

11:08 hash

11:08 Er..

11:08 (def pool (ref (sorted-map))) (dosync (commute pool conj {(:id @agt) agt}))

11:09 I get a ClassCastException: clojure.lang.PersistentHashMap because of the map after the conj.

11:09 Agent state looks like {:id 123, :conn jdbc.Connection...}.

11:11 Chouser: conj on map wants a vector I think

11:11 drewr: wm> (conj {} {:foo 1 :bar 2})

11:11 {:bar 2, :foo 1}

11:12 This even works:

11:12 wm> (conj (sorted-map :quux 3) {:foo 1 :bar 2})

11:12 {:bar 2, :foo 1, :quux 3}

11:12 I can't figure out what's unique about my situation.

11:12 Chouser: huh, I thought you needed "merge" for that. hm...

11:14 ozzilee: drewr: It looks like it's saying that pool is a PersistentHashMap, not a Ref.

11:15 Are you maybe doing (commute @pool ...) ?

11:15 Chouser: If I do: (def agt (ref {:id 123, :conn "aconn"})) ...and then run your example, it works fine for me.

11:16 or if I do (agent {...}) instead of (ref {...}), it also works.

11:16 ozzilee: This is a prime example of where clojure needs better error messages :-)

11:17 drewr: ozzilee: You aren't kidding. I can figure out that it's the conj that's complaining, but that's about it.

11:17 Chouser: there's some difference remaining between your real code and what you've posted here, since what you've posted works fine.

11:18 lisppaste8: drewr pasted "commute ref" at http://paste.lisp.org/display/67484

11:19 drewr annotated #67484 with "Actually, I'm using conj" at http://paste.lisp.org/display/67484#1

11:22 drewr: Call it like (add-agent *agent-pool* (make-agent conn)).

11:23 lisppaste8: drewr annotated #67484 with "better summary" at http://paste.lisp.org/display/67484#2

11:25 drewr: Am I off base here with the general idea?

11:25 lisppaste8: drewr annotated #67484 with "exception" at http://paste.lisp.org/display/67484#3

11:28 Chouser: what happens if you use a hash-map instead of a sorted-map?

11:30 drewr: Hm, that works.

11:31 Chouser: the clue is the PersistentTreeMap calling compare, and then failing

11:31 drewr: Yep.

11:31 Chouser: all the keys in a sorted-map have to be comparable to each other.

11:32 drewr: In this case they're integers, which should be straightforward.

11:32 That's why I threw in a sorted-map. I might as well have them in order when I'm looking at it.

11:33 Chouser: you can dump out the hash afterwards and see they's nothing unusual in there?

11:33 drewr: Yep, here's from an earlier version:

11:33 {1 clojure.lang.Agent@bad094, 2 clojure.lang.Agent@d3cae0, 3 clojure.lang.Agent@ac5024, ... }

11:34 That might have been before I used a sorted-map though.

12:47 ozzilee: drewr: java.lang.instrument.Instrumentation/getObjectSize

12:48 Can't get it to work here, though.

12:48 * ozzilee leaves for lunch

13:49 aperotte: Hello everyone, I had a quick question about agents

13:51 If you have an agent with a collection in it, and you want to change an element of that collection, does the collection need to be a collection of refs and does the change have to happen in the context of a transaction?

13:53 ozzilee: aperotte: Hopefully someone more knowledgeable will chime in, but I don't believe so.

13:54 aperotte: ok, thanks ozzilee. I have just started experimenting.

13:54 ozzilee: http://clojure.org/state I think the second paragraph under "Concurrency" will explain things.

14:00 avida: aperotte: collections are immutable, so you need to assign a new collection to it using send

14:01 shoover: aperotte: An agent's value can only be changed by sending an action to the agent. The action is passed the agent's current value as an argument, and then you can do whatever you want with that argument. Whatever the action returns becomes the agent's new value.

14:02 If the value is a collection containing Refs and you want to change those Refs, you'll need a transaction around the Ref changes

14:03 avida: aperotte: (def ag (agent '(1 2 3))) (send ag #(conj % 4)) @ag

14:03 aperotte: clojure that'll insert a 4 to the list (1 2 3) and assign it to the agent

14:03 aperotte: fantastic

14:04 that's exactly what I wanted to know

14:04 oh, one more thing

14:04 is it possible for two agents to share a collection?

14:05 avida: aperotte: yes, since they are immutable anyone can share a collection without worrying aboutchanges

14:05 when you send a change to an agent, you actually create a new collection (although efficiently shared behind-the-scenes)

14:05 aperotte: ahh, but if I make "changes" to the collection in one agent

14:05 it won't show up in the other

14:05 avida: so two agents can share one collection, but when you make changes to one agent, you actually end up with two different collections

14:05 aperotte: I see

14:06 Then, if I have two agents sharing a collection of refs, and change the refs within a transaction, will the changes show up in both?

14:08 avida: i think the refs stay the same, only the values they refer to have changed, so if your collection is of refs, then the collections wont change, however you can reference the changed values of the refs

14:08 aperotte: ok, I think I'm going to try it out

14:09 thanks avida, thanks shoover, thanks ozzilee

14:29 It worked ... albeit a bit of a round about way of getting at what I wanted

14:29 (def x [(ref 1) (ref 2) (ref 3)])

14:29 (def a1 (agent x))

14:29 (def a2 (agent x))

14:29 @(get @a1 1)

14:29 (dosync (ref-set (get @a1 1) 10))

14:29 @(get @a1 1)

14:29 @(get @a2 1)

14:31 avida: yep, you never changed the values of a1, a2 so you would always see the same refs

14:32 aperotte: The aim is to have the agents operate on themselves in this way asynchronously and independently

14:33 I'm not sure that this is correct, but this might be exactly the kind of thing clojure was made to avoid

14:36 avida: aperotte: ops on the agents are async but you're updatings refs which are synchrnous

14:37 aperotte: if I used send to tell an agent to (dosync (ref-set ... , that would make it asynchronous right?

14:37 avida: you could read from the refs, and change the refs themselves in the collection async

14:38 aperotte: how could I read and change the refs async without using agents?

14:39 avida: i think what you are doing is good, im not thinking this through all the way

14:53 shoover: updating refs in an agent action is asynchronous as far as the rest of the program is concerned. from the perspective of the action, its own changes to the refs are synchronous

14:54 * ozzilee is completely lost

14:54 * drewr is glad he's not the only one :-)

14:56 shoover: sorry. aperotte: if you tell an agent to (dosync (refset ..)), then yes, that makes an asynchronous change to the ref

14:57 but within the action that executes the ref-set, you'll see the new value right away

14:58 aking: Looks like Rich's slides from the jvm summit are up: http://wiki.jvmlangsummit.com/pdf/27_Hickey_clojure.pdf

14:59 lisppaste8: shoover pasted "Agent sees new value" at http://paste.lisp.org/display/67493

15:02 aperotte: thanks guys

15:02 shoover: That paste doesn't prove the async part of the above statements, but we'd need a real problem to solve for that

15:07 aperotte: The motivation behind this is to write a machine learning prototyping system where each element/layer contains and can operate async on an input vector to create an output vector that is shared with another element/layer as it's input layer (a connection)

15:08 the last "layer" should be "vector"

15:10 I wanted to make the elements operate async by using agents, but then the shared data that represents the connections between the elements becomes problematic

15:10 ozzilee: aperotte: Is this something like Cells? http://common-lisp.net/project/cells/

15:12 shoover: You should be able to get what you want by sharing refs. If you don't like that you may want to check out the agent watchers that were added recently in svn

15:14 joubert: rhickey: I started looking at the lib functionality that is now part of clojure proper. Will (in-ns) be deprecated since (ns) appears to supersede it?

15:14 aperotte: ok, thanks

15:15 ozzilee: I hadn't used cells before, but it sounds like something very similar

15:27 Chouser: joubert: no, in-ns is still recommended if you want to switch to a namespace without changing it at all.

15:28 joubert: chouser: ok, why the different usage of symbol vs. non-symbol?

15:28 (also, in-ns creates a namespace if it doesn't already exist)

15:29 (which seems to me to "change" a namespace :-)

15:29 Chouser: in-ns is from a kind of "lower layer" which also has things like use, require, import, etc -- all of which are functions instead of macros

15:30 each of those will continue to exist, I think, but if you can use (ns) instead that's preferred.

15:30 well, yeah, I guess in-ns does create the ns, but it doesn't muck about with the imports, referred symbols, etc.

15:32 joubert: ok, so when "declaring" a namespace, (ns) is now preferred over (in-ns), with the latter really more geared towards (switch-ns)

15:32 ozzilee: aperotte: There was some discussion about Cells on the mailing list (sorry, was afk): http://groups.google.com/group/clojure/browse_thread/thread/d79392e4c79f8cde#

15:32 Chouser: right, you've got it.

15:33 drewr: Is there a Java standard for JDBC -> XML serialization?

15:34 Like, taking a resultset and saving the text to a file?

15:35 joubert: cool; do you think it worthwhile to normalize the way namespace names are passed to these 2 forms? either always as symbol or as non-symbol?

15:35 aperotte: ozzilee: thanks, this might be exactly what I'm looking for

15:36 Chouser: joubert: probably not. I like that (ns) uses unquoted symbols -- uncluttered. But that means it has to be a macro, which can be clumsier to use programmatically if you're doing something really unusual.

15:37 joubert: but (ns) is just a wrapper around the underlying functions, which must have their args quoted since they are functions.

15:43 drewr: I keep getting java.lang.IncompatibleClassChangeErrors, but Clojure doesn't tell me what classes were involved in the exception.

15:45 Chouser: Huh, when I try to put the wrong type of key into a sorted hash, I get: java.lang.ClassCastException: clojure.lang.Keyword cannot be cast to java.lang.Integer

15:45 drewr: Chouser: Sorry, different problem :-)

15:46 I knew how to fix this one; just frustrated at the error message.

15:46 Chouser: oh, IncompatibleClassChangeErrors is not ClassCastException. I misread that, sorry.

15:50 * ozzilee might have to dig into Clojure tonight and work on error messages instead of putting a server together.

15:56 drewr: No complaints here. :-0

15:56 :-)

18:37 achim_p: hey. i began playing around with that jar-classpath-dependency-download-thing i talked about recently. here it is, for those interested: http://www.bitbucket.org/achimpassen/clojure-ties/

18:38 it's very much unfinished and untested, but i ran it successfully at least once. this was when i decided it needed a logo ;) raises the bar for absurd lambda iconography considerably.

18:39 any opinions on the approach in general? how do you deal with dependencies? i feel the existing java frameworks are way to bulky for what i want to do ...

18:40 Chouser: that's a fantastic logo

18:50 hsyn: hi there

18:54 Chouser: hsyn: hi

18:56 hsyn: i am very new clojure and also to lisp so i have allready problems with the getting start tutotial for java programmers as described in the clojure wiki.

18:56 if i try to load the script with

18:56 RT.loadResourceScript ( "C:/app/foo.clj" ) ;

18:56 i get a resouce not found exception.

18:56 i realized that this method only takes the name part of the given path. How can i load the script form a path which is not in the classpath?

19:03 Chouser: hsyn: are you sure you need to load clojure from within your own java app? It's a bit easier to run from the command line.

19:04 hsyn: i going to use the result in java.

19:04 in future

19:05 achim_p: hsyn: you could extend your classpath with RT.addURL

19:06 didn't try it though ...

19:07 hsyn: thx achim it work

19:07 Chouser: you could also use clojure.lang.Compiler.load(rdr, sourcePath, sourceName)

19:07 oh, good, nm then. :-)

19:19 pjb3: So I take it Clojure's instance? method doesn't map to Java's instanceof operator?

19:19 (instance? java.util.Map {}) => false

19:23 rhickey: Clojure maps are not java,util.Maps

19:44 pjb3: Clojure maps are not java.util.Maps? Hmm...that's a bummer when trying to work with Java APIs that expect maps

19:45 I assume there's a good reason why clojure maps aren't java.util.Maps?

19:46 Chouser: Clojure maps are persistent (immutable)

19:46 You can still work with java.util.Maps pretty easily if you're sure you want them.

19:48 pjb3: I want to pass a clojure map to a Java API that expects a java.util.Map

19:56 shoover: there was some discussion mentioning it as a possibility, but it seemed to get bogged down on incompatibilities between Collection and Map: http://clojure-log.n01se.net/date/2008-07-18.html

19:57 Chouser: (reduce (fn [m [k v]] (.put m k v) m) (java.util.HashMap.) {:a 1 :b 2})

20:00 pjb3: I'm trying to see if I can use clojure with velocity

20:00 Chouser: that function helps

20:00 but then the next problem is that what about maps that contain maps

20:01 Since clojure maps don't even have a get method, they wouldn't work in a velocity context

20:04 Chouser: (defn jum [cm] (reduce (fn [m [k v]] (.put m k (if (map? v) (jum v) v)) m) (java.util.HashMap.) cm))

20:04 (jum {:a 1 :b {:c 3 :d 4 :e {:f 5}}})

20:06 pjb3: Chouser: nice, I guess I'll try that for now

20:08 But I agree with meredydd from that earlier discussion, I think it would be really nice to not have to do a deep copy everytime you want to have a java library use a clojure map

20:26 hsyn: is there any ATN parser available

23:13 Chouser: rhickey: I saw your "agents/refs in use" slides. Very nice.

23:14 What kind of responses have you gotten to your talk?

23:14 rhickey: really great. I just got to the airport - an amazing conference

23:14 so many interesting people

23:15 Chouser: :-)

23:15 rhickey: Clojure was very well received

23:16 blackdog: congrats rhickey

23:16 Chouser: great! anybody promise you tagged numbers? ;-)

23:16 rhickey: no, but they and tail calls were the most requested changes

23:17 some hope in escape analysis for local optimization of boxed numbers in the JITs

23:17 Cliff Click ran the ACO TSP code on 600 cores - scaled well

23:18 Chouser: that's pretty cool

23:22 rhickey: he had some amazing perf tools for his boxes - so much fun to see Clojure pop the bar meters on 600 cores

23:25 blackdog: i didn't realise the jvm scaled like that, never mind clojure

23:25 v. impressive

23:26 rhickey: those boxes are amazig - the sim was allocating 18GB a second, GC wasn't sweating

23:26 amazing

23:26 blackdog: crikey, all in makes one feel the jvm is a good choice :)

23:27 does the .net vm scale like that?

23:27 rhickey: I don't know if there are boxes like that for .Net

23:28 these things are like 200 cores in 5U

23:28 1/2 TB RAM

23:29 blackdog: so this is really a milestone for clojure getting that coverage

23:29 avida: rhickey: are these the Azul boxes? my company trialled something like that

23:30 rhickey: lots of people there had only barely heard of Clojure, now they know what it's about

23:30 avida: yes, Azul

23:31 blackdog: congrats again, you must be feeling over the moon after the hard work, and innovation!

23:31 rhickey: I'm totally spent - 3 days of intense conversations

23:33 blackdog: i mean really all the years on clojure and then taking it to a conference like that and getting the recognition etc

23:33 tis very cool

23:33 rhickey: blackdog: yes, thanks

23:34 now I have to gear up for Monday at MIT - very sophisticated Lisp group

23:34 blackdog: hehe

23:35 rhickey: plus maybe some Sun STM folks I hear

23:35 blackdog: do you know when infoq will publish the videos?

23:35 of the conf

23:35 rhickey: blackdog: no idea

23:36 I had only 1/2 hour - I talked as fast as I could

23:36 blackdog: did you come away with any inspiration, ideas?

23:37 rhickey: It will take a while to filter through, but overall I felt good about where Clojure is at. Some perf ideas, some more research topics to pursue

23:37 blackdog: well obviously, but any new implementation stuff :P

23:38 nice

23:38 probably more folks took away from what you've already achieved i bet

23:39 rhickey: there was a lot of talk about functional programming

23:39 blackdog: esp after your successful 600 core test :)

23:46 well on a more modest scale (1 core), my brother is selling 3d training videos which has a simple clojure backend producing json, so it's my first production use of clojure

23:51 yangsx: java -cp clojure.jar clojure.lang.Compiler src/clj/clojure/boot.clj won't work, is that expected?

23:51 I'm using svn version

23:53 Or how can I compile a Clojure program into Java classes?

23:55 blackdog: yangsx, you can't do that yet, rhickey is investigating aot compilation to java though

23:57 yangsx: blackdog: OK, I got that from clojure.markdown in the svn repo.

Logging service provided by n01se.net