#clojure log - Jul 22 2009

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

0:44 ataggart: what's the best way to get something added to clojure-contrib? assembla? clojure-dev group?

0:46 arohner_: ataggart: yes, join assembla & clojure dev

0:46 ataggart: I have, just wondering what I do with this file thats on my machine

0:46 arohner_: ataggart: are you making a new thing, or patching an existing component?

0:46 ataggart: new thing

0:46 arohner_: ataggart: fork the github project, and commit your change to your fork

0:47 ataggart: k thx

0:47 arohner_: then you'll need to get your change from your fork into the trunk. I'm not sure about the process there

0:48 ataggart: yeah, I haven't really grokked git yet

0:49 arohner_: I'm familiar with the mechanics of git, I'm not sure about the clojure process for getting your change from your fork to the trunk

0:52 ataggart: it looks like I just need to create a patch and send it up via assembla (http://clojure.org/patches)

0:58 replaca: ataggart: contributions to coontrib should be discussed in the google group first (though contributors do ometimes sneak them in)

0:58 ataggart: so just attach the code to the thread?

1:00 replaca: well, I'd discuss the idea first more than the code (though you can attach the code, too)

1:01 folks tend to be more interested in the capability and the api than the actual implementation

1:01 ataggart: well I've now "applied" to join the clojure-dev group

1:02 replaca: if it seems to have consensus in the google group that "it's a good thing" to have in contrib, then open an assembla ticket and attach the patch

1:02 ataggart: k

1:02 replaca: ataggart: feature discussion should be in the general clojure group

1:02 not in dev

1:02 ataggart: ah ok

1:03 replaca: dev is more for mechanics and details of how things are getting done

1:03 (though it might be interesting to be part of)

1:03 also, have you done a CA?

1:03 ataggart: yep

1:03 replaca: then that's all!

1:05 Anniepoo: I'm creating a structure with some anonymous functions that are callbacks for a popup menu.

1:05 Some of them need the event and some don't

1:07 suppose I have #(some-menu-callback) and #(another-callback %)

1:07 the first doesn't need the event

1:07 now, if I have the callback in callback-fn and try to invoke it with

1:08 (callback fn the-event)

1:08 it freeks on the first one

1:08 replaca: try (fn [_} some-menu-callback)

1:08 try (fn [_] some-menu-callback)

1:09 Anniepoo: ah, thanks

1:09 8cD

1:09 replaca: otherwise it thinks it has the wrong number of args

1:09 Anniepoo: yup

1:10 y'all are wonderful, this is a really helpful community

1:10 replaca: np

1:11 lisppaste8: ataggart pasted "contrib logging" at http://paste.lisp.org/display/83982

1:21 Knekk: hmm, populating a large array is so slow...

1:21 ataggart: what's "large"?

1:21 Knekk: ,(* 512 512 16)

1:21 clojurebot: 4194304

1:21 ataggart: mhmm... large

1:22 replaca: ataggart: logging looks nice (no writing code to the specific mechanism). coouple of typos in the ns comment (log and send-log are macros, not funcs) and you mispelled "their"

1:23 ataggart: heh, just saw the typo.

1:23 and I wasn't sure what the convention was regarding discussing a function/macro

1:23 updating my local version now

1:24 thanks for looking at it

1:24 replaca: well, in this case it's important that it's a macro and probably should be emphasized. In other contexts it's probably OK to be more casual about it

1:25 np -> this looks valuable to me for helping make sure we have code interop

1:26 heterogenous logging expectations is always a headache for mixing java libs

1:27 Knekk: any ideas on how to speed up performance with large datasets?

1:28 ataggart: parallelize?

1:28 * ataggart makes an attaempt at hand-waving

1:28 * Knekk tries it and fails miserably

1:28 Knekk: nope

1:30 hiredman: performance of what?

1:30 Knekk: populating a large array of structs

1:32 hiredman: why did parallelizing fail?

1:33 Knekk: I didn't try it. I was joking.

1:33 hiredman: threads? agents?

1:34 hiredman: create a threadpoolexecutor, generate thunks to init ranges of the array, feed them to the executor

1:34 or use agents

1:35 Knekk: ok, I'll try that tomorrow, thanks.

1:35 ataggart: is pmap appropriate for this?

1:35 _mst: or pcalls?

1:37 hiredman: pcalls maybe

1:37 Knekk: they are lazy/semi-lazy, and I need a persistent, fully-instantiated array

1:37 hiredman: ~def pcalls

1:37 I don't believe pcalls is lazy

1:38 Knekk: returns a lazy seq

1:38 hiredman: ,(doc pcalls)

1:38 clojurebot: "([& fns]); Executes the no-arg fns in parallel, returning a lazy sequence of their values"

1:38 hiredman: the seq is lazy because the fns might take longer then others

1:39 clojurebot: you are broken

1:39 clojurebot: I don't understand.

1:39 lushen: is there a split function besides (.split)

1:39 hiredman: lushen: I'd check str-utils in contrib

1:40 lushen: it is returning me a Ljava.lang.String that might be causing bugs

1:40 ok thanks

1:40 hiredman: lushen: I doubt it

1:40 seq knows how to make seqs from arrays

1:40 and most sequence functions call seq on their args

1:40 clojurebot: functions are maps

1:40 hiredman: (first (.split "foo bar" " "))

1:41 ,(first (.split "foo bar" " "))

1:41 clojurebot: "foo"

1:41 ataggart: re-split in str-utils may be what you want

1:41 hiredman: ,(.split "foo bar" " ")

1:41 clojurebot: #<String[] [Ljava.lang.String;@c7c56a>

1:41 hiredman: .split takes a regex

1:41 ,(.split "foo bar" "\\s")

1:41 clojurebot: #<String[] [Ljava.lang.String;@ac8e62>

1:41 hiredman: ,(first (.split "foo bar" "\\s"))

1:41 clojurebot: "foo"

1:42 hiredman: it just takes the regex as a string because java does not have regex literals

1:43 lushen: okay let me try to explain what i'm going for

1:43 I've got a bunch of flatkeys like "a/b/c"

1:43 i'm feeding the split array into update-in

1:44 hiredman: uh

1:44 no

1:44 update-in takes a map or maybe a vector

1:44 lushen: (update-in output (.split "a/b/c" "/") (fn [x y] y) "hahahahaha")

1:45 hiredman: ,(doc update-in)

1:45 clojurebot: "([m [k & ks] f & args]); 'Updates' a value in a nested associative structure, where ks is a sequence of keys and f is a function that will take the old value and any supplied args and return the new value, and returns a new nested structure. If any levels do not exist, hash-maps will be created."

1:45 hiredman: you are missing a lot of things

1:45 and things are out of order

1:45 etc

1:45 ,(update-in {:foo 1} [:foo] + 5)

1:45 clojurebot: {:foo 6}

1:46 hiredman: ,(update-in {:foo {:bar 1}} [:foo :bar] + 5)

1:46 clojurebot: {:foo {:bar 6}}

1:47 Knekk: pcalls doesn't really make it faster on my laptop

1:48 hiredman: *shrug*

1:48 large array initialization could just be slow

1:48 lushen: does it defeat the point of using update-in to parse a flathash if i have to feed the function a nested map in the first place?

1:48 hiredman: lushen: you don't need to feed it a nested map

1:49 as my first example demoed

1:50 ah

1:50 ok

1:50 I understand you update-in line now

1:50 your

1:50 ,(vec (.split "foo/bar" "/"))

1:50 clojurebot: ["foo" "bar"]

1:50 lushen: ah!

1:50 hiredman: update-in takes a vector as it's second arg

1:51 a java array is not a vector

1:52 _mst: ,(assoc-in {} (.split "a/b/c" "/") "hahahahaha")

1:52 clojurebot: {"a" {"b" {"c" "hahahahaha"}}}

1:52 _mst: assoc-in might save the trouble of that extra (fn ...) too

1:53 lushen: but that might overwrite the existing?

1:53 thanks let me try these out

1:54 _mst: yep, it will replace anything that was already in your map at that key

2:12 lushen: i have run into another problem: http://pastie.org/554637

2:12 the two functions are mutually referrent

2:13 do i have to nest the shorter one in the longer one?

2:13 replaca: lushen: no, just use declare

2:14 (delare foo) (defn bar [] (foo)) (defn foo [] 7)

2:14 (bar) => 7

2:14 lushen: thnx

2:14 replaca: np

2:35 lushen: (i am new to functional programming) : how do I make something like [ 0 0 "something" 0 ]

2:36 hiredman: ,[0 0 "something" 0]

2:36 clojurebot: [0 0 "something" 0]

2:37 hiredman: pretty simple, no?

2:37 lushen: for instance if i have the structure [3 "something"] i want to get [0 0 0 "something"]

2:38 hiredman: ,([3 "something"] 1)

2:38 clojurebot: "something"

2:38 hiredman: ,(assoc [0 0 0 0] 3 ([3 "something"] 1))

2:38 clojurebot: [0 0 0 "something"]

2:39 lushen: thank you

2:40 is there something like (vector 0 4) -> [0 0 0 0] ?

2:41 hiredman: (vec (range 4))

2:42 ,(vec (range 4))

2:42 clojurebot: [0 1 2 3]

2:42 hiredman: ,(vec (repeat 0 4))

2:42 clojurebot: []

2:42 hiredman: ,(vec (repeat 4 0))

2:42 clojurebot: [0 0 0 0]

2:42 hiredman: ,(vector 0 0 0 0)

2:42 clojurebot: [0 0 0 0]

3:01 hiredman: ,((comp vec (partial apply repeat) reverse list) 0 4)

3:01 clojurebot: [0 0 0 0]

3:12 Fossi: hi

3:16 lushen: hi

4:45 gulagong__: ,(prn "good morning")

4:45 clojurebot: "good morning"

6:03 Jomyoot: ,(/ 0 0)

6:03 clojurebot: java.lang.ArithmeticException: Divide by zero

6:30 Lau_of_DK: ,(loop [] (recur))

6:30 clojurebot: Execution Timed Out

7:47 hamza: hi, i packed my application in to a jar i need to load an image inside the jar. how can i convert URL imgURL = getClass().getResource(imgName); to clojure?

8:23 kylesmith: hamza: (.. YourClass getClass getResource imgName) should work

8:47 Chousuke: isn't the getClass redundant?

8:57 durka42: ,(.getClass YourClass)

8:57 clojurebot: java.lang.Exception: Unable to resolve symbol: YourClass in this context

8:57 durka42: ,(.getClass String)

8:57 clojurebot: java.lang.Class

8:58 durka42: ,String

8:58 clojurebot: java.lang.String

8:58 Jomyoot: HOw do I append an element to end of list

8:58 opposite of cons

8:58 (1 2 3) + 4 = (1 2 3 4)

8:58 what's the function name?

8:59 gnuvince: ,(append [1 2 3] [4])

8:59 clojurebot: java.lang.Exception: Unable to resolve symbol: append in this context

8:59 gnuvince: err

8:59 concat?

8:59 cark: ,(concat [1 2 3] [4])

8:59 clojurebot: (1 2 3 4)

8:59 gnuvince: ,concat

8:59 clojurebot: #<core$concat__3872 clojure.core$concat__3872@14412b0>

9:00 gnuvince: Too many languages, too many words.

9:00 Jomyoot: in a non lazy way

9:00 i want to construct a new seq entirely

9:00 concat is lazy

9:00 cark: Jomyoot : if you need to append to the back of the list, a list might not be the proper data structure

9:01 Chousuke: Jomyoot: appending to the end of a list is O(n)

9:01 hm

9:01 cark: ,(-> [] (conj 1) (conj 2) (conj 3))

9:01 clojurebot: [1 2 3]

9:01 cark: and that's not lazy

9:03 Chousuke: I suppose if clojure had finger trees you could have an efficient deque

9:05 cark: does it need to be a finger tree ?

9:05 we have a persistent queue already

9:10 hamza: i have compiled my application and i can run it using java package.main but if i put it in a jar and try to run it java -jar i get java.lang.ExceptionInInitializerError caused by null pointer exception. my -main ony contains a println line

9:33 Jomyoot: ,(let [x 1] (let [x 2] (println x)))

9:33 clojurebot: 2

9:59 Jomyoot: i would never understand macro

10:04 Fossi: it'll grow on you

10:42 rzoom: having some trouble with format

10:42 anyone have an example?

10:43 user=> (format "blah to %1$s sir!" "rob")

10:43 java.lang.Exception: Unable to resolve symbol: in this context (NO_SOURCE_FILE:0)

10:43 "blah to rob sir!"

10:45 using %s instead of %1$s gives the same result

10:49 Chouser: ,(format "blah to %1$s sir!" "rob")

10:49 clojurebot: "blah to rob sir!"

10:49 Chouser: rzoom: there's something wrong with your environment, I think.

10:50 you, it returns the correct result after reporting an exception!?

10:50 s/you, //

11:02 rzoom: Chouser: yeah, pretty strange eh?

11:03 Chouser: can you paste the whole stack trace somewhere?

11:04 rzoom: this is really weird. when i re-type it, it works

11:05 Chouser: maybe you've got a weird unicode space in there somewhere

11:05 rzoom: there must be a hidden character in there

11:05 yah

11:06 thanks for the help, though.

11:47 gko: What's the best way to convert a list of bytes to a Java array of bytes?

11:47 Chouser: into-array

11:47 gko: not list of bytes, list of numbers

11:47 OK, I'll try.

11:47 Chouser: into bytes or Bytes?

11:48 gko: bytes

11:48 Chouser: ,(into-array Byte/TYPE (map byte [1 2 3 4 5]))

11:48 clojurebot: #<byte[] [B@af908f>

11:49 gko: was so simple :)

11:50 Fossi: ,(meta meta)

11:50 clojurebot: nil

11:50 Fossi: how am i supposed to get the metadata of a function?

11:51 gko: ,(meta #^meta)

11:51 clojurebot: Unmatched delimiter: )

11:51 Chouser: most functions have no metadata

11:51 gko: ^#^meta

11:51 ,^#^meta

11:51 clojurebot: EOF while reading

11:51 Chouser: vars do, though

11:51 gko: ,#^meta

11:51 clojurebot: EOF while reading

11:51 gko: ,^#meta

11:51 clojurebot: No dispatch macro for: m

11:51 gko: grr

11:51 Chouser: ,(meta (var meta))

11:51 clojurebot: {:ns #<Namespace clojure.core>, :name meta, :file "clojure/core.clj", :line 158, :arglists ([obj]), :doc "Returns the metadata of obj, returns nil if there is no metadata."}

11:52 Chouser: , ^ #' meta

11:52 clojurebot: {:ns #<Namespace clojure.core>, :name meta, :file "clojure/core.clj", :line 158, :arglists ([obj]), :doc "Returns the metadata of obj, returns nil if there is no metadata."}

11:53 Fossi: ah, ok

11:56 and the metadata of namespaces?

11:57 Chouser: ,^(the-ns 'clojure.core)

11:57 clojurebot: {:doc "Fundamental library of the Clojure language"}

11:57 Fossi: ah, cool

11:57 * Fossi is not into meta so much yet ;)

11:58 Chouser: getting is pretty much always just ^ but the problem is more often getting at the object that has the metadata

12:02 Fossi: works with clojure.core, but not with my namespace

12:02 looks like ns does something differently

12:02 Chouser: you're sure you have metadata on it?

12:04 Fossi: hm. this is getting interesting

12:19 cemerick: hrm, I might just use lucene as a key-value datastore *evil grin*

12:20 stuartsierra: cemerick: I've done it. It works up to about 10GB.

12:22 cemerick: yeah, I've used it as the single store for metadata (as opposed to having sqlite or whatever off to the side), but I've always had source material available for reindexing if necessary...

12:22 stuartsierra: Yes, that's necessary.

12:22 cemerick: stuartsierra: at which point the merges kill you?

12:23 stuartsierra: The merges kill you, and the index is more likely to get corrupted when it gets that big.

12:24 Now I've got only searchable fields in the index, and HTML content stashed on a filesystem.

12:24 * technomancy is looking at using katta for clustering lucene

12:24 technomancy: pretty rough around the edges, but the concept is sound

12:24 not that I condone using it as a key/value store!

12:25 stuartsierra: Someone was telling me about Cassandra last night at Philly Lambda.

12:25 * cemerick tries to remember how often lucene indexes wedged themselves the last time he used it seriously

12:28 technomancy: yeah, it's not like there's a shortage of key-value stores out there

12:29 cemerick: there's no clear winner for cross-platform, embedded usage as far as I can tell (unless you're happy with a copyleft license)...but we covered this yesterday :-)

12:35 * drewr is neck-deep in cassandra atm

12:36 gde: How do I check that an argument is a character?

12:36 drewr: I think I like it but I'm having a time fitting my data into it

12:36 ,(char? \d)

12:36 clojurebot: java.lang.Exception: Unable to resolve symbol: char? in this context

12:36 drewr: doh

12:36 ,(instance? Character d)

12:36 clojurebot: java.lang.Exception: Unable to resolve symbol: d in this context

12:36 drewr: ,(instance? Character \d)

12:36 clojurebot: true

12:37 gde: Thanks!

12:37 cemerick: huh, I thought there was a char?

12:37 gde: there is a string?

12:37 cemerick: yes, definitely

12:37 drewr: i wish we had some notation for fns

12:37 Chouser: no, but there is a 'str' and a 'string?'!

12:38 cemerick: drewr: I remember thinking that the data model looked overly complicated (at least for what we need)

12:39 gde: is there a way of doing (instance? Character d) without using any Java?

12:39 drewr: cemerick: i don't think it's *overly* complicated, but it is more so than hbase

12:40 cark: gde : yes you can define a character? predicate and use it =)

12:40 cemerick: drewr: heh, well, we thought HBase was overly complicated too :-)

12:40 couch is more our speed.

12:41 drewr: cemerick: if couch didn't fall over after 10K records we would probably still be exploring it

12:41 it's probably improved by now

12:41 stuartsierra: I'm still trying to find a good answer to the question, "What do I do with the output of a Hadoop job to make the records accessible on the web?"

12:41 cemerick: drewr: when was that happening? We haven't had any such issues, FWIW.

12:42 (not that we're beating it up particularly badly atm, but we've got a lot more than 10K records)

12:42 drewr: it was last year; it was bad enough that it turned me off at the time

12:42 i should probably look again

12:42 Chouser: gde: Not quite sure what you mean. You just want to avoid using the name of a Java class, or something else?

12:42 gde: chouse: yes I wanted the code to be platform independent.

12:43 drewr: gde: java is platform independent

12:43 Chousuke: Are you going to run the code on ClojureCLR or something?

12:43 in that case, you will need to define some functions twice.

12:44 Chouser: #ifndef ClojureCLR (defn char? [x] (instance? Character x)) #endif

12:44 gde: i have some code that doesn't use any Java directly and i'd like to keep it that way

12:44 rhickey: issue/patch welcome for char?

12:45 * drewr scurries off to pick the low-hanging fruit

12:45 Chousuke: heh :P

12:45 gde: thanks, Chouser!

12:45 drewr: gotta put that CA to use!

12:46 gde: "char?" would be very nice

12:46 technomancy: next week in the Seattle Functional languages meeting, if anyone else is interested.

12:46 Chousuke: Some reader conditional could be useful for writing portable clojure if ClojureCLR gains support :P

12:47 technomancy: *next week is

12:47 hlship: does anyone have a handy example of adding meta data?

12:48 That is, I have a (defview) macro that creates a function, but I want to add some extra meta-data to the defined function.

12:48 Chousuke: with-meta should help you

12:48 hlship: I keep getting lost between the meta-data on the name and the meta-data on the function itself.

12:48 Chousuke: functions don't support metadata, do they?

12:48 Chouser: hlship: you macro expands to 'defn' or something else?

12:49 stuartsierra: hlship: one thing to remember: "def" and all its variants copy metadata on the name to metadata on the Var being defined.

12:49 hlship: Expands to defn currently, but could be (def ~fn-name (fn [] ...)) if it should

12:49 that's where I keep getting lost ...

12:50 Chouser: (defmacro m [] `(defn foo {:my :meta} []))

12:50 technomancy: hlship: there's some examples of using def to add docstrings before defn is defined in core.clj

12:50 Chouser: defn supports a map of metadata after the fn name.

12:50 technomancy: or that

12:50 hlship: Good options!

12:51 That last one is probably the easiest one for me .. let me give that a try. Thanks as always!

12:53 technomancy: you should probably read through core.clj anyway; it's very instructive.

12:54 hlship: I have been reading it, core.clj, yes ... but the (def) stuff is very complicated/bootstrapped.

12:54 Chousuke: or you could do (defmacro defview [& fn-stuff] `(alter-meta! merge {:meta 'stuff} (defn ~@fn-stuff)))

12:54 so that users could supply their own metadata map with defview as well.'

12:55 jbondeson: we need a "Stump Chouser" irc game. winner gets a kewpie doll.

12:55 Chousuke: also, hm

12:55 my alter-meta! call is apparently wrong :)

12:55 should have the (defn...) part as its first argument instead of last.

12:57 Chouser: and alter-meta! returns the metadata map instead of the var. def's traditionally return var's, though I don't know if anybody ever uses that

12:57 stuartsierra: clojure.test/with-test uses it!

12:58 Chouser: well, there you go then.

12:58 :-)

13:00 yason: re

13:09 Chousuke: Chouser: hm :/

13:11 so there's no way to just stick some additional metadata on the var returned without using let :(

13:11 Chouser: sure -- both def and defn transfer meta from the name to the var

13:12 hlship: turns out it was super easy

13:12 thanks again for the help

13:12 just needed that little nudge

13:13 Chouser: a little harder to write a macro that puts meta on a def

13:14 (defmacro m [] `(def ~(with-meta 'foo {:my :meta}) "val"))

13:18 gko: For sockets, do you just use reuse Java API + duck_streams?

13:20 technomancy: gko: there's a server-socket library in contrib

13:21 gko: technomancy: for clients?

13:21 technomancy: yeah, you probably want the Java API + duck-streams

13:24 gko: ok

14:04 So, to read a response: (duck/to-array-byte (.getInputStream s)) ?

14:07 stuartsierra: that should work

14:08 it's to-byte-array

14:08 gko: right

14:09 but it's blocking...

14:13 hiredman: uh

14:13 why wouldn't it?

14:13 ,(doc to-byte-array)

14:13 clojurebot: "clojure.contrib.duck-streams/to-byte-array;[[arg]]; Converts argument into a Java byte array. Argument may be a String, File, InputStream, or Reader. If the argument is already a byte array, returns it."

14:14 hiredman: arrays cannot be lazy

14:14 so it has to read until the end of the input stream, then construct the byte array

14:14 possibly it could return a promise

14:14 Chouser: not sure if duck-streams are designed with sockets in mind

14:24 ok, anybody with JNI experience here?

14:25 hserus: i have worked on it long back

14:25 Chouser: I'm using swig on C++ code, and my .so builds and loads ok.

14:25 hserus: ok..

14:38 stuartsierra: Chouser: yes, duck-streams was not designed with sockets in mind.

14:39 hiredman's right, to-byte-array has to read to the end of the input stream before it returns anything.

14:39 gko: ok

14:41 Chouser: hserus: sorry, got called away.

14:41 hserus: Chouser: np

14:41 Chouser: ok, so (System/loadLibrary "my_lib") rns without error

14:41 runs

14:42 But when I try to create a object, I get an error: (my.lib.Foo. "tmp)

14:42 java.lang.UnsatisfiedLinkError: my.libJNI.new_Foo(Ljava/lang/String;)J

14:43 hserus: i guess you've verified that the library is in the path?

14:43 Chouser: well, it loads

14:44 if it's not in the path, if fails at loadLibrary

14:46 my.libJNI is a Java class generated by swig, and loads ok too -- I can use reflection to see all its methods

14:46 gko: For socket, something like this:

14:46 (defmethod copy [SocketInputStream OutputStream] [input output]

14:46 (let [buffer (make-array Byte/TYPE *buffer-size*)]

14:46 (loop []

14:46 (let [size (.read input buffer)]

14:46 (if (and (pos? size) (> (.available input) 0))

14:46 (do (.write output buffer 0 size)

14:46 (recur))

14:46 (if (and (pos? size) (= (.available input) 0))

14:46 (.write output buffer 0 size)

14:46 nil))))))

14:46 hserus: are the related libraries in path too?

14:47 Chouser: hserus: yes, I was having problems with that too, but all those failures are when I load the lib.

14:47 now that all the deps are in the path, loadLibrary is working. That's also the step where the swig docs seem to expect UnsatisfiedLinkError's

14:48 so I'm confused as to how I'm getting UnsatisfiedLinkError when trying to construct

14:49 and with no error message, just a Java method signature

14:49 java.lang.UnsatisfiedLinkError: my.libJNI.new_Foo(Ljava/lang/String;)J

14:50 hserus: i have no idea.. do you know what the class does?

14:50 Chouser: the Foo C++ class? I've got the source to it, so I can look at that. You think that may reveal the problem?

14:51 it seems unlikely it's getting as far as calling the constructor

14:51 hserus: probably, it might be using another dependency which isn't available in path

14:57 Chouser: any luck?

15:11 Chouser: hserus: I'm trying to add a simple static method to the C++ and see if I can get that working.

15:12 hserus: ok

16:24 rhickey: anyone looking to contrib towards the forkjoin stuff - let me know if you want to try to implement pvmap-index or pvreduce-index, which are passed fns taking index and val

16:25 also interested in any performance feedback on the par branch

16:31 esp if you have > 4 cores

16:33 technomancy: where would be a good place to read up on forkjoin?

16:33 rhickey: http://gee.cs.oswego.edu/dl/concurrency-interest/index.html

16:34 technomancy: thanks

16:34 rhickey: but I hope to hide the complexity from people, using pvmap/pvreduce requires no forkjoin knowledge

16:35 drewr: fj seems to me like mapreduce with threads instead of computers

16:50 arohner_: is there a way to just go run a fn in a separate thread, using the agent thread pool?

16:50 I don't have an agent, and I don't care about the return value

16:50 rhickey: ,(doc future)

16:50 clojurebot: "([& body]); Takes a body of expressions and yields a future object that will invoke the body in another thread, and will cache the result and return it on all subsequent calls to deref/@. If the computation has not yet finished, calls to deref/@ will block."

16:50 arohner_: but I would like to be able to use i.e. await

16:51 rhickey: thanks

17:04 rhickey: lisppaste8: url

17:04 lisppaste8: To use the lisppaste bot, visit http://paste.lisp.org/new/clojure and enter your paste.

17:05 rhickey pasted "pvmap rocks!" at http://paste.lisp.org/display/84027

17:06 rhickey: quad-core perf ^^

17:06 Chousuke: nice. :)

17:07 rhickey: game-changing

17:07 arohner_: wow

17:10 Chousuke: Are you going to merge the namespace with clojure.parallel?

17:10 par is not a very good name. :P

17:10 rhickey: clojure.parallel is going to be deprecated from the looks of this, I just didn't want to deal with that right now

17:11 Chousuke: right.

17:11 I wonder if anyone was using it anyway.

17:12 rhickey: the beautiful thing about this is it's persistent data structure -> persistent data structure

17:12 pvmap returns a regular Clojure vector

17:13 Chousuke: heh, yeah.

17:14 Though it's also fun that even in the linear case, ArrrayList is not that much faster :P

17:15 rhickey: depending on the gc patterns, there were some runs where linear (vec (map ... is just Elapsed time: 82.551 msecs, I used a more typical number

17:16 * rzoom looks forward to pvmap in master

17:16 drewr: any issue with converting pmap to pvmap?

17:17 rhickey: drewr: pmap works on any seq, pvmap only takes vectors

17:17 drewr: ahh

17:18 rhickey: parallelizing a seq has limits, e.g. the per-item work must be substantial

17:19 also pmap is semi-lazy, pvmap computes a full result

17:19 note this pvmap call is just for inc, a tiny per-item job

17:19 drewr: yeah

17:20 rzoom: wonder how long the pmap version would take

17:20 rhickey: pmap inc is a losing proposition, many times slower than map

17:20 rzoom: ah

17:20 makes sense

17:20 rhickey: pmap is for big per-item work

17:21 where it is still quite handy, being lazy and all

17:21 arohner_: rhickey: would it be possible to make a fast pvfilter?

17:21 rhickey: I'll be able to do similar ops for maps and sets

17:22 arohner_: pvfilter is a little trickier, I've been thinking about it

17:23 Chouser: sparse vectors?

17:23 Chousuke: hmm

17:23 rhickey: maps can be used as sparse vectors

17:23 Chousuke: how do I specify the classpath for ant for building with jsr166y?

17:23 rhickey: Chousuke: that's built into the par branch

17:24 just put jsr166y.jar into your Clojure base dir

17:24 i.e. next to build.xml

17:25 Chousuke: now it complains about the class version... I guess ant is using 1.5?

17:26 rhickey: Chousuke: you have to set up Java 6, what OS?

17:26 Chousuke: OS X 10.5.7, java is 1.6.0_15 (developer preview. Has escape analysis and I want to see if it has any effect :P)

17:27 rhickey: ant won't see that unless you do:

17:27 export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home

17:27 Chousuke: I see.

17:28 rhickey: Java Preferences panel is not enough

17:28 Chousuke: hmm :/

17:29 now I get "cannot find symbol" for ((ForkJoinPool)pooledExecutor).setAsyncMode(true); ... I wonder if I have the right version of the jsr jar.

17:29 rhickey: use: http://cloud.github.com/downloads/richhickey/clojure/jsr166y.jar

17:31 Chouser: were you saying the results of pvfilter should be a sparse vector?

17:31 Chouser: rhickey: must musing

17:32 rhickey: dunno if you'd want the old vector's indexes intact, or if you'd want new non-sparse indexes.

17:32 Chousuke: rhickey: that worked.

17:32 Chouser: the latter, I suppose, but not sure how that could be done in parallel.

17:33 rhickey: parallel map and set merges etc are pretty exciting

17:33 pmmap

17:33 hrm

17:39 Chousuke: rhickey: I'm not seeing as amazing results as you are. I think memory might be a bottleneck for me instead of CPU :/

17:40 I have a dualcore processor

17:41 it's still faster than ArrayList, though :)

17:42 hm

17:43 I enabled escape analysis, and it causes some rather inconsistent results for (time (def v1 (vec (map inc v))))

17:45 without escape analysis, I got speeds ranging from 550ms to 670ms now it's giving me 550-1200ms, with one 221ms result and one 1337ms :P

17:47 array map case goes 200-400ms, and pvmap 50(!) to 1400ms (again, !) (more often around 240ms though)

17:48 mebaran151: is there a library function to turn a bunch of nested key value lists into a hash?

17:48 Chouser: ,(apply hash-map [[:a 1] [:b 2]])

17:48 clojurebot: {[:a 1] [:b 2]}

17:48 Chouser: heh

17:48 Chousuke: hmm

17:48 Chouser: ,(into {} [[:a 1] [:b 2]])

17:48 clojurebot: {:b 2, :a 1}

17:49 Chousuke: performance is really weird with escape analysis :/

17:50 in general it looks like it improves things though. unless you get unlucky.

17:53 mebaran151: doesn't work with nesting though

17:53 as in [:a [:b 1]] {a {b 1}}

17:57 arohner_: does stuart sierra hang out here?

18:07 Chousuke: mebaran151: http://gist.github.com/152325 something like this?

18:08 mebaran151: I think I tried something like that

18:30 cemerick: arohner_: saw him earlier today *shrug*

20:10 krumholt__: i have a function i want to use on all elements of a seq. normally i would use map, but the function takes two arguments x and y. and i want the second argument to be the result of the last function evaluation. how can i do that?

20:11 ieure: krumholt__, reduce.

20:14 krumholt__: ieure, perfect thanks

20:38 ataggart: is there a fucntion to see if there is a currently running transaction?

20:46 cemerick: ataggart: (clojure.lang.LockingTransaction/isRunning)

20:52 hiredman: (alter (ref 1) inc) and see if there is an exception?

20:54 ataggart: hiredman: ugly solutions are verboten

20:54 cemerick: thanks

21:13 arohner_: I'm trying to write a version of test-ns that runs the tests in parallel. These are functional tests that can take 30-60s

21:13 I have

21:13 (dorun (map deref (map #(future (%)) (filter #(:test (meta %)) (vals (ns-interns ns)))))))

21:13 that's not faster than the current test-ns. am I doing anything stupid?

21:14 ykphuah: why a new lisp instead of taking from CL/scheme/ELisp?

21:15 arohner_: ykphuah: http://clojure.org/rationale

21:23 ykphuah: arohner_: hmm, I wanted to integrate emacs/elisp with the JVM, but using clojure means I need to learn another LISP.

21:23 so I can't really call clojure's functions from elisp.

21:24 arohner_: I guess so

21:34 yangsx: I'm wondering whether it's possible to map over a function that takes a rest argument.

21:34 (defn a [x & y]...)

21:35 arohner: yangsx: yes, it is

21:35 yangsx: like: (map #(a x %) coll)

21:35 aroher: ?

21:35 arohner: ?

21:35 arohner: yangsx: it is possible

21:35 that code looks good

21:37 a will get called once for every item in coll, with two arguments, x and an item

21:37 yangsx: arohner: no, not exactly what I want. The coll is a sequence of sequnce, and I want something like `(a x ~@%)

21:38 sequences*

21:38 arohner: try

21:38 (map #(apply a x %) coll)

21:38 assuming coll looks like [ [1 2 3] [4 5 6] [7 8 9]]

21:40 yangsx: I see, thanks arohner. I forgot apply.

21:52 Anniepoo: on this page, http://bc.tech.coop/blog/081029.html

21:52 the last green on black code section shows an ns definition with (:use (my.lib this that))

21:52 is this correct?

21:57 this also appears in the clojure API docs

21:57 (:use (my.lib this that))

22:28 Mark_Addleman: i have a list: ([1235 ([0 17])] [1236 ([0 15] [16 8])]) and i want to transform it to ([1235 0 17] [1236 0 15] [1236 16 8]). i'm having all sorts of trouble thinking about this in the right way.

22:54 Chouser: Mark_Addleman: break it down -- each second-level vector stands along, right?

22:54 then build it back up.

22:57 (def x '([1235 ([0 17])] [1236 ([0 15] [16 8])]))

22:57 (let [[p c] (second x)] (map #(cons p %) c))

23:00 (let [[p c] (second x)] (map #(vec (cons p %)) c))

23:01 (mapcat (fn [[p c]] (vec (map #(vec (cons p %)) c))) x)

Logging service provided by n01se.net