#clojure log - Jul 27 2009

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

0:01 mebaran151: are the hashCodes for clojure collections consistent between JVM runs?

4:14 rdsr: Hi all, I have a problem with which I'm struggling since morning. First off here's the code

4:14 (ns maxtemperature

4:14 (:gen-class)

4:14 (:import [org.apache.hadoop.io IntWritable Text]

4:14 [org.apache.hadoop.mapred

4:14 JobConf JobClient

4:14 FileInputFormat FileOutputFormat

4:14 Mapper Reducer MapReduceBase])

4:14 (:use [clojure.contrib.str-utils :only (re-split)]))


4:14 (gen-class

4:14 :name mapper

4:14 :extends org.apache.hadoop.mapred.MapReduceBase

4:14 :implements [org.apache.hadoop.mapred.Mapper]

4:14 :prefix "m-")


4:15 (defn m-map [this key value output-collector reporter]

4:15 (let [line (str value)

4:15 year (subs line 15 19)

4:15 air-temperature (if (= (nth line 87) \+)

4:15 (Integer/parseInt (subs line 88 92))

4:15 (Integer/parseInt (subs line 87 92)))

4:15 quality (subs line 92 93)]

4:15 (if (and (not (= air-temperature 9999))

4:15 (re-matches quality "[01459]"))

4:15 (.collect output-collector (Text. year) (IntWritable. air-temperature)))))

4:15 jdz: lisppaste: url

4:15 lisppaste8: To use the lisppaste bot, visit http://paste.lisp.org/new/clojure and enter your paste.

4:15 rdsr:

4:15 (gen-class

4:15 :name reducer

4:15 :extends org.apache.hadoop.mapred.MapReduceBase

4:15 :implements [org.apache.hadoop.mapred.Reducer]

4:15 :prefix "r-")

4:15 jdz: rdsr: please stop pasting code here

4:15 rdsr:

4:15 (defn r-reduce [this key values output-collector]

4:15 (.collect output-collector key (apply max values)))


4:16 (defn -main [s]

4:16 (let [conf (JobConf. maxtemperature)

4:16 args (re-split " +" s 2)]

4:16 (doto conf

4:16 (.setJobName "Max temperature")

4:16 (.setMapperClass mapper)

4:16 (.setReducerClass reducer)

4:16 jdz: rdsr: you're not helping yourself here

4:16 rdsr: (.setOutputKeyClass Text)

4:16 (.setOutputValueClass IntWritable))

4:16 (FileInputFormat/addInputPath conf (first args))

4:16 (FileOutputFormat/setOutputPath conf (second args))))


4:16 (compile 'maxtemperature)

4:16 The problem is a direct translation of a hadoop job from the Book Hadoop The definite guide

4:16 The problem I'm facing is whenever i try to evaluate the function main

4:16 it cribs saying the its unable to resolve maxtemperature

4:16 oh sorry

4:17 I paste it there

4:17 thks for the info

4:17 sorry I'll paste the code on paste.lisp.org

4:17 lisppaste8: rdsr pasted "hadoop example not working" at http://paste.lisp.org/display/84264

4:18 rdsr: Hi all, sorry about the previous paste

4:18 What i wanted to know was what is it i'm doing wrong when I try to evaluate the main function bu it cribs saying that "its

4:19 unable to resolve the symbol maxtemperature"

4:19 Without the main function the code compiles fine

4:20 I guess my classpath is set correctly since I'm able to compile my code without the main function

4:20 Fossi: you need to compile the gen-classes first

4:20 jdz: well

4:21 Fossi: but (as i just learned on the weekend) most likely you want to use proxz a whole lot more

4:21 jdz: what is the variable maxtemperature in the main function supposed to contain

4:21 ?

4:21 rdsr: its a class name which i define in the beginning of the code

4:21 I hope my syntax is correct

4:22 The JobConf expects a parameter of type Class

4:23 Fossi: I tried compiling the class before the main function but I guess you can only write the compile function at the end of your file

4:24 Fossi: Also I think proxies won't help in that they just give me an on the fly obj what I need is a class

4:29 Fossi: you need a class because of what?

4:29 do you actually pass the class object to the hadoop stuff?

4:30 rdsr: Fossi yes

4:30 Fossi: ok, then i'd put my gen-class things in a different file and compile them aot

4:31 rdsr: yes JobConf expects a class object

4:31 hmmm

4:38 Fossi: I think the problem is something else cos, if you see the compilation example at http://clojure.org/compilation

4:38 AWizzArd: ~seen kotarak

4:38 clojurebot: kotarak was last seen quiting IRC, 4945 minutes ago

4:38 rdsr: the main function does access the constructor of the generated class

4:40 even if i modify my code, like that of the example, the compiler again gives the same error

5:48 lbj: Top of the morning gents

5:57 cgrand: lbj: what happened to your old nick?

6:13 rottcodd: is it possible to access the metadata in this proxy? (proxy [java.util.Map] [] (metadata [] {:thing 1}))

6:16 Chousuke: Map does not have a metadata method, does it? :/

6:16 what are you trying to accomplish?

6:20 rottcodd: I want to attach some data to a proxy for dispatching on a multimethod

6:21 Chousuke: hm, well, you could always implement IMeta with it.

6:22 or you could use a map as a wrapper for the actual proxy object and have the metadata on the map.

6:39 rottcodd: (let [obj (proxy [java.util.Map] [] (metadata [] {:thing 1}))] ((get (.__getClojureFnMappings obj) "metadata") obj))

6:43 Chousuke: rottcodd: I'm still fairly certain java.util.Map has no metadata interface :/

6:43 rottcodd: if you want to implement Clojure-compatible metadata, you need clojure.lang.IMeta I think.

6:44 rottcodd: this has nothing to do with java.util.Map, it should work with any proxy object

6:47 Chousuke: Well, a proxy can implement multiple interfaces.

6:49 rottcodd: implementing clojure.lang.IMeta would be the right way to do it, but I want to avoid aot compilation

6:53 Chousuke: rottcodd: why would you need AOT compilation? you're using proxy, aren't you?

6:54 (proxy [your.class.Here clojure.lang.IMeta] ...)

6:54 rottcodd: ah, I wasn't thinking straight

6:55 thanks Chousuke

7:07 AWizzArd: Hello rhickey. I programmed a very simple lexer in Clojure. I did that manually because it is simple (natural language: whitespace, words, numbers and punctuation). Because I need this very often it must run extremly fast.

7:07 It is just a very small part of a whole system. To let it run as fast as possible I made a pure Java version. It's 25 LOC vs 34 in Java, so, no big deal. Now I see that Java is about 5x faster for many small texts.

7:07 But, this is strange, for very big texts the pure Clojure version is faster. Do you have an idea how this can be? Maybe the GC?

7:09 Chousuke: can you show the code?

7:11 AWizzArd: No, I can't show it (copyright issue). But what it does: tokenizing a string into words, numbers, whitespace and specials. A token is represented as a hashmap, for example {:token "Hello", :category :word, :start 0, :end 5}. In Java this is a java.util.HashMap.

7:11 Specials will be a hashmap where :token is a string of length 1.

7:12 The other groups (words, whitespace and digits) group together.

7:12 Chousuke: is the hashmap a structmap?

7:12 AWizzArd: So, partition-by is doing nearly the right job, with the exception of specials.

7:12 Chousuke: hmm

7:12 AWizzArd: Yes, the hashmap is a struct.

7:13 if you have a function get-category [char] which uses Character/isLetter(), Character/isWhitespace() and isDigit then partition-by nearly does the right job. So that's the code basically, and in Java a simple for loop.

7:14 But for some reason Java is slow for very big inputs.

7:14 maacl: Is there a good email library for Clojure? or does one use JavaMail?

7:17 Chousuke: AWizzArd: are you using substrings in java?

7:17 really hard to think of any reasons without seeing the code :P

7:18 are you sure the clojure version is fast for big inputs and not just lazy? :)

7:18 AWizzArd: Chousuke: yes, I use substring in Java and subs in Clojure. First I tried collecting the single chars and then applied str to that. But subs is a good bit faster, so I used the same strategy in Java as well.

7:19 Chousuke: right, it is not lazy, I count the number of tokens so that each is touched.

7:22 maacl: ah just found Postal http://github.com/drewr/postal/tree/master

7:22 Chousuke: I suppose it might be the laziness. you never need to hold on to the head, do you?

7:23 AWizzArd: I collect all tokens of my input text in a vector. And I count the number of elements.

7:23 Chousuke: okay, so you do hold on to the whole data :P

7:23 hm

7:23 AWizzArd: Timing example for one MB text: Clojure 1800 msecs, Java 2400 msecs

7:24 Timing example for 30 texts of 2kb - 25kb each: Clojure 100 msecs, Java 17 msecs

7:25 When I let my make-token function just return a string and not an instance of a struct map then Clojure performs in 1000 msecs in the first example.

7:25 When I remove the makeToken function in Java it reduces the above timing to 400 msecs.

7:25 Chousuke: how about java?

7:25 hm

7:26 AWizzArd: So, the real pain sits in the part where the 745k java.util.HashMap's are created.

7:26 Chousuke: you represent the tokens as HashMaps in java?

7:26 AWizzArd: yes

7:26 Chousuke: that would be it, I guess.

7:26 AWizzArd: I thought they would be so very fast? Not?

7:26 Maybe I should use Clojure structs directly...

7:26 Chousuke: StructMaps can share their base keys, leading to some speed and space efficiency.

7:26 HashMaps can't.

7:27 if you had a Token class with fields in Java, I bet it would be even faster

7:27 AWizzArd: I see. So using Clojures StructMaps in Java may speed things up and also make it more comfortable for me.

7:27 Chousuke: I will try both. I guess what you said about having a Token class will be fastest.

7:28 Does the wikibook say how one can use Clojures data structures in Java? :)

7:28 Chousuke: well, clojure maps are java.util.Maps

7:28 so as long as you don't need to modify them, they should work just fine with any function expecting those.

7:29 AWizzArd: For printing the direct Clojure class is nicer.

7:29 But right, I don't need to modify the tokens.

7:29 So, I could use a mutable datastructure.

7:30 Chousuke: or declare final fields :p

7:31 AWizzArd: How?

7:32 Chousuke: um, just class Token { public final String token; ...}?

7:32 I wonder if it works like that.

7:32 you obviously need to have a constructor that initialises them.

7:34 AWizzArd: Chousuke: do you know how I can use a clojure.lang.PersistentStructMap in Java?

7:34 I guess having the clojure.jar in the CP is the first step.

7:34 Chousuke: that's all you need as far as I know.

7:36 AWizzArd: But how would I then say the equivalent of (struct +token+ :token :category :start :end)?

7:36 uhm, defstruct of course

7:36 Chousuke: for that I suppose you need to do some magic to get to the clojure evaluattor

7:36 -r

7:36 er

7:36 -t :P

7:37 would probably be easier to :genclass something that offers java factory functions for whatever you need.

7:39 AWizzArd: in principle the PersistentStructMap will offer some constructors and methods. I will have a look at the sources, it will be there I guess.

8:29 cemerick: rhickey: I was about to reply on the list, but I thought (if you're available) that I'd confine my embarrassment to the channel :-)

8:31 Chouser: ~max

8:31 clojurebot: max people is 164

8:31 AWizzArd: not yet.. but soon again, probably in autum

8:32 Btw Chouser, do you know how I can create a clojure.lang.PersistentVector after I imported it into Java? It seems to offer two constructors...

8:33 Or is there a way how I can say: PersistentVector x = SomeClass/staticMethod("(vector 1 2 3)"); ?

8:37 Chouser: AWizzArd: the constructors that take a java.util.List and Object[] seem like good choices.

8:37 oh, sorry

8:37 factory methods.

8:38 AWizzArd: Chouser: ah, good

8:38 Chouser: clojure.lang.PersistentVector.create(Object[]{1,2,3}) or something, right?

8:39 AWizzArd: yes, the static methods look fine

8:39 Chousuke: I now made a Token class and use that in my Lexer instead of a HashMap. Blindingly fast stuff now.

8:40 Chouser: AWizzArd: those factory methods use the mutable vector state in the chunks branch.

8:44 AWizzArd: I don't know the chunk branch yet. I saw master and par.

8:53 rhickey: cemerick: there's nothing to be embarrassed about, it's not a bad thing to want (dynamic named classes), just not something Java was designed to deliver well

8:54 I had a big philosophical struggle with newnew over the weekend

8:54 on the surface, it looks like an interop thing, i.e. given its similarity to proxy

8:55 Fossi: [insert clever greek philosopher joke]

8:55 rhickey: but it really shouldn't be. It has to be looked at as a core Clojure construct

8:56 since Clojure is defined in terms of these interfaces, and the objective is to let Clojure define itself, Clojure needs to be able to define and implement interfaces

8:56 but it should do so minimally, not incidentally adopt the semantics of Java

8:58 concrete derivation in particular complicates things significantly

8:58 I'm seriously thinking about limiting newnew to implementing interfaces only

8:59 using closure semantics only for named values, no explicit notion of fields or ctors

9:00 I came up with a hybrid strategy for mutability - close over a value and declare it mutable/volatile in the newnew construct

9:01 cemerick: rhickey: the 'embarrassment' bit was definitely tongue-in-cheek :-)

9:02 rhickey: cemerick: actually I think public discussions of the named class issues are important, as few people understand the limitations - better understanding would engender more realistic expectations, also the value proposition of working well with modular systems

9:02 cemerick: I'm clearly not aware of a lot of the issues, especially any kind of design aesthetic w.r.t. low-level impl. My primary concern was: if there are no statically-named classes, perhaps especially w.r.t. core collections, then it seems that they would not be usable in any scenario where Serialization was a requirement: ORMs, JMS queues, etc.

9:03 Chouser: I like the idea of the volatile thing being named (again) in the newnew form, to clarify the syntactic scope.

9:03 rhickey: Chouser: the key thing is that it is not introduced there, just qualified

9:04 Chouser: yep, so it's still 'let' creating the things to be closed-over.

9:04 rhickey: cemerick: It is a good point, and I have to think about that

9:04 cemerick: OK, glad I'm not totally off the reservation.

9:05 MarkVolkmann: Hey Rich, hate to bother you and know your time is occupied by many other things, but if you get some time, can you take a look at my email question about LockingTransaction and a wait call? Hopefully it's an easy question for you.

9:05 rhickey: but it is not an argument for names in newnew, it is an argument against using newnew to define the core collections

9:05 cemerick: ah-ha.

9:06 Chouser: but surely if newnew is incapable of defining Clojure's own core collections, that indicates a critical weakness in newnew?

9:06 rhickey: MarkVolkmann: the reason for the wait call is to prevent optimistic spinning, i.e. the transaction is likely to get right to the same failure if it repeats immediately, by blocking on the current owner you prevent that waste

9:06 cemerick: rhickey: for a moment, assume no newnew usage would occur at runtime, and the entire body of clojure code is AOT compiled, then being able to define static names would be entirely harmless and only beneficial for interop, yes?

9:07 MarkVolkmann: Thanks Rich!

9:07 Chouser: or perhaps you are considering a second new construct for Java interop scenarios?

9:07 rhickey: MarkVolkmann: however, the use of wait specifically is an implementation detail, and likely to change in the near future, possibly to a countdown latch

9:08 cemerick: the problem with that assumption is the assumption

9:08 cemerick: yeah, I'm just firming up my conception of things.

9:09 rhickey: MarkVolkmann: I'm partial to the arguments made in this paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=

9:09 aargh: http://berkeley.intel-research.net/rennals/pubs/052RobEnnals.pdf

9:11 MarkVolkmann: Thank you very much! Looks like a good read.

9:12 rhickey: cemerick: also, even if intended for AOT, having to restart on bugfix would stink during development, but true too of genclass

9:12 cemerick: rhickey: I guess I'd say that, if my last supposition is correct, then newnew could be allowed to define static names at AOT-compile-time, to redefine classes associated with those names at runtime, but not be able to introduce new static names at runtime?

9:12 rhickey: above pdf is: Software Transactional Memory Should Not Be Obstruction-Free

9:13 cemerick: it's a helluva wrinkle, but it's an out for the interop issues, perhaps

9:13 AWizzArd: I would like to call the static public PersistentStructMap construct(Def def, ISeq valseq) method, to create a Struct, from within Java. How can I create a Def object. Def seems to be something like an inner class?

9:13 rhickey: cemerick: one can't: redefine classes associated with those names at runtime

9:14 cemerick: oh, righty, can't unload classes without classloader majick.

9:16 Actually, the class wouldn't have to be redefined, only new methods added.

9:16 AWizzArd: Calling PersistentStructMap.Def.createSlotMap(somePersistentVector) does not work.

9:16 rhickey: Chouser: not a second construct, as we have genclass, what's missing from it is indirection-free method invocation, but unlike proxy gen-class invocation is direct, not hashmap lookup

9:16 Chouser: AWizzArd: if you follow the code path from the clojure fns you know, you should be able to answer these questions yourself. That's all I'm doing for you anyway.

9:17 AWizzArd: Ah ok, I started in the PersistentStructMap.java file.

9:17 rhickey: cemerick: any kind of class reloading/redefining is too far off the beaten path for Clojure

9:18 cemerick: you mean, in order to work properly within osgi, et al.?

9:18 Chouser: AWizzArd: defstruct expands to create-struct; create-struct call PersistentStructMap.createSlotMap()

9:18 rhickey: cemerick: right, and everything else (tools etc)

9:18 cemerick: I don't want to take on 'fixing' Java

9:18 cemerick: no, certainly not.

9:19 But, it would be a pretty big deal if, to put a clojure map into a JMS queue, you had to copy it into a java.util.HashMap. :-/

9:19 rhickey: also I think you'd run into serialization issues with modified versions of the same-named class

9:19 AWizzArd: Chouser: how do you follow that code path? Do you use some kind of stepper?

9:20 Chouser: AWizzArd: no, I open core.clj, search for defstruct, read it, search for create-struct, etc.

9:20 rhickey: cemerick: you aren't using prn for that?

9:20 Chouser: AWizzArd: vim has a handy keystroke that finds other occurances of the word under the cursor. I use that a lot.

9:21 AWizzArd: Chouser: oki, I will see what I can find there. Thanks for the suggestion. I was hoping that I know enough Java by now to find it out by looking at the direct .java file. But Rich used some more complex constructs that I don't know the syntax of.

9:21 cemerick: rhickey: Nope. Just a PersistentVector, directly. Serialization works just fine for transient stuff.

9:22 Chouser: AWizzArd: but that path just leads to createSlotMap -- why did you say that doesn't work?

9:22 cemerick: (I only mentioned map above off-hand)

9:22 Actually, we don't use maps for that purpose because of the interning issues with keywords

9:22 rhickey: cemerick: I haven't audited Clojure's data structures for serializability at all

9:22 a todo item

9:23 cemerick: well, it works well enough at the moment :-)

9:23 AWizzArd: Chouser: I saw in PersistentStructMap.java an inner class Def. Only for that class I found the createSlotMap method.

9:24 So I thought it could be good to somehow mention Def in the call. Either Def.createSlotMap() or PersistantStructMap.Def.createSlotMap() sounded good.

9:24 cemerick: rhickey: and honestly, I never thought twice about it -- in my head, core datastructures should *always* be serializable. I would have thought it a bug if they weren't, actually. There's just too many APIs out there that depend on it.

9:24 rhickey: http://groups.google.com/group/clojure/browse_frm/thread/9b7ef55a719f4b0b/80897665ffed6f9f?lnk=gst&q=serializeable+keywords#80897665ffed6f9f

9:25 cemerick: you want to submit a patch for keywords?, the poster doesn't have a CA

9:26 cemerick: but AOT-compiled newnew classes *will* have names, so if you've got the same jar on both ends you should still be able to (de)serialize

9:26 cemerick: that's interesting, we've successfully serialized keywords plenty. It doesn't quite work, though, as the deserialized keyword instances aren't properly interned.

9:27 Maybe our framework uses reflection to back into a serialization.....?

9:27 rhickey: cemerick: reflection should work too, the only problem is code that explicitly uses the names

9:28 cemerick: rhickey: OK, but you shut down your JMS server (with messages enqueued), deploy a new version of clojure, and *boom*

9:29 rhickey: cemerick: I understand, but depending upon what changed, it could go boom anyway

9:30 serialized objects as messages being extremely fragile

9:30 cemerick: yeah, just illustrating

9:30 rhickey: prn/read!!

9:30 AWizzArd: Chouser: but now it works. I didn't know that Java allows to call static methods of inner static classes by not mentioning the name of those inner classes.

9:31 Chouser: AWizzArd: hm! I still don't know that. :-)

9:32 rhickey: cemerick: so it's not merely names, but serialVersionUIDs etc

9:33 AWizzArd: Chouser: yeah, but you just teamed up with me, and we found that out. We are becoming Java experts. Maybe.

9:33 rhickey: cemerick: perhaps the collections should define read/writeObject

9:33 cemerick: rhickey: I'm completely sympathetic, but prn/read can't be the answer to interop.

9:34 yeah, that's what I was offering as a potential solution in my first email, but still, there needs to be a static name, or the ObjectInputStream will never be able to find a corresponding readObject method.

9:34 rhickey: cemerick: ah, nevermind

9:36 * rhickey considers serialization for non-ephemeral use extremely fragile

9:37 cemerick: I think everyone would agree. However, definitions of 'ephemeral' differ (e.g. my example of halting a JMS server).

9:38 rhickey: neat: http://clojuratica.weebly.com/

9:38 cemerick: heh, that's pretty crazy

9:40 rhickey: cemerick: I'm just playing devil's advocate, the serialization issue you've raised is an important one that I haven't given enough consideration, being personally afraid of serialization-based solutions

9:42 cemerick: the really crazy thing being, serialization is baked into so much of what larger jvm apps/frameworks do...

9:43 rhickey: cemerick: that is the key argument, even if you don't choose it, it may be forced upon you, thus it is an interop issue

9:44 AWizzArd: that clojuratica seems to be a nice helper.. wow :)

9:47 cemerick: rhickey: I think serialization could be really great, especially if it were straightforward to get print-dup-driven read/writeObject impls on clojure objects. You'd then have the benefits of both, I think: future-proofed serialization output sitting on top of the well-tested Serialization backend. The only hurdle is those static names.

9:49 rhickey: cemerick: there's really no hurdle with gen-class style AOT

9:51 perhaps newnew could be made to fail/warn if named and re-evaluated, i.e. no new code will be associated with the class

9:51 until restart

9:51 Chouser: rhickey: so at the moment you're advocating that cinc uses gen-class to implement the core data structures?

9:52 cemerick: yes, you can certainly write your own read/writeObject impls, but if your object graph is entirely clojure-based, then you should be able to generate it based on classname/fields/etc

9:52 ("you" being the royal "you", not necessarily rhickey :-) )

9:53 rhickey: yes, that's what I was suggesting earlier: newnew produces stable names at AOT compile

9:53 rhickey: Chouser: Not at all, I think the data structures should be defined with something with significantly simpler semantics than gen-class

9:54 cemerick: that's not what I'm saying, I'm saying if named, will produce with that name, and if named and reloaded, can't reload so complains

9:54 produce class with that name

9:54 Chouser: gen-interface did that once upon a time

9:54 rhickey: so not limited to AOT

9:55 Chouser: right, which brings up the second set of problems, if name is used explicitly at runtime the dynamically-loaded nature of the bytecode will be a problem, since not visible from all classloaders

9:56 if named would have to use the compile-to-file-in-classpath-dir I mentioned in my clojure-dev message

9:57 Chouser: like AOT does now while compiling.

9:57 rhickey: that would let it be loaded by the same classloader that loaded clojure itself, so fine for most interactive/scripty things

9:58 but I still dread the - this isn't working dynamically under Tomcat scenarios

9:59 I don't know if people would understand the 'now that you've named this it is static' aspect

9:59 not having explicit names is much cleaner

10:02 Chouser: so maybe still two worlds, newnew is really generate an instance of an anonymous implementation of some interfaces, and some static AOT-only generate a named implementation of some interfaces

10:02 AWizzArd: When I have a static method FOO in a Java class which uses new to create some immutable objects, then I can run pmap over FOO without problems yes? I don't need to synchronize that method somehow right?

10:03 rhickey: in both cases substantially reduced semantics from proxy/gen-class

10:03 AWizzArd: One and the same static methods can run in 20 threads at the same time, and each thread works on its own new'ed objects?

10:03 rhickey: AWizzArd: yes

10:03 AWizzArd: good

10:04 rhickey: when you pmap inc it is calling new Integer at some point

10:06 * rhickey awaits complaints about only being able to implement interfaces

10:07 cemerick: rhickey: as long as proxy (or some corollary) remains, I don't think you'll hear too much

10:08 rhickey: I guess another option is that, if named, newnew is a no-op when not AOT

10:09 cemerick: (however, the newnew wiki page saying that it will replace proxy is definitely confusing if that's the case)

10:09 rhickey: cemerick: that is an ideas page, it will contain conflicting ideas until I've worked it out, then another documentation page will descibe what won

10:10 cemerick: rhickey: you mean ideas don't pop out of your head fully-formed and with three potential implementations?!? ;-)

10:10 rhickey: until this weekend I thought it would replace proxy, now I feel making some very simple subset semantics is important

10:12 cemerick: the wiki page is an experiment in exposing the true nature of how this stuff comes about, and an effort to get others involved, and input, but scarily revelatory

10:12 cemerick: maybe newnew never takes a name, but another form links a static name to a newnew impl, copying its bytecode?

10:13 rhickey: cemerick: all paths to such things involve ctors in a way I'd like to avoid. I'm currently thinking about these things as closure-objects

10:14 that too may become an interop issue

10:14 Chouser: if newnew doesn't help with core data structures, is it needed before cinc? Or do we just need the reduced gen-class?

10:15 rhickey: but I've been thinking about auto-generating static factory methods

10:15 Chouser: leaving newnew as a high-performance introp thing for later.

10:16 rhickey: Chouser: newnew is not about interop, it is about cinc - what's missing from Clojure to let it define itself?

10:16 right now part of Clojure is defined with Java semantics - those interfaces at least

10:18 a simple approach would be to say, well, use gen-class to implement those interfaces, and worry about making gen-class faster later

10:19 Chouser: interfaces and (what are they called...) abstract implementations

10:19 rhickey: but gen-class adopts many complex semantics of Java, necessary for interop perhaps but not the cinc job. I'd like newnew to be something whose semantics are defined by Clojure, not be reference to Java, and implementable easily in more places than the JVM

10:20 Chouser: right, and avoiding fields, access specifiers, ctors, implementation inheritance etc etc

10:20 but with an easily explained mapping to the host

10:21 Chouser: ok, so because all this is for cinc, ignore java interop for now and find the clean minimal requirements for named and (perhaps separately) anonymous things that can be mapped to high-performance host constructs.

10:22 rhickey: exactly

10:22 Chouser: is there anything in clojure's core .java stuff now that could be anonymous?

10:22 rhickey: its for cinc and its the story for writing high-perf constructs like the core data structures in (portable?) Clojure

10:23 it's

10:23 I think i could define all of Clojure this way, all the data structures, everything

10:24 the only hitch is the abstract base classes

10:24 Chouser: "this way" being a sort of simplified gen-class? or anonymous newnew thing?

10:25 rhickey: mine I could get rid of, but the java.util.AbstractMap et al are really useful

10:26 Chouser: I don't need the names but cemerick does in order to serialize them, but let's consider it one thing, a way to implement interfaces, given gen-interface as a way to define them. The runtime/AOT issues are orthogonal

10:27 I was working this weekend on trying to treat abstract classes as mixins

10:27 Chouser: yeah, I found that concept quite intruiging

10:27 rhickey: I took a look at Scala's Traits and was horrified

10:31 I decided I don't want or need a way to define abstract classes in Clojure, but on the fence about newnew being able to consume them (as if an interface), subject to a no-arg-ctor rule, which is true for the java.util AbstractBases

10:32 Chouser: huh. You'd rather repeat chunks of code for clojure core classes? What about Traits drove that home?

10:32 rhickey: Chouser: no, but there are other options for CLojure, using macro-like things

10:32 Chouser: ah, ok.

10:33 rhickey: the traits thought was separate, just unsettlingly complex and nasty

10:33 Chouser: sure, I've got no objection there.

10:37 so if cemerick didn't need names, all cinc would really need is a simple newnew (implementing only pure interfaces) and a handful of macros.

10:38 rhickey: Chouser: I think so

10:39 what about an :aot-compile-as fixed.class.Name flag on newnew?

10:39 Chouser: such a pity. #=(hash-map :a 1, :b 2) looks good to me. :-P

10:41 so that would use a gen'ed name at runtime if that class name was not found?

10:41 rhickey: it would always use a gen-ed name when not AOT compiling

10:42 or yes, if not found

11:04 AWizzArd: I compile my .java file while still editing it every two minutes. How can I reload the new class file (sits in my CP) in Clojure without restarting it?

11:05 stuartsierra: AWizzArd: you can't, not without using custom ClassLoaders or an IDE like Eclipse.

11:05 AWizzArd: can I add new classes that I compiled after Clojure started, but which I didn't use yet?

11:06 stuartsierra: I don't know; probably yes.

11:07 AWizzArd: maybe via add-classpath

11:07 I just put the compiled file into new paths and then load those classes ;-)

11:07 stuartsierra: No, that adds a JAR or directory containing compiled files.

11:07 maybe

11:11 AWizzArd: Btw stuartsierra, what stepper do you use for your Clojure programming?

11:11 Chouser: if the class has the same name, I don't think it matter what directory it's getting loaded from -- you can still only do it the first time.

11:11 AWizzArd: Chouser: yes, guess so. This means happy restarting of Clojure every few moments. The staticism of Java transports to CLJ.

11:12 Chouser: AWizzArd: when writing .java, yes. proxy and gen-class both do work for you to allow loading new implementations into a running jvm.

11:12 stuartsierra: This is what OSGi frameworks and Java IDEs were made to solve, but they're big, complicated mechanisms.

11:13 mebaran151: is there a simple functional way to turn an integer into an array of bytes or do I have to do it imperatively?

11:14 I want to cast Integer into a bigendian representation and put that in a byte array

11:14 stuartsierra: Look at the java.lang.Integer class; there should be a method that does that.

11:15 mebaran151: a couple methods will reverse the bytes, which I don't think I want, but none of them seem to return an array of bytes

11:16 I'm trying to make an efficient byte order sortable representation of integer

11:18 big endian integer encoding should work

11:23 achim`: ,(-> 12345 bigint .toByteArray)

11:23 clojurebot: #<byte[] [B@18119d7>

11:26 hiredman: busy morning

11:27 Chouser: no way to alias a classname yet, right?

11:30 Jomyoot: what is yield :javascript ?

11:30 <%= yield :javascipt %>

11:30 what does it do?

11:31 Chouser: Jomyoot: that looks like ruby to me. where are you seeing that?

11:36 Jomyoot: wrong channel

11:36 i am seriously sorry

11:36 Chouser: :-) np

11:44 rhickey: Chouser: not yet

11:51 Chouser: It'd be handy if proxy methods had fns named like the methods they implement

11:51 hiredman: eh?

11:52 aren't proxy methods fns?

11:52 Chouser: yes, anonymous fns

11:52 Wrong number of args passed to: Foo$array-input-stream--1231$fn

11:53 that error could have the method name in there.

11:53 hiredman: ah

11:56 Chouser: hm, well that was easy.

11:57 * Chouser awaits a ticket invitation

12:07 Chouser: how can I get the size of a Java array from Clojure?

12:07 hiredman: count?

12:07 ,(count (into-array [1 2 3]))

12:07 clojurebot: 3

12:09 gulagong: hi

12:09 i want to do fast matrix multiplikation

12:09 but i don't get it fast ;)

12:09 maybe on of you wizzards could have a look at the code?

12:10 stuartsierra: matrix multiplication is kind of specialized, you probably want a dedicated matrix library

12:10 Chouser: hiredman: ha! thanks.

12:10 hm, 'counted?' says false. But that's not right, is it?

12:10 gulagong: i want to do it myself...

12:10 stuartsierra: counted checks inheritance

12:11 Chouser: stuartsierra: right, but I wonder if it should have a special case for Java arrays

12:11 hiredman: ,(doc counted)

12:11 clojurebot: "/;nil; "

12:11 hiredman: ,(doc counted?)

12:11 clojurebot: "([coll]); Returns true if coll implements count in constant time"

12:12 gulagong: my concrete broblem is: (get (long-array '(1 2 3 4 5)) 2) is not guaranteed to be a long-value

12:12 so this slows me down

12:14 and i don't see a reason in having long-arrays at all if they don't return longs

12:14 Chouser: gulagong: you've looked at the type hint and java primitive info at http://clojure.org/java_interop ?

12:15 gulagong: chouser: yes, but i'll do again

12:15 chouser: hm, aget seems promising

12:16 chouser: next time i'll read better

12:16 Chouser: gulagong: np.

12:19 rhickey: anyone explicitly deriving from RestFn?

12:20 hiredman: ~def c.l.RestFn

12:23 gulagong: speedup 5 :)

12:23 Chouser: gulagong: 5x ?? nice

12:26 ,(expression-info '(let [a (atom 5)] (+ (int @a) @a)))

12:26 clojurebot: {:class java.lang.Number, :primitive? false}

12:26 Chouser: ,(expression-info '(let [a (atom 5)] (+ (int @a) (int @a))))

12:26 clojurebot: {:class int, :primitive? true}

12:34 rzoom: where should i post a request for comment on a contrib patch? here? a mailing list?

12:35 Chouser: rzoom: either one. You've read http://clojure.org/patches ?

12:36 rzoom: yes, i have it in a github repo

12:36 but it maybe easier to just attach the patch.

12:37 Chouser: sure, git-formatted patches are nice

12:38 as described there. I asked because that page answers your original question (bring it up on the google group) and also has a reminder about the CA

12:38 rzoom: yeah, sent CA last week

12:39 wasnt sure about emailing to google group since I am not a member

12:39 Chouser: hm. It probably won't let you if you're not a member. It might be sufficient to bring it up here, depending on the patch.

12:40 rzoom: okie, will bring it up in here tomorrow, when I am off, hopefully get some feedback.

12:42 lbj: rhickey: Rour reference to 'STM should not be obstruction free', leaves me wondering: Are you planning to optimise the STM per their advice, or have you already done that?

12:42 rhickey: lbj: Clojure's STM is (already) not a spinning design

12:45 lbj: rhickey: Alright, and your meta-data handling matches what they described?

12:46 rhickey: lbj: not at all

12:46 completely different implementation

12:46 lbj: Ok

12:46 To me, meta-data is meta-data, but I haven't got a clue about the low level handling of it

12:58 cemerick: Goodness, the innards of java serialization(+externalizable) are just *horrible*.

12:58 That said, the breadth and depth of APIs that depend on it is staggering.

13:35 Fossi: the {var :keyword} destructuring always gets me. why is it that way around?

13:36 Chouser: to allow for options like :or, :as, :keys, etc.

13:36 Fossi: hmmm. makes sense

13:37 Chouser: ,(let [{a :alpha, :keys [b c d], :as all} {:alpha 1, :b 2, :x 3}] [a b c d all])

13:37 clojurebot: [1 2 nil nil {:alpha 1, :b 2, :x 3}]

13:40 * Fossi can't find the docs on that on the clojure page

13:43 Chouser: http://clojure.org/special_forms#let

13:45 Fossi: ah, ok. i was looking for the defn variant, but i guess that's just a macro using let

13:47 Chouser: we don't have anything that encapsulates a blocking queue yet, right? seque notwithstanding

13:47 Fossi: ah, it works with anything which supports nth. that's neat

13:49 cemerick: Chouser: not that I'm aware of, no

13:51 interesting that APM.keySet() returns an anonymous AbstractSet.

14:09 Fossi: is there a nice way to mock a class that's normally generated by gen-class for testing?

14:10 Chouser: you can use 'binding' to replace (locally in a thread) the definitions of any of the method

14:11 Fossi: @(.state activity) i'm tring to mock that

14:11 the activity

14:12 Chouser: (binding [full.namespace.name.Activity/-state (fn mock-method-def [] ...)] @(.state activity)) ...

14:13 of course by now you should have asked yourself if gen-class really is the right solution...

14:13 Fossi: ah. the /- part does the magic :)

14:13 Chouser: well, that's the name of the Var that's implementing your class, right? That's what you defn'ed in the .clj file

14:13 Fossi: well, in this case this is my interface to the world, so i have to genclass that :)

14:14 Chouser: Fossi: I'd still highly recommend using gen-interface and proxy whenever possible.

14:14 Fossi: well, this is handed to android, so not possible :\

14:14 Chouser: Then gen-class only for static methods

14:14 oh. :-P

14:14 wait

14:14 why not?

14:15 which of those don't work with android?

14:15 Fossi: well, what would bootstrap the proxy?

14:16 Chouser: proxy macros AOT compile to regular named classes

14:17 Fossi: that i didn't understand

14:17 Chouser: have you tried using proxy with android?

14:17 Fossi: yes, it works for the inner things

14:17 clojurebot: for is not a loop

14:18 rhickey: gen-interface and proxy should work fine with Android when AOT compiled

14:18 * Fossi scratches head

14:18 Chouser: use gen-class to define a class with only static methods

14:19 Fossi: ah, ok. that might work

14:19 Chouser: those static method(s) can then act as factories -- using proxy to create new instances of objects that implement your gen-interface'd interfaces.

14:20 I pretty recently and for the first time created a clojure lib meant for consumption by java. Doing it this way is beautiful and clean.

14:20 Fossi: still, i need to hold my state somewhere inbetween calls from the android api

14:21 Chouser: I ended up with a single my/Foo.clj that had a gen-class and several gen-interfaces, but no actual implementation at all.

14:21 the gen-class names my.foo as the implementing namespace

14:21 sorry, my.foo.clojure

14:22 so then I had a my/foo/clojure.clj that had all the helper fns, macros, and the gen-class method implementations

14:23 Fossi: so, if you get called multiple times, where do you keep your state?

14:24 Chouser: proxies can close over state

14:25 (defn make-foo [a b c] (let [mutable-state-thing (atom d)] (proxy ...)))

14:25 Fossi: hmmm. i guess i'll look at that when i have something working with one dynamic gen-class :)

14:25 Chouser: so in there your proxy fns can use the a b c "constructor" values and do things with d

14:40 how do I specify a byte[] parameter to a gen-class method?

14:41 stuartsierra: try "[B"

14:41 Chouser: not: "[B" "byte[]" "B[]" #=(eval '(class (make-array Byte/TYPE 0)))

14:41 gulagong: (let [la (long-array '(1 2 3 4))] (do (aset la 1 10.0) (aget la 1)))

14:41 ,(let [la (long-array '(1 2 3 4))] (do (aset la 1 10.0) (aget la 1)))

14:41 clojurebot: java.lang.IllegalArgumentException: argument type mismatch

14:42 gulagong: d'oh

14:42 ,(reduce #(conj %1 (vec (repeat 5 val))) [] (range 3))

14:42 clojurebot: java.lang.IllegalArgumentException: Wrong number of args passed to: sandbox$eval--5137$fn

14:42 gulagong: i don't understand this ....

14:43 maybe a bug?

14:43 stuartsierra: reduce always takes a 2-argument function

14:43 gulagong: aye

14:43 i see and its anonymous

14:44 Chouser: maybe you wanted %2 instead of val ?

14:44 mebaran151: what's the best way of communicating something akin to an interface

14:45 I'd like to make it clear that if you implement a certain set of functions, you can plugin a different backend to my hashdb library and it will work

14:46 Chouser: mebaran151: using a few defmethod calls?

14:46 stuartsierra: Or just describe it in documentation.

14:46 mebaran151: yeah I have a nice big comment at the top

14:46 AWizzArd: Is it possible to create a new (Java) class at runtime? One would give it a name, say what attributes it has (all are private and final) and add a method to it?

14:47 Chouser: AWizzArd: no

14:47 AWizzArd: ah, too sad

14:47 Chouser: AWizzArd: http://groups.google.com/group/clojure-dev/msg/04ef3280f7678e54?hl=en

14:48 "So, no more pretending. If you want a class name to be visible you

14:48 must compile a .class file and put it in the classpath. "

14:48 gah. sorry.

14:48 AWizzArd: It would have been nice as an alternative to defstruct to have a defclass. That could give an ideal candidate for throw away objects.

14:49 Today I found how extremly fast it is to create hundreds of thousands of instances of a class, and gc it.

14:50 Pure Java often still has the edge in performance.

14:52 mebaran151: also I have some common support functions between database backends

14:52 * Chouser wishes for *compile-files* in Java

14:52 rhickey: Chouser: ?

14:52 mebaran151: however, these support macros expect certain functions to be defined in the package that imports

14:52 *in the package that requires them

14:53 rhickey: AWizzArd: you create classes in Clojure all the time with fn

14:53 angerman: how would I make items in a database user-positional? e.g. the user gets a list of items [a b c d ...] and can sort them, e.g. [a d b c ...]. Does anyone know a nice approach to that problem?

14:54 Chouser: rhickey: There's probably another solution, but I've got this jni lib auto-loading when a class is loaded now. This is nice except when compiling Clojure code that depends on it, it expects the whole runtime environment to be available.

14:54 AWizzArd: rhickey: you mean creating closures?

14:54 stuartsierra: AWizzArd: fn actually creates a class at run-time

14:55 AWizzArd: And how can I define the fields?

14:55 rhickey: Chouser: that's really an issue with Clojure compiler using reflection for type info, on the todo list for cinc compiler to get rid of that

14:55 AWizzArd: let

14:56 Chouser: rhickey: how else would it know what to compile?

14:56 rhickey: Chouser: jar files, like javac

14:56 stuartsierra: Huh, never thought of it that way. So (let [x 1, y 2] (fn [] (prn x y))) creates a class with public member fields x and y?

14:57 cemerick: rhickey: earlier, you mentioned something about gen-class not having the performance issues of proxy...did I misunderstand you? I thought gen-class' bytecode was 2/3 levels of indirection away from a fn (genclass method -> Var -> fn), similar to proxy?

14:57 Chouser: rhickey: oh. my.

14:57 rhickey: cemerick: proxy does a hashmap lookup

14:57 Chouser: stuartsierra: private member fields, populated in the constructor

14:57 AWizzArd: rhickey: will that also use type hints i gave in the let? And will it make the fields final?

14:58 cemerick: rhickey: right, forgot about that.

14:58 rhickey: cemerick: gen-class does no lookup, mere indirections aren't necessarily impediments to hotspot inlining

14:58 Chouser: oh. public final

14:59 * rhickey is changing fn as you guys are looking at how it used to work...

14:59 Chouser: :-)

14:59 rhickey: IntelliJ refactoring is awesome

14:59 AWizzArd: hmm

14:59 why?

15:00 Will it change all affected files for you automatically?

15:00 Chouser: whoa! public non-final!?

15:00 ,(let [foo 1, obj (fn [] (prn foo))] (set! (.foo obj) 10) (obj))

15:00 clojurebot: java.lang.NullPointerException

15:00 Chouser: ,(let [foo 1, obj (fn [] (prn foo))] (.foo obj))

15:00 clojurebot: 1

15:01 Chouser: anyway, I still can't figure out how to tell gen-class this method wants a byte array arg

15:01 clojurebot: this is not a bug

15:02 AWizzArd: What I did today was writing a very short Java class, where PF stands for private final class Token { PF String token; PF int start; PF int end; } plus getters for these three fields.

15:02 rhickey: "[B"

15:03 AWizzArd: My Lexer will create instances of Token. That was dramatically speeding up my code. Before I used java.util.HashMap instead of Token.

15:03 Chouser: java.lang.ClassNotFoundException: java/lang/[B

15:03 clojurebot:

15:04 AWizzArd: I guess this is still more performant than using a structmap, but of course those can't grow.

15:04 rhickey: will those fns make use of the type hints I gave in the let?

15:04 Chouser: hm... maybe it works with (gen-class ...) but not (ns ... (:gen-clas ...))

15:05 rhickey: AWizzArd: yes

15:05 AWizzArd: oh that's good then

15:05 mebaran151: how can one namespace export symbols into another?

15:07 rhickey: mebaran151: can't

15:07 * stuartsierra is finally working on a Clojure/Hadoop library.

15:08 Chouser: (gen-class :name my.Foo :methods [#^{:static true} [newArrayInputStream [int "[B"] java.io.InputStream]])

15:08 mebaran151: well then what would be the best way to setup a set of support macros that another namespace can require or use?

15:08 AWizzArd: mebaran151: you can use things that you def'ed or defn'ed in one namespace from others.

15:09 Chouser: mebaran151: if namespace a.A has (defmacro aaa ...), then in namespace b.B you can (require 'a.A) and use (a.A/aaa ...)

15:10 rhickey: ,(Class/forName "[B")

15:10 clojurebot: [B

15:10 mebaran151: I know that, but the problem is the macros wrap functions in the namespace to which they are imported

15:11 Fossi: wow. that is one ugly expression :D

15:11 hiredman: eh

15:11 Chouser: mebaran151: ah.... if the functions aren't marked private, it should Just Work.

15:11 hiredman: Chouser: he might not be using `

15:12 Chouser: hm. true.

15:12 mebaran151: let me set up the problem a little more clearly: I have a macro that wraps up transaction handling: it requires a definition of open-transaction commit and abort

15:12 hiredman: mebaran151: so use ` so your macros emit namespace qualified symbols

15:12 rhickey: so, (new [interfaces] :as this ...) or (obj this [interfaces] ...)

15:12 hiredman: the second one

15:12 mebaran151: I do use `

15:13 let me just do a pastie: it'll be easier to explain

15:13 cemerick: rhickey: first one

15:13 (with 'this' as default?)

15:14 Chouser: I prefer 'new' over 'obj'

15:14 rhickey: cemerick: no defaults since these will often be wrapped by macros (think fn) and thus injected names won't be apparent

15:14 anyone remember thisfn?

15:14 cemerick: indeed

15:14 rhickey: didn't work

15:15 hiredman: I'd prefer a minimum of :as :with :doing :etc

15:15 rhickey: obj could be object or instance

15:16 lisppaste8: mebaran151 pasted "Macro Injection" at http://paste.lisp.org/display/84280

15:16 cemerick: presuming this is going to be used often (still a proxy replacement...?), shorter is better

15:17 rhickey: a lot of uses don't need 'this'

15:17 Chouser: It seems plausible that a few common use cases could be implemented as macros on top of 'new', just like fn will be

15:17 rhickey: yes

15:17 cemerick: it's hard to get better than new. could be (new [interfaces] ...), skipping 'this'

15:18 mebaran151: see, those macros rely on a parent namespace supplying implementations of cursor, transaction, commit etc

15:18 rhickey: cemerick: this optional in either case

15:18 mebaran151: how would I get these functions to use the functions in the namespace into which they were imported

15:18 cemerick: then I'd go with option #2, but with 'new'

15:18 hiredman: oh

15:18 rhickey: cemerick: that's not being offered, ambiguous with current new

15:19 hiredman: mebaran151: so you *want* unsafe macros?

15:19 cemerick: Oh, I figured the current new was being retired!

15:19 nevermind then :-)

15:19 mebaran151: I think I may

15:19 rhickey: current new is not being retired

15:20 mebaran151: this code is useful for more than my bdb library: I thought it would be nice to somehow abstract it, though I haven't found a good way

15:20 rhickey: ,`~'foo

15:20 clojurebot: foo

15:21 rhickey: a name emitted like that will capture the name in the scope in which it is expanded

15:22 sgtarr_: ,`~'bar

15:22 clojurebot: bar

15:22 rhickey: ,`rest

15:22 clojurebot: clojure.core/rest

15:22 sgtarr_: oh.

15:22 rhickey: `~'rest

15:22 ,~~'rest

15:22 clojurebot: java.lang.IllegalStateException: Var clojure.core/unquote is unbound.

15:23 rhickey: `~'rest

15:23 ,`~'rest

15:23 clojurebot: rest

15:23 rhickey: there

15:23 :)

15:23 ataggart: lol

15:23 * rhickey can't type

15:23 sgtarr_: what if someone gives an infinite loop to the bot?

15:23 Chousuke: I'd still like the args to newnew be just a map literal :/

15:23 slashus2: sgtarr_: It times out.

15:23 sgtarr_: I see :)

15:24 hiredman: Chousuke: makes sense to me

15:25 ,(new [Thing] {methodname ([args] stuff doing)})

15:25 clojurebot: java.lang.IllegalArgumentException: Unable to resolve classname: [Thing]

15:26 Chousuke: hiredman: hmm, I meant the keyword args though.

15:26 hiredman: but that could work too :P

15:26 rhickey: and overloaded methods?

15:27 Chouser: #=(java.lang.Class/forName "[B") works

15:28 Chousuke: I mean, if newnew potentially could have parameters other than :as and :volatile, it would be easier to have them passed around in a real map, so macros using newnew can just look up the args they're interested in, or use merge to allow the user to override "defaults" easily.

15:28 rhickey: Chouser: seems like a bug in genclass' the-class

15:28 hiredman: (new [Thing] {methodname (([args] stuff doing) ([arg1 arg2] wild and crazy stuff))})

15:29 rhickey: (some #{\.} strx) => (some #{\. \[} strx)

15:32 Chouser: gah! I didn't even want a byte array. sheesh.

15:32 rhickey: Chousuke: so, not about the methods, (new [interfaces] {:as this ...} ...)

15:33 Chousuke: yeah, like that.

15:35 Chouser: rhickey: I'm curious if you'd entertain syntax that avoids the sub-form (foo ...) when defining a newnew method foo

15:35 proxy is currently one of the few places that (foo ...) means anything other than invoking foo.

15:35 rhickey: Chousuke: I don't care much, but those complaining about :as might

15:35 Chouser: what about fn?

15:35 mebaran151: , (fn inf [] (inf))

15:36 clojurebot: #<sandbox$eval__5175$inf__5177 sandbox$eval__5175$inf__5177@300522>

15:36 mebaran151: , ((fn inf [] (inf)))

15:36 clojurebot: java.lang.StackOverflowError

15:36 ataggart: don't poke clojurebot

15:36 Chousuke: mebaran151: you can't send it to an infinite loop that easily :)

15:36 Chouser: rhickey: as in (new [Foo] (fn fooMethod [] ...)) ? looks good to me.

15:36 rhickey: no, as in isn't fn an example?

15:37 ataggart: can someon paste the link to the newnew stuff pls

15:37 looked but couldnt find it

15:37 Chousuke: http://www.assembla.com/wiki/show/clojure/New_new

15:37 ataggart: thx

15:38 rhickey: that's just an idea scratchpad, btw

15:38 ataggart: yup, I just wanted to be a bit less ignorant while following along with your conversation

15:38 rhickey: Chouser: (fn foo ([bar] ...) ...)

15:39 Chousuke: I think that's exempt because you don't usually call a literal vector :)

15:39 Chouser: oh, yes, multi-body fn's are also a (regrettable ;-) example

15:40 but at least they start with a vector rather than a plain word. Still potentially confusing, but perhaps slightly less so.

15:40 * rhickey went through pages of alternatives in arriving at current multi-body fns

15:40 rhickey: :(

15:40 Chousuke: ([foo bar] 1) is going to make anyone scratch their head, even if it is valid :P

15:41 rhickey: Chouser: what's your alternative?

15:41 ataggart: thankfully the vector is usally a name, so it's not so jarring, e.g., (myvec 1)

15:41 cemerick: the evils of macros, staring us straight in the face... :-D

15:41 * Chouser nearly posts "rhickey: and yet you failed!? amazing..." and then thinks better of it.

15:42 Chousuke: though ([foo bar] bar) is potentially meaningful as a function call...

15:42 cemerick: Chouser: Quick! Pull out the :do-what-i-want option! :-P

15:42 Chouser: rhickey: I hadn't considered alternatives for mutli-body fn. But for 'proxy' and similar, I thought using (defm ...) or (method ...) might be nice

15:42 rhickey: Chouser: so for a multiple method interface you want to say method over and over?

15:43 Chouser: um. yes.

15:43 rhickey: not me

15:43 Chouser: I have to say def and defn over and over anyway

15:43 rhickey: Chouser: but they are not nested in anything

15:44 here we have context

15:44 hiredman: Chouser: so you want boiler plate?

15:44 Fossi: what's a nice way of converting [1 2] to {:foo 1 :bar 2}?

15:45 Chousuke: ,(zipmap [:foo :bar] [1 2])

15:45 clojurebot: {:bar 2, :foo 1}

15:45 hiredman: ,`zipmap

15:45 clojurebot: clojure.core/zipmap

15:45 Chouser: Clojure has this beautiful, possibly undervalued, and unique-among-lisps feature of (foo ...) in almost every context meaning 'use the function or macro named "foo"'

15:45 Fossi: knew it had to be one of those obscure functional things :)

15:46 rhickey: Chouser: I'm not disagreeing

15:46 Fossi: to back up Chouser: it occasionally stumps me with proxy as well

15:46 rhickey: but defm or method wouldn't be that either - you have to lose the parens

15:47 Chouser: perhaps better syntax hightlighting would be sufficient -- if the method names in 'proxy' and newnew stood out differently from fn calls.

15:47 Fossi: multiple ( below each other instinctively shouts 'do' to me

15:47 hiredman: ,((pl λx x) 1)

15:47 clojurebot: 1

15:48 hiredman: or use a map for maping names to functionality

15:48 Chouser: heh. funky

15:48 angerman: would clojure be a vaiable choice for a json-rpc-client?

15:48 Fossi: a map wpuld be great

15:49 hiredman: angerman: sure

15:49 angerman: hiredman: I'm thinking about designing the interface with NetBeans Mantissa (or what it's called now) and use clojure to drive the gui

15:50 Chouser: (fn foo {[] (foo 1 2)} {[a b] (+ a b)})

15:50 rhickey: Chouser: right now I'm building on my 'failure', each method clause will mimic named fn without the fn (methfoo [x] ...) (methbar ([x] ...) ([x y] ...)), but in no case are they doing anything the way calls or macros are

15:50 Chouser: hm

15:50 hiredman: Chouser: woa, that is really funky

15:51 Chouser: hiredman: oh, isn't that what you meant?

15:51 rhickey: Chouser: that will fail to support same-arity type overloading when types in metadata, as non-unique map keys

15:51 gulagong: i'm still having trouble with type-hints, i don't understand why won't compile this:

15:51 (defn tester [#^longs x] (loop [a (long 0)] (recur (aget x 1 1))))

15:51 hiredman: I mean something like (proxy [Thing] [] {someMethodname ([a] do stuff to a)})

15:52 Chouser: rhickey: no, it's not a macro or fn call, any more than 'catch' or 'finally' is, but still it's a word at the front of a list that controls the meaning of that list.

15:54 rhickey: oh, I see. you still want an (optional) level of nesting for same-named methods.

15:54 instead of repeating the method name for each method arity or arg type difference

15:54 rhickey: Chouser: the repetition kills it for me, like having to say public over and over again in Java

15:55 Chouser: ok

15:56 rhickey: I'd consider a paren-free structural alternative

15:56 but IMO fn and new are already saying 'what's in here is special'

15:56 i.e. not a block-o-code

15:57 Chouser: ok

15:57 hiredman: and any macro can change things

15:57 Chouser: yeah, but that's not the point

15:57 rhickey: I take your point but the alternative would have to be really good, I personally like this use of parens for defining callable things, in addition to calling them

15:57 hiredman: so it's not like fn or whatever else is unique

15:58 Chouser: the problem here is reading other people's proxy block, not writing my own.

15:58 rhickey: well, it's common in other lisps, right? (define (foo ...))

15:59 rhickey: Chouser: parens are used for everything there though

15:59 Chouser: so true

16:00 rhickey: I'm looking at future-call in core.clj, I don't get confused, but I could see if it was badly formatted...

16:01 I agree highlighting the method names would be awesome

16:02 Fossi: at a glance, to me it looks like a big do blob

16:02 rhickey: content-handler in xml.clj is bigger, but...

16:03 Fossi: proxy being almost as short as do and being highlighted, then parantheses and methodnames

16:04 Chouser: this is more of an issue at first glance, I guess. It doesn't take a whole lot of studying to figure out that (foo [a b c] ...) is not actually building a vector and passing it to foo

16:04 rhickey: if the method names were highlighted it would be a complete non-issue, and possible I guess?

16:05 Fossi: i'd say yes

16:05 Chouser: it's more a matter of the initial thought "calling foo!? But what *is* foo" -- a feeling that generally overwhelmed me when trying to read any chunk of CL code.

16:05 Fossi: changing the methodnames to :foo (highlighting them) makes it a whole lot more recognizable

16:06 Chouser: I haven't looked at the vimclojure highlighting stuff at all. I assume it would be possible.

16:06 rhickey: Chouser: as I said, I agree and that's why Clojure is the way it (mostly) is

16:06 the exceptions are just in contexts where you will be defining things you call with parens

16:07 Fossi: and sorry for interrupting, but clojure stacktraces *suck*.

16:08 Chouser: Fossi: huh. Well, nobody's ever complained before, so...

16:08 Fossi: well, it's halfways the jvm's fault

16:08 "java.lang.ClassCastException: clojure.lang.PersistentArrayMap"... so what.

16:08 Chouser: Fossi: I'm kidding, BTW. People make such statements quite frequently.

16:08 rhickey: Fossi: are you using Slime?

16:09 duncanm: i think the stacktraces is recent slime has gotten better

16:09 Fossi: unfortunately not when i develop for android :)

16:10 angerman: does clojure-mode honor some a classpath setting?

16:10 rhickey: Slime (used to/still does?) have the problem of not putting the root cause first, like the repl does

16:10 technomancy: rhickey: that's fixed as of about a month ago

16:10 but yeah, that was a big problem

16:11 rhickey: technomancy: great!

16:11 technomancy: also it colorizes clojure internals differently, which helps a lot.

16:11 duncanm: rhickey: i'm still interested in adding some type-coercion bits to the compiler so that a single Fn can be used in place of a one-method iface, and a Map<keyword, Fn> be used as a general interface

16:12 technomancy: angerman: you may be looking for the swank-clojure-extra-classpaths variable

16:12 duncanm: i've been writing some Swing code, and having to write out the Listeners explicitly in proxy form is getting tiresome

16:12 Fossi: also , the parser throws some real bad things at you. i mean "ArrayIndexOutOfBoundEexception"? come on :)

16:14 Chouser: duncanm: have you tried to write a macro to do that?

16:15 angerman: technomancy: thanks. yes that looks pretty much like what I want :)

16:16 duncanm: Chouser: but that's still one more set of parens, right?

16:16 clojurebot: Who??

16:16 arohner: I'm using a cond inside a let to determine the value of a variable. the variable is a boolean, and the default case is "true false". that looks really weird

16:16 i.e. (let [foo (cond (bar) true true false)])

16:16 is there any way to improve that?

16:17 Chouser: duncanm: not sure exactly what you're aiming for, but maybe (new-listener TheType [e] (do-something-with e)) ?

16:17 trb: is there a downloadable doc bundle or a way to generate one so i can have some sort of api reference offline?

16:17 duncanm: Chouser: but avoiding having to specify the listener type is part of the goal here

16:18 Chouser: huh

16:18 duncanm: if i need to type out (new-listener TheType ...), then I could just write out the proxy form

16:18 which is okay (except that clojure-mode in emacs kinda indents it a little funny)

16:19 Chouser: duncanm: what do you expect the compiler to do? use the type expected by the outer form to control the class generated by the inner form?

16:19 duncanm: right

16:19 StartsWithK: i have something like (on widget :actionPerformed (fn [e] ...))

16:19 rhickey: arohner: ?

16:19 cemerick: yeah, I can see that working. (.addActionListener button #(blah)) ends up being a reflective call anyway, so it seems possible to try to build a reasonable proxy in that situation

16:19 duncanm: i was looking at the compiler source

16:19 liron: re multi-body fns, i think it's wrong that this doesn't work: (fn ([] (recur 1))([x] (println x)))

16:19 cemerick: not sure if rhickey would appreciate whatever additional overhead that causes in Reflector, though.

16:20 Chousuke: arohner: (boolean (bar))?

16:20 duncanm: if you find a Fn param, and the matching arg is an interface (of some type T), then modify the arg list by replace the Fn with an interface instance

16:20 ataggart: arohner (let [foo (bar)] ...)

16:20 duncanm: and check arity

16:20 StartsWithK: as no two listeners on swing share the same name, with reflection you can find where :actionPerformed is defined and construct a proxy for that listener

16:20 Chousuke: arohner: usually, you use a keyword for the "true" case of cond though

16:20 duncanm: StartsWithK: that's a cute idea!

16:21 Chousuke: arohner: eg. (cond foo bar :else zonk)

16:21 arohner: Chousuke: ah! thanks

16:21 right, keywords are true

16:21 StartsWithK: duncanm: i don't have a time today, but if you'll be here in a day or two i can rip it out and add some docs

16:22 duncanm: StartsWithK: i'm always in this channel

16:22 rhickey: duncanm: I don't want to add that kind of gimmick to the language semantics. but a macro could look at the name addActionListener and figure out a lot

16:22 StartsWithK: duncanm: ok, i'll find you then

16:23 rhickey: not all listener handle just one event

16:23 *s

16:23 so reflection is needed in the end

16:23 rhickey: reflection can happen in a macro

16:25 duncanm: ha

16:25 StartsWithK: i used, my own multimethod, with :default finding a right listener and adding a new .addMethod so i can cache the results

16:25 never tested is there any improvement in speed

16:25 rhickey: a macro could know what classes are imported in the namespace, look for addActionListener methods...

16:41 angerman: technomancy: I see clojure-http-client is in your github repo, does it use httpclient from apache?

16:43 technomancy: angerman: no, it uses JDK functionality only

16:44 angerman: technomancy: was there a certain reasoning behind that? just asking, so I understand :)

16:44 technomancy: angerman: there's currently no good way to handle dependencies in Clojure

16:45 angerman: technomancy: you mean to handle the apache dep?

16:45 technomancy: I don't think putting jars in a lib/ directory is a good idea; as soon as you get more than a couple of them it becomes a lot of manual labor

16:45 yes.

16:45 angerman: technomancy: does it support ssl?

16:46 technomancy: angerman: heh; I'm not sure. It does if net.java.URL does. =)

16:49 angerman: technomancy: hmm seems to if one sets some properties

16:51 gulagong: i don't get it.... can somebody please tell me how to typehint arrays of more than one dimension?

16:51 i tried:

16:51 ,(defn test [#^"[[Ljava.lang.Long;" x] (loop [a (long 0)] (recur (aget x 1 1))))

16:51 clojurebot: DENIED

16:56 Chouser: 'aget' is only inlined by Clojure for one-dimentional arrays

16:56 Knekk: Chouser: why's that?

16:57 gulagong: ok, so wrap in a fn

16:57 ?

16:57 and it works

16:57 nope, thats silly

16:57 angerman: technomancy: so as far as I understand it, all jars in ~/.clojure/ will be added to the classpath right?

16:57 Chouser: Knekk: I couldn't say. possibly just hasn't been done yet.

16:57 gulagong: try another let level

16:58 ,(expression-info '(let [#^"[[J" x 5] (aget (let [#^"[J" y (aget x 1)] y) 1)))

16:58 clojurebot: {:class long, :primitive? true}

16:58 Knekk: not that it's hard to manage a one-dimensional array to support more-dimensions, if performance is an issue

16:58 technomancy: angerman: totally depends on how you invoke clojure

16:59 gulagong: you have to calculate the positions ;)

16:59 hui, chousers code is very compact

16:59 Knekk: sure, but it's not hard

17:00 Fossi: n8

17:00 Knekk: and probably only marginally slower than letting the compiler do it for you

17:00 angerman: technomancy: with slime

17:01 Chouser: gulagong: you really want a 2-dimentional array of Object like you have? Or of primitive long?

17:01 gulagong: chouser: no, i want primitive long

17:02 Chouser: (defn foo [#^"[[J" x] (loop [a (long 0)] (recur (aget (let [#^"[J" y (aget x 1)] y) 1))))

17:02 so something like that

17:03 technomancy: angerman: anything more than one command to put all the dependencies where they need to be is suboptimal in my book

17:03 Chouser: not sure why it needs another 'let' instead of just using #^"[J" on the inner (aget ...), but it apparently does.

17:03 Chousuke: Chouser: That's just horrible :P

17:03 * Chouser bows

17:04 gulagong: ok, i see that it works

17:04 Chousuke: but then again, dealing with arrays in Clojure never seems to lead to anything good.

17:04 gulagong: but isn't it overhead - that nested let

17:05 angerman: technomancy: No, I ment when I start slime with M-x slime. I thought it would be wrapped into the swak-clojure-extra-classpath args. so that every .jar in ~/.clojure would be added to the classpath

17:05 Knekk: isn't there a better way to represent #^"[J]" ?

17:05 Chousuke: gulagong: nah.

17:05 Chouser: gulagong: no, in this case it should run faster

17:06 gulagong: thanks guys

17:06 Chousuke: gulagong: let doesn't really "do" anything. it just establishes a binding.

17:06 gulagong: you are great :)

17:06 Chouser: Knekk: yes!

17:06 ,(defn foo [#^"[[J" x] (loop [a (long 0)] (recur (aget (let [#^longs y (aget x 1)] y) 1))))

17:06 clojurebot: DENIED

17:06 Chouser: Knekk: thanks for the push. :-)

17:06 * Knekk shoves.

17:07 Knekk: I mean, I like JVM type definition strings as much as the next guy... XNI rules my world.

17:07 err, JNI

17:07 technomancy: angerman: that's the default value of the extra-classpaths variable, but I never use the default value

17:07 Chouser: so #^longs is the same as #^"[J", but I don't think there's anything for #^"[[J"

17:07 Chousuke: it should be "longss" :P

17:08 but of course it isn't.

17:08 Chouser: Chousuke: my first thought too.

17:08 gulagong: isn't #^"[[J" short for #^"[[Ljava.lang.Long" ?

17:09 Chouser: no

17:09 because long is not the same as Long

17:09 angerman: technomancy: it somehow fails to set up the classpath in slime for me :(

17:10 gulagong: ok

17:10 i have to read a lot of code to learn all this ;)

17:11 Knekk: angerman: I ended up using a shell script to set up the path. Slime starts up the lisp using the script

17:11 angerman: technomancy: hmm ahh now

17:11 args

17:11 no

17:12 Chouser: gulagong: you and me both. 95% of the Java I know has been learned using Clojure.

17:13 gulagong: hehe

17:13 that gives hope :)

17:27 chopmo: This may be a stupid question, but what is the easiest way to read binary data from a file?

17:28 Chouser: the same way you would in Java

17:28 chopmo: I'm trying to store a PDF file as an attachment to a couchdb document.

17:28 Chouser: Thanks. I was trying to use slurp.

17:28 drewr: that only works for text

17:28 ..but I guess that's why you asked your q :-)

17:29 chopmo: drewr: Right :-)

17:29 drewr: <-- captain obvious

17:39 angerman: technomancy: I'm having issued to figure out why the swank variable is not passed to slime :(

17:41 technomancy: angerman: what is your swank-clojure-extra-classpaths variable set to?

17:43 angerman: technomancy: it's set to ~/.clojure/

17:43 what happes though is, that slime-lisp-implementations ends up with two items

17:43 for cojure

17:44 technomancy: angerman: that's probably not what you want. you probably want it set to a value of all the jars *in* ~/.clojure.

17:44 which is what it should be out of the box

17:45 angerman: technomancy: that is what I expected to happen too.

17:46 technomancy: angerman: my suggestion is to unpack all your dependencies inside your project directory. See what I wrote at http://technomancy.us/126#projects

17:46 that way your classpath will remain simple

17:48 angerman: technomancy: I think we have a communication issue :)

17:49 dysinger: that's a Led Zepplin song right ?

17:49 technomancy: angerman: I think I know what you're trying to do... I'm saying that's more complicated than my way.

17:49 angerman: Here's what I expected to happen: "When I do M-x slime" I expected slime to start up with the -cp set to swank-clojure-extra-classpaths

17:50 technomancy: angerman: is swank-clojure-extra-classpaths a list or a string?

17:51 angerman: now the issue seems to be that swank-clojure-extra-classpaths is for some reason not evaluated to what it's set (when (file-directory-p ....) ( ... ) )

17:52 later during load time (eval-after-load "slime" is invoked and clojure is added to the slime-lisp-implementations

17:53 at that point it binds the -cp to what it was at load time. :(

17:53 Therefore if I evaluate the (when (file...) (...)) and set swank-clojure-extra-classpaths

17:54 then run the (eval-after-load section from the swank-clojure-autoload.el

17:54 and finally run (setq slime-lisp-implementations (cdr slime-lisp-implementations)) the first clojure item with the wrong -cp is removed

17:54 technomancy: angerman: maybe if you pasted some code for what you're doing it would help.

17:55 angerman: and the second one is retained

17:55 technomancy: angerman: are you using slime with CL and clojure?

17:55 angerman: no. just clojure

17:56 technomancy: I'll try something

17:56 technomancy: angerman: I think you're making this more complicated than it needs to be. Did you read the article at http://technomancy.us/126?

18:00 dysinger: I hope I don't have that much trouble with I add CL to the slime/swank mix

18:00 Setting up for clojure only was easy

18:00 oh - angerman - you said you were just using clojure ?

18:01 Yeah setup a project elisp function like technomancies and you'll be good

18:01 lol s/technomancies/technomancy\'s

18:19 AWizzArd: rhickey: when closures are compiled into pojos, then Drools could maybe work on those?

18:22 ~seen kotarak

18:22 clojurebot: kotarak was last seen quiting IRC, 5769 minutes ago

18:22 hiredman: ,(/ 5769 60 24)

18:22 clojurebot: 641/160

18:23 hiredman: ,(int (/ 5769 60 24))

18:23 clojurebot: 4

18:32 AWizzArd: Btw, how long can a CP become? :)

18:33 Is there some size limit such as 1024 chars or something like that?

18:38 arohner: AWizzArd: what is a CP?

18:38 classpath?

18:39 AWizzArd: yes

18:40 arohner: well, your shell has a size limit

18:40 wouldn't surprise me if you could hit that limit before the classpath limit

18:41 dysinger: hmm - I think you are doing it wrong if you need to worry about CP length

18:42 AWizzArd: I just downloaded Drools. It comes with 19 .jars and 43 more dependencies.

18:43 So, that would mean adding funny 62 more .jar files to the classpath.

18:43 or maybe some wildcard syntax can help

18:43 arohner: JDK 6 supports * in cp

18:44 AWizzArd: Especially because I don't want to explicitly write out those damn names, or write a script which does so.

19:30 duncanm: so if i have a list of coords, Xs and Ys

19:30 (map #(Point. %1 %2) xs ys) will only get me the diagonal

19:31 is there a concise way to write it so i get all the points (x, y)

19:31 right now, i have (flatten (map (fn [x] (map (fn [y] (Point. x y)) ys)) xs)), which is ugly

19:32 arohner: (for [x (range ...) y (range ...)] #(Point. %1 %2))

19:32 duncanm: ahh

19:46 rhickey: very nice story, thanks: http://infolace.blogspot.com/2009/07/coming-to-clojure.html

21:19 ataggart: what's the trick to getting new code from an assembla ticket into github?

21:20 e.g., http://www.assembla.com/spaces/clojure-contrib/tickets/7-Logging-functions

21:22 arohner: http://msdn.microsoft.com/en-us/devlabs/ee334183.aspx

21:29 Chouser: ataggart: you just need a contrib committer to adopt your project

21:30 ataggart: and how does one foment that?

21:37 Chouser: good question. no committer asked for it on the google group?

21:38 ataggart: asked for it to be written? no, I needed it for myself and thought others might find it useful

21:40 at least 3 others did think it was useful, judging by the response on the groups thread

21:41 Chouser: I just don't know how this should all work.

21:42 ataggart: likewise

21:42 Chouser: I think your best bet is for someone with commit privs to suddenly need logging features and to discover your patch. :-)

21:43 I dunno if any committer should commit your code without evaluating it. Might be an okay policy, or maybe not.

21:44 ataggart: that certainly is one way to keep stuff out of contrib

21:44 hmm

21:44 is there a list of contrib commmitters beyond http://clojure.org/contributing

21:44 because I see Tom Faulhauber had last commit in github

21:45 Chouser: I don't think so, unfortunately. Used to be on google code, but I don't see anywhere on github to get a list of who has commit privs.

21:45 ataggart: and he was one who responded positively t the thread, but he's not on that list

21:45 hmph

21:46 even if he had github privs would that mean he also has some more authority in assembla?

21:46 this is all very confusing

21:47 Chouser: ok, sure enough. he has a contrib commit with no "sign-off", so he must have privs.

21:47 you should ping him, either directly or via the google group, and se if he'll put it in for you.

21:49 I think anyone with a CA in can ask for membership on clojure and contrib assembla groups

21:50 that's all the privs anyone there gets, other than rich himself. I think.

21:50 ataggart: well I am in assemabla

21:51 and reading the istructions, it says all code needs to be submitted via a ticket

21:51 kinda of dead-ends there

21:51 who knows, maybe I have commit privs in github

21:52 I'm really not incluned to bug eople about doing stuff for me though

21:56 Chouser: ataggart: well, that's how it is for now. If he wants it, as he seems to, he'd probably be happy to check it in for you.

21:58 I guess another option would be to write rich and ask for contrib commit privs. I guess that's how that works, not sure.

23:54 tomoj: what's fn*?

23:56 hiredman: tomoj: the real fn primitive

23:57 fn is listed as "primitive" but it really is a macro that emits a call to fn*

23:57 tomoj: ah, I see

23:57 trying to figure out the lazy-seq source

23:57 thanks

Logging service provided by n01se.net