#clojure log - Nov 12 2008

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

1:58 sohail: uhhh... what the f: http://commons.apache.org/jelly/

2:00 tWip: that's the logical conclusion to xml-mania... making it turing complete

2:01 Nafai: sohail: Yes, jelly is very painful

2:01 I've actually written a bit in it in the past

2:03 sohail: incroyable

2:04 Nafai: I've also done a bit in a workflow language called XPDL

2:05 It took 7 printed pages of XML to do a simple method implementation

2:05 sohail: wow

2:05 I bet that stuff pays well though

7:33 Lau_of_DK: Good afternoon gentlemen

8:07 gnuvince: morning

9:58 Cark: is there any equivalent to CL's labels in clojure ?

9:58 so that i can recursively call a let function

9:59 rhickey: Cark: self-recursion is possible with (fn my-name [] ... (my-name ...))

9:59 mutual local recursion a la labels/letrec is not currently supported

10:00 Cark: ah too bad

10:00 thanks though

10:00 rhickey: on the todo list

10:02 gnuvince: rhickey: would that mean that the order in which functions are defined would become unimportant?

10:03 rhickey: gnuvince: this is just about local/let-bound fns, but yes

10:04 gnuvince: ah ok.

10:37 rhickey: Chouser: http://clojure-log.n01se.net/ is down?

10:40 abrooks: rhickey: It appears to be. I'll check it out.

10:42 Chouser: abrooks: thanks

10:46 abrooks: Chouser: Shoot. I need to go interview a candidate now. I'll continue after lunch.

10:50 Chouser: rhickey: back up

10:50 rhickey: Chouser: thanks!

10:51 abrooks: Chouser: Was it Ruby?

10:52 Chouser: abrooks: maybe. I turned off mod_rails for clojurescript.n01se.net, and now it seems to work. Could be coincidence.

10:52 I'll have to try fast-cgi or something instead.

11:05 Seems like binding/set! might work for early termination of an enumerator

11:05 rhickey: Chouser: sounds scary

11:06 * rhickey doesn't like the term enumerator when no numbers are involved

11:12 Chouser: also Java uses enum to mean something completely different

11:18 rhickey: cgrand: thanks for the LFE post!

11:19 * rhickey woke up thinking about LFE for Clojure

11:22 cgrand: rhickey: you're welcome

11:27 chouser: I also thougt about using binding/set! and dismissed it immediatly

11:27 too scary

11:30 rhickey: cgrand: you spent a lot of effort to support map, I wonder if it is as important as 'chunking', i.e. reducing at other than 1:1 ratio

11:31 for me the inversion of control is much more about resource management than having someone else run the iteration

11:32 cgrand: yeah I also wondered about the practicality of suporting map (and came to no conclusion and since I was on my way...)

11:32 rhickey: map is definitely a challenge

11:35 as far as supporting ordinary functions - the underlying protocol can be richer as adapters are easy

11:37 cgrand: What du you mean by "reducing at other than 1:1 ratio" : reducing N items at a time (N:1) or , for example, transforming an "enum" of char[]into an enum of lines (N:M)?

11:38 rhickey: the 'slides' passed a stream to the reducing fn, not individual items

11:38 opens the door to consuming more than one item per step

11:39 Chouser: My (very weak!) attempt at doing this passed a seq to the step function, which then could return however much of the seq it chose to not consume.

11:39 rhickey: chars to lines would be an example

11:41 Chouser: yes, that's closer to the slides

11:42 there's a risk of leaking the seq, but that could be in the "don't do that" category AFAIC

11:42 the enum would close the resource and subsequent use of the seq would fail

11:43 Chouser: oh, if you passed the chunk-seq to outside the enum somehow?

11:43 rhickey: sure, it could be your return even

11:44 or passed to something in your worker code

11:44 the latter would still be in scope of enumeration

11:44 Chouser: ok, yep, I see that.

11:44 * rhickey still trying to think of a better word for enumeration

11:46 leafw: rhickey, try: wn enumeration -synsn

11:46 rhickey: These internal iterations have the potential to be very fast, much faster than seqs (i.e. allocation-free), that's why I had the io/stream primitive notion under them, in my model that's what I passed. Would then go back and rebuild all the seqs on io/streams

11:46 leafw: (wordnet)

11:46 * cgrand thought he had read thoses slides... but no

11:48 rhickey: reducer?

11:49 supporting map would blow my model :(

11:49 leafw: distiller?

11:49 rhickey: very tricky

11:49 Chouser: only multi-seq map, right?

11:49 rhickey: right

11:50 Chouser: similar problem in ruby iterators, which I was able to solve with continuations. :-/

11:52 cgrand: rhickey: we had different goals: mine was to come up with a general collection "framework", yours is better IO. Fun vs practical :-)

11:52 rhickey: cgrand: I want both

11:56 this would be a very important addition to Clojure

11:58 cgrand: Working on MFE made me thought wonder about replacing seqs by reducers as the basic building block...

11:58 LFE

11:59 rhickey: early termination is not laziness

11:59 I do think the seqs can be built on ios

11:59 should be

12:00 some ios are only available in resource managed contexts - reducers

12:00 those are the building blocks in my mind

12:01 I had played with seqs on iterators early on, but iterators are MT-broken

12:01 ios can be made atomic

12:01 more primitive than iterators, just need EIO protocol

12:07 cgrand: I'm lost between ios and reducers. Are reducers specialized ios or two different things (ios being generators)?

12:08 rhickey: ios are simple streams/generators, reducers can be thought of as things that make ios available in a resource-managed context

12:09 cgrand: so ios aren't directly exposed in user code?

12:43 cddr: general Java question coming up...

12:43 To make a library threadsafe, must one wrap all uses of iterators inside a synchronized block?

12:45 roblally: I think it is worse than that. Some Collections can't be modified whilst you iterate over them. So you have to protect the underlying collection as well as the iterator. Otherwise you get ( IIRC ) ConcurrentModificationExceptions thrown.

12:45 cddr: yeah those are what I'm trying to fix

12:46 although the underlying collections do seem to be Vectors rather than lists so maybe not so bad

12:47 roblally: Vector is one of the old collection classes that are synchronised by default. But that doesn't protect the iterators.

12:47 If it isn't too expensive, I copy the collection before iteration 'just to be sure'

12:48 Of course that may not give you the right answer in every case.

12:50 cddr: I tried the copy (clone) approach before but it didn't seem to work (i.e. still got the same exception)

12:52 roblally: I guess that you need to lock during the copy process ( since copying to another class that takes a Collection parameter in it's constructor will probably fall back to an internal iterator.

12:53 cddr: apologies for non #clojure related spam but I have asked this stuff on c.l.j and got nothing

12:53 I'll shut-up now, thanks roblally

12:54 roblally: But locking during copy should be less painful than locking during the whole iteration process. Basically you need to lock on every operation that touches the collection. Without the copy you need to lock on the collection plus the iterator ( which is much nastier in Java )

12:54 Happy to try to help.

12:55 (As an aside, having a conversation that shows the pain of not using Clojure on a Clojure list will probably just make everyone here feel all warm inside)

12:56 cddr: lol

12:57 Cark: well sometimes the java non-sense leaks to clojure too ...

12:57 those tons of different and incompatible listener interfaces .... i hate these

12:58 don't know how i could abstract that to a simple function call

12:59 so in the end, i make a proxy for ListSelectionListener, then another one for ChangeListener etc ... they all have the same single event parameter.. grrr

13:11 Chousuke: Cark: maybe you could use a multimethod?

13:12 Cark: that way, there would still be multiple functions for creating proxies but at least they'd be an implementation detail

13:19 rhickey: cgrand: sorry, got called away - ios could be exposed where safe

13:20 for instance an io could be passed to the fn inside a reducer

13:21 Cark: chousuke : yep that makes sense

13:23 chousuke : but i still need to declare the type of listener for each callback invocation

13:24 still shorter than a full proxy form tho

14:07 Chouser: Cark: could you write a macro to abstract away whatever's in common between those proxies?

14:14 Cark: chouser : yes i'm in the process of doing so

14:15 actually i'll do that tomorrow, time to kick back and watch a movie ... see you all tomorrow

14:51 drewr: I've got a lazy stream that is exhausting my heap space.

14:51 I have a (count) in front of it. Would that be keeping objects around so that they're not GC'ed?

14:51 rhickey: drewr: are you holding on to the head?

14:52 yeah, count will run it out

14:52 drewr: OK, let me get rid of that.

14:52 I've got a ref that I update with the count so I don't even need that.

15:03 What's the right way to kick something off asynchronously that itself uses agents to do other asynchronous things?

15:04 Clojure doesn't seem to allow nested send-offs.

15:05 rhickey: drewr: it does support nested actions

15:06 they don't fire until the sender's action completes

15:07 so lexically nested, but temporally chained

15:11 drewr: Is there any way that I could mess that up lexically so that they don't get chained temporally?

15:12 rhickey: nope, but if you send then block, those sends aren't going anywhere

15:12 drewr: OK.

15:17 rhickey: there's always (.run (Thread. f))

15:18 I guess I could add a release-pending-actions, waiting until action complete is usually a feature

15:22 drewr: Removing (count ...) kept my JVM from crashing, but now something is just blocking.

15:22 Ugh.

15:35 Chouser: oh, cool. You can look at queued agents: (.getQueue Agent/pooledExecutor)

15:35 well, queued actions.

15:36 abrooks: Ah, nifty. I've wanted to do that.

15:36 Chouser: but that's only for send, not send-off

15:36 I wonder why pooledExecutor is public but soloExecutor is not.

15:38 rhickey: Chouser: probably only public so I could debug something

15:38 Chouser: and the action has its agent, but that's also private.

15:38 ah. seems like runtime viewing of that stuff could be generally useful.

15:39 wouldn't want people calling .shutdown I suppose.

15:39 rhickey: I don't want people marrying implementation details

15:39 Chouser: I only want to flirt with them.

15:39 rhickey: I've swapped out the pools and queues a few times

15:41 Chouser: are you at all sympathetic to a higher-level view-only feature? a way to get a sense of how backed up the agent queue is, for example.

15:41 * abrooks clears what he was typing and agrees with Chouser

15:42 rhickey: Chouser: yes, definitely, both agents and refs need more introspective capabilities

15:46 kotarak: In how far, is it voodoo to access a private function from a macro via ns-resolve? Say foo.bar/macro uses foo.bar/private. Do I have to make private public? %)

15:47 Chouser: kotarak: excellent question. I've wondered that myself.

15:48 kotarak: One can use ns-resolve. But that feels hacky like hell...

15:48 Chouser: So far I've always just made such functions public, documented them, and assumed nobody would ever use the directly.

15:48 kotarak: Also my way up to now.

15:49 * rhickey adds macros emitting calls to private fns to todo list

15:49 kotarak: (ns foo) (defn- foo [] :Private) (in-ns 'use) ((ns-resolve 'foo 'foo)) => :Private, that works, but well, ... not really...

15:59 rhickey: AOT shaping up, in 1094:

15:59 AOT compiler support

15:59 breaking change to load - no longer takes extension

15:59 load will load from classfile if newer than source

15:59 to compile, source dir and compile dir must be in classpath

15:59 (compile 'my.cool.ns)

15:59 will compile my/cool/ns.clj and anything it loads directly or indirectly

16:00 require/use system should use classfiles if present and newer

16:00 just move your files up a dir

16:00 Chouser: sweet

16:01 kotarak: *thumbs up*, that saves my lots of directories containing a single file...

16:01 rhickey: just need to sync clojure.contrib and shake out anything we find

16:01 duck1123_: rhickey: when it's all done, will you be posting anything describing what needs to be changed if you haven't updated since before the AOT work?

16:01 rhickey: one thing I was able to preserve was a ns defined by multiple files - clojure.core is

16:02 duck1123_: that's a very nice fix, much more Java-like

16:02 rhickey: duck1123_: if you're using contrib you should let those guys shake that out first

16:02 other than moving your files up a dir, and dropping the extension on any manual load calls, AOT shouldn't change anything

16:02 * rhickey ducks

16:03 rhickey: the binding form changes are bigger from a source-level perspective

16:03 duck1123_: do you /have/ to move them up a level?

16:03 rhickey: yup, namespaces now correspond to classes, not packages

16:03 * drewr is glad.

16:04 * kotarak too

16:04 Chouser: hm I just created a namespace dir last night and was happy to be able to put a README and support files in there.

16:04 dumb, since I knew this was coming.

16:05 kotarak: Chouser: you still can, no? Then you have foo/bar.clj and foo/bar/README.txt foo/bar/support.clj...

16:05 rhickey: Chouser: those related resources are the only thing that doesn't work well here, was an argument against this arrangement initially

16:05 Chouser: yeah, I suppose. don't know if anyone will think to look in the subdir for a readme.

16:06 duck1123: this will mean less typing when C-x C-f ing from file to file

16:06 kotarak: Hmm.... How many projects, does a jar contain? Maybe /README.txt at the root is sufficient?

16:06 cemerick: rhickey: this will be huge. Many thanks.

16:07 rhickey: cemerick: genclass subsumption still to go

16:08 one thing to note is that this compile is a 'same-world' compiler, a side effect of compiling is loading, this to make macros and the fns they use avaiable

16:10 *compile-files* will be true when compiling and can be used like this: (def a (when-not *compile-files* (my-big-init)))

16:10 cemerick: oh, that'll be very nice, as well

16:10 rhickey: for stuff you only want at runtime, or just put that work in function bodies

16:11 so no eval-when yet

16:13 back later...

17:14 everyone ported to 1094?

17:14 :)

17:14 duck1123: was it all for nothing? :)

17:14 tomhickey: =)

17:15 kotarak: Hmm.. The hg mirror is still at 1091.

17:17 duck1123: github is up to date

17:22 AWizzArd: is 1094 the new 'stable' version?

17:22 rhickey: AWizzArd: nothing has been unstable, but recent version have had breaking changes

17:23 AWizzArd: seems that AOT made a lot of progress so far *thumbs up*

17:24 rhickey: looking for testers/feedback/bug reports

17:25 AWizzArd: sure, I'm in

17:25 kotarak: will try also

17:27 * duck1123 has to go grocery shopping, then class

17:35 Chouser: user=> (find-ns 'clojure.xml)

17:35 #<Namespace #<Namespace: clojure.xml>>

17:36 clojure.contrib mmap and gen-interface are on 1094, seem to be working fine.

17:36 rhickey: Chouser: I saw that the other day, still chewing on default print currently #<(class x) (str x)>

17:37 Chouser: ah, ok.

17:37 rhickey: Namespace's toString does too much

17:38 fixed

17:38 user=> (find-ns 'clojure.core)

17:38 #<Namespace clojure.core>

17:39 Chouser: thanks for the report on mmap and gen-interface

17:42 Chouser: .class files aren't generated unless asked for?

17:42 rhickey: right, you have to (compile 'clojure.contrib.xxx)

17:43 Chouser: main() methods will come later?

17:43 rhickey: main goes with the genclass fold-in

17:44 an easy option is to map main to main

17:45 Chouser: (compile ...) creates a classes dir in the CWD, not necessarily the one in your classpath.

17:47 rhickey: classes is just a default, you can set *compile-path* to whatever you want

17:47 is bound in repl, you can set! or use binding

17:47 Chouser: wild. So I just compiled a namespace, then imported it and used Java reflection on it.

17:48 rhickey: cool!

17:48 I want to get genclass folded in so it's just as easy

17:48 also AOT proxy

17:49 basically want to be able to deliver Clojure apps w/o any need for custom classloader

17:49 SimonAdameit: Hi, I have some problems with setting up clojure/slime. See http://paste.lisp.org/display/70249 for output. Any Idea whats going wrong?

17:50 Chouser: I have no idea what this is. a whole lot of const__xx that appear to vars and symbols. oh, return values from the top-level forms in my namespace .clj?

17:50 rhickey: yeah, that's a truly worth goal.

17:53 SimonAdameit: sorry, I don't use emacs, but many others here do. I'm sure someone will be along to help you.

17:54 rhickey: hm, but if I (compile foo) without the right classes dir in my classpath, it looks like Clojure is unhappy about being able to find the class.

17:56 SimonAdameit: rhickey: My congrats to you, Clojure looks alot like the lisp in my daydreams :) .. I mean realy great.

17:57 rhickey: SimonAdameit: great - thanks!

17:57 Chouser: right, you can put your classes wherever you want, and that dir has to be a)in your classpath, and b) set as *compiler-path*

17:59 Chouser: the const__xxs are any vars or literals from compilation, they turn into static members

17:59 Chouser: zip-filter and zip-filter.xml required no changes other than file renames.

17:59 rhickey: Chouser: great!

18:00 are you compiling everything as you move it?

18:00 Chouser: scgilardi is hard at it, too.

18:00 no

18:00 should I?

18:00 rhickey: Chouser: if you can, that way we know that everything works the same when compiled

18:01 Chouser: and then test the compiled version.

18:01 yeah, ok

18:01 gotta go. bbl.

18:03 rhickey: me too

18:04 SimonAdameit: Yay! - going back to revision 1088 solved the problem. as recommended in the group

18:46 rhickey: hmm... why didn't my VM complain about a class called core-print?

20:05 gnuvince_: Is there a prettier way to do something like: (loop [coll coll, n n] ...)?

20:05 (instead of inventing a bunch of names

21:41 _Jordan_: SLIME won't connect, just polls indefinitely even after the Clojure REPL prompt shows up... anyone seen this before?

21:48 yangsx: _Jordan_: you're using an interim subversion checkout, it seems

21:49 _Jordan_: yangsx: bummer... do I just wait then?

22:28 drewr: _Jordan_: No, just back up a few revs.

22:31 _Jordan_: drewr: thanks!

22:59 yangsx: _Jordan_: Now that rhickey completed AOT, you can fetch the latest. I actually can run it now, but a few changes to .emacs is necessary.

23:00 _Jordan_: yangsx: Thanks--I ended up just backing up a few revs as suggested, which I think will be sufficient for me to start having fun learning :)

23:01 yangsx: for a quick trial, in your *inferior-lisp* buffer: (add-classpath "path-to-clojure-classes") (add-classpath "path-to-clojure/src/clj") and M-x slime-connect, it works for me

23:06 _Jordan_: This is neat! Now I just need to get it into my fingers to not spell it with an 's', and I'm golden :D

23:43 larrytheliquid: is there a let-like function that accepts a list of symbol/value pairs? ie: bindings-list => '(one 1); (let-like-function bindings-list one) => 1

Logging service provided by n01se.net