#clojure log - Jun 10 2008

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

9:26 cgrand: Did you know that the function passed to pmap must not return nil?

9:36 Chouser: I did not.

9:37 cgrand: It seems to be an unintended consequence of using a LinkedBlockingQueue.

9:38 rhickey: any reason not to try parallel.clj?

9:39 cgrand: apart from being too lazy to install one jar? No :-)

9:40 rhickey: pmap may go away

9:43 cgrand: rhickey: oh, ok, I'll look at parallel.clj then.

9:44 blackdog: coming from the non lisp world i find the docs sometimes difficult to understand, e.g. -> macro, i just worked out is a kind of "member access" for a struct

9:44 it could be a good idea to note things like that in the docs

9:45 rhickey: except it's not, it's a more general "through" or "thread"

9:45 blackdog: yes, that's why i said "kind of"

9:45 but for the beginner

9:45 that's maybe the predominant use case?

9:46 i'm not sure, that's what I'm about to use it for :)

9:46 rhickey: I'm not saying there couldn't be more explanation, right now the docs are a reference, so a bit terse

9:46 but misaligned analogies can confuse more than enlighten

9:47 Chouser: You never *need* the -> macro, it just allows you to arrange your code a bit differently.

9:47 rhickey: examples would help, something I hoped would happen on the Wiki

9:47 blackdog: well, i'd like to do a simple member access type thing to a map and I don't want to use (get) all the time

9:47 is there another way?

9:48 rhickey: you hardly ever have to use get given maps are functions of their keys

9:48 blackdog: oh, this is something I've not understood then

9:48 rhickey: and keywords are functions of maps, if you're using them as keys

9:49 blackdog: ah ok, there are the missing links, thanks!

9:49 rhickey: user=> ({:a 1 :b 2} :b)

9:49 2

9:49 user=> (:b {:a 1 :b 2})

9:49 2

9:50 blackdog: or maybe I'm the missing link ;) cheers

14:52 rapido: erlang has proven that copy semantics make concurrency easy

14:53 immutable data structures have efficient copy semantics

14:53 but how do immutable data structures scale in a distributed setup?

14:54 rhickey: erlang unifies the local and distributed model

14:54 clojure doesn't

14:55 some reasons here: http://research.sun.com/techrep/1994/smli_tr-94-29.pdf

14:55 rapido: but how do you efficiently share huge data structures along different distributed nodes?

14:55 rhickey: erlang doesn't efficiently share huge data structures over the wire

14:55 rapido: erlang doesn't solve that

14:55 i believe there is a way to do that

14:56 using distributed hash tables

14:56 rhickey: I think the distributed case should be handled separately. I don't believe in transparent distribution

14:56 sure, tuplespaces linda etc. Mnesia too

14:57 rapido: i believe DHT is build to handle churn better than tuplespaces or mnesie

14:57 mnesie <- mnesia

14:58 rhickey: DHT is sort of an implementation detail, no?

14:58 model is still associative

14:58 rapido: no, i think it is an ideal substrate. you can leverage any datastructure on top of a DHT

15:00 rhickey: but is hashing important in some non-implementation-detail way?

15:01 rapido: there are dht implementations that leverage near-keyness as opposed to (scattered) hashes.

15:02 you can build a persistent vector on top of a dht

15:03 just store the sha-1 hash at every tree node

15:03 the value is the serialization of the tree node. the key is its sha-1 hash

15:04 replace pointers to sub-trees with their sha-1 hashes

15:05 does that make sense?

15:06 rhickey: it could, but I am unsure about whether you really want distributed versions of the same data structures you use in-proc

15:06 other options are to put Clojure under Terracotta

15:07 rapido: but such data structures could scale to tera bytes, going beyond jvm's

15:07 rhickey: so many other issues dominate - scalability, fault tolerance etc

15:08 not uninteresting, but a large problem space

15:09 rapido: i've had my fare share of faulty disks, nodes etc.

15:10 that's why i think immutability is so important

15:10 rhickey: no argument here :)

15:11 rapido: i generally don't like the 'encapsulation' argument

15:11 i don't want an object to hide that it is connecting to sybase, opening files, etc

15:12 because when such object throws an exception

15:12 it better close the sybase connection, file handle, etc

15:13 rhickey: I make that point in most of my talks, encapsulation just means "I'm in charge of this mess", not that there isn't a mess

15:13 not a substitute for immutability by any stretch

15:14 rapido: everyone is re-implementing the same hard stuff <- resource management.

15:15 try , catch .... oh, i forgot finally :)

15:17 you'd better don't catch c++ exception. it is almost certain that you've caught a memory leak at the same time

15:17 rhickey: programming is not a solved problem yet

15:19 rapido: not everybody agrees, but i think it was/is a good choice that clojure is build on top of the jvm

15:20 rhickey: I have no regrets

15:20 as a new language, Clojure has a lot of utility

15:20 excellent performance

15:20 great portability

15:21 rapido: its the magic of hotspot - not everybody appreciates the 1000 man-years that went into it

15:21 rhickey: exactly

15:22 rapido: hotspot goes back to self technology

15:22 the language self i mean

15:24 may be shez scheme is the exception

15:24 rhickey: I like the separation of concerns, I can focus on my language and not on code generation or socket libraries

15:25 rapido: schez scheme achieves amazing compiler feats

15:25 but it didn't took 1000 man-years to achieve those feats

15:26 rhickey: yes, but hotspot's dynamism sets it apart

15:26 rapido: agreed

15:27 i've implemented my own language (enchilada)

15:27 first try: java

15:27 second try: java

15:27 third: factor

15:27 fourth: factor :)

15:27 five to seven: haskell

15:28 and finally: scala

15:28 i got addicted to a typed language after the haskell experience :)

15:29 but i finally went for the jvm, exactly because i don't like to bother with native socket implementations

15:29 myself

15:31 i didn't consider clojure yet - sorry about that ;)

15:33 rhickey: that's ok, I'll likely not get around to reimplementing Clojure in Clojure either :)

15:34 rapido: aaahhhh, the art of bootstrapping

15:34 factor has done it

15:35 factor started on the jvm

15:36 eventually, the factor compiler was written in factor

15:36 generating native code

15:36 for various platforms

15:37 (still using the jvm implementation of factor i believe)

15:37 i think that's pretty cool

15:37 (but i could be wrong there)

15:38 hmm, according to the definition, that's not strict bootstrapping

15:39 ... bootstrapping is confusing ...

15:40 rhickey: what programming languages do you like? (apart from clojure of course)

15:41 rhickey: Common Lisp

15:42 I like the ideas in Haskell

15:42 but am concerned about type system expressivity

15:43 rapido: yeah, types can hinder

15:44 i like apl/j/k because it so different and still effective

15:44 it so <- it is so

15:44 Chouser: that's really key. It wasn't 'til I was trying to do an XML filter DSL in Scala that I realized how much static types can *really* get in your way.

15:44 rhickey: I disagree with types in many ways. They represent an extreme narrowing of our ability to categorize

15:44 Lau_of_DK: Syntax question: This simple decoder should be able to take anything from 1 integer, to an infinate number of integers, it works with either 1 or 2... whats wrong?

15:45 (defn decode

15:45 ([x] (char (+ 97 (- (/ x 2) 1))))

15:45 ([x &more] (apply str (list (decode x) (decode &more)))))

15:45 rhickey: I think the type theorists wis

15:45 I think the type theorists wish programming ws

15:45 was math, but it isn't

15:46 rapido: did you know that there are infinitely many types of numbers?

15:46 they are surreal numbers

15:46 (just teasing there)

15:46 la_mer: Speaking of distributed computing (look at me, creating a tangent of a conversation from a half hour ago!), has anyone come across any reasonable overview of all of the different distributed computing frameworks available for Java apps (hadoop, terracotta, gigaspaces, coherence, etc., etc., etc.)?

15:47 Lau_of_DK: rhickey, you wrote the language, can you see why my overload isnt working?

15:47 rhickey: & more

15:48 not &more

15:48 rapido: la_mer: check out: freepastry, overlayweaver, bamboo, etc

15:50 Lau_of_DK: rhickey, okay, if I change the param to "& more" and in the function I change it to "more" then it compiles, but it throws an error when I run it with more than 1 arg - Is this because I need to add some splicing operator?

15:51 rhickey: you don't need the list call

15:51 Chouser: (defn decode ([x & more] (str (char (+ 97 (- (/ x 2) 1))) (when more (apply decode more)))))

15:52 la_mer: rapido: Thanks for the names of more frameworks to look at. Do you know of any writeups comparing/contrasting them, their approaches, where they fit in the stack?

15:53 Lau_of_DK: Chouser, rhickey, thanks guys :) You really fill the gap between here and descent tutorials :)

15:53 rapido: la_mer: this page has many links: http://www.sics.se/~sameh/p2pComputing.php

15:54 i found it very valuable

15:54 la_mer: rapido: Thanks for that. p2p is a whole area unto itself, I suppose.

15:55 rapido: la_mer: my gut feeling is that p2p *is* distributed computing

15:55 la_mer: I saw on google groups that someone has put together a clojure wrapper for hadoop, so I suppose we'll start there

15:56 rhickey: people have played with NetKernel and Clojure too

15:56 I'm interested in trying Shoal (from Glassfish)

15:57 la_mer: rapido: p2p means no central control (to me, anyway) -- we'll be looking to deploy to a grid architecture.

15:58 rapido: hey! i didn't know about shoal! thanks for the link rhickey!

15:58 rhickey: the Shoal API is so simple, it's irresistible

15:58 rapido: la_mer: the world is made that way: there is no central control ;)

15:59 ahh, jtca

15:59 jtxa

16:00 rhickey: how did you find shoal?

16:01 rhickey: interested in trying, haven't tried

16:01 good enough for Glassfish clustering, apparently

16:01 rapido: la_mer: what's your stake on clojure and grid architectures? (who's *we* ?)

16:03 la_mer: we is http://snowtide.com

16:03 we do a lot that isn't publicly available / marketed, FWIW

16:04 rapido: to bad i don't live near western massachusetts

16:04 too bad

16:05 la_mer: We are developing a new service / product this summer, and I'm considering using clojure in part. The real meat of the functionality will be the processing of documents in varied and interesting ways, all of which are computationally (and storage) intensive. Not targeting a grid architecture would be foolish at this point.

16:05 rapido: Where are you now?

16:05 rapido: la_mer: the netherlands .. amersfoort (google that!)

16:07 la_mer: rapido: Looks very pleasant :-)

16:08 rapido: la_mer: you would consider using clojure in a production setting?

16:08 la_mer: rapido: Very close to it, yes.

16:08 rapido: that means you have power

16:09 la_mer: rapido: lol :-) In what way?

16:09 rapido: or you are good in convincing 'management' to try something new to beat the 'competition'

16:10 la_mer: I *am* management. :-D

16:10 rapido: la_mer: i rest my case...

16:10 la_mer: We're also a small company trying to solve tough problems, and we always need to be using the best raw materials we can lay our hands on.

16:11 rapido: la_mer: well, for me it is certain that you don't represent classical management

16:12 la_mer: No, I suppose not. Of course, it remains to be seen whether that's a good thing or not. ;-)

16:15 rapido: la_mer: managing people (=developers). that's difficult

16:15 herding cats

16:16 of course, managing non-developers is the real task

16:17 and managing your own company is the most interesting 'challange'

16:20 la_mer: It's definitely a challenge. And, FWIW, I actually think that sales and "business development" is *way* harder than software development. People are hard, machines are easy.

16:25 rapido: got to go..... i'm mutating to a different location ....

16:25 la_mer: rapido: good talking with you

17:11 Lau_of_DK: For string comparison, does Clj have something like if ("x" isin "extreme") ...true ?

17:12 rhickey: Clojure strings are Java Strings

17:13 Lau_of_DK: So I gotta go with (. String compare) or something like that?

17:14 rhickey: one of us will have to look it up, and it's not going to be me :)

17:15 dudleyf: (. "foobar" contains "foo")

17:15 Lau_of_DK: haha

17:15 rhickey, no I wasnt asking for the specific function, I would just prefer not to call anything (. java) if there already was something in lisp that did the same :)

17:15 Thanks dudleyf

17:16 Chouser: or (.contains "foobar" "foo") -- how much simpler could a lisp version be?

17:16 rhickey: I'll probably not be adding (contains x y) in order to avoid (.contains x y)

17:17 Lau_of_DK: Come one guys :)

17:17 rhickey: All of the Java stuff is nicely documented, were I to wrap it re-documenting it would fall on me - no point

17:17 Lau_of_DK: I dont know - Im just prone to picking stuff that arent Java, as far down the road as I can

17:17 rhickey: that's not going to let you leverage Clojure fully

17:17 Lau_of_DK: But I get the point, so forget the stupid question :)

17:18 Chouser: not a stupid question! I definitely went through an "ew, I have to touch Java?" phase.

17:19 But it's pointless. You're on the JVM, you have the Java libs -- use them!

17:19 Lau_of_DK: Youre right

17:19 And for IO and GUI Im all over them :)

17:19 Chouser: and string. ;-)

17:19 rhickey: everyone goes through that phase, I think

17:20 Lau_of_DK: Thanks guys, thats good to know

17:20 Chouser: regex is a bit of an exception, I think, because the Java API is so... wordy.

17:20 Lau_of_DK: Last night I redid some of the Eulers that I had already handled in SBCL or Python, I was pretty amazed at the speed of Clojure

17:20 rhickey: ew, Java libs --> ooh, Java libs

17:20 Lau_of_DK: hehe

17:20 The libs are nice, and huge

17:20 which is something I missed very badly in SBCL

17:21 dudleyf: I think a lot of Java API's could use some Clojure lipstick

17:21 rhickey: Chouser: regex still too wordy?

17:22 dudleyf: fair enough, but driven by need and added value

17:22 dudleyf: rhickey: Oh, absolutely.

17:22 rhickey: Clojure's wrapping of fork/Join is pretty sweet vs. the Java

17:23 IMO

17:24 .., doto etc a ripe area as well

17:25 dudleyf: I've always hated the Java IO API's

17:25 rhickey: not sure how to fix them...

17:26 dudleyf: No, me neither. It's kind of baked into their design.

17:26 Chouser: duckstream helps the IO libs a bit

17:26 Lau_of_DK: Whats to hate about them ?

17:26 rhickey: layers

17:26 Lau_of_DK: They seem pretty straight forward

17:26 rhickey: layers

17:26 Chouser: rhickey: no, re-seq and re-find are good. I wrote a re-split that I like too.

17:26 rhickey: more layers

17:26 dudleyf: I really don't like the decorator approach

17:26 Lau_of_DK: If the performance is descent, why worry about layers?

17:27 rhickey: wordy

17:27 Chouser: re-split does what?

17:27 Lau_of_DK: btw rhickey, wasnt it tonight you were giving that talk to the java group ?

17:28 rhickey: last Thursday, still haven't looked at the 3 hours of material

17:28 Lau_of_DK: Send them to me, I'll run through them in a jiffy

17:31 rhickey: 2gb

17:31 probably needs editing :)

17:32 Chouser: user=> (re-split #"\\W" "different/separators between:these")

17:32 ("different" "/" "separators" " " "between" ":" "these")

17:32 Lau_of_DK: Ok. How did they take it? Were you pitching Lisp in itself?

17:37 baggico: man. i blab so much about lisp to my colleagues at work...

17:37 well... when something sets me off anyway :)

17:38 people are generally either: "huh?" or receptive.

17:41 rhickey: Chouser: re-split wraps String.split?

17:42 or lazy?

17:42 Chouser: lazy

17:42 The implementation looks a lot like re-seq. ;-)

17:43 I generally use it in combination with (partition) to do search/replace in strings.

17:44 (for [[text url] (partition 2 (re-split #"..." text))] ...)

17:45 that's what converts plaintext urls in the irc log into <a> links

17:47 Lau_of_DK: this I dont get

17:47 (import '(java.lang.Math))

17:47 (. abs -5)

17:47 Error: Abs not found in this context

17:47 but it works fine in Java

17:47 Chouser: (import '(java.lang.Math abs))

17:48 Lau_of_DK: gives the same error

17:48 rhickey: all of java.lang is already imported

17:48 Chouser: oh, sorry.

17:48 (. Math abs 5)

17:48 rhickey: there you go

17:51 Chouser: Lau_of_DK: get it?

17:57 Lau_of_DK: Yes sir, thanks

17:58 Chouser: very good. :-)

Logging service provided by n01se.net