#clojure log - Nov 09 2008

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

0:12 albino: Has anyone created a duplicate of the api page with examples of each thing listed?

0:13 duck1123: can anyone point to any code that makes heavy use of regex?

0:13 albino: afaik, no

0:14 albino: I do so much better with examples

0:14 and grepping the code for them isn't as fast

0:39 _sk: hi all, is let equivalent to binding?

4:21 baggles: save me

6:02 Lau_of_DK: Is anyone here in possion of some basic Jetty skills ?

6:03 Lets say Im located in /x/y/z/jetty and wish to make /x/y/z/jetty/webapps/proj1 the default page to be displayed when my IP is accessed, I would imagine that I launch Jetty with this command 'java -Djetty.home=/x/y/z/jetty/webapps/proj1 -jar start.jar', but this does not seem to be the case

6:03 tWip: Do you mean Jetty spesific or servlet skills?

6:03 Lau_of_DK: Where do I go wrong ?=

6:04 tWip: no I think jetty home must be the root of jetty, it does not denote a "default" web app to run

6:05 Lau_of_DK: Understood

6:07 tWip: you need to put your app with name root.war or a directory called root

6:08 that should deploy it to / context path

6:08 at least that's what my webapps/README.TXT says

6:10 leafw: back

6:10 Lau_of_DK: Yes you are quite right. I did not gain that same understanding from the README, but thank you, its working now

6:10 baggles: has anyone had much success making applets with clojure?

6:11 i had a search through the mailing list subjects ... but the only message i found was mine in ... march or february

6:11 leafw: baggles: what is the problem? Can see why it shouldn't work just fine

6:11 baggles: createClassLoader permissions.

6:11 leafw: baggles: sorry, no idea about that method.

6:12 baggles: maybe i'll go back and try again

6:12 tWip: I think you will need the AllPermission for your codebase

6:12 to create class loaders

6:13 baggles: well, afaik clojure needs that, and making just a random applet for the web, you don't get those permissions

6:14 but it's been a while since i've tried, so i might just be blathering

6:14 tWip: the client needs to grant that policy, or you need to sign your applet jar

6:15 Lau_of_DK: Are there any blogs available, which talk about Clojure webapps, servlets, Comets, etc. ?

6:16 baggles: heh

6:16 when i put "clojure applet" into google, i just get a link to a post i made on the mailing list

6:17 tWip: AOT compilation should solve your classloader problems, but that's not done yet I think

6:17 baggles: aot?

6:17 tWip: ahead of time

6:17 Lau_of_DK: tWip, there is so much talk about AOT nowadays, do you have a link whereby I can educate myself on AOT and its relation to Clojure?

6:18 baggles: oh. as in: removing the compiler from clojure and just having byte code.

6:18 to be honest, i just want a language that is fairly close to java that i can write macros for.

6:19 tWip: I don't really know enough about the AOT spesifics in Clojure to say more on the matter :P

6:27 Fanda: Rich is currently working on AOT

6:27 http://richhickey.backpackit.com/pub/1597914

6:27 there is a relation with genclass, which is not resolved yet, I guess

6:27 it could simplify things

6:28 baggles: hm.

6:29 i have some vague notion with respect to java about having a thread with different permissions

6:33 i wonder how long it would take me to write a set of common lisp macros that just dumped java source code out.

6:36 tWip: baggles, like linj?

6:36 baggles: like that

6:37 * baggles downloads it

6:37 baggles: i was thinking of something closer to java syntax itself, but that looks good

6:37 i mean not syntax, but nomenclature

6:37 java, lispified

6:37 * baggles tries anyway

6:37 baggles: oh wait the website is broken

6:38 :(

7:22 Lau_of_DK: Can anybody here provide some insight, into how I will make Jetty load a Clojure-file when presenting pages?

7:24 tWip: Lau_of_DK: so are you doing from scratch?

7:24 not using webjure or compojure that is

7:24 Lau_of_DK: Yes, I've just installed Jetty and is looking to integrate Clojure from scratch - If there is much to be gained from forking those projects, I'll be happy to do so

7:25 I would just like to begin with some understanding of the underlying logic

7:25 tWip: you need to make a servlet

7:25 that is the basic unit of java web programming

7:26 on the other hand, if you are comfortable with being married to jetty, you can use its embedding system directly and define handlers

7:27 Lau_of_DK: Assuming that I dont fork any existing project?

7:27 tWip: I don't know why you would need to fork anything just to use it

7:28 but you are ofcourse welcome to do so

7:30 Lau_of_DK: I will probably need to extend certain features to match the needs of my projects, thus effectively making it a fork, at least as I have understood the term

7:32 tWip: well if they are good features, you could contribute them back

7:32 but gotta go now

7:33 Lau_of_DK: Ok, thank you for your input

7:37 duck1123: Lau_of_DK: if you are starting from scratch, you might want to look into using Tomcat or something similar as opposed to Jetty

7:38 Lau_of_DK: For what reason ?

7:38 duck1123: Jetty's good if you want something lightweight, but Tomcat will give you more features

7:38 Lau_of_DK: Alright, I'll have to look into it

7:38 duck1123, how much have you accomplished on your compojure project ?

7:39 duck1123: also, I'm secretly hoping you solve the tomcat/clojure issue so I can steal it for my own use :)

7:39 Lau_of_DK: Haha

7:39 The secrets out my friend

7:39 duck1123: not much. The wife and I played Guitar Hero last night, and I don't get much time durring the week

7:40 I was about to make some coffee and get back to work

7:40 Lau_of_DK: I'll rephrase. If I adopted your project, what features would that give me to start out with ?

7:42 (I'm gittin' it as we speak)

7:43 duck1123: unless you want exactly what I'm doing, go with weavejester's instead

7:43 Lau_of_DK: Alright

7:44 duck1123: although I have basic RESTful interaction with a mysql database, OpenID authentication, registration, and operations on a RDF model with Jena

7:45 if you want any of those, you could steal them from my project

7:45 Lau_of_DK: Well I thank you for the invitation, I'd like to look into your Mysql Interop actually

7:48 duck1123: also, if you want, there's a Compojure group on google groups

7:50 Lau_of_DK: Great, I'll sign up

8:09 If I remember correctly, I used to be able to use this approach (in-ns 'myutils :uses 'clojure), but this has been removed. Is anybody aware of how I would do this nowadays?

8:23 kotarak: Lau_of_DK: (ns myutils :refer-clojure :use (some.stuff a b) :require [clojure.zip :as zip] :import (some.other Class)), rough summary (:refer-clojure is optional)

8:23 Lau_of_DK: Thanks alto

8:23 alot

8:24 kotarak: oops. () missing around :use ... and :require ... and :import ...

8:24 and :refer-clojure also.

8:25 Lau_of_DK: (ns myutils (:refer-clojure) (:use (some.stuff a b)) (:require [clojure.zip :as zip]) (:import (some.other Class))), corrected

8:26 Lau_of_DK: Thanks

8:38 In working with Compojure it seems that when you update a .clj file, in order to see the change you need to restart the server - Is this correct?

9:43 cpc26: anyone having trouble with this mornings build?

9:45 duck1123: I think the build is still unstable

9:45 Rich is working on AOT

9:46 the past few commits have been labeled "DO NOT USE"

9:47 cpc26: thanks!

9:48 duck1123: np

9:52 Does anyone know if java regex supports lookaround?

9:53 I'm thinking it doesn't but I haven't been able to find anything that says one way or another

10:06 Chousuke: duck1123: the doc for Pattern says it does

10:06 http://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/Pattern.html and search for lookahead

10:09 duck1123: thanks

10:10 I don't know why my query didn't work before, but it does now

10:24 MattBowen: Hi all. I'm trying to use Clojure from svn, but I'm getting an exception. I did a checkout of r1092, ran mvn install, and added clojure-lang-1.0-SNAPSHOT.jar to my classpath, but I get this when I try "java clojure.lang.Repl": http://paste.lisp.org/display/70014

10:24 Chousuke: use r1088

10:25 duck1123: svn clojure is unstable atm

10:25 does anyone know what this might mean? cannot load '/view/entry/entry.clj' again while it is loading

10:26 MattBowen: Chousuke: thank you much

11:41 Cark: hum ...how to get a sequence from a java Enumeration ?

11:42 kotarak: Does it implement Iterable? Then (seq ...)

11:42 Cark: looks like it doesn't =/

11:43 damn that's almost the same interface Iterable and Enumeration

11:46 ah : (enumeration-seq enum)

11:57 Chouser: kotarak: if you specify no options to (:refer-clojure) you can just leave it out.

11:57 kotarak: Chouser: I think I wrote, that it's optional.

11:58 Chouser: oh, ok. sorry.

11:58 kotarak: np

11:58 Chouser: I should have mentioned the options, though...

12:09 AWizzArd: Moin

12:15 * Chouser waves

12:24 Chouser: writing is hard. I've started writing a blog entry on how to write a somewhat complicated macro. Does anyone think that's worth finishing?

12:25 kotarak: Chouser: I think so. I understand that this is some kind of step-by-step guide how you created the macro? Something like that is certainly interesting for someone new to macros.

12:26 Chouser: I tried to give something like this on the wiki. But well... It's a rather cheesy chapter.

12:26 Chouser: yeah, that's what it is. ok, I'll keep at it. Thanks for the encouragement. :-)

12:27 kotarak: Chouser: pure egoism. It's always good to look for some new technique for the own toolbox. ;)

12:27 AWizzArd: let us know when it is done

12:30 Chouser: well, me asking for encouragement is a bit egoist as well. Or something -- insecurity.

12:32 Chousuke: I think macro writing is still somewhat of an wizardly skill.

12:33 there's not that much material on it and whatever we have is not exactly easy...

12:33 leafw: Chousuke: ?? macro writing is just abstracting a function

12:33 Chousuke: see this example, which walks you through it: http://pacific.mpi-cbg.de/cgi-bin/gitweb.cgi?p=fiji.git;a=blob;f=plugins/Examples/Multithreaded_Image_Processing.clj;hb=HEAD

12:33 Chousuke: yeah but what are good techniques to use when writing a macro? how to choose when to write a macro? etc.

12:34 leafw: (or see it linked here: http://pacific.mpi-cbg.de/wiki/index.php/Clojure_Scripting#Example_Clojure_plugins_included_in_Fiji )

12:34 in my experience, one writes a macro when in need of abstracting a common set of similar functions. The same way that a function abstracts a common set of code blocks.

12:35 kotarak: Chouser: do you need control over the evaluation of the arguments => macro, otherwise => function is a rough guideline.

12:35 Opps

12:35 I meant Chousuke sorry.

12:36 Chousuke: well, yeah

12:37 kotarak: Chousuke: http://en.wikibooks.org/wiki/Clojure_Programming#Macros, scroll down a bit, the example with with-open-file is pattern, which I find reoccurring.

12:38 Chousuke: I know macros can be used to abstract away patterns, but that still doesn't tell much about what are good, general techniques to use when *writing* macros

12:38 like, using helper functions that return forms.

12:39 leafw: Chousuke: as many as macro writers ... although general guidelines: use gensym (or name# as shortcut) always!

12:40 Chousuke: even if you say that, to a beginner it's not evident *why* one should do that :)

12:40 Chouser: this is helpful, guys. Keep it up.

12:41 leafw: Chousuke: gigamonkeys lisp tutorial has some advice to give on macro writing. Same for every other lisp book out there.

12:41 Chouser: ?

12:41 kotarak: Chousuke: I tried to explain the effects of not using gensym with a bad example in the wiki link above.

12:41 Chousuke: it avoids capturing names, but it's easy to get confused with macros about when you want a symbol and when you want the value named by that symbol

12:42 kotarak: Chousuke: it the same with ref and @ref. When do I want the ref and when the deref'd value?

12:42 Chousuke: kotarak: I think that's much easier to reason about

12:43 Chouser: leafw: by having people describe what they find hardest to understand about writing macros I may be able to make my blog post more relevant to more people.

12:45 leafw: Chouser: great. Where is such blog post?

12:45 Chouser: I'm working on it.

12:47 leafw: always something good is cooking around clojure ..

12:47 Chousuke: for example, when you do (let [foo (gensym)] ... then, in a macro, the value of foo is some random symbol and in your macro, instead of binding into the name "foo", you need to use the *value* of foo as the naming symbol. worse yet, inside a syntax-quoted form "foo" is actually treated different from the "foo" outside of the syntax-quote. For simple examples, it's easy to keep track of them, but when it gets more complex I know *I* have problems

12:48 kotarak: If you loose track, then this is a sign, the macro is too complicated and better split up into some helpers.

12:49 Chousuke: yeah. but I find documentation about such macro writing techniques rather scarce

12:52 AWizzArd: Chousuke: in principle a macro is always then useful, when you don't want to put code into #(..)

12:52 Chousuke: for example map'ing a form-generating function over the arguments of a macro and then using ~@ to "splice" the result into the macro

12:53 no doubt common when writing macros but not exactly obvious.

12:53 AWizzArd: you never need macros, you can do everything exclusively with functional programming as well

12:53 it just looks not always as beautiful

12:54 Chousuke: oh, I'm not arguing against macros :)

12:54 AWizzArd: a macro is there to remove a ' or #(..) from your code

12:54 Chousuke: they're great. I just want to hear more about what tricks and techniques people use when writing macros.

12:54 AWizzArd: and all macros can be written in one line of code

12:55 their purpose is to introduce lazyness

12:55 when you have code that you don't want to be evaluated in some cases

12:55 Chousuke: they reduce repetition in code, yeah.

12:56 AWizzArd: what they reduce is the use of ' and #(..)

12:56 Chousuke: I view macros as basically code transformations.

12:56 I want to know, what are good techniques for doing code transformation :)

12:56 AWizzArd: yes, they can also be seen as some mini-compiler :-)

12:58 gnuvince_: Does anybody know a Java (or Clojure) library that would perform the same tasks as Python's BeautifulSoup?

13:00 AWizzArd: Chousuke: well yeah, that's a bit general.. but anyway, you should consider to read in this book: http://www.paulgraham.com/onlisp.html There is a download link for downloading it legally and free from the authors website.

13:00 This is more or less *the* book about macros.

13:06 Chouser: gnuvince_: http://home.ccil.org/~cowan/XML/tagsoup/

13:06 onlisp is great.

13:06 gnuvince_: Chouser: he doesn't seem to have any documentation though.

13:17 Chouser: yeah, it's just a sax-compatible parser. you can drop it into clojure.xml/parse or clojure.contrib.lazy-xml to parse HTML instead of XML

13:20 sohail: anyone have an example of gen-and-load-class?

13:22 hm, does clojure not have any tests? I can't seem to find any..

13:23 Lau_of_DK just the man I'm looking for

13:23 Lau_of_DK: sohail, :)

13:23 sohail: how did you do the anonymous function for the slot thing?

13:23 Lau_of_DK: Yes, if you want to test if i = 1, you can test like this (= i 2)

13:23 sohail: I'm thinking you need a gen-and-load-class

13:23 Lau_of_DK: Is pretty advanced, but you'll pick it up

13:23 sohail: Lau_of_DK :-P

13:23 Lau_of_DK: Yes you need gen-and-load-class

13:24 sohail: can you paste what you did for me?

13:24 Lau_of_DK: I can do one better, but lets go black-ops

14:33 jgracin: is there a function which checks whether a sequence contains an element?

14:33 Chouser: (some #{item} sequence)

14:35 jgracin: Chouser: thanks. I guess it's short enough. I've been bitten twice by (contains? [1 2] 1), though.

14:36 Chouser: yep, contains? is for key/value (a.k.a. associative) collections, not sequences.

14:36 jgracin: I keep forgetting that. Hopefully, two times will be enough to remember. :-)

14:38 Chouser: The idea of renaming it has been shot down before. I wonder if "has-key?" as considered.

14:39 * Chouser tries to find the thread.

14:43 Chouser: http://groups.google.com/group/clojure/msg/b26d88598b14254c

14:43 "I don't think contains-key? works for sets or vectors very well, and

14:43 am not inclined to change this much-used name"

14:57 bradbev: Is there an idiomatic way to spawn a thread in Clojure? It seems like it would be such a common thing to do that there should be, but apart from using Java interop I can't see it.

14:57 AWizzArd: bradbev: send / send-off

14:57 bradbev: Or agents, but that seems like a slightly different thing to me

14:57 Chouser: bradbev: you can either use and agent of Java's Executor framework.

14:58 bradbev: so for a server handling connections, I'd use send-off to do the work?

14:58 AWizzArd: or send

14:58 if you have enough cpus :-)

14:58 bradbev: and the agent is effectively my thread handle

14:58 AWizzArd: send-off has the disadvantage that it grows your thread pool.. that could be not so nice when your server gets attacked

14:59 send uses a thread pool of, I believe 2*number-of-cores threads

14:59 bradbev: AWizzArd: but sent cannot perform blocking ops, which I may need

14:59 sorry send & there should have been a ? in there :)

15:00 AWizzArd: send will put your tasks in a queue and work them off

15:00 Chouser: for blocking operations, you should use send-off, yes.

15:00 AWizzArd: send-off does the same, only that it will extend your queue if at call time there is no thread available in the pool

15:01 in priniple send-off should be the choice if you don't expect anyone to hammer your server with many requests

15:01 bradbev: Right, so Agents are kind of like thread handles, but with a built in thread-safe RPC facility?

15:02 AWizzArd: but if you have 300 cpus available anyway then send should work fine, as your default pool would consist of around 600 threads or so

15:02 Chouser: agents aren't tied to a particular thread -- when an agent's action is complete, the thread returns to a pool to be used by another agent action later.

15:02 rhickey: bradbev: agents work off thread pools

15:03 AWizzArd: the choice should really be made based on blocking or not - both use pools

15:04 AWizzArd: I see. I looked at the Java sources and understood it in that way, that send-off is doing the same, but only extends the pool if needed.

15:04 but I am not a Java guy, so...

15:06 bradbev: thanks guys, have to dash.

15:16 AWizzArd: So what is send-off doing else? I understand that send puts all tasks in a queue and a fixed number of threads works these tasks off. And send-off is doing the same, only that there is not a queue - when a new tasks comes in and the pool is empty, then a new thread will come into the pool and begin it's work.

15:17 rhickey: AWizzArd: yes, they are different kinds of pools

15:17 Chousuke: hmm

15:19 AWizzArd: If humans are involved then send-off has the advantage of not letting someone wait. But when I do for example work with my Genetic Programming engine then I could accept the blocking behaviour. I don't want 80k threads running at the same time, but instead let my 80k programs run, using all cores available.

15:20 But having more threads running would cause more scheduling overhead, so in the end it would take even more time to do all calculations. Is that +/- right?

15:22 rhickey: if you are doing calculations you should use send, there's no reason to have more than #cores threads

15:22 AWizzArd: yes oki, thx

15:24 Chousuke: there are examples on the wiki for parsing integers that use (. Integer parseInt ...). Any objections for converting these to use /?

15:25 rhickey: Chousuke: no that would be preferable

15:26 Chousuke: okay. Thought so :)

15:27 AWizzArd: Btw, in his book about Clojure Stuart Halloway says in the preface that "Wherever you need it, you can get the exact same performance that you could get from hand-written Java code.". Is that definitly correct or more or less advertisement? ;-)

15:29 rhickey: AWizzArd: you can get direct calls to Java methods (no reflection or other overhead), and direct primitive math

15:29 AWizzArd: I see

15:31 apropos math... a function expt could be nice as (.pow ...) requires explicit "casting" to bigint

15:35 Lau_of_DK: rhickey, I can see that you falling a bit behind the group in terms of contributions to clojure-euler.wikispaces.com - it'd be great if you could provide the solutions between 192-210, thanks

15:47 Chousuke: hmm, there are still few uses of . for method calls in the wikibook page, but as those are non-static access I guess I'll let them be. readers probably should become acquaintainted with it anyway :/

16:32 RadioApeShot: Does clojure provide some what of hooking into the printer so I can customize how certain objects are printed at the REPL?

16:32 Is there a multi-method I can extend or something?

16:33 Chouser: yep, print-method

16:33 RadioApeShot: Thanks

16:34 Chouser: user=> (defmethod print-method String [obj w] (.write w (str "I'm a string: " obj)))

16:34 user=> "hello"

16:34 I'm a string: hello

16:36 RadioApeShot: What is the dispatch function for the method?

16:36 Chouser: class of the obj

16:36 check boot.clj for the actual definition.

16:37 RadioApeShot: Hm

16:37 The java-class?

16:37 So it is not possible to extend this to print a particular kind of map, say

16:39 Or it seems you can set the meta-data of thing to give it a "class"

16:40 Looking at boot.clj, anyway

16:40 danlarkin: Chouser: is . syntax for method calls deprecated?

16:40 Chouser: that's really an rhickey question. My best recollection is that it is not deprecated for instance calls.

16:42 danlarkin: what would be the alternative, doto?

16:43 Chouser: (.method instance arg1 arg2)

16:43 RadioApeShot: Chouser: Looking at boot.clj more carefully it seems like the custom printer only dispatches on the Java class of clojure values, so there is no simple way to define a custom printed "class" in pure Clojure. Is this right?

16:44 danlarkin: Chouser: ah.

16:45 Chouser: RadioApeShot: No, I don't think that's right.

16:45 RadioApeShot: Well, the dispatch method is class

16:46 And class just returns the Java class of the item passed in

16:46 So, for instance, all clojure PersistentHashMaps are clojure.lang.PersistentHashMap

16:46 But suppose I want a PersistentHashMap to represent a "class"

16:47 the function class isn't a multi-method, so I can't customize it to dispatch different for certain kinds of things.

16:47 Chouser: oh, I see what you're saying.

16:48 RadioApeShot: Say I want to print every PersistentHashMap with a {:polynomial true} association in a particular way.

16:48 That is, in fact, the case in point.

16:48 I have a polynomial algebra library I am porting from Scheme

16:48 PLT Scheme, in fact.

16:48 And I had a nice custom printer that printed out polynomials in a 3x^2 + 2x - 4 style.

16:49 PLT Scheme has a nice implementation of CLOS with a custom printer (although I think it stopped working in PLT 4).

16:49 I just as soon move all my code over to Clojure anyway.

16:50 Chouser: well, let's see. you could dynamically rebind class to your own function...

16:50 RadioApeShot: Hm

16:50 Like "around" the REPL?

16:50 Chouser: That's probably not the best solution, though. hm...

16:51 RadioApeShot: Maybe class should have a custom behavior for maps.

16:51 Obviously I am not in a position to make that change, though

16:52 Or maybe I need to implement the very basic polynomial type as a Java class

16:52 That might be the "Clojure Way"

16:52 Chouser: or perhaps print-method should use something a bit more complicated than "class"

16:53 RadioApeShot: Otherwise you are pushing towards having a within-clojure object system in addition to the Java object system.

16:53 Chouser: well, we're already quite a ways down that road with the multi-taxonomy heirarchy system used by multimethods

16:54 RadioApeShot: It just seems like there is not a good way of meshing a full object system with a Lisp style with Java.

16:54 Chousuke: hmm :/

16:54 RadioApeShot: Which is what this is getting at.

16:55 Looks like there are already polynomial libraries in Java

16:55 Chouser: I wonder if rhickey'd be open to having print-method dispatch on (:print-type ^obj) instead of class (if such a meta-key exists).

16:55 RadioApeShot: Probably best to write a thin wrapper

16:56 Chousuke: Chouser: some solution based on metadata is probably the cleanest option.

16:56 RadioApeShot: I am just partial to my implementation because I was learning to work with foldl when I was writing it and some of the code is "clever"

16:57 Well

16:57 Thanks for the advice

16:57 Chouser: I pretty sure he'd reject a single :type key, but maybe a specific :print-type would be ok.

16:57 RadioApeShot: Hopefully a better solution will come out of the discussion

17:08 brill: Does anyone here use Emacs as IDE for clojure?

17:08 * Hun

17:08 Hun: for certain values of I

17:17 AWizzArd: brill: I use Emacs+Slime+Clojure

17:20 brill: AWizzArd: Do you use brill clementsons setup or the setup at clojure?

17:21 abrooks: Chouser: Is there some existing Java interface for printables? That seems the more Java + duck-type-y thing to do.

17:21 AWizzArd: brill: no, I set up my emacs before I saw his postings. You can check out this post one possible .emacs config: http://paste.lisp.org/display/68867

17:22 Chouser: abrooks: .toString ?

17:23 abrooks: Chouser: For example...

17:23 brill: AWizzArd: Thanks. I just noticed that there's a lot of different ways of settings things up.

17:23 abrooks: Chouser: Is that part of an interface or merely a convention across classes?

17:24 Chouser: abrooks: I believe it's provided by Object.

17:24 AWizzArd: brill: in emacs try also M-x customize-group swank-clojure

17:24 abrooks: Chouser: I guess it's still testable from Clojure.

17:24 Chouser: Hm. Is .toString more like __str__ or __repr__?

17:24 AWizzArd: it will pop up a "graphical UI" to set some settings of Clojure, as soon you have set up the right files

17:25 Chouser: abrooks: __str__ I think. Java doesn't provide a __repr__ which is why readable printing in Clojure is useful and also such a big task.

17:26 abrooks: Chouser: Most __repr__ strings aren't readable either.

17:26 Chouser: oh

17:27 brill: AWizzArd: I'll try that.

17:27 abrooks: Chouser: >>> open("/etc/hosts").__repr__()

17:28 "<open file '/etc/hosts', mode 'r' at 0x7f2afd57faf8>"

17:28 Chouser: __repr__()s mostly useful for human debug. Often it is the same as __str__().

18:32 AWizzArd: rhickey: in one of your videos you mentioned that Clojures STM is semi-optimistic. Can you outline in a few words the difference to fully optimistic? I tried to google that but couldn't find anything.

18:34 rhickey: AWizzArd: a fully optimistic STM might let transactions run to their end, only to find out they can't succeed and retry - when Clojure's STM finds a ref conflict it will block, subject to deadlock detection. This causes less churn

18:37 AWizzArd: I see thanks.

18:39 And is it guaranteed that if 100 threads are trying to update a ref at the same time one of them will definitly always win? Or could it happen from time to time that they all need to retry?

18:40 rhickey: AWizzArd: livelock is possible, although there are retry limits that will kick in

18:43 AWizzArd: So can one say: the more updates/reads happen inside a dosync, the higher the chance that it will take more and more time until one transaction succeeds?

18:44 bradbev: I would like to write a little server that listens on a thread & spawns workers to handle connections. From a previous conversation, agents and send-off is the idiomatic clojure way to do this. How would I terminate the main listening loop? Send a message to the agent, change a global var?

18:45 rhickey: AWizzArd: all transaction issues involve the degree of contention - 2 transactions that have no overlapping refs can both succeed. Long writing transactions contending for the same refs as short writing transactions will have problems. But MVCC reads-don't-impede characteristics and commute both work towards reducing the contention footprint substantially

18:45 bradbev: and, I guess, what initial states (if any) should the agent have?

18:46 AWizzArd: and if really needed.. one could still lock the transactions I guess.. Clojure does nothing to prevent an explicit lock as I see it

18:47 rhickey: bradbev: initial state is completely application specific, but yes, you could send an action for termination. Don't neglect java.util.concurrent for workflow issues, like queues or latches

18:47 AWizzArd: manual locking and transactions are a bad combination

18:48 AWizzArd: hmm

18:48 bradbev: rhickey: that's the thing that is confusing me. An agent doesn't quite feel right here & I wonder if it is me misusing the concept, or if agents don't fit my use-caes

18:49 rhickey: bradbev: 'spawn workers' certainly sounds like a good fit for agents

18:49 AWizzArd: bradbev: can't you run the listener in its own agent in a loop that checks if a global var is set to true and as soon someone connects this listener will send-off some worker?

18:50 rhickey: AWizzArd: you'd want to uses queues or latches, not spin loops and flags

18:50 bradbev: AWizzArd: yes, that's what I'm thinking. Just have never used Agents before :)

18:50 AWizzArd: rhickey: in your ant example you used this running flag in an endless loop

18:51 bradbev: send & send-off messages from the same thread - will they be executed in the order they were sent?

18:51 rhickey: AWizzArd: that flag is a hack for repl control during testing, the loop is the core activity loop, the ants do real work in the loop

18:53 bradbev: yes

18:54 bradbev: are agent messages executed concurrently, or strictly in sequence?

18:54 rhickey: bradbev: any single agent is only doing one action at a time, but multiple agents act concurently

18:55 bradbev: right, ok. I can view an agent as a single threaded message processor then.

18:55 AWizzArd: bradbev: and it is not guaranteed that they will run exactly in the order you sent them off

18:55 rhickey: AWizzArd: not true, if sent from the same thread or agent, will execute in order

18:56 AWizzArd: you can do a little experiment with printing a number and see that it won't always go from 1 to 10

18:56 rhickey: AWizzArd: ?

18:56 bradbev: I presume that there is a queue that is thread safe, how stuff gets put on that queue might be non-ordered, but the queue is processes in order

18:57 rhickey: bradbev: if put on from multiple threads, order between threads is non-rodered, but still ordered within each thread

18:58 bradbev: that makes sense too

19:03 So I could set my server agent up so the listen function will listen with a timeout, and reschedule itself with a send-off. At the start of the function, inspect the agent & if it is not running, abort. Then I can stop the server by sending the agent a message from another thread

19:05 rhickey: bradbev: right

19:05 bradbev: that sounds like a plan then :) Thanks!

19:05 rhickey: sure

19:19 AWizzArd: rhickey: I don't rememeber exactly what I did when I was playing around with agents for the first time some weeks ago. I remember that I was printing out numbers and they were not showing up in order. From that experiment I still knew only that stuff and incorrectly assumed that agents don't run in order.

19:20 so good to have this update, thanks :-)

19:22 but the behaviour is different when using send-off yes?

19:23 so if we have (def x (map agent (range 30))) and then do (doseq a x (send-off a #(println %))) we really do get some random order

19:24 so I just remembered that I did an experiment, but not that I used send-off, so that was the source of my confusion.

19:27 Chouser: both send and send-off should behave the same in this regard, which is that the actions will be in indeterminite order because they're on *different* agents.

19:28 order is only guaranteed when the actions are coming from one thread (or agent) and going to one agent.

19:29 AWizzArd: Chouser: now I am confused again

19:29 Chouser: great, happy to help. :-)

19:29 bradbev: AWizzArd: an agent executes functions off a queue, in the order they were added to the queue

19:29 AWizzArd: bradbev: yes

19:29 bradbev: if you only add from one thread, then your order is determined

19:29 AWizzArd: yes

19:30 but why does my send-off example from above produce random looking output?

19:31 is it not the repl thread that sends them all off?

19:31 rhickey: AWizzArd: because you have 30 different agents

19:31 they act in parallel

19:31 bradbev: so each agent gets one task

19:31 AWizzArd: ah okay, then I misunderstood bradbev's question from above

19:32 all clear now

19:34 although it's funny that when I use send instead of send-off in my example above I get the right order ;-)

19:35 bradbev: a smaller threadpool may restrict the ordering so it looks right

19:35 AWizzArd: I tried it with 50 agents a few times

19:35 when using send-off 10 agents are enough to regularily get a random order

19:36 Chouser: AWizzArd: how many cores on the computer you tried that on?

19:36 AWizzArd: 2 cores here

19:37 but even on a single core machine there should be at least two threads running, as I think the source initializes the threadpool which is used by send with 2*available-cores

19:38 Chouser: looks like 2 + availableProcessors

19:38 AWizzArd: or that way, ok

19:38 so, 4 threads here

19:38 Chouser: which is just another way to get to 4

19:38 :-)

19:39 AWizzArd: (def x (map agent (range 100))) (doseq a x (send a #(println %)))

19:41 but try (range 10) and send-off

19:41 bradbev: send runs the function for an agent on existing threads, right? So that means you will only have 2-4 existing threads to run the funcs. If the funcs all run at the same speed, then you're quite likely to get the correct order. Send-off will expand the thread pool, causing new threads to get created (slow). So lets say threads 1-4 are used up, so (print 5) gets a new thread, which is slow to start. In the mean time (print 6) is now

19:41 so you might see 1 2 3 4 6 5

19:44 AWizzArd: bradbev: yes, sounds good

19:45 even when I try it with 1000 agents it is the correct order

19:45 bradbev: yeah, if you added a small random sleep in the function it would differ

19:45 AWizzArd: only in areas of about 10k agents the order gets a little bit out of sync

19:45 gnuvince_: Quick question, is OpenJDK just the name Sun gave to their JDK when they GPL'ed it or is it something different?

19:46 bradbev: hmm, maybe not actually

19:47 Chousuke: gnuvince_: yes... and no.

19:47 gnuvince_: OpenJDK is not completely equal to the sun JDK; Sun was unable to open source everything.

19:49 I think most of the encumbered parts have since been replaced with free implementations though.

19:50 As far as I know sun's JDK isn't based on OpenJDK yet, but might be in the future.

19:54 gnuvince_: ok

19:54 I was reading about this G1 Garbage Collector

19:54 quite interesting

20:47 blackdog_: danlarkin, a useful addition to the json encoder would be an option to encode keywords as strings

20:48 so (name keyword) somewhere

20:48 for json keys

20:48 danlarkin: keywords are encoded as strings currently

20:48 oh, but you mean without the colon?

20:48 blackdog_: ah you're there :)

20:48 yes

20:49 i'm just returning data from a db

20:49 and it's great except the keys have :

20:50 danlarkin: yeah I wavered back and forth on keeping the colon or not

20:51 blackdog_: so maybe an option like :indent, :keepColon

20:53 danlarkin: Mmmmmm, I'd have to pass it through encode-coll and encode-map-entry

20:54 blackdog_: i'm not sure if I've ever seen json keys with : in the key

20:54 kind of confusing with the : separator too

20:55 danlarkin: yeah... perhaps I should just always use (name keyword)

20:55 blackdog_: IMO it's less suprising

20:56 danlarkin: of course that'll drop the hint that they were keywords and when I write the decoder (json->clojure) I'll have to make all keys as strings or keywords

20:57 blackdog_: yea, hence optional is maybe best

20:57 good point

20:58 danlarkin: I'm leaning towards dropping the colon and making all keys be keywords when I decode

20:58 how does that strike you

20:59 blackdog_: yes, sounds fine, and the user can always make them strings if required

21:00 i guess except for "x y" keys

21:01 nevermind

21:03 danlarkin: alright I've made the change

21:03 blackdog_: w00t

21:06 i think you left the reflection flag on I just got lot's of interesting messages

21:06 apart from that works for me!

21:06 thanks

21:07 danlarkin: oh, oops... yes I did

21:08 there, fixed that too

21:08 blackdog_: thanks!

21:11 danlarkin: you should watch out for reflection though, it'll slow ya down a lot in the wrong places

21:11 blackdog_: true

21:22 danlarkin: heh, I'm glad someone's finding it useful

21:22 blackdog_: yea, i'm a web developer mostly so servlets returning json is bread and butter

21:23 and this is perfect for returning db records

21:23 danlarkin: it should be quite fast, I spent a bit of time tweaking things to that effect

21:24 blackdog_: coolio

21:24 danlarkin: hopefully soon I'll have a decoder... I want to use kotarak's parser combinator library but right now it's in a state of flux

21:25 blackdog_: looking forward to it

21:28 lisppaste8: bradbev pasted "Agents sending to themselves?" at http://paste.lisp.org/display/70056

21:29 bradbev: Are agent's allowed to send to themselves?

21:29 If so, what am I doing wrong?

21:33 Chouser: bradbev: java.lang.Exception: Agent has errors (NO_SOURCE_FILE:0)

21:33 bradbev: hmm, I don't see that...

21:33 Chouser: yeah, that's the squirrely thing about agents. The exceptions hide inside them

21:34 bradbev: ick.

21:34 Chouser: So try (def ag (agent nil)) then (send ag at 3)

21:34 bradbev: oh, you called agent-errors

21:34 Chouser: then @ag or right, (agent-errors ag)

21:35 bradbev: but no real clue as to the error itself...

21:35 Chouser: (.printStackTrace (first (agent-errors ag)))

21:37 bradbev: oh, I see - I'm trying to send to the agent-state, not the agent

21:37 damnit

21:37 Chouser: ah, good -- I hadn't spotted it yet

21:37 you can use *agent*

21:37 bradbev: awesome, thanks

21:38 ok, got it now, thanks much

21:39 Chouser: great

Logging service provided by n01se.net