#clojure log - Jun 25 2008

The Joy of Clojure
Main Clojure site
Google Group
IRC
List of all logged dates

0:34 jgrant: anyone had any experience with jvm on FreeBSD multi-core systems ?

0:35 i have a multi-threaded app written in clojure and i'm not seeing it(probably the jvm) distribute threads across the 4 cores ?

0:36 Using jdk-1.6 on FreeBSD 7.0 SMP (4 cores)

0:36 surely the jvm uses native threads ?

0:37 the java process itself will shift between the 4 cores but top reports only 25% cpu usage (100% for the cpu that the process is currently running on)

1:33 albino: jgrant: are you using the jvm that comes with freebsd?

1:57 jgrant: albino : i'm using the port jdk16

2:00 ok i just figured out that it is not the jvm on freebsd7

2:00 the following simple code maxes out all 4 cores :

2:00 public class Test implements Runnable

2:00 {

2:00 private volatile int m_counter;

2:00 public void run()

2:00 {

2:00 for (;;)

2:00 for (int i = 0; i < 1000000; i++)

2:00 m_counter++;

2:00 }

2:00 public static void main(String[] args)

2:00 {

2:00 Test test1 = new Test();

2:01 Test test2 = new Test();

2:01 Test test3 = new Test();

2:01 Test test4 = new Test();

2:01 Thread thread1 = new Thread(test1);

2:01 Thread thread2 = new Thread(test2);

2:01 Thread thread3 = new Thread(test3);

2:01 Thread thread4 = new Thread(test4);

2:01 thread1.start(); thread2.start(); thread3.start(); thread4.start();

2:01 try {

2:01 thread1.join(); thread2.join(); thread3.join(); thread4.join();

2:01 } catch (InterruptedException iex) {

2:01 System.out.println(iex);

2:01 }

2:01 if (test1.m_counter == test2.m_counter)

2:01 System.out.println("Don't care, this is just to fool the optimiser");

2:01 }

2:01 }

2:05 albino: jgrant: for code of that size please use a pastebin

2:07 jgrant: sure

2:07 cgrand: jgrant: what is the problem? Maxing out 4 cores is expected: you put each core in a busy loop.

2:09 jgrant: no no, that is an example of what i want, i.e. to max out cores

2:10 but in my other code ( a socket listener ) it sticks to one core only even though it's using a newFixedThreadPool

2:10 the other code is in clojure too

2:11 i've heard the synchronized blocks can cause that sometimes

2:13 rich's example of parrallel map apparently works --> http://clojure.org/refs

2:13 cgrand: can you paste some code?

2:14 http://paste.lisp.org/new/clojure

2:14 lisppaste8: jgrant pasted "jgrant" at http://paste.lisp.org/display/62808

2:15 jgrant: it makes use of a single ServeSocket and pool is just pool (. Executors (newFixedThreadPool nthreads))

2:15 where nthreads is 64

2:19 handle-req function is trivial it just reads the input from sin and writes some simple text out

2:28 cgrand: how do you stress this server?

2:29 jgrant: ab -v1 -n 10000000 -c 400 http://127.0.0.1:8000/

2:29 using apache bench

2:29 it's picky, only accepts ips for url

2:32 cgrand: you said that this code "sticks to one core only" but does it max out during ab runs?

2:32 jgrant: it maxes out only one core

2:32 when running ab to stress it

2:32 cgrand: ok

2:34 and if you try to put some more "stuff to do" in your handle-req (eg a small busy loop) does it still stick to one core?

2:38 jgrant: (. Thread (sleep 100)) is what i tried at the end of handle-req and that makes no difference

2:39 another detail : each thread updates a global ref defined as (def *serve-count* (ref 0))

2:39 by calling (dosync

2:39 (commute *serve-count* inc))

2:39 could that be a problem ?

2:41 cgrand: (. Thread (sleep 100)) doesn't keep the cpu busy, so the scheduler can hand it another thread -- it just prevents the executor from sending a new job.

2:41 jgrant: do you have an example of 'busy loop' ?

2:42 cgrand: I haven't had any pb with (dosync (commute *something* inc)) so far (on windows, linux and solaris)

2:42 jgrant: me either on OS X / FreeBSD

2:42 i may have seen some on linux

2:44 cgrand: busy loop (reduce + (range 1000000))

2:46 jgrant: handle-req is already reading the input and generating a response then sending it back

2:47 surely that is 'busy' enough for each thread ?

2:50 cgrand: I can't tell. Does the pb go away when you remove the code updating the global ref?

2:53 jgrant: no

3:37 cgrand: jgrant: I tried to write a server that reads 3 lines and send them back mirrored (using yiour main loop), I stressed it with your ab command line, CPU usage stayed around 100%. I then added (reduce + (range 1000000)) at the start of handle-req and CPU usage maxed out (>190%, dual core -- I even saw 202% at some point :-))

3:38 jgrant: ah

3:38 so i guess just sending a response is not enough stress for the jvm to spread the threads across cpus then ?

3:39 how many threads did you use in ab and how many did you use for the thread pool ?

3:40 cgrand: for ab I used your command line as is and 64 threads in the pool

3:49 jgrant: cgrand : thanks i verified the same

3:49 cgrand : the socket handling is not enough to require more than one cpu it seems

3:50 cgrand: or it requires too much syscalls

3:51 jgrant: no don't think that's it because leaving the socket handling in but adding the reduce call makes use of multiple cpus

3:53 turning up ab concurrent clients may get multiple cpu usage

3:53 but the socket handling may be the bottle neck at that point

8:47 wwmorgan: (= (disj (sorted-set :a) :a) (sorted-set)) evaluates to false but it should be true

9:28 rhickey: that's a bug, right?

9:35 cgrand: wwwmorgan, in the meantime you can fix it in your copy by replacing ([] {}) with ([] (. clojure.lang.PersistentTreeSet EMPTY)) in the def of reduce-set in boot.clj, it should do the trick

9:43 wwmorgan: cgrand: thanks

9:44 cgrand: wwmorgan: it seems that you can even delete this line entirely

11:30 cemerick: looks like genclass.clj isn't loaded by default -- am I crazy, or wasn't that being loaded automatically before?

12:44 Lau_of_DK: Evening Gents

12:45 blackdog: cemerick, you can load the base cljs like this (import '(clojure.lang RT))

12:45 (. RT loadResourceScript "inspector.clj")

12:57 cgrand: Lau_of_DK: evening

13:11 Lau_of_DK: I remember somebody mentioning porting Clojure to ikvm - I just want to know, does this mean that Clojure can run on .Net without making any use of Java whatsoever?

13:13 blackdog: Lau_of_DK, it was cemerick who was talking about that

13:13 Lau_of_DK: cemerick: awake and educate :)

13:13 blackdog: although i think he was La_mer at the time

13:14 cemerick: blackdog: regarding genclass.clj, etc., I know about loadResourceScript and such -- I was just hoping to avoid having to bring up the "standard libraries" myself ahead of generating classfiles

13:14 blackdog: k

13:15 cemerick: And yes, I was la_mer until recently -- changed to cemerick so that people in the real world would know who I was and vice versa

13:15 Lau_of_DK: cemerick: Did I understand you correctly? You got Clojure running on pure .Net ? No Java...

13:15 cemerick: Lau_of_DK: ikvm/ikvmc allows you to use just about any java library in .NET without the JVM -- the java standard libraries are required, though (and are provided as DLL's by ikvm)

13:16 Lau_of_DK: Oh ok. That provides a gigantic performance boost doesnt iT?

13:18 cemerick: Lau_of_DK: Well, keep in mind that .NET is a runtime just like the jvm -- it's not like Clojure (or any other java lib) is being distilled to a native library.

13:19 So, performance is good as a ikvm-generated DLL -- good enough that I haven't benchmarked things in that area for a long time.

13:20 blackdog: cemerick, so you use ikvm for production stuff?

13:21 cemerick: blackdog: Yes: http://snowtide.com/PDFTextStream.NET

13:21 blackdog: have you used it for long running servers?

13:21 cemerick: Yes, although our customers beat it up more than we do....with good results. :-)

13:21 blackdog: ah ok,

13:21 Lau_of_DK: pretty awesome

13:22 cemerick: There was a hiccup recently related to the correspondence between classloaders and AppDomains, but that was fixed in ikvm v0.34, IIRC

13:22 blackdog: it's amazing, and ikvm done by one guy too

13:22 Lau_of_DK: There are a few guys gifted with unique intellects obviously

13:23 cemerick: We're pretty far behind in the ikvm releases (for PDFTextStream, anyway), simply because we have customers on .NET 1.1 still.

13:23 blackdog: i've seen snowtide befoer somewhere

13:23 Lau_of_DK: .Net is backwards compatible - why not upgrade?

13:25 blackdog: ah

13:25 that's where, i ported your difflib libaray to haxe

13:26 ah and you are chas emerick the author of the difflib

13:26 cemerick: Lau_of_DK: Yes and no -- ikvm v0.36 doesn't support .NET 1.1, only 2.0+

13:26 blackdog: Indeed. Nice to meet you. I'm glad to see that jsdifflib is getting around. :-)

13:27 I'm afraid it's pretty neglected at this point.

13:27 blackdog: :)

13:27 as is my port

13:27 cemerick: Thanks for mentioning haxe -- I hadn't seen it before.

13:27 blackdog: haxe was last years language du jour, now it's clo jure

13:28 it does rock

13:28 but the server side story is weak, hence i'm looking at jvm/clojure

13:28 cemerick: So, given that genclass and friends aren't loaded by default, is there a "recommended" way to bootstrap clojure maximally? I'd like to not reinvent the wheel.

13:29 (if a such a wheel exists, I suppose)

13:43 Lau_of_DK: cemerick: if the wheel does exist, it just stopped turning

13:43 rhickey: user.clj is auto-loaded

14:08 cemerick: rhickey: That's interesting; user.clj in pwd is autoloaded -- is there a standard search path for finding user.clj (pwd, ~/.user.clj, etc)?

14:10 rhickey: it uses the classpath

14:24 cemerick: A good and bad thing ;-)

15:28 rhickey: you mentioned (.get clojure.lang.Compiler/SOURCE_PATH) yesterday -- that works great when you're using load-file, but it's a relative path (relative to the classpath entry's root, I suppose) when the file being loaded is user.clp

15:30 I know it's an unofficial API point, but I thought I'd mention it anyway :-)

15:30 Lau_of_DK: Has anybody started any bigger projects in Clojure yet ?

15:30 (bigger = bigger than a port of msPaint)

15:31 cemerick: Lau_of_DK: I'm guessing that no one's going to spill the beans until there's something impressive to look at. ;-)

15:31 arbscht: I have difficulty making my projects look big when written in lisp :)

15:40 Lau_of_DK: cemerick: sure - It'd just be nice with some inspiration - Im not sure Im fully grasping the possibilities yet

15:40 Like Rich talked about Election Projection systems... I have no clue :)

16:13 abw_: hello guys, could someone tell me how to go through the sexp history in the clojure repl as rich hickey did in the screencast?

16:13 drewr: abw_: Got Emacs?

16:13 abw_: yes

16:14 and all the clojure code stuffs

16:15 drewr: Then M-p, my friend.

16:15 Just like anything else.

16:16 abw_: drewr: great, thanks!

16:17 cemerick: abw_: FWIW, it's Ctrl-up in Enclojure :-)

16:17 (just in case you want to join the herd someday ;-) )

16:17 * cemerick thinks that gen-class is absolutely the cat's nuts

16:24 abw_: cemerick:cool, I've noted that for when enclojure gets more mature

16:59 I'm thinking of writing a ClojureTutorial app once I know the Clojure language more completely and with more experience. It will draw ideas from Lisp Critic and the socratic method and work like a more computationally intelligent text based adventure game because of the Lisp Critic. Right now I am taking notes on what it's like to know less Clojure, how I learn it and what the knowledge dependancies are.

17:31 cemerick: rhickey: I'm having trouble defining a method for a class in gen-and-save-class that has as a return type the generated class. I'd like to do this, but gen-class complains that the 'pkg.Foo symbol doesn't have a getType method: (gen-and-save-class "." 'pkg.Foo :implements [Bar] :methods [['union [java.util.Collection] 'pkg.Foo]])

18:13 pjb3: Hey clojurers, I just wrote up a function to execute a sql query and return the first column of the first row as a string

18:13 http://pastie.org/222226

18:13 So if you execute (db/select-first "SELECT login FROM users")

18:13 You get "pbarry"

18:14 Any comments on the code? I'm wondering if there are better, more idiomatic ways to do that

18:18 specifically I'm wondering how to put a try/catch statement into do-with-connection, which will always make sure the connection gets closed

18:31 lisppaste8: blackdog pasted "sql" at http://paste.lisp.org/display/62844

18:31 blackdog: pjb3, i'm using that

18:33 i can't tell you if it's better or not as i'm new to clojure too, but it's different !

18:34 pjb3: What happens when getConnection throws an exception?

18:35 blackdog: oh i didn't hand;e that :)

18:36 pjb3: Yeah, this is the Java code: http://pastie.org/222245

18:37 You set con to null, then do the getConnection in the try, and then check to see if con is null in the finally

18:37 but I can't figure out how you would do that in clojure

18:37 blackdog: oh

18:37 pjb3: since you can't let [con nil] and then change it later

18:37 blackdog: yea i see

18:38 pjb3: rhickey: help?

18:38 blackdog: well i suppose it's the same as with-open in boot.clj

18:38 somehow it doesn't matter :)

18:39 i don't know

18:39 yes, now i realise i glossed over the matter

18:42 pjb3: To use with-open, wouldn't it be (with-open con (.getConnection DriverManager "jdbc:whatever") (stuff))

18:42 which gives you the same problem, because the call to .getConnection happens outside the try

18:51 lisppaste8: pjb3 pasted "with-connection" at http://paste.lisp.org/display/62845

19:14 jgrant: Is it really not possible to modify a hash map created with (hash-map ...) ?

19:14 (get mymap :key) works

19:14 (put mymap : val) fails ?

19:15 that is (put mymap :key val) fails ?

19:24 anyone ?

19:24 or should i just use Java's HashMap class instead ?

19:30 ok assoc works (i guess i should read the API more carefully)

21:16 abw_: the clojure-mode does not detect soft word wrapping of Clojure comments in Aquamacs highlighting the wrapped comments as code. Anyone else notice this bug?

21:22 pjb3: If I try to add metadata to a java object, specifically a java.sql.ResultSet, I get a java.lang.IncompatibleClassChangeError

21:22 Is that to be expected?

22:29 rhickey: pjb3: metadata is only supported on certain Clojure data structures - symbols and collections

22:29 pjb3: rhickey: Ok

22:31 cemerick: rhickey: Good evening. I don't suppose you noticed my note about regarding defining methods that take or return the type that is being generated....

22:32 rhickey: yes, when I saw it you weren't on - right now there is no way to do that

22:34 I'll look into it

22:38 cemerick: rhickey: That's what I thought. I've been idly working on understanding what's going on in gen-class, and I think I can see what needs to be done, though dimly. :-)

Logging service provided by n01se.net