#clojure log - May 23 2011

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

0:10 technomancy: wolfjb: you probably just want to use clojure.java.io/copy between a file and a zipoutputstream

0:16 wolfjb: technomancy: thanks! amalloy and ataggart have been very helpful in leading me toward that solution as well

0:33 I'm not understanding apply very well tonight, I thought I could call (apply addentry files zip-outputstream) which would turn into (addentry file zip-outputstream) for each file in files. however, that isn't working right.

0:33 technomancy: yeah; apparently I didn't read the backlog

0:33 ataggart: ,(apply + [1 2 3])

0:33 clojurebot: 6

0:34 ataggart: wolfjb: you probably want doseq

0:34 wolfjb: ah

0:34 ataggart: e.g., (doseq [f files] ...)

0:35 amalloy: wolfjb: apply "expands" the last argument (which must be a seq) in-place to result in a function call with more arguments

0:35 &(apply (partial list '+) 1 2 [3 4 5 6])

0:35 sexpbot: ⟹ (+ 1 2 3 4 5 6)

0:35 wolfjb: ah

0:35 (I feel like I'm saying that a lot)

0:35 heh

0:35 amalloy: then you're doing it right

0:35 wolfjb: thanks for the help

0:36 you guys are awesome!

0:37 woot! my zip stuff works now!

3:41 amalloy: do agents have a reset!-like functionality, or do i have to send them (constantly x)?

6:24 Vinzent: I'm using appengine-magic and have created a protocol and extended it to the appengine entity, but when I'm calling protocol fn on the entity queried from database, implementation for Object is chosen

6:29 example: http://paste.org/pastebin/view/33786

6:30 What could be the problem?

7:09 Cozey: Sorry for an OT, but I'd like to query Lisp programmers -- if you use/have to use Python or Ruby, which one do you choose and why?

7:14 manutter: Lisp programmers aren't awake this early ;)

7:15 Cozey: :-)))

7:16 vijaykiran: Hi all a quick question - I want to create a simple REST based app using compojure - which JSON library should I use ?

7:20 Vinzent: vijaykiran, take a look at clojure.contrib.json

7:22 raek: Vinzent: clojure.contrib.json has now been lifted to its own project: https://github.com/clojure/data.json

7:22 [org.clojure/data.json "0.1.0"]

7:22 (works with both clojure 1.2 and 1.3)

7:22 vijaykiran: ^

7:22 Vinzent: oh, ok, i thought it requires 1.3

7:22 vijaykiran: Vinzent & raek: thanks!

7:23 timvisher: hey all

7:23 i find myself pulling out google whenever i want to look for a reference to a namespace

7:24 `doc` is great if i know the function i'm looking up

7:24 `find-doc` is great if i don't, but it doesn't seem to work for namespace prefixes, i.e. `(find-doc "clojure.pprint/*")` doesn't return all the functions in that namespace

7:24 raek: Cozey: kinda OT answer: I like Python 3 because it has a good string model (i.e. strings are sequences of characters rather than bytes, so you don't work with the data in its encoded form)

7:24 timvisher: is there any way to get the references that i find at clojure.org at the repl?

7:25 raek: ...but I've heard that they have fixed the string model in Ruby too

7:25 manutter: timvisher: dunno about the repl, but I use clojuredocs.org for a lot of that kind of stuff

7:26 timvisher: manutter: thanks for the pointer! hadn't seen that before

7:26 raek: timvisher: print-namespace-doc

7:26 timvisher: i'm primarily interested in it because i want access to these docs offline, and i want them to be specific to what the version i'm working with

7:26 Vinzent: timvisher, not exactly what you want, but i use auto-complete which shows all public vars and doc-strings in the ns

7:27 Cozey: raek: will Ruby has similar migrating difficulties as python2->python3 ?

7:27 timvisher: hmm

7:28 raek: that looks like what i want, but calling it like `(print-namespace-doc 'clojure.pprint)` returns clojure.pprint \n nil

7:28 raek: Cozey: from what I have heard, they added the support in a non-major version

7:28 Cozey: mhm

7:28 Vinzent: timvisher, it prints docstring of the ns

7:28 raek: and that sounds scary to me

7:29 timvisher: vinzent: ah

7:29 cemerick: timvisher: there's also http://clojureatlas.com </self-promotion>

7:30 timvisher: cemerick: that looks cool to

7:30 i still want the offline thing.

7:30 raek: I'm so happy that Clojure got the string model thing right...

7:30 timvisher: maybe it just doesn't exist at the moment

7:31 i thought there may be a simple function that they call to generate the online docs

7:31 raek: well, it's mostly Java's effort...

7:31 cemerick: timvisher: you'll be able to download a static copy of your atlas very shortly :-)

7:31 timvisher: but it's probably akin to javadoc

7:31 raek: timvisher: you can clone the gh-pages branch of clojure

7:31 Vinzent: I'm using appengine-magic and have created a protocol and extended it to the appengine entity, but when I'm calling protocol fn on the entity queried from database, implementation for Object is chosen

7:31 raek: that way you get the same html files that are on the site

7:31 Vinzent: example: http://paste.org/pastebin/view/33786 What could be the problem?

7:32 timvisher: raek: that's an interesting option

7:32 Cozey: yeah. i need to code something in python and it is a pain in a way now. but from github i see many clojure people useing ruby as welll

7:32 timvisher: hadn't considered doing that yet

7:35 Vinzent: No ideas? Am I missing something obvious?

7:35 manutter: timvisher: btw, if you poke around clojuredocs.org, there's a downloadable standalone archive. I got it running on my laptop fairly easily (though iirc it's a Rails webapp!)

7:36 raek: Vinzent: maybe you extend the protocol to the wrong type or something...

7:37 also, I'm not very sure how ambiguities are handled (your type is both of the entity type and of the Object type)

7:37 timvisher: manutter: thanks. that's a litle like Apple using those windows mobile devices for awhile in there stores for their mobile point of sale devices in their stores. :)

7:38 manutter: Heh

7:38 timvisher: is there an easier way than including multiple Exceptions in a catch clause than multiple (catch...) forms?

7:38 Vinzent: raek, no, when i'm creating object explicitly it works well, so type can't be wrong...

7:38 timvisher: i.e. a seq of Exception types or something along those lines?

7:38 Vinzent: raek, but every type is an Object!

7:38 timvisher: (catch [FooException IndexException BlahException]...)?

7:38 raek: Vinzent: http://groups.google.com/group/clojure-dev/browse_thread/thread/fb3a0b03bf3ef8ca/d335cce2f83049b3?lnk=gst&q=prefer+mechanism+for+protocols#d335cce2f83049b3

7:39 manutter: timvisher: I guess you could just catch RuntimeException and then check for which subclass the actual exception is?

7:39 timvisher: manutter: mmm. trying to avoid that

7:40 i could always extract the catch function out and then call that from each catch statement

7:40 raek: timvisher: you can with a macro

7:40 manutter: true

7:43 Vinzent: raek, but my entity type is an descendant of the Object; i believe such cases are handled well

7:43 anyway i'll check this

7:55 raek, thanks, you are right: i've removed default implementations and all works fine now; i'm completely confused with that

8:00 raek: Vinzent: with 'extend', you can "mix" the implementation from multiple sources. it expects a map, so you use ordinary functions (assoc etc) to compose the new implementation

8:01 Vinzent: an example https://github.com/clojure/clojure/blob/b578c69d7480f621841ebcafdfa98e33fcb765f6/src/clj/clojure/java/io.clj#L158

8:04 Vinzent: raek, yes, i know that, just thought protocols "know" about inheritance

8:05 so user don't have to extend protocol on his type unless he needs special behaviour

8:06 raek: I know that they were not designed to provide inhertitance, but I cannot present any of the "inheritance is bad" arguments

8:06 well, except for reuse with maps in extend

8:07 all I know is that protocols were never meant to be a class system like that of java

8:13 dnolen: Vinzent: raek: a longish but excellent explication around the issues around inheritance. tl;dr, implementations have many assumptions backed in, inheritance locks you into those assumptions. Better to program against protocols. In Clojure the benefits are obvious, trivial to create things from scratch which interoperate cleanly w/ the entire system - i.e. your own IPersistentSet.

8:13 http://www2.parc.com/csl/groups/sda/projects/oi/towards-talk/transcript.html

8:15 inheritance only gets you convenience, protocols get you much further - scalability and sustainability.

8:35 Vinzent: dnolen, thanks, I'll definitely read it tonight!

8:39 dnolen: Vinzent: it's a good read. Objective-C also takes the inheritance is bad stance + explicit support for protocols. Java interfaces were basically lifted from Objective-C protocols as far as I understand. http://www.virtualschool.edu/objectivec/influenceOnJava.html

8:40 old ideas.

8:57 Vinzent: dnolen, interesting point. I don't know much about obj-c (just heard that it's "right" cpp), but it would be good to learn (at least briefly) other, non-lispy languages to understand clojure better

9:07 btw, can I somehow re-extend protocol dinamically?

9:10 dnolen: Vinzent: if you mean add methods to a protocol, then no.

9:12 Vinzent: dnolen, no, I mean redefining from the repl/recompiling, but looks like the problem is in appengine

9:12 dnolen: Vinzent: oh yeah, definitely.

10:17 amac_: hmm, anyone know how I can use contrib.sql/insert-records and find the auto-inc primary key the record was assigned... querying on the provided columns might bring back multiple rows

10:47 cemerick: OK, continuing on my stream of numerics-related ignorance: bigints are contagious, so (+ 1 XXXN) => XXYN. Why aren't BigDecimals contagious as well? As things stand, (+ 1.0 1e400M) => Infinity.

10:48 It seems like the loss of precision there is undesirable…?

11:08 joegallo: cemerick: 1.0 as a double isn't very precise, though, is it? wouldn't making the result be a bigdecimal mean a magical gain in precision? double + bigdecimal ==> double because you can't manufacture precision out of nowhere. the same doesn't apply for an integer because they are already precise. (caveat: i have no clue what i'm talking about.)

11:16 cemerick: joegallo: fair point, that may be it.

11:18 TimMc: or... maybe it's a bug :-)

11:18 cemerick: Perhaps the "right" thing to do might be to promote as necessary, and then flatten the precision of the result down to the least-precise input you have.

11:19 TimMc: I don't think so; there's never been any discussion of BigDecimal contagion, so I have to assume that that's intended. I figure I'll ask some stupid questions to understand why, though. :-)

11:20 TimMc: joegallo's explanation is pretty convincing from a semantics perspective, though

11:21 Does BigDecimal has a larger exponent range than doubles?

11:21 cemerick: arbitrarily so

11:22 TimMc: I suppose it has to.

11:22 Also, I can't English today. >_<

11:23 pjstadig: yeah i think it is basically what joegallo says

11:23 there's like essentially a subset relationship between the various ints

11:23 every long can also be represented by a big int

11:23 but i guess it's not the same way with floating point, or so i've been lead to understand

11:30 cemerick: pjstadig: NaN and infinity are unrepresentable in BigDecimal (AFAICT), so perhaps that's it.

11:32 pjstadig: there is also a difference between 1.1 and 1.10 in terms of precision (or accuracy i forget?) so upcasting from a double to a BigDecimal may essentially add zeroes onto the end of the base or something

11:33 TimMc: cemerick: Ooof, good point.

11:33 pjstadig: (caveat: i also have no clue what i'm talking about)

11:34 cemerick: pjstadig: Same here, but negatory on that last one:

11:34 ,(java.math.BigDecimal/valueOf 1.1)

11:34 clojurebot: 1.1M

11:34 cemerick: *shrug*

11:35 pjstadig: cemerick: well it prints that way, but how is in represented in memory?

11:35 meh

11:35 it's a problem i don't have :) (having to worry about all this precision stuff)

11:36 cemerick: ,(.precision (java.math.BigDecimal/valueOf 1.1))

11:36 TimMc: ,(java.math.BigDecimal. Double.POSITIVE_INFINITY)

11:36 clojurebot: 2

11:36 java.lang.ClassNotFoundException: Double.POSITIVE_INFINITY

11:36 TimMc: ,(java.math.BigDecimal. Double/POSITIVE_INFINITY)

11:36 cemerick: so, it's not inflating precision or anything

11:36 clojurebot: java.lang.NumberFormatException: Infinite or NaN

11:36 cemerick: yeah, there's no infinity in BD

11:36 TimMc: however

11:36 ,(.doubleValue 1e9999999999999999999999)

11:36 clojurebot: Infinity

11:37 pjstadig: i have on my reading list some paper (that was footnoted in JoC) about how floating points are evil...guess i should read it

11:44 dnolen: ,(/ (BigDecimal. 1.05) 3)

11:44 clojurebot: java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result.

11:45 dnolen: ^ a really good reason for no BigDecimal contagion

11:47 cemerick: dnolen: Good point.

11:48 I guess * and / involving BigDecimals could rationalize them first…

11:48 TimMc: dnolen: Not sure I see it...

11:48 * cemerick is kidding

11:48 TimMc: could pass in a rounding mode.

11:48 / could, that is

11:50 dnolen: TimMc: if it's rounding what's the point?

11:50 TimMc: hm

13:38 amalloy: technomancy: looks like http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8708#10 is fixed in trunk

13:41 technomancy: amalloy: sweet; thanks for following up on that

13:41 amalloy: technomancy: i couldn't bear to have the bug blamed on me, see :)

13:42 technomancy: heh. do you know how long the bug's been around?

13:42 wondering if I should drop the check or if I should assume for a while that all 24 users are affected

13:42 amalloy: technomancy: uh. well, i reported it last week. if you mean, how long it's been broken, i don't know

13:42 but git blame ought to reveal it pretty quickly

13:43 edw: Yay, amalloy and technomancy! Thanks for looking into this.

14:33 coopernurse: trying to get my clojure/emacs/slime setup going on Aquaemacs. Installed clojure-mode via marmelade, but when I restart emacs the mode isn't available. do I need to add anything to my .emacs file?

14:33 I'm a little confused where the .el files were installed to

14:35 technomancy: it should be in ~/.emacs.d/elpa, but in general forks of GNU Emacs are not supported.

14:35 * technomancy recommends http://emacsformacosx.com

14:35 coopernurse: ah, ok thanks

14:35 I see stuff here: /Users/james/Library/Application Support/Aquamacs Emacs/elpa

14:36 but it looks like they're not being loaded when Aquamacs starts

14:36 I'll try the other emacs distro

14:37 desertman: sorry i seem to have a connection problem

14:41 i don't know if i already posted this or not because of a connection problem but i am trying to undertstand contrib.sql http://richhickey.github.com/clojure-contrib/sql-api.html , ie (do-commands "show databases") returns a clojure.lang.ArraySeq$ArraySeq_int, I would like to get a list of the databases.

14:42 ordnungswidrig: *grr* is there any sensible imap mail client on osx. thunderbird eats my cpu

14:43 drewr: people like sparrowapp.com

14:43 raek: desertman: clojure.lang.ArraySeq *is* a seq. are you sure you're not using (a potentially implicit) str instead of some of the printing functions? (pr prn pr-str prn-str)

14:43 drewr: I find it doesn't support gmail's imap well

14:44 ordnungswidrig: drewr: I have a *real* imap server :)

14:44 raek: ,(pr-str (seq (to-array [1 2 3])))

14:44 clojurebot: "(1 2 3)"

14:44 raek: ,(str (seq (to-array [1 2 3])))

14:44 clojurebot: "(1 2 3)"

14:44 raek: hrm

14:44 ,(str (lazy-seq [1 2 3]))

14:45 clojurebot: "clojure.lang.LazySeq@7861"

14:45 drewr: ordnungswidrig: you could try it then

14:45 ordnungswidrig: drewr: I will. thanks for the point

14:45 coopernurse: I was missing: (package-initialize)

14:45 ordnungswidrig: I tried of measure the number of retries of a dosync block by incrementing a atom

14:45 coopernurse: in .emacs.. I'm a ELPA n00b

14:46 ordnungswidrig: But I do not really trust the numbers

14:46 manutter: ,(apply str (lazy-seq [1 2 3]))

14:46 clojurebot: "123"

14:46 manutter: hrm, what was that other command now...

14:46 raek: ,(apply str (lazy-seq [(lazy-seq [1]) (lazy-seq [2]) (lazy-seq [3])]))

14:46 clojurebot: "clojure.lang.LazySeq@20clojure.lang.LazySeq@21clojure.lang.LazySeq@22"

14:46 desertman: (type (do-commands "show databases")) returns clojure.lang.ArraySeq$ArraySeq_int , which is an array of integers

14:47 raek: desertman: not actually an array, but an array seq

14:47 ordnungswidrig: using pcalls to start 100 parallel executions on 2 refs on a 2 core machine should give some retries, shouldn't it?

14:47 raek: ,(seq (to-array [1 2 3]))

14:47 clojurebot: (1 2 3)

14:47 raek: ,(type (seq (to-array [1 2 3])))

14:47 clojurebot: clojure.lang.ArraySeq

14:48 raek: desertman: it works and prints like an ordinary sequence

14:48 manutter: desertman: ah, yes, you're getting back the results of executing a sql command

14:48 desertman: do-commands is intended for non-select sql commands like drops and inserts

14:49 desertman: You're probably getting back a 1 element array containing the number of rows returned by the "show databases" command

14:50 desertman: yes

14:51 manutter: If you want to see the actual database names, you need (with-query-results results ["show databases"] (doseq [row results] (println row)))

14:52 or something like that

14:53 desertman: ok

14:55 <manutter> - yes, that works, thank you

14:57 manutter: :)

15:00 desertman: <manutter> although the syntax is a bit strange to me, , because it seems that '(doseq [row results] (println row))' is being passed as an argument to with-query-results results ?

15:01 manutter: It's the body of the with-query-results macro -- wraps your code in some setup/cleanup stuff

15:03 ,(doc clojure-contrib.sql/with-query-results)

15:03 clojurebot: Cool story bro.

15:03 desertman: ok, i wouldn't have seen that in the docs, but i guess you need to look at the source to know what these things do, although i'm not there yet

15:04 coopernurse: Ok, got my emacs/slime setup working. thanks for the tip on the other emacs distro

15:04 I never cared for all the extra junk in aquamacs anyhow

15:05 technomancy: "training wheels on a humvee" is one description I've heard =)

15:05 manutter: desertman: there's a few other macros that do similar things, like with-out-str etc

15:06 coopernurse: technomancy: heh.. after years of Java development, this clojure/repl/slime setup is pretty interesting. love reloading code in the repl without restarting the vm

15:07 I'm going through some project euler problems and realizing I know _nothing_ about FP :-)

15:07 my solutions are 10x too long

15:10 * technomancy mumbles something cryptic about "mind like water"

15:11 manutter: ,(doc clojure-contrib.sql/with-query-results)

15:11 clojurebot: Cool story bro.

15:12 manutter: no, doh, that's what I typed last time

15:12 ,(doc clojure.contrib.sql/with-query-results)

15:12 clojurebot: "([results sql-params & body]); Executes a query, then evaluates body with results bound to a seq of the results. sql-params is a vector containing a string providing the (optionally parameterized) SQL query followed by values for any parameters."

15:12 manutter: there we go

16:03 markoman: hi. is there some standard way to formalize strings to ascii form? I need this for searching string from datastore

16:04 im thinking something like: Caté1 -> cate1

16:07 ampleyfly: you want cate1 and Caté1 to collide, then?

16:07 coopernurse: markoman: stackoverflow thread that is relevant to what you want to do

16:07 http://stackoverflow.com/questions/5651124/is-there-a-way-to-dumb-down-text-from-unicode-to-ascii

16:07 not clojure specific..

16:08 but it seems like your first challenge is to make a mapping, since ascii can't represent all unicode characters (obviously), so it's not clear what heuristic you want to use

16:08 TimMc: markoman: What should بينظير ڀٽو become?

16:08 coopernurse: TimMc: right

16:09 markoman: good question, i have no idea...

16:09 hiredman: you render each string as set of paths and then find best fits

16:09 markoman: and is there a upper and lower case version of بينظير ڀٽو?

16:09 TimMc: I work at a company that makes transliteration software, but I don't happen to know the open source answer to this.

16:09 hiredman: good luck

16:16 markoman: there is this example in java: String unicode = "Unicode: \u30e6\u30eb\u30b3\u30fc\u30c9"; byte[] bytes = unicode.getBytes("US-ASCII"); how could I test it on clojure repl?

16:17 TimMc: markoman: That will just get you munged data.

16:18 raek: ,(String. (.getBytes "Unicode: \u30e6\u30eb\u30b3\u30fc\u30c9" "US-ASCII") "US-ASCII")

16:18 clojurebot: "Unicode: ?????"

16:19 amalloy: better than Unrecognizable Rectangles

16:19 raek: the default behaviour seems to be to replace unrepresentable characters with \?

16:20 markoman: yeah, it replace nontransable chars to question marks, not good, but some start

16:21 amalloy: markoman: it is a start down the path of evil. unicode is a much larger set than ascii; trying to downconvert will bring only pain

16:21 raek: amalloy: tell that to "Mr. Yurukôdo"

16:21 ahem. "Yurikôdo", i.e. "Unicode"

16:21 * amalloy doesn't really get it

16:23 raek: I think that's it not very useful for the Japanese (the string contains Katakana) if their text is replaced with questionmarks

16:23 markoman: thats true

16:23 amalloy: that's what i said, right? don't convert unicode to ascii?

16:23 markoman: well then is string to upper and lower case smart enough, it works with unicode?

16:24 raek: amalloy: oh, sorry. I accidentally read "better than Unrecognizable Rectangles" as a comment on "Unicode: ?????"

16:24 amalloy: markoman: insofar as such a thing is possible, yes, it works

16:24 raek: yeah, sorry. better, but not actually good

16:25 "Unicode: ?????" neatly sums up the developer's feelings: "I clearly don't understand Unicode"

16:25 raek: :)

16:25 markoman: I could also try to make a map, as suggested, but for special punctuations only, partial solution

16:25 TimMc: markoman: http://gizmodo.com/382026/a-cellphones-missing-dot-kills-two-people-puts-three-more-in-jail

16:26 raek: markoman: if you only consider the latin alphabet, maybe you could normalize the string to NFKD form (Normalization Form Compatibility Decomposition) and strip all the combining characters

16:26 amalloy: TimMc: that too. but java's toLowercase gets that right, doesn't it?

16:27 TimMc: amalloy: What does "right" mean?

16:27 You have to know the language of origin.

16:27 amalloy: converts uppercase dotted-i to lowercase dotted-i, and likewise for dotless?

16:27 raek: markoman: this can problably do something useful: http://site.icu-project.org/

16:28 amalloy: i guess not in all cases

16:28 i see what you mean about language of origin

16:28 markoman: basically, if I use lower case to this: بينظير ڀٽو does the meaning change totally different?

16:28 raek: it basically knows the stuff about the letters that human know

16:28 markoman: it would be a problem for getting right results with searches...

16:29 raek: markoman: it won't change it, since the Arabic script does not have the lower/upper case distinction

16:29 ,(.toLowerCase "بينظير ڀٽو")

16:29 clojurebot: "بينظير ڀٽو"

16:29 TimMc: markoman: Are you beginning to see why there are are so few search engines? :-)

16:29 amalloy: TimMc: b/c for turkish lowercase of I is dotless, but for english the lowercase is dotted? that's the point you're making, right?

16:30 TimMc: amalloy: Indeed.

16:30 * amalloy is glad not to deal with this stuff

16:30 raek: ,(let [s "بينظير ڀٽو"] (= s (.toLowerCase s) (.toUpperCase s)))

16:30 clojurebot: true

16:30 pdk: turkish has two separate letters for I

16:30 coopernurse: different subject - anyone out there running clojure on a VPS with limited ram?

16:30 pdk: one has a dot on both lower/upper cases, one has a dot on neither

16:31 amalloy: coopernurse: for given values of "limited"

16:31 coopernurse: I'm interested in how small you were able to get the JVM - if you tried..

16:31 I was playing with compojure on a rackspace cloud 512mb instance and it worked fine

16:31 jvm rss was about 125mb

16:31 with hello world loaded

16:32 I was curious if anyone has tried to get that smaller.. I had start/max heap set to 32mb

16:32 hiredman: on osx I can't seem to get the rss of the jvm below 80mb no matter what settings I use

16:32 coopernurse: hiredman: yeah, that's about what I've seen too

16:32 hiredman: just fiddling around with it

16:32 markoman: simple things come easily complicated in this digital world

16:33 TimMc: markoman: You're trying to make a program that can deal with any language. :-P

16:33 markoman: but hey thanks, I got something to chew

16:33 TimMc: Rather... it turns out that search engines are not language-agnostic.

16:34 markoman: nah, Im finding limits and making compromises then :)

16:35 TimMc: I wish I could share some of the crazy stuff we're dealing with in our own software, but I'm not really sure what's confidential and what isn't.

16:36 markoman: *promises not to copy and paste everything on twitter*

16:37 I get gray hair even with basic character encodings

16:37 amalloy: haha i'd like to see someone try to distribute their whole codebase in tweet-sized chunks

16:37 TimMc: public static void main(String[] args) { #loloutofspace

16:41 raek: markoman: ##(apply str (remove #(= (Character/getType %) Character/NON_SPACING_MARK) (java.text.Normalizer/normalize "åäö" java.text.Normalizer$Form/NFKD)))

16:41 sexpbot: ⟹ "aao"

16:44 markoman: hmh... on intelliJ repl it gives questionmarks, need to test lein repl

16:45 TimMc: markoman: Make sure to have rlwrap installed -- it handles Unicode printing.

16:46 raek: markoman: install rlwrap if you don't have it. jline, which is the default, is broken and does not have support for UTF-8

16:46 markoman: thats clojure lib, or lein extension or java lib?

16:46 raek: markoman: OS package

16:47 then the test to confirm everything works: ##(seq "åäö") ;should return *three* characters

16:47 sexpbot: ⟹ (\å \ä \ö)

16:49 markoman: ok, found from mac port

16:52 cemerick: hiredman, coopernurse: I'll bet that mapped byte buffers are being used for your bootclasspath, so a very large chunk of that 80MB is rt.jar and friends.

16:52 technomancy: so... how hard would it be to create a lazy seq from which items could be dropped without calculating them?

16:53 coopernurse: cemerick: interesting. I'll google "mapped byte buffers" and learn something

16:53 hiredman: technomancy: very

16:53 technomancy: I'm thinking if drop rebound some kind of var, the lazy seq construction could check it and simply skip the work where possible if it knew the results could be discarded.

16:53 hiredman: (defprotocol internal-drop ...)

16:54 technomancy: seems easier to just have a lazy-seq of delays

16:54 technomancy: hiredman: did we have this conversation already? I have a vague idea I once determined it'd be harder than it sounds, but I don't recall why.

16:55 certainly easier, but less elegant.

16:55 cemerick: coopernurse: Looking at URLClassLoader, it's loading class byte[]s from ByteBuffers, and there's a comment that makes me think they're direct 99% of the time (would certainly make sense if the jars are on disk).

16:55 hiredman: technomancy: yeah, I vaguely recall that, but don't recall specifics

16:57 markoman: hmh, i still get questionmarks after installing rlwrap and restarting apps

16:57 hiredman: ~google java file encoding

16:57 clojurebot: First, out of 323000 results is:

16:57 utf 8 - Setting the default Java character encoding? - Stack Overflow

16:57 http://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding

16:57 coopernurse: cemerick: ah. so does that mean I'm seeing disk blocks that are mmap'd in the RSS figure?

16:57 markoman: is there something else I need to include when repl starts up?

16:57 coopernurse: I have a tenuous grasp on how mmap works

16:59 cemerick: coopernurse: Perhaps. And, if those jars are being mmap'ed from Java, then the corresponding memory allocation happens outside of the heap.

16:59 Thus, the disconnect between -Xmx and such and rss.

16:59 coopernurse: cemerick: ah, cool. I always wondered why that difference was so huge

16:59 looks like -XX:MaxDirectMemorySize may be a way to tune that?

17:01 cemerick: coopernurse: I'll bet that's a hard limit that will cause an exception to be thrown when the classloader attempts to mmap the files, not prompt it to fall back to a non-mmap code path.

17:01 coopernurse: which option reference are you looking at? I don't see that one on http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

17:01 coopernurse: cemerick: interesting. and it appears the default is 64MB

17:02 cemerick: http://download.oracle.com/docs/cd/E15289_01/doc.40/e15062/optionxx.htm

17:02 cemerick: oops, that's jrockit..

17:04 cermerick: normal java -X doesn't show it.. oh well. but I suspect you've explained why you can't really get a JVM down below 100MB RSS in practice.

17:05 cemerick: coopernurse: there's a *lot* of options that aren't listed on the page I linked. There's a more comprehensive reference, I just can't find it at the moment.

17:06 markoman: when repl starts on intelliJ it says "/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Dfile.encoding=UTF-8 " but still I get question marks when printing scandic letters

17:07 offby1: markoman: might have to do with your terminal program.

17:07 Or your choice of font.

17:09 coopernurse: cemerick: I see. my goal is to see how many JVMs you could reasonably fit in a given amount of RAM for small-ish webapps. My preference would be to use a separate JVM per app so I can restart them separately. I realize that's not Java's preference, but it would make the Compojure web apps more similar to Ruby/Python apps from an ops standpoint

17:10 cemerick: hot swapping the WAR files is another option, but my experience with that has been less than stellar (run out of PermGen due to classloader leaks, etc)

17:12 cemerick: coopernurse: undeploy/redeploy is the good middle ground; you could probably run a dozen or more small webapps in 128MB of heap, and deploy them independently.

17:12 coopernurse: cemerick: undeploy/redploy - is that a jetty feature?

17:13 cemerick: coopernurse: any (worthwhile) container supports undeploying an app while leaving the container and other apps undisturbed

17:13 jetty, tomcat, glassfish are all suitable

17:13 S11001001: except for the permgen thing

17:14 cemerick: turn on class GC and you should be fine

17:14 or, use a VM that doesn't have permgen (jrockit is now free)

17:16 coopernurse: I'm not saying you can have heroku-in-a-box on this route, but it certainly works for handfuls of apps. 1:1 app:container setups are never going to be efficient in any way (unless there's a magic config parameter that we've not discovered that will eliminate the rss issue).

17:17 coopernurse: cemerick: agreed. the JVM seems to favor long running processes. redeploying seems like a decent route -- I guess I've just been bit by that in the past. perhaps I should take a look at jrockit

17:18 cemerick: coopernurse: some containers (weblogic, IIRC?) have previously offered a "hot redeploy" option that, yes, have never worked as advertised. Doing discrete undeploy + deploy operations are a whole different (simpler) approach, though, and "just work".

17:19 markoman: yeah, on terminal i needed to do: export JAVA_OPTS="-Dfile.encoding=utf-8"

17:19 cemerick: Also, the shorter your uptime is, the less you'll get JITed code.

17:19 coopernurse: cemerick: interesting, I would have figured hot deploy does the same thing as undeploy/redeploy - or is the latter just not atomic, and therefore more reliable?

17:20 markoman: but intelliJ is harder to get functioning on repl, need to test it on clj files tomorrow, sometimes they still work, just repl not

17:21 thanks guys, more next time

17:22 cemerick: coopernurse: exactly, those are two operations that you run / script as part of your "redeploy" process. Things marketed as "hot redeploy" try to be tricky, probably attempting to do things like what JavaRebel et al. do.

17:23 (and, JavaRebel works well when it can — but there are always changes that cause it to fall over and force a full rebuild / deploy cycle)

18:34 ssideris: hello, has anyone seen the "make-dependency does not exist" error with leiningen native-deps 1.0.5?

18:35 it seems that native-deps is a bit sensitive to the version of leiningen or something...

18:36 technomancy: ssideris: what lein version?

18:36 ssideris: 1.5.0

18:36 (on windows if that's relevant)

18:36 technomancy: odd; that should work.

18:36 if you use lein 1.6.0-SNAPSHOT you don't need a plugin for native-deps

18:37 ssideris: oh, ok let me give it a go

18:37 thanks

18:37 and thanks for leiningen btw :-)

18:37 technomancy: sure =)

18:39 having make-dependency move was a long time ago though; 1.0.5 shouldn't be affected by that

18:43 ssideris: sorry just a clarification: does that mean that lein 1.6.0-SNAPSHOT does not differentiate between native and non-native deps?

18:43 do I put everything under :dependencies?

18:44 technomancy: ssideris: not yet; so far it works the same as the plugin

18:45 except you don't need a separate "lein native-deps" step

18:45 but you do need them in :native-deps in project.clj

18:45 hopefully by the time it's released you can put them in :dependencies though.

18:46 ssideris: hm, that's odd, there must be something really weird with my setup because it didn't seem to pick up :native-dependencies\

18:46 it's being ignored...

18:47 technomancy: hmm... it's very new code; not well-tested yet. could you open an issue?

18:47 ssideris: ok

18:49 technomancy: what's the dependency?

18:50 ssideris: [penumbra/lwjgl "2.4.2"]

18:54 technomancy: https://github.com/technomancy/leiningen/issues/207

18:54 opened!

18:54 thanks

19:47 technomancy: ssideris: if you can take a look it would help to get it fixed faster; I don't have any experience with native deps and have a few search features I need to finish first

19:47 but I'll get to it before the release if not

19:50 ssideris: technomancy: erm, wouldn't know where to start, but if you can direct me a bit I would be happy to give it a go

19:51 technomancy: here's the function in question: https://github.com/technomancy/leiningen/blob/master/src/leiningen/deps.clj#L144

19:51 pretty much taken verbatim from the plugin

19:53 ssideris: ok, I'll see if I find anything, but don't put too much faith in my clojure abilities :-)

21:16 mec: Is there anywhere that has the syntax for literal defrecords in alpha 7?

21:17 amalloy: mec: it's just #user.Foo[1 2] or #user.Foo{:a 1 :b 2}

21:18 ataggart: http://dev.clojure.org/display/design/defrecord+improvements

21:20 and it's not just for records, but a generic constructor literal.

21:31 mec: ataggart: so its good for any java class?

21:38 ataggart: yes, e.g. #java.util.Date[12345]

21:38 bear in mind that this is really intended for print-dup, not something you should be typing out.

22:12 WuHoUnited: hello clojurebot

22:14 (doc clojurebot)

22:14 clojurebot: It's greek to me.

22:14 WuHoUnited: (source clojurebot)

22:15 amalloy: clojurebot: source?

22:15 clojurebot: source is http://github.com/hiredman/clojurebot/tree/master

22:15 amalloy: WuHoUnited: ^

22:15 WuHoUnited: ^_^

22:15 amalloy: $whatis source

22:15 sexpbot: source is http://github.com/cognitivedissonance/sexpbot

22:15 amalloy: (also that if you're interested)

Logging service provided by n01se.net