#clojure log - Aug 09 2008

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

7:05 Chouser: kotarak: chimp works as advertised.

7:05 kotarak: I'll have to let it settle in a bit before I can provide any useful opinions.

7:53 kotarak: Chouser: thanks for testing :)

12:34 arohner: has anyone tried alternatives to an RDBS in clojure?

12:34 I'm working on a project using lots of bayes statistics, and using SQL to do my queries looks like it will be very messy

12:35 I would really love a DB I can query using lisp, but it seems like postgres/mysql is the only game in town

12:44 I guess I don't need an alternative to RDBS, I want an alternative to SQL

12:55 shoover1: I've been wondering how Berkeley DB would work as a key/value store for Clojure. There is a pure Java implementation, and supposedly it's supposed to work well with JavaBeans.

12:59 shoover: I'm just thinking out loud, as I don't know of an existing implementation for Clojure. But in this system I imagine being able to query using your keys, Clojure fns, or list comprehensions.

13:04 arohner: interesting

13:04 the problem is, my data actually fits in an RDBS well

13:05 lots of user sessions, each session is a bunch of questions

13:05 I just don't want to use SQL to get to it

13:06 the fact that SQL is not composable is very irritating

13:07 moxley: There's a lightweight persistence framework inside Compojure-- persist.clj

13:09 arohner: hmm..

13:09 then I would need to write the code to add an index to a set of rows

13:15 moxley: yeah, it doesn't help much there

13:17 arohner: I'm also slightly worried about what happens when the dataset becomes larger than RAM

13:22 joubert: I've used BDB as a datastore from Clojure

13:24 shoover: joubert: Oh?! What'd you think?

13:26 joubert: well, the way I persist values is to encode the data (structure) to the UTF-8 stream of bytes that make up the result of its (pr-str)

13:26 Then I decode doing the reverse

13:26 e.g.:

13:26 (defn decode-data-structure-from-persistence [persisted-representation]

13:26 (let [string-representation (new String persisted-representation *utf-8*)]

13:26 (read (new java.io.PushbackReader (new java.io.StringReader string-representation)))))

13:26 (defn encode-data-structure-for-persistence [data-structure]

13:26 (let [string-representation (pr-str data-structure)]

13:26 (. string-representation (getBytes *utf-8*))))

13:26 the limitation is that I cannot persist seqs

13:27 I can share some code if interested

13:27 shoover: Please paste that snippet into http://paste.lisp.org/new

13:28 joubert: the encode and decode?

13:28 shoover: yes, the functions above

13:29 lisppaste8: joubert pasted "Encode / Decode Clojure data for persistence (e.g. to BDB)" at http://paste.lisp.org/display/65076

13:31 joubert: I should also have shown: (def *utf-8* "UTF-8")

13:31 shoover: That's cool. And then you use Clojure/Java interop for creating and querying your db?

13:32 joubert: yeah, I have functions to put and get values, as well as the plumbing you need to setup the BDB environment & database

13:32 I also have support for keys

13:32 that allow referential logic

13:32 (BDB calls them SecondaryKeys)

13:36 shoover: So that would work for all of Clojure's readable data structures, but seqs you would have to evaluate to a list or vector first

13:36 joubert: right

13:36 maybe I should post my code to the clojure group files area

13:37 it is only about 170 lines, and would obviate a lot of grunt code you'd need to write

13:38 shoover: Yes, that or submit to clojure-contrib, as long as you're willing to license it appropriately

13:38 joubert: currently GPL 3, but I guess most clojure things are CPL?

13:38 I'm happy to change to CPL

13:39 shoover: clojure-contrib is CPL. I'm not sure, but I think things submitted to the group are automatically CPL

13:40 rhickey: joubert: to contribute to clojure-contrib you'll need to submit a CA: http://clojure.org/contributing

13:41 joubert: OK

13:41 arohner_: rich: is there any way to view the change history to the docs on clojure.org?

13:42 there have been several groups posts where someone asks you to change the docs, and you reply "done". I don't see a good way to find out what changed.

13:42 rhickey: arohner_: not right now, there might be a way with the wiki software...

13:51 arohner_: http://clojure.org/space/changes

13:52 arohner_: cool. thanks!

14:32 drewr: arohner_: You could look into an object persistence solution, like Terracotta.

14:34 arohner_: I'll check it out. Thanks

14:36 that's interesting, though not quite what I'm looking for

14:37 my biggest headache is SQL

14:38 the queries I'm writing are ugly, and I can't help but think it could be easier using map, filter and reduce

14:39 but then I want to be able to declare indexes on my dataset, and have the library handle moving stuff out of RAM when not necessary

14:39 basically a normal RDBMS that I can query using lisp

14:40 rhickey: arohner_: there is: http://clojure.org/api#index

14:42 arohner_: how do I call it?

14:42 index

14:42 java.lang.Exception: Unable to resolve symbol: index in this context

14:43 ah, nm

14:43 rhickey: it's in the clojre.set namespace

14:43 clojure.set

14:48 arohner_: I don't quite understand what it's supposed to do from the docstring

14:52 oh look, a helpful example at the bottom of set.clj :-)

15:07 all of the subtypes of map have the same literal syntax, {}, right?

15:07 so to find out what type of map you have, you should do (class map)?

15:07 rhickey: arohner_: right now, yes

16:28 jamii: rhickey: It looks like stream fusion might be really effective combined with seq. You couldnt do it statically because clojure is late binding, but maybe hotspot could be persuaded to use it?

16:33 rhickey: jamii: I have no idea if optimizations like that are present in hotspot

16:36 jamii: I wouldnt expect them to be useful in java. Its a simple compiler optmisations in an early binding language. It would work really well with clojures seqs but it would require some ability to hot swap code. I wonder if hotspot exposes any of its functionality.

Logging service provided by n01se.net