#clojure log - Jan 08 2017

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

6:34 lxsameer: is it ok to have a atom containing a hashmap which each value is an atom itself ?

6:35 dysfun: sure it's okay. but what are you trying to achieve?

6:37 lxsameer: dysfun: I'm polling some data from a remote service, I want to keep track of data changes, the data is a huge map, and i want to update the changed values only

6:38 dysfun: i don't understand the problem with using one atom

6:39 is it a concurrency problem?

6:39 lxsameer: dysfun: i wanted to create an atom for each key and set a validation and a watcher so validation only allows new values to be set, and watchers trigger the behavior which i needed

6:40 dysfun: using one big hashmap in an atom, i can't find out what key changed

6:40 dysfun: how would you design such thing, I need it to be scalable

6:40 dysfun: sorry, what sort of data will these be?

6:41 are you doing HFT or something?

6:41 lxsameer: dysfun: the data is a huge json containing stock market information

6:42 dysfun: okay. so then you process that and update your atom of atoms according to the new values?

6:42 lxsameer: dysfun: i had this in mind, but I'm open to new ideas

6:43 specially my atom of atoms seems not scalable

6:43 dysfun: okay. so each piece of data is like {:symbol 'FOO" :price 1.23 ...} ?

6:43 that sort of thing?

6:44 lxsameer: dysfun: yeah right

6:45 dysfun: i was thinking about creating a queue using kafka or something

6:45 dysfun: now you have two problems ;)

6:45 lxsameer: and put the values in the queue and have them process by nodes using core.async

6:45 dysfun: why ?

6:45 dysfun: well, message queuing theory says exactly once delivery is hard

6:46 so now we've got to check timestamps and such

6:46 lxsameer: dysfun: aha

6:46 dysfun: so what's you solution about this ?

6:47 dysfun: pick a database designed for storing exactly this sort of data. like riak

6:47 you can blissfully configure it in last-write-wins mode because that's exactly what you want

6:47 or, if you don't actually need storage, shove it in a megaref

6:48 lxsameer: dysfun: megaref ? is it a clojure thing ?

6:48 dysfun: it's a library

6:48 https://github.com/cgrand/megaref

6:48 lxsameer: dysfun: thanks man, i have to read about it, also the riak solution is cool too

6:49 dysfun: cool, have fun

7:03 osfameron: lxsameer: dysfun: kafka is pretty good as a queue

7:04 lxsameer: osfameron: i see

7:04 dysfun: sure, it is

7:04 but you have to know how it fails too

7:05 osfameron: ah - we haven't come across particular failure modes - do tell more

7:05 dysfun: at-least-once != exactly-once will be the first one you'll hit

7:06 osfameron: ah yes. we're making sure all of our consumers are idempotent, which must be as mitigation of that

7:06 I've not followed all the details

7:06 dysfun: someone gets it

7:07 osfameron: but yeah, that's easy enouhg for us to do with our particular use of it as a message bus, but might not work for other situations

7:08 dysfun: yeah. each situation needs it factoring in differently

8:58 hiro`: Anyone here read Web Development in Clojure? I've seen from the beta-version (which is floating around online) and the author's blog that he was using Yesql, but Luminus no longer uses it. Did the book get updated to use the Luminus stack (inc HugSQL, Yesql's replacement), does anyone know

8:58 ?

8:58 dysfun: that's a fairly niche question, suspect you'd be better asking in the work week

8:58 maybe you could check the book's errata online?

8:59 hiro`: Had a quick scan through that already, couldn't find any reference to it.

9:01 dysfun: i guess probably not then

9:01 shouldn't be too difficult to get up to date with hugsql though

9:01 avicenna: web development with clojure version 2 uses hugsql as seen in the pragprog website

9:01 dysfun: ah, there we go

9:01 hiro`: ah, that's what I was wondering.

9:01 thanks avicenna

10:36 machinewar: any convention for naming clojure tests with deftest. deftest fn-name-test? perhaps

10:36 dysfun: that's common

10:37 test-fn-name also

10:37 just pick a sane one and roll with it

10:37 * dysfun is going to try and build a JDK9 today

12:04 moncrey: hey room, looking for a little insight into a missing namespace in a project.... the code runs fine but i cant for the life of me figure out where this library is coming from....

12:04 line 9 of this file https://github.com/overtone/overtone/blob/master/src/overtone/sc/machinery/server/native.clj

12:05 points to a library that does not exist within that repository, or anywhere else within the organization's repositories.

12:06 not assuming anyone will have the answer but ANY insight would help. so confused.

12:06 if the library did not exist, the project would not run, correct?

12:07 or at least, if it hit that required method, woudnt it throw errors??

12:36 justin_smith: moncrey: require looks for a specific artifact inside a package, there'

12:36 s no rule saying that the name of the artifact require finds has any relationship to the artifact name

12:37 scriptor: I think they left

12:38 justin_smith: hmm

12:38 rhg135: So it can load runtime ns given it exists before hand?

12:41 justin_smith: well the ns won't compile unless it can be found - but the point I was making was that there is no relationship required between the name of the ns that require looks for, and the name of any artifact you use

12:44 rhg135: I see, so a file foo/bar.clj can have ns bar.baz and you require it as foo.bar?

12:44 justin_smith: rhg135: more like a dependency org.foo/bar can have a file src/baz/quux.clj and you can require it via baz.quux

12:45 but the code referencing baz.quux doesn't help you find foo/bar

12:45 it does appear something is funky here though? https://github.com/search?q=org%3Aovertone+nativescsynth&type=Code

12:46 scriptor: my guess is that it's from this dependency https://github.com/overtone/overtone/blob/master/project.clj#L62

12:46 can't find a github of that anywhere

12:47 justin_smith: based on the name, I would assume it is the piece of code that tells you which engine is actually present

12:47 since overtone uses a big c++ program for all the synthesis, and it needs a different version for every OS configuration

12:49 it would actually make sense for that data to be provided by the dependency itself (which wouldn't be part of the overtone github, it's a compiled binary)

12:53 FOUND IT [overtone/scsynth ""] https://clojars.org/overtone/scsynth/versions/

12:53 it's the only clj file in a big blob of native code blobs

12:56 that ns in one line: (ns overtone.nativescsynth.availability) (def native-scsynth-lib-availability {:windows {64 false 32 true} :linux {64 true 32 false} :mac {64 true 32 true}})

13:28 machinewar: going to post this function I'm working on in case anyone feels like helping refactor and make it more idiomatic clojure. https://gist.github.com/AlexWheeler/7d6a97978e51a036367de42936da8fae#file-clj-clj-L7. comments appreciated

13:31 osfameron: machinewar: body of copy-nil-vals could just be: {k (or v k)}

13:32 hmm, that's slightly different semantics, would also overwrite false

13:32 {k (if (nil? v) k v)} I suppose

13:33 why do you need to (apply merge) twice?

13:34 ridcully: machinewar: could you please also through in some small data example? e.g. going from [{:value 42 :mapped_value 666}] -> {42 666} ?

13:34 machinewar: yep

13:36 ridcully: my gutfeeling is, that you take a lot of "reshaping" steps in between, where something like (into {} (map (juxt :value :mappeD_value)) columns) brings you already close

13:37 throw in your nil handling around that juxt

13:37 machinewar: yep that's what I'm thinking. this was my intuitive way of doing it step by step, but figure there's a better way. Cool I'll play a bit with that thanks a lot

13:38 osfameron: oh, the juxt/into thing is nice ridcully!

13:39 I'd have approached it with a dumb (fn [{keys [:value :mapped_value]} ...)

13:39 * osfameron makes a mental note for next time

13:39 ridcully: well you have to deal with that nil check there still. so destructuring instead of the juxt is fine

17:18 osfameron: oh dear, I got lost in http://prog21.dadgum.com/ again

19:48 Vincentdm: Hi, I'm new and trying to understand multimethods. I would like to define (defmulti) in one file (namespace), and then use separate files for implementations (defmethod). However, this seems to require a circular dependency.

19:49 I thought the whole point of such dispatching was that code could require only the namespace that contains (defmulti), and it would call the right implementation automatically.

19:53 justin_smith: Vincentdm: why would that require a circular dependency?

19:53 the file that defines a multimethod doesn't need to know about the implementations

19:54 and no, only requiring the file with the defmulti is not the point, at all

19:54 you only need to require the implementations in one place, and every other location can just use the defmulti though

19:59 Vincentdm: Hi Justin, seems you are right. My code was faulty (I copt-pasted a ::keyword when moving the defmethod), which caused the dispatching to stop working. This lead me to believe it didn't "find" the defmethod implementation

20:01 So when calling a multimethod, Clojure will basically consider defmethods in whatever file, as long as it is included in the project dependencies?

20:01 justin_smith: Vincentdm: no

20:01 Vincentdm: the namespace with the defmethod needs to get loaded (usually via require)

20:01 it can be loaded anywhere, but it needs to be explicitly loaded, there is no implicit loading of code

20:02 Vincentdm: ok, good to know that

20:02 So that means that I cannot add a new implementation of a multimethod, without explicitly referring to it from a more "general" place (which is actually not supposed to know this specific implementation exists)

20:02 ?

20:06 justin_smith: Vincentdm: define "supposed"

20:06 Vincentdm: the general pattern is you have a bunch of code that refers to the multimethod, and then one place that loads all the defmultis

20:06 usually a top level namespace that loads your config and starts everything up will also load up all those implementations

20:08 err, make that "all the defmethods"

20:08 Vincentdm: Well, I need to work with API's of different vendors, and I was hoping that I could add implementations of new APIs just by adding a new file containing defmethods. But based on your answer I understand that I will also need some central place loading all the implementations

20:08 justin_smith: sure, you need a top level namespace that controls that stuff, clojure avoids magic

20:08 but I misspoke above - you refer to the defmulti everywhere, and only load the files with defmethods in one place (usually your top level ns that also sets up your config)

20:10 Vincentdm: Ok, that's clear. As I understand it now multimethods are more about providing a clean syntax (so you don't need to do if/else/.... to dispatch incoming data to the appropriate functions), but it is not really a solution for reducing dependencies.

20:10 Since I see no real difference between referring to my implementations from within the NS that does the defmulti compared to some other central namespace.

20:10 justin_smith: Vincentdm: yeah, we don't do implicit DI

20:10 it's just not a thing clojure does

20:11 Vincentdm: Alright. Thanks for clarifying that!

20:11 justin_smith: but all that requires is a single require

20:12 Vincentdm: for clarity, I usually find it useful to special case my top level namespace

20:12 Vincentdm: What do you mean with "special case"?

20:12 justin_smith: that is to say, that namespace breaks some rules the rest of my namespaces follow, because it has to configure everything else and decide how startup happens and what resources we use etc.

20:13 Vincentdm: ok, right

20:13 justin_smith: it doesn't do "just one thing" - it puts together all the pieces to make everything run (but ideally it does this without also implementing program logic)

20:14 so that would be the namespace that injects clojure dependencies (since unlike classes they are not injected by just having them on the classpath and referring to something they define)

20:15 Vincentdm: Ok, I see. Reminds me of the whole Stuart Sierra Component stuff. With an entrypoint that's responsible of setting everything up

20:15 justin_smith: exactly - I do that too

21:50 tjmaynes: hello

22:56 TimMc: puredanger: I really appreciate the clojure.org HTTPS redirect!

22:57 puredanger: np

22:57 TimMc: Is that TLS from Cloudfront back to the origin too?

22:59 I've been going around today poking people about HTTPS and it has been surprisingly productive. :-D

22:59 puredanger: the origin is S3 in this case

23:00 TimMc: Ah, got it. So that's good.

23:00 Interesting that you can't attach HSTS for CF. I understand it for S3, though.

23:00 puredanger: I found some places where people were complaining about it

23:01 sounds like it might be something they will add

23:01 if they do, I'll add it

23:02 TimMc: Oh, I remember this now, someone at work was dealing with this -- you can't add *any* custom headers to a CF distribution.

23:05 puredanger: yeah

Logging service provided by n01se.net