#clojure log - Oct 21 2014

The Joy of Clojure
Main Clojure site
Google Group
List of all logged dates

0:31 egghead: is there a good lib for wrapping up apis as cli apps in clj?

0:34 justin_smith: you should be able to do most of it with cheshire, clj-http, and tools.cli

0:35 maybe grenchman if the load-up time of an uberjar is still too long

0:40 trx|2: egghead: I use swig for accessing shared libraries (.dll,.so) from clojure. Swig is a command line tool but it wouldn't be difficult to create a custom leiningen plugin to run swig scripts from the command line.

0:41 rritoch: egghead: the problem I run into is with clojure class loaders so you need to register the generated .so or .dll from a static initializer

0:42 eggghead: because of that you typically need to adjust the .java files generated by swig before you compile them

0:43 justin_smith: rritoch: oh, you think he means library apis, I thought he meant http apis

0:44 rritoch: justing_smith: now that you mention it, he didn't specify

0:44 egghead: :)

0:45 as pitiful as the jvm is for cli apps, tools.cli is closest to what I was looking for with my vague question, just was wondering if there was anything ~fancier~

0:47 justin_smith: fancier means - integration with other tools? cursor control? colorful output?

0:49 egghead: i dunno what I was thinking justin_smith, I guess *in* and clojure.tools.cli are all I need

0:50 justin_smith: egghead: you'll likely find *out* and *err* pretty handy too

0:50 egghead: :p

0:51 justin_smith: plus stuff from clojure.string if you are taking input from stdin that comes from other tools, likely

0:56 rritoch: Does anyone have any suggestions for AOT compiled scripts? I'm developing a clojure web application framework and it seems I made a major performance mistake in the dispatcher. To mimic the behavior of other script languages like PHP I made a (script ...) macro which runs during compilation IF a dispatch variable is set as true.

0:56 arrdem: tbaldridge: watching the commit stream for pixie is an inordinate amount of fun :D

0:57 rritoch: example use case?

0:58 rritoch: my gut is that you're doing it wrong, but I'd like to quantify that

0:58 rritoch: Here is a condensed version of the macro (defmacro script [docstr & body] `(if *dispatch* ~(conj body 'do)))

0:59 Usage could be ... (script "HelloWorld" (println "hello world!"))

1:00 In the actual macro there is also some logging which dumps something like "running script: " docstr...

1:00 justin_smith: and the "script" would be something that optionally occurs as a stateful action during application template rendering while creating a project?

1:01 rritoch: The script is launched with either load-file or load-reader depending on if it is a local resource or a remote (URL)

1:01 justin_smith: in what context? when is the script run?

1:02 rritoch: The script is run from the servlet, this particular servlet sends all requests through the doPost method of the servlet.

1:02 justin_smith: OK that's a really bad idea

1:03 rritoch: Yep, I wanted to mimic the behavior of PHP before I realized how slow it would be

1:03 justin_smith: there is a lot of overhead on eval

1:03 yeah, eval is slow

1:03 rritoch: Now I want to undo that decision, preferably by just changing the macro and possibly the deployment method

1:03 justin_smith: what you want is one server process that stays running for months (maybe even years) at a time, that has a function that gets called for each request

1:04 instead of loading a script, call a function in a handler namespace

1:04 so convert the load-script logic to an invoke-function logic

1:05 this will involve defining the namespaces (of course), requiring them from your main servelet definition, and then invoking their functions - instead of having a file that imperatively does a sequence of steps, have a file that defines a function which another namespace can invoke

1:05 rritoch: Here is the complexity. Say there are more than one scripts in a single CLJ, this creates an issue since it would need to have multiple functions

1:05 justin_smith: rritoch: ok, that is the normal way to do clojure

1:06 I have namespaces with hundreds of functions in them

1:06 well, at least two

1:06 rritoch: Well, the point of this is to smooth the transition for developers coming from other script languages, such as PHP, CGI (Perl), etc.

1:07 * arrdem furrows brow

1:07 justin_smith: that's kind of odd

1:07 rritoch: I was thinking that maybe I could define an atom holding an array, and put the randomly generated function names in that array, and have a single initializer

1:07 justin_smith: why would you randomly generate function names?

1:08 and why would putting the functions in an array be in any way useful?

1:08 rritoch: The problem is the macro expansion would happen for each script in the file so I'm not sure if it's possible to have a different expansion for the first (and/or last) expansion in a given file.

1:08 It is useful because it simulates running a Perl or PHP script

1:10 arrdem: Okay so lemme get this straight

1:10 you have this (script) thing, that runs... when... at load time? at -main invocation time? at AOT time?

1:10 rritoch: So the purpose of using randomly generated function names is to avoid collisions between scripts, than there would be a single function like __run_scripts whic would run all of the scripts

1:10 But defining __run_scripts should only happen in one of the macro expansions

1:10 arrdem: that's a terrible reason. we have gensym. function name collisions are literally impossible.

1:11 justin_smith: rritoch: if they are in an array, they don't need any names at all

1:11 rritoch: arrdem: Yeah, I'm aware of gensym, it is what I plan on using

1:11 arrdem: rritoch: so... why do you have worries about name collision?

1:12 justin_smith: ,((juxt #(+ % 8) #(* % 3) #(mod % 2)) 12) ; rritoch - none of these have names, nor do they need them. They are all functions.

1:12 rritoch: But I would need to collect the list of functions which need to be run from the single script execution function (__run_sccripts)

1:12 clojurebot: [20 36 0]

1:13 arrdem: Why? a "script" is really just a function that sequentially calls a bunch of other anonymous functions in a straight line block..

1:13 justin_smith: rritoch: you can have a list of functions without any names

1:14 rritoch: Ok. so say I have some index.clj script setup which has 3 scripts in it, an init script, a form validation script and an output generation script as would be typical for a single page web application

1:15 justin_smith: what differentiates a script from a function?

1:15 rritoch: These need to be run in order, but the dispatcher has no idea about what index.clj is going to contain, or what functions it will contain

1:15 arrdem: this (if *script ... ) macro thing apparently

1:15 rritoch: arrdem: yes, basically

1:15 arrdem: rritoch: so... you have three functions in a file... go on...

1:16 rritoch: But the macro expansion just surrounds the code in an (if *dispatch* (do ...code...)

1:16 So as of now it runs at load time

1:16 But I want to AOT and simulate this same behavior

1:16 Assuming it's possible

1:16 justin_smith: rritoch: is the condition in the macro, or in the output of the macro?

1:17 rritoch: In the output of the macro

1:17 justin_smith: if the condition is in the output of the macro, and it checks *dispatch* at runtime, you are fine

1:17 just make sure *dispatch* is not resolved during compile time

1:17 rritoch: Yeah, that is how it currently works, but it is slow

1:18 justin_smith: it's slow because you are calling load

1:18 rritoch: I'd like to AOT compile this instead

1:18 justin_smith: you don't need to call load

1:18 just require the namespace, and call the function

1:18 wrap the whole "script" in one function you can call

1:18 rritoch: The dispatcher isn't going to know the names of the functions

1:18 The web designer is going to build index.clj

1:19 With whatever scripts they need to run to produce the web page

1:19 justin_smith: OK, then pass those function names in as an argument

1:19 this is all logic you can do at runtime, without needing to use the compiler to load new code

1:20 rritoch: The dispatcher won't know what the function names are, at least not on an initial HTTP GET request, I can see function names being passed in POST requests, or part of URL's but not for a home page

1:20 But you did give me an idea

1:21 Can the macro define the gensym function with metadata? So at runtime a (run-script) function could extract the list of functions which need to run?

1:22 ex. (run-script "index.clj")

1:22 justin_smith: rritoch: in order to know what to run, there needs to be shared information between the one deciding what runs and the one doing the running. In your case it was a file name.

1:22 rritoch: you can literally take every file name, and make it a function name, that the contents of each file you would load, and put it in a function body.

1:23 there is no extra information needed that was not needed in the original unperformant design

1:23 *that has the contents of each file

1:23 you can have a single namespace, with an "index" function, and a "validate" function, etc. etc.

1:23 no gensyms are needed

1:23 no runtime compilation is needed

1:24 rritoch: Hmm, it's going to take me a few minutes to process that one

1:24 justin_smith: rritoch: that's understandable

1:24 rritoch: So your suggesting I wrap the source code in a function that gets called, and compile that instead so I can just call the function?

1:25 * arrdem cheers

1:25 justin_smith: rritoch: clojure is not a scripting language like php or perl, but I promise that the clojure approach is very useful, and helps make good designs that scale well

1:25 arrdem: (inc justin_smith)

1:25 lazybot: ⇒ 100

1:25 justin_smith: rritoch: yes, that sounds about right

1:25 rritoch: Technically I think that is exactly what I want, but I've never considered that possiblity, nor am I sure where to start, but your right, that would be a fast AOT script

1:25 justin_smith: rritoch: and you don't need to explicitly compile the function, that should happen (as much as is needed) during generation of the uberjar and startup of the app

1:26 rritoch: you have a marvelous new world of app design ahead of you if you stick with it :)

1:27 rritoch: Lol, I have 30 years programming experience, and 10 years web development experience, I'm not new to LISP, but fairly new to clojure (less than a year)

1:27 justin_smith: OK

1:27 rritoch: I'm almost literally over-the hill

1:27 To the point where I'm just making code to pass on to the next generation

1:28 justin_smith: sorry, not meaning to be condescending, just saying, based on your level of (un)familiarity of how things are done here, and my own experience with scripting oriented stuff

1:29 rritoch: No, I understand your perspective. It is the clojure way of doing things, and the solution you came up with is almost exactly what I'm looking for

1:30 In addition to helping developers familiar with scripting languages, this also simplifies the task of translating existing web applications into clojure.

1:30 justin_smith: I think you'll find it's the way of doing things you'll find in any language that doesn't run a process that is expected to run for under half a second and then exit

1:30 rritoch: that is a promising thing, yeah

1:31 rritoch: maybe you could also blog about the adaptation process also - I bet there would be a lot to learn from that

1:33 rritoch: justing_smith: It has taken me a very long time to reach the proof of concept phase where clojure web apps could be run from tomcat, and standalone, deploying .jsp templates and using .clj scripts for deployment.

1:33 Sources can either be in the application, maven or OSGI bundles.

1:34 justin_smith: rritoch: ring makes things a lot easier

1:34 rritoch: I developed a leiningen plugin which copies all dependencies to the WEB-INF/lib folder

1:34 justin_smith: it connects regular clojure functions to the servlet api

1:34 ring does that part too

1:34 rritoch: justin_smith: Ring made things worse, I had to remove it

1:34 justin_smith: rritoch: I suspect you were not using it the way it was meant to be used - I have used it for years

1:34 rritoch: Ring doesn't support JSP

1:34 justin_smith: OK

1:35 rritoch: There is a conflict because Ring provides it's own javax/servlet/Servlet.class

1:35 justin_smith: OK

1:35 rritoch: That of course conflicts with what tomcat provides

1:36 justin_smith: were you using ring uberwar?

1:36 rritoch: It was a nightmare so I disabled ring support until I can find a solution to it

1:36 justin_smith: I have a few ring apps running in tomcat instances

1:36 they work great - but I am not explicitly using jsp, just the ring apis

1:37 rritoch: To support CGI this framework depends on tomcat

1:37 justin_smith: like I said, I have ring apps running via uberwars inside tomcat

1:37 rritoch: So it can be run from the command line, and still process .jsp files

1:38 ring uberwars wouldn't be able to support CGI execution of jsp scripts, simply because of the connnflict with the javax/servlet/Servlet.class

1:38 justin_smith: OK

1:39 I don't really know the jsp part, so I can't address that

1:39 rritoch: But I actually still have ring support in the code, I'm just waiting to find a solution to this conflict issue

1:41 In order to load code from OSGi modules I've needed to wrap tomcat's resources (such as ServletRequest, ServletContext, ServletResponse) in wrappers so I could .forward to code which isn't in the classpath but is in OSGi bundles

1:41 Needless to say this development process was a nightmare because around every corner is a new bug or conflict to deal with

1:42 Now that it is working I'm trying to undo some of the horrible design choices that were made just to get the thing functional.

1:42 justin_smith: rritoch: well, I'll just reiterate what I said above - I put functions into namespaces, compiled at uberjar or application startup time, and use function parameters to vary their behavior, and it works very nicely for me

1:42 arrdem: rritoch: so... what class conflict are you getting with ring? Because Ring doesn't actually package that class as far as I can tell.

1:43 rritoch: arrdem: I was getting a security violation

1:43 aardem: If I recall correctly its because the "signers" don't match between what ring utizes and what the tomcat library provides

1:44 arrdem: rritoch: what version of the servlet package are you using?

1:45 in production that is, ring has another version and I suspect that's where your issue is coming from.

1:45 rritoch: arrdem: but yes, that class is provided by ring, at least it was, I can give you it in a sec, but first I wanted to share the leiningen dep which is [org.apache.tomcat/tomcat-jasper "7.0.52"] and conflicts with ring

1:45 The tomcat dep is the only place of conflict

1:47 I don't know if this is new, but as you can see from the sources, ring won't allow tomcat to be used

1:47 https://github.com/weavejester/lein-ring/blob/master/src/leiningen/ring/uberwar.clj#L26-L33

1:48 Simply because tomcat contains javax/servlet/Servlet.class

1:48 Previously I was getting a security conflict, but with the latest version it looks like the tomcat dependency wouldn't even be loaded

1:49 I wasn't actually running ring from tomcat with uberwar though, I didn't make it that far

1:49 It was crashing while running it from the command line (CGI mode)

1:50 The conflict I was getting comes from this code... https://github.com/weavejester/lein-ring/blob/master/src/leiningen/ring/war.clj#L230-L233

1:51 So, until ring is willing to give up control of what servlet it uses, this application framework can't support ring.

1:52 Without tomcat, there is no easy way to execute .jsp templates

1:52 Making your own .jsp compiler is one option, but it would be an insanely huge project

1:53 justin_smith: rritoch: add a config to make that one step optional, it should be a trivial patch

1:53 rritoch: that seems so much easier than the alternatives

1:55 rritoch: Ring doesn't provide any configuration setting to bypass that step

1:56 Yeah, I could create my own "version" of ring which doesn't conflict but that basically eliminates any benefits to using ring in the first place, since I'd have to maintain a custom version

1:57 justin_smith: rritoch: it's open source, clone it, offer a patch

1:58 that process is so much easier than any other alternative I have seen so far

1:58 rritoch: justin_smith: That isn't a bad idea, but it isn't a priority. Optimization is the current priority but before going onto the next phase I really need to revisit the ring issues

1:59 Leaving code in the application that never gets used, while it happens more than anyone likes to admit, I'd prefer not doing it.

1:59 justin_smith: but this is the easy way to get things optimized. eval is very expeinsive, you want a long running process serving requests without runtime compilation

2:00 rritoch: Well, this script issue isn't the worse problem I face

2:00 I believe you helped me the other day with this fallback URL compiler

2:00 But technically the only reason that is required is the classloader isn't looking in the right places to find the pre-compiled classes on the remote server

2:02 justin_smith: rritoch: do the names of the namespaces reflect the location of the files relative to the effective classpath?

2:02 rritoch: This fallback compilation hack to compile the remote sources locally is functional, but I don't want to rely on it as the primary means of utilizing remote code bases.

2:02 justin_smith: compiling before deploying is not a hack

2:02 it is the right way to do it

2:02 compiling at runtime is the hack

2:03 rritoch: I agree, and that is what I have in place right now, a runtime compiler hack

2:03 But there are actually some benefits to that since it could respond to code changes in real time without a need to restart the server or servlet container.

2:04 arrdem: Clojure will do that out of the box with a simple defn, no "compile hacks" required.

2:04 rritoch: Regardless of the possible benefits, it is currently on my list of optimization issues to deal with

2:04 arrdem: The entire def infrastructure is designed to support dynamic code reloading both in development and production without having to defensively eval or load everything all the time.

2:05 rritoch: arrdem: Compile hacks are absolutely required for this case as source code can exist in the local app, in maven loaded jars, in OSGI bundles, or from remote URL's

2:06 This is an issue that was solved yesterday since load-reader doesn't generate __init.class when *compile-files* is enabled.

2:06 justin_smith: rritoch: and we have a classpath, so that the function (require) can abstract over all of those, as long as you set the classpath properly

2:07 rritoch: Since the compile function only accepts a local source, the only way around it was to generate a reader from the URL and call the Compiler/compile function directly, manufacturing an appropriate script path to ensure that the __init.class is properly generated.

2:07 justin_smith: rritoch: every method of loading code compiles it

2:07 rritoch: Justin, these are web applications running from tomcat, adjusting the classpath used by tomcat isn't how I plan on spending the next 3 years of my life

2:08 justin_smith: do you explicitly need to create the .class file?

2:08 rritoch: Tomcat doesn't simply classpath manipulation in any way

2:09 Regardless, what I've done so far is really the only solution until I can get the deployment to properly switch classloaders when controller (MVC) functions get called.

2:10 justin_smith: I'm referring to attaining classes (gen-class) from a source file located on a remote server

2:11 As I said, load-reader doesn't generate the __init.class, which is why the hack is required,

2:11 load-reader does a lot of compilation, but the generated class files are unusable since it doesn't include the __init.class

2:12 justin_smith: rritoch: it's late here, and I need to move on to other things. Good luck. I think there are likely much simpler ways to meet your root requirements than the elaborate things you are describing, but I can't go into it right now, sorry.

2:12 rritoch: This may be a bug or a feature of clojure, I'm not really sure, but the way around it was to call Compiler/compile directly

2:13 justin_smith: That is not a problem, this hack solution works, and I neeed to deal with the class loader issue anyhow.

2:14 I also need/want to implement this function/file solution so the scripts run faster by being AOT compiled

2:15 arrdem: rritoch: ... having spent the past summer working on the Clojure compiler implementation, I assure you that you are bludgeoning the Clojure runtime into supporting a sequentially executed imperative style based on code loading for the language is not designed.

2:15 rritoch: AOT is not your problem. Your problem is that you are misapplying the language as if it were PHP or Perl when it is not designed to be used as such.

2:16 rritoch: arrdem: That id debatable, since it is exactly how most versions of LISP function

2:16 err, is debatable

2:17 arrdem: rritoch: Clojure is hardly a "typical" lisp in a number of ways especially with regards to the total lack of an interpreter let alone the datastructure and software engineering philosophies involved.

2:18 rritoch: arrdem: You can't have your cake and eat it too, your explicitly contridicting the reasoning behind not allowing cyclical dependencies

2:19 Either clojure is meant to support Streaming source code, or it isn't

2:19 arrdem: rritoch: cyclical dependencies on what... namespaces? vars? classes?

2:20 ddellacosta: I guess it's not possible to do destructuring w/Schema in schema.core/defn, huh

2:20 rritoch: It may take me some time to find the article, but as I said, what your stating is directly contridicting the statements from clojure's developers as to why they won't support cyclical dependencies

2:21 It is right here: https://news.ycombinator.com/item?id=2467809

2:22 By Rich Hickey who supposedly has a big role to play with Clojure

2:22 arrdem: how does that have anything to do with the conversation at hand

2:22 by the way we do support circular dependencies on symbols via declare

2:22 with some restrictions

2:23 rritoch: You stated I was misapplying the language, but I'm actually directly applying the language per the reasoning by Rich as to why cyclical dependencies aren't being allowed

2:24 I may not know all of the clojure politics, but I suspect Rich is higher up in the food chain than anyone in this chatroom.

2:24 Raynes: ☜(⌒▽⌒)☞

2:24 rritoch: So, either Clojure is meant to be scriptable/streamable, or it isn't

2:24 arrdem: Raynes: lols

2:24 rritoch: I'm working under the assumption that it is

2:26 Since load-reader doesn't create the required __init.class files when *compile-files* is enabled the only way to compile a class from a stream is to pass it to Compiler/compile directly, as I have done.

2:27 sveri: Hi, I have a list of keywords and a map. For each keyword I want to update the map with a given function, how would I do that? I know that I can reduce over the list, but I don't know how to keep a copy of the new map the same time.

2:27 rritoch: If clojure is planning on moving away from streamability, than remove this cyclical dependency issue because it causes more problems than it solves.

2:30 amalloy: sveri: reduce keeps an accumulator; that's the whole point. here, your accumulator is the new map

2:30 rritoch: Better yet, just repair load-reader (Compiler/load) so that it generates the __init.class files when *compile-files* is enabled.

2:30 amalloy: (reduce (fn [m k] (change m somehow with k)) original-map list-of-keywords)

2:30 dbasch: ,(reduce (fn [x y] (assoc x y (rand-int 10))) {} [:a :b :c :d])

2:30 clojurebot: {:d 6, :c 5, :b 7, :a 0}

2:30 dbasch: ^ sveri

2:31 sveri: amalloy: dbasch thank you, obv. that's the whole point. My head still does not want to think about it that way, but it's clear if someone states the obvious, jeez

2:32 justin_smith: rritoch: to step back in for a moment: clojure supports runtime update, and aims to minimize the difference between entering code sequentially in a repl vs. loading from a required file. This is on order to streamtime development, but has nothing to do with hotswapping code in production. Also, this does not mean that streaming code and using eval on arbitrary sources at runtime is a) efficient b) simple or c) a sane way to get any r

2:32 work done. I'm not suggesting this is wrong because clojure doesn't or isn't meant to support scripting, but saying that the kind of complexity and brittle behavior you are seeing is symptomatic of using clojure in unusual ways.

2:34 if I dealt with the kind of complexity and frustration in my clojure development that you are describing, I would have ditched it long ago.

2:34 rritoch: justing_smith: I've walked away from this project a few dozen times for the exact reasons your stating

2:35 justing_smith: But the goal of this is to reduce the complexity, and learning curve, for developers using this web platform

2:35 If I do the hard work, they don't have to

2:35 For that matter, it also means that I don't ever need to deal with these problems again, once they're solved

2:36 justin_smith: my fear is that the design is importing external complexities that don't need to exist, and introducing an extra layer of complexity by trying to make clojure something it simply is not

2:38 rritoch: Flexibility is exactly why I'm utilizing clojure instead of Pike, clojure has far fewer limitations

2:38 In this particular case a real clojure developer can define their functions and have a (script) macro which calls their functions, escaping the scripting structure

2:39 But for programmers familiar with scripting languages, this hybrid functionality makes the transition easier on them

2:40 arrdem: At the expense of atrocious performance and shielding them from a language designed to support real software architecture rather than load order based imperative programming

2:41 rritoch: arrdem: That is why I'm now dealing with the performance issue, and I believe justins solution of wraping the file as a function will solve that problem

2:42 arrdem: I've never tried it before, and I'm not entirely sure if (ns .... (:import) is going to function properly within a function, but these are issues I'll learn when I try

2:44 Regardless, this platform has a long way to go, after the optimization step, I need to revisit the ring issue, and after that I need to establish a modern GUI. I'm considering using libGDX instead of JavaFX, but integrating that into a framework which is already complex is probably going to take years.

2:46 So far all I've solved is creating a MVC framwork that is script driven, which can utilize maven resources, OSGi bundles, and remote codebases (Such as how many .js files are delivered client-side).

2:46 I'm currently optimizing what I have so far because repairing these things after adding new layers of complexity will be a nightmare

2:47 amalloy: rritoch: importing inside of a function doesn't do anything. import doesn't load any classes, just gives the current file a shorthand to refer to them

2:48 rritoch: amalloy: :( that is truly sad then, because these scripts need a way of defining Imports to the scripts can be clean

2:49 justin_smith: rritoch: but that isn't what :import does

2:50 rritoch: justin_smith: I believe I know what import does, it likely generates bitecode so the class file contains the imports at a binary level, and it also cleans up the code because you don't have to namespace qualify all of your classes

2:50 justin_smith: rritoch: nope

2:51 rritoch: justin_smith: Ok?

2:51 justin_smith: it's a convenience so you don't have to type out the full package name

2:51 arrdem: rritoch: there is no JVM bytecode "import" operation.

2:52 rritoch: loading a class loads all of its transitive dependencies recursively. As justin_smith says, import just makes class names nicer to type.

2:52 rritoch: arrdem: If I recall correctly, the class file format includes a section for imports, but I could be wrong, it has been over 10 years since I dealt with class files at a binary level

2:52 justin_smith: rritoch: http://stackoverflow.com/questions/9680295/import-statement-byte-code-significance

2:53 rritoch: Well, I just rechecked the class file structure, and your right, there is no section for imports

2:53 http://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html

2:54 It's been well over 10 years since I dealt with class files at a binary level

2:54 arrdem: Right. Classes make fully qualified references to their depended classes, and unresolved dependencies forces classloading.

2:54 rritoch: arrdem: Ok, well using load-reader is allowing me to utilize imports

2:55 arrdem: what does "utilize imports" mean. We just went over the fact that (import) is simply syntactic sugar.

2:57 rritoch: arrdem: Are you trying to be helpful or are you just trolling?

2:57 dysfun_: is there a clojure library for handling renaming files with a known name into the format file.n.ext where n is an increasing number?

2:58 rritoch: dysfun_: http://clojuredocs.org/clojure.core/format ?

2:58 dysfun_: hrm, i suppose that works

2:59 rritoch: dysfun_: If you want it more automatic you may be able to find a way of wrapping format to make it automatic

2:59 dysfun_: that's overkill. it just seems like a common piece of code. i wondered if anyone knew a library that included it

3:00 it's not terribly difficult to write

3:45 rolfb: urh

3:45 oh nice, completely botched the window there

3:45 sorry all

4:43 tincho1: Hi, I'm having an issue with loading the optimus library as it uses core.memoize which relies on core.cache and that has some issue with it in core.cache/through.

4:44 I've found similar issues online but their solutions don't seem to work for me. I've tried tracking through the dependencies and :excluding core.cache when I can. But no luck.

4:45 If anyone has some advice it would be greatly appreciated

4:56 rurumate: Why does tools.analyzer depend on datomic-free?

4:57 It's surprising to find a part of the compiler need some sort of database

4:57 see https://github.com/clojure/tools.analyzer/blob/master/project.clj

4:58 AeroNotix: Bronsa^?

5:07 borkdude: rurumate https://github.com/clojure/tools.analyzer/search?utf8=✓&q=datomic

5:08 rurumate it has scope provided

5:12 rurumate: borkdude: yes but still lein pulls all the datomic deps when doing a lein install in tools analyzer. Also, there is no mention of datomic on README.md

5:13 borkdude: rurumate that is normal with scope provided. it won't be packaged with your uberjar

5:14 rurumate: if it's not packaged with the uberjar, apps may break at runtime. The circumstances for that are not mentioned in the README

5:16 maybe it makes more sense to put the datomic stuff in a separate project that depends on t.a. ?

5:18 cfleming: rurumate: Bronsa is a bit too clever for that I think. I haven't looked in detail, but I believe the datomic stuff is experimental, and it looks like it's conditionally compiled out if not present: https://github.com/clojure/tools.analyzer/blob/5a58c3ffd87da05e50924d2931b833544717ee55/src/test/clojure/clojure/tools/analyzer/query_test.clj#L11

5:18 rurumate: Is this actually causing you some problem?

5:20 rurumate: cfleming: yes, it bothers me

5:22 cfleming: rurumate: That sounds like a fairly mild sort of a problem :-)

5:23 rurumate: also it clogs my hard drive

5:28 dysfun_: gah, i keep getting "Variable binding depth exceeds max-specdl-size just trying to do (ns foo) and evaluating it under cider-ritz (via nrepl). any ideas?

5:29 or at the very least what that error even means to give me an idea where to look?

5:30 oh, seems to be on the emacs side

5:30 is there a channel for clojure-related emacs stuff?

5:34 phillord: so this error is normally caused by an illfounded recursion

5:34 although it can happen as a result of just deep recursion

5:34 One solution is to do (setq max-specpdl-size 4000)

5:35 which will increase the max allowable level, but if it still happens, then there is a bug somewhere. For which you need to turn debugging on, and get a stack trace

5:35 dysfun: oh, i think it might be something to do with the 5 nrepl connections i just closed

5:35 but then i was creating them because i was getting that, and sometimes it would just go away after a while

5:37 phillord: if it occurs randomly and not predicably, then it's quick likely just that you are hitting a deep recursion

5:37 so increasing the max size will work

5:37 dysfun: i've set it to 10 for now, and it's gone away. i suppose i'll have to either find the bug and fix it or just restart emacs periodically

5:37 phillord: if not, again, you need to get a stack trace

5:38 You have max-specpdl-size at 10?

5:38 dysfun: yes

5:38 phillord: well, the default on my OS is 1300

5:38 dysfun: that's bizarre then

5:38 cause it's working fine now

5:38 phillord: what OS?

5:38 dysfun: osx

5:38 plain gnu emacs

5:39 phillord: launch with "emacs -Q"

5:39 and then tell me the value?

5:40 dysfun: will do, once I've reproduced it

5:40 (if i reproduce it)

5:40 phillord: you don't need to reproduce anything -- just see whether max-specpdl-value is the same with -Q --

5:40 dysfun: oh right

5:40 rurumate: Is it possible to require more than one library at once, in the repl? Like the :require clause in the ns macro

5:41 dysfun: 1300

5:41 rurumate: you can use ns if you want

5:41 phillord: this variable controls how many times you can recurse -- because it controls how many times a variable binding can be pushed onto the stack

5:41 Emacs cannot know whether something is going to terminate, so it just sets this value

5:41 rurumate: oh sure, why didn't I do that before

5:42 dysfun: *nod*, most interpreted languages do that

5:42 so why didn't everything break completely when i set it to 10?

5:42 phillord: if you have rest it to 5 from the default 1300, then are you going to crashes in perfectly normal code

5:42 * phillord shrugs

5:42 phillord: I don't know

5:42 dysfun: software :)

5:42 phillord: where did it get set to 10?

5:42 dysfun: m-x set-variable

5:43 phillord: you set it to 10?

5:43 dysfun: yup

5:43 phillord: why?

5:43 clojurebot: why is startup slow is busy compiling the `for` macroexpansion

5:43 phillord: set it back to 13000

5:43 dysfun: first stackoverflow i answered

5:43 phillord: sorry 1300

5:43 dysfun: er saw

5:43 well i killed it now

5:43 so it's back to 1300

5:44 phillord: This value used to be smaller in the past -- it protects emacs from o-o-memory crashes

5:44 dysfun: i have 8 gigs on this machine. i'm not concerned about that :)

5:44 phillord: no

5:44 that's why it's got much bigger than it used to be

5:44 dysfun: i know it was written for a different day...

5:45 then again, the default configs for postgres and mysql are very conservative

5:45 the postgres one in particular is for a Sun IV

5:46 phillord: well, emacs is like Shakespeare -- not of a time, but for all time

5:47 cfleming: It's also like Shakespeare in that no-one really understands it :-)

5:48 dysfun: heh. i'm kind of hoping DEUCE will get somewhere, but i'm not interested enough to actually develop it

5:59 SagiCZ1: how does (swap! foo atom) work with the foo function? when i redefine foo while the application is running, the new behavior isnt reflected.

6:02 i mean the live re-compilation is advertised as a feature of clojure but it seems to me that you have to design carefuly so that you can use it and if you do would that produce idiomatic code?

6:12 dysfun: the code that uses it also has to be recompiled

6:13 it gets bound to a compiled function

8:02 mearnsh: dysfun: #clojure-emacs exists

8:14 dysfun: mearnsh: yeah, i discovered :)

8:16 mearnsh: ah yes

8:58 justin_smith: dysfun: you may want to mention on #clojure-emacs that you are using cider-ritz

8:59 dysfun: the issue has gone away

9:44 SagiCZ1: ~lazy-logs

9:44 clojurebot: lazy-logs is http://logs.lazybot.org/

9:45 justin_smith: SagiCZ1: the syntax is (swap! atom foo)

9:46 SagiCZ1: if you need t see a new definition of foo, you need to recompile the function that calls (swap! atom foo) OR you need to call (swap! atom #'foo)

9:46 that is guaranteed to always look up the definition (can be a bad idea because of the var lookup overhead)

10:02 SagiCZ1: thank you justin_smith

10:03 justin_smith: np

10:03 SagiCZ1: is that true for other calls as well? (foo (foo2 coll))

10:04 justin_smith: yes, if you need to see changes without redefining, then you can change it to (#'foo (#'foo2 @#'coll)) ; assuming all 3 were global defs

10:05 in practice you should only need explicit var lookup like the above in stateful functions that have internal mutation (like a server process, or if they wrap some long running stateful java thing)

10:06 with pure functions, just re-evaluate the definition to capture the new definitions for things it uses

10:06 SagiCZ1: my case is a long running doseq

10:06 which is consuming a lazy-seq stream

10:06 if i want to recompile it, i would have to stop the consuming and start over

10:06 justin_smith: yeah, if you need to see new defs while it runs, it makes sense to use var lookup explicitly then

10:07 mdrogalis: That sounds like some funky code.

10:08 SagiCZ1: and now that i mention it, i am not sure i should use lazy-seq to act as a stream... i am streaming some data from the internet and it seems to work fine, but i wonder if there is some better clojure way to do streams.. non mutable..

10:08 mdrogalis: it probably is kinda funky, i would like to improve it

10:08 justin_smith: SagiCZ1: what about using core.async and putting the data elements on a channel?

10:08 mdrogalis: SagiCZ1: Can you use a real queue and reload your code in a normal way?

10:09 justin_smith: then you can redefine the consumer of the channel

10:09 or yeah, a queue

10:09 mdrogalis: Aye, both are good solutions.

10:09 SagiCZ1: what is a normal queue?

10:09 *real

10:09 rurumate: queue = channel in core.async lingo

10:09 justin_smith: SagiCZ1: there are a few options in java.util.concurrent

10:09 mdrogalis: SagiCZ1: Oh, I mean something like HornetQ or ActiveMQ.

10:10 justin_smith: mdrogalis: oh, an inter-process queue?

10:10 mdrogalis: The reason I say that you might want an inter-process queue is that it will buy you the ability to kill off your old jar and reload a new one from scratch. Avoids redefining vars.

10:11 SagiCZ1: mdrogalis: yeah i would probably end up with that later, because i might need some automatic back up in case of connection drop, but for now i would like to be able to use just clojure stuff.. of course writing abstract enough code to be able to swap in activeMQ alter

10:12 mdrogalis: Maybe an interface between your code and a channel would be best. Sounds like you get the idea.

10:12 SagiCZ1: yeah so i will probably use java's blocking queue or something.. or get familiar with clojure.async

10:13 mdrogalis: :)

10:13 SagiCZ1: thank you both for ideas

10:14 mdrogalis: Yep

10:18 mmeix: trying to solve 4clojure #58, need a hint

10:18 see: https://www.refheap.com/92101

10:20 clgv: mmeix: do not construct lists but call functions in the reduce step

10:21 mmeix: hm, tried that ... but obviously the wrong way

10:21 ok ... more thinking - thanks for the hint

10:21 (but it's not way off, right?)

10:23 SagiCZ1: if i want to apply a function to each element of a coll but i only care about the side effects, is there a better way than (dorun (map foo coll)) ?

10:23 puredanger: maybe the new run! in 1.7?

10:24 I guess that does reduce-y things so maybe not

10:24 dorun seems fine

10:27 SagiCZ1: puredanger: alright, just wondering

10:28 zerokarmaleft: SagiCZ1: I prefer doseq

10:29 SagiCZ1: but map can do pmap

10:32 puredanger: if you're heading in that direction it might be more direct to just make your own ExecutorService

10:33 jonathanj: how does one do networking in Clojure?

10:33 i'm hoping for something more high-level than java.net.Socket

10:34 * clgv checks run!

10:35 clgv: ok, `run!` just uses the benefits of reducible collections if possible

10:35 sam_: Hi, When I do (class (clojure.instant/validated [ 1 1 1 1 1 1 1 1 1 1 1 ])) , I get clojure.instant$validated$fn__6196. Could someone explain what this means?

10:36 ToxicFrog: sam_: does clojure.instant/validated return a function?

10:37 Jaood: jonathanj: haven't use it but there's aleph

10:37 ToxicFrog: Because that's the type of an anonymous function.

10:37 stuartsierra: puredanger: What's the first argument to `run!`

10:38 puredanger: based on the source, I'd say it's a 1-arg function to be invoked with an item from coll

10:38 SagiCZ1: (doc run!)

10:38 clojurebot: "([proc coll]); Runs the supplied procedure (via reduce), for purposes of side effects, on successive items in the collection. Returns nil"

10:38 puredanger: ,(source run!)

10:38 clojurebot: Source not found\n

10:38 jonathanj: Jaood: thanks, i actually just stumbled on to aleph and it looks fairly useful at first glance

10:39 puredanger: ,(run! println (range 5))

10:39 clojurebot: 0\n1\n2\n3\n4\n

10:41 stuartsierra: OK, it's just `(reduce #(proc %2) nil coll))`

10:42 Like `(dorun (map …))` but without the intermediate seq.

10:42 sam_: Oh. I was trying to annotate namespaces.

10:42 So, under what type would this come?

10:42 And the [ 1 1 1 1 1 1 1 1 1 1] is a persistent vector, right?

10:44 stuartsierra: sam_: Every Clojure function compiles to its own class with a generated name.

10:45 `clojure.instant$validated$fn__6196` is probably the class of an anonymous fn created in the body of a function named `validated` in the namespace `clojure.instant`

11:38 numberten: is there a function that lazily concats but with a non-deterministic order?

11:41 Bronsa: numberten: no but it's trivial to make one

11:41 numberten: (defn shufcat [& args] (apply concat (shuffle args)))

11:47 mdrogalis: Bronsa: shufcat. I love it.

11:48 numberten: Bronsa: (shuffle args) would realize the lists in args right?

11:49 Bronsa: numberten: no

11:49 numberten: it realizes args, which is not lazy anyway. but the elements in args are still not realized

11:49 ,(defn shufcat [& args] (apply concat (shuffle args)))

11:49 clojurebot: #'sandbox/shufcat

11:50 Bronsa: ,(first (shufcat (map #(do (println %) %) (range 10)) (range 10 20) (range 20 30)))

11:50 clojurebot: 20

11:51 numberten: very cool

11:52 i guess I thought the inner most arguments were evaluated first, so any strict list processing call would realize everything

12:01 sveri: ,(midje)

12:01 clojurebot: #<CompilerException java.lang.RuntimeException: Unable to resolve symbol: midje in this context, compiling:(NO_SOURCE_PATH:0:0)>

12:01 sveri: (midje)

12:01 nullptr: ~midje

12:01 clojurebot: #<RuntimeException java.lang.RuntimeException: Invalid token: :>

12:04 EvanR: so is there a way to use reflection intentionally without a reflection warning

12:04 without disabling the warn option for the whole file

12:04 Bronsa: EvanR: probably only by using clojure.lang.Reflect directly

12:05 EvanR: thats what im doing, clojure.lang.Reflector/invokeInstanceMember

12:05 and i get reflection warning, clojure.lang.Reflector/invokeInstanceMember cant be resolved

12:06 Bronsa: EvanR: you have a reflection warning on your usage of c.l.R/invokeInstanceMember then, not on the method you're trying to invoke

12:06 EvanR: confused

12:06 Bronsa: EvanR: I see the 3-arity takes either an array or an Object

12:06 EvanR: i have an object

12:08 putting ^Object does not help

12:09 sam_: Hi!

12:09 Bronsa: EvanR: hint the first arg as ^String

12:09 EvanR: oh

12:09 sam_: Are functions/namespaces to work with data structures?

12:10 EvanR: the second arg to invokeInstanceMember? it didnt work

12:10 Bronsa: EvanR: I really can't figure out why the compiler thinks there's an ambiguity there, might be a reflector bug

12:10 EvanR: http://sprunge.us/XeeE

12:11 EvanR: the first arg

12:11 why string?

12:12 Bronsa: because that's its signature, the defined arities are (Object,String), (String, Object, Object[]) and (String, Object, Object)

12:12 EvanR: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Reflector.java (String, Object, Object[]) ?

12:13 oh wrong method

12:13 dbasch: it could match this method too perhaps: public static Object invokeInstanceMember(String name, Object target, Object... args)

12:14 EvanR: i had the arguments reversed

12:14 Bronsa: dbasch: no, in my repl (clojure.lang.Reflector/invokeInstanceMember ^String a b c) doesn't cause a reflection warning while (clojure.lang.Reflector/invokeInstanceMember a b c) does

12:14 dbasch: that’s not object[], it’s variadic

12:15 Bronsa: dbasch: what? Object.. x compiles to Object[]

12:16 EvanR: currently i have this

12:16 (clojure.lang.Reflector/invokeInstanceMember (str "get" (name k)) t (int i))

12:16 dbasch: Bronsa: what I mean is that when calling the method you don’t call it with an array

12:17 EvanR: (int i) doesnt look right

12:17 Bronsa: dbasch: from clojure you do

12:19 dbasch: Bronsa: but how would the compiler know whether c is Object or Object[] ?

12:19 noncom: can anyone explain, why is it a "while" cycle here instead of a simple "if" that would suffice in my opinion? https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Namespace.java#L58

12:20 EvanR: (clojure.lang.Reflector/invokeInstanceMember ^String (str "get" (name k)) t (int i)) is still giving me a warning

12:21 but

12:21 Bronsa: EvanR: it's the int the issue there, you need to remove it

12:21 EvanR: (clojure.lang.Reflector/invokeInstanceMember ^String (str "get" (name k)) ^Object t ^Object (int i)) worked

12:21 remove it?

12:21 Bronsa: dbasch: the java or clojure compiler?

12:21 dbasch: Bronsa: clojure

12:21 Bronsa: EvanR: either remove the (int ) call or box it

12:22 EvanR: box it?

12:22 Bronsa: dbasch: it might just default to Object

12:22 EvanR: why do you need that (int) call?

12:22 dbasch: Bronsa: but I can see how both methods could be considered

12:23 Bronsa: dbasch: there are some cases where the compiler just picks one of possible methods, I think this is one of those

12:23 EvanR: Bronsa: it appears to be used as the argument to the dynamically named method?

12:23 Bronsa: dbasch: it probably picks the Object[] arity only if you tag the arg as ^objects

12:23 myguidingstar: hi all, is it possible to run a lein task from existing nrepl?

12:25 joegallo: it's all just clojure code, so it is *possible*

12:25 technomancy: myguidingstar: if you set :eval-in :nrepl, Leiningen will try to connect to an existing nrepl server instead of starting a new jvm

12:25 (it will fall back to starting a new JVM if it can't though)

12:26 Bronsa: EvanR: look, (int x) in that case returns a primimtive int. the invokeInstanceMember method takes an Object. if you pass a primitive to it the compiler will not know how to invoke it, just remove the (int) call

12:26 technomancy: and you can't initiate it from within the repl (unless you bring in leiningen as a dep)

12:28 myguidingstar: (inc technomancy)

12:28 lazybot: ⇒ 150

12:29 Bronsa: noncom: the compareAndSet might fail

12:32 noncom: Bronsa: ah, i see.. :)

Logging service provided by n01se.net