[xwiki-devs] adding a new jsr-223 scripting language to xwiki (clojure)
In http://code.xwiki.org/xwiki/bin/view/Snippets/DisplayFormatedCodeAttachments...
VincentMassol | 2009/05/05 20:03
Niels, no there's no plan but we can now support any language using macros so feel free to provide a clojure macro if you're interested. Shouldn't be too hard to do, especially if there's a JSR-223 implementation (in which case simply dropping the jar in WEB-INF/lib should be enough - you'd then use it using the script macro).
Clojure <http://groups.google.com/group/clojure> has aJSR-223 implementation according to: http://github.com/pmf/clojure-jsr223/tree/master http://groovy.codehaus.org/JSR-223+access+to+other+JVM+languages http://sayspy.blogspot.com/2009/03/interacting-between-jvm-lang-here-and.htm... Are there any examples, documentation, or suggestions of writing a "script macro" to call a new jar in WEB-INF/lib ? Any other special magic I should know? Does anything from the above suggest that Clojure couldn't be used as an Xwiki scripting language, replacing cases where Groovy scripting might normally be employed? Niels http://nielsmayer.com PS: Here's examples of easy "scripted java" programming you can do in Clojure (note the helpful parallelism constructs): http://travis-whitton.blogspot.com/2009/07/network-sweeping-with-clojure.htm... http://travis-whitton.blogspot.com/2009/06/clojures-agents-scientists-monkey... It could be very useful to employ massive parallelism via such Clojure scripts, which could achieve a xwiki-based web portal performance akin to Yahoo's, Google's, etc. For example, the following describes how Yahoo works -- and would be quite easy to implement this kind of processing "for free" in Clojure with very little code: http://research.yahoo.com/files/pnuts.pdf The component responsible for multi-record requests is called the
scatter-gather engine, and is a component of the router. The scatter-gather engine receives a multi-record request, splits it into multiple individual requests for single records or single tablet scans, and initiates those requests in parallel. As the requests return success or failure, the scatter-gather engine assembles the results and then passes them to the client. In our implementation, the engine can begin streaming some results back to the client as soon as they appear. We chose a server-side approach instead of having the client initiate multiple parallel requests for several reasons. First, at the TCP/IP layer, it is preferable to have one connection per client to the PNUTS service; since there are many clients (and many concurrent processes per client machine) opening one connection to PNUTS for each record being requested in parallel overloads the network stack. Second, placing this functionality on the server side allows us to optimize, for example by grouping multiple requests to the same storage server in the same web service call.
Range queries and table scans are also handled by the scatter gather engine. Typically there is only a single client process retrieving the results for a query. The scatter gather engine will scan only one tablet at a time and return results to the client; this is about as fast as a typical client can process results. In the case of a range scan, this mecha- nism simplifies the process of returning the top-K results (a frequently requested feature), since we only need to scan enough tablets to provide K results. After returning the first set of results, the scatter-gather engine constructs and returns a continuation object, which allows the client to re- trieve the next set of results. The continuation object con- tains a modified range query, which, when executed, restarts the range scan at the point the previous results left off. Con- tinuation objects allow us to have cursor state on the client side rather than the server. In a shared service such as PNUTS, it is essential to minimize the amount of server- side state we have to manage on behalf of clients.
Hi, On Sun, Jul 26, 2009 at 1:43 PM, Niels Mayer <nielsmayer@gmail.com> wrote:
In
http://code.xwiki.org/xwiki/bin/view/Snippets/DisplayFormatedCodeAttachments...
VincentMassol | 2009/05/05 20:03
Niels, no there's no plan but we can now support any language using macros so feel free to provide a clojure macro if you're interested. Shouldn't be too hard to do, especially if there's a JSR-223 implementation (in which case simply dropping the jar in WEB-INF/lib should be enough - you'd then use it using the script macro).
Clojure <http://groups.google.com/group/clojure> has aJSR-223 implementation according to: http://github.com/pmf/clojure-jsr223/tree/master http://groovy.codehaus.org/JSR-223+access+to+other+JVM+languages
http://sayspy.blogspot.com/2009/03/interacting-between-jvm-lang-here-and.htm...
Are there any examples, documentation, or suggestions of writing a "script macro" to call a new jar in WEB-INF/lib ?
1. Put the Clojure jar file<http://repo1.maven.org/maven2/org/clojure/clojure/1.0.0/clojure-1.0.0.jar>inside WEB-INF/lib directory 2. Build the Clojure JSR223 engine from http://github.com/pmf/clojure-jsr223/tree/master and place it inside WEB-INF/lib 3. Restart your xwiki and use the script macro as below: {{script language="Clojure"}} // Put your clojure script here. {{/script}} (Note that since context is a reserved binding in JSR-223 specifications used for Scripting Macros, the XWiki Api Context is now accessible throught xcontext.) I haven't myself tried out clojure inside xwiki but I built a ruby macro<http://code.xwiki.org/xwiki/bin/view/Macros/RubyMacro>(for 2.0M2) just by following the steps I described to you :) Enjoy! - Asiri
Thanks! Vincent and Asiri! On Sun, Jul 26, 2009 at 1:43 AM, Asiri Rathnayake < asiri.rathnayake@gmail.com> wrote
I haven't myself tried out clojure inside xwiki but I built a ruby macro<http://code.xwiki.org/xwiki/bin/view/Macros/RubyMacro>(for 2.0M2) just by following the steps I described to you :)
So is 2.0M2 required for me to implement a clojure macro similar to your ruby macro? When will 2.0M2 be released? Vincent Massol wrote:
Well all you need is drop the JSR223 jar in WEB-INF/lib and then you
can use the new language using the script macro: http://code.xwiki.org/xwiki/bin/view/Macros/ScriptMacro The above link suggests I should be using at least 2.0M1 to get the following option, which seems necessary for generating content-data other than HTML pages w/o extra wiki markup messing up the syntax: wikiyestrue/falsetrueindicate if the result of the script execution has to be parsed by the current wiki parser. If not it's put in a verbatim block.2.0 M1 Therefore, should I try this out with 2.0M1, or a current snapshot? I'm definitely +1 to make it part of the XWiki platform (same as for jruby) if you develop it. Similar to jruby I don't think we should package it in the default WAR though but as an optional download to install (will be even easier with the app manager). I'm willing to try. Please let me know what version of XE will give the best success at this integration. Once I get things working I will look more closely at creating a package analogous to Asiri's Ruby, except it will integrate Clojure. And yes I'd be willing to develop this package once I get things working. One of the things I'll need to do is generate content-type other than HTML as the result of a page containing a Clojure script. I'm particularly interested in outputting JSON (plaintext). I assume this will be feasible in 2.0M1 and later? I Looking forward to using a language designed explicitly for "list processing" to create lists of JSON data! Niels http://nielsmayer.com
On Tue, Jul 28, 2009 at 10:49, Niels Mayer<nielsmayer@gmail.com> wrote:
Thanks! Vincent and Asiri!
On Sun, Jul 26, 2009 at 1:43 AM, Asiri Rathnayake < asiri.rathnayake@gmail.com> wrote
I haven't myself tried out clojure inside xwiki but I built a ruby macro<http://code.xwiki.org/xwiki/bin/view/Macros/RubyMacro>(for 2.0M2) just by following the steps I described to you :)
So is 2.0M2 required for me to implement a clojure macro similar to your ruby macro? When will 2.0M2 be released?
Vincent Massol wrote:
Well all you need is drop the JSR223 jar in WEB-INF/lib and then you
can use the new language using the script macro:
http://code.xwiki.org/xwiki/bin/view/Macros/ScriptMacro
The above link suggests I should be using at least 2.0M1 to get the following option, which seems necessary for generating content-data other than HTML pages w/o extra wiki markup messing up the syntax: wikiyestrue/falsetrueindicate if the result of the script execution has to be parsed by the current wiki parser. If not it's put in a verbatim block.2.0 M1 Therefore, should I try this out with 2.0M1, or a current snapshot?
I'm definitely +1 to make it part of the XWiki platform (same as for
jruby) if you develop it. Similar to jruby I don't think we should
package it in the default WAR though but as an optional download to
install (will be even easier with the app manager).
I'm willing to try. Please let me know what version of XE will give the best success at this integration. Once I get things working I will look more closely at creating a package analogous to Asiri's Ruby, except it will integrate Clojure. And yes I'd be willing to develop this package once I get things working.
One of the things I'll need to do is generate content-type other than HTML as the result of a page containing a Clojure script. I'm particularly interested in outputting JSON (plaintext). I assume this will be feasible in 2.0M1 and later? I Looking forward to using a language designed explicitly for "list processing" to create lists of JSON data!
Yes you can work with 2.0M1 or even 1.9.2. 2.0M2 is supposed to be released today. You don't need to do anything special at macro level, to generate plain text (for JSON for example) you need to use plain text renderer (xhtml renderer is the default). See http://platform.xwiki.org/xwiki/bin/view/AdminGuide/URL+resolution#HoutputSy....
Niels http://nielsmayer.com _______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
(I forgot to send this out a while back, probably because I didn't want to distract myself with thinking about integrating Xwiki and Clojure any more at the time :-) ) Got an interesting reply back from the Clojure list regarding JSR-223 support: http://groups.google.com/group/clojure/msg/ad811b69d448e3db (see below). Regarding this message, and Xwiki's JSR-223 API usage, a question: Does Xwiki's JSR-223 usage rely on the "Invocable interface" or the "Compilable-interface" ? Looking into what this all means, http://java.sun.com/developer/technicalArticles/J2SE/Desktop/scripting/ states Invokable is optional:
Script engines are not required to support the Invocable interface. However, the Rhino JavaScript technology implementation included in JDK 6 does. If your script contains a function called sayHello, you could invoke it repeatedly by casting your ScriptEngine object to an Invocable object and by calling its invokeFunction method. Alternatively, if your script defines objects, you can call object methods using the invokeMethod method.
Meanwhile., the author, Phil Frank, claims that Compilable is not implemented for his JSR-223 bridge, and may not be necessary for Clojure:
The Compilable-interface is a bit harder to do, and given Clojure's AOT compilation, I'm not sure if there's really a use case (because if you use require within a small script, Clojure will already out of the box transparently choose a compiled implementation, given a proper classpath setup). It's not quite the same, but comes pretty close.
Does Xwiki need anything from this Compilable interface, or will it's lack of implementation be a problem? As to the "active instance" issues brought up -- will these apply when running multiple active instances out of a web container like tomcat? According to http://tomcat.apache.org/tomcat-6.0-doc/class-loader-howto.html there's a separate classloader per web application, which means, I guess, that the JSR-223 clojure implementation will share any static initialization from the Clojure implementation:
WebappX - A class loader is created for each web application that is deployed in a single Tomcat 6 instance. All unpacked classes and resources in the /WEB-INF/classes directory of your web application archive, plus classes and resources in JAR files under the /WEB-INF/lib directory of your web application archive, are made visible to the containing web application, but to no others. ... As mentioned above, the web application class loader diverges from the default Java 2 delegation model (in accordance with the recommendations in the Servlet Specification, version 2.3, section 9.7.2 Web Application Classloader).
So if all Clojure under Xwiki is shared via a common "WebAppX" classloader under the Servlet Spec, does this mean that all Clojure instances would share a :"symbol table" and their values/properties? This could either become a useful feature, or an egregious security hole in a multi-wikli environment. ---------------- From: Phil Frank To: Clojure <clojure@googlegroups.com> On Aug 1, 2:36=A0am, Niels Mayer <nielsma...@gmail.com> wrote:
PS: I've always seen xwiki as the "emacs of webapps" (and wikis)... So I'm looking forward to having a real emacsish type language -- clojure -- to extend it via a more appropriate language for scripting.
I'm the author of the JSR 223 bridge for Clojure. At the moment, I have several uncommitted changes (mainly related to wrapping it up in a nice OSGi service component (to be used optionally). The Invocable- stuff is not implemented at the moment, but will be painless to implement. The Compilable-interface is a bit harder to do, and given Clojure's AOT compilation, I'm not sure if there's really a use case (because if you use require within a small script, Clojure will already out of the box transparently choose a compiled implementation, given a proper classpath setup). It's not quite the same, but comes pretty close. Aside from that, I have some unresolved design issues regarding isolation of engines; this works differently for Clojure than for other dynamic languages for the reasons explained below. Due to the static initialization of the Clojure runtime, it is much harder to create different independent Clojure-engine instances, since this would require loading each Clojure-runtime under a separate classloader, which I am not opposed to, but was unable to implement. At the moment, I'm using a weird kind of isolation in which each Clojure-engine runs in its own thread, using a shared runtime, and bindings are isolated by pushing and popping them. I'm going to abandon this rather stupid approach and use a simpler "one namespace per engine"-scheme. This change should (hopefully) be transparent for little scripts that don't have their own explicit namespaces. If your scripts use user-defined namespaces, I'll just assume you break this isolation intentionally (which is true to the meaning of namespaces, after all). If anybody would like to look into the classloader-based isolation, I'd happily to integrate this change (even if this means a severe startup-overhead for the creation of each engine-instance; it's simply the proper way to do it). For production (at least if you plan to have more than one active engine instance), I'd recommend to wait for either the namespace-based isolation or (should anybody get involved) the classloader-based approach. I'll reply to this thread again if one of these approaches is implemented.
Hi, On Mon, Aug 31, 2009 at 04:01, Niels Mayer<nielsmayer@gmail.com> wrote:
(I forgot to send this out a while back, probably because I didn't want to distract myself with thinking about integrating Xwiki and Clojure any more at the time :-) )
Got an interesting reply back from the Clojure list regarding JSR-223 support: http://groups.google.com/group/clojure/msg/ad811b69d448e3db (see below). Regarding this message, and Xwiki's JSR-223 API usage, a question: Does Xwiki's JSR-223 usage rely on the "Invocable interface" or the "Compilable-interface" ? Looking into what this all means, http://java.sun.com/developer/technicalArticles/J2SE/Desktop/scripting/ states Invokable is optional:
Script engines are not required to support the Invocable interface. However, the Rhino JavaScript technology implementation included in JDK 6 does. If your script contains a function called sayHello, you could invoke it repeatedly by casting your ScriptEngine object to an Invocable object and by calling its invokeFunction method. Alternatively, if your script defines objects, you can call object methods using the invokeMethod method.
Meanwhile., the author, Phil Frank, claims that Compilable is not implemented for his JSR-223 bridge, and may not be necessary for Clojure:
The Compilable-interface is a bit harder to do, and given Clojure's AOT compilation, I'm not sure if there's really a use case (because if you use require within a small script, Clojure will already out of the box transparently choose a compiled implementation, given a proper classpath setup). It's not quite the same, but comes pretty close.
Does Xwiki need anything from this Compilable interface, or will it's lack of implementation be a problem?
For now XWiki simply call ScriptEngine.eval(String, StringContext). The plan for latter is to use Compilable (when possible, a ScriptEngine is not required to implement it AFAIK so XWiki would still support implementation without Compilable) to be able to do some caching of the compiled version.
As to the "active instance" issues brought up -- will these apply when running multiple active instances out of a web container like tomcat? According to http://tomcat.apache.org/tomcat-6.0-doc/class-loader-howto.html there's a separate classloader per web application, which means, I guess, that the JSR-223 clojure implementation will share any static initialization from the Clojure implementation:
WebappX - A class loader is created for each web application that is deployed in a single Tomcat 6 instance. All unpacked classes and resources in the /WEB-INF/classes directory of your web application archive, plus classes and resources in JAR files under the /WEB-INF/lib directory of your web application archive, are made visible to the containing web application, but to no others. ... As mentioned above, the web application class loader diverges from the default Java 2 delegation model (in accordance with the recommendations in the Servlet Specification, version 2.3, section 9.7.2 Web Application Classloader).
So if all Clojure under Xwiki is shared via a common "WebAppX" classloader under the Servlet Spec, does this mean that all Clojure instances would share a :"symbol table" and their values/properties? This could either become a useful feature, or an egregious security hole in a multi-wikli environment.
I don't know, it depends in the Clojure implementation of JSR223 plus i'm not a classloade/servlet expert.
---------------- From: Phil Frank To: Clojure <clojure@googlegroups.com>
On Aug 1, 2:36=A0am, Niels Mayer <nielsma...@gmail.com> wrote:
PS: I've always seen xwiki as the "emacs of webapps" (and wikis)... So I'm looking forward to having a real emacsish type language -- clojure -- to extend it via a more appropriate language for scripting.
I'm the author of the JSR 223 bridge for Clojure. At the moment, I have several uncommitted changes (mainly related to wrapping it up in a nice OSGi service component (to be used optionally). The Invocable- stuff is not implemented at the moment, but will be painless to implement. The Compilable-interface is a bit harder to do, and given Clojure's AOT compilation, I'm not sure if there's really a use case (because if you use require within a small script, Clojure will already out of the box transparently choose a compiled implementation, given a proper classpath setup). It's not quite the same, but comes pretty close.
Aside from that, I have some unresolved design issues regarding isolation of engines; this works differently for Clojure than for other dynamic languages for the reasons explained below.
Due to the static initialization of the Clojure runtime, it is much harder to create different independent Clojure-engine instances, since this would require loading each Clojure-runtime under a separate classloader, which I am not opposed to, but was unable to implement.
At the moment, I'm using a weird kind of isolation in which each Clojure-engine runs in its own thread, using a shared runtime, and bindings are isolated by pushing and popping them.
I'm going to abandon this rather stupid approach and use a simpler "one namespace per engine"-scheme. This change should (hopefully) be transparent for little scripts that don't have their own explicit namespaces. If your scripts use user-defined namespaces, I'll just assume you break this isolation intentionally (which is true to the meaning of namespaces, after all).
If anybody would like to look into the classloader-based isolation, I'd happily to integrate this change (even if this means a severe startup-overhead for the creation of each engine-instance; it's simply the proper way to do it).
For production (at least if you plan to have more than one active engine instance), I'd recommend to wait for either the namespace-based isolation or (should anybody get involved) the classloader-based approach. I'll reply to this thread again if one of these approaches is implemented. _______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
participants (3)
-
Asiri Rathnayake -
Niels Mayer -
Thomas Mortagne