XWiki and SVN as a backend
Hi, I'm investigating what it would take to store (or duplicate) some of the data in XWiki's database in Subversion. Rationale: - A plain text version of wiki pages stored in SVN buys you: - off-line edit, merge, etc - access to the pages using other tools, editors and so on - replication options - versionning, branching etc, and version display via e.g. ViewCVS - insulation from database crashes, with easier incremental backup - a kind of poor-man's content management system - SVN is very robust, reliable, and has a mature Java interface (SVNKit) - if Tomcat crashes a static version of the site can still be served - I have a large static site that is managed in SVN (8 Gb in 40,000 files) which I want to migrate to a more dynamic system. This looks like a significant size job, but we have a fair amount of resources in my team, and possibly we might fund some of the work in the XWiki community if anyone is interested. I wonder if this type of idea has been discussed before, and if anyone has comments on the best way to go about the problem? It seems to me that trying to replace the database entirely with a file-based store is too much work and would lead to performance problems, but perhaps it would be possible to have a process which pushed changes from the database into an SVN working copy and vice versa? Where would be good places to start looking at the code from this point of view? I also need to include a JavaCC-based wiki-to-html parser that we have as an option, but I'm betting that's an easier task? Thanks, best, Hamish -- Prof. Hamish Cunningham Department of Computer Science University of Sheffield Regent Court 211 Portobello St. Sheffield S1 4DP United Kingdom http://www.dcs.shef.ac.uk/~hamish/
Hi Hamish, On Mar 30, 2007, at 4:06 PM, Hamish Cunningham wrote:
Hi,
I'm investigating what it would take to store (or duplicate) some of the data in XWiki's database in Subversion.
Rationale:
- A plain text version of wiki pages stored in SVN buys you: - off-line edit, merge, etc - access to the pages using other tools, editors and so on - replication options - versionning, branching etc, and version display via e.g. ViewCVS - insulation from database crashes, with easier incremental backup - a kind of poor-man's content management system - SVN is very robust, reliable, and has a mature Java interface (SVNKit) - if Tomcat crashes a static version of the site can still be served - I have a large static site that is managed in SVN (8 Gb in 40,000 files) which I want to migrate to a more dynamic system.
Yep I agree with most of these but there are limitations (see below).
This looks like a significant size job, but we have a fair amount of resources in my team, and possibly we might fund some of the work in the XWiki community if anyone is interested.
That would be real cool.
I wonder if this type of idea has been discussed before, and if anyone has comments on the best way to go about the problem?
Yes we've discussed this in the past. Someone even said he had started working on it but I tried to get a status of the work and never got an answer so we can consider this dead. That said, we've also opened a Google Summer of Code project for this (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/Subversion +Integration). If you're interested you could work with your team and that person on it. We could create a branch in SVN for this work.
It seems to me that trying to replace the database entirely with a file-based store is too much work and would lead to performance problems, but perhaps it would be possible to have a process which pushed changes from the database into an SVN working copy and vice versa?
Yes, this is needed for several reasons: 1) Performances but that's not really the main issue as we could use caching for searching (lucene, etc). 2) XWiki pages can contain objects and we don't currently have a textual representation of a full page with it's objects, etc apart from an XML representation but that's not nice for editing at all. Thus 2 thoughts: 1) It's probably better to decide what pages are stored in SVN and which pages are stored in the wiki. Indeed some pages such as Classes probably don't need to be stored in SVN 2) We need to allow storing the page content separately from the rest of the metadata (class definition, objects, etc) Based on this, we could possibly implement in 2 ways (haven't thought more about it): 1) by implementing a XWikiStoreInterface. That means it would get stored both in the DB and in SVN for some docs. We would chain the storage systems (that probably requires some minor changes to the core) 2) by using the XWiki notification service. I know there's an event called when storing. I don't know if there's an event for getting back a doc from storage. This possibly requires some small changes to the core. Obviously one issue is that if we do a svn update before editing a document (to check if there's a newer version) that'll affect performances. Thus we might want to schedule updates regularly or offer a button for the user to do that. In any case it means we need to cater for SVN merge conflicts and offer a solution for that. Anyone has any other idea? I suggest we gather ideas here on the list and when we have some agreement we can start putting those on a wiki page on xwiki.org. Hamish, I'd be happy if you wanted to drive this effort.
Where would be good places to start looking at the code from this point of view?
See above for some ideas.
I also need to include a JavaCC-based wiki-to-html parser that we have as an option, but I'm betting that's an easier task?
Actually Christian and Nam are developing one right now. It would be cool if you could have a look and possibly help them. It's in SVN in xwiki/branches/XWIKI_WYSIWYG_NEWARCHI/core/src/main/java/com/xpn/ xwiki/wysiwyg Thanks -Vincent
vincent
That said, we've also opened a Google Summer of Code project for this (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/Subversion+Integratio...). If you're interested you could work with your team and that person on
sounds good!
1) It's probably better to decide what pages are stored in SVN and which pages are stored in the wiki. Indeed some pages such as Classes probably don't need to be stored in SVN 2) We need to allow storing the page content separately from the rest of the metadata (class definition, objects, etc)
I agree that not all of the data needs to be in SVN, but I think it would be nice if some of the textual content of pages was available for external editing and so on. An advantage of pushing all the data out into some textual form (even if it isn't very readable) is that you could then use SVN replication (or just checking out another copy) to duplicate a site, and branching for experimenting with a new approach, etc.
Obviously one issue is that if we do a svn update before editing a document (to check if there's a newer version) that'll affect performances. Thus we might want to schedule updates regularly or offer a button for the user to do that. In any case it means we need to cater for SVN merge conflicts and offer a solution for that.
Anyone has any other idea?
One option is to get the SVN server to hit a URL whenever anything changes and trigger import of the changes at that point. Some other mechanism (like the scheduled update you propose) is also necessary to cope with network outages etc.
I suggest we gather ideas here on the list and when we have some agreement we can start putting those on a wiki page on xwiki.org. Hamish, I'd be happy if you wanted to drive this effort.
I'd like to participate; at present I don't understand well how xwiki works, so I'm not sure I'm the right person to drive it.
Actually Christian and Nam are developing one right now. It would be cool if you could have a look and possibly help them. It's in SVN in xwiki/branches/XWIKI_WYSIWYG_NEWARCHI/core/src/main/java/com/xpn/xwiki/wysiwyg
We're using a different syntax (see attached) and allowing embedding of controlled language statements for ontology editing (http://gate.ac.uk/sale/lrec2006/clie/clie.pdf). Actually if I was going to do this again (write a wiki parser) I'd be tempted not to use JavaCC, which is great for fully deterministic parsing but difficult when you want to return a reasonable translation of messy data even when the author has made mistakes. Our JAPE regular expressions over annotation graphs tool might have been easier (http://gate.ac.uk/, imperfectly documented in http://gate.ac.uk/sale/tao/index.html), but I've invested too much time in the JavaCC version now! So this wouldn't replace your existing parser efforts, but I'd like to try and make it an optional syntax. Best, -- Hamish http://www.dcs.shef.ac.uk/~hamish/
Hi, A quick&simple solution would be to make a cron task that dumps the content of the wiki periodically (it is done now in XML format) and performs a SVN commit. Of course, this only resolves the backup part. See below for some more comments. On 3/30/07, Hamish Cunningham <hamish@dcs.shef.ac.uk> wrote:
vincent
That said, we've also opened a Google Summer of Code project for this ( http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/Subversion+Integratio... ). If you're interested you could work with your team and that person on
sounds good!
1) It's probably better to decide what pages are stored in SVN and which pages are stored in the wiki. Indeed some pages such as Classes probably don't need to be stored in SVN 2) We need to allow storing the page content separately from the rest of the metadata (class definition, objects, etc)
I agree that not all of the data needs to be in SVN, but I think it would be nice if some of the textual content of pages was available for external editing and so on. An advantage of pushing all the data out into some textual form (even if it isn't very readable) is that you could then use SVN replication (or just checking out another copy) to duplicate a site, and branching for experimenting with a new approach, etc.
Since the wiki documents can be retrieved as XML, separating the document content and document metadata is an easy task, which can be performed with many possible methods: SAX, DOM, XSLT. Moreover, since your documents will mainly contain text, without the need for objects and classes, it might be a good idea to store just the content and simple metadata (author and modification time) in SVN. Even more, XWiki has the notion of spaces, or a group of related documents. You can put all your documents in spaces which don't have any XWiki-related content (special class documents, or templates), and only sync these spaces with the SVN repository. About branching XWiki can be of use in here, since it supports setting up virtual wikis on the same server.
Obviously one issue is that if we do a svn update before editing a
document (to check if there's a newer version) that'll affect performances. Thus we might want to schedule updates regularly or offer a button for the user to do that. In any case it means we need to cater for SVN merge conflicts and offer a solution for that.
Anyone has any other idea?
One option is to get the SVN server to hit a URL whenever anything changes and trigger import of the changes at that point. Some other mechanism (like the scheduled update you propose) is also necessary to cope with network outages etc.
This is a good idea. I know that SVN allows registering a kind of listeners (Hook Scripts is the term they use) which can be triggered before or after committing files. One hook could be used to re-get the content from the wiki when starting the commit, and one could be used to update the wiki after the commit. See http://svnbook.red-bean.com/en/1.0/ch05s02.html#svn-ch-5-sect-2.1 for more details.
I suggest we gather ideas here on the list and when we have some
agreement we can start putting those on a wiki page on xwiki.org. Hamish, I'd be happy if you wanted to drive this effort.
I'd like to participate; at present I don't understand well how xwiki works, so I'm not sure I'm the right person to drive it.
Actually Christian and Nam are developing one right now. It would be cool if you could have a look and possibly help them. It's in SVN in
xwiki/branches/XWIKI_WYSIWYG_NEWARCHI/core/src/main/java/com/xpn/xwiki/wysiwyg
We're using a different syntax (see attached) and allowing embedding of controlled language statements for ontology editing (http://gate.ac.uk/sale/lrec2006/clie/clie.pdf). Actually if I was going to do this again (write a wiki parser) I'd be tempted not to use JavaCC, which is great for fully deterministic parsing but difficult when you want to return a reasonable translation of messy data even when the author has made mistakes. Our JAPE regular expressions over annotation graphs tool might have been easier (http://gate.ac.uk/, imperfectly documented in http://gate.ac.uk/sale/tao/index.html), but I've invested too much time in the JavaCC version now!
So this wouldn't replace your existing parser efforts, but I'd like to try and make it an optional syntax.
This is easy to accomplish. For most of these items there are existing filters, only the syntax must be changed, and the rest of them are easy to add. In order to change the syntax, you just need to change a file, core/src/main/resources/radeox_markup_xwiki.properties. In order to add new filters, I think there is a page describing this on www.XWiki.org. Anyway, it's not hard at all. Best,
-- Hamish http://www.dcs.shef.ac.uk/~hamish/ <http://www.dcs.shef.ac.uk/%7Ehamish/>
Regards, Sergiu Dumitriu -- http://purl.org/net/sergiu
participants (3)
-
Hamish Cunningham -
Sergiu Dumitriu -
Vincent Massol