XWiki

The end of the year 2009 is getting close and it's time to look back at what the XWiki open source project has achieved during this year.

2009 has been a tremendous year for the XWiki project, establishing XWiki as one of the best enterprise wiki (if not the best, but I'll let you be the judge on that ). I'm eager to see what 2010 will yield.

I'd like to take the opportunity to congratulate all the XWiki committers and all the XWiki contributors and users who've made this possible. Thanks everyone!

Releases in 2009

The XWiki dev team has released 58 versions in total for XWiki Enterprise, XWiki Enterprise Manager, XWiki Office and XWiki Eclipse. Not bad hey! More than 1 release per week as average. Here are the details:

XWiki Enterprise Features in 2009

Here's a list of major features developed in 2009 for XWiki Enterprise:

• WYSIWYG editor based on GWT
• Skin (Colibri) + complete action menus overhaul
• Color themes for easily customizing colors
• Rendering engine, allowing Polyglotism
• More powerful wiki syntax (XWiki Syntax 2.0)
• Dashboard
• Livetables allowing filtering, searching, sorting of data extracted from wiki pages in real time
• Lots of UI improvements, among which:
  • Document footer overhaul
  • Jump to page
  • AJAX save
  • Full screen editing
  • Threaded comments
  • Class/Object editor overhaul
  • Multiple attachment support
• Performance improvements (greater than 30% speed improvements)
• Office Importer (Word, Powerpoint, Excel, Open Office)
• REST API
• Blog Application rewrite
• Watchlist improvements (scalability, support for wiki farms, improved email format)
• Lots of new Macros and Applications available on the Code Zone
• Introduction of Wiki Macros to develop macros directly inside wiki pages (and have them available from the WYSIWYG editor)
• Support for more scripting languages (in addition to Groovy which has been supported since the beginning): Ruby, Python, PHP
• Professional PDF export
• Clustering and Distributed events

I'm pretty sure I've forgotten a lot of new features but there are just too many to list them all.

Downloads in 2009

Downloads have kept increasing from the past year (almost doubled).

Mailing list Activity in 2009

The mailing lists have continued to see their activity increase, which means more users and more developers/contributors. Here's the figure of emails posted on the XWiki lists (each value is per month):

Note that this screenshot was taken on the 16th of December 2009 which explains why the last month has a low number of emails.

Active Committers and Contributors in 2009

We have 14 active committers, amongst which a good portion committing daily.

One area of improvement for 2010 is to redesign xwiki.org to make it more attractive and to value more people's contributions, and make people be more active on it, especially in the area of contributing macros, applications, code snippets, etc.

So far we have on Code Zone:

• 84 code snippets available
• 81 macros available
• 30 plugins available
• 50 applications available

I wish you all a very good Christmas season and a very good new year with lots of good wishes for improving XWiki in 2010.

Starting with XWiki Enterprise 1.7 we've released a new wiki syntax for XWiki (Since XWiki is now Polyglot you can continue to use the old XWiki syntax or use other wiki syntaxes). We're calling this new syntax by the delicious and original name of "XWiki 2.0 Syntax".

Here are some salient points of this new syntax:

• More consistent syntax and less error prone. All special syntax characters such as bold symbols, italics symbols, link symbols, etc all use 2 characters to prevent ambiguity in case you wish to use that character in your text. For example for bold you can write: 

• Very close to Creole 1.0 (see the Creole 1.0 syntax). The goal of Creole is "a common wiki markup language to be used across different wikis. It's not replacing existing markup but instead enabling wiki users to transfer content seamlessly across wikis, and for novice users to contribute more easily".
• Meaningful new lines and spaces. This mean new lines are honored unlike traditional wiki-style new lines which are ignored. This was the number one feature asked by our users.
• Ability to enter wiki syntax in link labels. For example to add an image link you'd write: 

• Ability to pass parameters to any syntax elements including the ability to easily style them. For example to style a portion of text or a section header you can use: 

• HTML must now be entered using the HTML macro. This allows to enter non-interpreted HTML in the main text. It also provides security control to wiki admins if they want to prevent users from entering HTML (and thus potentially harmful javascript). In addition the HTML macro automatically converts invalid HTML into valid XHTML.
• Easy linking to attachments using the link syntax. For example: 

• Strong table syntax including ability to have table headers as columns or as rows.
• Added scripting macros to enter XWiki scripts (We support JSR 223). For now we have official support for Velocity, Groovy and Jython, meaning that the full XWiki API is available directly to you using any of these scripting languages.
• Email-style quotations. For example: 

• Ability to group syntax elements together and thus support very complex use cases such as a table inside a list item inside a table. For example: 

• The syntax is powerful enough to represent most HTML elements thus allowing us to have a 100% bidirectional WYSIWYG editor with no information loss.
• Syntax for monospace, verbatim, definition lists, and more.

See the XWiki 2.0 Syntax Guide for more details. Note that we have an automatic converter that converts XWiki 1.0 syntax into 2.0 syntax.

We hope you'll enjoy this new syntax which we've tried to make the most powerful wiki syntax out there. Feedbacks welcome!

XWiki Enterprise allows users to leave comments on pages. However in order to prevent spam on your public wiki instance you usually only want to allow registered users the right to add comments. Thus we need a solution that still allows guest users to leave comments while preventing spam.

I'm proposing 2 solutions that I've both tried on this blog and that have worked well: Solution 1: create a special guest account that can be used transparently to leave comments. This can be achieved by creating a custom skin and tweaking the comments form Solution 2: integrate with an external comment web service such as IntenseDebate. This also requires a custom skin in order to override some templates.

Solution 1: Special Guest account

• Create a "Guest" user by going to the Administration page.
• Modify the Skins class so that the comments property points to the commentsinline.vm file (instead of the comments.vm one). Go to the XWiki.XWikiSkins page and change the Name field for the Comments property to "commentsinline.vm". Save.
• In the Administration page, click on the Presentation icon and then click on the "Customize" button in the Skin section. Click "Edit this Skin" to edit the skin.
• Modify the "Style" property field for better avatar display with the following content.
• Modify the "Comments" property field with the following content. This change adds several fields to the comments form that guest users can fill and provide a direct link to log in guest users.

When done you should get the following comment feature:

Solution 2: Integration with IntenseDebate

• Register on IntenseDebate to get a key to interact with their system.
• Modify the Skins class to add 2 new fields so that we can override them. Go to the XWiki.XWikiSkins page and use the right panel to add 2 text area fields named "htmlfooter.vm" and "docextra.vm". Save.
• In the Administration page, click on the Presentation icon and then click on the "Customize" button in the Skin section. Click "Edit this Skin" to edit the skin.
• Modify the newly added "htmlfooter.vm" property field by adding the following content.
• Modify the newly added "docextra.vm" property field by adding the following content. Make sure you modify the replace-with-your-intense-debateid-here value in the pasted content with your IntenseDebate id.

Here's what you'd get:

IntenseDebate is really nice and it offers the following features:

• Comment threading
• Reply by email
• Importing / exporting comments
• Commenter profiles
• Reputation points and comment voting
• Moderation / blacklisting, Profanity filter and Akismet spam filtering
• Widgets
• RSS feeds and reader tracking
• Twitter / friendfeed integration
• OpenID authentication
• Commenter gravatar
• HTML formatting

Most people probably think that XWiki is a wiki. This is the same as saying that Eclipse is a Java IDE.

In the same manner that Eclipse is a generic platform for developing applications, XWiki is a platform for developing any type of collaborative web applications. You can view it as an Application Server offering high level services relevant to developing collaborative applications. Example of such services are: Versioning service, Document service, Storage service, Attachment service, Authentication/Authorization service, WYSIWYG editing service, Wiki Service, and more as shown on the diagram below

Note that the list of services listed on the diagram is far from exhaustive and there are more since the image was created (like the GWT API, Scheduling service, REST API, etc).

So what type of collaborative applications can you develop with XWiki? Here are some examples:

• Collaborative RSS reader. By the way there's an instance running here for monitoring everything about XWiki that is said on the web.
• Workspace-based collaboration tool
• Merchant website. I'm sure you didn't recognize XWiki on that site! If you're like me and don't recognize XWiki check this link.
• Project Management tool. Note that development for it has been stopped.
• Online educational course creation and sharing
• A lot more... See other examples of XWiki usages

I was invited to present the XWiki project at the GlassFish TV yesterday. 

That was fun: I was at home with a headset on a conf call (I was using skype), presenting a presentation I had sent earlier on to my host (Jacob Kessler from Sun). Jacob was recording live the audio and the presentation slides. I'm now eager to see the result and I hope the voice quality isn't too bad.

I presented quickly what XWiki is and moved to show XWiki's growth in 2008 and then moved on to the meat of the talk which were the new features the XWiki project is releasing with XWiki Enterprise 1.8.

Here are the slides (note that Slideshare didn't do such a good job of converting my ODP presentation so you might want to download the slides):

I had the pleasure of presenting the XWiki 2009 Roadmap at a breakfast event organized by XWiki SAS.

I started the presentation by showing XWiki project stats for 2008, showing how active XWiki development is (see slides 3 and 4):

• Downloads in 2008: > 130K (double the downloads of 2007)
• Number of releases in 2008: 65 (13 final releases and 52 milestone releases)
• Active committers: 14 (double the active committers in 2007)
• Mailing list activity is exploding since 2008

Live stats can be also be found on SVNSearch (See this blog post for details on SVNSearch).

Here are the slides I presented (sorry they are in French and I did several demos you cannot see obviously):

As of January 2009 there are now 14 active open source committers who work daily on the XWiki code. The number of active committers have grown tremendously in the past year (we were roughly half of that a year ago). As a consequence the upcoming XWiki Enterprise release will be feature rich (some of the features are already in XWiki Enterprise 1.7) and the following features are all nearing completion:

• New Wiki Syntax 2.0
• New Rendering engine
  • Much faster
  • Way more powerful. We're now parsing document content and generating an internal object structure for it. Amongst other things it allows scripts to act on page content (semantic transformations are now possible and easy).
  • Polyglot. We now support several wiki syntaxes at once (you can choose the syntax on a page by page basis).
• New WYSIWYG editor (based on GWT). We've rewritten from scratch our WYSIWYG editor to provide an editor that is performant, robust and that just works (none of the WYSIWYG editors we've tried out there were working! More on that in a future post)
• Full WebDAV support
• Office Import. All types of documents are going to be supported (old Microsoft Office formats, Open Office, etc).
• Microsoft Office Plugin for editing and publishing wiki content directly from Microsoft Word.
• New Query Manager (called XWQL, it's a JPQL superset) to easily create queries to query anything in the wiki. For example: 

• XWiki IDE (a.k.a XEclipse) with Syntax Coloring and auto completion on APIs:{

... and more (subject for another post)

All these are planned for March 2009. 2009 is looking to be a great year for the XWiki project.

We're just finishing rewriting from scratch the XWiki rendering engine (it's already usable in XWiki Enterprise 1.7 and will be completely finished in 1.8). We were using Radeox but it's been dead for several years and it was just too limitating and slow (based on regex) for our need. As a consequence we now have a state of the art rendering engine that supports multiple wiki syntaxes, hence making XWiki the first Polyglot Wiki.

Supported syntaxes other than the XWiki syntax are: Confluence, JSPWiki, MediaWiki, TWiki, Creole and HTML/XHTML (for the intrepid ones). In addition we'll be adding new syntaxes as we progress since it's now very easy to do so.

What this means is that if you like XWiki and you want to switch to it but you don't want to learn another wiki syntax, you will be able to switch and keep using your favorite syntax. XWiki is even able to convert on the fly between one syntax and another (currently implemented only for some syntaxes).

Technically we have parsers for each syntax which transforms text into an internal representation using Java Objects (called XDOM) which we then traverse and render using one the available Renderers (XHTML Renderer, XWiki Syntax 2.0 Renderer to convert to XWiki Syntax 2.0, PDF/RTF Renderer, etc), as shown in the diagram below.

Note: Not all syntax elements are fully implemented for all syntaxes and we still have lots of fine tuning to work on so don't expect something perfect yet for syntaxes other than XWiki 1.0, XWiki 2.0 or HTML/XHTML syntaxes. In term of Renderers we currently have the XHTML and XWiki 2.0 Syntax ones implemented. The others are planned for later.

Some of you may know about the XWiki open source project but may not know about the XWiki SAS company (Note that "SAS" simply refers to a certain type of company in France) or if you do you may wonder what are the relationship between them.

History

A bit of history: Ludovic Dubost created the XWiki open source project in 2003 on SourceForge. In 2004 Ludovic also created a company named XPertNet for offering support and services around the XWiki software. I joined the XPertNet company in December 2006 and XPertNet was renamed in XWiki SAS in 2008. 

Committers

As of December 2008 the XWiki SAS company has 30 employees (not all full time, about 25 FTE), out of which 15 are open source developers and 10 out of these 15 are committers on the XWiki open source project. Out of these 15 employees, 4 were recruited from the Google Summer of Code and several were recruited after being active committers on the open source project.

In total the open source project currently has about 12 active committers (about 70 in total since 2003). This means that the majority of committers are also employees of XWiki SAS. There are 2 reasons for this:

• As you have probably understood, XWiki SAS has recruited most of the active committers over the years.
• It's very hard for someone who's working on an open source project in his free time to work on an open source project sponsored by a company. The reason is that committers who are employees are working full time on the code and thus progressing quickly on it and it's hard for someone working 1 or 2 hours a day to play catch up. This is true for core development but it is fortunately not true for contributions in peripheral domains such as Macros, Code Snippets, Plugins or Applications. In these domains it's easy for anyone to contribute and we'd like to see even more growth in this area in the future.

Sponsoring

As mentioned above, XWiki SAS is paying 13 developers to develop for the open source project thus sponsoring a large portion of the project. In practice XWiki SAS reinvest more than 50% of its earnings in the growth of the XWiki open source project (the rest being spent on supporting the company): most of it in paying committers, in paying research developers (there are 2 currently) for French and European research projects and the rest in paying for the infrastructure and its support (xwiki.org, free XWiki farm, etc).

Update 20/02/2010: About a year ago XWiki SAS has published a Manifesto explaining its relationship with the open source project and highlighting its values.

Meritocracy

You may have noticed that not all XWiki open source developers from XWiki SAS are committers. This is because we've defined some meritocracy rules for participating to the XWiki open source project and we're strictly following them for anyone, be them from XWiki SAS or for external people interested in contributing. We're a real open source project and we're very clearly separating the company from the open source project. 

More precisely our open source project rules are the Apache Software Foundation ones (from where I come from: I had been working in my free time on open source projects for 7 years before joining the XWiki project and later joining XWiki SAS).

We do this since we all believe strongly in real open source and we're passionate about it. We certainly don't want to do like several other companies who jump on the open source bandwagon just to use it as a marketing ploy.

Conclusion

All our developments are fully done in the open and we're proud of that. We believe this generates a strong and level-playing ecosystem where people don't fear to contribute for fear of their contributions being "ripped" by a company.

We'd be happy to answer any question you may have about the company.

At long last I'm resurrecting my old blog and moving it to the XWiki platform (I've left the old blog posts there since I didn't want to loose comments and create broken links for people referring to the posts). 

Funnily I stopped blogging when I joined the XWiki open source project in December 2006. I think the main reason is that I have been busy understanding the existing code base, refactoring it, managing the development team and unconsciously I wanted to become an XWiki expert before I could start talking about it and evangelizing it in a more open manner. Since I'm reading "Outliers" from Malcolm Gladwell I think I now understand how everything fits together:

• I've been working on the XWiki project for 2 years. Since I spend 70+ hours per week on it, that's close to 8000 hours since I joined the project.
• Malcolm Gladwell says that you need roughly 10,000 hours to become an expert in a given field

So I'm now close to becoming an XWiki expert and this makes me understand the project pretty well and as a consequence I now feel entitled and mature to blog about it! Okay this is a bit contrived and there's probably more to it but it feels good to relate these concepts together.

So starting today you'll see lots of posts about the XWiki platform in general or the lessons I learn while developing it.