L-P Huberdeau's blog

After nearly 4 years, I am finally done with the translation synchronization problem. The Cross Lingual Wiki Engine project is far from completed, but at least the change tracking mechanism is. Looking in my own archives, I first wrote about this issue in August 2004. The solution I ended up implementing is in fact very close to what I wrote the second time around in 2006, except that I dropped most of the input required by the user.

The implementation is now part of the TikiWiki CVS and will be included in the 1.10 release (or however it’s going to be called) in a few weeks from now. As it’s licenced under LGPL, feel free to rip the implementation apart and include it in your own projects. If you do so, please tell us on Wiki-Translation. I will be glad to help out.

The architecture actually uses a single table and large SQL queries to do the work. In fact, the implementation is only around 125 lines of SQL and maybe 300 lines of simple PHP to work around the TikiWiki way of handling things. The core implementation took around 6 hours including a proof of concept, then I probably worked an other 10 hours working out the queries and extracting meaningful information.

The tasks accomplished by the SQL is fairly dense. You can also read the article detailing the theory and implementation.

I definitely spent more time writing the documentation than working in the implementation. In fact, writing is short enough. Reviewing is long. Finding problems is hard. Luckily, I had Alain Désilets and Sébastien Paquet to help me in that task. There may still be room for improvements and comments are welcome.

Why didn’t I do it back in 2006? Well, I thought it would have been more complicated, and the problem was mostly theory as I didn’t have an actual test site that would need to handle more than two languages. In fact, the current implementation is the child of many discussions I had along the way. If I had done it before, it wouldn’t have been as good. Couldn’t have been without all those discussions at RoCoCo and WikiSym2007. They brought special use cases and influenced the results to cover much broader situations and do it with simplicity.

In the past few days, I spent a lot of time working on the Cross Lingual Wiki Engine Project. I finally found the solution to change tracking across multiple languages without restricting contributions. In fact, the core of it only took a few hours to write. Then I had to spend a few more to mine information out of the table. Most of the time I spent on the project recently was to write an article on how the thing works.

My primary purpose in writing the article in the first place was to explain the article to other people working with me on the project. I made a few attempts before writing it down and it didn’t work out so well. I figured I was better off trying to structure my mind before trying to communicate, and I know of no better way to do so than by writing. Since the article is part of an academic project, I figured I should give it an academic feel (and I don’t mean to make it boring). I started writing it in LaTeX using LyX. The tools are just amazing. Using them kicks me in flow in a matter of minutes. During my writing session at the Pub, I never noticed 5 hours had passed, sun had went down and the room became crowded.

Anyway, in giving it some academic feel, I went down to basic math concepts like sets and graphs. It’s quite nice because the theory maps really well to the problem. In fact, it was probably a strong influence. In all those years thinking about the problem, I followed multiple math courses and one of them was discreet maths, so I guess it oriented me towards the solution. It turned out just mapping the architecture to those concepts allowed me to explain even more details. Mathematics are a very expressive language and it has a set of solutions to many problems if you can express them correctly.

So for pages and pages, I go on explaining how it all works in math terms. Then comes the implementation section where I actually explain how it works using real technologies. Today, after the review of a first draft, I decided to add an explanation of an additional query, which happens to be quite central. I didn’t know that before. I went on to describe it and explain in which ways it is correct. That was until I realized it was wrong. Simply not accurate. False in all possible ways. I had the query rewritten twice already because I forgot some corner conditions the two first times. I was pretty certain it was good this time. Mapping it back to theory, I realized how wrong I was.

Then I went back to my code and made an attempt to catch that newly discovered corner condition. After expanding the query significantly (it had 4-5 levels of subqueries by that time), I realized that last level of nesting was really close to the real purpose. I went on and removed some code, and then more. The final expression is so simple. All I did before was run in the wrong direction.

I can’t be certain it’s right now, but at least it fits the model. My advice of the day: when working on a though data modeling problem, try to explain it in terms an academic would understand. Write the formulas and prove them. Draw diagrams using GraphViz or restrict your diagrams to 2 primitives. Write it in LaTeX to give it all an old-school academic look. The final document will be good, but nothing of value compared to what you will learn on the way.

I still need to make a few changes based on the last review, and will probably go for a second round of review, but that article will eventually be published somewhere. Stay tuned.