L-P Huberdeau's blog

Most software design out there is a matter of personal taste. There are very few widely agreed upon rules. It happened to all of us. You get to read a particularly bad piece of code and think it requires a complete rewrite. In most cases, it wouldn’t be hard to get people to agree with you. Rewriting would make everything more beautiful and allow easier modification. However, it has a terrible cost. It will always take longer than you expected. Bad code has this ability to hide features inside. While it may happen that some portions of code are dead, most of the time, they serve a very specific purpose your great new design wouldn’t have considered.

Major backend rewrites also tend to leave the front-end part behind. It will break user experience. All the polishing that was made on the interface is likely to be gone because it was not re-implemented or simply left broken. It would probably take months to reach the same level of external quality (compared to days or weeks to improve the internal quality).

When you are at point A and think it’s not the best place to be, there is nothing wrong with trying to go to point B. However, teleportation does not exist in the world will live in, and your app won’t just appear in point B on the next day. Even if you rewrite everything, get a perfect backend and a better front-end. All this legacy data won’t just transfer itself. Data conversions are a pain. One of the reason the code was so bad in the first place is probably because the data model was messed up. Converting data for all the edge conditions is a very long process and is always prone to break, which leads to users complaining. Of course, this implies that your upgrade process even has a way to handle data conversions as part of the regular upgrade process.

Before thinking about your grand new design, think about transition. It will probably expense more effort than the development efforts. After you rewrote everything, will you be able to make it from A to B?

Recently, I started working on restructuring the wiki plugin API in Tikiwiki. The plugins are great. They allow to find the different features in the wiki and create create applicative wikis. The problem is that they are hard to use. The best ones have too many parameters and not all of them are really well documented in the UI (while sometimes documentation is better on the doc site). When too many parameters are used, it just becomes unreadable. I decided to rework these during TikiFestStrasbourg after all of us learnt one new thing about plugin capabilities during discussions.

These were some of the issues:

• Documentation in UI was a short blob of text containing HTML. It was not meaningful to users and a pain for translators to manage. Parameters were rarely detailed.
• Plugin list was not filtered. All plugins were listed, even if related features were turned off. This created too much noise.
• The syntax was hard to understand and a user interface would help a lot.
• Caching does not behave nicely with some plug-ins.

To solve these issues, more meta-data is required about the plugins. The naive solution is to rewrite the plugin API entirely and make it good. After all, at this time, each plug-in is stored in a separate file containing two functions with a naming conventions. This is not a modern GoF-endorsed design!

Well, rewriting is a bad idea. Tikiwiki ships with around 75 plugins, plus a few in a separate folder to be enabled by sys-admins in controlled environments, plus an unknown amount in mods, and all those custom plugins written for specific applications we have no control on. If we only had to consider our own work, it would still take around 40 hours to convert them all, considering no clean-up is made and documentation is only entered as-is without improvements. Rewriting the API would be a 3-4 hour job at most. The conversion is uncertain the result would break upgrades on all customized installations.

I’d much rather compromise a little bit on beauty to save some pain. Adding an extra function to provide all the meta-data and making sure that the functions who use these act conditionally based on if the new way of doing things is available. I can keep working on improvements without having to convert everything at once. No functionality is broken.

Of course, the transition still has to be made, but at least it provides us with more time to do it. No one will scream that their favorite plugin is broken. I will be able to merge back in trunk much faster and get help from the rest of the community.

There is more than just code involved.

At this time, I think there is nothing more important in an Open Source project than to have face to face meetings. I spent the last week in Strasbourg for the TikiFest. I don’t think any of us really knew what was going to happen. I had this idea that it was a perfect moment to release the next major version of Tikiwiki. Some thought it was about security or other technical aspects. We were all wrong. During those days, we got to know each other a little more. For the last years, we have been discussing over IRC, exchanging an occasional email, but we really didn’t know each other. I think that this alone will change the project’s dynamics in the future.

Sitting together in the same room working on the project made it clear that we were all facing similar issues. We could discuss them and see how to improve the situation in the future. No mailing list or chat session could have done as much. We would get up early in the morning and not really see time fly by, alternating between discussions and code. Only our stomachs could bring us back to reality, and that was usually very late.

One of the most important changes that we made is change the release schedule, or create one for that matter. It has been over 3 years since the last major release. Trunk changed so much, we barely know what has been added. The changelog is terrific and certainly not useful to anyone due to the very long length. We decided to release twice a year. The primary reason why we could never really release is that new stuff kept being added in. The reason it came to be that way was that everyone needed that other feature to be added before the release. Otherwise, it would be years before it would be part of a stable release. Well, it has been years since we had a major stable release.

Getting more frequent releases was an easy decision. The discussion was in fact, very short. I think we had everything settled in the morning of the very first day. Later on, this allowed us to assign dates for some major changes, like fully dropping PHP 4 support. Timed-based releases have huge advantages when it comes to open source projects. It allows to observe how things are evolving, take decisions and still provide users with a decent amount of information. It’s not about saying that somewhere in the future we will drop PHP 4. We can provide a month and base it on the growth of PHP 5.

I really have the feeling I am now working on a new project, even if very little of the code changed during those last few days. Just like many other events where developers gather and discuss critical issues, I have some doubts about wether everything discussed will end up being done. However, I have a good feeling. The team is committed and deeply involved in the project. It’s a great thing to know we all share a common vision.

During the entire week, we didn’t have so many hot debates (except maybe one where I exaggeratedly bursted out). Some things are just plain good for a project, and when a project ran without any major guidance for years, some basic, typical, structures are welcome by everyone. One of the only issues we couldn’t quite agree on was what to accept in stable branches. Tikiwiki has a long history of being free for all, which was convenient for every major contributors. With the new model, stable branches are required. Not accepting new features is obvious. Some minor enhancements are acceptable, some not. UI enhancements are even harder to decide on. What is a bug anyway? Is it a major one? What is major? What to accept is an entire gray zone. It’s extremely subjective. I had to rollback commits. I hate doing that. Without people to discuss the issue in front of me, I don’t think I could have done it. In total, we reviewed around 8 commits made after RC1. It took us at least 2 hours to decide which one we couldn’t accept.

Rejecting someone’s contribution is hard. At least I could motivate myself by the fact that this was only the stable branch. They could still apply their patch on trunk. However, with faster release cycles, we will have to keep trunk as somewhat stable. We already have this idea of experimental branches (or feature branches) for new things to mature before being merged, but it really adds overhead. Subversion is quite hard to handle on the branching and merging issues. If you are familiar with version control, it really all makes sense, but it’s a barrier to entry. I started writing scripts to do the hard work, but they still require people using a shell to use them.

Establishing a software process to ensure quality without breaking the team dynamics is hard. Doing it on a community of volunteers is even harder. Some well known best practices just don’t apply. Unit testing? Forget it. Iterative development? We don’t even know what people will be working on or when they will have time to do it. With at least 6 years of development, people got used to their way and, let’s face it, it worked so far. Only part that didn’t work is that the software could never quite release. The decisions we took in Strasbourg were required to improve the releasability of Tikiwiki. I hope it will work out with the community.

For every other decision other than the release cycle, we basically had to take accountability for it. We built some sort of road map, but it can’t really apply to anyone else as we can’t assign tasks to anyone. People will keep volunteering on what they want and we have no control over that. This leads to a strange phenomenon in the roadmap. The roadmap is more about dropping things than about adding things. Tikiwiki is huge. It has features for everything. Letting people add whatever they want has always been very important. It lead to features no one could have imagined with incremental changes and clever combination of features. However, for all the great things that happened, some things were added and forgotten by their authors, remained obscur, unmaintained and unused. For all we know, it might not work at all as it rot and broke. These things are a burden for the project administrators, so they will gradually be removed. Ideally, things like removing unmaintained features and dropping support for older technologies will reduce the weight we have to carry and allow to build greater things.