After the couple hours sleep (’Is this good for productivity in any way?), we started the last bit of the API review. We reviewed the akonadi changes between the KDE SC 4.4 release and current trunk, renamed method names, watched for const’s and made sure we did not do any binary incompatible changes.

After lunch, we had a small talk about deprecating all old API KDE has which deal with kresources. As Akonadi is the successor of those and that is approaching rapidely, it is time for developers to port all the remaining kresources usages. With marking that API as deprecated, they get nice little warnings while compiling….

After that we talked a bit about the development of kdepim in the next months, where in SVN that will happening, and how that matches the KDE policy around the upcoming freezes. More news on that later. We first need to talk to some more people.

In the meanwhile I adapted the accountwizard to the changes i described yesterday. And with good progress. We are now beta testing it and fixing the remaining bugs. Should be ready for the KDE-PIM 4.5.0 release.

Steven hacked on unit tests and proxymodels (what else, whahaha). Tobias fixed most issues brought up by the API-review and some small fixes in KAdressBook (of course). Volker fixed a couple of KMail bugs, answered loads of questions and stated just now ‘Now that you asked me, i did not really do that much’.

Kevin Krammer worked on the conversion tool for transferring the old data from kmail to the new akonadi based akonadi. It has challenges like reading the old cache folder, which can contain a mix between mbox and maildir files and directories. He promised to blog about that later on.

Sérgio spended his time on KOrganiser, fixed a bunch of bugs and made the journal editor working again. Matthew fixed the default layout for KMail and a bunch of bugs, for example he fixed the hated bug about the ‘kontact special date summary plugin’ (if scrabble had spaces, this would have been a winner) hang now and then. Now ported to Akonadi properly.

Thomas deprecated the KResources API and bugxing and porting on KMail. Kevin Ottens worked the past days on the kimap implementation, writing unit tests, heck a whole framework. Users can activate a logfile to provide debug output about the imap process. That logfile can then be replayed with his framework, to exactly reproduce bugs.

You can see, that we are working in very many area’s at the same time. preparing our software for the next release. More tomorrow…

In Berlin there is the Akonadi meeting again. We do it twice a year and that seems to be a good interval. Just before each freeze (well, in this case, slightly after). Ah freeze, that reminds me of a) the temperature inside this cabin and b) the silence which arrived after the small argument… Back to topic. I’ve not hacked much the last couple of months, work work gets in the way sometimes. The spare time I had, I spent mostly on sysadmin tasks, of which Eike Hein is now doing most.

The Akonadi team has a few problems, or challenges if you like. Not really technical, but rather on the marketing side of stuff. I think the last couple of months have shown that we are not really communicating a lot with ‘the outside’, and in the days before this meeting, I’ve heard a lot of questions about the current state of KMail and other Akonadi apps. As with each meeting, I again hope to have time to report from the meeting and give you updates about the state of Akonadi within KDE.

I also will explain what I did manage to do last months and I hope to get started on a project related to it. More about that later.

One of the things I do want to talk about is Mailody. It is dead. With tears in my eyes I’ve deleted it a month ago. Why? Because I decided to do so. This was triggered by a flow of people wanting to use Mailody. Still puzzled? Yes? Ok. Let me explain. I created Mailody years ago, because I wanted to change the mail reading experience of my users. I never ever wanted to go after KMail users. They do the things they do their way and that’s fine. I wanted to create a mail client which had an alternative interface, alternative way of doing things, like saving the sent mail not in a separate folder, but in the same folder to preserve threading, having the ability to do a quick reply to a message, without launching a full blown composer, etc. (I can go on, but it’s pretty pointless for a dead app). I’ve accomplished some of my goals, but failed in some area’s, like seducing other developers to help me.

I’ve also failed to create a substantial userbase to keep it alive. I think I was the only user. I never made the switch to Mailody4, because it was never ready for production use. I feel the users approaching me to start using Mailody4 were doing it for the wrong reasons. Not because they liked what I created, no, they wanted it because KMail in svn trunk isn’t ‘ready’. Although everyone is Free to use their own motivation, I don’t think Mailody4 should be used for production. It will reflect badly on Mailody, Akonadi and me. Although I told people not to use it, they still insisted, ‘it is in keg, so must work somehow’. Right. The only option I had left was to kill it from svn, and so I did.

Is it really dead, dead? No. There still is a chance I will pick it up. I’ve still some unfinished business there. But it won’t be anytime before KMail trunk has been released and everyone is happy using that again. That means the whole stack should be ready for usage again (imap, mailtransport, identies, kmime, akonadi). Then I will make sure the glue between them (Mailody) is working fine again. But not now.

As said, I’ll be a frequent blogger the next days, hoping to bring you some news from what’s happening in Berlin at the Akonadi Meeting.

After that I worked to get the manual expunge code, which I explained on Friday. The DBus interfaces were added by Kevin and I adapted Mailody to use them. So that was one of the last goals I had for the meeting. In the meanwhile Thomas McGuire yelled that his rewrite of the pop3-resource was pretty much complete and passed his unit tests.

Day 3 was somehow a bit more relaxed than the other days. We briefly looked over the last bit of API we had to review. It was coming from the KMime library. KMime is a library that has been around for ages and was written with the intend to replace mimelib which is used by KMail. But that never happened.

KMime is the library that is responsible for creating the actual message after you press the send button in an email composer, but also deals with extracting the different parts from a message to display them, extracting the plain part, the HTML part, maybe some attachments, but also dealing with all kinds of encoding and decoding problems which are used to send the actual mail in.

When I started Mailody, I first tried to do that all on my own, but I soon realized that was not something I wanted to do, and Volker pointed me to the – at that moment unused – KMime library. After we removed the dust, we discovered some bad API and for KDE4, we removed most of the ugly API (like methods returning QList* (yes returning pointers to a list with pointers). Mailody then turned into a happy user with KMime. With the akonadification of KMail, it is needed that KMail also used KMime internally. Akonadi delivers the messages in that form to KMail, so it needs to. Andras Mantia has been actively working on this conversion for KMail. I’ve big respect for that work. I know it is a big task and probably not the most fun to do.

That also mean there is a new user of that stuff, and that means some API changes are wanted. We are still not completely happy with existing API, so that makes the extensions to that API somewhat less critical to get absolutely perfect. This is one of the candidates for a rewrite for KDE5. Still reading? Sorry to be a bit verbose on this stuff, just want to make sure you get a feeling why an akonadified KMail is not available yet. From one big monolithic application, it is being teared down to separate components, which will make the codebase a lot easier.

You have to have bad api’s to realize the value of good API’s.

After this mandatory part of the meeting, we quickly went over to the other mandatory part and did the Group Photo. I’ve not seen it yet, but I’m sure it will appear on the Dot or Planet soonish. There were a couple of discussions between various people. It’s fun to see that the base of Akonadi works nicely for a lot of people and we are slowly arriving at the phase were we can extend it further. Hot Topics in that area are:

• Search. You can basically split up two types of searches. The first one is in the messages within Akonadi. We will use Nepomuk for that. Especially now Virtuoso back-end for Nepomuk seems to be performing nicely, we can go ahead and implement them further. The ground work is already present; Akonadi supports creating virtual search folders, containing links to messages in other folders. Also each message passed into Akonadi is being indexed by Nepomuk. Just need to finish up that work as far as I know, like creating a user interface for creating such a search folder. The second type of searches are server side searches. Take IMAP. It allows you to search for messages directly on the server. We need to support that too.
• Filtering. Basically you have the same difference. There is a local filtering of messages, which is implemented by a summer of code student. It means that a special type of resource gets the messages before it gets added to the final destination folder. That special resource can filter it to another folder for example. This should all be finished, but it’s not completely ready yet. There is also filtering on the server side. For example IMAP filtering is (and should) be done on the server side, for example with Sieve. Volker and I briefly brainstormed about the possibility to create an application which can deal with that. Should not be too much work if we can base it on the work already done by the SOC student.

I briefly brought up Google Wave. I think Akonadi can easily provide a resource for it, although there wasn’t much interest from the participants of the meeting to work on it, so we probably will not work on it ourselves. Of course if the resource is created, you’d still need an application to edit the waves. So if anyone wants Google Wave integration into KDE, please use Akonadi as base. We can help you with the stuff, we just don’t have time to do it ourselves.

That pretty much concludes my coverage from the meeting. I know there will be a dot story outlining more stuff that has been done during the meeting, outside the little box I have lived in. It was fun to be there again.

After that we started with API-Review of the Mailtransport classes. Around 1920 everyone used to use their local sendmail application to deliver mail. That was ordered to deliver the mail as soon as the 14k4 modem has made a connection to the internet. A bit later people started to depend on the SMTP method to send mail. The application send the email to the provider which provides your internet connection and then it’s done. But now we are starting to support Microsoft Exchange Servers, we need a new way of sending mail. The feature Microsoft Exchange offers is that it also deals not only with incoming mail but also with outgoing mail. That means we needed to adapt our Mailtransport library to know which Akonadi Resources are capable of sending email.

When you compose a mail you have a selection bar, where you can indicate which provider/transport should be used for sending. That now has been extended to also show the resources that are capable of sending it. And that required a lot of changes. We combined it with nice features as sending messages after a specific date and time and moving messages after they have been send to a certain folder. To our surprise the whole morning was filled with reviewing the API and we have a pretty long list of stuff that has to change there.

After lunch we continued with the API review of IdentityManager. In this KDE cycle support has been added for images within signatures. Signatures in the context of mail messages which can hold company disclaimers for example. A much requested feature was that people (or company policy) needed to insert there company logo in the signature. A couple of KDE releases ago we added support for html based signatures and now we added the support for images. And those images need to be stored somewhere. But in a application independent way. The identities you setup are shared between KMail and Mailody and we need to prevent that breaks with the set image in the signature when you switch between those clients.

Continuing with yet more API review (really a weekend is to short for this). We continued with the API review of the widgets and classes provided by the new address book application. The classes are all in a library so other application can make use of them. One of the discussion points was about the dialog. When you for example click on a phone number and you have a voip phone, it can actually dial that phone number, the same goes for an email address. When you click it the default mail client is opened and the composer is launched. But if you show the dialog in an open composer, you want to add that address to the addressee list of that composer. Implementing that without making the API ugly needed some thinking.

In the evening Brad showed us the progress on Exchange support. He demonstrated Akonadi fetching mails, addresses and appointments from the Exchange server. Although it is in pre-alpha stage, it already shows that the resource is on the right track. From his side, there are pending requests for free-busy listing support and support for filtering. Both will be implemented sooner or later. His biggest problem, the mail sending part, is solved recently.

Hm, now the speaker is speaking about a deviation, who knows where I’ll end up now. Because people are talking loudly in this cabin, I can not decipher the German announcement yet.

I’m traveling to Berlin because the KDAB building in that city once again hosts an Akonadi Sprint. I wanted to organize it this time in my beautiful office, but considering that KDAB employes lots of people that work on Akonadi, it does not make much sense to let them all come over to the NL. That would be a waste of money.

I’ve set a few goals for this meeting. First I want to see if I can adapt the imap Akonadi resource (the thing that feeds Akonadi with data from imap servers) to properly deal with expunge. The IMAP protocol deletes messages in two stages. First the message gets a flag as being deleted and then you expunge the mailbox, which permanently removes the messages which are tagged for deletion.

This is one area KMail and Mailody differ. Mailody uses the above method to. When you delete a message, it is removed from the message list, but it is still there, you can undelete it when you hit the menu item to show the deleted messages. KMail does not work that way. I’m not sure what it does currently, but in the past it moved deleted message to the trash folder and immediately expunges the mailbox. We need to discuss how we can make the resource support both systems, without biting each other. Maybe I can convince Thomas to adapt to the Mailody method ;-).

Another goal is to see if I can get the IMAP resource to work with my mail server, somewhere there is a bug in the IMAP library or my mail server which prevents this. And that is not only blocking my implementation of the goal above, but it also prevents me from using Mailody4 full time. At the end of the meeting, I would like to be able to use Mailody4. I know I’ve set that as goal for previous meetings, but who knows, maybe it will work out this time.

There will also be some api reviews for the changes between 4.3 and 4.4. And there are a lot of changes, so that will take a lot of time I guess. Also, some of the API changes done to the identity manager (the thing that manages your identities in both KMail and Mailody) are not how I would have done them. I like them to be changes and I would like to work on them to correct those problems.

The speaker is still rambling about the deviation we are going to take. I don’t think that’s of any interest for me though.

Last goal is that I would like to know the current state of the porting of the applications to Akonadi and what the time line will be. That’s of course the stuff I will be blogging about this weekend. I know you are waiting for that info ;-)

Here you can see Mailody embedded in Kontact. You can see it fits in there, as it always have been there. On the top of Mailody you can put your favorite folders. That way you can collapse the regular folder sidebar and have a lot more room to see your mails and the message list.

In the past I had a copy of the kontact interface classes in Mailody, because they were living in kdepim. Last month the issue was raised again why it was located there. Because, when  living in kdepim it shielded from the outside world and application developers were discouraged to using it. The main reason from my point of view was that with the upcoming move to Akonadi we would like to keep the option open to adjust Kontact if needed. Moving it to kdepimlibs would mean staying binary compatible.

On the other hand we have seen little development to kontact the last few years. I argued that the reason there has been little development might be in the fact it is not really out there for people to use. The end result of the discussion is that we all agreed to moving it to kdepimlibs. The versioning of the plugins should prevent loading binary incompatible plugins.

That means from KDE 4.4 on people can embed their application to Kontact without nasty tricks. By making that possible I hope we can see a healthy amount of application which can be embedded. One point to note is that if your application is a uniqueapplication, you should use pimuniqueapplication so launching your application for a second time will result in kontact raising and showing your part within kontact.

With opening up Kontact for external application I also hope we see some more development for Kontact itself. I think the summary view is an excellent candidate to be plasmafied. Using Plasma and writing plasmoids for each application make sense. People can put up Lionmail applet on the summary view, a nice calendar widget, but also for example the current weather or a currency converter. Currently we have no takers for this feature, if you are interested, drop by the #kontact channel.

Yesterday I wrote that Akonadi is made up out of two parts. First is the database layer (15K loc) and second are the KDE libraries to communicate with that database / cache (35K loc). The database part differs between QMF and Akonadi. QMF has chosen to use SQLite and Akonadi has chosen MySQL.  Why we did not use SQLite is pointed out in our FAQ:

We tried. Really. It can’t handle the concurrent access very well, in the best case this means very slow operations, but we’ve also seen deadlocks and failing transactions. Once that’s fixed in sqlite, adjusting Akonadi to use it again instead of MySQL is no problem.

Currently there is some work done trying to let Akonadi work with SQLite. The database setup is a isolated layer within Akonadi. So as long as it understands SQL and it has a socket or port to connect to, it can be used in Akonadi. Some PostgreSQL patches has been committed too, these were for queries that had a little different syntax than MySQL. And that’s what is needed to get it to work. Some people trying it out, having knowledge about that particular database solution and being able to adapt Akonadi to it if there is a need to.

But when the Akonadi project started, we have had in mind that it should be able to be used as replacement for the file-based backend KMail uses at this moment. That means high speed and high volume data going in and out of the database on a constant bases.  This is very much another perspective than QMF.  You probably won’t open kde-core-devel archives on your telephone, while you will want to quickly see the new messages on that list. Tobias Koenig described the lists of requirements for each project:

Nokia requirements

• mobile phone centric pim handling (mail, sms, mms etc.)
• small memory footprint -> inprocess communication
• low bandwidth consumption
• low data volume handling (results from the previous points)

KDEPIM requirements

• API to access general PIM data (should be easy to extend)
• memory footprint can be larger
• inter process communication to be more robust
• handling of huge amount of data (as usual for desktop system)

As we have seen from KMail from the past, taking one solution to fit the other will result in nasty design (sorry KMail ;-)). Akonadi will work in the future on the somewhat fatter mobile devices and that’s probably where QMF and Akonadi will meet each other. I don’t see QMF used on desktops and I don’t see Akonadi used on thin mobile devices.

The second part of Akonadi are the awesome libraries which are created to access the database. For an application developer there is no need to learn sql and do tough queries, no the libraries deal with that. These contain of classes with a KJob API to manipulate data on a low level. Somewhat above that there are models , views and proxies to present the data to the user. They are in the library because we want a consistent look between all PIM applications.  The folder view of Mailody will look roughly the same as the one from KMail, at the minimum it will contain the same data. If you mark a message in Mailody as important, it will show up in KMail as important instantly and also in Lionmail (the plasma mail viewer written by Sebas) too.  On an even higher level it provides dialogs. So editing a contact in KAddressBook is exactly 100% the same as editing it in Mailody when you choose that option from the address book in the composer.

A library to this extend is not provided by QMF and there is probably no need for that. Last point I would like to make is that QMF is specialised in Messaging. Akonadi is type independent. It really does not matter if you want to store mails, contacts, appointments in there, you can even store bookmarks,  chat logs or kittens in it.  That last one has not been tried though.

Mailody is a mail client based on Akonadi. You can compare Mailody with KMail for example. KMail is a great mail application, but for my work flow it does not really fit. Some random differences between KMail and Mailody:

• Mail sent is by default stored in the folder which is selected. That means it will not break your thread, you always have the whole conversation together. This is optional in KMail.
• Mailody has a split delete and expunge setup for IMAP. That means that deleting is a two stage sequence, first you delete it, the message will disappear from the user interface. If you want to delete them permanently, you expunge it. This way there is no need for a trash. This is how the IMAP protocol is meant to be.
• Mailody supports server based flags. That means that if I share a folder with my co-worker, he can mark a message Red in his Thunderbird. Mailody will pick up that change and color the message Red in Mailody.
• Mailody has Nepomuk support, this means you can make comments to mails like ‘Need to talk to Xyz about this’.
• Thousands of smaller changes.

Why I not contributed to KMail is history, but in the meanwhile KMail has become a lot better. The code of Mailody is very clean and small and fun to work on. This is also because Mailody uses Akonadi. Akonadi can be divided into two sections.

First there is this database. This database is a cache. If you start Mailody you don’t want to retrieve 20.000 from the IMAP server every time you open kde-core-devel. This cache solves this problem. The 20.000 messages are cached there and when you open that folder, it retrieves it from that cache. Which is of course much faster. The database is fed by plugins (they are called resources). There is an IMAP plugin, a POP3 plugin, a VCard plugin etc. Each of these resources are responsible for feeding the cache. Each store it in their own part of the database. This means that not every application has to be able to have a library for talking IMAP. No, Akonadi knows how to do that.

The second part of Akonadi are the convenience classes for KDE. There is no need for each application to learn how to talk to the database. We made some classes for that. The Mailody view which displays all the messages in a folder is a model/view which is coming from Akonadi. You simply indicate which part of the database you want to view. Every item can be manipulated through KJobs for example.

Although the above is a very short introduction into Akonadi, I can give some powerful examples. Because Akonadi is central, it will share stuff. If Mailody changes an item, it will be instantly visible in KMail. Accounts are shared between Mailody and KMail. Mailody can easily show Newsgroups (folders are the newsgroups, the rest is the same). We can also let daemons run in the background. Take threading. KMail and Mailody thread the messages on display. This a very hard job to do right and causes delays in opening folders. We can let a daemon run in the background which already threads all messages. When Mailody opens a folder it will receive a threaded model/view with all the work done already.

I hope this will give you some insight about what Mailody and Akonadi are. Mailody is not ready for general usage though. Hopefully it will be ready in a few months for a beta. If you want more explanations about a certain subject, just drop a comment, I can blog a Part II if people want.

Screenie of the main Mailody Window:

Main Window Mailody

Some things might be visible for everyone. The tooltip has been added, so you can see the state of a certain folder, but more important, Mailody now has a Favorite Folders section. As Mailody prefers, it is a QDockWidget, so you can move it to the bottom, right or top, whatever you prefer. That means you can close your Folders list and only open it when needed, saving precious real estate.

I also made some changes to Composer. It now looks like this:

Composer Mailody

On the left of the composer there now is a list of all your addressbooks, in the past it only showed your standard addressbook. Whenever you hover for a while a popup appears with the contact details, that way you can also select a secondary e-mail address of the contact.

All credits go to the Akonadi people, I just glued it together. Mailody remains in alpha state, so don’t try it on anything else than some test folders.

The rest of this blog is about the presentations I saw, so if you don’t need those details, skip the rest ;-). The first presentation I saw was about the Spider engine for MySQL. Most of you know MyISAM and InnoDB. Spider is another one. This one makes it possible to store tables on different servers. The spider enabled server forwards the request to the remote server and passes it back to the requester. Not so useful you might think. Well, it also has the ability to forward sections of a table to different servers. Say you have 2000 records, you can store id 0-1000 on one server and 1000-2000 on another one. When a client requests id 1500, it will be passed on to the right server. As that server only holds 1000 records, the record is easily found and returned. For the client this whole process is transparent. You can imagine that searching for a keyword on two servers with half the records in parallel is much faster than searching it in one server which holds all records.

The second talk was about phpunit. This framework makes it possible to make unit tests for your php applications and libraries. He talked a lot about the reasons to do it and best practices. As I work on quite some php applications during day time, I think it makes sense to start using such a framework when possible. I think it also tells a bit about the matureness of an application when it contains unit tests. There are also some tools that can show which lines of your code are unittested. Something I did not know existed. I wonder if such a tool exists for C++ and if it does, what amount of lines are unittested within KDE libs.

After that I attended the keynote about Cloud Computing. Simon Wardley is a prefect keynote speaker. He brought statistical graphs, fighting cats and Tom Cruise together in something like 369 slides, without getting boring for a second. Although the topic was about Cloud Computing, it gave a good insight about the evolution of a technological invention. From the invention through a product until it reaches a service and about how that cycle has all the room for improvements, related inventions. I think I’m going to watch the video of the presentation to get a good understanding of everything he explained. Presentation of the day in my book.

After that the only possibility is disappointment of course. And it was. The presentation about a new mysqltuner was not so good. She arrived late, the beamer would not cooperate nicely, the laptop had issues with the mysql server and the presentation was largely about what the current mysqltuner script is doing wrong and at 5 minutes before the finish line she started to very quickly show the version she has written. I don’t consider the new version so much better, the mysqltuner script gives a nice summary of the state of your mysql server, after that you can start investigating what to do to make the thing faster. I don’t think any script can give a complete overview about how to tune the mysql-server best. Ah well, she has improved the output of some things, so it’s not that bad, I just expected more. Like output in html, a database which stores results over time, a deeper analysis of the most common queries, or whatever.

After that I returned to the main track where Alexandra Leisse had a talk about Community Building 101. It was something completely different and that was nice. I enjoyed the presentation very much, I think she used 5 slides for the whole presentation and they mostly contained pictures, hurray! She talked a lot about the aspects of maintaining a healthy community within a project. Some of that was really obvious, at least for me, but it’s nice to have it spelled out, some things are not so obvious and I learned some new tricks. Maybe I’ll now manage to create a community behind RSIBreak or Mailody ;-).

Finally I ended with a presentation about common mistakes in php unit testing. unfortunately it contained quite some overlap with the earlier presentation about phpunit, but that one too contained some things I did not know and things to remember when I ever start using testcases in PHP.

So tomorrow will be day 2. I’m looking forward to that. I made a list of things I want to attend, now the challenge is to not forget that list, like i did today.