KDE PIM Sprint Report

The KDE PIM Sprint is over (unfortunately…I could do this every day Smilie: :-)), so now it’s time for some recap of what has been done. I’ll try to cover the Akonadi side, and leave the rest up to others to cover their projects Smilie: ;-)

Hacking time! (Photo by Martin Klapetek)

Hacking time! (Photo by Martin Klapetek)

Akonadi Server optimizations

We finished and reviewed Volker’s old branch with a big optimization of the database schema. On my computer it reduces size of the file with the largest table by 30% and it speeds up all queries on that database, because the WHERE condition now has to perform only integer comparision, instead of string.

This however means, that we have to migrate user’s database on start. During the migration it is not  be possible to use any Akonadi-based applications.  We improved the code so that the migration takes about 10 minutes on my computer (used to take 20 and more). I personally think that it’s acceptable “downtime” for a one-time migration, so after I finish testing the migration code on other backends, I’ll merge the branch to master and we’ll ship it with KDE 4.13.

There's always time for a beer!

There’s always time for a beer!

Server-side Search

When using online IMAP, only headers are in Akonadi, the body is downloaded on-demand when the message is opened in KMail. This means that Nepomuk can’t index these emails and thus can’t include them in search results. To fix this case, we want to make use of IMAP’s SEARCH functionality. We simply ask Nepomuk to search it’s database of indexed emails, at the same time we send IMAP server the same search query and then we just merge results and show them to users. Most of the infrastructure in Akonadi Server has been in place for a long time now, so I’ll just undust it, adopt it to our current needs and we should be good to go Smilie: ;-)

Using Akonadi to store tags

Working on tags!I already mentioned this in a previous report: we want to cache tags in the Akonadi database and write them to storage backends if they support it (for instance as additional flags to emails on IMAP server, as CATEGORIES into events in iCal, etc.). Thanks to it it will be possible to share tags between multiple computers, yay! We just need to modify the Nepomuk libraries, so that when you ask Nepomuk for all data tagged with “Holiday”, Nepomuk can search it’s own database and also query Akonadi. Another benefit will be that filtering emails in KMail by tags will be much much faster, because the relation will be stored locally in Akonadi and we won’t have to talk to Nepomuk, which is very slow (mostly because of Virtuoso).

Storing conversations and threads in Akonadi

In his comment under my last blog, Till Adam said:

[...] KMail has the second best threading in the world, I think, second only to mutt because that is faster. [...]

Why can’t KMail just have the very best threading in the world? Because right now it has to fetch the entire folder from Akonadi in order to be able to perform Subject comparision when building threads. That’s both very slow and CPU-intensive operation. So we thought: let’s store information about relations between emails in Akonadi, and when KMail asks for content of a folder, we give back only first few conversations just to fill the screen, and then fetch remaining conversations on demand when user scrolls down. This should make opening even massive folders superfast and should save a lot of memory, too.

Akonadi and KDE Frameworks

Hopefully David's code is more stable than his towers

Hopefully David’s code is more stable than his towers Smilie: :)

The most-awaited discussion of the entire sprint was about KDE PIM and KDE Frameworks. When should we start? What has to be done? What can we use this opportunity for? From Akonadi point of view I want to do several things: remove deprecated API, change some API so that we use consistent naming and separate UI and non-UI stuff. Volker Krause suggested that we could move the client libraries into Akonadi repository with Akonadi server, so thatwe could share some of the code (protocol parsers for example), which I like, so we’ll go for that, too.

A bit unrelated, but still: the Akonadi server already compiles with Qt 5 for a while, so possibly during this development cycle we might switch to supporting only Qt 5 (and making use of all the C++11 awesomeness). There’s a little library that kdepimlibs link against, so we just build both Qt 4 and Qt 5 versions of it. Akonadi depends only on QtCore and QtDBus, so we only need distros to ship qt5-qtbase, which we believe most of them do by now.

Gmail resource

Vishesh and Àlex (Photo by Lukáš Tinkl)

Vishesh and Àlex (Photo by Lukáš Tinkl)

I’ve been promising this for ages, now I finally discussed this with others, got some input and can start hacking Smilie: :-) Let’s see if I can do something before Christmas Smilie: ;-) Gmail resource would store all your mails in one folder and would create virtual folders for each label and just link emails from the “All mails” folder into respective labels. This way the emails will share flags (read/unread), and you will even be able to manage the labels by linking or unlinking emails from label folders in KMail.

Here I’d like to thank everyone for coming to Brno – if was a lot of fun and great pleasure to see all of you again, and also thank Red Hat for letting us use the office. Looking forward to see you all again on FOSDEM, next sprint, Akademy or anywhere else Smilie: :-)

Category(s): KDE, Uncategorized
Tags: , , , , , ,

8 Responses to KDE PIM Sprint Report

  1. awesome, i’m looking forward for the 4.13 release!
    one thing I’ve noticed on my laptop is that emails with long text (mostly quotes from the previous 40-50 emails) it takes a bit to load, I hope you’ll be able to improve also the rendering speed of emails!

    thank you for your work!

  2. I’m curious about how would you guys keep akonadi and virtuoso in sync.. my experience that keep same data in two different database and keep them in sync really sucks and easy to have bug..

    • Nepomuk feeds new stuff into Akonadi (so Akonadi always knows about new stuff) and Akonadi emits notifications when something changes, so it’s only matter of Nepomuk listening to Akonadi. This is pretty much how KMail and other apps are kept “in sync” with what’s hapenning in Akonadi.

  3. Awesome, looking forward to see all the goodies you mentioned running.

  4. What I would like to see, is possibility to set KMail from start to store every email as own file to user wanted directory where directories are created like they are shown in the KMail.

    So every account would get a own directory, sub-directories being “sent”, “inbox” etc.
    And there every email file would be named by way as user has configured it, like
    “sender – subject.mbx”
    “date – sender – subject.mbx”
    “subject – date – sender.mbx”
    and so on.
    Files dates would be when they were received and so on saved on disk.

    And then get KMail to be used for viewing emails and even send emails from there launched from file manager.

    As it would be awesome to get easily access to all emails, backup them easily with Dolphin just by tarring directory and so on.

    (ps. I hope there is coming now the support for Ctrl+M for writing/reading email window and not just for the main window, as by default if user has hided menubar in main window, newly created windows for writing or reading should have hided menubar as well).

    • AFAIK, there is no email client which stores single emails in single files and for a reason: this would lead to a massive overhead and would be very, very … very slow. Databases are there for a reason…

      However, what I’m missing is indeed offline storage (but not in single files!) of IMAP mails. Because currently when you are offline (or have a buggy conection), there is no way to work with your emails. Or did this already changed? (because I’m using thunderbird of this.)

      Please take this into account, when redoing the search function.

      • In IMAP resource configuration, you can enable “Disconnected mode”, so it will fetch entire emails instead of just headers. This allows working with all your emails even when you are offline. Thanks to Akonadi you can work with them as if you were online, and when you really become online, Akonadi will replay all the changes you did to the server.

        Offline emails are also indexed by Nepomuk, so you would get search functionality even without server-side search.

        Per-file storage is not a problem: maildir works this way for instance, and Akonadi does that too for big files that don’t fit into the database. What is important is to have an index, that you can quickly search and get list of files that belong to each folders etc, as iterating over many files on a filesystem is very ineffective, as you already mentioned.

    • Sorry, I don’t see any use in providing access to emails via filesystem. That’s what email clients are for Smilie: ;-) Also going through 20k+ large folder in Dolphin just would not work Smilie: :-)