First technological preview of KDE Frameworks 5 has been released few weeks ago, a clear signal for us in Fedora KDE SIG to roll up our sleeves, heat up the builders and start packaging. For the last few days Martin Bříza, Jan Grulich, Lukáš Tinkl, Siddhart Sharma and I were working on bringing KDE Frameworks and Plasma 2 to Fedora. Now we are done and you can try out current technological preview of KF5 and Plasma 2 directly on your system without having to compile anything or messing up your default profile.
If you are a KDE developer, a Qt developer interested in using KF5 in your application or just a curious user, all you have to do is adding kde-frameworks repo to yum and install the packages! The repository is available in Copr, just download the repo file to /etc/yum.repos.d and you can start using Frameworks and Plasma 2. All Frameworks packages are prefixed with kf5- to prevent conflicts with regular packages and at this point they are installed into /opt/kf5 prefix instead of /usr
If you install kde5-workspace package, you can also have a sneak peek at what Plasma guys are doing for Plasma 2! Your display manager should have “KDE Plasma Workspace 2” entry in session types list, which will log you into Plasma 2 session. Just remember that Plasma 2 is still under heavy development and not yet ready for day to day use.
We will now start making the KDE Frameworks 5 packages co-installable with KDE 4 into regular prefix and hopefully we will ship KDE Frameworks 5 as a feature in Fedora 21 available to everyone.
Btw if you are coming to Brno for DevConf (February 7th – 9th) and want to know more about KDE Frameworks and Plasma 2, make sure to visit “KDE Frameworks 5” talk by me and Siddhart Sharma
The KDE PIM Sprint is over (unfortunately…I could do this every day ), so now it’s time for some recap of what has been done. I’ll try to cover the Akonadi side, and leave the rest up to others to cover their projects
Hacking time! (Photo by Martin Klapetek)
Akonadi Server optimizations
We finished and reviewed Volker’s old branch with a big optimization of the database schema. On my computer it reduces size of the file with the largest table by 30% and it speeds up all queries on that database, because the WHERE condition now has to perform only integer comparision, instead of string.
This however means, that we have to migrate user’s database on start. During the migration it is not be possible to use any Akonadi-based applications. We improved the code so that the migration takes about 10 minutes on my computer (used to take 20 and more). I personally think that it’s acceptable “downtime” for a one-time migration, so after I finish testing the migration code on other backends, I’ll merge the branch to master and we’ll ship it with KDE 4.13.
There’s always time for a beer!
When using online IMAP, only headers are in Akonadi, the body is downloaded on-demand when the message is opened in KMail. This means that Nepomuk can’t index these emails and thus can’t include them in search results. To fix this case, we want to make use of IMAP’s SEARCH functionality. We simply ask Nepomuk to search it’s database of indexed emails, at the same time we send IMAP server the same search query and then we just merge results and show them to users. Most of the infrastructure in Akonadi Server has been in place for a long time now, so I’ll just undust it, adopt it to our current needs and we should be good to go
Using Akonadi to store tags
I already mentioned this in a previous report: we want to cache tags in the Akonadi database and write them to storage backends if they support it (for instance as additional flags to emails on IMAP server, as CATEGORIES into events in iCal, etc.). Thanks to it it will be possible to share tags between multiple computers, yay! We just need to modify the Nepomuk libraries, so that when you ask Nepomuk for all data tagged with “Holiday”, Nepomuk can search it’s own database and also query Akonadi. Another benefit will be that filtering emails in KMail by tags will be much much faster, because the relation will be stored locally in Akonadi and we won’t have to talk to Nepomuk, which is very slow (mostly because of Virtuoso).
[...] KMail has the second best threading in the world, I think, second only to mutt because that is faster. [...]
Why can’t KMail just have the very best threading in the world? Because right now it has to fetch the entire folder from Akonadi in order to be able to perform Subject comparision when building threads. That’s both very slow and CPU-intensive operation. So we thought: let’s store information about relations between emails in Akonadi, and when KMail asks for content of a folder, we give back only first few conversations just to fill the screen, and then fetch remaining conversations on demand when user scrolls down. This should make opening even massive folders superfast and should save a lot of memory, too.
Akonadi and KDE Frameworks
Hopefully David’s code is more stable than his towers
The most-awaited discussion of the entire sprint was about KDE PIM and KDE Frameworks. When should we start? What has to be done? What can we use this opportunity for? From Akonadi point of view I want to do several things: remove deprecated API, change some API so that we use consistent naming and separate UI and non-UI stuff. Volker Krause suggested that we could move the client libraries into Akonadi repository with Akonadi server, so thatwe could share some of the code (protocol parsers for example), which I like, so we’ll go for that, too.
A bit unrelated, but still: the Akonadi server already compiles with Qt 5 for a while, so possibly during this development cycle we might switch to supporting only Qt 5 (and making use of all the C++11 awesomeness). There’s a little library that kdepimlibs link against, so we just build both Qt 4 and Qt 5 versions of it. Akonadi depends only on QtCore and QtDBus, so we only need distros to ship qt5-qtbase, which we believe most of them do by now.
Vishesh and Àlex (Photo by Lukáš Tinkl)
I’ve been promising this for ages, now I finally discussed this with others, got some input and can start hacking Let’s see if I can do something before Christmas Gmail resource would store all your mails in one folder and would create virtual folders for each label and just link emails from the “All mails” folder into respective labels. This way the emails will share flags (read/unread), and you will even be able to manage the labels by linking or unlinking emails from label folders in KMail.
Here I’d like to thank everyone for coming to Brno – if was a lot of fun and great pleasure to see all of you again, and also thank Red Hat for letting us use the office. Looking forward to see you all again on FOSDEM, next sprint, Akademy or anywhere else
we have a little favor to ask from you On the KDE PIM Sprint we discussed how to improve email threading in KMail by using Akonadi to store the information, so that KMail does not have to compute it every time. This would make opening a folder almost instant, all threads would be reconstructed immediately and it would massively improve CPU and memory consumption (so it’s totally something worth helping us with ) More details on what else we discussed and implemented will follow in another blog post tomorrow.
To implement the threading caching, we need to know, whether in these days it still makes sense to support threading by Subject. It’s used as a fallback when grouping by standardized email headers (In-Reply-To, References) are missing, which used to be a case with buggy email clients years ago, but hopefully is better now, so we could drop it, which would massively simplify the algorithms.
So we would like you to disable threading by Subject, observe how much it breaks your threading (and potentially your workflow) and report back to us. To disable it, go to View ->Message List ->Aggregation -> Configure. There go to Groups & Threading tab and in Threading combobox select Perfect and by Reference. If the combo boxes are disabled, you have to click Clone Aggregation to clone the default settings, and use the clone.
If removing threading by subject would break threading and workflow for too many users, we will keep the settings and we will try to figure out another way to do it.
So please configure your KMails, and let us know in comments below this post, on IRC, kde-pim mailing list or through any other communication means (just please try to avoid using smoke signals and pigeons )
The KDE PIM sprint in Brno, Czech republic starts this Friday, but some KDE developers just could not wait and decided to come to Brno already on Monday to work with the Red Hat KDE Team. Some of the stuff we are hacking right now is PIM related, but we also want to use this few days to work on other projects that we are involved in, but that are not strictly related to KDE PIM.
So I’m just sitting right now in the office with Àlex Fiestas, David Edmundson, Vishesh Handa, Martin Klapetek and my colleagues Jan Grulich and Lukáš Tinkl. I’m waiting for Àlex to finish polishing his port of BlueDevil to BlueZ5, so that we can start hacking on KScreen – there are far too many bugs that need our attention and we’ve been neglecting KScreen quite a lot in the past few months. We want to fix the annoying crash in our KDED module, solve a regression that my bold attempt to fix an another crash in KDED caused and discuss the future direction of KScreen – me and Àlex have different opinions on where we should go next with KScreen so this is a great opportunity to find a common path
Vishesh has been relentlessly working on improving the semantic technologies in KDE and from what I’ve seen, it’s something to really look forward to
Yesterday, me and Vishesh discussed the possibility of using Akonadi for handling tags of PIM data (emails, contacts, events, … and I implemented the feature into Akonadi and the Akonadi client libraries – only as a proof of concept though, I have no intention of shipping it at this point – much more work and discussion is needed about this. I also made further progress with implementing the IDLE extension to the Akonadi protocol. It allows the Akonadi server to send notifications about changes to clients using the Akonadi protocol, instead of D-Bus (performance++)
KDE hackers fighting bugs and bringing the awesome to our users
David Edmundson and Martin Klapetek have been working on creating Plasma theme for SDDM (a new display manager that for example Fedora intends to ship instead of KDM), and today they’ve been improving KPeople, the meta-contact library used by KDE Telepathy and that they will also integrate with KDE PIM.
My colleagues Lukáš Tinkl and Jan Grulich are working on plasma-nm, the new Plasma applet for network management in KDE.
More people will arrive to Brno tomorrow and the rest of KDE PIM sprint attendees will arrive during Friday, when the real sprint begins. Stay tuned for more news (not just) from the PIM world
Akonadi 1.10.3 has been released, fixing (among other things) support for PostgreSQL 9.2 with Qt 4.8.5.
I write a blog specially about this update, because the fix requires updating the Akonadi database schema. The update will be performed on next Akonadi start and it can take some time, especially on larger databases, so we kindly ask users for patience.
Update: if your distribution ships Qt 4.8.5 with the PSQL driver patch reverted, please make sure to remove the revert before pushing Akonadi 1.10.3 to repositories.
Users of other backends (MySQL or SQLite) will not be affected by this update.
Big thank you for investigating and fixing this problem to Cédric Villemain.
Fix crash when there are no flags to update during flags change
Fix crash on Akonadi shutdown when using PostgreSQL
Fix notification to clients about database upgrade
Send dummy requests to MySQL from time to time to keep the connection alive
I arrived back home from Akademy just a day ago and I already miss it. I enjoyed every single moment of it and had lots and lots of fun. Thanks everyone for making this such an awesome event, and especially to the local team. They did an incredible job!
This blog however will not be about Akademy (I will write one maybe later), but about Akonadi, core component of our PIM suite. As you probably already read or heard, Volker Krause has handed over to me maintainership of Akonadi. It means really lot to me and I’ll do my best to be at least as good maintainer as he was (if that’s even possible) and I would like to thank him for his outstanding job he did writing and maintaining Akonadi.
I really believe in Akonadi, I like design of the framework and admire all the work guys have done on it since the beginning, long before I even dreamed about becoming a KDE contributor. I also believe that having a powerful and well-working PIM suite is the key for success of KDE (not just) in the enterprise world. Akonadi is more powerful than most of the competition out there, we just now need to focus bit more on stability and performance. I think we are doing pretty fine with stability, so I want to focus mainly on the performance side of Akonadi. In this bit more technical blog post I want to write about what I did recently, what I’m doing know and what are (some of) my future plans with Akonadi. As usually, huge thank you to Volker for his ideas, suggestions and comments about my ideas. We had a great discussion during Akademy and we (theoretically) solved many problems and bottlenecks that were bothering Akonadi for a long time, but nobody had time to look into it.
I started working on batch notifications after the KDE PIM sprint in March, merged it in May (I think) and hopefully have resolved all regressions by now. Akonadi server uses notifications to inform all clients about changes, like new items, changed items, removed items, etc. Notification is a simple data structure that is transferred via D-Bus to all clients. Before this patch there was one notification per each item which means that marking 500 emails as read had generated 500 notifications that had to be transferred via D-Bus. With batch notifications the server can create a single notification that references all 500 items. This saves a lot of D-Bus traffic and allows faster processing on the client side. This feature will be available in KDE 4.11.
MTime-based item retrieval
This was written during Akademy and will allow Vishesh to improve start-up time of the Akonadi Nepomuk Feeder. Until now the feeder had to fetch all items from Akonadi and all items from Nepomuk on start just to compare whether everything is up-to-date. With mtime-based item retrieval the feeder can ask Akonadi to hand over only items that have been changed since some given timestamp. In most cases that means 0 items. You have to agree that fetching no or just a few items instead of all of them will give us a notable performance boost during start. Albert allowed us to backport this to 4.11, so you will get this improvement in 4.11 as well.
External payload files without path
Another thing that has been implemented during Akademy is aimed to save some of your precious disk space. Because Akonadi is a cache for your data, it means that it has to store all your emails, contacts, events etc. somewhere. Smaller records are stored directly in the database. Larger items are stored in external files on your hard drive and there is only file path stored in the database. However the path is always the same and is usually around 50 characters long, while the file name is only around 10 characters long. This patch makes sure that only the file name without path is stored in the database, saving some disk space. Clients now also have the capability to understand file names without path in server replies, so we can even reduce size of traffic between server and clients a bit. I know that 50 bytes is not much, but multiply it by tens or hundreds of thousands of items, and it’s already worth it. As a side effect, all newly created databases will be relocatable, because all file paths are relative, not absolute. There is no plan to make an automated migration to strip the path from existing records, but I might one day implement this in the Janitor, so that users can migrate their database manually if they want to. But it’s not a priority now.
Previously I explained that Akonadi sends notifications via D-Bus to all clients to inform them about changes. The problem here is that not all clients are usually interested in all changes. For instance KOrganizer does not care about new emails and KMail does not care about modified appointments (there are exceptions like the Nepomuk Feeder, which listens to everything for obvious reason). Yet each notification is “broadcasted” to all Akonadi clients. Each client then decides whether it’s are interested in the notification and wants to process it further or just drops it. The average number of clients is around 16, but in most cases only 3 or 4 clients are actually interested in each notification. That means that other 12 or 13 clients just drop the notification. What I’m working on right now is to move this filtering code to the Akonadi server, so that the clients can tell the server what kinds and types of notifications they want and the server will only send notifications to those clients that are really interested in it. This should save us a lot of D-Bus traffic and will fix the awkward situation when all clients are consuming CPU, even when you are just syncing one of your IMAP accounts.
Server-side change recording (IDLE)
The biggest task ahead of me. Some Akonadi clients are using feature called change recording. That means that every notification that is not dropped by the client is stored into a binary file (every client has one such file) and is removed again when the client confirms that the notification has been processed. This is used for example by the IMAP resource. When you are offline the resource is recording all notifications (about items being deleted, moved between folders, marked as read, etc) into the file and when you connect to the internet and the resource is switched to online all notifications are replayed from the file. My plan is to implement something similar to IMAP’s IDLE. Changes will be recorded on the server instead of the clients and all clients will be able to connect to the server and request all pending notifications. After that they send “IDLE” command + explanation of what kind of notifications they are interested in and the server will automatically feed them with new notifications. This is essentially continuation of the “Server-side filtering” feature, but it takes it to a completely new level. With this feature the Akonadi framework will generated almost no D-Bus traffic at all and the whole thing will be much much faster. I’m really looking forward to work on this because it’s a very challenging task and the result will definitely be worth the effort.
Volker has also submitted a few patches to reduce size of the messages sent between clients and server even more and started working on optimizing some SQL queries so that we don’t query the database for data we don’t actually use anywhere.
Of course there are more smaller ideas and improvements in the queue, but I need to keep something for next blog posts so stay tuned – there’s more coming soon!
I haven’t blogged about my involvement in KDE PIM in a while, so let’s see what’s new there, especially in the Google integration part….
Reborn Google Resources
Just before the KDE PIM sprint in Berlin this month, I’ve sat down and written completely new API for LibKGAPI (the library that implements Google API and is used by the Akonadi resources for Google services). The new API is job-based, and therefore much more awesome than the old one (which is known to suck). Anyway – what does this mean? It means that the new resources are awesome as well!
Google Contacts resource now has a full support for contacts groups. All contacts are stored in the top-level collection and are linked to the respective groups, so it does not matter where you edit the contact, you are still modifying the same instance. Like in the web interface.
Google Calendar now supports limited sync, so you can choose to only sync events from last year, or last two years (the default is last 3 years) instead of the full history.
Both resources have improved status reporting, error handling, are more stable (no more mystery crashes due to unhandled exceptions thrown from LibKGAPI) and subjectively synchronization is faster too.
Murdered Google Resources
As most of you probably noticed by now, Google is planning to shut down Google Reader by July 1. It’s pitty, because I already had a fully working Akonadi resource for Google Reader ready in the akregator_port branch. Cost me lot of time and nerves. Well, the resource is not there anymore and the only memory of it is greader branch with API implementation in LibKGAPI (which will die as well sooner or later). The good news however is that I can now help Alessandro and Frank with ownCloud News and the ownCloud Akonadi resource, so that we rock when Akregator2 is out I can’t wait to see what has changed in ownCloud since I installed 3.0.0 some time ago…
Upcoming Google Resources
I have two feature requests in bugzilla: one is to support Google Bookmarks, which is fairly complicated because of missing official API and absolutely no write API. So this is not going to happen soon. The second feature request is for Google Drive KIO slave. This is much more interesting task. I already tried writing Google Docs KIO slave about three years ago and I failed epically. Retribution! There’s almost complete API implementation by Andrius in LibKGAPI git, so I plan to port it to LibkGAPI2 and see whether we can together fight the Dark side and create a nice and shiny KIO slave.
Finally, deep in the dark corners of my mind, my so far the most evil plan is slowly shaping. The plan includes modifying the current IMAP resource, reusing most of it’s code and subclassing some specific parts to build a native GMail Akonadi resource that would support some GMail-specific IMAP extensions. The main idea is to support one-mail-in-multiple-folders-at-once case. Right now the IMAP resource handles that by creating a new instance of the same email in multiple folders. My bold plan is to store all emails in Inbox and link them to respective folders. This means that marking an email as read in one folder, will automatically mark it as read in all other folders (because it’s a single instance). The IMAP resource looks scary though, so I don’t know yet when I’ll get the courage (and time) to sit down and actually start coding…I guess probably after Akademy, after I talk to some people.
Batch Operations in Akonadi
I have talked to Volker Krause during the KDE PIM sprint about how to effectively handle “Mark feed as read” in the Google Reader resource. Currently, Akonadi creates a new notification for every change, therefore marking 300 items as read generates 300 notifications, which are delivered to the Akonadi resource, which should then create 300 HTTP request to store 300 changes. You probably agree that this slightly suboptimal. (I temporarily solved the problem by caching the notifications in the resource itself and then sending a big request to Google Reader at once). The solution that Volker suggested sounds fairly simple (it’s not) – batch notifications – i.e. a single notification about single change involving multiple items. The supported changes can be flags change, deleting or linking of items. By being able to deliver single notification about mass-change to Akonadi clients and to Akonadi resources represents new possibilities for optimizations. For instance the IMAP resource could simply send a single command to add a flag to multiple emails at once, instead of doing it one by one. The same goes for other operations and other resources that are dealing regularly with operations on larger sets of items. The obvious result: performance boost! After two weeks the work is in semi-working state – it works, but it goes nuts if more than 5 items are involved. The cause is known, but solution not (but I’ll get there eventually )
I’m occasionally helping with Akregator2 (Akonadi port of Akregator). Recently (ok, it was two months ago… ) I’ve written Akonadi Nepomuk Feeder plugin that is feeding RSS Articles into Nepomuk and a Search window (slightly inspired by the one in KMail) in Akregator2 where you can do full-text search (+ search via other criteria, including author’s name and date of publishing) based on data indexed in Nepomuk. Obviously, when I wanted to demo that on the KDE PIM sprint I found out that it’s not working as good as I thought, so there’s still some work to be done. But in general I’m happy to say, that from time to time it finds something .
Ok, so that’s about what I was, am and will be working on in KDE PIM. Here I’d like to say big thank you to all KDE PIM devs, because they are doing an incredible job. Thank you!
We have done some changes to our KDE Telepathy repositories recently, that might be interesting for packagers.
We have consolidated all Plasma applets into a single repository called ktp-desktop-applets (originally ktp-contact-applet). The ktp-presence-applet repository is now empty, the master branch contains just a README, the stable branch kde-telepathy-0.5 is preserved.
We have also normalized names of the applets, so they are now called org.kde.ktp-presence, org.kde.ktp-chat, org.kde.ktp-contactlist and org.kde.ktp-contact. The ktp-desktop-applets installs an update script for Plasma Desktop that will automatically update user configuration on next login.
All declarative plugins are now located in ktp-common-internals, so that all our QML models and utils are available for developers.
These changes will be first available in 0.6 release.
Every year Red Hat Czech offers a list of topics for bachelor and diploma theses to students from local universities. The topics are usually suggested and posted by Red Hat employees who then work with the students as technical consultants and mentors.
We see this as a good opportunity for students to learn about open source software, communities and collaboration and for us to find potential future employees (so there’s a great motivation too). We want students not to see their diploma or bachelor thesis as some boring obligation, but rather a “fun” way to learn new things, meet people and gain experience. And last but not least they are potential contributors to the projects they are involved in.
This year I managed to get several students I’m mentoring to pick a topic related to KDE. Although all projects are somehow related to KDE Telepathy, it’s a good start
Today I want to introduce work of first two students, Stanislav Láznička and Artur Dębski.
Stanislav Láznička is doing his bachelor thesis on Brno University of Technology. His topic is “Shared Board – Makneto“, more specifically, his task is to port Makneto to Telepathy. Makneto is a whiteboard-sharing application for KDE, similar to KWhiteboard. It has been written some time ago as a diploma thesis. Since then several students have based their theses on this project (not all of them successfully though). Stanislav is now researching how to use the Telepathy framework and preparing to start the actual port soon. If you are interested in further progress, follow his blog and his git repo on bitbucket.
Artur Dębski (IRC: mentero), a student from Silesian University of Technology in Poland. His topic is “Touch interface for KDE Telepathy“. Yes, hang on to your hat, we are going to have a totally awesome Plasma Active application for Instant Messaging. Artur has written an extensive blogpost about his work together with some mockups. You can checkout the existing code from here, there’s already a demo app to play with!
What is your excuse for still using Kopete instead of KDE Telepathy? Oh, you can’t live without your old conversation logs? Not a problem anymore!
KDE Telepathy is now able to import your AIM, MSN, ICQ, Yahoo, Jabber or GaduGadu logs from Kopete into Telepathy logger.
When you add a new account in the Accounts KCM, we will convert the new account ID into Kopete account ID and check, whether there are any logs in Kopete folder for this account. And if there are, we ask you whether you want to import them or not.
That’s nice, right? But what about our current users, who just silently weep, longing for their old Kopete logs? Well, we thought about them, too! After starting the KDE IM Log Viewer, you will be prompted with initial logs import dialog. The dialog will appear only once. Whether you click “Import Logs” or “Cancel”, we won’t bother you with this never again. Currently I’m considering adding a main menu to the log viewer, which would contain an action to invoke this dialog again, in case you accidentally click “Cancel” for instance.
When you are removing an existing account from KDE Telepathy, you are prompted the “Are you sure you want to remove the account?” dialog. As a bonus, I have enriched it by a “Remove conversations logs” checkbox. Guess what will happen if you check it
All these features will be available in KDE Telepathy 0.6.
And as usually, we are still looking for new contributors! There are some junior jobs in the bugzilla, which are a perfect start for newcomers. If you would need any help with fixing these bugs or wishes, just come talk to us on #kde-telepathy IRC channel or mailing lists.