Improved Gmail integration in KDE PIM 4.14

Standard

Hi all,

I already teased publicly about the new Gmail resource on Google+ yesterday, now it’s time for some more explanations and…screenshots!

What is this about?

A native Gmail Resource for Akonadi that will bring much better integration of Gmail features into Kmail.

I’ve been talking about it and promising it for quite some time. But now thanks to some changes to the regular IMAP Resource that Christian Mollekopf has done recently, I could finally start working on it!

But…why?

Those who use Gmail know that Gmail does some things differently than most normal mail services. The biggest difference is that there are not really any folders with emails. Instead there’s one folder with all your emails and then there are labels, that you can assign to emails and then you can just filter your emails by the labels. And you can assign multiple labels to one email.

Yeah, but why bother? It already works quite well with the normal IMAP resource, right?

Yes, Gmail is able to hide this specialities from regular email clients so that they can still work with Gmail like with any other generic mail server, but at the cost of losing some features.

More and more often you can hear today that desktop email clients are dead and the future is in webmail (and cloud). And for many users who only have one email account this is true – why having KMail and KOrganizer etc. running, when they can have all this and more opened constantly in a single tab in their web browser? And the truth is, that Gmail is simply the largest mail provider in the world today. So if we want to persuade all these users to keep using KMail, we need to provide a user experience that is as close as possible to the native web interface. And for that we need a native Akonadi Resource ;-)

(Note: I’d like to avoid flamewars about “desktop clients are not dead vs. are dead” – I believe they are not dead for people who use more than one email account. They will cling to desktop clients until the dawn of the Gods, and even longer, but for normal users with just one mail account, it might be just matter of years to leave desktop clients. But who knows. impossible to see the future is).

Ok, so what’s the difference between Gmail and IMAP Resources?

The Gmail Resource supports some Gmail-specific IMAP extensions. In other words, it can speak and understands Gmail’s IMAP dialect. This means that the Gmail resource can handle the Gmail specifics better than the generic IMAP Resource:

1) Flattened folder hierarchy. This is best shown on screenshots: on the left, there’s a folder hierarchy as shown in KMail when using the current IMAP resource, on the right there’s the same account but synced via the new Gmail resource.

Folder hierarchy synced via the IMAP resource
Folder hierarchy synced via native Gmail Resource

2) One email to rule them all! As explained above, one email on Gmail can have multiple labels. But when you sync the mail via normal IMAP, you get a copy of that email in each folder. That means, that marking the mail as read will only mark as read that copy, but not the other copies in other folders, simply because KMail does not know they are effectively the same mail.

The Gmail resource is aware of this, an it syncs all your emails into one hidden folder and then just links them to the actual folders representing Gmail labels that you can see in KMail, so when you mark an email as read in any folder, it will mark it as read in all folders it’s linked into. Awesome, right?

3) OAuth authentication. The regular IMAP resource only supports the regular username-password authentication (and GSSAPI), which means that your password is stored in the computer somewhere, and if you use 2-step verification, you need to generate an app-specific password.

The Gmail resource has support for Gmail’s OAuth2, so you only enter your password once into Google’s web login, and the resource will then use a special tokens issued by Gmail with limited life-span to authenticate all your requests.

Gmail Resource authenticationGmail Resource: 2-step authentization support

The authentication is actually powered by LibKGAPI, a Qt/KDE library that implements various Google APIs, so it has the same look and uses the same code as the Akonadi Resources for Google Calendars and Google Contacts.

(Funny story and a question: I actually had to write a custom plugin for Cyrus-SASL to support XOAUTH2 mechanism, as upstream does not support it. Does anyone know whether there’s an existing implementation somewhere on the interwebs that I could use instead my crappy plugin?)

4) Simpler configuration. This is not really that big, bacause you don’t open the dialog very often, but I really like the configuration dialog a lot: simple and without complex options like encryption, mechanisms… This is simply because Gmail supports only a specific set of IMAP features, so I could just remove lot’s of stuff from the configuration dialog making it thinner and much easier to understand (IMO).

Gmail Resource configuration dialog

5) Push-notifications for all folders. The IMAP Resource can only monitor one folder for changes (using IMAP IDLE), because of certain technical restrictions. The folder usually is the Inbox. So if you have server-side filtering, you will never get push-notifications about new emails arriving to your other folders.

The limitation for watching only one folder applies to the Gmail Resource too. However since we understand Gmail, we can watch the “All Mail” folder, instead of the Inbox, so this way we get push notifications about emails from absolutely all folders (except for Trash and Junk folders, but who cares about these). Thinking about it, I could even remove the “Check Interval” option from configuration now.

Uhm ok, so what’s the state of the resource?

Currently the resource is still in a branch, waiting for some more features to be finished and for Christian to approve some of my changes to the IMAP resource (I’ll bribe him with some beer during Akademy, if necessary ;-)), and some changes must be done in KMail to properly support copying and moving of the linked emails, but other than that, it already works quite nicely :-)

That’s about it, see you next time :-)

28 thoughts on “Improved Gmail integration in KDE PIM 4.14

    • Not really :-) I’m still confused about web accounts vs online accounts (or whatever it is called), so I’m going to wait for Àlex to make a definitive stable release of this thing and then I’ll implement it :-)

  1. Lindsay Mathieson

    Looks good, glad it retains the folder hierarchy :) Worth it for the push notification alone. Will its share credentials with the Contacts & Calendar resources?

    There’s a bug in the IMAP resource where marking a search folder read that contains messages from multiple folders severely confuses the message unread count status. This will probably resolve that?

    Hows the integration with the existing IMAP resource? will you have to manually push fixes from it to you goggle resource?

    Is it ready for testing yet? happy to build from a branch

    Re Desktop Clients :) I don’t think it comes down to only having one email account or not – lots of people just don’t want to view email in the in browser, issues of integration with the desktop, reduced functionality etc.

    Lastly – the edit boxes in this form are impossible to see unless you actually click on them – even then very difficult to see the boarders. Not finding that very usable.

    Thanks for the ongoing work on the google resources! much appreciated.

  2. Alejandro Nova

    MARVELOUS. This is a game changer for me, since I use GMail tags heavily, but… will the KMail filter system support the GMail tagging system?

    • Each Gmail tag (called ‘label’ in Gmail terminology) is represented as a regular folder in kmail, similar to what you see in the web interface, if that’s what you meant.

  3. Almost every few months I try to use Kmail on my system but then eventually move back to Gmail web-client. The reason being that Gmail just provides so much better experience. Although it would be nice to use a desktop client so that emails are downloaded and viewed at my pleasure and I can also write my emails in a consistent environment.

    Here I are the things that I feel are missing in Kmail-Gmail.

    1. I have noticed that when you delete a message in Kmail it is moved to the local trash. Email isn’t moved to Gmail trash.

    2. Same is for drafts. I would like emails to be saved in Gmail drafts so that I can start writing in Kmail and finish email on my Android phone.

    3. If I get 3 security warnings from an email in short period of time Gmail clubs them together. No such things happen in Kmail.

    4. This is again related to threading emails. If a mailing list thread see 40 replies, they will all be shown as a tree taking space of 10 emails. Gmail clubs them together with showing number of unread emails. When you click this thread you will be shown emails from where you left reading. So only unread emails are shown.

    That looks like a comprehensive Gmail wish list. Thanks for your work.

    PS:- I hope this lands in Fedora 20 official repository soon.

    • Also I would like for Kmail to keep always a local copy so that everything can be searched locally and there is no delay in downloading email from server. That is the whole purpose of using a desktop client.

      When I add my Gmail to Kmail, my system temperature would stick around critical temperature of 96C, I hope this is because of Nepomuk and no longer valid with Baloo. I have 20,000 emails.

      I hope you plugin also works with Google Apps accounts.

      • Settings -> Configure KMail -> Acounts -> select your Gmail/IMAP account -> Modify -> Advanced -> Enable Disconnected Mode

        This way Kmail (Akonadi) will automatically download full messages from the server when they arrive and will have them available locally even when you are offline.

        With Baloo, you won’t notice anything is happening. Just your search will suddenly work :-)

        Since it’s using native Google authentication, it will work with any kind of account (that has Gmail available).

    • KMail already supports most of what you mentioned here, it just have to be configured.

      1) Settings -> Configure KMail -> Accounts -> select your Gmail/IMAP account -> Modify -> Advanced -> Trash Folder

      2) Settings -> Configure KMail -> Identities -> select your identity for Gmail -> Modify-> Advanced -> Drafts Folders (same for templates)

      3) Not sure what security warnings from emails mean? It it’s like notifications about new email, then KMail (since 4.12 I think) does that natively too

      4) Not sure we can do this in Kmail without substantial changes to the UI and the way it works. Maybe one day – I think it would work nicely for non-Gmail accounts too.

      It will go with KDE 4.14, but I don’t know whether we will ship 4.14 in F20. Definitely F21 :-)

      • Alejandro Nova

        Whenever I try to configure the trash folder as you said, I get the following when I restart KMail: “Fatal error: Could not create collection trash resourceId: 12″

      • 3. Duh! should have explained that properly. Let’s say some service sends you similar emails in a short period of time. In Gmail all those similar emails are clubbed together and shown as one email but in Kmail they are shown as individual emails.

        BTW can you create labels or folders in Kmail just locally that aren’t replicated back to Gmail. Gmail’s new Inbox is amazingly adept at sorting emails which made me remove all filtering rules and labels from Gmail and I would like to keep that for now.

        Thanks for providing this information. Configuring KDE can be quite daunting. I will add these info to the KDE wiki for future reference.

        • Thanks for explanation. I’m afraid this can not be done in KMail without rather large changes to the way it works. Gmail’s interface is more fluid and free-formed than traditional widget-based UI in KMail (or any other app), which makes it much easier to do things like what you described.

          For the local-only labels (or anything local-only for that matter): no. Everything you create in Kmail is automatically written back to the end storage (IMAP, Gmail, maildir, …). the only exception being the Search folders.

  4. Sinclair

    Looks really good, I have been hesitant to include my gmail account (that I use a lot) in KMail for a number of reasons, “half-baked” integration being one. Hope you make this part of 4.14 release!

    As for desktop clients: not ALL of us live where “constant online” is a reality. And to then call up old email, work on draft mail and so on for work (or personal) reasons demand offline storage such as desktop clients. For me a webmail interface is great when I am online. But not when I am offline…

  5. Have you considered making a “Google” resource that does mail, calendar and contacts?
    I.e. treats the Google account as a groupware server account?

      • Lindsay Mathieson

        2nd’ing that, would be really cool. Especially if you could unify the authentication.

        • AFAIK, LibKGAPI already does most of the work regarding contacts, calendar and even tasks. I use it everyday for most than half a year now and it works great for me. With LibKGAPI I’m able to create and sync everything but the mail from Google.

          Dan, might be a good idea to have a chat with yourself and Laurent Montel to make this Gmail integration an integral part of LibKGAPI so it becomes the unified Akonadi resource for all Google PIM services. I think that will be most welcome than ‘Google Blogger API v3′ commit from yesterday.

          One thing I would love to see would be the filters (message filters) working together. Unfortunately, this can’t be done with Google’s lack of attention to things like Sieve. Shame on you, Google!

          Besides everything, I would like to thank you very much for LibKGAPI and other code that you’re involved.

          • Hi,

            I think there is some confusion :-) LibKGAPI is a library that just implements Google APIs, nothing else – no real functionality, or Akonadi integration there. And I don’t want to add support for Gmail there, as Gmail is using IMAP, for which we already have KIMAP library (which just needs little tweaks here and there to support their features properly).

            What you probably meant are the Akonadi Resources for Google Services (Google Contacts and Google Calendars and Tasks Resources). I will indeed work on merging those together with the Gmail resource to have a “single resource to rule them all” :-) providing full-scale integration with Google PIM services.

            Blogger API support in LibKGAPI is to have better support for blogspot.com in Blogilo, which will be welcomed by many users who use Blogilo.

            Yeah, configuring Gmail filters is not possible right now (maybe the web API could be reverse-engineered, but that’s a lot of work and Google tends to change their internals from time to time, so our code would break) – you will still have to go to GMail web interface to configure them.

          • Dan, thank you very much for all those clarifications.

            You’re 100% right: I was talking about Akonadi Resources for Google Services, which is now merged to KDE PIM and in turn use LibKGAPI and KIMAP to provide userland functionality for Google Services.

            The reason I’m so focused on LibKGAPI is because I’m a Slackware user. Slackware don’t have a package for LibKGAPI yet and I have to keep it myself. Every time there is a new KDE release I have to recompile kdepim-runtime package so it could detect and be compiled against your LibKGAPI. This is what makes me good to go for Google Services within Slackware.

            About the Blogger API, don’t bother with my comment. I was just being annoying. :P I’m glad that you care about the people who want to blog something out there.

            Regarding Gmail filters, I totally agree with you. Reverse engineering their web API is a huge amount of work that could easily get broken. It doesn’t worth the effort. That’s why I blame the chocolate factory for that.

  6. BartOtten

    First of all: Thanks! Having a kinda stupid tree in Kmail for Gmail bothered me. So that’s on it’s own a +1 . I could say much more but I think SUDHIR KHANGER already said most or all of it.

    And the truth is, that Gmail is simply the largest mail provider in the world today. So if we want to persuade all these users to keep using KMail, we need to provide a user experience that is as close as possible to the native web interface.

    It’s the first time I read this argument in the KDE Sphere. And it sounds gooddddd! KDE folks tend to work at ‘nerd protocols’ rather than ‘popular protocols’ (Jabber instead of Facebook chat for example) but imho that keeps us on our own ehhh KDE planet ;-) Very glad to read how you think about it and your work!

    Respect (from someone that can’t write one line of code :P)

Comments are closed.