Demystifying Akonadi.

The exotic-sounding ‘Akonadi’ refers to both a mythological figure and the KDE platform’s central information framework. This article will dispel some of the mystery about how Akonadi will improve performance and integration, and how it is being rolled out into KDE applications. I’ll also provide some insight how the technology works, and what will become possible with this new PIM framework.

Many people have been asking what the status of the new, Akonadi-based Kontact Groupware suite is. As I’ve been working closely with the PIM hackers, I thought I’d give my readers a heads-up on what’s going on and what to expect. In this article, I will often take KMail as an example for the port, but similar things apply to the other PIM applications that form the Kontact suite as well.

The What & How?

I’m sure many of you haven’t heard the name Akonadi yet, so let me quickly explain what it is. Let’s get technical.

Akonadi is a groupware cache that runs on the local machine, a shared data store for all your personal informatio. Akonadi offers a unified API to receive and synchronise data with groupware, email servers or other online services. Agents called “resources” are responsible for communicating with the remote service, say your email server. These resources run out-of-process and communicate via separate control and data channels with the mothership (your local Akonadi). Resources are easy to implement and can interface any data source with Akonadi, be it your local calendar file, your companies groupware, email servers or address directories, or anything else you can come up with. More on that specifically later.

The Akonadi groupware cacheA common misunderstanding is that Akonadi is some sort of groupware server. In fact, Akonadi does not store any data itself, but just provides a common means to access data to your local applications.

So Akonadi does not store user data, it caches it. The user data is still stored in the traditional formats, be it on an online server (for example IMAP) or local files (ICAL calendar files). Locally, Akonadi provides a cache to speed up access and to make collections (email folders, for example) and their items available offline. To allow Akonadi to work on both powerful desktops and lean mobile devices, Akonadi can use different databases for its cache. Currently, the most complete backing store for Akonadi is MySQL, but PostGreSQL and sqlite backends are also available. In the case of MySQL, the database is started and handled by Akonadi itself, using a local socket, and no network access. This is intentional, for speed and security, since Akonadi’s database is really only a detail of the implementation.

The storage concept of Akonadi is straightforward. The team looked at many types of PIM data and found that items stored in folders are common to all of them. In Akonadi, Items represent mails, contacts or other individual pieces of data Folders are generally referred to as Collections, which can contain other Collections. Items themselves carry a type (using the freedesktop.org standard mimetype definitions), metadata and the actual data payload. Items can be identified by URLs. This URL is of course only valid locally, but it allows passing references to Akonadi items and collections around without copying the actual data. This makes Drag and Drop across applications (or in my favourite case, from the email notifier in Plasma into KMail) very easy. The receiving application can use any Akonadi client library to take the Akonadi URL and fetch its headers, or data. Akonadi Items may be retrieved partially, so if an app wants tod display a list of emails, it doesn’t have to copy around the whole inbox, attachments and all, but can just ask for a list of headers of those emails.

In order to access the Akonadi cache, and more importantly the underlying data, you can use one of the Akonadi access libraries. To my knowledge, there are Akonadi bindings for GTK+, Python and the Qt-style Akonadi classes already available. As you can see in the diagram, the design allows for different ways of accessing the Akonadi data, in the diagram the examples are called the “GNOME API” and “KDE API”.

As you’d expect, I’ve mostly worked with the KDE API, which you can find in kdepimlibs. This Qt-style library has been available for a couple of KDE Platform releases already, and is being further enhanced for more coding convenience, stability and performance all the time. There is a bunch of job classes, that allow for async access to Akonadi items. Relatively new are the MVC classes, notably EntityTreeModel and friends. The ETM and its friends and API sugar around it also provide async access to Akonadi data as well, and also allow for easier sorting, querying and filtering of all the data and metadata. Metadata handling is another very interesting aspect of Akonadi in itself, more on that later, as well.

Current Status

Plasma's calendar displaying calendaring info from AkonadiMany people are interested in the current status of Kontact’s Akonadi port. Initially, KDE had planned to release the new Kontact along with the rest of the KDE Applications 4.5. This did not quite work out, so we pushed the release back a bit, and are planning to release it along with one of the 4.5.x updates. The current plan is to release the Akonadi port of Kontact still this year. In contrast to our usual releases, this step is a bit different. Since PIM data is critically important, we are extending the beta phase until the Akonadi port of Kontact passes a much wider range of QA tests. When we are able to release depends a lot on the feedback we get from users. We are therefore making available monthly beta releases of the new Kontact suite. Data loss in this late phase of the port is extremely unlikely, and we made sure that trying the new Kontact doesn’t mean you must now also do the switch. You can in fact just reinstall the old one and use that again, since separate configuration files are used.

KAddressBook is already available in its Akonadi incarnationThe traditional Kontact, is of course still fully usable and we currently recommend this to end users. Kontact 4.4 is still actively maintained and supported, and is shipped by distributions along with KDE 4.5.0, so the current stable Kontact is 4.4.5. We did ship a new release of kdepimlibs, which are tested with Kontact 4.4.5 and are the basis for Kontact2 as well.

For normal workloads, KMail2 which is the heart of Kontact’s Mail component is already pretty usable. The focus of the stabilization and improvement efforts currently lies in the complex use cases common to hackers and email power users, such a different, high-volume email accounts, many large folders and a paranoid bunch of identities. Another area of focus is the migration of data, including the possibility to rollback to your “traditional” Kontact if you are not satisfied with the quality yet (please don’t forget to file bugs, so we can take proper care of those nasty insects).

Migration

Kontact2 only reads Kontact1′s configuration, but doesn’t change the original copy. Instead, a new configuration file derived from your old one will be used. So when first starting Kontact2, your “old” configuration, account setup, identities and filtering rules will be imported. KMail2 will also import locally cached emails, so you don’t have to download them all again. In the current state, user feedback from migration and usage is extremely valuable to the developers, so please give the next beta a whirl and report back to us, so we can improve on your experience. Of course, there are tools for importing and exporting data. During the migration, Kontact2 uses Kontact1′s existing downloaded email, so a lengthy re-download for offline reading is unnecessary.

If you’re not yet happy with the new Kontact, you can switch back to the old one, by re-installing the 4.4 Kontact.

What will KMail2 look like?

This might come as a little surprise to some, but in the initial version of KMail2, you won’t notice many differences to the traditional KMail1. This has a number of reasons: First of all, KMail’s UI is the result of years of polishing by the developers and a lot of feedback by the community. This won’t be thrown away for something that’s novel and cool, but might not satisfy most users. So KMail 2 will be a very straightforward port of KMail1, the UI will be mostly the same, while the underlying technology has changed completely. In the porting process to Akonadi, most of KMail1′s familiar UI has been kept.

The Akonadi-based Kontact betaYou might have noticed the first parts of KMail1 being converted. That is to say that the Kontact developers have worked towards the Akonadi port of KMail. The first, and actually one of the most central parts of KMail has already been introduced a while ago: the new listview. This listview is a rework of KMail’s list of messages to use the underlying Model-View-Controller design patterns that match Akonadi well. In Kontact 4.4, Kontact’s address book has been switched to use Akonadi. This first step in the migration was a bit painful, since it involved introducing a new infrastructure below applications that you use daily, and which you rely upon. The new possibilities are already making their way to the end users’ systems, for example in the calendar integration with the Plasma Desktop, which you can see in the screenshot. By clicking on the clock, you get the Plasma calendar which shows you your daily events. In the sceenshot, you can see the new version of KMail. As I’m using a full-HD display, I’ve enabled the widescreen layout of KMail. This makes it possible to see the whole email and a long enough list of others at once. A nice touch, which has been available for ages in KMail — just in case you wondered.

… but why the port then?

Simply put, traditional email clients don’t satisfy today’s expectations and work flows around personal data. Well, this of course needs some explanation: Already in times when KDE 3 was state-of-the-art, we noticed that more and more applications became interested in PIM data. Popular examples are Kopete, the instant messenger which held its own list of contacts, data which is mostly duplicated, including the inefficiency and maintenance nightmare you’re facing when you duplicate frequently changing information. So you want some kind of interface for contacts, and it should be something service-like, after all, you don’t want to run a seemingly unrelated application(your address book), just to get some more rich information about your chat contacts. Then, there’s of course my favourite example: Email monitoring. In essence, a full-fledged email client is a bit of an overkill, if you just want to know if there’s new email in your inbox. On of those overkill aspects is performance, or rather resource consumption. The solution to this is of course to share all this data. By using one central storage, and an easy to use access layer we can share the data across applications, and enable applications to make use of already available personal data. Enter Akonadi.

Akonadi has been built with performance and memory consumption in mind. Will Stephenson has put this very nicely: “In the 2.0 <= KDE <= 4.4 days, each program loaded the entire address book, calendars, and more specialised stuff like email, RSS feeds, and IM chat logs into its memory, so memory usage for PIM data increased linearly with the number of PIM apps running. Same goes for non-PIM apps using PIM data (the Kickoff menu's contact search data, Konqueror's Copy To IM Contact feature). Because Kontact is just a shell for KMail, KAddressbook, KOrganizer etc, it caused the same memory multiplication even though it's all one process.

With the Akonadi design, only the Akonadi process loads all the data into memory. Each PIM app then displays a portion of that data as it needs it, so the amount of extra resources taken by each extra PIM app is smaller, and the initial amount of memory used by each app is less. It should also provide extra stability, because each app no longer has to maintain its own data storage infrastructure, with all the caching, integrity and performance gotchas that keep computing science graduates employed.

Plasma’s new email notifier

Plasma's Email Notifier (alpha version)During Akademy, I’ve picked up the work on Lion Mail again. It is not quite done yet, but already looking very good. I’m nearing feature completion for a first release now, so I’m almost sure it will become part of Plasma 4.6 in January, currently it’s in alpha state but quite fun already. Interested users can of course check out the source from SVN, it’s currently located in playground unti the code is ready for review by my eagle-eyed fellow hackers. Lion Mail is a set of Plasma widgets that can be used to display and manipulate emails in Plasma. In Tampere, Finland, during Akademy, I’ve discussed design and workflow of email notifications with a lot of people, after that I sat down for some serious hacking and have already implement most of what I would like to see, email-wise, in Plasma 4.6. The main goal is to be non-intrusive, and making it as easy as possible to manage your email in your daily workflow. The email notifier provides a queue of your emails, but does so without the need to switch to the full-blown KMail, so no full context / attention switch is needed if you just quickly want to see what’s going on in your inbox.

The new email notifier sits, as you might expect, in your panel’s notification area. It’s hidden by default, but becomes visible when there are new emails in your inbox. When you click on it, you get a list of your new messages. Those messages are expandable, so you can peek into the email to quickly judge if it’s something you want to act upon right away, or not. The individual messages are interactive. When hovered with the mouse, four buttons overlay the email. These buttons allow you to mark an email as read, or important (and of course remove those flags). Disappearing emails slowly fade out, so you have a couple of seconds to undo your action before the list is cleaned up.

You can choose to display new email in arbitrary foldersUnfortunately, most normal email notifiers are pretty useless for high-volume emailers, especially if you use server-side filtering (which you should because it’s much more convenient when using multiple clients, especially mobile ones). In my case, I have about 60 folders, on3 different email accounts (work-work, private, gmail). Emails are filtered before they reach the client. I am personally far less interested in new emails in all folders called “inbox”. The Plasma email notifier allows you to choose the folder you want it to monitor. You can also set up multiple folders. As a nice extra, you can choose to also display emails marked as important, either merged or in a separate list (actually, the latter is not implemented yet, but on my short-term TODO list :-)). Emails are draggable, so you can drag an email from the email notifier into a folder in KMail2 if you want to copy or move it there. As I mentioned before, we’re not actually dragging the email around, but an Akonadi reference as a URL. This is fully transparent between applications, and even across toolkit and access libraries. I’ve also written a full-fledged single email Plasmoid, which allows you to put individual emails on your desktop (or dashboard) for quick reference. Just drag the email from the list onto your desktop, and it’ll appear as Plasma widget there, expandable, with HTML if you’re into that. The missing bits are rather overseeable at this point: clearing the list, separate list for important emails, refreshing logic for individual folders and queuing the re-jigging of the list until the mouse moves out, so items don’t change under your mouse while clicking on them. Not just minor bugfixes, but at the current pace, also not a lot of work left to do.

PUSH IMAP

As you can imagine, the email notifier’s design suits itself very well for PUSH IMAP. PUSH IMAP means that instead of checking in intervals for new email, the server notifies your client when a new email comes in. This means less useless mail checks and more importantly instant notification when a new email arrives. With “instant” I mean within a couple of seconds. In my tests, it took between 3 and 17 seconds from pushing the “Send” button on one machine until the email showed up in the email notifier. That’s pretty neat compared to checking your email every 30 minutes or so. So it’s all the more important that new email notifications become too annoying, hence the non-intrusive approach to the UI. PUSH IMAP is currently only enabled for the inbox folder of a given IMAP account, and of course your server needs to support it. The first email received in Lion Mail using PUSH IMAP

Kontact Mobile

The migration of the desktop version of Kontact is another big step in Akonadi’s existence. Akonadi has its uses outside of Kontact on the desktop as well. The new Kontact Mobile suite builds on top of Akonadi as well, but offers a completely different UI, optimized for smartphones and touch-screen devices. Kontact mobile is part of the upcoming Kontact suite as well. The infrastructure is this way shared across the device spectrum while the user interface is optimized for a certain device and use case. Akonadi does the hard work of talking to all kinds of groupware servers in the background, and caching of this data if you want to make it available on the go. The Dot has an excellent article about Kontact mobile, including a cool screencast.

Beyond Groupware – Akonadi and the Social-Semantic Desktop

The new email notifier is a good example what Akonadi makes possible in the near future, but there are more things brewing in the kitchen. As Akonadi is a generic cache, it comes handy in a much wider number of use cases. In the future, Akonadi can take care of managing and caching all kinds of interesting data, as you can stuff into it what you want. One interesting case is managing your online photo collection. Akonadi can provide standardised photo streams locally on your machine which are backed up by online services. In the same vain, microblogging can be handled through Akonadi, free caching, searchability and semantic linking to your contact are made very easy this way. There is actually already a microblog resource for Akonadi available, I’ve heard rumours of a FlickR one as well…

Emails now also in the desktop searchAkonadi plugs into the Nepomuk semantic framework for its indexing and searching needs. Items in Akonadi are therefore magically available for applications using Nepomuk to query and display data. Tags and other metadata is shared across the desktop, arbitrary items can be linked semantically (think emails and attachments linked to contacts in your address book). Akonadi in Kontact does not only mean that Akonadi is coming to full bloom, but also the semantic desktop built on top of Nepomuk. One nice example is shown in the screenshot, where the KRunner mini-commandline (hit ALT+F2!) also finds emails now. Part of the semantic desktop are also the Activities, which provide context to applications. You can think of ‘context’ as a project your working on, your current location, and many other “metadata” of your workflow. One features of Lion Mail which has been part of its idea from the beginning is showing different sets of emails per activity, the email notifier is built with activities in mind.

Plasma’s dataengines provide another fantastic opportunity for Akonadi to shine. The Plasma team is working on a generic cache for data supplied by dataengines, the idea is to transparently allow caching of arbitrary data from dataengines, so offline usage becomes completely transparent for many Plasma widgets. Akonadi forms one of the cornerstones of Project Silk which aims at deeper integration of online services and content into the user experience.

Concluding

With all the above in mind, there’s little less than a revolution going on in the groupware area. Akonadi matures further and makes possible a full-fledged groupware client in the form of Kontact, with excellent scalability and extensibility. Akonadi is built with a whole spectrum of target devices in mind, which shows in the Kontact Mobile suite running successfully on a N900. With more applications being available in their Akonadi versions, Akonadi will become a lot more useful, and enhance many other applications in the process. Akonadi also allows for a better user experience around email and calendaring in the primary workspace. Groupware is becoming a generic service on the local client. The upcoming new Kontact groupware suite is only the tip of the iceberg of what’s coming thanks to Akonadi. Quite inspiring, isn’t it?

Special thanks go to Will Stephenson for proof-reading the article.

Tags: , , , ,

24 Responses to “Demystifying Akonadi.”

  1. Marc says:

    Hi sebas,

    many thanks for this article.

    So do I get it right that the akonadi database can be excluded from backups without a problem?

    If it gets lost, its not a problem, because its only a cache and will be filled again, right?

    Otherwise I would see a Problem with incremental Backups…

    TIA
    Marc

    • sebas says:

      You’re correct.

      There’s one catch to this of course, if you have changes made that haven’t been synced back to the server, they’ll get lost. Akonadi does change recording, so if you change a couple of flags of your emails for example, they’ll be synced to the server once it becomes available again.

  2. Blackpaw says:

    I’ve seen that circular diagram a few times now and its quite pretty but I think confusing when non-technical people are try to decipher information flow – Resources appear all mixed up with clients.

    Maybe a more linear Layout?, e.g

    IMAP Akonadi Kmail

    etc

    • Blackpaw says:

      Or even:

      IMAP Server Akonadi IMAP Resource Akonadi KDE API KMail

    • Blackpaw says:

      Crap – just noticed the HTML :(

      IMAP <=> Server <=> Akonadi IMAP Resource <=> Akonadi <=> KDE API <=> KMail

    • This is the original architecture diagram so its main intent is not do show data flow but how pieces fit together, what dependencies which layer has and which it provides.

      Like different APIs only depending on the storage access protocol, not each other.
      Or Resources also being clients, just with less or no user interaction

  3. Dion Moult says:

    The promo team has gotten this document up and running for quite some time now – it could do to be read a little bit more by users :) However your post was very comprehensive as well.

    http://community.kde.org/Promo/Strategy/Akonadi

  4. sreich says:

    Looks pretty nice, LionMail looks especially fun.

    Maybe I will eventually be able to move from gmail’s web-app crap to a real application. Yes – I hate browsers :)
    (especially since they never obey my color schemes).

  5. blindauer says:

    Sad to see this direction of “storing/caching all in a DB”. this slows down the system. yes it caches informations for the user, so faster perhaps, but you need to execute some sql server, which isn’t costless. (CPU & storage)
    Sad to see that all is stored in the user home. this broke all NFS users as sqlite, mysql , postgresql doesn’t like to run on NFS. (don’t speak about lock, I’ve tested even nfs4 to be sure).
    This makes KDE go out of business for our university in a near future.
    We have loved the fast and simple kmail for my users compared to thunderbird, but now these email clients are as at same bloat level….

    • sebas says:

      Do you have any backing for your claim that “it slows down the system”? Akonadi is actuallly built to speed up the access to PIM-related data (be it cached or not). Using a common storage mechanism such as MySQL or SQLite makes it actually a lot faster to get at your email for example, especially when we’re talking about sorting or filtering. While filtering through a reasonably sized mailbox took something in the range of minutes in the traditional Kmail, it’s now a matter of seconds. Many other access schemes are also orders of magnitude faster. That’s the effect of sharing the storage and access.
      Remember, Kmail *has* to have some kind of data storage and caching. Using something hand-rolled for this does not magically make it faster. The opposite is true, by re-using an existing database, Kmail benefits from all the optimizations that have gone in there.
      Here’s a question to think about: Is something that brings its own storage mechanism more bloated than an app that uses one that’s commonly available and shares it with other apps?

      The solution to running Akonadi off of a NFS-shared homedir is to use an external MySQL or PostGreSQL server, which can run for example on your terminal server. That way you don’t need to store the data in the homedir, and you don’t get the locking problems you’re talking about. You can configure which backing store to use for Akonadi in the Akonadi configuration module.

      • The impression I get is that akonadi is mostly meant to be a unified cache of what’s available from other sources. This suggests a few things to me:

        It should probably be stored locally, rather than over yet another network connection (IMAP on server->terminal->MySql on server doesn’t sound like it’d be very good for your corporate network).

        It should probably be resilient to the db being deleted from under it.

        What happens if you just move it into /tmp and delete it when the user logs out? Is akonadi well-designed enough to deal with that efficiently, or would it cause the entire IMAP directory to be pulled down on each new login?

        • sebas says:

          I don’t know what will happen then. My assumption would be that your cached data would need to be restored from the backing server. To be sure about that, you’ll better ask on the kde-pim-users list.

      • Stijn Hoop says:

        That so-called ‘solution’ for NFS homedirectories really is not a solution in our case. Running another database server with it’s own backup/upgrade solutions just to run a desktop environment? And you suggest this, just for data that is only there as a “local cache”?

        I have commented multiple times on bugs 182292 and 179006 that we already have a pretty unworkable situation with Akonadi on Fedora 12 / KDE 4.4, but this article makes me afraid of what will happen once we switch to Fedora 14. I hope there’s time for a proper fix, instead of either suggesting edits to hundreds of configuration files, or setting up more servers.

        https://bugs.kde.org/show_bug.cgi?id=182292
        https://bugs.kde.org/show_bug.cgi?id=179006

  6. sebas says:

    This blog has moved. Not all nameservers might have updated yet. If you see this comment, you’re on the new site already.

  7. Jucato says:

    Great post! Take away some of the time/release-related portions and you have the makings of a nice high-level article about Akonadi. IMHO, we need these kinds of bird’s eye-view stuff to communicate better to users and developers the benefits of what our developers have been working on. Give a users why they would want to have Akonadi and attract developers in using the framework.

    We also need these kinds of write-ups for NEPOMUK and Strigi (and maybe perhaps Phonon and Solid?). I saw some content on the community/promo wiki, but something a bit more formatted (or to use aaron’s new favorite word, “elegant”) would probably be good. Maybe something like the old plasma.kde.org or phonon.kde.org sites?

    • sebas says:

      True. I might do a “Demystifying Nepomuk” at some point. There are some other things in life right now that take attention, though. And other important KDE stuff, and work-work.

  8. Ak says:

    Hi,

    This is a great post.

    One question I have about email though is that a lot of the scenarios above are about IMAP. I use KDE on a laptop and it has all my email on it and it is all local (downloaded via POP and removed from the server).

    Is Akonadi built with this scenario in mind too? (It sounds like it is, but safe to ask, I guess!) Also, I currently back up my email by simply backing up my .kde directory. Where would the Akonadi caches etc get stored and would they be quite large potentially?

    Thanks!

    • sebas says:

      I’m not using POP personally, so all I say has to be taken with a grain of salt.

      POP is just as well supported as IMAP, indeed I just took IMAP as an example. So yes, you’ll be able to use Kontact/Akonadi with your setup, too. (Caching emails locally is a generic feature of Akonadi, so that’ll work as well.) As to backing up, there’s a tool which allows you to do easy and safe backups of your Akonadi data, so you’re golden there as well. As to diskspace consumption, I don’t expect any surprises there. (Note that if you use MySQL as backing store, the administrative data of the InnoDB storage mechanism in MySQL will take about 100MB on disk, independent of the amount of data you carry.)

      • Ak says:

        Thanks for taking the time to reply. Makes sense. Sounds like I can continue backing up my main email (if ever I had to restore to a new machine without the Akonadi data, Akonadi would re-index anyway, so whether I back that bit up or not is probably optional I would think?)

      • mps says:

        Hi Sebas,

        This is a great article, many thanks. I have one question – what information does akonadi actually cache for emails? Just the mail headers? Only part of the mail headers? The whole email + attachments? I use a local maildir store for my mails, and when I tried kmail 2 this morning, it used up a huge amount of disk space. What is the estimated cache size in this case? (Maildir volume + 100Mb)? or more like (number of emails x Y kb per email + 100Mb)? It might be good to warn users about this as I was rather surprised by the amount of disk space required.

        cheers
        M.