KDE Frameworks 5: Plugin Factory Guts

In this article, I explain changes to the plugin loading mechanism in KDE Frameworks 5. The article is intended for a technical audience with some affinity to KDE development.

Over the past weeks, I’ve spent some time reworking the plugin system in KDE Frameworks 5. The original issue I started with is that we are shifting from a plugin system that is mostly KDE specific to making more use of Qt’s native plugin system. Qt’s plugin system has changed in a few ways that caused many of our more complex plugins to not work anymore. On the other side, moving closer to Qt’s plugins makes our code easier to use from a wider range of applications and reduces dependencies for those that just want to do plugin loading (or extending their app with plugins). A mostly complete, and I must say spiffy, solution is now in place, so here’s a good opportunity to tell a little about the technical background of this, what the implications for application developers are, and how you can use a few new features in your plugins.

Bye bye, K_EXPORT_PLUGIN

In the KDE Platform 4, the K_EXPORT_PLUGIN macro did two things. It provided an entry point (qt_plugin_instance()) function which loads the plugin. With Qt5, the need for the entry point is gone, since plugins are now QObject based, so the methods defined in Q_INTERFACE can be relied on as entry points. K_EXPORT_PLUGIN also provided PLUGIN_VERIFICATION_DATA, which can be used to coarsely identify if a plugin was built against the right version. In most cases, this wasn’t very useful, as it would only catch a relatively small class of errors. The plugin verification data is missing in the new implementation so far, but we plan to get it back in another form: being able to specify the version in the plugin, and checking against that. This part is not yet there, but it’s also not a problem for now, as it’s not required and won’t produce fatal errors.

K_PLUGIN_FACTORY

The heavy lifting is done by this macro, which is often used together with K_EXPORT_PLUGIN: You create a factory class using this macro, and then, in the old world, you’d use K_EXPORT_PLUGIN to create the necessary entry points. Since we’re already defining the plugin factory instance using Q_DECLARE_INTERFACE, Qt is happy about that, and the stuff in K_EXPORT_PLUGIN becomes useless. Basically, we’ve moved the interesting bits from K_EXPORT_PLUGIN to K_PLUGIN_FACTORY. For porting that means, in the vast majority of cases, you can just remove K_EXPORT_PLUGIN from your code, and be done. (If you don’t remove it, it’ll warn during build, but will still work, so it’s source-compatible. Mostly, in some cases, .moc can’t pick up the macro, in this case, either move it into the .h file, or include the corresponding .moc file in your .cpp code.)

K_PLUGIN_FACTORY, or rather its base class, KPluginFactory is pretty neat. It’s mostly assembled by macros and templates, which makes it a bit hard to read and understand, but once you realize what kind of effort is saved for you by that, you’ll happily go for it (you don’t have to care about its internals as it is well encapsulated, of course). The really interesting piece is this:

template
T *create(const QString &keyword, QObject *parent = 0, const QVariantList &args = QVariantList());

This is a method available in the factory (generated by K_PLUGIN_FACTORY) that is the base of your plugin, basically what you get from QPluginLoader::instance() from your plugin once you’ve loaded the .so file. You basically call (roughly)

MyFancyObject* obj = pluginLoader->instance()->create<MyFancyObject*>(this);

to load your code into the app hosting the plugin. (Of course, MyFancyObject can be either the class actually defined in the plugin, or, more commonly, the baseclass of it (you don’t want to include your plugin’s header in the app, as that defeats the point of the plugin in the first place). You only do the above if you go through QPluginLoader directly, KService and Plasma::PluginLoader can do most of this work for you (also, here the API didn’t change, so no worries).

K_PLUGIN_FACTORY_WITH_JSON or where is the metadata?

Qt5’s new plugin system allows you to bake metadata into the plugin binary itself. They’re specified as an extra argument to the Q_PLUGIN_METADATA macro, and basically point to a json file containing whatever info you want in the plugin. The metadata is compiled into the ELF section of the plugin, can be found very fast, and the plugin itself doesn’t need to be dlopened in order to read it. With Qt’s previous plugin system, the plugin shared object files would have to be loaded, which significantly impacts performance.

This mechanism is very useful for something we’ve been doing in KDE for a long time, namely the data included in the .desktop files. Those are being installed separately, into a services install dir, indexed by ksycoca for faster access and searching. These .desktop files (which really are the plugin’s metadata contain all the usual stuff, name, icon, author, etc., but also the plugin name, dependencies, and most importantly, the ServiceType (e.g. Plasma/DataEngine). KService uses them to find a plugin (often by service type) and load it from the plugin name.

Having the metadata baked into the plugin allows us to not use KServiceTypeTrader (which handles the searching through the sycoca cache) but to ask QPluginLoader directly. Right now, we’re still using sycoca for the lookup, but this mechanism allows us to move away from it in the future.

Something we do use the metadata for already, at least in Plasma::DataEngine is the creation of a KPluginInfo object. (This object basically exposes the metadata, and can be instantiated from a .desktop file. With the above changes, I also added a constructor to KPluginInfo that instantiates a KPluginInfo object from the json metadata baked into the plugin. This is one nail in the coffin of KServiceTypeTrader (and in extension KSyCoCa), but obviously not its death blow.

K_PLUGIN_FACTORY_WITH_JSON simply takes an extra argument, the metadata file, and bakes that into the plugin (by inserting it, internally, into the Q_PLUGIN_METADATA macro which is included in the KPluginFactory implementation.

kservice_desktop_to_json

In order to ease the transition from .desktop files to baked-in metadata, we introduced a cmake macro to help you with that. It’s pretty simple, you just write (in your CMakeLists.txt):

kservice_desktop_to_json(mypluginmetadata.desktop)

and during build time, a file called mypluginmetadata.json will be generated. You can include this file using the K_PLUGIN_FACTORY_WITH_JSON macro in your code, and the metadata will be baked in. When the plugin is loaded, your ctor will have a QVariantList as argument, which you can just pass to KPluginInfo, and get a valid plugininfo object back. If you’re interested what the .json file looks like, either peak into your build directory, or use the command

$ desktoptojson -i mydesktopfile.desktop

to generate a json file. (You usually want to run this at build time, and not put it in your repo, since otherwise, changes to the .desktop file, for example translations, will not be picked up.)

So, tl;dr

  • Changes are largely source-compatible (K_EXPORT_PLUGIN can just go away, you might have to include the .moc file explicitely)
  • You can optionally use JSON metadata in your plugin to create KPluginInfo objects

If you want to create a plugin, do the following:

  • In your CMakeLists.txt file, convert your old .desktop file at build-time using kservice_desktop_to_json() and use the resulting file (replace .desktop with .json) in the following step
  • In your plugin .cpp file, add a K_PLUGIN_FACTORY macro, this does Q_DECLARE_INTERFACE and Q_PLUGIN_METADATA for you. Optionally pass a .json file
  • Use QPluginLoader, KServiceTypeTrader or Plasma::PluginLoader to load your plugin.

These changes are documented in the API documentation of KPluginFactory and in our KDE5Porting document.

Personal thanks go out to kdelibs hacker extraordinaire David Faure, who has been patiently guiding me through making these changes to our plugin system.

Update by David Faure


One thing it doesn’t detail (because you didn’t directly work on that) is differences in where plugins get installed (the install dir changed), and how they are found ($QT_PLUGIN_PATH).

One of the porting ideas is: if you don’t need the trader to find your plugins, don’t use it anymore. E.g. if you can put all your plugins into a subdirectory of the plugin path, and you don’t need any filtering (you just want to load them all), iterate over that, no servicetype and no trader needed. We still have to solve the use case of filtering/querying though, ideally in Qt (so that the json metadata can actually be useful).

9 Responses to “KDE Frameworks 5: Plugin Factory Guts”

  1. Shaheed Haque says:

    Interesting. So what is story as far as i18n is concerned?

    I ask because today, the Python plugin hosting logic (aka “Pate”) in Kate uses custom logic not based on KTrader or KService to find the plugins. When Pate finds them, it extracts certain descriptive strings from the Python source text, and displays them in the plugin management UI. Sadly, this mechanism bypasses the normal i18n support workflow provided by .desktop files. Thus, there is a pending “TODO” to migrate Pate towards .desktop files (though not using KTrader as its search algorithms are not suitable), and I wonder if there is something here that might be more amenable to cross-language/interpreter-based “plugins”.

    • sebas says:

      Doesn’t sound like it is to me, but I really don’t know details about that. The plugin system I’m talking about is really about C++ plugins.

  2. Phil Thompson says:

    How should proxy plugins work if .desktop meta-data is to be embedded in the plugin? For example, one that acts as a proxy for multiple Python plugins. In other words, when there is not a one-to-one relationship between meta-data and plugin.

    • sebas says:

      That’s two types of plugins then, no? How does it work currently?

      • Nicolas says:

        I assume they have multiple .desktop files referring to the same .so file, with some other line in the .desktop saying what .py file to load.

        • sebas says:

          Ah ok. This doesn’t seem covered by the json metadata in .so file, so another case that needs solving before we’re able to put KSyCoCa asleep.

      • Phil Thompson says:

        Why is that two types of plugin? If a C++ application can be extended with plugins then you can normally get it to support plugins written in pure Python by providing a proxy plugin written in C++. The proxy plugin loads the interpreter and executes the Python code as appropriate. There is only one proxy and it has to have some means of determining which Python code to load – exactly how isn’t important here. The point is that you don’t create a new C++ proxy for each new Python plugin. If there is (Python) plugin-specific meta-data stored in the .desktop file then embedding it in the C++ proxy plugin is going to break things.

        • sebas says:

          Well, it’s a Factory, and the plugin itself.

          I really don’t know enough about how this works, how it worked with the old system, and how it can work with the new system. You know much better what you need, so maybe you could have a look?

          • Shaheed Haque says:

            What would be needed does not exist today, and indirectly underlies my question too.

            I think what is needed is a standard way, either in .desktop files, or in this new mechanism, to get the framework to pick up the metadata via a level of indirection. I assume the difficulty with that concept is the implied need for some level of “activation” of the proxy in order to get to the “real” plugin whereas .desktop files (and this replacement) have till now always contained this metadata directly.

            I’m not sure how best to reconcile the discrepancy.

            The solution in Kate is that the hosting logic Pate is the only real plugin, and has to be activated before it can look for and deal with the real Python plugins using custom logic.