Virtuoso, here I come!

Nepomuk, which is KDE’s semantic desktop framework is a very interesting new technology, and has the potential to move many applications forward. Nepomuk needs an RDF store to save and retrieve semantic information, such as data from your filesystem indexer, tags, ratings, and other, much more complex bits. Up until now, there were performance problems. There were two possible ways to store your information: redland and sesame2. Redland is written in C++, but is very simple and by far not meeting the performance requirement we need for Nepomuk’s use-cases. The Sesame2 backend is a lot better in terms of performance, though still not quite up to the task. What is more, it requires certain Java files which have licensing problems — in the end many distributors just omit shipping this backend. That way performance of Nepomuk goes from bad to worse for many users.
Sebastian Trueg, KDE’s main developer of the Semantic Desktop is of course aware of this problem, so he has been working hard to find a solution for the RDF storage part which is both up to the task in terms of performance and stability. This solution is Virtuoso, and Sebastian has now a working backend for Soprano (which is the Qt-style storage interface for Nepomuk). This means that the Redland or Sesame2 storage mechanisms for Nepomuk can now be replaced by the much faster Virtuoso store. For the user this has important advantages: faster search, tagging, rating and longer battery life on mobile devices.
So my project for tonight was: Switch the Nepomukserver on my development machine to Virtuoso. And apparently, I succeeded.

Here’s my quick walkthrough if you want to do the same:

  1. Download Virtuoso 5.0.12 or later here
  2. ./configure --prefix=/home/sebas/kdesvn/install (or whereever you install your KDE trunk)
    make -j3
    make install 
  3. install libiodbc2-dev (on Kubuntu or libiodbc-dev on Debian)
  4. rebuild kdesupport/soprano (possibly delete CMakeCache.txt),
    make sure Virtuoso backend is built

    -- Found iODBC 3.52.6: libs - /usr/lib/libiodbc.so; includes - /usr/include
    -- Performing Test __SOPRANO_HAVE_GCC_VISIBILITY
    -- Performing Test __SOPRANO_HAVE_GCC_VISIBILITY - Success
    ---------------------------------------------------------------------------------------
    -- Soprano Components that will be built:
       * Sesame2 storage backend (java-based)
       * Virtuoso storage backend (Run-time dependancy: Virtuoso)
       * Raptor RDF parser
         (including TriG parser)
       * The CLucene-based full-text search index library
       * D-Bus server/client support
    
    -- Soprano Components that will NOT be built:
       * Redland storage backend
       * Raptor RDF serializer
    
  5. Rebuild kdebase.
  6. Edit ~/.kde{4}/share/config/nepomukserverrc and put Soprano Backend=virtuosobackend ins the [Basic Settings] section. Mine looks like:

    [Basic Settings]
    Configured repositories=main
    Start Nepomuk=true
    Soprano Backend=virtuosobackend
    

    Obviously, the virtuosobackend line is the important one.

  7. That’s it for the setup. Now you need to restart the nepomukserver.

    // Check which backend we're currently using
    luna.sebas(~): qdbus org.kde.NepomukStorage /nepomukstorage org.kde.nepomuk.Storage.usedSopranoBackend                                                                                                                                                                                sesame2
    
    // Stop the nepomukserver
    luna.sebas(~): qdbus org.kde.NepomukServer /nepomukserver org.kde.NepomukServer.quit
    
    // Start the nepomukserver
    luna.sebas(~): nepomukserver
    
    // Make sure the Virtuoso backend is used
    luna.sebas(~): qdbus org.kde.NepomukStorage /nepomukstorage org.kde.nepomuk.Storage.usedSopranoBackend
    virtuosobackend
    
    

    You’ll notice that it automatically starts converting your data from sesame2 (or possibly redland) format to the new Virtuoso format, this will show up as a job in your plasma notification area, and it might take a long time, depending on how much data you have. Go get a coffee now.

This worked for me. I was glad to find information on The Other Sebastian’s blog and on techbase.

Please leave a comment reporting your findings.

5 Responses to “Virtuoso, here I come!”

  1. openSUSE users can get the packages to build and run the Virtuoso backend (including minimal 5.0.12 packages) by registering KDE:KDE4:UNSTABLE:Desktop and installing virtuoso-drivers and libiodbc-devel.

  2. Does this also work with non-trunk (i.e. 4.3 branch)? I gave a try with a newer version of Soprano and Virtuoso, but no dice. I just managed to crash nepomukserver if I asked for the backend.

  3. Anon says:

    ” Redland is written in C++”

    Actually straight C, as far as I can tell.

  4. I have already compiled virtuoso support without having it installed because it’s a runtime dep, so I don’t think I need to rebuild kdesupport, and then I wonder why should I rebuild kdebase, i.e. is there anything in kdebase that depends directly on the virtuoso backend? Sounds unlikely. Anyway I’ll try to switch to Virtuoso too, thanks for the howto Sebas =).

  5. trooper says:

    My biggest problem setting up was getting other KDE components to compile, including kdebase (kde svn was a mess of broken compiles when I did it). Finally got it going and seemed to convert ok. I have 5gb index and 500mb metadata. Strigi still seemingly stalls indefinitely parsing certain directories. Dolphin froze randomly browsing and using nepomuksearch:/ guaranteed, occasionally. But when it is working right, virtuoso is visibly much better performance than sesame2 and Dolphin/gwenview were actually usable again though a little laggy. Still work to go, but a great improvement.