Using pygraphviz to plot OSGi bundle dependencies

I’ve been working on an OSGi project for the last few years. As with any project, evolutionary changes will eventually require some cleanup. As new bundles have been added over time, the graph of dependencies is starting to get unwieldy in places. Even with good management of these dependencies, a nice visual layout of things can really help you see how your bundles are interconnected and give you the power to start separating some connections if you graph starts to hit cyclical dependencies.

I drove around a few different visual tools in Eclipse (PDE visualization tool, m2eclipse dependency graph) but needed something that would narrow the view to just my project. I not only wanted to see what things my bundles depend on in the project but I wanted to see what depends on a given bundle.

This was my first project with pygraphviz but after I figured out which way the grain runs, I was able to give it a basic graph of my project and generate the diagrams I wanted. I finally let pygraphviz handle the graph traversing and things got better (less code, faster results).

Since I wanted to analyze the runtime dependencies of my server, I installed the Felix Remote Shell bundle along with the Felix Shell bundle to allow remote connections to management functionality. With these in place, I was able to connect via telnet and query for package level information to build my graph.

Using the code below, I was able to generate a graph of the entire project (successors) and a graph for each bundle in the graph (predecessors and successors). Some sample images are below the source code. Eventually I’ll add saving and opening of dot files to allow for analysis without a running server.

#! /usr/bin/env python
import re
from sets import Set
import os
import pygraphviz as pgv
import sys
import telnetlib

# [   1] [Active     ] [   15] org.sakaiproject.nakamura.messaging (0.11.0.SNAPSHOT)
bundle_from_ps = re.compile('^\[\s*(?P<bundle_id>\d+)\]\s\[.+\]\s(?P<bundle_name>.+)\s')

# org.sakaiproject.nakamura.api.solr; version=0.0.0 -> org.sakaiproject.nakamura.solr [11]
bundle_from_req = re.compile('^.*-\> (?P<bundle_name>.*) \[(?P<bundle_id>.*)\]$')

def get_sakai_bundles():
    """Get a list of bundles that are create as part of Sakai OAE.
    Returns a dictionary of dict[bundle_name] = bundle_id.
    """
    tn = telnetlib.Telnet('localhost', '6666')
    tn.write('ps -s\nexit\n')
    lines = [line for line in tn.read_all().split('\r\n') if 'org.sakaiproject' in line]

    bundles = {}
    for line in lines:
        m = bundle_from_ps.match(line)
        bundles[m.group('bundle_name')] = m.group('bundle_id')
    return bundles

def get_package_reqs(bundle_id):
    """Gets the requirements (imports) for a given bundle.
    Returns a dictionary of dict[bundle_name] = bundle_id.

    Keyword arguments:
    bundle_id -- Bundle ID returned by the server in the output of
                 get_sakai_bundles()
    """
    tn = telnetlib.Telnet('localhost', '6666')
    tn.write('inspect package requirement %s\nexit\n' % (bundle_id))
    lines = [line for line in tn.read_all().split('\r\n') if line.startswith('org.sakaiproject') and line.endswith(']')]

    reqs = {}
    for line in lines:
        m = bundle_from_req.match(line)
        reqs[m.group('bundle_name')] = m.group('bundle_id')
    return reqs

def build_bundle_graph():
    """Build a graph_attr (nodes, edges) representing the connectivity
    within Sakai bundles
    """
    sakai_bundles = get_sakai_bundles()
    bundles = {}
    for b_name, b_id in sakai_bundles.items():
        reqs = get_package_reqs(b_id)
        bundles[b_name] = reqs.keys()
    return bundles

def draw_subgraph(name, graph, filename, successors = True):
    """Draw (write to disk) a subgraph that starts with or ends with
    the specified node.

    Keyword arguments:
    name -- name of the node to focus on
    graph -- graph of paths between bundles
    filename -- filename to write subgraph to
    successors -- whether to lookup successors or predecessors
    """
    if successors:
        nbunch = graph.successors(name)
    else:
        nbunch = graph.predecessors(name)

    if nbunch:
        nbunch.append(name)
        subgraph = graph.subgraph(nbunch)
        subgraph.layout(prog = 'dot')
        subgraph.draw('graphviz/%s' % filename)

def main():
    if not os.path.isdir('graphviz'):
        os.mkdir('graphviz')

    bundles_graph = build_bundle_graph()

    # print the whole graph
    graph = pgv.AGraph(data = bundles_graph, directed = True)
    graph.layout(prog = 'dot')
    graph.draw('graphviz/org.sakaiproject.nakamura.png')

    for b_name, reqs in bundles_graph.items():
        draw_subgraph(b_name, graph, '%s.png' % b_name)
        draw_subgraph(b_name, graph, '%s-pred.png' % b_name, False)

if __name__ == '__main__':
    main()
Sakai OAE Bundle Dependency Graph
Sakai OAE Bundle Dependency Graph
Sakai OAE Solr Bundle Predecessors
Sakai OAE Solr Bundle Predecessors
Sakai OAE Presence Bundle Successors
Sakai OAE Presence Bundle Successors

Basics of OSGi and Declarative Services

This is a slide deck for a talk I gave at the Sakai 2010 conference in lovely Denver, Colorado. The title to the talk was “Using OSGi in Nakamura” which was my original intention, but after reviewing the slides I seemed to have given a talk about the basics of OSGi. Good information under a wrong marquee.

No matter the intention, I found today that I needed to refer to the slides for setting up an activator, so I’m reposting it here to give myself and hopefully others faster access. Be on the lookout for a repeat performance of this presentation at the Sakai 2011 conference in Los Angeles. I might even try to show it around a bit once the European conference is announced.

When To Immediately Activate An OSGi Component

OSGi has a fantastic feature for immediate components [1] and delayed components [2]. This allows components to delay their possibly expensive activation until the component is first accessed. At the very least it allows the OSGi platform to consume resources as needed. No sense is sucking up those server resources for something that can wait.

As developers start their adventure into OSGi, I’ve noticed a pattern creeping up that I’m not sure folks are aware of: just because you like your component doesn’t mean it needs to start immediately. Unless your component has something it needs to do before other components can do their work, there’s little need to start a component immediately. A lot of times, the work can be moved into the bundle activator or is fine being delayed.

People often see that a component does something in its activation method and assume it should start immediately. The questions to ask yourself are:

  • Will the things that need to use this component have a reference to it?
    Yes? Delayed activation.
  • Before activation, will functionality be missing that should be available as soon as possible?
    Yes? Immediate activation.
  • Can something else do the job of activating the component at a later time?
    Yes? Delayed activation.

Registering with an external service, such as JMS, is a perfect scenario for an immediate component. If the JMS service were in OSGi, declarative services could do the work of making sure service trackers know your component exists, but OSGi has no jurisdiction over external resources. Binding to a JMS topic requires manual interaction and therefore should happen by way of immediate component activation.

Getting and setting of properties is not a case for immediate component activation. Neither is because the service is referenced by something else.

When using declarative services, a reference to your not-yet-started component will be given to anything that has said it wants to collect an interface you implement. This requires no effort on the part of the service being referenced and thusly does not require the referenced service to be activated immediately. Fret not! On first access of this service by any consumer the service will get activated.

OSGi has not been the fastest thing for me to learn but understanding the nuances of little things like this really helps me understand why it is a good approach at decoupled services and modular software construction.

1. OSGi compendium spec version 4.2, section 112.2.2
2. OSGi compendium spec version 4.2, section 112.2.3

LDAP Authentication & Authorization Dissected and Digested

LDAP is one of those things that I’ve integrated with a few times but never put enough energy into to really get the details or understand it much.  There’s always been someone I can bounce questions off of and thankfully those people were available again as I started working out the details of performing LDAP authentication.

The steps below are general enough to be used by anyone and will hopefully shed some light into the steps performed in LDAP authentication.  The process below also includes some steps for authorization.

Authentication

1.  Get a connection to the LDAP server.
With the host and port for the LDAP server, create a connection to it.  This can be  a simple direct connection or a pooled connection.  If more than a basic test, it is best to use a pooled connection.  Log and fail if a connection cannot be created.

2.  Bind as the application user.
Bind the connection to the application user.  This user should have enough permissions to search the area of LDAP where users are located.  This user may also have permissions to search for things outside of the users area (groups, authorization).  Log and fail if the application user cannot bind.

3.  Search for the DN (distinguished name) of the user to be authenticated.
This is where we verify the username is valid.  This does not authenticate the user but simply makes sure the requested username exists in the system.  Log and fail if the user’s DN is not found.

4.  Bind as user to be authenticated using DN from step 3.
Now for the moment of truth.  Bind to the connection using the DN found in step 3 and the password supplied by the user.  Log and fail if unable to bind using the user’s DN and password.

Authorization

5.  Re-bind as application user.
To check the authorization of a user, we need to read attributes from the user’s account. To do this, we need to re-bind as the application user.

6.  Search for user and require attributes.
A filter is used to search for the user like was done in step 3 but we’ll add an extra check to the query to look for the attributes that show we’re authorized.

Example Code

The code shown here is using the JLDAP library from Novell.  Interesting includes are noted at the top. The utility class used for escaping the search filter can be found in the Sakai Project’s Nakamura codebase.

import com.novell.ldap.LDAPAttribute;
import com.novell.ldap.LDAPConnection;
import com.novell.ldap.LDAPEntry;
import com.novell.ldap.LDAPException;
import com.novell.ldap.LDAPSearchResults;
// ...
String baseDn = "ou=People,o=MyOrg";
String userFilter = "uid = {}";
String authzFilter = "authzAttr=special:Entitlement:value";
// ...
public boolean authenticate(Credentials credentials) throws RepositoryException {
    boolean auth = false;
    if (credentials instanceof SimpleCredentials) {
      // get application user credentials
      String appUser = connMgr.getConfig().getLdapUser();
      String appPass = connMgr.getConfig().getLdapPassword();

      // get user credentials
      SimpleCredentials sc = (SimpleCredentials) credentials;

      long timeStart = System.currentTimeMillis();

      String userDn = LdapUtil.escapeLDAPSearchFilter(userFilter.replace("{}",
          sc.getUserID()));
      String userPass = new String(sc.getPassword());

      LDAPConnection conn = null;
      try {
        // 1) Get a connection to the server
        try {
          conn = connMgr.getConnection();
          log.debug("Connected to LDAP server");
        } catch (LDAPException e) {
          throw new IllegalStateException("Unable to connect to LDAP server ["
              + connMgr.getConfig().getLdapHost() + "]");
        }

        // 2) Bind as app user
        try {
          conn.bind(LDAPConnection.LDAP_V3, appUser, appPass.getBytes(UTF8));
          log.debug("Bound as application user");
        } catch (LDAPException e) {
          throw new IllegalArgumentException("Can't bind application user [" + appUser
              + "]", e);
        }

        // 3) Search for username (not authz).
        // If search fails, log/report invalid username or password.
        LDAPSearchResults results = conn.search(baseDn, LDAPConnection.SCOPE_SUB, userDn,
            null, true);
        if (results.hasMore()) {
          log.debug("Found user via search");
        } else {
          throw new IllegalArgumentException("Can't find user [" + userDn + "]");
        }

        // 4) Bind as user.
        // If bind fails, log/report invalid username or password.

        // value is set below. define here for use in authz check.
        String userEntryDn = null;
        try {
          LDAPEntry userEntry = results.next();
          LDAPAttribute objectClass = userEntry.getAttribute("objectClass");

          if ("aliasObject".equals(objectClass.getStringValue())) {
            LDAPAttribute aliasDN = userEntry.getAttribute("aliasedObjectName");
            userEntryDn = aliasDN.getStringValue();
          } else {
            userEntryDn = userEntry.getDN();
          }

          conn.bind(LDAPConnection.LDAP_V3, userEntryDn, userPass.getBytes(UTF8));
          log.debug("Bound as user");
        } catch (LDAPException e) {
          log.warn("Can't bind user [{}]", userDn);
          throw e;
        }

        if (authzFilter.length() > 0) {
          // 5) Return to app user
          try {
            conn.bind(LDAPConnection.LDAP_V3, appUser, appPass.getBytes(UTF8));
            log.debug("Rebound as application user");
          } catch (LDAPException e) {
            throw new IllegalArgumentException("Can't bind application user [" + appUser
                + "]");
          }

          // 6) Search user DN with authz filter
          // If search fails, log/report that user is not authorized
          String userAuthzFilter = "(&(" + userEntryDn + ")(" + authzFilter + "))";
          results = conn.search(baseDn, LDAPConnection.SCOPE_SUB, userAuthzFilter, null,
              true);
          if (results.hasMore()) {
            log.debug("Found user + authz filter via search");
          } else {
            throw new IllegalArgumentException("User not authorized [" + userDn + "]");
          }
        }

        // FINALLY!
        auth = true;
        log.info("User [{}] authenticated with LDAP in {}ms", userDn,
            System.currentTimeMillis() - timeStart);
      } catch (Exception e) {
        log.warn(e.getMessage(), e);
      } finally {
        connMgr.returnConnection(conn);
      }
    }
    return auth;
  }

Setting Up RVM, Ruby, Gem and Rails on Ubuntu

This is centered on Ubuntu only because aptitude is used once. If you’re comfortable with your package manager feel free to replace the aptitude command with your local command.

I’m beginning to dig deeper into ruby, rails and the whole suite of tools available for them and have hit my knee on several things along the way. I’m no stranger to downloading things to download things to do things; I’ve been doing that in Java since 1996. I’ve also been kicking the tires on Ubuntu sense back before they got their naming scheme together and Debian even before that. I totally dig the Debian/Ubuntu way of install by deb packages. But with ruby, like Java, I feel it is best setup a machine by minimizing the use of apt (apt-get, aptitude, etc) and downloading the tools you need directly. I don’t let aptitude install Eclipse, Xalan, servlet-api or any of the other tools/libraries I use on top of the JVM and this is the route I have ended up going with when setting up ruby as well.

TL;DR: short list of what to do

This short list is for Ruby 1.9.2. If you want to install an earlier version, be sure to follow the extra steps about installing gem from rubygems.org.

  1. Install RVM [details]:
    bash < > ~/.bashrc  # This loads RVM into a shell session.
    source ~/.bashrc
  2. Install a C compiler for RVM [details]:
    sudo aptitude install gcc
  3. Install Ruby 1.9.2 using RVM [details]:

    See below for instructions on installing RubyGems when using Ruby < v1.9.2

    rvm pkg install zlib
    rvm install 1.9.2 --default -C --with-zlib-dir=$HOME/.rvm/usr
  4. Install Rails using RubyGems [details]:
    gem install rails

Details:

Install RVM

Note: The full install instructions have much better details.

RVM (Ruby Version Manager) gives us a way to install Ruby without getting it from aptitude. Being that Ruby is our starting point for everything after RVM and RVM is our means of getting Ruby, everything is downhill after getting RVM. You could install ruby using aptitude then install rvm as a gem but what’s the point of downloading Ruby just to download stuff to eventually download Ruby (again) and remove the system local stuff that was installed the first time. Let’s just cut out all the cruft in the middle.

Install a C compiler for RVM

Since RVM setups up the Ruby interpreter, it has to act 1 level deeper than the interpreter and C is that next level down. GCC is a very common tool to have on a linux machine, so this step is mostly for setting up a new machine. It’s always a good idea to have GCC installed. And vi. But this is a tech blog and not for theology, so I won’t go into that.

 

Install Ruby 1.9.2 using RVM

The big moment! All this work to get Ruby installed so we can begin our journey into RoR-land. This will take some time to perform as it needs to download, configure and compile Ruby.

 

Install RubyGems from rubygems.org for Ruby < v1.9.2

Note: The full install instructions have much better details.

wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz
tar zxf rubygems-1.3.7.tgz
cd rubygems-1.3.7
ruby setup.rb

Ruby 1.9.2 includes RubyGems 1.3.7, so we don’t have to install it when we install Ruby, but for previous versions we’ll need to install this bit ourselves. You can install it from aptitude but it locks you from doing system updates and some other functionality I wasn’t keen on losting. Installing it from rubygems.org gives back the freedom to break my system in whatever way I choose.

Install Rails using RubyGems

This is where things get down to Ruby’s style of easy. Package management that looks like an already very easy package manager (aptitude) is a bonus in my book. Installing gems will become something you most likely get very comfortable with as this is the easiest way to get the libraries you need for your Ruby installation. Read more into RVM Gemsets to see where this stuff gets even cooler.
Note: If you get this error message:

ERROR:  Loading command: install (LoadError)
    no such file to load -- zlib


take a look at this page for installing Ruby with zlib. I tripped over this nicely.

Setting “quoted-printable” In A commons-email Body Part

I’m revamping the inner workings of a mail sending tool to use commons-email.  I wanted to make sure I set the text body parts to the appropriate transfer encoding to allow for non-ASCII languages to be used. My failed attempts and final success are documented below.

1. [bad] Set Message ‘Content-transfer-encoding’

The problem with this is that everything in the message is processed as “quoted-printable” which will turn your multipart boundaries into useless wastes of text.

// this chews up your multipart boundary
HtmlEmail email = new HtmlEmail();
email.addHeader("Content-transfer-encoding", "quoted-printable");

So rather than having this in your raw email (note reported and actual boundary match):

Content-Type: multipart/alternative;
boundary="----=_Part_7_1164835397.1283397465477"

------=_Part_7_1164835397.1283397465477
Content-Type: text/plain; charset=UTF-8

You end up with this in your raw email (note: actual boundary has = replaced by =3D):

Content-Type: multipart/alternative;
boundary="----=_Part_7_1164835397.1283397465477"

----=3D_Part_7_1164835397.1283397465477
Content-Type: text/plain; charset=UTF-8

2. [useless] Setting Each Body Part’s transfer encoding

I’ll save you the pain. This begins to turn commons-email back into javamail. It saves you nothing, adds lots of code, doesn’t really work and, given the next solution, is the wrong approach.

3. [Best!] Let commons-email do it’s job

The solution I ended up with is to just let commons-email figure it out. My test text is “You get that thing I sent you?\n3x6=18\nåæÆД. This has several non-ASCII characters that are just ripe for conversion. The raw bit of email I find is:
------=_Part_4_644193719.1283397465436
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

You get that thing I sent you?
3x6=3D18
=C3=A5=C3=A6=C3=86=C3=90

The content-transfer-encoding is set as I expected and the content has been properly escaped. Kudus to the commons-email team for making this just happen.