Cloud-friendly Classloading with RabbitMQ
One of the things everyone who deploys artifacts into the cloud has to deal with is the issue of classloading. If you have multiple nodes out there, listening to your RabbitMQ server, waiting to do work, you have to have pre-deployed all the dependencies you need. This means some system to either copy them out there automatically (in the case of deployable artifacts), or you simply have to copy the JAR files into a lib/ directory somewhere that the listener has access to.
None of these solutions is ideal.
I was contemplating this on my way to work the other day and I've come up with a solution that I'm most of the way finished coding: write a ClassLoader that uses RabbitMQ to load the class data from a "provider" (just a listener somewhere in the cloud that actually does have that class in its CLASSPATH).
There are two moving parts: a Provider and a ClassLoader. The Provider has a number of message listeners and binds them to the configured exchange with routing keys that could be "com.mycompany.cloud.#", or "com.thirdparty.#", or simply "#". The routing key is the class or resource name, so you could have different providers for different areas of responsibility. Third-party classes could come from one provider, while your own internal class files could come from an entirely different provider (ostensibly running on a different VM).
Some potential uses:
- You could provide added layers of class file security because you could control exactly where class files come from without exposing those class files to be copied to the file system.
- Providing class files dynamically to nodes that come up and down based on system demand but still need to do work that requires those individual classes. Amazon EC2 instances would not need to be pre-populated with JAR files, simply configured to use the cloud classloader pointed to your RabbitMQ server.
- Wrap normal classloading with some AOP hooks that would cloud-ify an entire installation without touching the source code or using special configurations.
Point number 3 is the most interesting to me. Using Spring AOP, one could wrap normal classloading with a cloud-friendly version, which would alter the way all your classloaders work, without having to hack on the Tomcat source code (or whatever application you're deploying).
I suspect I'll write a Maven-aware Provider that will search maven repositories for requested class files. I'm sure there are other possibilities here.
UPDATE: Pushed this code to Github: http://github.com/jbrisbin/vcloud/tree/master/amqp-classloader/
As always, patches and feedback are eagerly sought and heartily welcomed.
