Findka + libpython-clj

New blog post: Serving Findka's recommendations via libpython-clj

Now that those updates to Findka’s algorithm are done, I’m going to, for the third time, work on that Biff-Materialize integration. It’s high priority this time (since it’ll help with analytics which is currently the top priority for me), so it should actually happen now.

Crux includes some code for starting up an embedded Kafka instance in the same process. My plan is to add one of those to Biff, pipe Crux’s transactions to it, and then have Materialize pull from Kafka. It should work nicely on a single machine for low-scale, and then there’s an easy way to scale it out later: put Materialize and Kafka on their own machines.

The main thing I have to get working is piping the Crux transactions into Kafka. Crux has a Kafka plugin to do just that, but sadly it crashes on startup for me. I’ve brought it up on #crux, but in the mean time I think I’ll just reproduce the same functionality from within the Biff process. Most of the code I can simply copy from the Kafka plugin.