Last week, we released a new sample application: kafka-python-console-sample. It demonstrates how to use the Kafka API using the confluent-kafka Python library. It works exactly the same way as our previous console samples, Java and Node.js, and can be deployed in Bluemix or run locally on macOS and Linux.

The confluent-kafka-python library is still very new but it is based on the excellent librdkafka C/C++ client and it’s maintained by Confluent, the company created by the Kafka founders.

Like our Node.js sample, in order to work in Bluemix, it currently uses a prebuilt binary for confluent-kafka-python.

How to create the prebuilt binary for Bluemix ?

In the sample, we’re providing a prebuilt binary for Python 3.4 with librdkafka 0.9.3. If you want to rebuild it yourself, follow these instructions on Ubuntu 14.04:

  • Install librdkafka (see https://github.com/ibm-messaging/message-hub-samples/blob/master/docs/librdkafka.md)
  • Install confluent-kafka-python:
    pip3 install -r requirements/dev.txt
  • In your project directory create a confluent_kafka_prebuilt sub-directory and copy inside:
    • ¬†¬†¬† librdkafka.so.1 (by default located at /usr/local/lib/)
    • ¬†¬†¬† cimpl.cpython-34m.so and __init__.py from [PYTHON_INSTALL]/site-packages/confluent_kafka
  • Ensure the LD_LIBRARY_PATH in manifest.yml matches the directory name you created above:

    env:
    LD_LIBRARY_PATH: /home/vcap/app/confluent_kafka_prebuilt/

Join The Discussion

Your email address will not be published.