Last week, we released a new sample application: kafka-python-console-sample. It demonstrates how to use the Kafka API using the confluent-kafka Python library. It works exactly the same way as our previous console samples, Java and Node.js, and can be deployed in Bluemix or run locally on macOS and Linux.
The confluent-kafka-python library is still very new but it is based on the excellent librdkafka C/C++ client and it’s maintained by Confluent, the company created by the Kafka founders.
Like our Node.js sample, in order to work in Bluemix, it currently uses a prebuilt binary for confluent-kafka-python.
How to create the prebuilt binary for Bluemix ?
In the sample, we’re providing a prebuilt binary for Python 3.4 with librdkafka 0.9.3. If you want to rebuild it yourself, follow these instructions on Ubuntu 14.04:
- Install librdkafka (see https://github.com/ibm-messaging/message-hub-samples/blob/master/docs/librdkafka.md)
- Install confluent-kafka-python:
pip3 install -r requirements/dev.txt
- In your project directory create a
confluent_kafka_prebuiltsub-directory and copy inside:
librdkafka.so.1(by default located at /usr/local/lib/)
- Ensure the LD_LIBRARY_PATH in
manifest.ymlmatches the directory name you created above:
env: LD_LIBRARY_PATH: /home/vcap/app/confluent_kafka_prebuilt/