h2o ๋ฐ h2o4gpu๋ฅผ open source๋กœ๋ถ€ํ„ฐ buildํ•˜๋ ค๋ฉด xgboost4j_gpu.so๊ฐ€ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋Ÿฐ๋ฐ ๊ทธ๊ฑธ buildํ•˜๋ ค๋ฉด ๋˜ pyarrow๊ฐ€ ํ•„์š”ํ•˜์ง€์š”. ํ•˜์ง€๋งŒ, ppc64le ์•„ํ‚คํ…์ฒ˜์—์„œ pyarrow๋ฅผ ์„ค์น˜ํ•˜๋ ค๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์ด error๊ฐ€ ๋‚˜๋Š” ๊ฒƒ์„ ๋ณด์…จ์„ ๊ฒƒ์ž…๋‹ˆ๋‹ค.

$ pip install pyarrow
Collecting pyarrow
Using cached https://files.pythonhosted.org/packages/be/2d/11751c477e4e7f4bb07ac7584aafabe0d0608c170e4bff67246d695ebdbe/pyarrow-0.9.0.tar.gz

[ 66%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/tmp/pip-install-kil31a/pyarrow/build/temp.linux-ppc64le-2.7/lib.cxx:592:35: fatal error: arrow/python/platform.h: No such file or directory
#include “arrow/python/platform.h”
^
compilation terminated.
make[2]: *** [CMakeFiles/lib.dir/lib.cxx.o] Error 1
make[1]: *** [CMakeFiles/lib.dir/all] Error 2
make: *** [all] Error 2
error: command ‘make’ failed with exit status 2

์ด ๋ฌธ์ œ์— ๋Œ€ํ•ด์„œ ์ตœ๊ทผ์— arrow community ๋„์›€์„ ๋ฐ›์•„ ํ•ด๊ฒฐ์„ ํ–ˆ์Šต๋‹ˆ๋‹ค.

https://github.com/apache/arrow/issues/2281

์ข€๋” ๊ฐ„๋‹จํ•˜๊ฒŒ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ •๋ฆฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋จผ์ €, Redhat์—์„œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์‚ฌ์ „ ํ•„์š” fileset๋“ค์„ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค.

[dhkim@ING ~]$ sudo yum install jemalloc jemalloc-devel boost boost-devel flex flex-devel bison bison-devel

[dhkim@ING ~]$ mkdir imsi
[dhkim@ING ~]$ cd imsi

์ด error๋ฅผ ํ•ด๊ฒฐํ•˜๋Š” ํ•ต์‹ฌ์€ ๋จผ์ € arrow์™€ parquet-cpp๋ฅผ source์—์„œ build ํ•˜๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค.

[dhkim@ING imsi]$ git clone https://github.com/apache/arrow.git

[dhkim@ING imsi]$ git clone https://github.com/apache/parquet-cpp.git

[dhkim@ING imsi]$ which python
~/anaconda2/bin/python

์—ฌ๊ธฐ์„œ๋Š” anaconda2๋ฅผ ์‚ฌ์šฉํ•˜๋Š”๋ฐ, anaconda3๋„ ๋™์ผํ•˜๊ฒŒ buildํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋จผ์ € conda ๋ช…๋ น์–ด๋กœ ๋‹ค์Œ๊ณผ ๊ฐ™์€ package๋“ค์„ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค.

[dhkim@ING imsi]$ conda install numpy six setuptools cython pandas pytest cmake flatbuffers rapidjson boost-cpp thrift snappy zlib gflags brotli lz4-c zstd -c conda-forge

์—ฌ๊ธฐ์„œ๋Š” user home directory ๋ฐ‘์— dist๋ผ๋Š” directory์— arrow์™€ parquet-cpp๋ฅผ ์„ค์น˜ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค.

[dhkim@ING imsi]$ mkdir dist
[dhkim@ING imsi]$ export ARROW_BUILD_TYPE=release
[dhkim@ING imsi]$ export ARROW_HOME=$(pwd)/dist
[dhkim@ING imsi]$ export PARQUET_HOME=$(pwd)/dist

[dhkim@ING imsi]$ mkdir arrow/cpp/build && cd arrow/cpp/build

[dhkim@ING build]$ cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE -DCMAKE_INSTALL_PREFIX=$ARROW_HOME -DARROW_PYTHON=on -DARROW_PLASMA=on -DARROW_BUILD_TESTS=OFF ..

[dhkim@ING build]$ make -j 8

[dhkim@ING build]$ make install

— Installing: /home/dhkim/imsi/dist/include/arrow/python/platform.h
— Installing: /home/dhkim/imsi/dist/include/arrow/python/pyarrow.h
— Installing: /home/dhkim/imsi/dist/include/arrow/python/type_traits.h
— Installing: /home/dhkim/imsi/dist/lib64/pkgconfig/arrow-python.pc

[dhkim@ING build]$ cd ~/imsi
[dhkim@ING imsi]$ mkdir parquet-cpp/build && cd parquet-cpp/build

[dhkim@ING build]$ cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME -DPARQUET_BUILD_BENCHMARKS=off -DPARQUET_BUILD_EXECUTABLES=off -DPARQUET_BUILD_TESTS=off ..

[dhkim@ING build]$ make -j 4

[dhkim@ING build]$ make install

— Installing: /home/dhkim/imsi/dist/include/parquet/util/memory.h
— Installing: /home/dhkim/imsi/dist/include/parquet/util/stopwatch.h
— Installing: /home/dhkim/imsi/dist/include/parquet/util/visibility.h

[dhkim@ING build]$ cd ~/imsi/arrow/python

[dhkim@ING python]$ MAKEFLAGS=-j8 ARROW_HOME=/home/dhkim/imsi/dist PARQUET_HOME=/home/dhkim/imsi/dist python setup.py build_ext –build-type=$ARROW_BUILD_TYPE –with-parquet –inplace

[dhkim@ING python]$ export LD_LIBRARY_PATH=/home/dhkim/imsi/dist/lib64:$LD_LIBRARY_PATH

์ด์ œ pyarrow๋ฅผ buildํ•  ์ค€๋น„๊ฐ€ ๋๋‚ฌ์Šต๋‹ˆ๋‹ค. ๋‹ค๋งŒ, arrow ์ชฝ์˜ ์‚ฌ์†Œํ•œ bug๋กœ ์ธํ•ด, ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋์— ์ (.)์ด ๋‹ฌ๋ฆฐ *.so. ๋ผ๋Š” soft link๋“ค์„ ๋งŒ๋“ค์–ด์ฃผ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

[dhkim@ING python]$ ln -s /home/dhkim/imsi/dist/lib64/libarrow_python.so.10.0.0 /home/dhkim/imsi/dist/lib64/libarrow_python.so.
[dhkim@ING python]$ ln -s /home/dhkim/imsi/dist/lib64/libarrow.so.10.0.0 /home/dhkim/imsi/dist/lib64/libarrow.so.
[dhkim@ING python]$ ln -s /home/dhkim/imsi/dist/lib64/libparquet.so.1.4.1 /home/dhkim/imsi/dist/lib64/libparquet.so.
[dhkim@ING python]$ ln -s /home/dhkim/imsi/dist/lib64/libplasma.so.10.0.0 /home/dhkim/imsi/dist/lib64/libplasma.so.

์ด์ œ wheel file์„ build ํ•ฉ๋‹ˆ๋‹ค.

[dhkim@ING python]$ python setup.py build_ext –build-type=release –with-parquet –bundle-arrow-cpp bdist_wheel

๋‹ค์Œ๊ณผ ๊ฐ™์ด dist directory ๋ฐ‘์— ๋งŒ๋“ค์–ด์ง‘๋‹ˆ๋‹ค.

[dhkim@ING python]$ ls -l dist/pyarrow-0.9.1.dev423+g8e6af29-cp27-cp27mu-linux_ppc64le.whl
-rw-rw-r– 1 dhkim dhkim 7195829 Jul 26 16:03 dist/pyarrow-0.9.1.dev423+g8e6af29-cp27-cp27mu-linux_ppc64le.whl

์ด๊ฑธ pip๋กœ ์„ค์น˜ํ•˜๊ณ , import๊นŒ์ง€ ์ž˜ ๋˜๋Š” ๊ฒƒ์„ ํ™•์ธํ•˜์‹ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

[dhkim@ING python]$ pip install dist/pyarrow-0.9.1.dev423+g8e6af29-cp27-cp27mu-linux_ppc64le.whl

[dhkim@ING python]$ pip list | grep pyarrow
pyarrow 0.9.1.dev423+g8e6af29

[dhkim@ING python]$ python
Python 2.7.15 |Anaconda, Inc.| (default, May 1 2018, 23:32:32)
[GCC 7.2.0] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import pyarrow
>>>

ํ† ๋ก  ์ฐธ๊ฐ€

์ด๋ฉ”์ผ์€ ๊ณต๊ฐœ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ํ•„์ˆ˜ ์ž…๋ ฅ์ฐฝ์€ * ๋กœ ํ‘œ์‹œ๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค.