The NFS Gateway supports NFSv3 and allows HDFS to be mounted as part of the clientâ€™s local file system. Currently NFS Gateway supports and enables the following usage patterns:
- Users can browse the HDFS file system through their local file system on NFSv3 client compatible operating systems.
- Users can download files from the the HDFS file system on to their local file system.
- Users can upload files from their local file system directly to the HDFS file system.
- Users can stream data directly to HDFS through the mount point. File append is supported but random write is not supported.
The NFS gateway machine requires the same dependencies as a HDFS client; Hadoop JAR files in PATH, HADOOP_CONF directory. The NFS gateway can be on the same host as DataNode, NameNode, or any HDFS client.
On a user command-line, the speed improvement is significantly noticeable:
|Command||Hadoop fs – Time (user)||NFS Mount – Time (user)|
1. Install necessary packages via yum
yum install nfs-utils -y
2. If mounting hdfs as user root, add the following properties via ambari to HDFS > Configs
3. Stop the OS-provided nfs and rpcbind services
service nfs stop
service rpcbind stop
4. Start the hadoop-provided portmap and nfs3 services ( Add these lines to /etc/rc.local to mount hdfs on system startup )
/usr/iop/220.127.116.11/hadoop/sbin/hadoop-daemon.sh --script /usr/iop/18.104.22.168/hadoop/bin/hdfs start portmap
/usr/iop/22.214.171.124/hadoop/sbin/hadoop-daemon.sh --script /usr/iop/126.96.36.199/hadoop/bin/hdfs start nfs3
5. Create a local directory to use at the mount point, and mount hdfs – replace 0.0.0.0 with IP address shown in step 4
mkdir -p /data/hdfs
mount -t nfs -o vers=3,proto=tcp,nolock 0.0.0.0:/ /data/hdfs/
6. Enjoy native linux commands, as well as speed for accessing HDFS =)