Hadoop : WARN util.NativeCodeLoader

This is one of my daily adventure posts.In this post i am going to discuss the problem and the solution for a problem that occured while running Word Count job on my Hadoop installation on my Ubuntu 13.04.


So this is what,I encountered every time while running the job as well as starting start-all.sh script.

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

This was because the bundled native libraries are for 32-bit systems. We need to build the 64-bit libraries ourselves from the source.

So after googling and trying half a dozen tutorials.I finally found the answer.


Prepare

1. Install utilities for building.

   sudo apt-get install build-essential
   sudo apt-get install g++ autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev
   sudo apt-get install maven

2. Ensure that protobuf 2.5 is installed. Download from https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

   tar xzvf protobuf-2.5.0.tar.gz
   cd protobuf-2.5.0
   /configure --prefix=/usr
   make
   sudo make install

Download Sources

1. Download the source tarball hadoop-2.2.0-src.tar.gz from one of the mirrors.
2. Extract the sources to a convenient location – this path is mentioned as hadoop-2.2.0-src below.

Compile Hadoop

1. There is a known compilation issue with the 2.2.0 sources. Let’s fix it first. Edit hadoop-common-project/hadoop-auth/pom.xml, and insert

   
   org.mortbay.jetty
   jetty-util
   test
   

   above the dependency block for jetty.
2. Compile sources. Make sure you are in the source folder (hadoop-2.2.0-src).

   mvn package -Pdist,native -DskipTests -Dtar

3. It will take a while to complete the compilation process. Back up the files from hadoop-2.2.0/lib/native, and copy the newly generated files in hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/lib/native to hadoop-2.2.0/lib/native.
4. Run the examples again to see if the performance is better. The warnings related to the non-native libraries should be gone. For me, the performance improved significantly after building native 64-bit libraries.