Step 2: Map the volume in VirtualBox and permanently mount the volume in boot2docker
> docker-machine-nfs default --shared-folder=/Projects/hadoop
[INFO] Configuration:
- Machine Name: default
- Shared Folder: /Projects/hadoop
- Mount Options: noacl,async
- Force: false[INFO] machine presence ... OK
[INFO] machine running ... OK
[INFO] Lookup mandatory properties ... OK
- Machine IP: 192.168.99.100
- Network ID: vboxnet2
- NFSHost IP: 192.168.99.1
[INFO] Configure NFS ...
!!! Sudo will be necessary for editing /etc/exports !!!
The nfsd service does not appear to be running.
Starting the nfsd service
OK
[INFO] Configure Docker Machine ... OK
[INFO] Restart Docker Machine ... OK
[INFO] Verify NFS mount ...
OK
--------------------------------------------
The docker-machine 'default'
is now mounted with NFS!
ENJOY high speed mounts :D
--------------------------------------------
As seen in the log, sudo is needed to create/update /etc/exports.
Content:
This post shows how to share any MacOS volume in a Docker container. We use
docker-machine together with VirtualBox on MacOS. See an older
post of mine how this can be done.
Share user home directory
Let’s map a directory in the user home /Users/peter/Hadoop to /usr/local/hadoop/test-home
in the Docker container. This should be easy, as the user directory is mounted
by boot2docker.
Step 0: Setup
First initialize the directory and create a test file test.txt in it that can be
used to test if the mapping was successful:
> cd /Users/peter
> mkdir Hadoop
> cd Hadoop
> touch test.txt
> ls
test.txt
Step 1: Map the volumes
Run Docker and map this directory to /usr/local/hadoop/test-home in the
Docker container. For this we use the -v option as follows:
> docker-machine ssh default
...
Boot2Docker version 1.10.2, build master : 611be10 - Mon Feb 22 22:47:06 UTC 2016
Docker version 1.10.2, build c3959b1
docker@default:~$ ls /Users/peter/Hadoop/
test.txt
Check the directory in the Docker container:
bash-4.1# cd /usr/local/hadoop
bash-4.1# ls
LICENSE.txt NOTICE.txt README.txt ... test-home
bash-4.1# ls test-home/
test.txt
/usr/local/hadoop/test-home was generated in the Docker container and we
find the test.txt file.
This was easy, the -v did the job.
Share any directory
Mapping a directory in the user home was easy, as this directory is mounted
automatically in boot2docker. However, this is not the case for directories
outside of the user home directory as we will see below.
Step 0: Setup
First, let’s see what’s in the MacOS /Projects/hadoop directory that we want to map:
> ls /Projects/hadoop
build/ build.gradle settings.gradle src/
In our case, we are especially interested in the build directory with the
generated Hadoop Map/Reduce libraries we want to use in the Hadoop Docker
container.
Step 1: Map the volumes
Start Docker and map the project directory /Projects/hadoop to
/usr/local/hadoop/test-project as above:
Note, that DEVICE must correspond to the logical name of the shared folder defined
in VirtualBox above. If you missed the name, then following error message is returned:
docker@default:~$ sudo mount -t vboxsf -o defaults,uid=`id -u docker`,gid=`id -g docker`\
wrong-logical-name /Projects/hadoop
mount.vboxsf: mounting failed with the error: Protocol error
NODE must correspond an existing file path in boot2docker (not MacOS!). If you missed this name, then
following error message is returned:
docker@default:~$ sudo mount -t vboxsf -o defaults,uid=`id -u docker`,gid=`id -g docker`\
projects-git /wrong-name
mount.vboxsf: mounting failed with the error: No such file or directory
mount: mounting projects-git on /wrong-name failed: No such file or directory
Double-click the DMG file and follow the instructions.
This will install:
VirtualBox
docker-machine
Step 2: Create boot2docker
Use docker-machine to create a default boot2docker
environment for VirtualBox driver. boot2docker is a Linux distribution
based on a stripped down Tiny Core Linux:
Check if the image has been built and is registrated:
> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
sequenceiq/hadoop-docker 2.7.1 9e7e80f84015 9 minutes ago 1.769 GB
Alternatively, you may just pull the image instead of building it yourself (lazy way):
> docker pull sequenceiq/hadoop-docker:2.7.1
Step 7: Run the Docker container
Run the Docker container:
> docker run -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash
/
Starting sshd: [ OK ]
16/02/28 05:13:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [4c2d09c596b0]
4c2d09c596b0: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-4c2d09c596b0.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-4c2d09c596b0.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-4c2d09c596b0.out
16/02/28 05:13:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn--resourcemanager-4c2d09c596b0.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-4c2d09c596b0.out
bash-4.1#
The bash shell is started and is ready to receive your commands.
Check the Java version:
bash-4.1# java -version
java version "1.7.0_71"
Step 8: Check status of Docker container
> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4c2d09c596b0 sequenceiq/hadoop-docker:2.7.1 "/etc/bootstrap.sh -b" 4 hours ago Up 4 hours 2122/tcp, 8020/tcp, 8030-8033/tcp, 8040/tcp, 8042/tcp, 8088/tcp, 9000/tcp, 19888/tcp, 49707/tcp, 50010/tcp, 50020/tcp, 50070/tcp, 50075/tcp, 50090/tcp compassionate_khorana
bash-4.1# ./bin/hadoop version
Hadoop 2.7.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
Compiled by jenkins on 2015-06-29T06:04Z
Compiled with protoc 2.5.0
From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
This command was run using /usr/local/hadoop-2.7.1/share/hadoop/common/hadoop-common-2.7.1.jar
Run the mapreduce test:
bash-4.1# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'
16/02/27 05:54:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/27 05:54:52 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/02/27 05:54:53 INFO input.FileInputFormat: Total input paths to process : 31
...
Check the test output:
bash-4.1# bin/hdfs dfs -cat output/*
16/02/27 05:57:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
6 dfs.audit.logger
4 dfs.class
3 dfs.server.namenode.
2 dfs.period
2 dfs.audit.log.maxfilesize
2 dfs.audit.log.maxbackupindex
1 dfsmetrics.log
1 dfsadmin
1 dfs.servers
1 dfs.replication
1 dfs.file
List output directory:
bash-4.1# bin/hdfs dfs -ls
16/02/28 06:05:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
drwxr-xr-x - root supergroup 0 2016-02-28 05:03 input
drwxr-xr-x - root supergroup 0 2016-02-28 06:04 output
> gradle tasks
[...]
Template tasks
--------------
createGradlePlugin - Creates a new Gradle Plugin project in a new directory named after your project.
createGroovyClass - Creates a new Groovy class in the current project.
createGroovyProject - Creates a new Gradle Groovy project in a new directory named after your project.
createJavaClass - Creates a new Java class in the current project.
createJavaProject - Creates a new Gradle Java project in a new directory named after your project.
createScalaClass - Creates a new Scala class in the current project.
createScalaObject - Creates a new Scala object in the current project.
createScalaProject - Creates a new Gradle Scala project in a new directory named after your project.
createWebappProject - Creates a new Gradle Webapp project in a new directory named after your project.
exportAllTemplates - Exports all the default template files into the current directory.
exportGroovyTemplates - Exports the default groovy template files into the current directory.
exportJavaTemplates - Exports the default java template files into the current directory.
exportPluginTemplates - Exports the default plugin template files into the current directory.
exportScalaTemplates - Exports the default scala template files into the current directory.
exportWebappTemplates - Exports the default webapp template files into the current directory.
initGradlePlugin - Initializes a new Gradle Plugin project in the current directory.
initGroovyProject - Initializes a new Gradle Groovy project in the current directory.
initJavaProject - Initializes a new Gradle Java project in the current directory.
initScalaProject - Initializes a new Gradle Scala project in the current directory.
initWebappProject - Initializes a new Gradle Webapp project in the current directory.
[...]
As you can see, there are many different templates you may use. For our purpose, the createWebappProject task seems to be interesting.
Step 1: Create the web project
Use the createWebappProject task for creating a new fresh web project. Invoke the task and answer some questions:
> gradle createWebappProject
:createWebappProject
templates> Project Name: (WAITING FOR INPUT BELOW)> Building 0% > :createWebappProjecttestweb
templates> Project Parent Directory: [/Development/java/projects/projects-test] (WAITING FOR INPUT BELOW)> Building 0% > :createWebappProject
templates> Use Jetty Plugin? (Y|n)[n] (WAITING FOR INPUT BELOW)> Building 0% > :createWebappProjectY
templates> Group: [testweb] (WAITING FOR INPUT BELOW)> Building 0% > :createWebappProject
templates> Version: [0.1] (WAITING FOR INPUT BELOW)> Building 0% > :createWebappProject
BUILD SUCCESSFUL
Total time: 25.877 secs
> gradle tasks
Build tasks
-----------
[...]
war - Generates a war archive with all the compiled classes, the web-app content and the libraries.
[...]
Web application tasks
---------------------
jettyRun - Uses your files as and where they are and deploys them to Jetty.
jettyRunWar - Assembles the webapp into a war and deploys it to Jetty.
jettyStop - Stops Jetty.
[...]
Step 2: Create your first page and run it
Create the landing page:
> echo"Hello" > src/main/webapp/index.html
And finally, run the web application in Jetty:
> gradle jettyRun
Starting a new Gradle Daemon for this build (subsequent builds will be faster).
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
> Building 75% > :jettyRun > Running at http://localhost:8080/testweb