Share volumes in a Docker Container on MacOS - simplified

In the last post I showed how to share a volume in a Docker container on MacOS. There was a way, but it was not really easy.

Alternatively, there is another way that’s much easier: use docker-machine-fs.

Step 1: Install docker-machine-nfs

Standalone:

curl -s https://raw.githubusercontent.com/adlogix/docker-machine-nfs/master/docker-machine-nfs.sh |
  sudo tee /usr/local/bin/docker-machine-nfs > /dev/null && \
  sudo chmod +x /usr/local/bin/docker-machine-nfs

Or using Homewbrew:

brew install docker-machine-nfs

Step 2: Map the volume in VirtualBox and permanently mount the volume in boot2docker

> docker-machine-nfs default --shared-folder=/Projects/hadoop
[INFO] Configuration:

	- Machine Name: default 
	- Shared Folder: /Projects/hadoop 
	- Mount Options: noacl,async 
	- Force: false 

[INFO] machine presence ...                  OK 
[INFO] machine running ...                   OK 
[INFO] Lookup mandatory properties ...       OK 

	- Machine IP: 192.168.99.100 
	- Network ID: vboxnet2 
	- NFSHost IP: 192.168.99.1 

[INFO] Configure NFS ... 

 !!! Sudo will be necessary for editing /etc/exports !!! 
The nfsd service does not appear to be running.
Starting the nfsd service
                                             OK 
[INFO] Configure Docker Machine ...          OK 
[INFO] Restart Docker Machine ...            OK 
[INFO] Verify NFS mount ... 			


OK 

--------------------------------------------

 The docker-machine 'default'
 is now mounted with NFS!

 ENJOY high speed mounts :D

--------------------------------------------

As seen in the log, sudo is needed to create/update /etc/exports. Content:

/Projects/hadoop 192.168.99.100 -alldirs -mapall=501:20

Note, that in boot2docker the file /mnt/sda1/var/lib/boot2docker/bootlocal.sh is overridden:

docker@default:~$ cat /mnt/sda1/var/lib/boot2docker/bootlocal.sh
#!/bin/sh
sudo umount /Users
sudo mkdir -p /Projects/hadoop
sudo /usr/local/etc/init.d/nfs-client start
sudo mount -t nfs -o noacl,async 192.168.99.1:/Projects/hadoop /Projects/hadoop

I.e., /Projects/hadoop in MacOS is mapped to /Projects/hadoop in boot2docker.

Test it:

> docker-machine ssh default
docker@default:~$  ls /Projects/hadoop
build/  build.gradle  settings.gradle  src/

Step 3: Map the volumes

Run the Docker container:

> docker run \
  -v /Projects/hadoop:/usr/local/hadoop/test-project  \
  -it sequenceiq/hadoop-docker:2.1 /etc/bootstrap.sh -bash

List the project directory in the Docker container:

bash-4.1# cd /usr/local/hadoop
bash-4.1# ls
LICENSE.txt  NOTICE.txt  README.txt ...  test-project
bash-4.1# ls test-project
build  build.gradle  settings.gradle  src

It works.

Tags docker macos

Share volumes in a Docker Container on MacOS

This post shows how to share any MacOS volume in a Docker container. We use docker-machine together with VirtualBox on MacOS. See an older post of mine how this can be done.

Share user home directory

Let’s map a directory in the user home /Users/peter/Hadoop to /usr/local/hadoop/test-home in the Docker container. This should be easy, as the user directory is mounted by boot2docker.

Step 0: Setup

First initialize the directory and create a test file test.txt in it that can be used to test if the mapping was successful:

> cd /Users/peter
> mkdir Hadoop
> cd Hadoop
> touch test.txt
> ls
test.txt

Step 1: Map the volumes

Run Docker and map this directory to /usr/local/hadoop/test-home in the Docker container. For this we use the -v option as follows:

> docker run \
  -v /Users/peter/Hadoop:/usr/local/hadoop/test-home \ 
  -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash

Check Docker mounts:

> docker inspect <container-id>
...
    "Mounts": [
        {
            "Source": "/Users/peter/Hadoop",
            "Destination": "/usr/local/hadoop/test-home",
            "Mode": "",
            "RW": true,
            "Propagation": "rprivate"
        }
        ],
...       

Check the home directory in boot2docker:

> docker-machine ssh default
...
Boot2Docker version 1.10.2, build master : 611be10 - Mon Feb 22 22:47:06 UTC 2016
Docker version 1.10.2, build c3959b1
docker@default:~$ ls /Users/peter/Hadoop/
test.txt

Check the directory in the Docker container:

bash-4.1# cd /usr/local/hadoop
bash-4.1# ls
LICENSE.txt  NOTICE.txt  README.txt ... test-home
bash-4.1# ls test-home/
test.txt

/usr/local/hadoop/test-home was generated in the Docker container and we find the test.txt file.

This was easy, the -v did the job.

Share any directory

Mapping a directory in the user home was easy, as this directory is mounted automatically in boot2docker. However, this is not the case for directories outside of the user home directory as we will see below.

Step 0: Setup

First, let’s see what’s in the MacOS /Projects/hadoop directory that we want to map:

> ls /Projects/hadoop
build/  build.gradle  settings.gradle  src/

In our case, we are especially interested in the build directory with the generated Hadoop Map/Reduce libraries we want to use in the Hadoop Docker container.

Step 1: Map the volumes

Start Docker and map the project directory /Projects/hadoop to /usr/local/hadoop/test-project as above:

docker run \
  -v /Users/peter/Hadoop:/usr/local/hadoop/test-home \ 
  -v /Projects/hadoop:/usr/local/hadoop/test-project  \
  -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash

Check Docker mounts:

> docker inspect <container-id>
...
    "Mounts": [
        {
            "Source": "/Users/peter/Hadoop",
            "Destination": "/usr/local/hadoop/test-home",
            "Mode": "",
            "RW": true,
            "Propagation": "rprivate"
        },
        {
            "Source": "/Projects/hadoop-test",
            "Destination": "/usr/local/hadoop/test-project",
            "Mode": "",
            "RW": true,
            "Propagation": "rprivate"
        }
        ],
...     

This looks as expected.

List the mapped volume in the Docker container:

bash-4.1# cd /usr/local/hadoop/test-home
bash-4.1# ls
LICENSE.txt  NOTICE.txt  README.txt ... 	test-home  test-project
bash-4.1# ls test-project
bash-4.1# 

The directory was generated in the Docker container but it is empty!

The reason for this is as mentioned above, that only the user home directory is mounted in boot2docker per default and not any other.

Two additional steps are needed:

  • Map the volume in VirtualBox
  • Mount the volume in boot2docker

Step 2: Map the volume in VirtualBox

Stop VirtualBox with docker-machine stop default and share the /Projects/hadoop folder with:

VBoxManage sharedfolder add default --name "hadoop" --hostpath /Projects/hadoop 

Start the VirtualBox again with docker-machine start default.

Alternatively, you may use the VirtualBox UI (Settings -> Shared Folders).

Step 3: Mount the volume in boot2docker

Use mount to mount the MacOS directory in boot2docker.

Usage:

docker@default:~$ mount --help
BusyBox v1.23.1 (2015-02-22 15:52:11 UTC) multi-call binary.
Usage: mount [OPTIONS] [-o OPTS] DEVICE NODE
...

where:

  • DEVICE is the logical name of the shared folder defined in VirtualBox
  • NODE is the hostpath of the directory to be shared in boot2docker (not MacOS!)

Use it:

> docker-machine ssh default
docker@default:~$ sudo mkdir -p /hadoop
docker@default:~$ sudo mount -t vboxsf -o defaults,uid=`id -u docker`,gid=`id -g docker` \
  hadoop /hadoop
docker@default:~$ ls /hadoop
build/  build.gradle  settings.gradle  src/  

Note, that DEVICE must correspond to the logical name of the shared folder defined in VirtualBox above. If you missed the name, then following error message is returned:

docker@default:~$ sudo mount -t vboxsf -o defaults,uid=`id -u docker`,gid=`id -g docker` \
  wrong-logical-name /Projects/hadoop
mount.vboxsf: mounting failed with the error: Protocol error

NODE must correspond an existing file path in boot2docker (not MacOS!). If you missed this name, then following error message is returned:

docker@default:~$ sudo mount -t vboxsf -o defaults,uid=`id -u docker`,gid=`id -g docker` \
  projects-git /wrong-name
mount.vboxsf: mounting failed with the error: No such file or directory
mount: mounting projects-git on /wrong-name failed: No such file or directory

Run the Docker container again as above:

> docker run \
  -v /hadoop:/usr/local/hadoop/test-project  \
  -it sequenceiq/hadoop-docker:2.1 /etc/bootstrap.sh -bash

Note, that /hadoop corresponds to NODE set above (not the original path in MacOS!).

List the project directory in the Docker container:

bash-4.1# cd /usr/local/hadoop
bash-4.1# ls
LICENSE.txt  NOTICE.txt  README.txt ...  test-project
bash-4.1# ls test-project
build  build.gradle  settings.gradle  src

Yes, it works. However, if you restart boot2docker, the mounted volumes disappear.

Step 4: Permanently mount MacOS volumes in boot2docker

In boot2docker, edit/create (as root) /mnt/sda1/var/lib/boot2docker/bootlocal.sh to add an automount:

mkdir -p /hadoop                         
mount -t vboxsf -o defaults,uid=`id -u docker`,gid=`id -g docker` hadoop /hadoop  

More information

See this StackOverflow post.

Tags docker macos

Running Oracle XE in a Docker Container on MacOS

This post shows how to build and start Oracle XE on MacOS using the Docker image provided by https://github.com/wnameless/docker-oracle-xe-11g.

Step 1: Start Docker

Start formerly installed default Docker instance using docker-machine:

> docker-machine ls
NAME      ACTIVE   DRIVER       STATE   URL   SWARM   DOCKER    ERRORS
default   -        virtualbox   Saved                 Unknown   
> docker-machine start default

Check if the Docker instance is running:

> docker-machine ls    
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
default   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.10.2   

Set enviromnental parameters to be able to commumicate with the Docker instance:

eval $(docker-machine env)

Step 2: Pull or build Docker image

Build the Docker image on your own:

> git clone https://github.com/wnameless/docker-oracle-xe-11g
> cd docker-oracle-xe-11g
> docker build  -t wnameless/oracle-xe-11g .

Check the image:

docker images     
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
wnameless/oracle-xe-11g    latest              45bd9967896a        About an hour ago   2.389 GB

Or just pull the image from the repository (lazy way):

> docker pull wnameless/oracle-xe-11g

Step 3: Run

Run the container with 22 and 1521 ports opened:

>  docker run -d -p 49160:22 -p 49161:1521 wnameless/oracle-xe-11g

Step 4: Test

Connect to database with DB client, e.g. SQLDeveloper, using default settings:

hostname: ~~localhost~~ <IP>
port: 49161
sid: xe
username: system
password: oracle

Using docker-machine on MacOS, localhost does not work. Use the <IP> adress given by following command instead:

> docker-machine ip

Connect with SSH to the container (password: admin), using the same <IP> adress as above:

> ssh root@~~localhost~~<IP> -p 49160    

Tags docker macos database

Running Hadoop in a Docker Container on MacOS

Instead of installing Hadoop directly on my MacBook, I decided to use a Docker container.

Fortunatelly, there is already one Docker image ready to install that can be found at https://github.com/sequenceiq/hadoop-docker.

Step 1: Download and install Docker infrastucture

Download Docker installer:

Double-click the DMG file and follow the instructions.

This will install:

  • VirtualBox
  • docker-machine

Step 2: Create boot2docker

Use docker-machine to create a default boot2docker environment for VirtualBox driver. boot2docker is a Linux distribution based on a stripped down Tiny Core Linux:

> docker-machine create --driver virtualbox default

Check if the default boot2docker environment is successfully created:

> docker-machine ls 
NAME      ACTIVE   DRIVER       STATE     URL   SWARM   DOCKER    ERRORS
default   *        virtualbox   Stopped                 Unknown   

Step 3: Start default boot2docker

> docker-machine start default 

This starts:

  • VirtualBox
  • boot2ocker Linux

Check status:

> docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
default   -        virtualbox   Running   tcp://192.168.99.100:2376           v1.10.2   

Step 4: Init environmental parameters

Init the Docker specific environmental parameters in the shell where the Docker image will be built:

> eval "$(docker-machine env default)"

Check the environmental parameters:

> env | grep DOCKER
DOCKER_HOST=tcp://192.168.99.100:2376
DOCKER_MACHINE_NAME=default
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/Users/peter/.docker/machine/machines/default

Step 5: Clone the GIT project

> git clone https://github.com/sequenceiq/hadoop-docker.git

The hadoop-docker has been created. Step into it for the further processing:

> cd hadoop-docker

Step 6: Build the Docker image

Build the the Hadoop Docker image:

> docker build  -t sequenceiq/hadoop-docker:2.7.1 .
Sending build context to Docker daemon 229.9 kB
Step 1 : FROM sequenceiq/pam:centos-6.5
centos-6.5: Pulling from sequenceiq/pam
b253335dcf03: Pull complete 
a3ed95caeb02: Pull complete 
69623ef05416: Pull complete 
8d2023764774: Downloading [========================================>          ] 71.89 MB/88.09 MB
...
Successfully built 9e7e80f84015

Check if the image has been built and is registrated:

> docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
sequenceiq/hadoop-docker   2.7.1               9e7e80f84015        9 minutes ago       1.769 GB

Alternatively, you may just pull the image instead of building it yourself (lazy way):

> docker pull sequenceiq/hadoop-docker:2.7.1

Step 7: Run the Docker container

Run the Docker container:

> docker run -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash
/
Starting sshd:                                             [  OK  ]
16/02/28 05:13:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [4c2d09c596b0]
4c2d09c596b0: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-4c2d09c596b0.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-4c2d09c596b0.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-4c2d09c596b0.out
16/02/28 05:13:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn--resourcemanager-4c2d09c596b0.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-4c2d09c596b0.out
bash-4.1# 

The bash shell is started and is ready to receive your commands.

Check the Java version:

bash-4.1# java -version
java version "1.7.0_71"

Step 8: Check status of Docker container

> docker ps
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                    NAMES
4c2d09c596b0        sequenceiq/hadoop-docker:2.7.1   "/etc/bootstrap.sh -b"   4 hours ago         Up 4 hours          2122/tcp, 8020/tcp, 8030-8033/tcp, 8040/tcp, 8042/tcp, 8088/tcp, 9000/tcp, 19888/tcp, 49707/tcp, 50010/tcp, 50020/tcp, 50070/tcp, 50075/tcp, 50090/tcp   compassionate_khorana

Step 9: Test Hadoop: Grep

Go into the Hadoop directory:

bash-4.1# cd $HADOOP_PREFIX
bash-4.1# pwd
/usr/local/hadoop

Check the Hadoop version:

bash-4.1# ./bin/hadoop version
Hadoop 2.7.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
Compiled by jenkins on 2015-06-29T06:04Z
Compiled with protoc 2.5.0
From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
This command was run using /usr/local/hadoop-2.7.1/share/hadoop/common/hadoop-common-2.7.1.jar

Run the mapreduce test:

bash-4.1# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'
16/02/27 05:54:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/27 05:54:52 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/02/27 05:54:53 INFO input.FileInputFormat: Total input paths to process : 31
...

Check the test output:

bash-4.1# bin/hdfs dfs -cat output/*
16/02/27 05:57:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
6	dfs.audit.logger
4	dfs.class
3	dfs.server.namenode.
2	dfs.period
2	dfs.audit.log.maxfilesize
2	dfs.audit.log.maxbackupindex
1	dfsmetrics.log
1	dfsadmin
1	dfs.servers
1	dfs.replication
1	dfs.file

List output directory:

bash-4.1# bin/hdfs dfs -ls         
16/02/28 06:05:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
drwxr-xr-x   - root supergroup          0 2016-02-28 05:03 input
drwxr-xr-x   - root supergroup          0 2016-02-28 06:04 output

Step 10: Find more info

Tags docker macos hadoop

Gradle: From zero to war

The well known gradle init task is an easy way to create a Java project:

> gradle init --type java-library

However, there is no support for initializing a web application. Alternatively, you may use the cool gradle-templates of townsfolk: https://github.com/townsfolk/gradle-templates.

How this may be done, is described below.

Step 0: Create master gradle script

Create a master build.gradle script. This may be done anywhere. However, I put it in the project parent directory where all my projects are found:

buildscript {
    repositories {
        maven {
            url 'http://dl.bintray.com/cjstehno/public'
        }
    }
    dependencies {
        classpath 'gradle-templates:gradle-templates:1.4.1'
    }
}
apply plugin:'templates'

List the tasks:

> gradle tasks
[...]
Template tasks
--------------
createGradlePlugin - Creates a new Gradle Plugin project in a new directory named after your project.
createGroovyClass - Creates a new Groovy class in the current project.
createGroovyProject - Creates a new Gradle Groovy project in a new directory named after your project.
createJavaClass - Creates a new Java class in the current project.
createJavaProject - Creates a new Gradle Java project in a new directory named after your project.
createScalaClass - Creates a new Scala class in the current project.
createScalaObject - Creates a new Scala object in the current project.
createScalaProject - Creates a new Gradle Scala project in a new directory named after your project.
createWebappProject - Creates a new Gradle Webapp project in a new directory named after your project.
exportAllTemplates - Exports all the default template files into the current directory.
exportGroovyTemplates - Exports the default groovy template files into the current directory.
exportJavaTemplates - Exports the default java template files into the current directory.
exportPluginTemplates - Exports the default plugin template files into the current directory.
exportScalaTemplates - Exports the default scala template files into the current directory.
exportWebappTemplates - Exports the default webapp template files into the current directory.
initGradlePlugin - Initializes a new Gradle Plugin project in the current directory.
initGroovyProject - Initializes a new Gradle Groovy project in the current directory.
initJavaProject - Initializes a new Gradle Java project in the current directory.
initScalaProject - Initializes a new Gradle Scala project in the current directory.
initWebappProject - Initializes a new Gradle Webapp project in the current directory.
[...]

As you can see, there are many different templates you may use. For our purpose, the createWebappProject task seems to be interesting.

Step 1: Create the web project

Use the createWebappProject task for creating a new fresh web project. Invoke the task and answer some questions:

> gradle createWebappProject
:createWebappProject

templates> Project Name:  (WAITING FOR INPUT BELOW)
> Building 0% > :createWebappProjecttestweb

templates> Project Parent Directory: [/Development/java/projects/projects-test]  (WAITING FOR INPUT BELOW)
> Building 0% > :createWebappProject

templates> Use Jetty Plugin? (Y|n) [n]  (WAITING FOR INPUT BELOW)
> Building 0% > :createWebappProjectY

templates> Group: [testweb]  (WAITING FOR INPUT BELOW)
> Building 0% > :createWebappProject

templates> Version: [0.1]  (WAITING FOR INPUT BELOW)
> Building 0% > :createWebappProject

BUILD SUCCESSFUL

Total time: 25.877 secs

Check the generated files and directories:

> cd testweb/
> find .
.
./build.gradle
./gradle.properties
./LICENSE.txt
./src
./src/main
./src/main/java
./src/main/resources
./src/main/webapp
./src/main/webapp/WEB-INF
./src/main/webapp/WEB-INF/web.xml
./src/test
./src/test/java
./src/test/resources

List the tasks:

> gradle tasks
Build tasks
-----------
[...]
war - Generates a war archive with all the compiled classes, the web-app content and the libraries.
[...]

Web application tasks
---------------------
jettyRun - Uses your files as and where they are and deploys them to Jetty.
jettyRunWar - Assembles the webapp into a war and deploys it to Jetty.
jettyStop - Stops Jetty.
[...]

Step 2: Create your first page and run it

Create the landing page:

> echo "Hello" > src/main/webapp/index.html 

And finally, run the web application in Jetty:

 
> gradle jettyRun
Starting a new Gradle Daemon for this build (subsequent builds will be faster).
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
> Building 75% > :jettyRun > Running at http://localhost:8080/testweb
 

Open your browser http://localhost:8080/testweb/ and you should see the welcome message.

Step 3: Create war

> gradle clean war

Check if the web archive is generated:

> find build/libs/
build/libs/
build/libs//testweb-0.1.war

Original post: http://peter-on-java.blogspot.com/2016/01/gradle-from-zero-to-war.html

Tags buildtool