21 Mar 2016
In the last post I showed how to
share a volume in a Docker container on MacOS. There was a way, but it was not
really easy.
Alternatively, there is another way that’s much easier: use
docker-machine-fs .
Step 1: Install docker-machine-nfs
Standalone:
curl -s https://raw.githubusercontent.com/adlogix/docker-machine-nfs/master/docker-machine-nfs.sh |
sudo tee /usr/local/bin/docker-machine-nfs > /dev/null && \
sudo chmod +x /usr/local/bin/docker-machine-nfs
Or using Homewbrew:
brew install docker-machine-nfs
Step 2: Map the volume in VirtualBox and permanently mount the volume in boot2docker
> docker-machine-nfs default --shared-folder= /Projects/hadoop
[ INFO] Configuration:
- Machine Name: default
- Shared Folder: /Projects/hadoop
- Mount Options: noacl,async
- Force: false
[ INFO] machine presence ... OK
[ INFO] machine running ... OK
[ INFO] Lookup mandatory properties ... OK
- Machine IP: 192.168.99.100
- Network ID: vboxnet2
- NFSHost IP: 192.168.99.1
[ INFO] Configure NFS ...
!!! Sudo will be necessary for editing /etc/exports !!!
The nfsd service does not appear to be running.
Starting the nfsd service
OK
[ INFO] Configure Docker Machine ... OK
[ INFO] Restart Docker Machine ... OK
[ INFO] Verify NFS mount ...
OK
--------------------------------------------
The docker-machine 'default'
is now mounted with NFS!
ENJOY high speed mounts :D
--------------------------------------------
As seen in the log, sudo is needed to create/update /etc/exports
.
Content:
/Projects/hadoop 192.168.99.100 -alldirs -mapall= 501:20
Note, that in boot2docker the file /mnt/sda1/var/lib/boot2docker/bootlocal.sh
is overridden:
docker@default:~$ cat /mnt/sda1/var/lib/boot2docker/bootlocal.sh
#!/bin/sh
sudo umount /Users
sudo mkdir -p /Projects/hadoop
sudo /usr/local/etc/init.d/nfs-client start
sudo mount -t nfs -o noacl,async 192.168.99.1:/Projects/hadoop /Projects/hadoop
I.e., /Projects/hadoop
in MacOS is mapped to /Projects/hadoop
in boot2docker.
Test it:
> docker-machine ssh default
docker@default:~$ ls /Projects/hadoop
build/ build.gradle settings.gradle src/
Step 3: Map the volumes
Run the Docker container:
> docker run \
-v /Projects/hadoop:/usr/local/hadoop/test-project \
-it sequenceiq/hadoop-docker:2.1 /etc/bootstrap.sh -bash
List the project directory in the Docker container:
bash-4.1# cd /usr/local/hadoop
bash-4.1# ls
LICENSE.txt NOTICE.txt README.txt ... test -project
bash-4.1# ls test -project
build build.gradle settings.gradle src
It works.
Tags
docker
macos
19 Mar 2016
This post shows how to share any MacOS volume in a Docker container. We use
docker-machine together with VirtualBox on MacOS. See an older
post of mine how this can be done.
Share user home directory
Let’s map a directory in the user home /Users/peter/Hadoop
to /usr/local/hadoop/test-home
in the Docker container. This should be easy, as the user directory is mounted
by boot2docker.
Step 0: Setup
First initialize the directory and create a test file test.txt
in it that can be
used to test if the mapping was successful:
> cd /Users/peter
> mkdir Hadoop
> cd Hadoop
> touch test.txt
> ls
test.txt
Step 1: Map the volumes
Run Docker and map this directory to /usr/local/hadoop/test-home
in the
Docker container. For this we use the -v
option as follows:
> docker run \
-v /Users/peter/Hadoop:/usr/local/hadoop/test-home \
-it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash
Check Docker mounts:
> docker inspect <container-id>
...
"Mounts" : [
{
"Source" : "/Users/peter/Hadoop" ,
"Destination" : "/usr/local/hadoop/test-home" ,
"Mode" : "" ,
"RW" : true ,
"Propagation" : "rprivate"
}
] ,
...
Check the home directory in boot2docker:
> docker-machine ssh default
...
Boot2Docker version 1.10.2, build master : 611be10 - Mon Feb 22 22:47:06 UTC 2016
Docker version 1.10.2, build c3959b1
docker@default:~$ ls /Users/peter/Hadoop/
test.txt
Check the directory in the Docker container:
bash-4.1# cd /usr/local/hadoop
bash-4.1# ls
LICENSE.txt NOTICE.txt README.txt ... test -home
bash-4.1# ls test -home/
test.txt
/usr/local/hadoop/test-home
was generated in the Docker container and we
find the test.txt
file.
This was easy, the -v
did the job.
Share any directory
Mapping a directory in the user home was easy, as this directory is mounted
automatically in boot2docker. However, this is not the case for directories
outside of the user home directory as we will see below.
Step 0: Setup
First, let’s see what’s in the MacOS /Projects/hadoop
directory that we want to map:
> ls /Projects/hadoop
build/ build.gradle settings.gradle src/
In our case, we are especially interested in the build
directory with the
generated Hadoop Map/Reduce libraries we want to use in the Hadoop Docker
container.
Step 1: Map the volumes
Start Docker and map the project directory /Projects/hadoop
to
/usr/local/hadoop/test-project
as above:
docker run \
-v /Users/peter/Hadoop:/usr/local/hadoop/test-home \
-v /Projects/hadoop:/usr/local/hadoop/test-project \
-it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash
Check Docker mounts:
> docker inspect <container-id>
...
"Mounts" : [
{
"Source" : "/Users/peter/Hadoop" ,
"Destination" : "/usr/local/hadoop/test-home" ,
"Mode" : "" ,
"RW" : true ,
"Propagation" : "rprivate"
} ,
{
"Source" : "/Projects/hadoop-test" ,
"Destination" : "/usr/local/hadoop/test-project" ,
"Mode" : "" ,
"RW" : true ,
"Propagation" : "rprivate"
}
] ,
...
This looks as expected.
List the mapped volume in the Docker container:
bash-4.1# cd /usr/local/hadoop/test-home
bash-4.1# ls
LICENSE.txt NOTICE.txt README.txt ... test -home test -project
bash-4.1# ls test -project
bash-4.1#
The directory was generated in the Docker container but it is empty!
The reason for this is as mentioned above, that only the user home directory
is mounted in boot2docker per default and not any other.
Two additional steps are needed:
Map the volume in VirtualBox
Mount the volume in boot2docker
Step 2: Map the volume in VirtualBox
Stop VirtualBox with docker-machine stop default
and share the
/Projects/hadoop
folder with:
VBoxManage sharedfolder add default --name "hadoop" --hostpath /Projects/hadoop
Start the VirtualBox again with docker-machine start default
.
Alternatively, you may use the VirtualBox UI (Settings -> Shared Folders).
Step 3: Mount the volume in boot2docker
Use mount
to mount the MacOS directory in boot2docker.
Usage:
docker@default:~$ mount --help
BusyBox v1.23.1 ( 2015-02-22 15:52:11 UTC) multi-call binary.
Usage: mount [ OPTIONS] [ -o OPTS] DEVICE NODE
...
where:
DEVICE
is the logical name of the shared folder defined in VirtualBox
NODE
is the hostpath of the directory to be shared in boot2docker (not MacOS!)
Use it:
> docker-machine ssh default
docker@default:~$ sudo mkdir -p /hadoop
docker@default:~$ sudo mount -t vboxsf -o defaults,uid= ` id -u docker` ,gid= ` id -g docker` \
hadoop /hadoop
docker@default:~$ ls /hadoop
build/ build.gradle settings.gradle src/
Note, that DEVICE
must correspond to the logical name of the shared folder defined
in VirtualBox above. If you missed the name, then following error message is returned:
docker@default:~$ sudo mount -t vboxsf -o defaults,uid= ` id -u docker` ,gid= ` id -g docker` \
wrong-logical-name /Projects/hadoop
mount.vboxsf: mounting failed with the error: Protocol error
NODE
must correspond an existing file path in boot2docker (not MacOS!). If you missed this name, then
following error message is returned:
docker@default:~$ sudo mount -t vboxsf -o defaults,uid= ` id -u docker` ,gid= ` id -g docker` \
projects-git /wrong-name
mount.vboxsf: mounting failed with the error: No such file or directory
mount: mounting projects-git on /wrong-name failed: No such file or directory
Run the Docker container again as above:
> docker run \
-v /hadoop:/usr/local/hadoop/test-project \
-it sequenceiq/hadoop-docker:2.1 /etc/bootstrap.sh -bash
Note, that /hadoop
corresponds to NODE
set above (not the original path in MacOS!).
List the project directory in the Docker container:
bash-4.1# cd /usr/local/hadoop
bash-4.1# ls
LICENSE.txt NOTICE.txt README.txt ... test -project
bash-4.1# ls test -project
build build.gradle settings.gradle src
Yes, it works. However, if you restart boot2docker, the mounted volumes disappear.
Step 4: Permanently mount MacOS volumes in boot2docker
In boot2docker, edit/create (as root) /mnt/sda1/var/lib/boot2docker/bootlocal.sh
to
add an automount:
mkdir -p /hadoop
mount -t vboxsf -o defaults,uid= ` id -u docker` ,gid= ` id -g docker` hadoop /hadoop
See this StackOverflow
post.
Tags
docker
macos
04 Mar 2016
This post shows how to build and start Oracle XE on MacOS using the Docker image provided by
https://github.com/wnameless/docker-oracle-xe-11g .
Step 1: Start Docker
Start formerly installed default
Docker instance using docker-machine
:
> docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default - virtualbox Saved Unknown
> docker-machine start default
Check if the Docker instance is running:
> docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default * virtualbox Running tcp://192.168.99.100:2376 v1.10.2
Set enviromnental parameters to be able to commumicate with the Docker instance:
eval $( docker-machine env)
Step 2: Pull or build Docker image
Build the Docker image on your own:
> git clone https://github.com/wnameless/docker-oracle-xe-11g
> cd docker-oracle-xe-11g
> docker build -t wnameless/oracle-xe-11g .
Check the image:
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
wnameless/oracle-xe-11g latest 45bd9967896a About an hour ago 2.389 GB
Or just pull the image from the repository (lazy way):
> docker pull wnameless/oracle-xe-11g
Step 3: Run
Run the container with 22 and 1521 ports opened:
> docker run -d -p 49160:22 -p 49161:1521 wnameless/oracle-xe-11g
Step 4: Test
Connect to database with DB client, e.g. SQLDeveloper, using default settings:
hostname: ~~localhost~~ <IP>
port: 49161
sid: xe
username: system
password: oracle
Using docker-machine
on MacOS, localhost
does not work. Use the <IP> adress given by
following command instead:
> docker-machine ip
Connect with SSH to the container (password: admin), using the same <IP> adress as above:
> ssh root@~~localhost~~<IP> -p 49160
Tags
docker
macos
database
27 Feb 2016
Instead of installing Hadoop directly on my MacBook, I decided to use a Docker
container.
Fortunatelly, there is already one Docker image ready to install that can be found
at https://github.com/sequenceiq/hadoop-docker .
Step 1: Download and install Docker infrastucture
Download Docker installer:
Double-click the DMG file and follow the instructions.
This will install:
VirtualBox
docker-machine
Step 2: Create boot2docker
Use docker-machine to create a default boot2docker
environment for VirtualBox driver. boot2docker is a Linux distribution
based on a stripped down Tiny Core Linux :
> docker-machine create --driver virtualbox default
Check if the default boot2docker environment is successfully created:
> docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default * virtualbox Stopped Unknown
Step 3: Start default boot2docker
> docker-machine start default
This starts:
VirtualBox
boot2ocker Linux
Check status:
> docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default - virtualbox Running tcp://192.168.99.100:2376 v1.10.2
Step 4: Init environmental parameters
Init the Docker specific environmental parameters in the shell where the Docker image
will be built:
> eval " $( docker-machine env default) "
Check the environmental parameters:
> env | grep DOCKER
DOCKER_HOST = tcp://192.168.99.100:2376
DOCKER_MACHINE_NAME = default
DOCKER_TLS_VERIFY = 1
DOCKER_CERT_PATH = /Users/peter/.docker/machine/machines/default
Step 5: Clone the GIT project
> git clone https://github.com/sequenceiq/hadoop-docker.git
The hadoop-docker
has been created. Step into it for the further processing:
> cd hadoop-docker
Step 6: Build the Docker image
Build the the Hadoop Docker image:
> docker build -t sequenceiq/hadoop-docker:2.7.1 .
Sending build context to Docker daemon 229.9 kB
Step 1 : FROM sequenceiq/pam:centos-6.5
centos-6.5: Pulling from sequenceiq/pam
b253335dcf03: Pull complete
a3ed95caeb02: Pull complete
69623ef05416: Pull complete
8d2023764774: Downloading [======================================== > ] 71.89 MB/88.09 MB
...
Successfully built 9e7e80f84015
Check if the image has been built and is registrated:
> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
sequenceiq/hadoop-docker 2.7.1 9e7e80f84015 9 minutes ago 1.769 GB
Alternatively, you may just pull the image instead of building it yourself (lazy way):
> docker pull sequenceiq/hadoop-docker:2.7.1
Step 7: Run the Docker container
Run the Docker container:
> docker run -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash
/
Starting sshd: [ OK ]
16/02/28 05:13:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin -java classes where applicable
Starting namenodes on [ 4c2d09c596b0]
4c2d09c596b0: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-4c2d09c596b0.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-4c2d09c596b0.out
Starting secondary namenodes [ 0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-4c2d09c596b0.out
16/02/28 05:13:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin -java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn--resourcemanager-4c2d09c596b0.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-4c2d09c596b0.out
bash-4.1#
The bash shell is started and is ready to receive your commands.
Check the Java version:
bash-4.1# java -version
java version "1.7.0_71"
Step 8: Check status of Docker container
> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4c2d09c596b0 sequenceiq/hadoop-docker:2.7.1 "/etc/bootstrap.sh -b" 4 hours ago Up 4 hours 2122/tcp, 8020/tcp, 8030-8033/tcp, 8040/tcp, 8042/tcp, 8088/tcp, 9000/tcp, 19888/tcp, 49707/tcp, 50010/tcp, 50020/tcp, 50070/tcp, 50075/tcp, 50090/tcp compassionate_khorana
Step 9: Test Hadoop: Grep
Go into the Hadoop directory:
bash-4.1# cd $HADOOP_PREFIX
bash-4.1# pwd
/usr/local/hadoop
Check the Hadoop version:
bash-4.1# ./bin/hadoop version
Hadoop 2.7.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
Compiled by jenkins on 2015-06-29T06:04Z
Compiled with protoc 2.5.0
From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
This command was run using /usr/local/hadoop-2.7.1/share/hadoop/common/hadoop-common-2.7.1.jar
Run the mapreduce test:
bash-4.1# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'
16/02/27 05:54:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin -java classes where applicable
16/02/27 05:54:52 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/02/27 05:54:53 INFO input.FileInputFormat: Total input paths to process : 31
...
Check the test output:
bash-4.1# bin/hdfs dfs -cat output/*
16/02/27 05:57:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin -java classes where applicable
6 dfs.audit.logger
4 dfs.class
3 dfs.server.namenode.
2 dfs.period
2 dfs.audit.log.maxfilesize
2 dfs.audit.log.maxbackupindex
1 dfsmetrics.log
1 dfsadmin
1 dfs.servers
1 dfs.replication
1 dfs.file
List output directory:
bash-4.1# bin/hdfs dfs -ls
16/02/28 06:05:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin -java classes where applicable
Found 2 items
drwxr-xr-x - root supergroup 0 2016-02-28 05:03 input
drwxr-xr-x - root supergroup 0 2016-02-28 06:04 output
Step 10: Find more info
Tags
docker
macos
hadoop
15 Jan 2016
The well known gradle init
task is an easy way to create a Java project:
> gradle init --type java-library
However, there is no support for initializing a web application. Alternatively, you may use the cool gradle-templates of townsfolk: https://github.com/townsfolk/gradle-templates .
How this may be done, is described below.
Step 0: Create master gradle script
Create a master build.gradle
script. This may be done anywhere. However, I put it in the project parent directory where all my projects are found:
buildscript {
repositories {
maven {
url 'http://dl.bintray.com/cjstehno/public'
}
}
dependencies {
classpath 'gradle-templates:gradle-templates:1.4.1'
}
}
apply plugin:'templates'
List the tasks:
> gradle tasks
[ ...]
Template tasks
--------------
createGradlePlugin - Creates a new Gradle Plugin project in a new directory named after your project.
createGroovyClass - Creates a new Groovy class in the current project.
createGroovyProject - Creates a new Gradle Groovy project in a new directory named after your project.
createJavaClass - Creates a new Java class in the current project.
createJavaProject - Creates a new Gradle Java project in a new directory named after your project.
createScalaClass - Creates a new Scala class in the current project.
createScalaObject - Creates a new Scala object in the current project.
createScalaProject - Creates a new Gradle Scala project in a new directory named after your project.
createWebappProject - Creates a new Gradle Webapp project in a new directory named after your project.
exportAllTemplates - Exports all the default template files into the current directory.
exportGroovyTemplates - Exports the default groovy template files into the current directory.
exportJavaTemplates - Exports the default java template files into the current directory.
exportPluginTemplates - Exports the default plugin template files into the current directory.
exportScalaTemplates - Exports the default scala template files into the current directory.
exportWebappTemplates - Exports the default webapp template files into the current directory.
initGradlePlugin - Initializes a new Gradle Plugin project in the current directory.
initGroovyProject - Initializes a new Gradle Groovy project in the current directory.
initJavaProject - Initializes a new Gradle Java project in the current directory.
initScalaProject - Initializes a new Gradle Scala project in the current directory.
initWebappProject - Initializes a new Gradle Webapp project in the current directory.
[ ...]
As you can see, there are many different templates you may use. For our purpose, the createWebappProject
task seems to be interesting.
Step 1: Create the web project
Use the createWebappProject
task for creating a new fresh web project. Invoke the task and answer some questions:
> gradle createWebappProject
:createWebappProject
templates> Project Name: ( WAITING FOR INPUT BELOW)
> Building 0% > :createWebappProjecttestweb
templates> Project Parent Directory: [ /Development/java/projects/projects-test] ( WAITING FOR INPUT BELOW)
> Building 0% > :createWebappProject
templates> Use Jetty Plugin? ( Y|n) [ n] ( WAITING FOR INPUT BELOW)
> Building 0% > :createWebappProjectY
templates> Group: [ testweb] ( WAITING FOR INPUT BELOW)
> Building 0% > :createWebappProject
templates> Version: [ 0.1] ( WAITING FOR INPUT BELOW)
> Building 0% > :createWebappProject
BUILD SUCCESSFUL
Total time : 25.877 secs
Check the generated files and directories:
> cd testweb/
> find .
.
./build.gradle
./gradle.properties
./LICENSE.txt
./src
./src/main
./src/main/java
./src/main/resources
./src/main/webapp
./src/main/webapp/WEB-INF
./src/main/webapp/WEB-INF/web.xml
./src/test
./src/test/java
./src/test/resources
List the tasks:
> gradle tasks
Build tasks
-----------
[ ...]
war - Generates a war archive with all the compiled classes, the web-app content and the libraries.
[ ...]
Web application tasks
---------------------
jettyRun - Uses your files as and where they are and deploys them to Jetty.
jettyRunWar - Assembles the webapp into a war and deploys it to Jetty.
jettyStop - Stops Jetty.
[ ...]
Step 2: Create your first page and run it
Create the landing page:
> echo "Hello" > src/main/webapp/index.html
And finally, run the web application in Jetty:
> gradle jettyRun
Starting a new Gradle Daemon for this build ( subsequent builds will be faster) .
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
> Building 75% > :jettyRun > Running at http://localhost:8080/testweb
Open your browser http://localhost:8080/testweb/ and you should see the welcome message.
Step 3: Create war
> gradle clean war
Check if the web archive is generated:
> find build/libs/
build/libs/
build/libs//testweb-0.1.war
Original post: http://peter-on-java.blogspot.com/2016/01/gradle-from-zero-to-war.html
Tags
buildtool