Spark with Docker on Synology

If you’d like to run Spark in Docker on your Synology, this might be one way to do it. I’m providing a Dockerfile and a docker-compose file, use them on your Synology.

Dockerfile

FROM ubuntu:16.04
RUN apt-get -y update
RUN apt-get -y install curl
# Install Java
ARG JAVA_BINARY=http://download.oracle.com/otn-pub/java/jdk/8u111-b14/server-jre-8u111-linux-x64.tar.gz
ENV JAVA_HOME /usr/local/jdk1.8.0_111
ENV PATH $PATH:$JAVA_HOME/bin
RUN curl -L -H "Cookie: oraclelicense=accept-securebackup-cookie" -k ${JAVA_BINARY} | tar -xz -C /usr/local/ && ln -s $JAVA_HOME /usr/local/java
# Install Spark 2.x
ARG SPARK_BINARY=http://d3kbcqa49mib13.cloudfront.net/spark-2.0.2-bin-hadoop2.7.tgz
ENV SPARK_HOME /usr/local/spark-2.0.2-bin-hadoop2.7
ENV PATH $PATH:${SPARK_HOME}/bin
RUN curl -s ${SPARK_BINARY} | tar -xz -C /usr/local/
WORKDIR $SPARK_HOME

docker-compose.yml

spark-master:
 image: moszinet/spark-docker
 command: bin/spark-class org.apache.spark.deploy.master.Master -h spark-master
 hostname: spark-master
 environment:
 MASTER: spark://spark-master:7077
 SPARK_CONF_DIR: /conf
 SPARK_PUBLIC_DNS: 127.0.0.1
 expose:
 - 7001
 - 7002
 - 7003
 - 7004
 - 7005
 - 7006
 - 7077
 - 6066
 ports:
 - 4040:4040
 - 6066:6066
 - 7077:7077
 - 8080:8080
 volumes:
 - ./conf/spark-master:/conf
 - ./data:/tmp/data
spark-worker:
 image: moszinet/spark-docker
 command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
 hostname: spark-worker
 environment:
 SPARK_CONF_DIR: /conf
 SPARK_PUBLIC_DNS: 127.0.0.1
 SPARK_WORKER_CORES: 2
 SPARK_WORKER_MEMORY: 2g
 SPARK_WORKER_PORT: 8881
 SPARK_WORKER_WEBUI_PORT: 8081
 links:
 - spark-master
 expose:
 - 7012
 - 7013
 - 7014
 - 7015
 - 7016
 - 8881
 ports:
 - 8081:8081
 volumes:
 - ./conf/spark-worker:/conf
 - ./data:/tmp/data

My home server “room”

IMG_0044

From top to bottom in the rack:

  • Unifi Access Point (white little thingy on the top of the rack, barely visible)
  • Unifi Switch, PoE
  • Dell Server with Hyper-V, 16G RAM
  • Synology RS418+, 6G RAM, 5TB RAID5
  • Unifi CloudKey
  • VoiP phone base station
  • rPI w/ XBee (testing purposes)
  • UPS

On the left:

  • mPower from Ubiquiti
  • mPort from Ubiquiti
  • a camera connected to Synology

Passed exam: AWS Certified Solution Architect – Associate

After quite some learning I managed to pass the AWS Certified Solutions Architect – Associate exam. This is only the first of the solutions architect exams at Amazon, I really hope I will be able to do the second one this year.

Nevertheless, the exam was quite all right; I had to do it in Vienna because in Budapest there is only one single place where you can do an AWS exam (Masterfield training center), but they only had one single date I could book for January, which date wasn’t okay for me. I will give them another chance with the AWS Certified Solutions Architect – Pofessional exam once I’m ready for it 😉 …

Contributors Wanted: Home Automation Project

[Please Share: Contributors Wanted:, Home Automation Open Source Project]

Hey People of the world,

I’ve started a Home Automation project [you know, sensors in your home, remote switches that can be controlled from your smartphone, etc. – pretty exciting, interesting stuff wink emoticon … ] – and I’m looking for contributors who would be dedicated in spending some time with the project, contribute their own ideas and be part of something cool ;]

I have a solid roadmap in my mind, but nothing is carved in stone, so if you bring your own ideas it could easily turn out that the whole project takes a different path.

That being said, I’m looking for contributions in the following areas:

– Mobile Phone / Tablet development with Xamarin
– Web development to create a proper website
– .NET development for the system’s “brain” (buzzwords: Raspberry PI 2, Windows IoT Core, .NET Core)
– HW specialist to create devices for the solution

Let me know if you are interested, I would be happy to give you more details about the project. The project is already on github, please check it out. Disclaimer: it’s still work in progress in every aspect of it: https://github.com/moszinet/HomeAutomation/wiki

[ Even if you do not wish to contribute it would be extremely helful to share this post so it reaches a wider audience. ]

Thanks a lot,
Moszi

PS. I can be reached at: moszi@moszi.net

Synology DS713+ 2G RAM

So I’ve bought about 2 weeks ago a nice Synology DS713+ NAS server (hahh: first I wrote NSA ;]] ) – and I have to say, that I love it. It’s really fast.

Also – I’ve installed 2G RAM in it – so it is even faster now :) – Synology doesn’t say it’s possible, but when you open the box, you’ll see they’ve installed a simple 1G DDR3 SODIMM RAM which I you simply have to replace with a bigger one.
Installed, works perfectly!

 

Synology auto block IP list export

I’ve just spent some time finding the list of the IP addresses that Synology DSM blocked  – and if anyone is wondering, it can be found in /etc/synoautoblock.db . This is a simple SQLite database – the AutoBlockIP table contains all the IP addresses that are blocked.