dockerbasedhadoopprovisi鹏oning.pptVIP

* * * * * Why we have started this at all – there are so many options. We repeated the same steps over and over – and scripted. Still, we felt that there is something missing. See bullet points * * * * * Been through many different approaches. Bare metal, cloud VM, so on – ended up using Docker. Tested many provisioning frameworks – Ambari is the one. * * * * * Quick question - How many of you have used Docker before. Docker is a container based virtualization framework. Unlike traditional virtualization Docker is fast, lightweight and easy to use. Docker allows you to create containers holding all the dependencies for an application. Each container is kept isolated from any other, and nothing gets shared. * * * * * I can run 5-6 containers – less overhead than 1 virtualbox. No SOCKS proxy, etc. * * * * * The ‘provisioning’ framework. No need to enter details, there were pretty good sessions about Ambari. Blueprints 1.5.1 tech preview, 1.6 fully supported. Blueprint = stack definition + component layout. REST API – we have created, open sourced Ambari client + shell (come and join the Ambari Meetup today at 3:30) * * * * * * * * * * Now, the issues. Do it again and again – for each cloud provider. Create the image – but how do you know what’s the requirement, building an image each and every time? Network – this is a big issue. EC2 has API, Azure his own. Open Stack has a network as a service component – Neutrom. SDN – Software define network!!! Everything is dynamic – how do you do service discovery? Extra features – fully dynamic Hadoop cluster. * * * * * Will expand on these shortly. Sounds too easy – lets get into details. * * * * * A Docker image is described by a Dockerfile – like a Vagrant file for virtualbox for example. You want trusted build – use Docker.io Faster provisioning – a 100+ node Hadoop cluster in less than 5 minutes? Come and join the Ambari meetup. Licensing –Ganglia or Nagios (BSD and GPL). Hortonworks Hadoop – Apache 2 Bigtop is