4

Getting Started with Hetzner Cloud Service (IaaS)

A week ago, the press has announced a new Cloud Service: Hetzner Cloud is based in Germany but they offer Infrastructure as a Service (IaaS) in the US as well.  You can rent computing resources at “unbeatable prices” as they point out; e.g. 1 vCPU and 2 GB RAM + 20 GB SSD cost less than 3 € per month (or around US$ 3), currently. Internet transfer of 20 TB is included. In this blog post, I am capturing the first experience I have gained with the service.

Main Features

Super-quick Deployment

A server instance can be started in less than 10 seconds. This might not be too important for most of us, since there is a high probability to run the service 24hrs/7d a week, considering the low price of the service. Anyhow, the deployment time of 1o sec is convenient and it is very, very competitive, I think.

Note, that a server costs the same money, whether or not it is switched on. In the Q&A section, Hetzner argues that they need to reserve resources since they want to keep their promise that your server is available in less than 10 sec after you hit the “power” button. My guess is, that the virtual machines are always up and running. Or they are using container technologies like LXD, which allow for quick boot-ups paired with the convenience features you are used to from the virtual machine’s world (e.g. suspend/resume).

Unbeatable prices

As Hetzner’s web page points out, Hetzner’s cloud services are targeting a low price segment. A quick comparison shows a price difference between AWS and Hetzner’s Cloud of factor 7 on-demand hours for small servers

  • Hetzner cloud: 2.49 US$ + VAT for 1 CPU 2 GB RAM server with a 20 GB SSD disk and 20 TB transfer
  • AWS: 16.82 US$ + VAT for 1 CPU, 2 GB in US East with disk and Internet transfer to be paid separately.

As heise.de is stating, you need to pay for any extra services, even if you are used to getting those services for free in case of other cloud providers.

The largest available server in Hetzner’s cloud service has 8 vCPU and 32 GB RAM. On the other side, AWS offers instances with up to 128 CPUs and 3904 GB RAM (x1e.32large). This is a factor of 16 with respect to CPU and a factor > 100 (!) with respect to RAM. If you want to go with Hetzner, but you need larger servers with up to 256 GB RAM, you might consider to make use of their dedicated root server service. You can get such a such a root server for 139 € per month in Germany and ~117 €/month in the US, currently. This is more than competitive.

Browser Access to the Command Line

This is a feature not many cloud providers offer to their clients: you can access the command line console of your servers via a web browser.

However, there is a big BUT: the access requires you to set a password for the user, which can be used in the SSH access as well. I would not recommend doing so, because of security reasons. For me, the feature would be usable only, if they’d implement a solution that does not rely on password authentication. Here, a single sign-on solution would be perfect.

Attachable ISO Images

Quite some time ago, I have created a blog post on my search for a provider that allows installing a virtual machine instance from CD/ISO image. I was disappointed to see that AWS as the number one IaaS provider in the market does not offer such a service. I had found another provider, but at the end, I was not too happy with the support of this provider. Now, we can see, that Hetzner cloud offers this functionality as well.  You even can send your own ISO to Hetzner’s team, and they will make it available privately to your account, as they point out.

Simple API

Hetzner offers a simple Cloud API to control all of their functionality:

First, I got it wrong and I thought that they charge 50 € if you want to write integration code using their API. However, as Katie Snow has pointed out in her comment, it is the other way round: “If you are developing integrations based on our API and your product is Open Source you may be eligible for a free one time €50 (excl. VAT) credit on your account.” I.e., Hetzner wants to promote the usage of their API, which makes perfect sense for a newcomer in the IaaS space.

Good Support

I have experienced a good responsiveness to support questions, even if you are posing technical questions and no error reports

Internet Reachability of all Servers

All servers are reachable via the Internet. This might be convenient for some among us, but it also might be considered as being too insecure for others. The server is open to brute force intrusion and denial of service attacks from the Internet. The recommendation of Hetzner’s support hotline: Install an onboard firewall (e.g. iptables). In addition, I recommend to never create a user with password.

Still, a provider provisioned firewall in front of the server would be a better solution, in my opinion.

Missing Services: Load Balancers, Firewalls, …

I guess, it is not fair to compare the feature richness of Hetzner’s cloud with those of the big IaaS providers AWS and Azure. Anyhow, if you decide to go with Hetzner’s Cloud service, do not expect to be provided with infrastructure and services you are used to be available for AWS and Azure. Configurable firewalls and load balancers you not in the portfolio of Hetzner’s cloud (yet). You also will not find other services like templating, docker orchestration, database service, autoscaling, etc. to name only a few. File-based storage seems to be available as so-called storage-boxes, but the attachment to Hetzner cloud servers and its price model for the data transfer between box and cloud is still to be investigated.

Conclusion

Hetzner’s cloud offers computing resources for “unbeatable prices”, as they point out. On-demand computing resources are offered for prices AWS users can only dream of. However, you cannot expect a new cloud provider like Hetzner to offer all the features you are used to finding in clouds run by AWS, Azure or Google engine. You will get fewer compute options to choose from and much lower maximum compute resources per server (only up to 8 vCPU and 32 GB RAM). And your search for services like configurable firewalls, load balancers, templating a.s.o. will be in vain.

However, if you are looking for an unbeatable price paired with some nice, convenient features like browser console access and user-defined ISOs, together with a responsive support hotline, Hetzner does seem to be a good candidate.

 

0

Cassandra “Hello World” Example

 

 

Today, we will introduce Cassandra, a distributed and resilient, highly scalable noSQL database. For simplicity, we will run a cluster it within Docker containers and test the resiliency functions by killing one of two containers and verifying that all data is retained.

What is Cassandra?

Apache Cassandra is a fast, distributed noSQL database that can be used for big data use cases. A short comparison of Apache Cassandra with Apache Hadoop can found in this Cassandra vs Hadoop blog post:

  • Hadoop is a big data framework for storing and analyzing a vast amount of unstructured, historic data. Why ‘historic’? The reason is that the search capabilities of Hadoop rely on long-running, CPU-intensive MapReduce processes that are running as batch processes.
  • Cassandra is a distributed noSQL database for structured data, and is ideally suited for structured, “hot” data, i.e. Cassandra is capable of processing online workloads of a transactional nature.

I have found following figure that compares Cassandra with SOLR/Lucene and Apache Hadoop:

Source: https://docs.datastax.com/en/datastax_enterprise/4.5/datastax_enterprise/srch/srchIntro.html

Target Configuration for this Blog Post

In this Hello World blog post, we will create two Cassandra server containers and a Cassandra client container. For sake of this test, the Cassandra databases are stored within the containers (in a productive environment, you would most likely store the database outside the container). We will add data to the cluster and make sure the data is replicated to both servers. We can test this by shutting down one server container first, starting a new server container to restore the redundancy, shutting down a second container and make sure, that the data is still available.

Tools used

  • Vagrant 1.8.6
  • Virtualbox 5.0.20
  • Docker 1.12.1
  • Cassandra 3.9

Prerequisites:

  • > 3.3 GB DRAM (Docker host: ~0.3 GB, ~1.5 GB per Cassandra node, < ~0.1 GB for Cassandra client)

Step 1: Install a Docker Host via Vagrant and Connect to the Host via SSH

If you are using an existing docker host, make sure that your host has enough memory and your own Docker ho

We will run Cassandra in Docker containers in order to allow for maximum interoperability. This way, we always can use the latest Logstash version without the need to control the java version used: e.g. Logstash v 1.4.x works with java 7, while version 5.0.x works with java 8 only, currently.

If you are new to Docker, you might want to read this blog post.

Installing Docker on Windows and Mac can be a real challenge, but no worries: we will show an easy way here, that is much quicker than the one described in Docker’s official documentation:

Prerequisites of this step:

  • I recommend to have direct access to the Internet: via Firewall, but without HTTP proxy. However, if you cannot get rid of your HTTP proxy, read this blog post.
  • Administration rights on you computer.

Steps to install a Docker Host VirtualBox VM:

Download and install Virtualbox (if the installation fails with error message “<to be completed> see Appendix A of this blog post: Virtualbox Installation Workaround below)

1. Download and Install Vagrant (requires a reboot)

2. Download Vagrant Box containing an Ubuntu-based Docker Host and create a VirtualBox VM like follows:

basesystem# mkdir ubuntu-trusty64-docker ; cd ubuntu-trusty64-docker
basesystem# vagrant init williamyeh/ubuntu-trusty64-docker
basesystem# vagrant up
basesystem# vagrant ssh

Now you are logged into the Docker host and we are ready for the next step: to create the Ansible Docker image.

Note: I have experienced problems with the vi editor when running vagrant ssh in a Windows terminal. In case of Windows, consider to follow Appendix C of this blog post and to use putty instead.

Step 2 (optional): Download Cassandra Image

This extra download step is optional, since the Cassandra Docker image will be downloaded automatically in step 3, if it is not already found on the system:

(dockerhost)$ docker pull cassandra
Using default tag: latest
latest: Pulling from library/cassandra

386a066cd84a: Already exists
e4bd24d76b78: Pull complete
5ccb1c317672: Pull complete
a7ffd548f738: Pull complete
d6f6138be804: Pull complete
756363f453c9: Pull complete
26258521e648: Pull complete
fb207e348163: Pull complete
3f9a7ac16b1d: Pull complete
49e0632fe1f1: Pull complete
ba775b0b41f4: Pull complete
Digest: sha256:f5b1391b457ead432dc05d34797212f038bd9bd4f0b0260d90ce74e53cbe7ca9
Status: Downloaded newer image for cassandra:latest

The version of the downloaded Cassandra image can be checked with following command:

(dockerhost)$ sudo docker run -it --rm --name cassandra cassandra -v
3.9

We are using version 3.9 currently. If you want to make sure that you use the exact same version as I have used in this blog, you can use the imagename cassandra:3.9 in all docker commands instead of cassandra only.

Step 2: Run Cassandra in interactive Terminal Mode

In this step, we will run Cassandra interactively (with -it switch instead of -d switch) to better see, what is happening. In a productive environment, you will use the detached mode -d instead of the interactive terminal mode -it.

We have found out by analyzing the Cassandra image via the online imagelayer tool, that the default command is to run /docker-entrypoint.sh cassandra -f  and that cassandra uses the ports 7000/tcp 7001/tcp 7199/tcp 9042/tcp 9160/tcp. We keep the entrypoint and map the ports to the outside world:

(dockerhost)$ sudo docker run -it --rm --name cassandra-node1 -p7000:7000 -p7001:7001 -p9042:9042 -p9160:9160 cassandra

INFO  15:30:17 Configuration location: file:/etc/cassandra/cassandra.yaml
INFO  15:30:17 Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=172.17.0.4; broadcast_rpc_address=172.17.0.4; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=; cluster_name=Test Cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@2928854b; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=172.17.0.4; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_coalescing_strategy=TIMEHORIZON; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=172.17.0.4}; server_encryption_options=; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@27ae2fd0; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO  15:30:17 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  15:30:17 Global memtable on-heap threshold is enabled at 251MB
INFO  15:30:17 Global memtable off-heap threshold is enabled at 251MB
WARN  15:30:18 Only 22.856GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO  15:30:18 Hostname: 4ba7699e4fc2
INFO  15:30:18 JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_111
INFO  15:30:18 Heap size: 1004.000MiB/1004.000MiB
INFO  15:30:18 Code Cache Non-heap memory: init = 2555904(2496K) used = 3906816(3815K) committed = 3932160(3840K) max = 251658240(245760K)
INFO  15:30:18 Metaspace Non-heap memory: init = 0(0K) used = 15609080(15243K) committed = 16252928(15872K) max = -1(-1K)
INFO  15:30:18 Compressed Class Space Non-heap memory: init = 0(0K) used = 1909032(1864K) committed = 2097152(2048K) max = 1073741824(1048576K)
INFO  15:30:18 Par Eden Space Heap memory: init = 167772160(163840K) used = 73864848(72133K) committed = 167772160(163840K) max = 167772160(163840K)
INFO  15:30:18 Par Survivor Space Heap memory: init = 20971520(20480K) used = 0(0K) committed = 20971520(20480K) max = 20971520(20480K)
INFO  15:30:18 CMS Old Gen Heap memory: init = 864026624(843776K) used = 0(0K) committed = 864026624(843776K) max = 864026624(843776K)
INFO  15:30:18 Classpath: /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.5.4.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.39.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.3.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.3.jar:/usr/share/cassandra/lib/primitive-1.0.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.0.jar:/usr/share/cassandra/lib/reporter-config3-3.0.0.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-3.9.jar:/usr/share/cassandra/apache-cassandra-thrift-3.9.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar
INFO  15:30:18 JVM Arguments: [-Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms1024M, -Xmx1024M, -Xmn200M, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/usr/share/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/var/log/cassandra, -Dcassandra.storagedir=/var/lib/cassandra, -Dcassandra-foreground=yes]
WARN  15:30:18 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
INFO  15:30:18 jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
WARN  15:30:18 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
WARN  15:30:18 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO  15:30:18 Initializing SIGAR library
WARN  15:30:18 Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true
WARN  15:30:18 Directory /var/lib/cassandra/data doesn't exist
WARN  15:30:18 Directory /var/lib/cassandra/commitlog doesn't exist
WARN  15:30:18 Directory /var/lib/cassandra/saved_caches doesn't exist
WARN  15:30:18 Directory /var/lib/cassandra/hints doesn't exist
INFO  15:30:18 Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
INFO  15:30:19 Initializing system.IndexInfo
INFO  15:30:20 Initializing system.batches
INFO  15:30:20 Initializing system.paxos
INFO  15:30:20 Initializing system.local
INFO  15:30:20 Initializing system.peers
INFO  15:30:20 Initializing system.peer_events
INFO  15:30:20 Initializing system.range_xfers
INFO  15:30:20 Initializing system.compaction_history
INFO  15:30:20 Initializing system.sstable_activity
INFO  15:30:20 Initializing system.size_estimates
INFO  15:30:20 Initializing system.available_ranges
INFO  15:30:20 Initializing system.views_builds_in_progress
INFO  15:30:20 Initializing system.built_views
INFO  15:30:20 Initializing system.hints
INFO  15:30:20 Initializing system.batchlog
INFO  15:30:20 Initializing system.schema_keyspaces
INFO  15:30:20 Initializing system.schema_columnfamilies
INFO  15:30:20 Initializing system.schema_columns
INFO  15:30:20 Initializing system.schema_triggers
INFO  15:30:20 Initializing system.schema_usertypes
INFO  15:30:20 Initializing system.schema_functions
INFO  15:30:20 Initializing system.schema_aggregates
INFO  15:30:20 Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO  15:30:20 Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
INFO  15:30:21 Initializing key cache with capacity of 50 MBs.
INFO  15:30:21 Initializing row cache with capacity of 0 MBs
INFO  15:30:21 Initializing counter cache with capacity of 25 MBs
INFO  15:30:21 Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  15:30:21 Global buffer pool is enabled, when pool is exhausted (max is 251.000MiB) it will allocate on heap
INFO  15:30:21 Populating token metadata from system tables
INFO  15:30:21 Token metadata:
INFO  15:30:21 Initializing system_schema.keyspaces
INFO  15:30:21 Initializing system_schema.tables
INFO  15:30:21 Initializing system_schema.columns
INFO  15:30:21 Initializing system_schema.triggers
INFO  15:30:21 Initializing system_schema.dropped_columns
INFO  15:30:21 Initializing system_schema.views
INFO  15:30:21 Initializing system_schema.types
INFO  15:30:21 Initializing system_schema.functions
INFO  15:30:21 Initializing system_schema.aggregates
INFO  15:30:21 Initializing system_schema.indexes
INFO  15:30:21 Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
INFO  15:30:21 Completed loading (5 ms; 1 keys) KeyCache cache
INFO  15:30:21 No commitlog files found; skipping replay
INFO  15:30:21 Populating token metadata from system tables
INFO  15:30:21 Token metadata:
INFO  15:30:22 Cassandra version: 3.9
INFO  15:30:22 Thrift API version: 20.1.0
INFO  15:30:22 CQL supported versions: 3.4.2 (default: 3.4.2)
INFO  15:30:22 Initializing index summary manager with a memory pool size of 50 MB and a resize interval of 60 minutes
INFO  15:30:22 Starting Messaging Service on /172.17.0.4:7000 (eth0)
WARN  15:30:22 No host ID found, created 1c7f41f6-4513-4949-abc3-0335af298fc8 (Note: This should happen exactly once per node).
INFO  15:30:22 Loading persisted ring state
INFO  15:30:22 Starting up server gossip
INFO  15:30:22 This node will not auto bootstrap because it is configured to be a seed node.
INFO  15:30:22 Generated random tokens. tokens are [295917137465811607, -302115512024598814, -4810310810107556185, -7541303704934353556, -6783448374042524533, -304630524111773314, 1533898300851998947, 3941083117284553885, -6988940081331410223, -4534890749515718142, 4308460093122238220, 7581503118159726763, -6723684273635062672, 1884874259542417756, 7788232024365633386, 5915388149540707275, -6016738271397646318, 1614489580517171638, -3947302573022728739, 1923926062108950529, -9108925830760249950, -9060042955322751814, -2000084340205921073, -6132707306760563996, -6883241210703381902, 8740195267857701913, 8041076389604804564, -6303053730206256533, 598769270772963796, -2041316525404296230, -3860133938689713137, 4497202060050914136, 8955694409023320159, 3976567367560776009, -9165275604411410822, 1012738234769039757, 7642490246886963502, -3432866062799149095, 2519199740046251471, 2388427761311398841, -6886560953340875448, -4905186677634319463, -2365025404983910008, -8627402965240057817, -7397018902928602797, 1108859385650835241, 5281094978891453223, 6855360109813178939, -7807165856254133598, 1028026472880211244, 16660751466306624, 4072175354537615176, 2046113304521435920, -4044116572714093080, 98476484927120434, -5650328009808548456, -1384196055490211145, 8269412378242105431, -3207741555694791033, 8461694112286132273, 7684200390338837062, -3510258253288340132, 8994172498716928245, 5962685449452339479, -6226929694277901237, -3500613953333362662, -1492091879021245132, -947060640443297664, -6146648192398214417, -4464345544784150661, 6672100140478851757, -1340386687486760416, -3402143056639425167, -8508238454664137195, 964918476456248216, -7768463348829798026, 7756599010305739999, 1151692851232028639, 5052762745532257677, -6938353080763027108, 6683494536543705153, -6365889230484419309, 7384531241040909254, -4442336472114294091, -3750727149103368055, -8877412501490432986, 2647020543458892072, 6274135164775101483, -3649936680386055010, -7567305039792354763, -2172430473128016611, -2414729292194865719, 1408230014943390277, 4364677156300888178, 755861929549178049, 8235690776885053324, 8581387345513151684, 5002718336674882238, -870258856608853484, -1483711216472527900, -1255676054139266272, 331419834776310203, 1622392659577676198, 1187388789833685773, -5932747467864101101, 8122153151262337345, -2146380548913123401, 8197662599537401443, 8506067402867065505, 3090918727224804345, 3744225329829803414, 7619357059829297568, 556700409131325501, 5248429818045721574, 4765015544140971772, 2971482486644427028, -9173245872558505964, -2210735674653180475, 1181488914969268296, 2089494377150191570, 2047582435108024564, -6175545876351053551, -3298063022651817995, -1325629347910090158, 7488863875459007234, -5497017350454887793, -6756613781665488411, -6330009014934080380, 7681124670001326945, -7376366050502109636, 7992819870754351976, 3544290132427354974, -254827227758952719, -5704064381235954635, -836110888355241863, 7698549346624041319, 2301405470858849916, -481871362892611650, 6645744400280051944, -3818320263511106188, 1562581647772413229, -8160175779708883692, -2739834834172049430, 3510749139267324868, -5348896967283946783, 3527384472005761253, 4400799032050497147, -8651238311044541754, -5523410360681048732, 7071021940179800806, -5960796444158211925, 8420370346185308708, -2728886487595348029, 3105537168230717181, -1517621972941887996, 8452927690910375980, 6016490440494310456, -6889421189750345676, -2831286529760432055, -2189711506599834998, -7186866319154931067, -7009440320556973546, -2037384534764248619, -2220440490024002478, -377216044270617087, -1134884987470025768, 5192116499655548796, -3347230676841655272, -9130715416947308740, 9204760499816567337, -2439108211250476827, 1538934571518472975, -337514931320682527, -1674538086055718391, -3843322791290462622, 953749838960659962, 3330174901016008157, -2756012697370451081, -3602025464158100127, 3704439510960864841, 3752924436635734010, -5000939990480386852, 6154714044831917923, -8885087254969833946, 8407434459532892399, 4101903548525500975, 7904189481335560232, -2940053311648165067, 7494278666585169078, 1192828145405490948, 4470543315284180590, 654377960824023051, 2686967977432480840, 1411203428069491170, -8646993717939343792, 1159570865425141646, 8797484166341348183, 1079738560110059198, 1312350127490152747, -5189555431814227920, -2519118820283044758, 6059756840747708677, 5774693484122764099, -4349072189170425833, 4035740869403628813, 1511153166937753622, -3218856330350607949, 7304305360157341382, 3235867109258004764, -8951317005617098076, -162420324859555355, -793345512783903889, 5117521076123648029, -5882312494461926993, 8597264656412748201, -2877683839203210639, 9189818776605217015, 3313374825585251422, -7874810056424078419, 5674307591120690376, 9164898553319153477, 4358330615806437879, -5310359817210733626, 3113922769030946482, -659865237366019522, -9119890847611597075, -9205810881436744029, 8288333514535517283, 110170749276212955, 4325548561407427018, -1734212991042000302, -6873916426971903298, -7698545135503972364, -6954734571985878843, 1921094010145318263, 8877598562894529515, 241048672326064469, 900676715600069606, 5777523205257439109, 3010110724142136055, 8665660702093987211, 8608092300575511901, 7093280185971788300, 2944189561076742298, -2953386007626714319, -4900156269772444277, -5634850246813770872, 2948626453088923273, 2176870549249253374, -7387349836523930484, -9134092894261200380, 3875564163537339084, 6061299752516911114, -8854152481465861942, -9205171033569700009, 1363364174055650687]
INFO  15:30:22 Create new Keyspace: KeyspaceMetadata{name=system_traces, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}}, tables=[org.apache.cassandra.config.CFMetaData@1ef21588[cfId=c5e99f16-8677-3914-b17e-960613512345,ksName=system_traces,cfName=sessions,flags=[COMPOUND],params=TableParams{comment=tracing sessions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@cc79ec64, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [client command coordinator duration request started_at parameters]],partitionKeyColumns=[session_id],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[client, command, session_id, coordinator, request, started_at, duration, parameters],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@43aa6aac[cfId=8826e8e9-e16a-3728-8753-3bc1fc713c25,ksName=system_traces,cfName=events,flags=[COMPOUND],params=TableParams{comment=tracing events, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@cc79ec64, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [activity source source_elapsed thread]],partitionKeyColumns=[session_id],clusteringColumns=[event_id],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[activity, session_id, thread, event_id, source, source_elapsed],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO  15:30:22 Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
INFO  15:30:22 Initializing system_traces.events
INFO  15:30:22 Initializing system_traces.sessions
INFO  15:30:22 Create new Keyspace: KeyspaceMetadata{name=system_distributed, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}}, tables=[org.apache.cassandra.config.CFMetaData@7a49fac6[cfId=759fffad-624b-3181-80ee-fa9a52d1f627,ksName=system_distributed,cfName=repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@cc79ec64, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [coordinator exception_message exception_stacktrace finished_at parent_id range_begin range_end started_at status participants]],partitionKeyColumns=[keyspace_name, columnfamily_name],clusteringColumns=[id],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[status, id, coordinator, finished_at, participants, exception_stacktrace, parent_id, range_end, range_begin, exception_message, keyspace_name, started_at, columnfamily_name],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@19525fa0[cfId=deabd734-b99d-3b9c-92e5-fd92eb5abf14,ksName=system_distributed,cfName=parent_repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@cc79ec64, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [exception_message exception_stacktrace finished_at keyspace_name started_at columnfamily_names options requested_ranges successful_ranges]],partitionKeyColumns=[parent_id],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.TimeUUIDType,columnMetadata=[requested_ranges, exception_message, keyspace_name, successful_ranges, started_at, finished_at, options, exception_stacktrace, parent_id, columnfamily_names],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@59907bc9[cfId=5582b59f-8e4e-35e1-b913-3acada51eb04,ksName=system_distributed,cfName=view_build_status,flags=[COMPOUND],params=TableParams{comment=Materialized View build status, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@cc79ec64, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[] | [status]],partitionKeyColumns=[keyspace_name, view_name],clusteringColumns=[host_id],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[status, keyspace_name, view_name, host_id],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO  15:30:22 Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized
INFO  15:30:22 Initializing system_distributed.parent_repair_history
INFO  15:30:22 Initializing system_distributed.repair_history
INFO  15:30:22 Initializing system_distributed.view_build_status
INFO  15:30:22 Node /172.17.0.4 state jump to NORMAL
INFO  15:30:22 Create new Keyspace: KeyspaceMetadata{name=system_auth, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[org.apache.cassandra.config.CFMetaData@2bcd7a78[cfId=5bc52802-de25-35ed-aeab-188eecebb090,ksName=system_auth,cfName=roles,flags=[COMPOUND],params=TableParams{comment=role definitions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@cc79ec64, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [can_login is_superuser salted_hash member_of]],partitionKeyColumns=[role],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[role, salted_hash, member_of, can_login, is_superuser],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@14f7f6de[cfId=0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d,ksName=system_auth,cfName=role_members,flags=[COMPOUND],params=TableParams{comment=role memberships lookup table, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@cc79ec64, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[role],clusteringColumns=[member],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[role, member],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@53491684[cfId=3afbe79f-2194-31a7-add7-f5ab90d8ec9c,ksName=system_auth,cfName=role_permissions,flags=[COMPOUND],params=TableParams{comment=permissions granted to db roles, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@cc79ec64, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | [permissions]],partitionKeyColumns=[role],clusteringColumns=[resource],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[resource, role, permissions],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@24fc2ad0[cfId=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec,ksName=system_auth,cfName=resource_role_permissons_index,flags=[COMPOUND],params=TableParams{comment=index of db roles with permissions granted on a resource, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@cc79ec64, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[resource],clusteringColumns=[role],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[resource, role],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO  15:30:23 Not submitting build tasks for views in keyspace system_auth as storage service is not initialized
INFO  15:30:23 Initializing system_auth.resource_role_permissons_index
INFO  15:30:23 Initializing system_auth.role_members
INFO  15:30:23 Initializing system_auth.role_permissions
INFO  15:30:23 Initializing system_auth.roles
INFO  15:30:23 Waiting for gossip to settle before accepting client requests...
INFO  15:30:31 No gossip backlog; proceeding
INFO  15:30:31 Netty using native Epoll event loop
INFO  15:30:31 Using Netty Version: [netty-buffer=netty-buffer-4.0.39.Final.38bdf86, netty-codec=netty-codec-4.0.39.Final.38bdf86, netty-codec-haproxy=netty-codec-haproxy-4.0.39.Final.38bdf86, netty-codec-http=netty-codec-http-4.0.39.Final.38bdf86, netty-codec-socks=netty-codec-socks-4.0.39.Final.38bdf86, netty-common=netty-common-4.0.39.Final.38bdf86, netty-handler=netty-handler-4.0.39.Final.38bdf86, netty-tcnative=netty-tcnative-1.1.33.Fork19.fe4816e, netty-transport=netty-transport-4.0.39.Final.38bdf86, netty-transport-native-epoll=netty-transport-native-epoll-4.0.39.Final.38bdf86, netty-transport-rxtx=netty-transport-rxtx-4.0.39.Final.38bdf86, netty-transport-sctp=netty-transport-sctp-4.0.39.Final.38bdf86, netty-transport-udt=netty-transport-udt-4.0.39.Final.38bdf86]
INFO  15:30:31 Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO  15:30:31 Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
INFO  15:30:33 Scheduling approximate time-check task with a precision of 10 milliseconds
INFO  15:30:33 Created default superuser role 'cassandra'

Step 3: Create a second Cassandra Node

We want to start a second Cassandra container on the same Docker host for simple testing. We will connect to the container running on the first node via IP. For that we need to find out the IP address as follows:

(dockerhost)$ sudo docker inspect --format='{{ .NetworkSettings.IPAddress }}' cassandra-node1
172.17.0.2e

This information can be used in the next command by setting the CASSANDRA_SEEDS variable accordingly:

Note also that we have changed the port mapping in order to avoid port conflicts with the first Cassandra node:

(dockerhost)$ sudo docker run -it --rm --entrypoint="bash" --name cassandra-node2 \
              -p27000:7000 -p27001:7001 -p29042:9042 -p29160:9160 \
              -e CASSANDRA_SEEDS="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' cassandra-node1)" \
              cassandra

Note that we have overridden the default entrypoint, so we get access to the terminal.

We now start Cassandra on the second node:

(container):/# /docker-entrypoint.sh cassandra -f
...
INFO 17:37:03 Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO 17:37:03 Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
INFO 17:37:05 Created default superuser role 'cassandra'

On the first Cassandra node, we will see following additional log lines:

INFO 17:36:21 Handshaking version with /172.17.0.3
INFO 17:36:21 Handshaking version with /172.17.0.3
INFO 17:36:22 InetAddress /172.17.0.3 is now DOWN
INFO 17:36:22 Handshaking version with /172.17.0.3
INFO 17:36:23 Handshaking version with /172.17.0.3
INFO 17:36:54 [Stream #beb912f0-bca3-11e6-a935-4b019c4b758d ID#0] Creating new streaming plan for Bootstrap
INFO 17:36:54 [Stream #beb912f0-bca3-11e6-a935-4b019c4b758d, ID#0] Received streaming plan for Bootstrap
INFO 17:36:54 [Stream #beb912f0-bca3-11e6-a935-4b019c4b758d, ID#0] Received streaming plan for Bootstrap
INFO 17:36:55 [Stream #beb912f0-bca3-11e6-a935-4b019c4b758d] Session with /172.17.0.3 is complete
INFO 17:36:55 [Stream #beb912f0-bca3-11e6-a935-4b019c4b758d] All sessions completed
INFO 17:36:55 Node /172.17.0.3 state jump to NORMAL
INFO 17:36:55 InetAddress /172.17.0.3 is now UP

Note: if you get following error message:

Exception (java.lang.RuntimeException) encountered during startup: A node with address /172.17.0.3 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.

you need to start the service using the following line instead:

(container):/# /docker-entrypoint.sh cassandra -f -Dcassandra.replace_address=$(ip addr show | grep eth0 | grep -v '@' | awk '{print $2}' | awk -F"\/" '{print $1}')

The error will show up, if you have connected a Cassandra node to the cluster, then you destroy the node (by stopping the container) and re-start a new container. The new container will re-claim the now unused IP address of the destroyed container. However, this address is marked as unreachable within the cluster. We would like to re-use the IP address in the cluster, which requires the -Dcassandra.replace_address option.

The term

(container):/# ip addr show | grep eth0 | grep -v '@' | awk '{print $2}' | awk -F"\/" '{print $1}'
172.17.0.3

will return the current IP address of eth0 of the docker container and helps to feed in the correct IP address to the-Dcassandra.replace_address option.

Step 4: Start a CQL Client Container

Now we want to add some data to the distributed noSQL database. For that, we start a third container that can be used as CQL Client (CQL=Cassandra Query Language similar to SQL). We can start a CQL shell like follows:

(dockerhost)$ sudo docker run -it --rm -e CQLSH_HOST=$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' cassandra-node1) --name cassandra-client --entrypoint=cqlsh cassandra
Connected to Test Cluster at 172.17.0.2:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh>

Step 5: Create Keyspace

Now let us create a keyspace. A keyspace is the pendant for a database in SQL databases:

cqlsh> create keyspace mykeyspace with replication = {'class':'SimpleStrategy','replication_factor' : 2};
cqlsh>

Upon successful creation, the prompt will be printed without error.

Step 6: Create Table

For adding data, we need to enter the keypace and

cqlsh> use mykeyspace;
cqlsh:mykeyspace> create table usertable (userid int primary key, usergivenname varchar, userfamilyname varchar, userprofession varchar);
cqlsh:mykeyspace>

Step 7: Add Data

Now we can add our data:

cqlsh:mykeyspace> insert into usertable (userid, usergivenname, userfamilyname, userprofession) values (1, 'Oliver', 'Veits', 'Freelancer');
cqlsh:mykeyspace>

The CQL INSERT command has the same syntax as an SQL INSERT command.

Step 8 (optional): Update Data

We now can update a single column as well:

cqlsh:mykeyspace> update usertable set userprofession = 'IT Consultant' where userid = 1;

Now let us read the entry:

qlsh:mykeyspace> select * from usertable where userid = 1;

 userid | userfamilyname | usergivenname | userprofession
--------+----------------+---------------+----------------
      1 |          Veits |        Oliver |  IT Consultant

(1 rows)
cqlsh:mykeyspace>

Step 9 (optional): Query on Data other than the primary Index

In Cassandra, you cannot select upon a column without an index, if data filtering is not enabled:

cqlsh:mykeyspace> select * from usertable where userprofession = 'IT Consultant';
InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING"

Let us add a secondary index to enable the query without a performance impact:

cqlsh:mykeyspace> create index idx_dept on usertable(userprofession);

Now the same query should be successful:

cqlsh:mykeyspace> select * from usertable where userprofession = 'IT Consultant';
 userid | userfamilyname | usergivenname | userprofession
--------+----------------+---------------+----------------
      1 |          Veits |        Oliver |  IT Consultant
(1 rows)

Yes, perfect.

Step 10: Test Resiliency

In the moment, we have following topology (with all nodes and the client being Docker containers on the same Docker host):

Now, we will test, whether the data is retained, if the Cassandra application on node2 is stopped first. For that we stop the application on node2 by pressing Ctrl-C.

On node1 we see:

INFO 18:06:50 InetAddress /172.17.0.5 is now DOWN
INFO 18:06:51 Handshaking version with /172.17.0.5

On the client we see that the data is still there:

cqlsh:mykeyspace> select * from usertable where userprofession = 'IT Consultant';
 userid | userfamilyname | usergivenname | userprofession
--------+----------------+---------------+----------------
      1 |          Veits |        Oliver |  IT Consultant
(1 rows)

Now let us start the Cassandra application on node 2 again and wait some time until the nodes are synchronized. On node1 we will get a log similar to:

INFO  17:36:35 Handshaking version with /172.17.0.4
INFO  17:38:58 Handshaking version with /172.17.0.4
INFO  17:38:58 Handshaking version with /172.17.0.4
INFO  17:39:00 Node /172.17.0.4 has restarted, now UP
INFO  17:39:00 Node /172.17.0.4 state jump to NORMAL
INFO  17:39:00 InetAddress /172.17.0.4 is now UP
INFO  17:39:00 Updating topology for /172.17.0.4
INFO  17:39:00 Updating topology for /172.17.0.4

Now we can stop Cassandra on node1 by pressing Ctrl-C on terminal 1. On node2, we will get a message similar to:

INFO  17:41:32 InetAddress /172.17.0.3 is now DOWN
INFO  17:41:32 Handshaking version with /172.17.0.3

At the same time, the node1 container is destroyed, since we have not changed the entrypoint for node1 and we have given the --rm option in the docker run command in step 2.

Now, we verify that the data is still retained:

cqlsh:mykeyspace> select * from usertable where userprofession = 'IT Consultant';
NoHostAvailable:

Oh, yes, that is clear: we have used node1’s IP address and port, when we have started the client.

Let us now connect to node2 by entering “exit" and starting a new client container like follows:

(dockerhost)$ sudo docker run -it --rm -e CQLSH_HOST=$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' cassandra-node2) -e CQLSH_PORT=9042 --name cassandra-client --entrypoint=cqlsh cassandra
Connected to Test Cluster at 172.17.0.4:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh>

To be honest, I was a little bit confused here and would have expected that I need to connect to port 29042 instead, since I have started node2 with a port mapping from 29042 (outside port) to 9042 (container port). But this is wrong: from the Docker host, we can directly access the node2 container IP address with all its ports, including port 9042. Only, if we want to access container from outside the Docker host, we need to access port 29042 of the Docker host IP address instead of port 9042 of the node2 container:

(dockerhost)$ netstat -an | grep 9042
tcp6       0      0 :::29042                :::*                    LISTEN

Now let us check on the second node that the data is retained:

Anyway, we are connected to the second node now and can check the data:

cqlsh> use mykeyspace;
cqlsh:mykeyspace> select * from usertable where userid = 1;
 userid | userfamilyname | usergivenname | userprofession
--------+----------------+---------------+----------------
      1 |          Veits |        Oliver |  IT Consultant
(1 rows)

Perfect! The data is still there.

Appendix A: No keyspace has been specified.

If you get an error message like follows,

cqlsh> select * from usertable where userprofession = 'IT Consultant';
InvalidRequest: Error from server: code=2200 [Invalid query] message="No keyspace has been specified. USE a keyspace, or explicitly specify keyspace.tablename"

Then you have forgotten to prepend the “use mykeyspace; command:

cqlsh> use mykeyspace;
cqlsh:mykeyspace> select * from usertable where userid = 1;
 userid | userfamilyname | usergivenname | userprofession
--------+----------------+---------------+----------------
      1 |          Veits |        Oliver |  IT Consultant
(1 rows)

Summary

In this blog post we have performed following tasks:

  1. Introduced Cassandra with a little comparison with Hadoop
  2. Started a Cassandra node in a Docker container
  3. Started a second Cassandra node and build a Cassandra cluster
  4. Started a Cassandra Client in a container
  5. Added Data with replication factor 2 and performed some CQL commands for a warm-up
  6. shut down node 2 and verified that the data is still available
  7. started node 2 again and wait some seconds
  8. shut down node1 and verified that the data is still available on node2

With this test, we could verify that the data replication between nodes in a Cassandra cluster works and no data is lost, if a node fails.

 

3

Java Build Automation Part 2: Create executable jar using Gradle

Original title: How to build a lean JAR File with Gradle

In this step by step guide, we will show that Gradle is a good alternative to Maven for packaging java code into executable jar files. In order to keep the executable jar files “lean”, we will keep the dependent jar files outside of the jar in a separate folder.

Tools Used

  1. Maven 3.3.9
  2. JDK 1.8.0_101
  3. log4j 1.2.17 (downloaded automatically)
  4. Joda-time 2.5 (downloaded automatically)
  5. Git-2.8.4 with GNU bash 4.3.42(5)

Why using Gradle for a Maven Project?

In this blog post, we will show how Gradle can be used to create a executable/runnable jar. The task has been accomplished on this popular Mkyong blog post by using Maven. Why would we want to do the same task using Gradle?

By working with both, Maven and Gradle, I have found that:

  • Gradle allows me to move any resource file to outside of the jar without the need of any additional Linux script or alike;
  • Gradle allows me to easily create an executable/runnable jar for the JUnit tests, even if those are not separated into a separate project.

Moreover, while Maven is descriptive, Gradle is procedural in nature. With Maven, you describe the goal and you rely on Maven and its plugins to perform the steps you had in mind. Whereas with Gradle, you have explicit control on each step of the build process. Gradle is easy to understand for programmers and it gives them fine-grained control over the build process.

The Goal: a lean, executable JAR File

In the following step by step guide, we will create a lean executable jar file with all dependent libraries and resources.

Step 1 Download Hello World Maven Project of Mkyong

Download this hello world Maven project you can find on this popular HowTo page from Mkyong:

curl -OJ http://www.mkyong.com/wp-content/uploads/2012/11/maven-create-a-jar.zip
unzip maven-create-a-jar.zip
cd dateUtils

Logs:

$ curl -OJ http://www.mkyong.com/wp-content/uploads/2012/11/maven-create-a-jar.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7439  100  7439    0     0  23722      0 --:--:-- --:--:-- --:--:-- 24963

olive@LAPTOP-P5GHOHB7  /d/veits/eclipseWorkspaceRecent/MkYong/ttt
$ unzip maven-create-a-jar.zip
Archive:  maven-create-a-jar.zip
   creating: dateUtils/
  inflating: dateUtils/.classpath
  inflating: dateUtils/.DS_Store
   creating: __MACOSX/
   creating: __MACOSX/dateUtils/
  inflating: __MACOSX/dateUtils/._.DS_Store
  inflating: dateUtils/.project
   creating: dateUtils/.settings/
  inflating: dateUtils/.settings/org.eclipse.jdt.core.prefs
  inflating: dateUtils/log4j.properties
  inflating: dateUtils/pom.xml
   creating: dateUtils/src/
   creating: dateUtils/src/main/
   creating: dateUtils/src/main/java/
   creating: dateUtils/src/main/java/com/
   creating: dateUtils/src/main/java/com/mkyong/
   creating: dateUtils/src/main/java/com/mkyong/core/
   creating: dateUtils/src/main/java/com/mkyong/core/utils/
  inflating: dateUtils/src/main/java/com/mkyong/core/utils/App.java
   creating: dateUtils/src/main/resources/
  inflating: dateUtils/src/main/resources/log4j.properties
   creating: dateUtils/src/test/
   creating: dateUtils/src/test/java/
   creating: dateUtils/src/test/java/com/
   creating: dateUtils/src/test/java/com/mkyong/
   creating: dateUtils/src/test/java/com/mkyong/core/
   creating: dateUtils/src/test/java/com/mkyong/core/utils/
  inflating: dateUtils/src/test/java/com/mkyong/core/utils/AppTest.java
olive@LAPTOP-P5GHOHB7  /d/veits/eclipseWorkspaceRecent/MkYong/ttt
$ cd dateUtils/

olive@LAPTOP-P5GHOHB7  /d/veits/eclipseWorkspaceRecent/MkYong/ttt/dateUtils
$ 

Step 2 (optional): Create GIT Repository

In order to see, which files have been changed by which step, we can create a local GIT repository like follows

git init
# echo "Converting Maven to Gradle" > Readme.txt
git add .
git commit -m "first commit"

After each step, you then can perform the last two commands with a different message, so you can always go back to a previous step, if you need to do so. If you have made changes in a step that you have not committed yet, you can go back easily to the last clean commit state by issuing the command

# go back to status of last commit:
git stash -u

Warning: this will delete any new files you have created since the last commit.

Step 3 (required): Initialize Gradle

gradle init

This will automatically create a file build.gradle file from the Maven POM file with following content:

apply plugin: 'java'
apply plugin: 'maven'

group = 'com.mkyong.core.utils'
version = '1.0-SNAPSHOT'

description = """dateUtils"""

sourceCompatibility = 1.7
targetCompatibility = 1.7

repositories {

     maven { url "http://repo.maven.apache.org/maven2" }
}
dependencies {
    compile group: 'joda-time', name: 'joda-time', version:'2.5'
    compile group: 'log4j', name: 'log4j', version:'1.2.17'
    testCompile group: 'junit', name: 'junit', version:'4.11'
}

Step 4 (required): Gather Data

Since we are starting from a Maven project, which is prepared to create a runnable JAR via Maven already, we can extract the needed data from the POM.xml file:

MAINCLASS=`grep '<mainClass' pom.xml | cut -f2 -d">" | cut -f1 -d"<"`

Note: In cases with non-existing maven plugin, you need to set the MAINCLASS manually, e.g.

MAINCLASS=com.mkyong.core.utils.App

We also can define, where the dependency jars will be copied to later:

DEPENDENCY_JARS=dependency-jars

Logs:

$ MAINCLASS=`grep '<mainClass' pom.xml | cut -f2 -d">" | cut -f1 -d"<"`
$ echo $MAINCLASS
com.mkyong.core.utils.App
$ DEPENDENCY_JARS=dependency-jars
echo $DEPENDENCY_JARS
dependency-jars

Step 5 (required): Prepare to copy dependent Jars

Here, we will add instructions to the build.gradle file, which dependency JAR files are to be copied into a directory accessible by the executable jar.

We will need to copy the jars, we depend on, to a folder the runnable jar will access later on. See e.g. this StackOverflow question on this topic.

cat << END >> build.gradle

// copy dependency jars to build/libs/$DEPENDENCY_JARS 
task copyJarsToLib (type: Copy) {
    def toDir = "build/libs/$DEPENDENCY_JARS"

    // create directories, if not already done:
    file(toDir).mkdirs()

    // copy jars to lib folder:
    from configurations.compile
    into toDir
}
END

Step 6 (required): Prepare the Creation of an executable JAR File

In this step, we define in the build.gradle file, how to create an executable jar file.

cat << END >> build.gradle
jar {
    // exclude log properties (recommended)
    exclude ("log4j.properties")

    // make jar executable: see http://stackoverflow.com/questions/21721119/creating-runnable-jar-with-gradle
    manifest {
        attributes (
            'Main-Class': '$MAINCLASS',
            // add classpath to Manifest; see http://stackoverflow.com/questions/30087427/add-classpath-in-manifest-file-of-jar-in-gradle
            "Class-Path": '. dependency-jars/' + configurations.compile.collect { it.getName() }.join(' dependency-jars/')
            )
    }
}
END

Step 7 (required): Define build Dependencies

Up to now, a task copyJarsToLib was defined, but this task will not be executed, unless we tell Gradle to do so. In this step, we will specify that each time, a Jar is created, the copyJarsToLib task is to be performed beforehand. This can be done by telling Gradle that the jar goal depends on the copyJarsToLib task like follows:

cat << END >> build.gradle

// always call copyJarsToLib when building jars:
jar.dependsOn copyJarsToLib
END

Step 8 (required): Build Project

Meanwhile, the build.gradle file should have following content:

apply plugin: 'java'
apply plugin: 'maven'

group = 'com.mkyong.core.utils'
version = '1.0-SNAPSHOT'

description = """dateUtils"""

sourceCompatibility = 1.7
targetCompatibility = 1.7

repositories {

     maven { url "http://repo.maven.apache.org/maven2" }
}
dependencies {
    compile group: 'joda-time', name: 'joda-time', version:'2.5'
    compile group: 'log4j', name: 'log4j', version:'1.2.17'
    testCompile group: 'junit', name: 'junit', version:'4.11'
}

// copy dependency jars to build/libs/dependency-jars
task copyJarsToLib (type: Copy) {
    def toDir = "build/libs/dependency-jars"

    // create directories, if not already done:
    file(toDir).mkdirs()

    // copy jars to lib folder:
    from configurations.compile
    into toDir
}

jar {
    // exclude log properties (recommended)
    exclude ("log4j.properties")

    // make jar executable: see http://stackoverflow.com/questions/21721119/creating-runnable-jar-with-gradle
    manifest {
        attributes (
            'Main-Class': 'com.mkyong.core.utils.App',
            // add classpath to Manifest; see http://stackoverflow.com/questions/30087427/add-classpath-in-manifest-file-of-jar-in-gradle
            "Class-Path": '. dependency-jars/' + configurations.compile.collect { it.getName() }.join(' dependency-jars/')
            )
    }
}

// always call copyJarsToLib when building jars:
jar.dependsOn copyJarsToLib

Now is the time to create the runnable jar file:

gradle build

Note: Be patient at this step: it can appear to be hanging for several minutes, if it is run the first time, while it is working in the background.

This will create the runnable jar on build/libs/dateUtils-1.0-SNAPSHOT.jar and will copy the dependency jars to build/libs/dependency-jars/

Logs:

$ gradle build
:compileJava
warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning
:processResources
:classes
:copyJarsToLib
:jar
:assemble
:compileTestJava
warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning
:processTestResources UP-TO-DATE
:testClasses
:test
:check
:build

BUILD SUCCESSFUL

Total time: 3.183 secs

$ ls build/libs/
dateUtils-1.0-SNAPSHOT.jar dependency-jars

$ ls build/libs/dependency-jars/
joda-time-2.5.jar log4j-1.2.17.jar

Step 9: Execute the JAR file

It is best practice to exclude the log4j.properties file from the runnable jar file, and place it outside of the jar file, since we want to be able to change logging levels at runtime. This is, why we had excluded the properties file in step 6. In order to avoid an error “No appenders could be found for logger”, we need not specify the location of the log4j.properties properly on the command-line.

Step 9.1 Execute JAR file on Linux

On a Linux system, we run the command like follows:

java -jar -Dlog4j.configuration=file:full_path_to_log4j.properties build/libs/dateUtils-1.0-SNAPSHOT.jar

Example:

$ java -jar -Dlog4j.configuration=file:/usr/home/me/dateUtils/log4j.properties build/libs/dateUtils-1.0-SNAPSHOT.jar
11:47:33,018 DEBUG App:18 - getLocalCurrentDate() is executed!
2016-11-14

Note: if the log4j.properties file is on the current directory on a Linux machine, we also can create a batch file run.sh with the content

#!/usr/bin/env bash
java -jar -Dlog4j.configuration=file:`pwd`/log4j.properties build/libs/dateUtils-1.0-SNAPSHOT.jar

and run it via bash run.sh

Step 9.1 Execute JAR file on Windows

In case of Windows in a CMD shell all paths need to be in Windows style:

java -jar -Dlog4j.configuration=file:D:\veits\eclipseWorkspaceRecent\MkYong\dateUtils\log4j.properties build\libs\dateUtils-1.0-SNAPSHOT.jar
11:45:30,007 DEBUG App:18 - getLocalCurrentDate() is executed!
2016-11-14

If we run the command on a Windows GNU bash shell, the syntax is kind of mixed: the path to the jar file is in Linux style while the path to the log properties file needs to be in Windows style (this is, how the Windows java.exe file expects the input of this option):

$ java -jar -Dlog4j.configuration=file:'D:\veits\eclipseWorkspaceRecent\MkYong\dateUtils\log4j.properties' build/libs/dateUtils-1.0-SNAPSHOT.jar
11:45:30,007 DEBUG App:18 - getLocalCurrentDate() is executed!
2016-11-14

Inverted commas have been used in order to avoid the necessity of escaped backslashes like D:\\veits\\eclipseWorkspaceRecent\\… needed on a Windows system.

Note: if the log4j.properties file is on the current directory on a Windows machine, we also can create a batch file run.bat with the content

java -jar -Dlog4j.configuration=file:%cd%\log4j.properties build\libs\dateUtils-1.0-SNAPSHOT.jar

To run the bat file on GNU bash on Windows, just type ./run.bat

Yepp, that is it: the hello world executable file is printing the date to the console, just as it did in Mkyong’s blog post, where the executable file was created using Maven.

Download the source code from GIT.

Note: in the source code, you also will find a file named prepare_build.gradle.sh, which can be run on a bash shell and will replace the manual steps 4 to 7.

References

Next Steps

  • create an even leaner jar with resource files kept outside of the executable jar. This opens the opportunity to changing resource files at runtime.
  • create an executable jar file that will run the JUnit tests.

 

3

LXD vs. Docker — or: getting started with LXD Containers

Container technology is not new: it had existed long before the Docker hype around container technology has started after 2013. Now, with Docker containers having reached mainstream usage, there is a high potential of getting confused about available container types like Docker, LXC, LXD and CoreOS rocket. In this blog post we will explain, why LXD is not competing with Docker.

We will show in a few steps how to install and run LXC containers using LXD container management functions. For that, we will make use of an automated installation process based on Vagrant and VirtualBox.

What is LXD and why not using it as a Docker Replacement?

After working with Docker for quite some time, I have stumbled over another container technology: Ubuntu’s LXD (say “lex-dee”). What is the difference to Docker, and do they really compete with each other, as an article in the German “Linux Magazin” (May 2015) states?

As the developers of LXD point out, a main difference between Docker and LXD is that Docker focuses on application delivery from development to production, while LXD’s focus is system containers. This is, why LXD is more likely to compete with classical hypervisors like XEN and KVM and it is less likely to compete with Docker.

Ubuntu’s web page points out that LXD’s main goal is to provide a user experience that’s similar to that of virtual machines but using Linux containers rather than hardware virtualization.

For providing a user experience that is similar to that of virtual machines, Ubuntu integrates LXD with OpenStack through its REST API.  Although there are attempts to integrate Docker with OpenStack (project Magnum), Ubuntu comes much closer to feature parity with real hypervisors like XEN and KVM by offering features like snapshots and live migration. As any container technology, LXD offers a much lower resource footprint than virtual machines: this is, why LXD is sometimes called lightervisor.

One of the main remaining concerns of IT operations teams against usage of the container technology is that containers leave a “larger surface” to attackers than virtual machines. Canonical, the creators of Ubuntu and LXD is tackling security concerns by making LXD-based containers secure by default. Still, any low level security feature developed for LXC potentially is available for both, Docker and LXD, since they are based on LXC technology.

What does this mean for us?

We have learned that Docker offers a great way to deliver applications, while LXD offers a great way to reduce the footprint of virtual-machine-like containers. What, if you want to leverage the best of both worlds? One way is to run Docker containers within LXD containers. This and its current restrictions are described in this blog post of Stéphane Graber.

Okay; one step after the other: let us postpone the Docker in LXD discussion and let us get started with LXD now.

LXD Getting Started: a Step by Step Guide

This chapter largely follows this getting started web page. However, instead of trying to be complete, we will go through a simple end to end example. Moreover, we will add some important commands found on this nice LXD cheat sheet. In addition, we will explicitly record the example output of the commands.

Prerequisites:

  • Administration rights on you computer.
  • I have performed my tests with direct access to the Internet: via a Firewall, but without HTTP proxy. However, if you cannot get rid of your HTTP proxy, read this blog post.

Step 1: Install VirtualBox

If not already done, you need to install VirtualBox found here. See appendix A, if you encounter installation problems on Windows with the error message “Setup Wizard ended prematurely”. For my tests, I am using the already installed VirtualBox 5.0.20 r106931 on Windows 10.

Step 2: Install Vagrant

If not already done, you need to install Vagrant found here. For my tests, I am using an already installed Vagrant version 1.8.1 on my Windows 10 machine.

Step 3: Initialize and download an Ubuntu 16.0.4 Vagrant Box

In a future blog post, we want to test Docker in LXD containers. This is supported in Ubuntu 16.0.4 and higher. Therefore, we download the latest daily build of the corresponding Vagrant box. As a preparation, we create a Vagrantfile in a separate directory by issuing the following command:

vagrant init ubuntu/xenial64

You can skip the next command and directly run the vagrant up command, if you wish, since the box will downloaded automatically, if nocurrent version of the Vagrant box is found locally. However, I prefer to download the box first, and run the box later, since it is easier to observe, what happens during the boot.

vagrant box add ubuntu/xenial64

Depending on the speed of your Internet connection, you can take a break here.

Step 4: Boot the Vagrant Box as VirtualBox Image and connect to it

Then, we will boot the box with:

vagrant up

Note: if you encounter an error message like “VT-x is not available”, this may be caused by booting Windows 10 with Hyper-V enabled or by nested virtualization. According to this stackoverflow Q&A, running Virtualbox without VT-x is possible, if you make sure that the number of CPUs is one. For that, try to set vb.cpus = 1 in the Vagrantfile. Remove any statement like vb.customize ["modifyvm", :id, "--cpus", "2"] in the Vagrantfile. If you prefer to use VT-x on your Windows 10 machine, you need to disable Hyper-V. The Appendix: “Error message: “VT-x is not available” describes how to add a boot menu item that allows to boot without Hyper-V enabled.

Now let us connect to the machine:

vagrant ssh

Step 5: Install and initialize LXD

Now we need to install LXD on the Vagrant image by issuing the commands

sudo apt-get update
sudo apt-get install -y lxd
newgrp lxd

Now we need to initialize LXD with the lxd init interactive command:

ubuntu@ubuntu-xenial:~$ sudo lxd init
sudo: unable to resolve host ubuntu-xenial
Name of the storage backend to use (dir or zfs) [default=zfs]: dir
Would you like LXD to be available over the network (yes/no) [default=no]? yes
Address to bind LXD to (not including port) [default=0.0.0.0]:
Port to bind LXD to [default=8443]:
Trust password for new clients:
Again:
Do you want to configure the LXD bridge (yes/no) [default=yes]? no
LXD has been successfully configured.

I have decided to use dir as storage (since zfs was not enabled), have configured the LXD server to be available via the default network port 8443, and I have chosen to start without LXD bridge, since this article was pointing out that the LXD bridge does not allow SSH connections per default.

The configuration is written to a key value store and, which can be read with the lxc config get commands, e.g.

ubuntu@ubuntu-xenial:~$ lxc config get core.https_address
0.0.0.0:8443

The list of available system config keys can be found on this Git-hosted document. However, I have not found the storage backend type “dir”, I have configured. I guess, the system assumes that “dir” is used as long as the zfs and lvm variables are not set. Also, it is a little bit confusing that we configure LXD, but the config is read out via LXC commands.

Step 6: Download and start an LXC Image

Step 6.1 (optional): List remote LXC Repository Servers:

The images are stored on image repositories. Apart from the local repository, the default repositories have aliases images, ubuntu and ubuntu-daily:

ubuntu@ubuntu-xenial:~$ sudo lxc remote list
sudo: unable to resolve host ubuntu-xenial
+-----------------+------------------------------------------+---------------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    | PUBLIC | STATIC |
+-----------------+------------------------------------------+---------------+--------+--------+
| images          | https://images.linuxcontainers.org       | simplestreams | YES    | NO     |
+-----------------+------------------------------------------+---------------+--------+--------+
| local (default) | unix://                                  | lxd           | NO     | YES    |
+-----------------+------------------------------------------+---------------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | YES    | YES    |
+-----------------+------------------------------------------+---------------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | YES    | YES    |
+-----------------+------------------------------------------+---------------+--------+--------+
Step 6.2 (optional): List remote LXC Images:

List all available ubuntu images for amd64 systems on the images repository:

ubuntu@ubuntu-xenial:~$ sudo lxc image list images: amd64 ubuntu
sudo: unable to resolve host ubuntu-xenial
+-------------------------+--------------+--------+---------------------------------------+--------+---------+------------------------------+
|          ALIAS          | FINGERPRINT  | PUBLIC |              DESCRIPTION              |  ARCH  |  SIZE   |         UPLOAD DATE          |
+-------------------------+--------------+--------+---------------------------------------+--------+---------+------------------------------+
| ubuntu/precise (3 more) | adb92b46d8fc | yes    | Ubuntu precise amd64 (20160906_03:49) | x86_64 | 77.47MB | Sep 6, 2016 at 12:00am (UTC) |
+-------------------------+--------------+--------+---------------------------------------+--------+---------+------------------------------+
| ubuntu/trusty (3 more)  | 844bbb45f440 | yes    | Ubuntu trusty amd64 (20160906_03:49)  | x86_64 | 77.29MB | Sep 6, 2016 at 12:00am (UTC) |
+-------------------------+--------------+--------+---------------------------------------+--------+---------+------------------------------+
| ubuntu/wily (3 more)    | 478624089403 | yes    | Ubuntu wily amd64 (20160906_03:49)    | x86_64 | 85.37MB | Sep 6, 2016 at 12:00am (UTC) |
+-------------------------+--------------+--------+---------------------------------------+--------+---------+------------------------------+
| ubuntu/xenial (3 more)  | c4804e00842e | yes    | Ubuntu xenial amd64 (20160906_03:49)  | x86_64 | 80.93MB | Sep 6, 2016 at 12:00am (UTC) |
+-------------------------+--------------+--------+---------------------------------------+--------+---------+------------------------------+
| ubuntu/yakkety (3 more) | c8155713ecdf | yes    | Ubuntu yakkety amd64 (20160906_03:49) | x86_64 | 79.16MB | Sep 6, 2016 at 12:00am (UTC) |
+-------------------------+--------------+--------+---------------------------------------+--------+---------+------------------------------+

Instead of the “ubuntu” filter keyword in the image list command above, you can use any filter expression. E.g. sudo lxc image list images: amd64 suse will find OpenSuse images available for x86_64.

Step 6.3 (optional): Copy remote LXC Image to local Repository:

This command is optional, since the download will be done automatically with the lxc launch command below, if the image is not found on the local repository already.

lxc image copy images:ubuntu/trusty local:
Step 6.4 (optional): List local LXC Images:

We can list the locally stored images with the following image list command. If you have not skipped the last step, you will find following output:

ubuntu@ubuntu-xenial:~$ sudo lxc image list
sudo: unable to resolve host ubuntu-xenial
+-------+--------------+--------+-------------------------------------------+--------+----------+------------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                DESCRIPTION                |  ARCH  |   SIZE   |         UPLOAD DATE          |
+-------+--------------+--------+-------------------------------------------+--------+----------+------------------------------+
|       | 844bbb45f440 | no     | Ubuntu trusty amd64 (20160906_03:49)      | x86_64 | 77.29MB  | Sep 6, 2016 at 5:04pm (UTC)  |
+-------+--------------+--------+-------------------------------------------+--------+----------+------------------------------+
Step 6.5 (mandatory): Launch LXC Container from Image

With the lxc launch command, a container is created from the image. Moreover, if the image is not available in the local repository, it will automatically download the image.

ubuntu@ubuntu-xenial:~$ lxc launch images:ubuntu/trusty myTrustyContainer
Creating myTrustyContainer
Retrieving image: 100%
Starting myTrustyContainer

If the image is already in the local repository, the Retrievien image line is missing and the container can start within seconds (~6-7 sec in my case).

Step 7 (optional): List running Containers

We can list the running containers with the lxc list command, similar to a docker ps -a for those, who know Docker:

ubuntu@ubuntu-xenial:~$ lxc list
+-------------------+---------+------+------+------------+-----------+
|       NAME        |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+-------------------+---------+------+------+------------+-----------+
| myTrustyContainer | RUNNING |      |      | PERSISTENT | 0         |
+-------------------+---------+------+------+------------+-----------+

Step 8: Run a Command on the LXC Container

Now we are ready to run our first command on the container:

ubuntu@ubuntu-xenial:~$ sudo lxc exec myTrustyContainer ls /
sudo: unable to resolve host ubuntu-xenial
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Step 9: Log into and exit the LXC Container

We can log into the container just by running the shell with the lxc exec command:

ubuntu@ubuntu-xenial:~$ sudo lxc exec myTrustyContainer bash
sudo: unable to resolve host ubuntu-xenial
root@myTrustyContainer:~# exit
exit
ubuntu@ubuntu-xenial:~$

The container can be exited just by issuing the “exit” command. Different from Docker containers, this will not stop the container.

Step 10: Stop the LXC Container

The container can be stopped with the sudo lxc stop command:

ubuntu@ubuntu-xenial:~$ sudo lxc stop myTrustyContainer
sudo: unable to resolve host ubuntu-xenial
ubuntu@ubuntu-xenial:~$ lxc list
+-------------------+---------+------+------+------------+-----------+
|       NAME        |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+-------------------+---------+------+------+------------+-----------+
| myTrustyContainer | STOPPED |      |      | PERSISTENT | 0         |
+-------------------+---------+------+------+------------+-----------+

Summary

We have discussed the differences of Docker and LXD. We have found that Docker focuses on application delivery, while LXD seeks to offer Linux virtual environments as systems.

We have provided the steps needed to get started with LXD by showing how to

  • install the software,
  • download images,
  • start containers from the images and
  • running simple Linux commands on the images.

Next steps:

Here is a list of possible next steps on the path to Docker in LXC:

  • Networking
  • Docker in LXC container
  • LXD: Integration into OpenStack
  • Put it all together

Appendix A: VirtualBox Installation Problems: “Setup Wizard ended prematurely”

  • Download the VirtualBox installer
  • When I start the installer, everything seems to be on track until I see “rolling back action” and I finally get this:
    “Oracle VM Virtualbox x.x.x Setup Wizard ended prematurely”

Resolution of the “Setup Wizard ended prematurely” Problem

Let us try to resolve the problem: the installer of Virtualbox downloaded from Oracle shows the exact same error: “…ended prematurely”. This is not a docker bug. Playing with conversion tools from Virtualbox to VMware did not lead to the desired results.

The Solution: Google is your friend: the winner is: https://forums.virtualbox.org/viewtopic.php?f=6&t=61785. After backing up the registry and changing the registry entry

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Network -> MaxFilters from 8 to 20 (decimal)

and a reboot of the Laptop, the installation of Virtualbox is successful.

Note: while this workaround has worked on my Windows 7 notebook, it has not worked on my new Windows 10 machine. However, I have managed to install VirtualBox on Windows 10 by de-selecting the USB support module during the VirtualBox installation process. I remember having seen a forum post pointing to that workaround, with the additional information that the USB drivers were installed automatically at the first time a USB device was added to a host (not yet tested on my side).

Appendix B: Vagrant VirtualBox Error message: “VT-x is not available”

Error:

If you get an error message during vagrant up telling you that VT-x is not available, a reason may be that you have enabled Hyper-V on your Windows 10 machine: VirtualBox and Hyper-V cannot share the VT-x CPU:

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'thesteve0/openshift-origin' is up to date...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
 default: Adapter 1: nat
 default: Adapter 2: hostonly
==> default: Forwarding ports...
 default: 8443 (guest) => 8443 (host) (adapter 1)
 default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["startvm", "8ec20c4c-d017-4dcf-8224-6cf530ee530e", "--type", "headless"]

Stderr: VBoxManage.exe: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component ConsoleWrap, interface IConsole

Resolution:

Step 1: prepare your Windows machine for dual boot with and without Hyper-V

As Administrator, open a CMD and issue the commands

bcdedit /copy "{current}" /d "Hyper-V" 
bcdedit /set "{current}" hypervisorlaunchtype off
bcdedit /set "{current}" description "non Hyper-V"

Step 2: Reboot the machine and choose the “non Hyper-V” option.

Now, the vagrant up command should not show the “VT-x is not available” error message anymore.

4

IT Automation Part IV: Ansible Tower “Hello World” Example

This is part IV of a little “Hello World” example for Ansible, an IT automation (DevOps) tool. This time, we will get acquainted with Ansible Tower, a web front end for Ansible. The post has following content:

  • Quickest way of “installing” an Ansible Tower trial system (via Vagrant this time)
  • Step by step guide for a minimalistic “Hello World!” example, following closely the official documentation with following differences:
    • more insights on the dependencies of the steps (discussed with a simple dependency diagram)
    • shorter, simple page step by step tutorial with links to the official documentation at each step
  • Ansible Tower Benefits

Posts of this series:

  • Part I: Ansible Hello World with a comparison of Ansible vs. Salt vs. Chef vs. Puppet and a hello world example with focus on Playbooks (i.e. tasks), Inventories (i.e. groups of targets) and remote shell script execution.
  • Part II: Ansible Hello World reloaded with focus on templating: create and upload files based on jinja2 templates.
  • Part III: Salt Hello World example: same content as part I, but with Salt instead of Ansible
  • Part IV: Ansible Tower Hello World: investigates Ansible Tower, a professional Web Portal for Ansible

Why not “Installing” Ansible Tower via Docker?

In parts I to III of this blog series, I have made good experience downloading Docker images from Docker Hub instead of installing the software in question. This is, how we had “installed” Ansible and Saltstack. Also this time, I have tried to do so, but the attempts were less successful. I have tested following images:

  1. leowmjw/ubuntu-ansible-tower is a manual build and 8 months old at the time of writing. I had succeeded to accomplish almost all steps, but at the end, the launched jobs were stuck in the “pending” state and I could not find out, what is wrong.
  2. ybalt/ansible-tower is an automated build, but still 6 months old. The services seem to start, but first greeting I got was a “Fatal Error” welcome screen (maybe, since I did not map the certs private folder, but I did not want to mess around with certifications)

At this point, I had given up following the Docker path for now, even though there are many other public Ansible Tower images on Docker Hub I could test. Instead, I have found that installing Ansible Tower on a VirtualBox image by using Vagrant is a very convenient way of tackling the installation part. It is recommended for proof of concept purposes only, but it is an officially offered way of creating an Ansible Tower service; see this official documentation.

Installing Ansible Tower via Vagrant

Prerequisites

  • VirtualBox 5 is installed. You can get the software from here. See the Appendix A of part 1 of this series for some possible workarounds, if you have problems installing VirtualBox on a Windows system.
  • Vagrant is installed. Get the software from here.
  • Enough resources on the host system: additional 2 GB RAM and 2 vCPUs. However, the top command on my Notebook shows that the host system is running with less than 20% CPU load, although the Ansible Tower is running.

Step 1: install Ansible Tower via Vagrant and connect via SSH

Following this instructions, you can issue the following commands in order to download, start and connect to the Ansible Tower image:

$ vagrant init ansible/tower # creates the Vagrantfile
$ vagrant box add ansible/tower # optional; downloads the Vagrant box 

As an addition to the official documentation, I have added the second command. The download might take a long time,  and the next command will require your attention.

give you some time for a long coffee break, while the next command requires your immediate attention: I had been prompted for the Administrator password twice. If you fail to provide the password in time, you might come back from the coffee break and be surprised by an error with a message like: VBoxManage.exe: error: Failed to create the host-only adapter.

$ vagrant up --provider virtualbox # the --provider option is optional, since virtualbox is the default provider
$ vagrant ssh # connects to the machine

The following (or similar) content will be shown:

$ vagrant ssh
Last login: Sat Jun 11 00:16:51 2016 from gateway

  Welcome to Ansible Tower!

  Log into the web interface here:

    https://10.42.0.42/

    Username: admin
    Password: <some_random_password>

  The documentation for Ansible Tower is available here:

    http://www.ansible.com/tower/

  For help, email support@ansible.com
[vagrant@ansible-tower ~]$

Step 2: follow the official quick start guide (with minor changes)

We closely follow the official quick start guide with a minor adaption at step “4. Create the Credential” below. This is, because Vagrant does not allow for password login of the root user.

Step 2.1: Log into Ansible Tower

We now can connect via browser to http://www.ansible.com/tower/ and log in with the credentials:

Step 2.2 Import a License

Since we are logged in the first time ever, we will be provided with a pop-up like follows:

Click on “Get a Free Tower Trial License” and choose the appropriate license version. I have applied for the permanent free license for up to 10 nodes, and I have received an email from support@ansible.com with the license file. You can either drag&drop the license file to the browser, or you can copy the license file content into the corresponding field,  so the content of the License File field looks similar to:

{
    "company_name": "YourCompanyName", 
    "contact_email": "youremail@company.com", 
    "contact_name": "Your Name", 
    "hostname": "yourhostname", 
    "instance_count": 10, 
    "license_date": 2128169547, 
    "license_key": "your_license_key", 
    "license_type": "basic", 
    "subscription_name": "Basic Tower up to 10 Nodes"
}

After checking the “I agree …” checkbox and submitting the form, you will be rewarded with a License Accepted pop-up window:

After clicking OK, you will reach the Dashboard:

Step 2.3: Add all elements needed for a minimum Hello World Example

Now: where to start? For that, let us have a look at the data model and the dependencies. The data model of Ansible Tower, as provided by the Ansible Tower documentation looks like follows:

Note that (at least for new versions of Ansible), users can be directly attached to an organization with no need to create a team. The same holds for inventory groups. Let us get rid of those and perform a “Hello World” in following simplified model:

Okay, the figure might look slightly more complex than the one in the Ansible Tower documentation. This is, because I have added all mandatory parent to child dependencies, similar to the Unified Modeling Language notation.

Note: In the Ansible Tower model, each child has only a single parent, while each parent can have an arbitrary number of children. The only exception I have found so far are the Projects: each project can be assigned to one or more organizations.

As in the real world, you need to follow the arrows, if you want to reach your goal:

  1. Users, inventories and projects depend on organizations. However an organization named “default” is already pre-installed, so we do not need to do anything here.
  2. we need to create a user before we can create a credential
  3. we need to create an inventory before we can create a host
  4. we need to add a playbook directory, before we can define a project
  5. we need to create a playbook file, before we can define a job template. In addition, a credentials, an inventory and a project needs to be available.
  6. A job can be launched from a job template panel only

The official quick start guide goes through this model like follows:

Let us follow the same order. Nothing needs to be done for the organization  (“Step 0”): a “default” organization exists already. So, let us start with Step 1:

  1. Create a User
    Click on  and then on Users and then on  and add and save following data:
  2. Create an Inventory
    Click on Inventories, then  and and and save following data:
  3. Create a Host
    Click on Inventories, then on the newly created inventory, and then on  on the far right. Add and save following data:
  4. Create a Credential
    Click on  and then on Credentials and then on  . Here we deviate a little bit from the official documentation. The documentation asks to add and save following data:

    However, we have installed Ansible Tower using Vagrant, and per default, the root user cannot log in with a password. Therefore we will use the vagrant user and log in with the private SSH key like follows:

    The private SSH key can be found on the host system, from where you have started the ansible tower. From the directory containing the Vagrantfile, you need to navigate to CODE .vagrant/machines/default/virtualbox CODE and open the file CODE private_key. The content needs to be cut & paste into the Private Key field above.
  5. Create a Playbook Directory
    Start an ssh session to the Ansible Tower host. In our case, this is done by issuing
    CODE vagrant ssh CODE
    as shown above. Issue the command
    CODE sudo mkdir /var/lib/awx/projects/helloworld CODE
  6. Create a Playbook
    In the ssh session, start an editor session, e.g.
    CODE vi /var/lib/awx/projects/helloworld/helloworld.yml CODE:
    cut&paste following content into the file:

    ---
    - name: Hello World!
      hosts: all
    
      tasks:
    
      - name: Hello World!
        shell: echo "Hi! Tower is working!"
  7. Create a Project
    Click on Projects and on  and add and save following data:
  8. Create a Job Template
    Click on Job Templates and on  and add and save following data:

    Note that you need to use the other credential named “Vagrant User per SSH Key” we have created in Step 4 above.
  9. Create (Launch) a Job
    Click on Job Templates and on  in the corresponding line of the job template. In case of a password credential, add the password into the prompt (not needed for SSH key authentication):

This will lead to following result:

The command line output can be checked by clicking the  button:

Why Ansible Tower?

When we compare the many steps needed to run a simple “Hello World” example we could have accomplished on the command-line in less than half of the time (see e.g. part 1 of this blog series), we could ask ourselves, why someone would like to use Ansible tower at all. The main reasons (as I see it) are like follows:

  • Fine-grained role-based access control: in a real-world example, most steps above would be performed by an Ansible Tower administrator. As an example, you could define a team that is allowed to launch certain jobs only. Moreover, Ansible Tower seems to allow a quite fine-grained access control on what a team or user is allowed to do with Inventories and Job Templates (see  -> Teams -> Permissions):
  • Unlike StaltStack, Ansible (without Tower) does not have any notion of immediate or scheduled background jobs. In addition, it does not automatically record the execution time and results of any performed jobs, making audit trails a complex task. Ansible Tower is filling this gap.
  • Nice and easy graphical handling including statistics etc.
  • Ansible Tower offers a modern RESTful API, which allows it to integrate with existing tools and processes

Summary

In this blog post we have shown how Vagrant can help to quickly set up a local Ansible Tower trial. We have discussed the Ansible Tower object model in its simplest form, and could explain, why and which minimal steps are needed to perform a simple “Hello World”. At the end, we have compared Ansible Tower with simple command-line based Ansible (without Tower).

2

Choosing the right IaaS Provider for a custom Appliance or: how hard is it to install from ISO in the cloud?

Which Cloud Infrastructure provider allows to install custom appliances via ISO? The short answer is: none of the IaaS market leaders Amazon Web Serviese (AWS), Microsoft Azure offer the requested functionality, but they offer the workaround to locally install the virtual machine (VM) and upload the VM to the cloud. The cheaper alternative DigitalOcean does not offer any of those possibilities.

At the end, I thought I have found  the perfect solution to my problem: Ravello Systems a.k.a. Oracle ravello is a meta cloud infrastructure provider (they call it “nested virtualization provider”), which is re-selling the infrastructure from other IaaS providers like Amazon AWS and Google Engine. They offer a portal that supports the installation of a VM from an ISO in the cloud. Details see below. They write:

However, Ravello was ignoring my request for a trial for more than two months.

Ravello’s trial seems to be open for companies only? I even told them that I am about to found my own company, this did not help.

If you are representing a large company and if you are offering them a prospect to earn a lot of money, they might be reacting differently in your case, though. Good luck.

I am back at installing the image locally and uploading it to Amazon AWS. Maybe this is the cheaper alternative, anyway. I am back at the bright side of life…

At the end, after more than 2 months, I have got the activation link. The ISO Upload tool has some challenges with HTTP proxies, but is seems to work now.

Document Versions

v1.0 (2016-03-14): initially published version
v1.1 (2016-03-21): added a note on ravello’s nested virtualization solution, which makes the solution suitable for VMware testing on public non-VMware clouds
v1.2 (2016-03-23): added a note of my problems of getting a trial account; I have created a service ticket.
v1.3 (2016-03-30): Ravello has chosen to close my ticket without helping me. I am looking for alternatives.
v1.4 (2016-04-09): After I have complained about the closed ticket, they wrote a note that they are sorry on 2016-03-30. However, I have still not got an account. I have sent a new email asking for the status today.
v1.5 (2016-05-25): I have received an activation link on May, 11th. It has taken more than 2 months to get it. I am not sure, if I am still interested…

The Use Case

Integration of high tech systems with legacy systems is fun. At least, it is fun, if you have easy access from you development machine to the legacy systems. In my case, I was lucky enough: the legacy systems I am dealing with are modern communication systems that can be run on VMware. Using a two year old software version of the system, I have run the legacy system on my development machine. With that I could run my integration software against the real target system.

But why have I used a two year old software version of the legacy system? That is, why: the most recent versions of that system have such a high demand on the virtual resources (vCPU, DRAM) that it has outgrown my development machine: it was quite a bit…

…overloaded.

Possible Solutions

How to deal with this? Some thoughts of mine are:

  • I could buy a new notebook with, say 64 GB RAM.
    • this is an expensive option. Moreover, I am a road warrior type of developer and do a lot of coding in the train. Most notebooks with 64GB RAM are bulky and heavy and you need to take a power plant with you if you do not want to run out of energy during your trip.
  • I could develop a lightweight simulator that is mocking the behavior of the legacy system.
    • In the long run, I need to do something along those lines anyway: I want to come closer to Continuous Integration+Deployment process and for the automated tests in the CI/CD system, it is much simpler to run a simulator as part of the software than to run the tests against bulky legacy systems.
  • I could develop and test (including integration tests) in the IaaS cloud.

The Cloud Way of Providing a Test Environment

Yes, the IaaS cloud option is a particularly interesting one; especially, if development is done as a side job because:

  • I need to pay only for resources I use.
  • For most functional tests, I do not need full performance. I can go with cheaper, shared resources.
  • I can pimp up the legacy system and reserve resources for performance tests, while freeing up the resources again after finish of the test.
  • Last but not least, I am a cloud evangelist and therefore I should eat my own dog food (or drink my own champagne, I hope).

However: which are the potential challenges?

  1. Installation challenges of the legacy system in the cloud.
  2. How much do you pay for the VM, if it is shut down? Open Topic, but will not (yet) investigated in this blog post.
  3. How long does it take from opening the lid of the development notebook until I can access the legacy system? Open Topic, but will not (yet) investigated in this blog post.

First things first: in this post, I will concentrate on challenge 1.

The Cloud Way of installing (a custom appliance from ISO)

In my case, the legacy system must be installed from ISO. From my first investigation, it seems that this is a challenge with many IaaS providers. Let us have a closer look:

Comparison of IaaS Providers

  • DigitalOcean: they do not support the installation from ISO. See this forum post.
    • there is no workaround like local installation and upload of the image, see here. Shoot. 🙁

  • AWS: same thing: no ISO installation support.
    1. For AWS, the workaround is to install the system locally and to upload and convert the VM. See this stackoverflow post.
      One moment: didn’t I say the legacy system is too large for my notebook? Not a good option. 🙁
    2. Another workaround for AWS is to use a nested virtualization provider like ravello systems: they claim here that the installation of an AWS image from ISO is no problem.
      Note: ravello’s nested virtualization solution places an additional Hypervisor on top of AWS’ XEN hypervisor, in order to run VMware VMs on public clouds that do not support VMware VMs natively. This will not increase the performance, though and is intended for test environments only. However, this is exactly, what I am aiming at (for now).

Ravello claims: “With Ravello, uploading an ISO file is as simple as uploading your files to dropbox. Once the file is in your library in Ravello simply add the CD-ROM device to your VM and select your customer ISO file from your library.”

  • Microsoft Azure: not fully clear…
    • I have found here the information that an ISO can be attached to an existing VM. I do not know, though, whether or not the VM can be installed from the ISO by booting from ISO.
    • you can create a local image in VHD format and upload it to the cloud. However, the only (convenient) way to create this image is to install the VM on Hyper-V. I do not have access to Hyper-V and I do not want to spend any time on this for now. 🙁

Among those options, it seems like only AWS and ravello are possible feasible for me.

Even so, I need to take the risk caused by the fact that my legacy systems are supported on VMware only. However, this is a risk I need to accept, if I want to go with a low cost mainstream IaaS provider. A private cloud on dedicated VMware infrastructure is prohibitive with respect to effort and price.

Decision:

I have a more powerful notebook at home and I could install the image locally. However, I will give the meta IaaS provider Ravello Systems a try and I will install the legacy system via their overlay cloud. Within Ravello systems, I will choose AWS as the backend IaaS provider, because AWS is the number one IaaS provider (see this article pointing to the Gartner report) and therefore I want to gain some experience with AWS.

Note about the pricing comparison between AWS and ravello: I believe that ravello comes at higher rates (estimated 30-50%). But please do not take this for granted and calculate yourself, using the AWS monthly calculator and the ravello pricing page.

HowTo

More than 2 months after my application, I finally got an activation link. Looking for how to import the ISO, I have found this ravello link. However, the documentation is not good. They write:

To download the tool, click Download VM Import Tool on the Library page.

However, there is no Download VM Import Tool on the Library page. Instead, you can choose Library->Disk Images ->Import Disk Image in order to reach the import tool download page (or click this direct link).

After installing the GUI tool on Windows using the exe file, I am redirected to the browser login page of the tool:

If you are behind a proxy, you will receive the following connectivity problem error:

The link will lead here. The process is to create a config.properties file on a folder named .ravello in the user’s home directory (%HOME%\.ravello).

Note: be sure to use %HOME%\.ravello and not %USERPROFILE%\.ravello, if those two pathes differ in your case (in my case they do: %HOME% is my local Git directory on F:\veits\git).

The file config.properties needs to have following content:

[upload]
proxy_address = <ip address of proxy server>
proxy_port = <port on which the proxy server accepts connections>

The nasty thing is, that you need to kill the RavelloImageImportServer.exe task in case of Windows or the ravello-vm-upload process in case of Linux.

The problem is, that

  1. they do not tell you how to restart the process. In my case, I have found RavelloImageImportServer.exe on C:\Program Files (x86)\Ravello Systems\Ravello import utility. I have restarted it.
  2. Even though I have created the properties file, the import tool does not find the proxy configuration on %USERPROFILE%\.ravello. Crap! I have found out that the import tool is looking for %HOME%\.ravello instead, which has been set by my local git installation to be on F:\veits\git. I was close to giving up…

Finally, I have managed to upload the ISO:

From there, it should be possible to create an empty VM, attach the ISO on it and boot the VM from ISO…

No luck: after some time, the upload is stopped due to no apparent reason:

The pause button as well as the resume button are greyed out. No way to resume the upload. Well thought, but not so good implemented. Okay, the service is quite new. Let us see, how ravello works, if we give them a few additional months…

After connection to the Internet without HTTP proxy (my notebook was in standby for a while), I have seen, that I could not log into the local GUI upload tool anymore. The process was consuming a constant 25% of my dual core CPU. Workaround: renaming the config.properties file (or maybe remove/comment out its content), killing and restarting of the process brought back the GUI upload process to normal.

Summary

I have shortly investigated, which options I have to run a legacy system on an IaaS provider cloud network.

Before I found out that ravello’s service times are sub-optimal, I initially thought that the meta IaaS provider called Ravello Systems is the winner of this investigation:

However, I see following problems:

  • it has taken ravello more than two (!) months to provide me with an activation link.
  • An ISO or VM upload requires the installation of a local tool
  • the GUI tool has problems to handle HTTP proxies. I have followed their instructions, but I could not get it to work, initially. At the end, I have found out, that the tool is not looking in Maybe the tool is looking in %USERPROFILE%\.ravello, but in %HOME%\.ravello, which is a GIT home directory and does not match C:\Users\myusername in my case.
  • another problem might be that Ravello is running the VMware VMs on top of a Hypervisor layer, which in turn translates the VM to the underlying infrastructure. There is a high risk that this will work only for test labs with low CPU consumptions. This is to be tested.

In the short time I have invested into the investigation, I have found that

  1. ravello had seemed to be the best alternative, since the system can be installed in the cloud with
  2. A reader of my blog suggests to check out Vultr. Since ravello has its own drawbacks (service: long time to respond, longer time to help, GUI import tools seems to have weaknesses: I could not get it to work from behind a HTTP proxy, even if I follow the instructions), Vultr might be a real good alternative with low pricing.
  3. Amazon AWS is an alternative, if it is O.K. for you not to install from ISO, but to install locally and upload the created custom VM.

The following alternatives have major drawbacks:

  • Microsoft Azure requires the local installation of the VM using Hyper-V and I do not have such a system. I have not found a statement, whether it is possible to boot a Microsoft Azure VM from ISO (do you know that?).
  • DigitalOcean neither supports an installation from ISO, nor does it support the upload of custom VMs.

See also:

Next steps:

  • once the ISO is uploaded, create a VM and try to boot the VM from ISO.
  • Try out Vultr.

Update 2016-03-21: I have applied for a trial with ravello on March 17th, but no reaction so far, apart from the automatic email reply. I have opened a ticket yesterday and I got an email that they will come back to me…

Update 2016-03-23: still waiting…

Update 2016-03-30: instead of helping me, Ravello’s support has sent an email that they did not get any response from me (response about what?) and they have closed the ticket, along with a link with the possibility to give feedback. My feedback was “not satisfied”. Let us see, how they react.

Update 2016-05-11: I have received the activation link, more than 2 months after my application. I have signed in although I do not know, if I am still interested. I have added the HowTo chapter, but I have failed to upload the ISO via a HTTP proxy, even though I have followed the instructions closely.

Meanwhile, I have signed up for a native AWS account. The intent of this blog was to find a provider that makes it more easy to install an image from ISO: I did not want to install locally, and then upload and convert the image, because my SSD disk is notoriously full. Ravello was the only alternative I had found in a quick Internet research. However, Ravello had failed to provide me with a valid registration within 2 months.

14

IT Automation Part I: Ansible “Hello World” Example using Ansible on Docker

This is part I of a little “Hello World” example using Ansible, an IT automation tool.

The post has following content:

  • Popularity of Ansible, Salt, Chef and Puppet
  • Installation based on Docker
  • Playbooks (i.e. tasks) and Inventories (i.e. groups of targets)
  • Remote shell script execution

As a convenient and quick way of Ansible installation on Windows (or Mac), we are choosing a Docker Ansible image on an Vagrant Ubuntu Docker Host that has won a java performance competition against CoreOS and boot2docker (see this blog post).

NEW: Try it out!

Get a feeling for Ansible in a real console without the need to install anything!

  1. Quick sign-in to Katacoda via Github, LinkedIn, Twitter Google or email
  2. click the console below or on https://www.katacoda.com/oliverveits/scenarios/ansible-bootstrap

Posts of this series:

  • Part I: Ansible Hello World (this post) with a comparison of Ansible vs. Salt vs. Chef vs. Puppet and a hello world example with focus on Playbooks (i.e. tasks), Inventories (i.e. groups of targets) and remote shell script execution.
  • Part II: Ansible Hello World reloaded with focus on templating: create and upload files based on jinja2 templates.
  • Part III: Salt Hello World example: same content as part I, but with Salt instead of Ansible
  • Part IV: Ansible Tower Hello World: investigates Ansible Tower, a professional Web Portal for Ansible

 

Versions

2015-11-09: initial release
2016-06-08: I have added command line prompts basesystem#, dockerhost# and container#, so the reader can see more easily, on which layer the command is issued.
2017-01-06: Added linked Table of Contents

Table of Contents

Why Ansible?

For the “hello world” tests, I have chosen Ansible. Ansible is a relatively new member of a larger family of IT automation toolsThis InfoWorld article from 2013 compares four popular members, namely Puppet, Chef, Ansible and Salt. Here you and here find more recent comparisons, if not as comprehensive as the InfoWorld article.

In order to explore the popularity of the software, let us look at a google trend analysis of those four tools (puppet/ansible/chef/salt + “automation”) (status November 2015; for a discussion of the somewhat more confusing recent results, please consult Appendix F below):

Okay, in the google trends analysis we can see that Ansible is relatively new and that is does not seem to replace Puppet, Chef or Salt. However, Ansible offers a fully maintained, RESTful API on the Ansible Web application called Ansible Tower (which comes at a cost, though).  Moreover, I have seen another article stating that Ansible is the very popular among docker developers. Since I have learned to love docker (it was not love at first sight), let us dig into Ansible, even though Puppet and Chef seems to be more popular in google searches.

For a discussion of REST and Web UI capabilities of the four tools, see Appendix D.

Ansible “Hello World” Example – the Docker Way

We plan to install Ansible, prepare Linux and Windows targets and perform simple tests like follows:

  1. Install a Docker Host
  2. Create an Ansible Docker Image
    • Download an Ansible Onbuild Image from Docker Hub
    • Start and configure the Ansible Container
    • Locally test the Installation
  3. Remote Access to a Linux System via SSH
    • –> Create a key pair, prepare the Linux target, access the Linux target
    • Note: we will use the Docker Host as Linux Target
  4. Working with Playbooks
    • Ansible “ping” to single system specified on command line
    • Run a shell script on single system specified on command line
  5.  Working with Inventory Files
    • Ansible “ping” to inventory items
    • Run a shell script on inventory items

1. Install a Docker Host

Are you new to Docker? Then you might want to read this blog post.

Installing Docker on Windows and Mac can be a real challenge, but no worries: we will show an easy way here, that is much quicker than the one described in Docker’s official documentation:

Prerequisites:
  • I recommend to have direct access to the Internet: via Firewall, but without HTTP proxy. However, if you cannot get rid of your HTTP proxy, read Appendix B.
  • Administration rights on you computer.
Steps to install a Docker Host VirtualBox VM:

Download and install Virtualbox (if the installation fails with error message “Setup Wizard ended prematurely” see Appendix A: Virtualbox Installation Workaround below)

1. Download and Install Vagrant (requires a reboot)

2. Download Vagrant Box containing an Ubuntu-based Docker Host and create a VirtualBox VM like follows (assumed a Linux-like system or bash on Windows):

(basesystem)$ mkdir ubuntu-trusty64-docker ; cd ubuntu-trusty64-docker
(basesystem)$ vagrant init williamyeh/ubuntu-trusty64-docker
(basesystem)$ vagrant up
(basesystem)$ vagrant ssh

Now you are logged into the Docker host and we are ready for the next step: to create the Ansible Docker image.

Note: I have experienced problems with the vi editor when running vagrant ssh in a Windows terminal. In case of Windows, consider to follow Appendix C and to use putty instead.

2. Create an Ansible Docker Image

1. Download an Ansible Onbuild Image from Docker Hub

In order to check that the docker host has Internet access, we issue following command:

dockerhost# docker search ansible

This command will lead to an error, if the you work behind a HTTP proxy, since we have not (yet) configured the docker host for usage behind a HTTP proxy. I recommend to get direct Internet access without HTTP proxy for now. However, if you cannot get rid of your HTTP proxy, read Appendix B.

Now we download the ansible image:

dockerhost# docker pull williamyeh/ansible:ubuntu14.04-onbuild

2. Start and configure the Ansible Container

dockerhost# docker run -it williamyeh/ansible:ubuntu14.04-onbuild /bin/bash

The -it (interactive terminal) flags starts an interactive session in the container. Now you are logged into the Docker Container and you can prepare the Ansible configuration files, namely the inventory file (hosts file) and the playbook.yml:

3. Locally test the Installation

container# ansible all -i 'localhost,' -u vagrant -c local -m ping

Note that the -i is a CSV list of target hosts and needs to end with a comma. With -u, we define the remote user, and the private key file needs to be specified. The response should look like follows:

localhost | success >> {
 "changed": false,
 "ping": "pong"
}

3. Remote Access to a Linux System via SSH

Now let us test the remote access to a Linux system.

We could perform our tests with any target system with a running SSH service and installed Python >v2.0 on /usr/bin/python (see the FAQs). However, the Ubuntu Docker host is up and running already, so why not use it as target system? In this case, the tested architecture looks like follows:

Ansible is agent-less, but we still need to prepare the target system: Ansible’s default remote access method is to use SSH with public key authentication. The best way is to create an RSA key pair on the Ansible machine (if not already available) and to add the corresponding public key as “authorized key” on the target system.

1. Create an RSA key pair on the Ansible container:
container# ssh-keygen -t rsa

and go through the list of question. For a proof of concept, and if you are not concerned about security, you can just hit <enter> several times. Here is a log from my case:

root@930360e7db68:/etc/ssh# ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
fe:eb:bf:de:30:d6:5a:8c:1b:4a:c0:dd:cb:5f:3b:80 root@930360e7db68
...

On the target machine (i.e. the ubuntu-trusty64-docker system), create a file name /tmp/ansible_id_rsa.pub and copy the content of ansible’s ~/.ssh/id_rsa.pub file into that file. Then:

dockerhost# cat /tmp/ansible_id_rsa.pub >> ~/.ssh/authorized_keys
2. test remote access via SSH:

Now, we should be able to access the target system from the ansible container using this key. To test, try:

container# ssh vagrant@192.168.33.10 -C "echo hello world"

This should echo a “hello world” to your screen. Here 192.168.33.10 is a reachable IP address of the docker host (issue ifconfig on the docker host to check the IP address in your case).  For troubleshooting, you can call ssh with the -vvv option.

3. Remote access via Ansible:

Check that Python version > 2.0 (better: > 2.4) is installed on your target machine. In our case of the ubuntu-trusty64-docker image, this is a pre-installed package and we get:

dockerhost# python --version
Python 2.7.6

Now also a remote Ansible connection should be possible from the container:

container# ansible all -i '192.168.33.10,' -u vagrant -m ping

which results in following output:

192.168.33.10 | success >> {
 "changed": false,
 "ping": "pong"
}

This was your first successful Ansible connection via SSH. Now let us also perform a change on the remote target. For that, we perform a remote shell command:

container# ansible all -i '192.168.33.10,' -u vagrant -m shell -a "echo hello world\! > hello_world"

This time the module is a “shell” and the module’s argument is a echo hello world command. We should get the feedback

192.168.33.10 | success | rc=0 >>

On the target, we can check the result with:

dockerhost# cat hello_world
hello world!

4. Working with Playbooks

This was your first remote shell action via Ansible. Now we want to have a look to playbooks, which are the Ansible way to document and automate the tasks.

1. Create a playbook

On the ansible container terminal, we create a playbook.yml file:

container# vi playbook.yml

and we add and save the following content:

---
# This playbook uses the ping module to test connectivity to Linux hosts
- name: Ping
  hosts: 192.168.33.10
  remote_user: vagrant 

  tasks: 
  - name: ping 
    ping: 
  - name: echo 
    shell: echo Hello World! > hello_world

Note: If you have problems with formatting of the characters in the terminal (I have experienced problems in a Windows terminal), then I recommend to use a putty terminal instead of using vagrant ssh. For that, see Appendix C.

Note also that the number of white spaces are relevant in a yml file. However, note that the ‘!’ does not need to be escaped in the playbook (it was necessare on the command line, though). Now, we perform following command on the Ansible container:

container# ansible-playbook -i '192.168.33.10,' playbook.yml

This time, the -u flag is not needed (it is ignored, if specified), since we have specified the user in the playbook. We get the following feedback:

PLAY [Ping] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.33.10]

TASK: [ping] ******************************************************************
ok: [192.168.33.10]

TASK: [echo] ******************************************************************
changed: [192.168.33.10]

PLAY RECAP ********************************************************************
192.168.33.10 : ok=3 changed=1 unreachable=0 failed=0

We check the result on the target machine again:

dockerhost# cat hello_world
Hello World!

We see that the file hello_world was overwritten (“Hello World!” with capital letters instead of “hello world!”).

5. Working with Inventory Files

Instead of specifying singular hosts in the playbook.yml, Ansible offers the more elegant way to work with groups of machines. Those are defined in the inventory file.

More information about inventory files can be found the official documentation. However, note that this page describes new 2.0 features that do not work on the Docker image (currently ansible 1.9.4). See Appendix E for more details on the problem and for information on how to upgrade to the latest development build. The features tested in this blog post work on ansible 1.9.4, though; so you do not need to upgrade now.

1. Now we do the same as before, but we use an inventory file to define the target IP addresses:

container# vi /etc/ansible/hosts

and add the following lines:

[vagranthosts]
192.168.33.10

In the playbook.yml we replace 192.168.33.10 by a group name, e.g. vagranthosts

---
# This playbook uses the win_ping module to test connectivity to Windows hosts
- name: Ping
  hosts: vagranthosts
  remote_user: vagranthost

  tasks:
  - name: ping
    ping:
  - name: echo
    shell: echo HELLO WORLD! > hello_world

In order to see the difference, we also have changed the hello world to all capital letters.

Now we perform:

container# ansible-playbook -i /etc/ansible/hosts playbook.yml

Here, we have replaced the list of hosts by a reference to the inventory file.

The output looks like follows:

PLAY [Ping] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.33.10]

TASK: [ping] ******************************************************************
ok: [192.168.33.10]

TASK: [echo] ******************************************************************
changed: [192.168.33.10]

PLAY RECAP ********************************************************************
192.168.33.10 : ok=3 changed=1 unreachable=0 failed=0

We check the result on the target machine again:

dockerhost# cat hello_world
HELLO WORLD!

We see that the file hello_world was overwritten again (all capital letters). Success!

Summary

We have shown, how you can download and run Ansible as a docker container on any machine (Windows in my case). We have prepared the Ansible container and the target for SSH connections and have shown, how to perform connectivity tests and shell scripts on the remote system. In addition, we have introduced Playbooks as a means to document and run several tasks by one command. Moreover, Inventory files were introduced in order to manage groups of target machines.

Next steps

Following topics are looked at in the next two parts of this series:

  • Part II: Ansible Hello World reloaded will show
    • how to upload files with Ansible
    • how to create dynamic file content with Ansible using jinja2 templates
    • bind it all together by showing a common use case with dynamic shell scripts and data files
  • Part III: Salt Hello World example: same content as part I, but with Salt instead of Ansible
  • Part IV: Ansible Tower Hello World: investigates Ansible Tower, a professional Web Portal for Ansible

Open: Window support of Ansible

Appendix A: Virtualbox Installation (Problems)

  • Download the installer. Easy.
  • When I start the installer, everything seems to be on track until I see “rolling back action” and I finally get this:
    “Oracle VM Virtualbox x.x.x Setup Wizard ended prematurely”

Resolution of the “Setup Wizard ended prematurely” Problem

Let us try to resolve the problem: the installer of Virtualbox downloaded from Oracle shows the exact same error: “…ended prematurely”. This is not a docker bug. Playing with conversion tools from Virtualbox to VMware did not lead to the desired results.

The Solution: Google is your friend: the winner is:https://forums.virtualbox.org/viewtopic.php?f=6&t=61785. After backing up the registry and changing the registry entry

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Network -> MaxFilters from 8 to 20 (decimal)

and a reboot of the Laptop, the installation of Virtualbox is successful.

Note: while this workaround has worked on my Windows 7 notebook, it has not worked on my new Windows 10 machine. However, I have managed to install VirtualBox on Windows 10 by de-selecting the USB support module during the VirtualBox installation process. I remember having seen a forum post pointing to that workaround, with the additional information that the USB drivers were installed automatically at the first time a USB device was added to a host (not yet tested on my side).

Appendix B: HTTP Proxy Configuration

If you need to work behind a HTTP proxy, you need to consider several levels that need to know of it:

  • the physical host for both, your browser as well as your terminal session (http_proxy and https_proxy variables) for successful vagrant init commands and download of the vagrant boxes.
  • the docker host (if it differs from the physical host) for both, in the docker configuration files as well as on the bash. Note that the configuration files differ between CoreOS, boot2docker and Ubuntu.
  • the docker client for the terminal session; needed for apt-get update+install.

Ubuntu Docker:

sudo vi /etc/default/docker

add proxy, if needed like follows (adapt the names and ports, so it fits to your environment):

export http_proxy='http://proxy.example.com:8080'
export https_proxy='http://proxy.example.com:8080'

then:

sudo restart docker

CoreOS:

sudo mkdir /etc/systemd/system/docker.service.d
sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf

and add something like (adapt the names and ports, so it fits to your environment):

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080"

Then:

sudo reboot

or try:

sudo sysctl docker restart

Appendix C: How to use putty for accessing Vagrant Boxes

What is to be done:

  1. locate and convert the vagrant private key to ppk format using puTTYgen.
    1. locate the vagrant private key of the box. In my case, this is C:\Users\vo062111\.vagrant.d\boxes\williamyeh-VAGRANTSLASH-ubuntu-trusty64-docker\1.8.1\virtualbox\vagrant_private_key
    2. start puTTYgen -> Conversions -> import -> select the above path.
    3. press “Save private key” and save vagrant_private_key as vagrant_private_key.ppk
  2. In putty,
    1. create a connection to vagrant@127.0.0.1 port 2201 or port 2222 (the port vagrant uses is show in the terminal during “vagrant up”)
    2. specify the ppk key Connection->SSH->Auth->(om the right)Private key file for authentication
    3. Click on Session on the left menu and press Save
    4. Press Open and accept the RSA fingerprint -> you should be able to log in without password prompt. If there is still a password prompt, there is something wrong with the private key.

Appendix D: REST APIs of Ansible, Puppet, Chef and Salt

Here, I have made a quick research on the RESTful interfaces and Web UI Interfaces of Ansible, Puppet, Chef and Salt. I have not found this information on the  Feature comparison table on Wikipedia:

Appendix E: Install the latest Ansible Development Version

The Ansible version in the docker image has the problem that it has a version 1.9.4 (currently), but the Ansible documentation is describing the latest v2.0 features. E.g. in version 1.9.4, variables in the inventory file described in the documentation are ignored (see e.g. the example “jumper ansible_port=5555 ansible_host=192.168.1.50″) and this leads to a “Could not resolve hostname” error ; see also this stackoverflow post).

Here, we will show, how to install the latest Ansible version in the container. For that, run the container:

docker -it williamyeh/ansible:ubuntu14.04-onbuild /bin/bash

ansible --version

will result in an output similar to:

ansible 1.9.4

If you get a version >=2.0, you might not need to upgrade at all. In all other cased, perform the following steps:

If you are behind a proxy, perform sth. like:

export http_proxy=http://proxy.example.com:8080
export https_proxy=http://proxy.example.com:8080
apt-get update; apt-get install git
git clone git://github.com/ansible/ansible.git --recursive
cd ./ansible
source ./hacking/env-setup
ansible --version

should give you some output like:

ansible 2.0.0 (devel 9b9fb51d9d) last updated ...

Now also the v2.0 features should be available. If you want to update the version in future, you will need to perform the git command

git pull

In the /ansible directory.

(chapter added on 2016-04-11)

In order to explore the popularity of the software, we have looked at a google trend analysis of those four tools (puppet/ansible/chef/salt + “automation”) (status November 2015):

Note that the google trends result looks quite differently 5 months later (2016-04-11). Google seems to have changed their source data and/or algorithm. With the same search terms as we had used in Nov. 2015 (puppet/ansible/chef/salt + “automation”; note that the link works only, if you are logged into your google account), in April 2016 we got following non-convincing graph:

Especially the analysis of Salt’s and Chef’s popularity for 2011 and before does not look very convincing.

If we are searching for “Software” instead via this google trends link (works only, if you are logged into your google account), we get something like the following:

Also this data does not look reliable: according to Wikipedia’s Vagrant page, Vagrant’s initial version was March 2010. Why do we see so many search hits before that time? That is not plausible. The same with Puppet, which has started 2005 and has many hits on 2004.

To be honest, google trends analysis used to (at least) look reliable in November 2015, but it does not look reliable anymore. What a pity: I used to work a lot with google trends in the past for finding out, which technology is trending, but looking at the more recent results, I have lost the confidence that I can rely on the data. If you know an alternative to google trends, please add a comment to this blog post.

In any case; for the time after 2013, it looks like the popularity of Ansible is rising quickly (if we believe it).

0

Getting Started with Ansible

This is part 1 of a little “Hello World” example using Ansible, an IT automation tool.

The post has following content:

  • Popularity of Ansible, Salt, Chef and Puppet
  • Installation based on Docker
  • Playbooks (i.e. tasks) and Inventories (i.e. groups of targets)
  • Remote shell script execution

As a convenient and quick way of Ansible installation on Windows (or Mac), we are choosing a Docker Ansible image on an Vagrant Ubuntu Docker Host that has won a java performance competition against CoreOS and boot2docker (see this blog post).

Posts of this series:

  • Part 1: Ansible Hello World (this post) with a comparison of Ansible vs. Salt vs. Chef vs. Puppet and a hello world example with focus on Playbooks (i.e. tasks), Inventories (i.e. groups of targets) and remote shell script execution.
  • Part 2: Ansible Hello World reloaded with focus on templating: create and upload files based on jinja2 templates.
  • Part 3: Salt Hello World example: same content as part I, but with Salt instead of Ansible
  • Part 4: Ansible Tower Hello World: investigates Ansible Tower, a professional Web Portal for Ansible

 

Versions

2015-11-09: initial release
2016-06-08: I have added command line prompts basesystem#, dockerhost# and container#, so the reader can see more easily, on which layer the command is issued.
2017-01-06: Added linked Table of Contents

Table of Contents

Why Ansible?

For the “hello world” tests, I have chosen Ansible. Ansible is a relatively new member of a larger family of IT automation toolsThis InfoWorld article from 2013 compares four popular members, namely Puppet, Chef, Ansible and Salt. Here you and here find more recent comparisons, if not as comprehensive as the InfoWorld article.

In order to explore the popularity of the software, let us look at a google trend analysis of those four tools (puppet/ansible/chef/salt + “automation”) (status November 2015; for a discussion of the somewhat more confusing recent results, please consult Appendix F below):

Okay, in the google trends analysis we can see that Ansible is relatively new and that is does not seem to replace Puppet, Chef or Salt. However, Ansible offers a fully maintained, RESTful API on the Ansible Web application called Ansible Tower (which comes at a cost, though).  Moreover, I have seen another article stating that Ansible is the very popular among docker developers. Since I have learned to love docker (it was not love at first sight), let us dig into Ansible, even though Puppet and Chef seems to be more popular in google searches.

For a discussion of REST and Web UI capabilities of the four tools, see Appendix D.

Ansible “Hello World” Example – the Docker Way

We plan to install Ansible, prepare Linux and Windows targets and perform simple tests like follows:

  1. Install a Docker Host
  2. Create an Ansible Docker Image
    • Download an Ansible Onbuild Image from Docker Hub
    • Start and configure the Ansible Container
    • Locally test the Installation
  3. Remote Access to a Linux System via SSH
    • –> Create a key pair, prepare the Linux target, access the Linux target
    • Note: we will use the Docker Host as Linux Target
  4. Working with Playbooks
    • Ansible “ping” to single system specified on command line
    • Run a shell script on single system specified on command line
  5.  Working with Inventory Files
    • Ansible “ping” to inventory items
    • Run a shell script on inventory items

1. Install a Docker Host

Are you new to Docker? Then you might want to read this blog post.

Installing Docker on Windows and Mac can be a real challenge, but no worries: we will show an easy way here, that is much quicker than the one described in Docker’s official documentation:

Prerequisites:
  • I recommend to have direct access to the Internet: via Firewall, but without HTTP proxy. However, if you cannot get rid of your HTTP proxy, read Appendix B.
  • Administration rights on you computer.
Steps to install a Docker Host VirtualBox VM:

Download and install Virtualbox (if the installation fails with error message “Setup Wizard ended prematurely” see Appendix A: Virtualbox Installation Workaround below)

1. Download and Install Vagrant (requires a reboot)

2. Download Vagrant Box containing an Ubuntu-based Docker Host and create a VirtualBox VM like follows (assumed a Linux-like system or bash on Windows):

(basesystem)$ mkdir ubuntu-trusty64-docker ; cd ubuntu-trusty64-docker
(basesystem)$ vagrant init williamyeh/ubuntu-trusty64-docker
(basesystem)$ vagrant up
(basesystem)$ vagrant ssh

Now you are logged into the Docker host and we are ready for the next step: to create the Ansible Docker image.

Note: I have experienced problems with the vi editor when running vagrant ssh in a Windows terminal. In case of Windows, consider to follow Appendix C and to use putty instead.

2. Create an Ansible Docker Image

1. Download an Ansible Onbuild Image from Docker Hub

In order to check that the docker host has Internet access, we issue following command:

dockerhost# docker search ansible

This command will lead to an error, if the you work behind a HTTP proxy, since we have not (yet) configured the docker host for usage behind a HTTP proxy. I recommend to get direct Internet access without HTTP proxy for now. However, if you cannot get rid of your HTTP proxy, read Appendix B.

Now we download the ansible image:

dockerhost# docker pull williamyeh/ansible:ubuntu14.04-onbuild

2. Start and configure the Ansible Container

dockerhost# docker run -it williamyeh/ansible:ubuntu14.04-onbuild /bin/bash

The -it (interactive terminal) flags starts an interactive session in the container. Now you are logged into the Docker Container and you can prepare the Ansible configuration files, namely the inventory file (hosts file) and the playbook.yml:

3. Locally test the Installation

container# ansible all -i 'localhost,' -u vagrant -c local -m ping

Note that the -i is a CSV list of target hosts and needs to end with a comma. With -u, we define the remote user, and the private key file needs to be specified. The response should look like follows:

localhost | success >> {
 "changed": false,
 "ping": "pong"
}

3. Remote Access to a Linux System via SSH

Now let us test the remote access to a Linux system.

We could perform our tests with any target system with a running SSH service and installed Python >v2.0 on /usr/bin/python (see the FAQs). However, the Ubuntu Docker host is up and running already, so why not use it as target system? In this case, the tested architecture looks like follows:

Ansible is agent-less, but we still need to prepare the target system: Ansible’s default remote access method is to use SSH with public key authentication. The best way is to create an RSA key pair on the Ansible machine (if not already available) and to add the corresponding public key as “authorized key” on the target system.

1. Create an RSA key pair on the Ansible container:
container# ssh-keygen -t rsa

and go through the list of question. For a proof of concept, and if you are not concerned about security, you can just hit <enter> several times. Here is a log from my case:

root@930360e7db68:/etc/ssh# ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
fe:eb:bf:de:30:d6:5a:8c:1b:4a:c0:dd:cb:5f:3b:80 root@930360e7db68
...

On the target machine (i.e. the ubuntu-trusty64-docker system), create a file name /tmp/ansible_id_rsa.pub and copy the content of ansible’s ~/.ssh/id_rsa.pub file into that file. Then:

dockerhost# cat /tmp/ansible_id_rsa.pub >> ~/.ssh/authorized_keys
2. test remote access via SSH:

Now, we should be able to access the target system from the ansible container using this key. To test, try:

container# ssh vagrant@192.168.33.10 -C "echo hello world"

This should echo a “hello world” to your screen. Here 192.168.33.10 is a reachable IP address of the docker host (issue ifconfig on the docker host to check the IP address in your case).  For troubleshooting, you can call ssh with the -vvv option.

3. Remote access via Ansible:

Check that Python version > 2.0 (better: > 2.4) is installed on your target machine. In our case of the ubuntu-trusty64-docker image, this is a pre-installed package and we get:

dockerhost# python --version
Python 2.7.6

Now also a remote Ansible connection should be possible from the container:

container# ansible all -i '192.168.33.10,' -u vagrant -m ping

which results in following output:

192.168.33.10 | success >> {
 "changed": false,
 "ping": "pong"
}

This was your first successful Ansible connection via SSH. Now let us also perform a change on the remote target. For that, we perform a remote shell command:

container# ansible all -i '192.168.33.10,' -u vagrant -m shell -a "echo hello world\! > hello_world"

This time the module is a “shell” and the module’s argument is a echo hello world command. We should get the feedback

192.168.33.10 | success | rc=0 >>

On the target, we can check the result with:

dockerhost# cat hello_world
hello world!

4. Working with Playbooks

This was your first remote shell action via Ansible. Now we want to have a look to playbooks, which are the Ansible way to document and automate the tasks.

1. Create a playbook

On the ansible container terminal, we create a playbook.yml file:

container# vi playbook.yml

and we add and save the following content:

---
# This playbook uses the ping module to test connectivity to Linux hosts
- name: Ping
  hosts: 192.168.33.10
  remote_user: vagrant 

  tasks: 
  - name: ping 
    ping: 
  - name: echo 
    shell: echo Hello World! > hello_world

Note: If you have problems with formatting of the characters in the terminal (I have experienced problems in a Windows terminal), then I recommend to use a putty terminal instead of using vagrant ssh. For that, see Appendix C.

Note also that the number of white spaces are relevant in a yml file. However, note that the ‘!’ does not need to be escaped in the playbook (it was necessare on the command line, though). Now, we perform following command on the Ansible container:

container# ansible-playbook -i '192.168.33.10,' playbook.yml

This time, the -u flag is not needed (it is ignored, if specified), since we have specified the user in the playbook. We get the following feedback:

PLAY [Ping] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.33.10]

TASK: [ping] ******************************************************************
ok: [192.168.33.10]

TASK: [echo] ******************************************************************
changed: [192.168.33.10]

PLAY RECAP ********************************************************************
192.168.33.10 : ok=3 changed=1 unreachable=0 failed=0

We check the result on the target machine again:

dockerhost# cat hello_world
Hello World!

We see that the file hello_world was overwritten (“Hello World!” with capital letters instead of “hello world!”).

5. Working with Inventory Files

Instead of specifying singular hosts in the playbook.yml, Ansible offers the more elegant way to work with groups of machines. Those are defined in the inventory file.

More information about inventory files can be found the official documentation. However, note that this page describes new 2.0 features that do not work on the Docker image (currently ansible 1.9.4). See Appendix E for more details on the problem and for information on how to upgrade to the latest development build. The features tested in this blog post work on ansible 1.9.4, though; so you do not need to upgrade now.

1. Now we do the same as before, but we use an inventory file to define the target IP addresses:

container# vi /etc/ansible/hosts

and add the following lines:

[vagranthosts]
192.168.33.10

In the playbook.yml we replace 192.168.33.10 by a group name, e.g. vagranthosts

---
# This playbook uses the win_ping module to test connectivity to Windows hosts
- name: Ping
  hosts: vagranthosts
  remote_user: vagranthost

  tasks:
  - name: ping
    ping:
  - name: echo
    shell: echo HELLO WORLD! > hello_world

In order to see the difference, we also have changed the hello world to all capital letters.

Now we perform:

container# ansible-playbook -i /etc/ansible/hosts playbook.yml

Here, we have replaced the list of hosts by a reference to the inventory file.

The output looks like follows:

PLAY [Ping] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.33.10]

TASK: [ping] ******************************************************************
ok: [192.168.33.10]

TASK: [echo] ******************************************************************
changed: [192.168.33.10]

PLAY RECAP ********************************************************************
192.168.33.10 : ok=3 changed=1 unreachable=0 failed=0

We check the result on the target machine again:

dockerhost# cat hello_world
HELLO WORLD!

We see that the file hello_world was overwritten again (all capital letters). Success!

Summary

We have shown, how you can download and run Ansible as a docker container on any machine (Windows in my case). We have prepared the Ansible container and the target for SSH connections and have shown, how to perform connectivity tests and shell scripts on the remote system. In addition, we have introduced Playbooks as a means to document and run several tasks by one command. Moreover, Inventory files were introduced in order to manage groups of target machines.

Next steps

Following topics are looked at in the next two parts of this series:

  • Part II: Ansible Hello World reloaded will show
    • how to upload files with Ansible
    • how to create dynamic file content with Ansible using jinja2 templates
    • bind it all together by showing a common use case with dynamic shell scripts and data files
  • Part III: Salt Hello World example: same content as part I, but with Salt instead of Ansible
  • Part IV: Ansible Tower Hello World: investigates Ansible Tower, a professional Web Portal for Ansible

Open: Window support of Ansible

Appendix A: Virtualbox Installation (Problems)

  • Download the installer. Easy.
  • When I start the installer, everything seems to be on track until I see “rolling back action” and I finally get this:
    “Oracle VM Virtualbox x.x.x Setup Wizard ended prematurely”

Resolution of the “Setup Wizard ended prematurely” Problem

Let us try to resolve the problem: the installer of Virtualbox downloaded from Oracle shows the exact same error: “…ended prematurely”. This is not a docker bug. Playing with conversion tools from Virtualbox to VMware did not lead to the desired results.

The Solution: Google is your friend: the winner is:https://forums.virtualbox.org/viewtopic.php?f=6&t=61785. After backing up the registry and changing the registry entry

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Network -> MaxFilters from 8 to 20 (decimal)

and a reboot of the Laptop, the installation of Virtualbox is successful.

Note: while this workaround has worked on my Windows 7 notebook, it has not worked on my new Windows 10 machine. However, I have managed to install VirtualBox on Windows 10 by de-selecting the USB support module during the VirtualBox installation process. I remember having seen a forum post pointing to that workaround, with the additional information that the USB drivers were installed automatically at the first time a USB device was added to a host (not yet tested on my side).

Appendix B: HTTP Proxy Configuration

If you need to work behind a HTTP proxy, you need to consider several levels that need to know of it:

  • the physical host for both, your browser as well as your terminal session (http_proxy and https_proxy variables) for successful vagrant init commands and download of the vagrant boxes.
  • the docker host (if it differs from the physical host) for both, in the docker configuration files as well as on the bash. Note that the configuration files differ between CoreOS, boot2docker and Ubuntu.
  • the docker client for the terminal session; needed for apt-get update+install.

Ubuntu Docker:

sudo vi /etc/default/docker

add proxy, if needed like follows (adapt the names and ports, so it fits to your environment):

export http_proxy='http://proxy.example.com:8080'
export https_proxy='http://proxy.example.com:8080'

then:

sudo restart docker

CoreOS:

sudo mkdir /etc/systemd/system/docker.service.d
sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf

and add something like (adapt the names and ports, so it fits to your environment):

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080"

Then:

sudo reboot

or try:

sudo sysctl docker restart

Appendix C: How to use putty for accessing Vagrant Boxes

What is to be done:

  1. locate and convert the vagrant private key to ppk format using puTTYgen.
    1. locate the vagrant private key of the box. In my case, this is C:\Users\vo062111\.vagrant.d\boxes\williamyeh-VAGRANTSLASH-ubuntu-trusty64-docker\1.8.1\virtualbox\vagrant_private_key
    2. start puTTYgen -> Conversions -> import -> select the above path.
    3. press “Save private key” and save vagrant_private_key as vagrant_private_key.ppk
  2. In putty,
    1. create a connection to vagrant@127.0.0.1 port 2201 or port 2222 (the port vagrant uses is show in the terminal during “vagrant up”)
    2. specify the ppk key Connection->SSH->Auth->(om the right)Private key file for authentication
    3. Click on Session on the left menu and press Save
    4. Press Open and accept the RSA fingerprint -> you should be able to log in without password prompt. If there is still a password prompt, there is something wrong with the private key.

Appendix D: REST APIs of Ansible, Puppet, Chef and Salt

Here, I have made a quick research on the RESTful interfaces and Web UI Interfaces of Ansible, Puppet, Chef and Salt. I have not found this information on the  Feature comparison table on Wikipedia:

Appendix E: Install the latest Ansible Development Version

The Ansible version in the docker image has the problem that it has a version 1.9.4 (currently), but the Ansible documentation is describing the latest v2.0 features. E.g. in version 1.9.4, variables in the inventory file described in the documentation are ignored (see e.g. the example “jumper ansible_port=5555 ansible_host=192.168.1.50″) and this leads to a “Could not resolve hostname” error ; see also this stackoverflow post).

Here, we will show, how to install the latest Ansible version in the container. For that, run the container:

docker -it williamyeh/ansible:ubuntu14.04-onbuild /bin/bash

ansible --version

will result in an output similar to:

ansible 1.9.4

If you get a version >=2.0, you might not need to upgrade at all. In all other cased, perform the following steps:

If you are behind a proxy, perform sth. like:

export http_proxy=http://proxy.example.com:8080
export https_proxy=http://proxy.example.com:8080
apt-get update; apt-get install git
git clone git://github.com/ansible/ansible.git --recursive
cd ./ansible
source ./hacking/env-setup
ansible --version

should give you some output like:

ansible 2.0.0 (devel 9b9fb51d9d) last updated ...

Now also the v2.0 features should be available. If you want to update the version in future, you will need to perform the git command

git pull

In the /ansible directory.

(chapter added on 2016-04-11)

In order to explore the popularity of the software, we have looked at a google trend analysis of those four tools (puppet/ansible/chef/salt + “automation”) (status November 2015):

Note that the google trends result looks quite differently 5 months later (2016-04-11). Google seems to have changed their source data and/or algorithm. With the same search terms as we had used in Nov. 2015 (puppet/ansible/chef/salt + “automation”; note that the link works only, if you are logged into your google account), in April 2016 we got following non-convincing graph:

Especially the analysis of Salt’s and Chef’s popularity for 2011 and before does not look very convincing.

If we are searching for “Software” instead via this google trends link (works only, if you are logged into your google account), we get something like the following:

Also this data does not look reliable: according to Wikipedia’s Vagrant page, Vagrant’s initial version was March 2010. Why do we see so many search hits before that time? That is not plausible. The same with Puppet, which has started 2005 and has many hits on 2004.

To be honest, google trends analysis used to (at least) look reliable in November 2015, but it does not look reliable anymore. What a pity: I used to work a lot with google trends in the past for finding out, which technology is trending, but looking at the more recent results, I have lost the confidence that I can rely on the data. If you know an alternative to google trends, please add a comment to this blog post.

In any case; for the time after 2013, it looks like the popularity of Ansible is rising quickly (if we believe it).