Category Archives: Infrastructure

vCloud Air: OnDemand IaaS by VMware

Introduction

Cloud computing is currently a highly discussed topic in many IT strategy meetings. The ability to spin workloads in third party cloud and pay only for what you actually use certainly make sense from financial point of view. There are many great use cases for that.

Imagine you have a big event, something like Super Bowl and your current infrastructure would likely struggle to handle such load for the duration of the event. You have two options how to prepare for it, you can buy new assets – compute, storage and network spending a lot of dollars or you can build your application stacks in third party cloud and leave it running for certain period of time and then tear down when the event is over. The second option also gives you virtually unlimited resources if you need to use them. How cool is that?

There are many Infrastructure as a Service offerings in the market, one of the most popular include Amazon AWS, Microsoft Azure or Google Cloud Compute. It was only matter of time that other vendors would start offer the same thing to address the competition.

One of such vendors is VMware. Initially, the popular hypervisor vendor offered two flavors of their public cloud offering named vCloud Air. First was a Dedicated Cloud, targeted on customers who wanted to essentially lease the entire servers for better physical isolation and security. The second was a Virtual Private Cloud, which is cheaper, but the hypervisor would run other companies’ VMs as well and provide logical isolation.

In either case, both of the offering were targeted on larger customers and you and me did not have an easy option to just spin couple of VMs for testing without committing to some long term payment plant.

With the new OnDemand access flavor that model changes and you now have a true option to pay as you go. This flavor allows any individual from flesh and bone (with credit card) to spin a workload in VMware’s data center and pay only for what he uses.

safe_image.php

vCloud Air Common Use Cases

To support this idea, they even get you a nice started pack of $300 dollars worth of resources. It is like with gas provider, where they would get you certain number of gas for free to try their service out. Good move to attrack people like me to poke around and spin some workloads.

I think that many enterprise companies will consider this option, as their infrastructure teams are familiar with vSphere and this cloud is built upon the same foundations as their data centers, making migration and interoperability a little bit easier for both sides.

How does it feel to spin a couple of VMs in this environment? Lets have a look.

Signing Up

First, you need to go to super secret URL only know by 5 individuals in the world. I like my audience so I am going to share that URL with you and it is at http://vcloud.vmware.com.

The landing pages gives you couple of information what vCloud is, couple of testimonial and general information. Click on Service Offering/Virtual Private Cloud OnDemand to get started.

Lets take advantage of that $300 voucher shall we? For that we need a VMware Account First. You you have one log in now, otherwise make the clickie-clickie action and create one.

During the registration you need to enter a valid credit card and billing information.

vcloud-air-1

Creating a new account for vCloud Air

At this point you will have an options to select the Support Level Plan. I’ll go with the basic one OnDemand Online Support. Make sure that the promo code ondemand2015 has been applied. As so many people are trying to get the hands on, you likely end up in the queue with Sing up request pending. So wait for it.

Ok, it took around 5 minutes to get the confirmation email with the login page URL and initial links to set the password.

vcloud-air-2

You need to reset your temporary password

Entering to the vCloud

Now that you we have set the credentials, lets go back to to main login page at https://vca.vmware.com. If you would like to use VMware Remote Console, install it & enable the plugin in your browser.

vcloud-air-3

The Login Screen looks nice and clean, thumbs up for that.

Building our first vPC

After login you are presented by main dashboard, that includes Services, Subscriptions and Tools. After Clicking the Virtual Private Cloud Ondemand, you have an option to select in which physical location you want to build your vPC. Since I am located in UK I’ll go with UK Slough 1 6 option. Slough is small town in Berkshire.

vcloud-air-4

After couple of seconds a new vPC instance will be build. After the gears stop spinning you will receive the following handy infrastructure to play with.

Default vPC topology

Default vPC topology

The infrastructure is composed of one Gateway, that provides access to your private cell. This gateway is connected to public segment which is according Ripe a chunk of larger pool that VMware allocated for this particular cloud.

The second network, is a private segment where would build your VM. We will get to that later. I want to show you around the interface.

The first tab you find under vPC is the Resource Usage which shows you how much you resources you consume and how much for last hour/day or month. You also have option to view detailed report. Give that your financial department to sponsor your vCloud adventures.

vcloud-air-6

Creating new virtual machine

Next, looking at Virtual Machines tab, you have an option to spin your first workload or migrate it from your private data center. Lets keep things simple for now and select the first option.

Creating the first workload

Creating the first workload

As any good cloud offering, you will be presented by a catalog of virtual machines that you can select from. Most of the Linux flavors are free, and you pay same extra fees for Windows VMs for licensing. You also have an option to create an empty machine, called shell VM from scratch. I’ll go with CentOS 6.4 32 Bit for now.

Selecting a VM from catalog

Selecting a VM from catalog

In customization page, you have the options to name your VM and specifie resource it will consume. You also get a nice cost calculation to get an idea how much your new puppy it will cost you. In production workload size should meet the application demand that this VM will run. I am testing the functionality so I have selected the minimums, send me some bitcoins and next we will go crazy with 16vCPUs and 120GB RAM.

I’ll attach the VM to the default-routed-network.

Workload customization and estimated cost

Workload customization and estimated cost

The creation of this small VM took roughly 3 minutes. And the status is shown under main Virtual Machine tab.

New VM is up and running

New VM is up and running

If you got your hands dirty with Amazon AWS, you know that after creating a workload it will receive an elastic IP address that is publicly routable, and using an RSA key pair you can log in though SSH.

The vCloud Air, by default works a little bit differently, are you recalling some mumbling earlier in this post about Remote Console Plugin? That is exactly what we are going to use to access the VM. At least initially. While selecting the VM, open Actions menu and select Open In Console.

Accessing the console through Remote Console Plugin

Accessing the console through Remote Console Plugin

Allow pop-ups and vuala, you are at the VM console. It is that easy.

Sitting at the console

Sitting at the console

I was not able to figured out that the default credentials looking at my magic ball that I have on my table, but I know where to look for it.

For that we need to look further at the VM details.

Discovering the auto generated password

Discovering the auto generated password

Go back to Remote Console and login. To confirm that we indeed ended in the default routed network, lets look at NIC settings and try to reach default gateway.

vcloud-air-16

Tip: You are stuck in Remote Console, press CTRL+ALT to escape the window.

The default Edge configuration will not respond to ping, but you can verify layer 2 by examining the VM ARP table.

Connecting VM to Internet

Our VM is very lonely at the moment, it can only speak to the Edge Gateway in some sort of way. Wouldn’t be great if it could speak to everyone on the Internet? For that to happen, we need to perform some additional configuration on Edge Gateway.

First, we enable the communication from the VM to the internet by configuring Dynamic NAT Translation. On the main page navigate to Gateways tab and select the existing gateway.

Default vPC Gateway

Default vPC Gateway

You will be presented by Gateway specific options, such as NAT Rules, Firewall Rules, Networks, and Public IPs. Before you can add a NAT Translation, you need to add new Public IP address. So start by requesting one.

In the background a new job will be initiated in vCloud Director, which is doing the heavy lifting under this light web UI.

Note: I had some problems assigning a public IP address in my first VDC1, where the job would never finish and I could not do anything with the gateway anymore, therefore I have opened a ticket with vCloud support and they were able to fix the issue with public IP assignments

After the task finishes you are actually assigned the same public address that your edge gateway currently uses.

Public IP successfully assigned

Public IP has been successfully assigned

Lets revisit the  NAT Rules tab and create our first entry that will dynamically translate our internal VM to the public IP address above.

Adding Source NAT entry

Adding Source NAT entry

Simple as that, click Next and Finish.

Although the dynamic NAT rule is in place we are still unable to reach any external resource. We need to modify the default edge firewall policy to allow this communication.

Go to Firewall Rules tab and Add a new entry called Internal-to-Internet.
g

Allowing default internal network to talk to anything on the Internet

Click Next and Finish. With all per-requisites in place the VM can finally reach the internet.
vcloud-air-25

VM is happy happy now

The default vDC private network automatically assigns an address from a pool to newly create virtual machines. These pools are configured in vCloud Director under each Organization VDC Networks respectively. But by default they not include DNS server configuration. For now I will cheat a little bit and edit the list of servers manually in VM at /etc/resolv.conf. You are now fully equip to install additional packages.

Coping and Pasting to virtual console sucks, wouldn’t be great if we could SSH to our box? For that we need three things in place. OpenSSH installed and configured on the box, static NAT entry and a Firewall Policy. There are bunch of great tutorials out there showing how to setup the first part.

For the second part, we are going to create a DNAT entry for VM that will map an external IP address and its port 22 to VM internal address. The second entry will show in the list.

s

Newly added destination NAT for our SSH traffic.

Finally, add a new firewall rule to allow communication from outside on port TCP/22. For added security define only a single or a range of public addresses that you are connecting from.

s

Newly added firewall rule to allow SSH inbound.

Lets connect to our VM via SSH and install Apache web server shall we?

vcloud-air-28

Installing and starting apache

As with any new service, we need to add SNAT entry and Firewall rule to permit communication from the Internet.

You also need to modify the host firewall, iptables in this case to allow communication from outside to httpd service.

After repeating the same steps as above you have a web server running in vCloud Air. How cool is that?

Your first vSphere VM running in cloud.

Your first vSphere VM running in cloud.

Lets stop for a moment and imagine the possibilities, if you can build VM you can build an entire application stack. If you can build an entire application stack, you can build an entire virtual data center. And that is the way to go my friends.

This concludes the basics how to build inside this third party cloud. In next article I am going to focus on scaling and creating more complex network topologies as well as exploring some additional features and parameters available exclusively through the vCloud Director interface.

Advertisements

What you can learn from Networking for VMware Admins

Introduction

It was not so long ago when I was searching for good resource that would uncover what is happening at the hypervisor level from network point of view and how it all clicks together to provider connectivity to ever increasing number of virtual machines. I am happy to say that I have found that resource. It is called Networking for VMware Administrators by Chris Wahl and Steve Pantol.

Networking-for-VMware-Administrators 1

The book starts by discussing the very foundation of networking, what networks is and what benefits it provides. Continuing with with common models and protocol stacks like ISO OSI and TCP/IP and the concepts of layering.

Comparing ISO OSI to TCP/IP

Comparing ISO OSI to TCP/IP

Diving more deeply into the individual layers authors start with physical layer. Ethernet technology is explained in great detail as well as common physical connectivity options like copper or fiber. You will also gain some knowledge about most used network connectors such as RJ-45 and modules like older GBIC or SFP.

10Gbps Twinax Cable used for short interconnections

10Gbps Twinax Cable used for short connections

The chapters build on top of each other and after the foundation and physical network properties next chapter covers data-link operations in great detail. You learn about switching and common network challenges like preventing network loops with spanning tree or increasing network through put by utilizing link aggregation technologies.

Another layer that could not be forgotten is Layer 3 or Network layer. In this chapter IP addressing and routing is explained in detail. Other common services such as automatic address configuration thought DHCP or name resolution with DNS are well touched giving you as a reader better overall perspective.

With the foundation lied down in first 5 chapters the book continues to touch popular converged network infrastructures. Concept of stateless computing from Cisco is explained – the Unified Computing System as well as the HP’s Blade Chassis C7000. Both are compared to give you better insight on one over the other.

Cisco Unified Computing System

Cisco Unified Computing System

The true discussion on virtual networking begins with Chapter 7: How Virtual Switching Differs from Physical Switching. This is an excellent entry point chapter, which describes similarities and differences between both. It touches on common virtual vSwitch terms such as virtual machines’s NIC cards (vNIC), Port-Group, physical uplinks (pNIC), VM kernel ports (vmk) and generally how does the virtual architecture fits with physical.

vSwitch Architecture

vSwitch Architecture

vswitch-logic

vSwitch Forwarding Logic

Better yet, this chapter outlines various configuration options on vSwitch like number of uplinks, MTU and Security Settings. Last but not least trunking and VLAN tagging options are explained.

Chapter 9 focus on vSphere Distributed Switch which is commonly found in enterprise environments. It explains how it differs from Standard vSwitch in control and data plane operations. And elaborates on many extra features it provides. You can expect to gain knowledge on link discovery protocols CDP and LLDP, exporting traffic flows with NetFlow, monitoring traffic in virtual environment using Port Mirroring, segmenting traffic using VLANs and Private VLANs and finally Load Based teaming and Network IO Control for intelligent traffic management.

After you gain this strong foundation, you are free to enter to realm of third party virtual switch. Cisco Nexus 1000V is the topic of next chapter. Authors explain the reasons why you might consider using this third party switch from Cisco in your environment. It touches on core architecture concepts like Virtual Supervisor Module (VSM) and Virtual Ethernet Module (VEM) and various modes of deployment options.

Nexus 1000V deployment options

Nexus 1000V deployment options

If you are more practical type of person, you will definitely like the lab scenarios that authors put together. A step by stem approach is outlined how to build a basic vSphere environment using Cisco UCS as main computing platform. In later chapters you will also discover how to migrate workloads from standard virtual switch to distributed virtual switch without causing downtime.

After discussing general networking technologies with relevant examples, Chapter 14 moves our direction toward IP based storage, starting with iSCSI. General uses cases are explained as well the idea of initiators and targets. Best practices for setting up iSCSI storage adapters are also well explained giving you good confidence when planning in production environment.

The storage topics are then closed by discussing NFS based storage and its uses cases. I especially like the right depth of topics around storage. Practical demonstration at the end of the chapter is also a huge benefit to better put things together.

The next to last chapter deals with additional vSwitch design options, showing many different scenarios with or without IP based storage in place and using 1 Gbps or 10 Gbps network adapters.

One of many vSwitch design options

One of many vSwitch design options

Finally, the last chapter discusses additional design options when dealing with heavy load vMotion migrations. You will learn how to design multiple VM kernel adapters for moving workloads around in case you need to. Network IO Control is also revisited in relation of egress traffic shaping and protecting v host from traffic overload, in case of multiple hosts decide to migrate loads onto same destination hypervisor.

Although I am primarily a network guy I must admin I enjoyed this well written book from the first page to last. It gave exactly what I was looking for, a good foundation of vSphere networking which is the base for advanced technologies like virtual overlays with VXLAN.

OpenWRT: Spinning up Authoritative DNS server

Introduction

During last few months my small home infrastructure has grown in numbers. First there came a beefe virtualization host with 2x AMD Opteron CPUs and 48 GB RAM which runs all labs and is awesome for learning. I named it Sonic after the hedgehog. Then the idea for centralized storage was born, and Synology DS213J together with 2x WD 3TB REDs came on board and is packed with features. My popular ones are Download Station, File Station and Integrated IPSec VPN Server.

As the Internet of Things rises, I bought couple of Raspberry Pi, which one of them runs Openelec as home media center. The others are waiting for new exciting projects that will soon come.

Besides the physical infrastructure, the virtual one have grown even faster with more and more complex labs. Overall there was a need to manage all this stuff more easily and provide layer of abstraction, and home DNS service would make that happen.

After reading number of resources that explained the basics about DNS such as what is the difference between Authoritative, Cache and Forward server, I was confident enough to begin the process of changing the default combined DHCP+DNS service (dnsmasq) in my home router running Openwrt with more capable software. I have chosen bind for the job.

Installing packages

The installation process is very easy, first log into your router via SSH. I am running Barrier Breaker 14.07 on TP-LINK TL-WR941ND. Then using the package manager, install bind-server, bind-tools and isc-dhcp-server-ipv4.

By default, Openwrt includes vi test editor by default, if you are more a nano person, install that package as well. We will use to edit the configuration files later.

BusyBox v1.22.1 (2014-09-20 22:01:35 CEST) built-in shell (ash)
Enter ‘help’ for a list of built-in commands.

  _______                     ________        __
 |       |.—–.—–.—–.|  |  |  |.—-.|  |_
 |   –   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 —————————————————–
 BARRIER BREAKER (14.07, r42625)
 —————————————————–
  * 1/2 oz Galliano         Pour all ingredients into
  * 4 oz cold Coffee        an irish coffee mug filled
  * 1 1/2 oz Dark Rum       with crushed ice. Stir.
  * 2 tsp. Creme de Cacao
 —————————————————–

root@soultrap:~# opkg update
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/base/Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker_base.
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/luci/Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker_luci.
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/packages/Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker_packages.
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/routing/Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker_routing.
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/telephony/Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker_telephony.
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/management/Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker_management.
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/oldpackages/Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker_oldpackages.

root@soultrap:~# opkg install bind-server bind-tools Installing bind-server (9.9.4-1) to root... Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/pa ckages/oldpackages/bind-server_9.9.4-1_ar71xx.ipk. Installing bind-libs (9.9.4-1) to root... Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/pa ckages/oldpackages/bind-libs_9.9.4-1_ar71xx.ipk. Installing libopenssl (1.0.2-1) to root... Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/pa ckages/base/libopenssl_1.0.2-1_ar71xx.ipk. Installing zlib (1.2.8-1) to root... Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/base/zlib_1.2.8-1_ar71xx.ipk. Installing bind-tools (9.9.4-1) to root... Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/oldpackages/bind-tools_9.9.4-1_ar71xx.ipk. Configuring zlib. Configuring libopenssl. Configuring bind-libs. Configuring bind-tools. Configuring bind-server.
root@soultrap:~# opkg install isc-dhcp-server-ipv4 Installing isc-dhcp-server-ipv4 (4.2.4-3) to root... Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/oldpackages/isc-dhcp-server-ipv4_4.2.4-3_ar71xx.ipk. Configuring isc-dhcp-server-ipv4.

root@soultrap:~# opkg install nano
Installing nano (2.3.6-1) to root...
Downloading http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/packages/nano_2.3.6-1_ar71xx.ipk.
Configuring nano.

Dnsmasq is a combined DHCP and DNS server and it would interfere with bind by default. To still have DHCP capabilities, we will configure the isc-dhcp server. But first uninstall dnsmasq. Clean up by removing the old dhcp leases file.

root@soultrap:~# /etc/init.d/dnsmasq stop
root@soultrap:~# opkg remove dnsmasq
Removing package dnsmasq from root...
Not deleting modified conffile /etc/config/dhcp.
root@soultrap:~# mv /etc/config/dhcp /etc/config/dhcp.backup
root@soultrap:~# rm /var/dhcp.leases

Configuring dhcp and bind

After package installation, we are going to configure dhcpd daemon, and the right place to do that is at /etc/dhcpd.conf.

root@soultrap:~# nano /etc/dhcpd.conf
# dhcpd.conf

authoritative;

default-lease-time 3600;
max-lease-time 86400;

option domain-name-servers 10.0.2.1, 8.8.8.8;
option domain-search “papuckovo.home”;

subnet 10.0.2.0 netmask 255.255.255.0 {
  range 10.0.2.128 10.0.2.191;
  option routers 10.0.2.1;
}

Then enable automatic service start after reboot and start the service.

root@soultrap:~# /etc/init.d/dhcpd enable
root@soultrap:~# /etc/init.d/dhcpd start

We will now focus on bind configuration and the main file to look at is located under /etc/bind/ directory.  The file named.conf will be in our particular interest.

First we will configure DNS forwarders that will be used when looking for something outside our authoritative domain. Popular choices are OpenDNS servers or Google Public DNS Servers, I will use the later one.

Next, we will add one forward and one reverse zone for our home domain papuckovo.home. As you can see they will point to files that will hold our records, and we need to create them.

root@soultrap:~# cat /etc/bind/named.conf
// This is the primary configuration file for the BIND DNS server named.
acl “RFC1918” {
        10.0.0.0/8;
};

options {
        directory “/tmp”;
        recursion yes;
        allow-recursion { RFC1918; };
        allow-transfer { none; };
        // If your ISP provided one or more IP addresses for stable
        // nameservers, you probably want to use them as forwarders.  
        // Uncomment the following block, and insert the addresses replacing
        // the all-0’s placeholder.

        forwarders {
                8.8.8.8;
                8.8.4.4;
        };

        auth-nxdomain no;    # conform to RFC1035
};
// prime the server with knowledge of the root servers
zone “.” {
        type hint;
        file “/etc/bind/db.root”;
};

// be authoritative for the localhost forward and reverse zones, and for
// broadcast zones as per RFC 1912

zone “localhost” {
        type master;
        file “/etc/bind/db.local”;
};
zone “papuckovo.home” {
        type master;
        file “/etc/bind/zones/db.papuckovo.home”;
};
zone “127.in-addr.arpa” {
        type master;
        file “/etc/bind/db.127”;
};
zone “0.in-addr.arpa” {
        type master;
        file “/etc/bind/db.0”;
};
zone “255.in-addr.arpa” {
        type master;
        file “/etc/bind/db.255”;
};
zone “2.0.10.in-addr.arpa” {
        type master;
        file “/etc/bind/zones/db.2.0.10”;
}

Now we will create new folder zones and copy example files from main bind directory.

root@soultrap:~# mkdir /etc/zones
root@soultrap:~# cp /etc/bind/db.local /etc/bind/zones/db.papuckovo.home
root@soultrap:~# cp /etc/bind/db.127 /etc/bind/zones/db.2.0.10

Edit the copied files to suit your needs. The SOA record needs to include domain name and the administrator’s email. You should also increment the Serial number after each change to zone files.

root@soultrap:~# nano /etc/bind/zones/db.papuckovo.home
;
; BIND data file for local loopback interface
;
$TTL    604800
@       IN      SOA     papuckovo.home. root.papuckovo.home. (
                              5         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      soultrap.papuckovo.home.
@       IN      A       10.0.2.1
soultrap        IN     A       10.0.2.1
www01           IN     A       10.0.2.2

Do the same thing for reverse zone

root@soultrap:~# nano /etc/bind/zones/db.2.0.10
;
; BIND reverse data file for local loopback interface
;
$TTL    604800
@       IN      SOA     papuckovo.home. root.papuckovo.home. (
                              5         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      soultrap.papuckovo.home.
1       IN      PTR     soultrap.papuckovo.home.
2       IN      PTR     www01.papuckovo.home.

Finally, enable automatic service start after boot and start the service

root@soultrap:~# /etc/init.d/named start
Starting isc-bind
root@soultrap:~# /etc/init.d/named enable

Verification

You can verify the operations of DHCP server by jumping on an internal client and issues ipconfig /renew. Or directly from the router listing dhcpd leases.

root@soultrap:~# head /var/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.2.4

lease 10.0.2.129 {
  starts 1 2015/02/23 08:19:57;
  ends 1 2015/02/23 08:21:57;
  tstp 1 2015/02/23 08:21:57;
  cltt 1 2015/02/23 08:19:57;
  binding state free;
  hardware ethernet 14:7d:c5:11:19:7f;

You can also verify that DNS server is operating correctly using nslookup or dig.

root@soultrap:~# dig ANY papuckovo.home @localhost
; <<>> DiG 9.9.4 <<>> ANY papuckovo.home @localhost
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53030
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;papuckovo.home.                        IN      ANY

;; ANSWER SECTION:
papuckovo.home.         604800  IN      SOA     papuckovo.home. root.papuckovo.home. 5 604800 86400 2419200 604800
papuckovo.home.         604800  IN      NS      soultrap.papuckovo.home.
papuckovo.home.         604800  IN      A       10.0.2.1

;; ADDITIONAL SECTION:
soultrap.papuckovo.home. 604800 IN      A       10.0.2.1

;; Query time: 74 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Feb 24 16:04:56 CET 2015
;; MSG SIZE  rcvd: 139

Troubleshooting

Sometimes, things do not go as we expected, it would be great to narrow down the problem.

root@soultrap:~# /etc/init.d/named start
Starting isc-bind
  isc-bind failed to start

The default error message is not very useful and you can investigate further in log file. In this case I have mistyped the allow-transfer keyword in the main configuration file.

root@soultrap:~# logread
Tue Feb 24 16:20:30 2015 daemon.notice named[13361]: BIND 9 is maintained by Internet Systems Consortium,
Tue Feb 24 16:20:30 2015 daemon.notice named[13361]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
Tue Feb 24 16:20:30 2015 daemon.notice named[13361]: corporation.  Support and training for BIND 9 are
Tue Feb 24 16:20:30 2015 daemon.notice named[13361]: available at https://www.isc.org/support
Tue Feb 24 16:20:30 2015 daemon.notice named[13361]: —————————————————-
Tue Feb 24 16:20:30 2015 daemon.info named[13361]: using 1 UDP listener per interface
Tue Feb 24 16:20:30 2015 daemon.info named[13361]: using up to 4096 sockets
Tue Feb 24 16:20:30 2015 daemon.info named[13361]: loading configuration from ‘/etc/bind/named.conf’
Tue Feb 24 16:20:30 2015 daemon.err named[13361]: /etc/bind/named.conf:10: unknown option ‘allow-trasfer’
Tue Feb 24 16:20:30 2015 daemon.crit named[13361]: loading configuration: failure
Tue Feb 24 16:20:30 2015 daemon.crit named[13361]: exiting (due to fatal error)

MACs – Moved Adds and Changes

When adding new records in reverse and forward zone files it is needed to increase the serial number and then reload bind with new configuration.

root@soultrap:~# /etc/init.d/named reload
Stopping isc-bind
Starting isc-bind

Conclusion

Adding name resolution service to your home router will give you great flexibility and easy to use, your everyday users won’t need to be bothered with IP addresses, they can simple type names.

Further Reading

https://www.digitalocean.com/community/tutorials/an-introduction-to-dns-terminology-components-and-concepts

Cisco ASAv firewall HA Pair

Introduction

In previous post I have introduced and demonstrated the ASA in virtual form factor. This post will built on top of previous that one and will show you how to setup redundant high available pair of these firewalls.

The HA setup is ideal in situations where you are building virtual infrastructure hosted on private or public cloud that needs to be available all the time, surviving failure of one of the firewalls that run as virtual machines. Standard vSphere VM placement best practices should be also considered, such as anti-affinity rules and resource allocation. Consult these with you VMware administrators.

The following diagram outlines the final setup. My home lab only has one ESXi hosts, so everything is running over there.

ASAv HA Setup

Active Firewall rollout

Start with the deployment of first firewall which will be the active one. After downloading the OVA package from Cisco go to vSphere\Virtual Machines\Right Click on Cluster\Deploy OVF Template.

asav-ha-step1

After browsing and selecting the OVA package, I am using asav931.ova, you will be ask to accept the extra configuration options and estimated disk size requirements.

asav-ha-step2

Next, accept the license agreement and and click next. You will have an option to select the name of the new primary ASA which will be ASAv03-Active and the location of the VM.

asav-ha-step3

You will now have a choice to select deployment size, from 1vCPU Standalone all the way up to 4vCPU. If you are deploying in production, you should consult the deployment guide for the right flavor. I am deploying in lab, therefore I have selected 1vCPU HA Primary.

asav-ha-step4

After selecting the deployment size, we need to specify the resource that ASAv will consume. If you select a cluster DRS can automatically place the VM to least utilized hypervisor. You can also specify certain ESXi hosts. I have just one ESXi host, so the choice will be obvious.

asav-ha-step5

To same some storage space, change the default virtual disk format from Thick Provisioned to Thin Provision.

On the next page, we need to configure network mapping. ASAv has by default 10 vNIC adapters which first of them is the Management and the last Gig0/8 is used by HA Heartbeat. Remaining interfaces can be used for production traffic. We leave unused interface in DMZ for now.

asav-ha-step6

Now you have an option to Customize the template by typing configuration parameters such as Management Interface Settings, Device Manager IP Settings and HA Connection Settings.

The Management address of Primary Unit will be 172.16.2.1/24 and Secondary will be 172.16.2.2/24. The default gateway will not be required at this moment, our management station sits on the same virtual switch. To allow remote access from day 1, specify that the allowed management subnet is 172.16.2.0/24

The Primary Unit HA address will be 172.16.3.1/24 and the Secondary will be 172.16.3.2/24.

asav-ha-step7

In Ready to Complete page, you get the summary of all configuration options, and you will have a choice to Power VM after deployment. Congratulations you just deployed your first HA Pair Firewall. After first power on, ASAv will perform initial configuration and reboot.

If everything worked as expected, you should be able to log in from management station via SSH.

asav-ha-step8

By default, however, not username was configured in template therefore it is still needed to jump to virtual console and create one and point the AAA authentication for SSH to local user database.

It is also worth to mention that although ASDM https access was enable, it would still need to have an account and aaa authentication configured properly to work.

ciscoasa(config)# username admin password cisco privilege 15
ciscoasa(config)# enable password cisco
ciscoasa(config)# aaa authentication ssh console LOCAL

Now you can actually connect to the ASA remotely. If you examine the configuration a little bit you find that it has setup the failover interface and peer and currently shows that its mate is down/unknown.

ciscoasa# sh failover
Failover On
Failover unit Primary
Failover LAN Interface: fover GigabitEthernet0/8 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 61 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.3(1), Mate Unknown
Last Failover at: 10:16:56 UTC Feb 20 2015
        This host: Primary – Active
                Active time: 873 (sec)
                slot 0: empty
                  Interface management (172.16.2.1): Unknown (Waiting)
        Other host: Secondary – Failed
                Active time: 0 (sec)
                  Interface management (172.16.2.2): Unknown (Waiting)
 

ciscoasa# show failover state

State          Last Failure Reason      Date/Time
This host  –   Primary
Active         None
Other host –   Secondary
Failed         Comm Failure             10:17:13 UTC Feb 20 2015

====Configuration State===
====Communication State===

Standby Firewall rollout

Before we are going to configure the other interfaces, lets set up the secondary unit. Again we are going to start by deploying a template from .OVA. To save some space, I will only show difference from deploying the primary unit.

The VM name will be set to ASAv03-Standby, and the deployment configuration will be set to 1vCPU HA Secondary. Selecting the right resource in production should be also considered to minimize that a single hypervisor failure will cause both firewalls go down. Therefore I recommended anti-affinity rules so those two VMs will never run on same machine.

The interface mapping will be exactly same as with primary unit. See the reference above. In the customization template page, enter the Management IP address and HA IP address settings for this unit.

asav-ha-step9

Deployment of this small flavor took no longer than 30 seconds, and after the initial reboot the Standby firewall is up an running.

While still setting remotely on primary unit we are going to check the failover pair state again.

Beginning configuration replication: Sending to mate.
ciscoasa# show failover
Failover On
Failover unit Primary
Failover LAN Interface: fover GigabitEthernet0/8 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 61 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.3(1), Mate 9.3(1)
Last Failover at: 10:16:56 UTC Feb 20 2015
        This host: Primary – Active
                Active time: 2111 (sec)
                slot 0: empty
                  Interface management (172.16.2.1): Normal (Monitored)
        Other host: Secondary – Standby Ready
                Active time: 0 (sec)
                  Interface management (172.16.2.2): Normal (Waiting)

ciscoasa# sh failover state

               State          Last Failure Reason      Date/Time
This host  –   Primary
               Active         None
Other host –   Secondary
               Standby Ready  Comm Failure             10:17:13 UTC Feb 20 2015

====Configuration State===
        Sync Done
====Communication State===
        Mac set

====VM Properties Compatibility===
vCPUs – This host:  1
        Other host: 1
Memory – This host:  2048 Mhz
         Other host: 2048 Mhz
Interfaces – This host:  9
             Other host: 9

Looking much better now. To see failover in action, lets first complete the configuration of other interfaces to get some traffic flowing through the firewall

prompt hostname priority state
!
interface GigabitEthernet0/0
 description OUTSIDE
 nameif outside
 security-level 0
 ip address 10.0.2.91 255.255.255.0 standby 10.0.2.92
!
interface GigabitEthernet0/1
 description INSIDE
 nameif inside
 security-level 100
 ip address 172.16.0.1 255.255.255.0 standby 172.16.0.2
!
interface GigabitEthernet0/2
 description DMZ
 nameif dmz
 security-level 50
 ip address 172.16.1.1 255.255.255.0 standby 172.16.1.2
!
monitor-interface outside
monitor-interface inside
monitor-interface dmz
!
route outside 0 0 10.0.2.1
!
policy-map global_policy
 class inspection_default
  inspect icmp

My outside router does not have static routes to these private network behind ASA so a object NAT will help to mitigate that.

object network inside_net
 subnet 172.16.0.0 255.255.255.0
nat (inside,outside) source dynamic inside_net interface

Now we should have connectivity from inside host to Internet. We get verify that by pulling google web page from client and also checking the NAT or connection table.

ciscoasa# sh nat detail
Manual NAT Policies (Section 1)
1 (inside) to (outside) source dynamic inside_net interface
    translate_hits = 12, untranslate_hits = 0
    Source – Origin: 172.16.0.0/24, Translated: 10.0.2.91/24

Now start continuous ping from inside machine and power off the primary firewall.

root@deb01:~# ping -c 100 8.8.8.8
100 packets transmitted, 85 received, 15% packet loss, time 99250ms
rtt min/avg/max/mdev = 10.771/11.988/21.956/2.426

From the output, I examined that from 100 packets, 15 was lost during switchover. With default pool and hold times it was not amazingly fast. We can do better.

failover polltime unit 1 holdtime 3
failover polltime interface 1 holdtime 5

Repeat the 100 ping test again and examine the results.

Now with adjusted timers, from 100 packets we lost 4 during switchover. Getting better :-). Lets see how this HA pair will cope with TCP sessions.

We are going to initiate a large file download from inside host and then shutdown the secondary (now active) ASA and examine the result on traffic flow.

The primary ASA took back the role of Active Firewall, but the TCP session has died. There is one another parameter that we need to tweak for HTTP sessions and to explicitly enable them.

ciscoasa/pri/act(config)# failover replication http
Now initiate the download again, and check that it will be re-established after failover.

Conclusion

After tweaking some default settings you have a highly available firewall cluster running in fully visualized environment.

s

VCP-DCV Series Part 1: Introduction

In the current “cloudy” era of computing, there is growing need for engineers that are able to break the silos between different infrastructure teams such as network, compute and storage to delivery the next generation apps and services to consumers.

This approach brings many benefits to the business. Suddenly network people are interested and understand what server people mean by implementing a distributed virtual switch. Server people now better understand storage people when talking about IO performance and benefits of local storage caching.

In the end it will eventually make all sense. We are not living in closed isolated enviroments any more. People and teams are getting closer together to create a unified engineer who is specialized in one particular area but has presense in other technology towers as well and is able to speak to his peers to reach a common understanding. This is what gives such people the edge they need to be succesfull in this new world of thinking.

This is what the next generation engineer looks like (or atleast feels like :D)

I come with a networking background, in the old world you could call me “the network guy”. The idea of putting togerther a virtualization series came from experimenting and fuzzing with cumpute virtualization itself. I have realized many benefits of virtualization long time ago before knowing that it is one of key elements of every modern data center.

There are many virtualiation vendors out there, from Microsoft, Citrix, or VMware. I have selected the last one, VMware as it has a major share in the market, and I have seen it used by many customers.

If you also want to break the silos, learn something new and have fun, this series is the right place to start. It will give you perspective on data center virtualization from a different angle, from someone who is primarily specialized on networks that sits in the middle and connects these compute pieces together.

The series will explain the nuts and bolts of a typical VMware vSphere implementation in real world. I hope you will also find it useful when preparing for your VCP-DCV exam if that is your goal.

Cisco ASAv firewall installation

Introduction

The data center networking trend is clear with every network service being slowly virtualized. Network devices that are virtual provide big advantage over their physical counterparts. First, VMs can be provisioned much more quicker and be part of a larger virtual infrastructure, you can easily scale them adding more virtual CPU or memory and you can snapshot them to save their actual state to a file and move them around.

Cisco has also introduced their virtual version of their popular firewall product ASA. It simple called ASAv and runs under popular hypervisors such as VMware vSphere or KVM. You can find the product home page here. This is different product and idea than Cisco ASA 1000V Cloud Firewall.

First you need to get hold of the ova package. You need to be entitled with Cisco to download the software from here or you can do a Google search and find it elsewhere. I had some problems with the latest release 9.3.2(200) where it would stuck at booting loop, the kernel complained about Illegal Instruction. Looks like it did not like my dual Opteron 4180 host. Therefore in this demo we are going to use release 9.3.1 which worked just fine.

Sharping the axe

Before we are going to deploy the actual virtual firewall, lets make some solid ground for it. Firewalls usually divide network into multiple security zones so first we are going to create some, and we use vSwitches for that. In my vSphere deployment I already have default vSwitch called vSwitch0, with a Portgroup called Native that has a connection to outside world.

We are going to create additional two vSwitches that will have following Portgroups: ASAv-inside and ASAv-DMZ respectively, and we are going to attach two linux instances to them. So in the end we end up with simple topology like this:

asav-network

To get started, log in to vSphere and go to Hosts\Configuration\Networking\Virtual Standard Switch, click add Add networking.

Select New Standard Switch, there is no need to assign physical adapter for breakout, we will attach this vSwitch to one of ASAv interfaces. The first port group will be called ASAv-inside with no VLAN tag. Follow the same steps for DMZ vSwitch and its ASAv-DMZ PortGroup.

asav1

Chopping the tree

Back to ASAv, after downloading, log into the vCenter and go to VMs and File\Deploy OVF Template.

Note: I tried to deploy the asav932-200.ova directly into ESXi, however I received and error that The OVF package requires support for OVF properties. Details: Line 264: Unsupported element ‘Property’

Answer the usual OVF deployment question such as name of VM, which Data Center and Cluster will be used. I only have one so it is no brainier. Deployment configuration specifies the number of vCPU that the VM will have and whether it will be part of HA pair. By default ASAv will come with 1 Management Interface Management0/0 and 9 regular interfaces GigabitEthernet0/0 – 0/8. You need to map each of them to correct port group created in previous step.

I am only really using 4 interfaces at this point, so I left the rest connected in ASAv-DMZ portgroup.

asav2

1 Mgmt Interface and 9 Regular Interfaces

Some basic configuration parameters such as IP configuration of management interface can also be entering during wizard. That makes me wonder if those parameters can be passed to template while deploying automatically via script.

asav3

Initial Configuration Options

After quick OVF deployment, you can look at default resource requirements which correspond to deployment size selected in wizard.

asav6

Resource utilization

And finally the ASAv console is available directly through vCenter.

asav3

ASAv Virtual Console

Before you can take full advantage of all ASAv features in your lab you need to license the box. If you are lucky you can ask a Cisco representative for a temporary license or *hint* do a smart Google search for a little piece of software.

Initial Configuration

To actually verify that the ASAs has indeed network connectivity, we will perform initial configuration and test reach ability to Google DNS servers.

#First virtual interface mapped to Native PortGroup
#
interface GigabitEthernet0/0
nameif outside
security-level 0
ip address dhcp
#
#Second virtual interface mapped to ASAv-inside PortGroup
#
interface GigabitEthernet0/1
nameif inside
security-level 100
ip address 192.168.1.1 255.255.255.0
#
#Management virtual interface mapped to Backend PortGroup
#
interface Management0/0
management-only
nameif management
security-level 0
ip address 10.0.1.41 255.255.255.0
#
# DNS, SSH and routing
#
dns server-group DefaultDNS
name-server 8.8.8.8
!
route outside 0.0.0.0 0.0.0.0 10.0.2.1 1
aaa authentication ssh console LOCAL
ssh 10.0.1.0 255.255.255.0 management
username cisco password

Verification

asav5

If you are currently aiming for CCIE Security this is an excellent way how to build your own virtual lab for practice. Coupled with virtual ACS server and IPS appliances it is very easy to test and learn new features, validate syntax for scripts and many more without harming your production environment.

I draw the line in the sand here and leave your imagination what you can do with multiple of these virtual firewalls bundled with couple of virtual routers and virtual machines to re-create complete data center infrastructure sandbox.

Resources

Introduction to ASAv

http://www.cisco.com/c/en/us/td/docs/security/asa/asa92/asav/quick-start/asav-quick/intro-asav.html

ASAv Product Overview

http://www.cisco.com/c/en/us/products/collateral/security/adaptive-security-virtual-appliance-asav/datasheet-c78-733399.pdf

Deploying ASAv

http://www.cisco.com/c/en/us/td/docs/security/asa/asa92/configuration/general/asa-general-cli/intro-asav.pdf

Digging into DNS

For many years I have used CLI tool nslookup for grabbing any DNS related information. Recently I have switched to different tool called Domain Infromation Grabber or simply DIG. It is like a next gen nslookup as you will see it has many cool features. This short tutorial will show you how to install it under Windows machine.

First, you need to visit the ISC site at http://www.isc.org/downloads. Dig is part of bind installation package. At the time of writing this artice the current stable version is 9.9.6-P1. Choose you your architecture, I’ll go now with x64. Download & Unzip.

Dig requires to have MS C++ Redistributable installed, so install vcredist_x64 first. Now copy dig.exe and all .dll files into C:\Windows\System32\

Now when you are able to execute dig directly from command line:

dig-1Some basic task you can do with dig are standard IP name resolution.dig-3

As you see the output shows several sections. The information we are looking for is under ANSWER SECTION. This querry shows the IPv4 address of opendns.com. Without specifying any options the default DNS server is used. In this case it is 10.0.2.1.

If you want to point the DNS querry to particular DNS server, use @ option. In next example I am asking opendns DNS server about the same domain.

dig-4

To perform a reverse DNS querry, use -x option. Here I am asking the same server what does IPv4 address 67.215.92.210 resolve to. The answer is api.opendns.com.

dig-5

Last but not least, you can ask for mail server records MX. Which will return the mail server addresses that opendns.com domain uses.

dig-6

Dig has many great features, but these few should keep you going for a while. There are also sites such as kloth.net that provide DIG services.