Category Archives: SDN

vCloud Air: OnDemand IaaS by VMware

Introduction

Cloud computing is currently a highly discussed topic in many IT strategy meetings. The ability to spin workloads in third party cloud and pay only for what you actually use certainly make sense from financial point of view. There are many great use cases for that.

Imagine you have a big event, something like Super Bowl and your current infrastructure would likely struggle to handle such load for the duration of the event. You have two options how to prepare for it, you can buy new assets – compute, storage and network spending a lot of dollars or you can build your application stacks in third party cloud and leave it running for certain period of time and then tear down when the event is over. The second option also gives you virtually unlimited resources if you need to use them. How cool is that?

There are many Infrastructure as a Service offerings in the market, one of the most popular include Amazon AWS, Microsoft Azure or Google Cloud Compute. It was only matter of time that other vendors would start offer the same thing to address the competition.

One of such vendors is VMware. Initially, the popular hypervisor vendor offered two flavors of their public cloud offering named vCloud Air. First was a Dedicated Cloud, targeted on customers who wanted to essentially lease the entire servers for better physical isolation and security. The second was a Virtual Private Cloud, which is cheaper, but the hypervisor would run other companies’ VMs as well and provide logical isolation.

In either case, both of the offering were targeted on larger customers and you and me did not have an easy option to just spin couple of VMs for testing without committing to some long term payment plant.

With the new OnDemand access flavor that model changes and you now have a true option to pay as you go. This flavor allows any individual from flesh and bone (with credit card) to spin a workload in VMware’s data center and pay only for what he uses.

safe_image.php

vCloud Air Common Use Cases

To support this idea, they even get you a nice started pack of $300 dollars worth of resources. It is like with gas provider, where they would get you certain number of gas for free to try their service out. Good move to attrack people like me to poke around and spin some workloads.

I think that many enterprise companies will consider this option, as their infrastructure teams are familiar with vSphere and this cloud is built upon the same foundations as their data centers, making migration and interoperability a little bit easier for both sides.

How does it feel to spin a couple of VMs in this environment? Lets have a look.

Signing Up

First, you need to go to super secret URL only know by 5 individuals in the world. I like my audience so I am going to share that URL with you and it is at http://vcloud.vmware.com.

The landing pages gives you couple of information what vCloud is, couple of testimonial and general information. Click on Service Offering/Virtual Private Cloud OnDemand to get started.

Lets take advantage of that $300 voucher shall we? For that we need a VMware Account First. You you have one log in now, otherwise make the clickie-clickie action and create one.

During the registration you need to enter a valid credit card and billing information.

vcloud-air-1

Creating a new account for vCloud Air

At this point you will have an options to select the Support Level Plan. I’ll go with the basic one OnDemand Online Support. Make sure that the promo code ondemand2015 has been applied. As so many people are trying to get the hands on, you likely end up in the queue with Sing up request pending. So wait for it.

Ok, it took around 5 minutes to get the confirmation email with the login page URL and initial links to set the password.

vcloud-air-2

You need to reset your temporary password

Entering to the vCloud

Now that you we have set the credentials, lets go back to to main login page at https://vca.vmware.com. If you would like to use VMware Remote Console, install it & enable the plugin in your browser.

vcloud-air-3

The Login Screen looks nice and clean, thumbs up for that.

Building our first vPC

After login you are presented by main dashboard, that includes Services, Subscriptions and Tools. After Clicking the Virtual Private Cloud Ondemand, you have an option to select in which physical location you want to build your vPC. Since I am located in UK I’ll go with UK Slough 1 6 option. Slough is small town in Berkshire.

vcloud-air-4

After couple of seconds a new vPC instance will be build. After the gears stop spinning you will receive the following handy infrastructure to play with.

Default vPC topology

Default vPC topology

The infrastructure is composed of one Gateway, that provides access to your private cell. This gateway is connected to public segment which is according Ripe a chunk of larger pool that VMware allocated for this particular cloud.

The second network, is a private segment where would build your VM. We will get to that later. I want to show you around the interface.

The first tab you find under vPC is the Resource Usage which shows you how much you resources you consume and how much for last hour/day or month. You also have option to view detailed report. Give that your financial department to sponsor your vCloud adventures.

vcloud-air-6

Creating new virtual machine

Next, looking at Virtual Machines tab, you have an option to spin your first workload or migrate it from your private data center. Lets keep things simple for now and select the first option.

Creating the first workload

Creating the first workload

As any good cloud offering, you will be presented by a catalog of virtual machines that you can select from. Most of the Linux flavors are free, and you pay same extra fees for Windows VMs for licensing. You also have an option to create an empty machine, called shell VM from scratch. I’ll go with CentOS 6.4 32 Bit for now.

Selecting a VM from catalog

Selecting a VM from catalog

In customization page, you have the options to name your VM and specifie resource it will consume. You also get a nice cost calculation to get an idea how much your new puppy it will cost you. In production workload size should meet the application demand that this VM will run. I am testing the functionality so I have selected the minimums, send me some bitcoins and next we will go crazy with 16vCPUs and 120GB RAM.

I’ll attach the VM to the default-routed-network.

Workload customization and estimated cost

Workload customization and estimated cost

The creation of this small VM took roughly 3 minutes. And the status is shown under main Virtual Machine tab.

New VM is up and running

New VM is up and running

If you got your hands dirty with Amazon AWS, you know that after creating a workload it will receive an elastic IP address that is publicly routable, and using an RSA key pair you can log in though SSH.

The vCloud Air, by default works a little bit differently, are you recalling some mumbling earlier in this post about Remote Console Plugin? That is exactly what we are going to use to access the VM. At least initially. While selecting the VM, open Actions menu and select Open In Console.

Accessing the console through Remote Console Plugin

Accessing the console through Remote Console Plugin

Allow pop-ups and vuala, you are at the VM console. It is that easy.

Sitting at the console

Sitting at the console

I was not able to figured out that the default credentials looking at my magic ball that I have on my table, but I know where to look for it.

For that we need to look further at the VM details.

Discovering the auto generated password

Discovering the auto generated password

Go back to Remote Console and login. To confirm that we indeed ended in the default routed network, lets look at NIC settings and try to reach default gateway.

vcloud-air-16

Tip: You are stuck in Remote Console, press CTRL+ALT to escape the window.

The default Edge configuration will not respond to ping, but you can verify layer 2 by examining the VM ARP table.

Connecting VM to Internet

Our VM is very lonely at the moment, it can only speak to the Edge Gateway in some sort of way. Wouldn’t be great if it could speak to everyone on the Internet? For that to happen, we need to perform some additional configuration on Edge Gateway.

First, we enable the communication from the VM to the internet by configuring Dynamic NAT Translation. On the main page navigate to Gateways tab and select the existing gateway.

Default vPC Gateway

Default vPC Gateway

You will be presented by Gateway specific options, such as NAT Rules, Firewall Rules, Networks, and Public IPs. Before you can add a NAT Translation, you need to add new Public IP address. So start by requesting one.

In the background a new job will be initiated in vCloud Director, which is doing the heavy lifting under this light web UI.

Note: I had some problems assigning a public IP address in my first VDC1, where the job would never finish and I could not do anything with the gateway anymore, therefore I have opened a ticket with vCloud support and they were able to fix the issue with public IP assignments

After the task finishes you are actually assigned the same public address that your edge gateway currently uses.

Public IP successfully assigned

Public IP has been successfully assigned

Lets revisit the  NAT Rules tab and create our first entry that will dynamically translate our internal VM to the public IP address above.

Adding Source NAT entry

Adding Source NAT entry

Simple as that, click Next and Finish.

Although the dynamic NAT rule is in place we are still unable to reach any external resource. We need to modify the default edge firewall policy to allow this communication.

Go to Firewall Rules tab and Add a new entry called Internal-to-Internet.
g

Allowing default internal network to talk to anything on the Internet

Click Next and Finish. With all per-requisites in place the VM can finally reach the internet.
vcloud-air-25

VM is happy happy now

The default vDC private network automatically assigns an address from a pool to newly create virtual machines. These pools are configured in vCloud Director under each Organization VDC Networks respectively. But by default they not include DNS server configuration. For now I will cheat a little bit and edit the list of servers manually in VM at /etc/resolv.conf. You are now fully equip to install additional packages.

Coping and Pasting to virtual console sucks, wouldn’t be great if we could SSH to our box? For that we need three things in place. OpenSSH installed and configured on the box, static NAT entry and a Firewall Policy. There are bunch of great tutorials out there showing how to setup the first part.

For the second part, we are going to create a DNAT entry for VM that will map an external IP address and its port 22 to VM internal address. The second entry will show in the list.

s

Newly added destination NAT for our SSH traffic.

Finally, add a new firewall rule to allow communication from outside on port TCP/22. For added security define only a single or a range of public addresses that you are connecting from.

s

Newly added firewall rule to allow SSH inbound.

Lets connect to our VM via SSH and install Apache web server shall we?

vcloud-air-28

Installing and starting apache

As with any new service, we need to add SNAT entry and Firewall rule to permit communication from the Internet.

You also need to modify the host firewall, iptables in this case to allow communication from outside to httpd service.

After repeating the same steps as above you have a web server running in vCloud Air. How cool is that?

Your first vSphere VM running in cloud.

Your first vSphere VM running in cloud.

Lets stop for a moment and imagine the possibilities, if you can build VM you can build an entire application stack. If you can build an entire application stack, you can build an entire virtual data center. And that is the way to go my friends.

This concludes the basics how to build inside this third party cloud. In next article I am going to focus on scaling and creating more complex network topologies as well as exploring some additional features and parameters available exclusively through the vCloud Director interface.

Cisco ASAv firewall REST API: Bulk Requests

Introduction

Continuing on the same track with ASA REST API we are going to explore how to be more efficient when firing up the json code toward our firewalls. In previous example we created a small number of service and network objects and groups, today we are going to experiment with larger number of objects created by smaller number of calls. Lets get the ball rolling.

Bulk service object creation

To create a bulk request we need to adjust some parameters such as API endpoint URL. As you recall, when we created a new service object we were pointing to this URL: https://<management-ip>/api/objects/networkservices.

Now we are going to redesign the call a little bit and will always point to same endpoint regardless of object type. We will simply use https://<management-ip>/api and the more specific resource URI will be part request body. This allows us for example create bulk of service and network objects in one call.

[
  {
    "resourceUri": "/api/objects/networkservices",
    "data": {     
  "kind": "object#TcpUdpServiceObj",
  "name": "TCP-21",
  "value": "tcp/21"
}, "method": "Post" }, { "resourceUri": "/api/objects/networkservices",    "data": {
  "kind": "object#TcpUdpServiceObj",
  "name": "TCP-22",
  "value": "tcp/22"
}, "method": "Post" }, { "resourceUri": "/api/objects/networkservices",    "data": {
  "kind": "object#TcpUdpServiceObj",
  "name": "TCP-23",
  "value": "tcp/23"
}, "method": "Post" }, { "resourceUri": "/api/objects/networkservices", "data": {
  "kind": "object#TcpUdpServiceObj",
  "name": "TCP-24",
  "value": "tcp/24"
}, "method": "Post" }
]

If everything is hunky dory you will receive a Status Code 200 OK and response body will contain additional details.

{
"entryMessages":
[
     {
     "resourceUri":"/api/objects/networkservices",
     "method":"POST",
     "selfLink":"http://10.201.230.5/api/objects/networkservices/TCP-21",
     "messages":
     [
         {
         "level":"Info",
         "code":"201",
         "details":"Created (201) - The request has been fulfilled and resulted in a new resource being created"
          }
     ]
     },
     {
     "resourceUri":"/api/objects/networkservices",
     "method":"POST",
     "selfLink":"http://10.201.230.5/api/objects/networkservices/TCP-22",
     "messages":
     [
         {
         "level":"Info",
         "code":"201",
         "details":"Created (201) - The request has been fulfilled and resulted in a new resource being created"
         }
     ]
     },
     {
     "resourceUri":"/api/objects/networkservices",
     "method":"POST",
     "selfLink":"http://10.201.230.5/api/objects/networkservices/TCP-23",
     "messages":
     [
         {
         "level":"Info",
         "code":"201",
         "details":"Created (201) - The request has been fulfilled and resulted in a new resource being created"
         }
     ]
     },
     {
     "resourceUri":"/api/objects/networkservices",
     "method":"POST",
     "selfLink":"http://10.201.230.5/api/objects/networkservices/TCP-24",
     "messages":
     [
         {
         "level":"Info",
         "code":"201",
         "details":"Created (201) - The request has been fulfilled and resulted in a new resource being created"
         }
     ]
     }
],
"commonMessages":[]
}

Ideally your deployment script would examine the response and capture the self links that can be used to un-deploy this call. Also the message details are important to verify if objects were successfully created.

Now we are going to look at error handling by creating another request that will have one already existing object and one new in it.

[
  {
    "resourceUri": "/api/objects/networkservices",
    "data": {     
  "kind": "object#TcpUdpServiceObj",
  "name": "TCP-24",
  "value": "tcp/24"
}, "method": "Post" }, { "resourceUri": "/api/objects/networkservices", "data": {
  "kind": "object#TcpUdpServiceObj",
  "name": "TCP-25",
  "value": "tcp/25"
}, "method": "Post" }
]

The response header will contain 400 Bad Request status code and body will reveal some clues.

{
"entryMessages":
[
     {
     "resourceUri":"/api/objects/networkservices",
     "method":"POST",
     "messages":
     [
         {
         "level":"Error",
         "code":"DUPLICATE",
         "context":"objectId",
         "details":"TCP-24"
         }
     ]
     },
     {
     "resourceUri":"/api/objects/networkservices",
     "method":"POST"
     }
],
"commonMessages":[]
}

The result result is that even the second object did not exists it was not created. When you reverse the object order in request, it will not help either. This means that the entire request need to create unique objects otherwise it will fail. So watch for return codes.

I have tried to create 103 unique objects in one request and I have found that the upper limit for single request is 100 entries.

{
"commonMessages":
[
     {
     "level":"Error",
     "code":"MAX-ENTRIES-LIMIT-EXCEEDED",
     "details":"The number of entries found are, 103. The bulk entries limit:100"
     }
]
}

Lets go back to our first four service objects and undeploy them using four unique DELETE Requests. The resource URIs are as follows:

Note: So far I have not found way of performing multiple object removal in single request. It seems that bulk operation only supports POST method toward service objects at the moment. The following below did not work.

[
  {
    "resourceUri": "/api/objects/networkservices/TCP-21",
    "method": "Delete"
  },
  {
    "resourceUri": "/api/objects/networkservices/TCP-21",
    "method": "Delete"
  }
]

Response had status code 400 Bad Request even if those URI are indeed valid and you can query them with GET request.

{
"entryMessages":
[
     {
     "resourceUri":"/api/objects/networkservices/TCP-21",
     "method":"DELETE",
     "messages":
     [
         {
         "level":"Error",
         "code":"INVALID-INPUT",
         "details":"Invalid resourceUri:/api/objects/networkservices/TCP-21"
         }
     ]
     },
     {
     "resourceUri":"/api/objects/networkservices/TCP-21",
     "method":"DELETE",
     "messages":
     [
         {
         "level":"Error",
         "code":"INVALID-INPUT",
         "details":"Invalid resourceUri:/api/objects/networkservices/TCP-21"
         }
     ]
     }
],
"commonMessages":[]
}

The 100 objects limitations also affect how are the responses constructed. When you query the following resource https://<management-ip>/api/objects/networkservices you will see that the body contains only 100 items even if 110 exists. You can see this by examining the offset, limit and total fields on body.

{
   "kind": "collection#NetworkServiceObjects",
   "selfLink": "https://10.201.230.5/api/objects/networkservices",
   "rangeInfo":
   {
       "offset": 0,
       "limit": 100,
       "total": 110
   },

Funny enough, the list will not start with the first configure service objects, but started from TCP-100 up till TCP-200. To see other items you need to manipulate the offset value. We can demonstrate this using API Console at documentation page.

rest-api-offset

Now the item list will start from TCP-110 and will include our older objects such as TCP-21, TCP-22 etc. You can also play with the limit parameter to constrain number of returned objects. If you look at Inspector, you will see the request parameters have been modified to include our query parameters.

rest-api-offset-body

Conclusion

Bulk object creation and retrieval handling should help you build more efficient communication patters between the firewall and automation tool or network controller.

Cisco ASAv firewall HA Pair

Introduction

In previous post I have introduced and demonstrated the ASA in virtual form factor. This post will built on top of previous that one and will show you how to setup redundant high available pair of these firewalls.

The HA setup is ideal in situations where you are building virtual infrastructure hosted on private or public cloud that needs to be available all the time, surviving failure of one of the firewalls that run as virtual machines. Standard vSphere VM placement best practices should be also considered, such as anti-affinity rules and resource allocation. Consult these with you VMware administrators.

The following diagram outlines the final setup. My home lab only has one ESXi hosts, so everything is running over there.

ASAv HA Setup

Active Firewall rollout

Start with the deployment of first firewall which will be the active one. After downloading the OVA package from Cisco go to vSphere\Virtual Machines\Right Click on Cluster\Deploy OVF Template.

asav-ha-step1

After browsing and selecting the OVA package, I am using asav931.ova, you will be ask to accept the extra configuration options and estimated disk size requirements.

asav-ha-step2

Next, accept the license agreement and and click next. You will have an option to select the name of the new primary ASA which will be ASAv03-Active and the location of the VM.

asav-ha-step3

You will now have a choice to select deployment size, from 1vCPU Standalone all the way up to 4vCPU. If you are deploying in production, you should consult the deployment guide for the right flavor. I am deploying in lab, therefore I have selected 1vCPU HA Primary.

asav-ha-step4

After selecting the deployment size, we need to specify the resource that ASAv will consume. If you select a cluster DRS can automatically place the VM to least utilized hypervisor. You can also specify certain ESXi hosts. I have just one ESXi host, so the choice will be obvious.

asav-ha-step5

To same some storage space, change the default virtual disk format from Thick Provisioned to Thin Provision.

On the next page, we need to configure network mapping. ASAv has by default 10 vNIC adapters which first of them is the Management and the last Gig0/8 is used by HA Heartbeat. Remaining interfaces can be used for production traffic. We leave unused interface in DMZ for now.

asav-ha-step6

Now you have an option to Customize the template by typing configuration parameters such as Management Interface Settings, Device Manager IP Settings and HA Connection Settings.

The Management address of Primary Unit will be 172.16.2.1/24 and Secondary will be 172.16.2.2/24. The default gateway will not be required at this moment, our management station sits on the same virtual switch. To allow remote access from day 1, specify that the allowed management subnet is 172.16.2.0/24

The Primary Unit HA address will be 172.16.3.1/24 and the Secondary will be 172.16.3.2/24.

asav-ha-step7

In Ready to Complete page, you get the summary of all configuration options, and you will have a choice to Power VM after deployment. Congratulations you just deployed your first HA Pair Firewall. After first power on, ASAv will perform initial configuration and reboot.

If everything worked as expected, you should be able to log in from management station via SSH.

asav-ha-step8

By default, however, not username was configured in template therefore it is still needed to jump to virtual console and create one and point the AAA authentication for SSH to local user database.

It is also worth to mention that although ASDM https access was enable, it would still need to have an account and aaa authentication configured properly to work.

ciscoasa(config)# username admin password cisco privilege 15
ciscoasa(config)# enable password cisco
ciscoasa(config)# aaa authentication ssh console LOCAL

Now you can actually connect to the ASA remotely. If you examine the configuration a little bit you find that it has setup the failover interface and peer and currently shows that its mate is down/unknown.

ciscoasa# sh failover
Failover On
Failover unit Primary
Failover LAN Interface: fover GigabitEthernet0/8 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 61 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.3(1), Mate Unknown
Last Failover at: 10:16:56 UTC Feb 20 2015
        This host: Primary – Active
                Active time: 873 (sec)
                slot 0: empty
                  Interface management (172.16.2.1): Unknown (Waiting)
        Other host: Secondary – Failed
                Active time: 0 (sec)
                  Interface management (172.16.2.2): Unknown (Waiting)
 

ciscoasa# show failover state

State          Last Failure Reason      Date/Time
This host  –   Primary
Active         None
Other host –   Secondary
Failed         Comm Failure             10:17:13 UTC Feb 20 2015

====Configuration State===
====Communication State===

Standby Firewall rollout

Before we are going to configure the other interfaces, lets set up the secondary unit. Again we are going to start by deploying a template from .OVA. To save some space, I will only show difference from deploying the primary unit.

The VM name will be set to ASAv03-Standby, and the deployment configuration will be set to 1vCPU HA Secondary. Selecting the right resource in production should be also considered to minimize that a single hypervisor failure will cause both firewalls go down. Therefore I recommended anti-affinity rules so those two VMs will never run on same machine.

The interface mapping will be exactly same as with primary unit. See the reference above. In the customization template page, enter the Management IP address and HA IP address settings for this unit.

asav-ha-step9

Deployment of this small flavor took no longer than 30 seconds, and after the initial reboot the Standby firewall is up an running.

While still setting remotely on primary unit we are going to check the failover pair state again.

Beginning configuration replication: Sending to mate.
ciscoasa# show failover
Failover On
Failover unit Primary
Failover LAN Interface: fover GigabitEthernet0/8 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 1 of 61 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.3(1), Mate 9.3(1)
Last Failover at: 10:16:56 UTC Feb 20 2015
        This host: Primary – Active
                Active time: 2111 (sec)
                slot 0: empty
                  Interface management (172.16.2.1): Normal (Monitored)
        Other host: Secondary – Standby Ready
                Active time: 0 (sec)
                  Interface management (172.16.2.2): Normal (Waiting)

ciscoasa# sh failover state

               State          Last Failure Reason      Date/Time
This host  –   Primary
               Active         None
Other host –   Secondary
               Standby Ready  Comm Failure             10:17:13 UTC Feb 20 2015

====Configuration State===
        Sync Done
====Communication State===
        Mac set

====VM Properties Compatibility===
vCPUs – This host:  1
        Other host: 1
Memory – This host:  2048 Mhz
         Other host: 2048 Mhz
Interfaces – This host:  9
             Other host: 9

Looking much better now. To see failover in action, lets first complete the configuration of other interfaces to get some traffic flowing through the firewall

prompt hostname priority state
!
interface GigabitEthernet0/0
 description OUTSIDE
 nameif outside
 security-level 0
 ip address 10.0.2.91 255.255.255.0 standby 10.0.2.92
!
interface GigabitEthernet0/1
 description INSIDE
 nameif inside
 security-level 100
 ip address 172.16.0.1 255.255.255.0 standby 172.16.0.2
!
interface GigabitEthernet0/2
 description DMZ
 nameif dmz
 security-level 50
 ip address 172.16.1.1 255.255.255.0 standby 172.16.1.2
!
monitor-interface outside
monitor-interface inside
monitor-interface dmz
!
route outside 0 0 10.0.2.1
!
policy-map global_policy
 class inspection_default
  inspect icmp

My outside router does not have static routes to these private network behind ASA so a object NAT will help to mitigate that.

object network inside_net
 subnet 172.16.0.0 255.255.255.0
nat (inside,outside) source dynamic inside_net interface

Now we should have connectivity from inside host to Internet. We get verify that by pulling google web page from client and also checking the NAT or connection table.

ciscoasa# sh nat detail
Manual NAT Policies (Section 1)
1 (inside) to (outside) source dynamic inside_net interface
    translate_hits = 12, untranslate_hits = 0
    Source – Origin: 172.16.0.0/24, Translated: 10.0.2.91/24

Now start continuous ping from inside machine and power off the primary firewall.

root@deb01:~# ping -c 100 8.8.8.8
100 packets transmitted, 85 received, 15% packet loss, time 99250ms
rtt min/avg/max/mdev = 10.771/11.988/21.956/2.426

From the output, I examined that from 100 packets, 15 was lost during switchover. With default pool and hold times it was not amazingly fast. We can do better.

failover polltime unit 1 holdtime 3
failover polltime interface 1 holdtime 5

Repeat the 100 ping test again and examine the results.

Now with adjusted timers, from 100 packets we lost 4 during switchover. Getting better :-). Lets see how this HA pair will cope with TCP sessions.

We are going to initiate a large file download from inside host and then shutdown the secondary (now active) ASA and examine the result on traffic flow.

The primary ASA took back the role of Active Firewall, but the TCP session has died. There is one another parameter that we need to tweak for HTTP sessions and to explicitly enable them.

ciscoasa/pri/act(config)# failover replication http
Now initiate the download again, and check that it will be re-established after failover.

Conclusion

After tweaking some default settings you have a highly available firewall cluster running in fully visualized environment.

s