Step-by-step VMware Cloud Foundation 4.3 design and install MGMT domain

As VMware cloud foundation 4.3 is around for a while and 4.3.1 is already available, I thought I should write this piece on how to design and deploy step by step. So with out wasting any time lets directly jump on to the product. VMware cloud foundation is available for some time now and many enterprises are adopting it because of ease of management it provides, in terms of a complete suite which includes all required/necessary products for a true software defined datacenter. But if you are new to VMware Cloud Foundation then be aware VMware cloud foundation is a VMware validated suite of products such as vSphere for compute virtualization, vSAN for storage virtualization and NSX for network virtualization along with other products to ease day 2 operations. Interoperability of these products is extensively tested by VMware and finally made available for general use. It is based on VMware validated designs so all solution designing principle are accounted for.

If you are installing it fresh or you need to upgrade from a previous version of VCF I would recommend reading the release notes. Below are few sections I focus.

What's new  

(Source:VMware.com)

The VMware Cloud Foundation 4.3 release includes the following:

• Flexibility in Application Virtual Networks (AVN):  Application Virtual Networks (AVN)s, which include the NSX Edge Cluster and NSX network segments, are no longer deployed and configured during bring-up. Instead they are implemented as a Day-N operations in SDDC Manager, providing greater flexibility.
• FIPS Support: You can enable FIPS mode during bring-up, which will enable it on all the VMware Cloud Foundation components that support FIPS.
• Scheduled Automatic Password Rotations: In addition to the on-demand password rotation capability, it is now possible to schedule automatic password rotations for accounts managed through SDDC Manager (excluding ESXi accounts). Automatic password rotation is enabled by default for service accounts.  
• SAN in Certificate Signing Requests (CSR) : You can now add a Subject Alternative Name (SAN) when you generate a Certificate Signing Request (CSR) in SDDC Manager.
• Improvements for vSphere Lifecycle Manager images:  For workload domains that use vSphere Lifecycle Manager images, this release includes several improvements. These include: prechecks to proactively identify issues that may affect upgrade operations; enabling concurrent upgrades for NSX-T Data Center components; and enabling provisioning and upgrade of Workload Management. 
• Add vSphere Clusters in Parallel: You can add up to 10 vSphere clusters to a workload domain in parallel, improving the performance and speed of the workflow.
• Add and Remove NSX Edge Nodes in NSX Edge Clusters: For NSX Edge clusters deployed through SDDC Manager or the VMware Cloud Foundation API, you can expand and shrink NSX Edge clusters by adding or removing NSX Edge nodes from the cluster.
• Guidance for Day-N operations in NSX Federated VCF environments: You can federate NSX-T Data Center environments across VMware Cloud Foundation instances. You can manage federated NSX-T Data Center environments with a single pane of glass, create gateways and segments that span VMware Cloud Foundation instances, and configure and enforce firewall rules consistently across instances. Guidance is also provided for password rotation, certificate management, backup and restore, and lifecycle management for federated environments.
• Backup Enhancements: You can now configure an SDDC Manager backup schedule and retention policy from the SDDC Manager UI.
• VMware Validated Solutions: VMware Validated Solutions are a series of technical reference validated implementations designed to help customers build secure, high-performing, resilient, and efficient infrastructure for their applications and workloads deployed on VMware Cloud Foundation. Each VMware Validated Solution will come with detailed design with design decisions, implementation guidance consisting of manual UI-based step-by-step procedures and, where applicable, automated steps using infrastructure as code. These solutions based on VMware Cloud Foundation will be available on core.vmware.com. The first set of validated solutions, that can be applied on vSAN ReadyNodes, include the following:
• Identity and Access Management for VMware Cloud Foundation 
• Developer Ready Infrastructure for VMware Cloud Foundation 
• Advanced Load Balancing for VMware Cloud Foundation 
• Private Cloud Automation for VMware Cloud Foundation 
• Intelligent Operations Management for VMware Cloud Foundation 
• Intelligent Logging and Analytics for VMware Cloud Foundation 
• Documentation Enhancements: The content from VMware Validated Design documentation has now been unified with core VMware Cloud Foundation documentation or has been integrated into a VMware Validated Solution. Additional documentation enhancements include:
• Design Documents for VMware Cloud Foundation foundational components with design decisions 
• Design for the Management Domain     
• Design for the Virtual Infrastructure Workload Domain     
• Design for vRealize Suite Lifecyle and Access Management     
• Getting Started with VMware Cloud Foundation publication   
• Procedure enhancements through unification of content between VMware Validated Design and VMware Cloud Foundation publications
• Capacity Planner tool: Administrators can use the VCF Capacity Planner online tool to model and generate a Software Defined Data Center build of materials. This interactive tool generates detailed guidance of hyper-converged server, storage, network, and cloud software SKUs required to successfully deploy an on-premises cloud.
• Private APIs: Access to private APIs that use basic authentication is deprecated in this release. You must switch to using public APIs.
• BOM updates: Updated Bill of Materials with new product versions.

Bill of Materials (BOM)

If you are upgrading from previous version then please check Installation and Upgrade Information Section.

Resolved and known Issues:

Last section I focus on is resolved issues and refer attached link for known issues.

Host Creation

In my previous post about VMware Cloud Foundation version 4.2.1, I have explained the steps for creation of Nested ESXi host, and steps for preparing hosts in production also remains same.

Hence before we proceed with the next step please install and configure ESXi hosts and keep that in mind that esxi build number should be same as BOM. For version 4.3 esxi build needed is "VMware-VMvisor-Installer-7.0U2a-17867351.x86_64.iso" download customized OEM image for physical esxi hosts based on hardware vendor.

In previous versions of esxi default multipathing plugin used was NMP, but with update2a VMware has made HPP (High-Performance plugin) as default multipathing plugin hence commands which we used in my previous post about vcf 4.2.1 will no more set disk as SSD, please use these comands to set device as SSD.

esxcli storage hpp device set --mark-device-ssd=1 -d mpx.vmhba1:C0:T1:L0

esxcli storage hpp device set --mark-device-ssd=1 -d mpx.vmhba1:C0:T2:L0

esxcli storage hpp device set --mark-device-ssd=1 -d mpx.vmhba1:C0:T3:L0

esxcli storage hpp device set --mark-device-ssd=1 -d mpx.vmhba1:C0:T4:L0

NOTE: This step (marking HDD as SSD) is only needed in nested environment, do not run these commands in production environment until suggested by GSS.

Step by step video of ESXi host preparation for VCF.

Preparing Deployment parameter sheet.

Well in order to do the VCF deployment we need to do the planning, as we did in my previous post for version 4.2.1. we can use excel or json file. I used excel as its easy for each administrator. Download the excel parameter file as explained in my post or from vmware customer connect portal. Please read instructions clearly on the excel file and populate information accordingly. This sheet has only 4 tabs which needs to be populated based on the design decisions. 

These 4 tabs are Introduiction, Credentials, Host and Networks and deploy parameters.

Cloud Builder Deployment

Once our hosts are ready and parameter sheet is filled, DNS entries and NTP are in place we start deployment of cloud builder appliance. Now keep in mind that you need to provide a password for root and admin user as per complexity requirements. Or else you might have to re-deploy the cloud builder appliance, as services wont come up. Please refer my post for steps by step deployment of cloud builder.

Steps by step video of Cloud Builder appliance deployment.

Setting number of NSX-T managers for lab deployment.

If you are deploying VCF in lab environment then you would want to save resources, hence my recommendation is to keep NSX-T manager size small and change the number of managers to one instead of three. 

Note:In production environments do not reduce the number of NSXT managers, always run them in cluster of three.

In order to reduce number of manager appliances we need to first convert our parameter excel into .json file and then update the values in it.

To do that login to cloud builder UI using its url or ip address and login with admin credentials.

Accept EULA and move next.

Select platform according to hardware used.

Review pre-reqs and confirm they are validated, dont worry VCF will again validate it for us.

If you did not download parameter sheet from VMware website then you can download it now and populate it.

Now move to step 3 and upload completed excel file and press next.

Once you move next it will start validating and converting the excel file into json, hence wait for JSON Spec validation task to complete.

Once task is complete connect to Cloud builder using WINSCP and navigate to "/tmp" directory, you will find one excel and one JSON file, but as you are logged in using admin account you will not have access on it.

Now login to cloud builder using ssh with admin credentials. Elevate privileges to root with command sudo -s and navigate to /tmp directory, list all files, select json file and change permission to allow all with command "chmod 777 <filename>".

Now refresh WINSCP window and download the file.

Once file is downloaded its time to clean up current execution task. Use these commands to clean up current execution. Same commands can be used to clean up cloud builder if you frequently deploy it for your customers and use same cloud builder for bringup.

sudo psql -U postgres -d bringup -h localhost

delete from execution;

delete from "Resource";

\q

Now edit the JSON file for defining number of NSXT managers appliances. In JSON file search for "nsxtspec" and you will find below output with IP schema you have mentioned in excel sheet.

Now Remove two appliances and save the file final output should look like this.

Now we are ready to start bring up in our lab.

Step by step video of reducing NSX-T managers.

Management Domain Bring-up

As we have performed clean up of previous task, we will initiate management domain bringup. In production we will use the deployment parameter excel sheet which we have populated with the information and in Lab environment we will use the updated JSON file.

Browse cloud builder url or ip in browser and follow same steps as above for uploading the respective file as per environment and start validation.

Once validation is over if there are any warnings then review them and fix if required, finally acknowledge that and click next.

Now SDDC management domain bring up with start.

Monitor the entire process from GUI as well from the bringup log, please refer my post for steps.

Step by Step video of management domain bring up

In my next post Step-by-step VMware Cloud Foundation 4.3 design and install Edge Nodes I going to cover EDGE nodes configuration which is very essential for VMware cloud foundation north-south communication.

I hope I was able to add value, if your answer is yes, then don't forget to share and follow. 😊

If you want me to write on specific content or you have any feedback on this post, kindly comment below.

If you want, you can connect with me on Linkedin, and please like and subscribe my youtube channel VMwareNSXCloud for step by step technical videos.