Below is ARM template as well as instructions how to deploy fully managed docker swarm into Azure based off Windows hosts for both managers and workers.
Solution along with all required files is available at following GitHub repo artisticcheese/dockerswarmarm
Clone that repo and follow with steps below
End Results
Result of following through with steps below will be:
- Virtual Machine Scale set with worker nodes which is joined to swarm
- VM hosting docker swarm manager role
- Application Gateway which will point to worker nodes for layer 7 load balancing as well HTTP/HTTPS termination, web application firewall etc
- Azure load balancer with mapped entries for RDP access to worker nodes
- Azure Key Vault which will hold secrets
- Azure Automation Account which will hold DSC configurations for both worker nodes and for a swarm manager
Operation of ARM template and resources
- ARM template consists of main template and nested template. Main template deploys
- Virtual Machine Scale Set (VMSS) with worker nodes
- Application Gateway with backend pointing to VMSS for HTTP/HTTPS based termination for L7 load balancing
- Azure load balancer which points to VMSS for RDP access to worker nodes and alternative way to load balance on L4
- Network security group allowing RDP connectivity to both swarm manager and VMSS
- DSC configuration to be tied to VMSS with swarm manager IP which is output of nested template below
- Nested template contains deployment of artifacts for swarm manager
- Deploys swarm manager VM
- DSC configuration for swarm manager which deploys:
- xNetworking module to Automation Account (firewall operation)
- cChoco (third party software installation)
- cDSCdockerswarm module (operation automation of docker swarm)
- Create configuration for node in Automation Account and compiles it
- Output of nested template is internal IP for swarm manager VM which is used in main template to compile DSC configuration for VMSS
Once ARM template is completely deployed following steps will be performed on both swarm manager VM and VMSS machines:
- Swarmanager VM boots up and registers with Automation Account with provided automation account key and pulls DSC configuration. You can find DSC script here. DSC makes sure
- Pulls TLS server CA, cert, key as well as TLS client cert and key from automation account. Put those into specified local file system location so local docker daemon will use those for secure local TLS endpoint
- Configures environment variable DOCKER_CERT_PATH to point to client TLS certs above
- Disables Windows firewall
- Uses cDockerSwarm resource to initialize swarm
- Installs following packages via cChocoPackageInstallerSet resource
- Classic-Shell
- 7zip
- visualstudiocode
- sysinternals
- VMSS nodes boot up and connect and register with Automation Account with provided automation account key and pull respective DSC configuration. You can find DSC script here. DSC performs following:
- Copies TLS client certificates from Automation Account and saves them to local file system
- Configures environmental variable DOCKER_CERT_PATH to point to folder where TLS client certs were saved.
- Disables Windows Firewall
- Uses cDockerSwarm resource to connect to existing swarm and promote themselves to managers if number of managers are below specified threshold
Prerequisites
Before ARM template can be executed some prerequisites needs to be created manually. The reason they are done manually is because this is something you want to take care like pets rather than cattle. Both Azure Automation and KeyVault is not worth automating via ARM template.
Create resource group to hold all closely guarded artifacts for docker swarm. This resource group will hold Azure KeyVault as well as Automation Account
PS C:\gd\Documents\dockerswarmarm> New-AzureRmResourceGroup -Location SouthCentralUS -Name Utility-RG ResourceGroupName : Utility-RG Location : southcentralus ProvisioningState : Succeeded Tags : ResourceId : /subscriptions/b55607ab-c703-4044-a526-72bd701b0d48/resourceGroups/UtilityRG
Create KeyVault to store all the secrets in a Group. Make sure you use unique name for a vault
PS C:\gd\Documents\dockerswarmarm> New-AzureRmKeyVault -VaultName GregKeyVault -ResourceGroupName Utility-RG -Location SouthCentralUS -EnabledForTemplateDeployment Vault Name : GregKeyVault Resource Group Name : Utility-RG Location : SouthCentralUS Resource ID : /subscriptions/b55607ab-c703-4044-a526-72bd701b0d48/resourceGroups/Utility-RG/providers/Microsoft.KeyVault/vaults/GregKeyVault Vault URI : https://GregKeyVault.vault.azure.net Tenant ID : c0de79f3-23e2-4f18-989e-d173e1d403d6 SKU : Standard Enabled For Deployment? : False Enabled For Template Deployment? : True Enabled For Disk Encryption? : False Soft Delete Enabled? : Access Policies : Tenant ID : c0de79f3-23e2-4f18-989e-d173e1d403d6 Object ID : 6c19805a-8757-42ae-92de-02897cd7ccf9 Application ID : Display Name : Gregory Suvalian (artisticcheese_gmail.com#EXT#@artisticcheesegmail.onmicrosoft.com) Permissions to Keys : get, create, delete, list, update, import, backup, restore, recover Permissions to Secrets : get, list, set, delete, backup, restore, recover Permissions to Certificates : get, delete, list, create, import, update, deleteissuers, getissuers, listissuers, managecontacts, manageissuers, setissuers, recover Permissions to (Key Vault Managed) Storage : delete, deletesas, get, getsas, list, listsas, regeneratekey, set, setsas, update
Create WindowsPasswordSecret and add it to KeyVault which will be used for login to both swarmmanager nodes and VMSS machines
PS C:\gd\Documents\dockerswarmarm> Set-AzureKeyVaultSecret -VaultName GregKeyVault -Name WindowsPasswordSecret -SecretValue (ConvertTo-SecureString A123456! -AsPlainText -Force) Vault Name : gregkeyvault Name : WindowsPasswordSecret Version : fbaf487667d5495e8b15c6d564f53e38 Id : https://gregkeyvault.vault.azure.net:443/secrets/WindowsPasswordSecret/fbaf487667d5495e8b15c6d564f53e38 Enabled : True Expires : Not Before : Created : 4/19/2018 4:10:07 PM Updated : 4/19/2018 4:10:07 PM
Create new Azure Automation account which will be used both as pull server as well as reporting server for all nodes in docker swarm
PS C:\gd\Documents\dockerswarmarm> New-AzureRMAutomationAccount -Name AzureAutomation -Location SouthCentralUS -ResourceGroupName Utility-RG
SubscriptionId : b55607ab-c703-4044-a526-72bd701b0d48
ResourceGroupName : Utility-RG
AutomationAccountName : AzureAutomation
Location : SouthCentralUS
State : Ok
Plan : Basic
CreationTime : 4/19/2018 11:14:56 AM -05:00
LastModifiedTime : 4/19/2018 11:14:56 AM -05:00
LastModifiedBy :
Tags : {}
Get PrimaryKey from Automation Account and create secret in KeyVault to provide to swarm nodes during build to pull their information
PS C:\> $PrimaryKey = (Get-AzureRmAutomationRegistrationInfo -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation).PrimaryKey PS C:\> Set-AzureKeyVaultSecret -VaultName GregKeyVault -Name AzureAutomationKey -SecretValue (ConvertTo-SecureString $PrimaryKey -AsPlainText -Force) Vault Name : gregkeyvault Name : AzureAutomationKey Version : 2bbab4453863413880d1607f06dc3c18 Id : https://gregkeyvault.vault.azure.net:443/secrets/AzureAutomationKey/2bbab4453863413880d1607f06dc3c18 Enabled : True Expires : Not Before : Created : 4/19/2018 4:34:32 PM Updated : 4/19/2018 4:34:32 PM Content Type : Tags :
VMSS members is using TLS connection to swarm manager to pull information how to join swarm. Swarm manager docker daemon is TLS secured. For this architecture to work we would need 5 files. You can find details how to create those at following post (https://artisticcheese.wordpress.com/2017/06/10/using-pure-powershell-to-generate-tls-certificates-for-docker-daemon-running-on-windows/
Total of 5 files will be required to be added to Automation Account. I have them under /certs folder for server and under /certs/clientcerts for VMSS worker nodes.
Create Automation variables which hold RSA keys for communication between nodes and swarm manager to pull secrets out.
PS C:\> New-AzureRmAutomationVariable -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation ` >> -Name ca -Value (get-Content C:\gd\Documents\dockerswarmarm\certs\ca.pem | Out-string) -Encrypted:$true Value : Encrypted : True ResourceGroupName : Utility-RG AutomationAccountName : AzureAutomation Name : ca CreationTime : 4/19/2018 11:59:31 AM -05:00 LastModifiedTime : 4/19/2018 11:59:31 AM -05:00 Description : PS C:\> New-AzureRmAutomationVariable -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation ` >> -Name privatekey -Value (get-Content C:\gd\Documents\dockerswarmarm\certs\key.pem | Out-string) -Encrypted:$true Value : Encrypted : True ResourceGroupName : Utility-RG AutomationAccountName : AzureAutomation Name : privatekey CreationTime : 4/19/2018 5:34:49 PM -05:00 LastModifiedTime : 4/19/2018 5:34:49 PM -05:00 Description : PS C:\> New-AzureRmAutomationVariable -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation ` >> -Name servercert -Value (get-Content C:\gd\Documents\dockerswarmarm\certs\cert.pem | Out-string) -Encrypted:$true Value : Encrypted : True ResourceGroupName : Utility-RG AutomationAccountName : AzureAutomation Name : servercert CreationTime : 4/19/2018 5:35:10 PM -05:00 LastModifiedTime : 4/19/2018 5:35:10 PM -05:00 Description : PS C:\> New-AzureRmAutomationVariable -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation ` >> -Name VMSSclientkey -Value (get-Content C:\gd\Documents\dockerswarmarm\certs\clientcerts\key.pem | Out-string) -Encrypted:$true Value : Encrypted : True ResourceGroupName : Utility-RG AutomationAccountName : AzureAutomation Name : VMSSclientkey CreationTime : 4/19/2018 12:02:14 PM -05:00 LastModifiedTime : 4/19/2018 12:02:14 PM -05:00 Description : PS C:\> New-AzureRmAutomationVariable -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation ` >> -Name VMSSclientcert -Value (get-Content C:\gd\Documents\dockerswarmarm\certs\clientcerts\cert.pem | Out-string) -Encrypted:$true Value : Encrypted : True ResourceGroupName : Utility-RG AutomationAccountName : AzureAutomation Name : VMSSclientcert CreationTime : 4/19/2018 12:02:39 PM -05:00 LastModifiedTime : 4/19/2018 12:02:39 PM -05:00 Description :
Get following information required to populate ARM template parameters file:
- KeyVault Resource ID (Get-AzureRMKeyVault | select ResourceId)
- Automation Account Endpoint (Get-AzureRmAutomationRegistrationInfo -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation | select Endpoint)
ARM template and instructions
ARM template expects following pieces of information
- vmssName Name what scale set will be called and what each nodes will have as a prefix to their name
- instanceCount Number of nodes to be created by default in VMSS
- adminPassword Password which will be assigned to administrators account for worker nodes and swarm manager, by default pulled from KeyVault (KeyVault Resource ID above)
- registrationURL URL to be used for DSC registration from steps above (Automation Account Endpoint)
- registrationKey Key to register nodes with DSC pull server (PrimaryKey obtained above)
- hostVMProfile What type of server to be used for virtual machine scale set
- LicenseType Whether to use Hybrid Benefit for servers which are being deployed
- AutomationAccountName Name of automation account obtained from steps above
- AutomationAccountRGName Resource Group name of automation account
- WorkerNodeDSCConfigURL This is URL DSC script which contains desired state for worker nodes
- SwarmManagerNodeDSCConfigURL This is URL DSC script which contains desired state for swarm manager
- swarmanagerdeploymenturi This is URL for nested deployment of swarm manager
Example of template file is below with relevant information filled in.
Deployment script is below which creates initial resource group and then deploys entire solution into it.
Entire solution deployment takes about 1.5 hours due to requirement to do certain steps sequentially since you can not for example create VMSS for worker nodes before swarm manager is initialized etc.
Working with deployed swarm
You can find out if your entire swarm is properly deployed by examining Automation Account and verifying that all nodes have green checkmark next identifying successful verification of DSC configuration.
Log into manager and verify that your swarm looks healthy and showing 3 initial nodes.
Since application gateway and Azure load balancer is only bound to VMSS then it’s necessary to drain primary swarm manager node to prevent it from hosting any containers
Create service with global distribution mode and host based port mapping
docker service create –name iis –publish published=80,target=80,mode=host –mode global microsoft/iis:windowsservercore-1709
Verify that service was successfully created and distributed by checking number of active replicas (you shall be having 2/2 since out of 3 nodes only 2 are active)
Check if worker nodes respond to HTTP calls
Access those containers through provisioned Application Layer Gateway
Expand Virtual Machine Scale set to 5 members. Since new machines in virtual machines scale set are using the same ARM template as original 2 they will automatically provision all necessary software and join swarm.
You can verify that nodes joined to docker swarm and service is running on all nodes
