ARM template for deploying windows based docker swarm in Azure

Below is ARM template as well as instructions how to deploy fully managed docker swarm into Azure based off Windows hosts for both managers and workers.

Solution along with all required files is available at following GitHub repo artisticcheese/dockerswarmarm

Clone that repo and follow with steps below

End Results

Result of  following through with steps below will be:

  • Virtual Machine Scale set with worker nodes which is joined to swarm
  • VM hosting docker swarm manager role
  • Application Gateway which will point to worker nodes for layer 7 load balancing as well HTTP/HTTPS termination, web application firewall etc
  • Azure load balancer with mapped entries for RDP access to worker nodes
  • Azure Key Vault which will hold secrets
  • Azure Automation Account which will hold DSC configurations for both worker nodes and for a swarm manager

Operation of ARM template and resources

  1. ARM template consists of main template and nested template. Main template deploys
    • Virtual Machine Scale Set (VMSS) with worker nodes
    • Application Gateway with backend pointing to VMSS for HTTP/HTTPS based termination for L7 load balancing
    • Azure load balancer which points to VMSS for RDP access to worker nodes and alternative way to load balance on L4
    • Network security group allowing RDP connectivity to both swarm manager and VMSS
    • DSC configuration to be tied to VMSS with swarm manager IP which is output of nested template below
  2. Nested template contains deployment of artifacts for swarm manager
    • Deploys swarm manager VM
    • DSC configuration for swarm manager which deploys:
      • xNetworking module to Automation Account (firewall operation)
      • cChoco (third party software installation)
      • cDSCdockerswarm module (operation automation of docker swarm)
    • Create configuration for node in Automation Account and compiles it
    • Output of nested template is internal IP for swarm manager VM which is used in main template to compile DSC configuration for VMSS

Once ARM template is completely deployed following steps will be performed on both swarm manager VM and VMSS machines:

  1. Swarmanager VM boots up and registers with Automation Account with provided automation account key and pulls DSC configuration. You can find DSC script here. DSC makes sure
    • Pulls TLS server CA, cert, key as well as TLS client cert and key from automation account. Put those into specified local file system location so local docker daemon will use those for secure local TLS endpoint
    • Configures environment variable DOCKER_CERT_PATH to point to client TLS certs above
    • Disables Windows firewall
    • Uses cDockerSwarm resource to initialize swarm
    • Installs following packages via cChocoPackageInstallerSet resource
      • Classic-Shell
      • 7zip
      • visualstudiocode
      • sysinternals
  2. VMSS nodes boot up and connect and register with Automation Account with provided automation account key and pull respective DSC configuration. You can find DSC script here. DSC performs following:
    • Copies TLS client certificates from Automation Account and saves them to local file system
    • Configures environmental variable DOCKER_CERT_PATH to point to folder where TLS client certs were saved.
    • Disables Windows Firewall
    • Uses cDockerSwarm resource to connect to existing swarm and promote themselves to managers if number of managers are below specified threshold

Prerequisites

Before ARM template can be executed some prerequisites needs to be created manually. The reason they are done manually is because this is something you want to take care like pets rather than cattle. Both Azure Automation and KeyVault is not worth automating via ARM template.

Create resource group to hold all closely guarded artifacts for docker swarm. This resource group will hold Azure KeyVault as well as Automation Account

PS C:\gd\Documents\dockerswarmarm> New-AzureRmResourceGroup -Location SouthCentralUS -Name Utility-RG

ResourceGroupName : Utility-RG
Location : southcentralus
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/b55607ab-c703-4044-a526-72bd701b0d48/resourceGroups/UtilityRG

Create KeyVault to store all the secrets in a Group. Make sure you use unique name for a vault


PS C:\gd\Documents\dockerswarmarm> New-AzureRmKeyVault -VaultName GregKeyVault -ResourceGroupName Utility-RG -Location SouthCentralUS -EnabledForTemplateDeployment

Vault Name : GregKeyVault
Resource Group Name : Utility-RG
Location : SouthCentralUS
Resource ID : /subscriptions/b55607ab-c703-4044-a526-72bd701b0d48/resourceGroups/Utility-RG/providers/Microsoft.KeyVault/vaults/GregKeyVault
Vault URI : https://GregKeyVault.vault.azure.net
Tenant ID : c0de79f3-23e2-4f18-989e-d173e1d403d6
SKU : Standard
Enabled For Deployment? : False
Enabled For Template Deployment? : True
Enabled For Disk Encryption? : False
Soft Delete Enabled? :
Access Policies :
 Tenant ID : c0de79f3-23e2-4f18-989e-d173e1d403d6
 Object ID : 6c19805a-8757-42ae-92de-02897cd7ccf9
 Application ID :
 Display Name : Gregory Suvalian (artisticcheese_gmail.com#EXT#@artisticcheesegmail.onmicrosoft.com)
 Permissions to Keys : get, create, delete, list, update, import, backup, restore, recover
 Permissions to Secrets : get, list, set, delete, backup, restore, recover
 Permissions to Certificates : get, delete, list, create, import, update, deleteissuers, getissuers, listissuers, managecontacts,
 manageissuers, setissuers, recover
 Permissions to (Key Vault Managed) Storage : delete, deletesas, get, getsas, list, listsas, regeneratekey, set, setsas, update

Create WindowsPasswordSecret  and add it to KeyVault which will be used for login to both swarmmanager nodes and VMSS machines


PS C:\gd\Documents\dockerswarmarm> Set-AzureKeyVaultSecret -VaultName GregKeyVault -Name WindowsPasswordSecret -SecretValue (ConvertTo-SecureString A123456! -AsPlainText -Force)

Vault Name : gregkeyvault
Name : WindowsPasswordSecret
Version : fbaf487667d5495e8b15c6d564f53e38
Id : https://gregkeyvault.vault.azure.net:443/secrets/WindowsPasswordSecret/fbaf487667d5495e8b15c6d564f53e38
Enabled : True
Expires :
Not Before :
Created : 4/19/2018 4:10:07 PM
Updated : 4/19/2018 4:10:07 PM

Create new Azure Automation account which will be used both as pull server as well as reporting server for all nodes in docker swarm


PS C:\gd\Documents\dockerswarmarm> New-AzureRMAutomationAccount -Name AzureAutomation -Location SouthCentralUS -ResourceGroupName Utility-RG

SubscriptionId : b55607ab-c703-4044-a526-72bd701b0d48
ResourceGroupName : Utility-RG
AutomationAccountName : AzureAutomation
Location : SouthCentralUS
State : Ok
Plan : Basic
CreationTime : 4/19/2018 11:14:56 AM -05:00
LastModifiedTime : 4/19/2018 11:14:56 AM -05:00
LastModifiedBy :
Tags : {}

Get PrimaryKey from Automation Account and create secret in KeyVault to provide to swarm nodes during build to pull their information


PS C:\> $PrimaryKey = (Get-AzureRmAutomationRegistrationInfo -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation).PrimaryKey
PS C:\> Set-AzureKeyVaultSecret -VaultName GregKeyVault -Name AzureAutomationKey -SecretValue (ConvertTo-SecureString $PrimaryKey -AsPlainText -Force)

Vault Name : gregkeyvault
Name : AzureAutomationKey
Version : 2bbab4453863413880d1607f06dc3c18
Id : https://gregkeyvault.vault.azure.net:443/secrets/AzureAutomationKey/2bbab4453863413880d1607f06dc3c18
Enabled : True
Expires :
Not Before :
Created : 4/19/2018 4:34:32 PM
Updated : 4/19/2018 4:34:32 PM
Content Type :
Tags :

VMSS members is using TLS connection to swarm manager to pull information how to join swarm. Swarm manager docker daemon is TLS secured. For this architecture to work we would need 5 files. You can find details how to create those at following post (https://artisticcheese.wordpress.com/2017/06/10/using-pure-powershell-to-generate-tls-certificates-for-docker-daemon-running-on-windows/

Total of 5 files will be required to be added to Automation Account. I have them under /certs folder for server and under /certs/clientcerts for VMSS worker nodes.

Create Automation variables which hold RSA keys for communication between nodes and swarm manager to pull secrets out.


PS C:\> New-AzureRmAutomationVariable -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation `
>> -Name ca -Value (get-Content C:\gd\Documents\dockerswarmarm\certs\ca.pem | Out-string) -Encrypted:$true

Value :
Encrypted : True
ResourceGroupName : Utility-RG
AutomationAccountName : AzureAutomation
Name : ca
CreationTime : 4/19/2018 11:59:31 AM -05:00
LastModifiedTime : 4/19/2018 11:59:31 AM -05:00
Description :

PS C:\> New-AzureRmAutomationVariable -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation `
>>  -Name privatekey -Value (get-Content C:\gd\Documents\dockerswarmarm\certs\key.pem | Out-string) -Encrypted:$true

Value                 :
Encrypted             : True
ResourceGroupName     : Utility-RG
AutomationAccountName : AzureAutomation
Name                  : privatekey
CreationTime          : 4/19/2018 5:34:49 PM -05:00
LastModifiedTime      : 4/19/2018 5:34:49 PM -05:00
Description           :

PS C:\> New-AzureRmAutomationVariable -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation `
>>  -Name servercert -Value (get-Content C:\gd\Documents\dockerswarmarm\certs\cert.pem | Out-string) -Encrypted:$true

Value                 :
Encrypted             : True
ResourceGroupName     : Utility-RG
AutomationAccountName : AzureAutomation
Name                  : servercert
CreationTime          : 4/19/2018 5:35:10 PM -05:00
LastModifiedTime      : 4/19/2018 5:35:10 PM -05:00
Description           :

PS C:\> New-AzureRmAutomationVariable -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation `
>> -Name VMSSclientkey -Value (get-Content C:\gd\Documents\dockerswarmarm\certs\clientcerts\key.pem | Out-string) -Encrypted:$true

Value :
Encrypted : True
ResourceGroupName : Utility-RG
AutomationAccountName : AzureAutomation
Name : VMSSclientkey
CreationTime : 4/19/2018 12:02:14 PM -05:00
LastModifiedTime : 4/19/2018 12:02:14 PM -05:00
Description :

PS C:\> New-AzureRmAutomationVariable -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation `
>> -Name VMSSclientcert -Value (get-Content C:\gd\Documents\dockerswarmarm\certs\clientcerts\cert.pem | Out-string) -Encrypted:$true

Value :
Encrypted : True
ResourceGroupName : Utility-RG
AutomationAccountName : AzureAutomation
Name : VMSSclientcert
CreationTime : 4/19/2018 12:02:39 PM -05:00
LastModifiedTime : 4/19/2018 12:02:39 PM -05:00
Description :

Get following information required to populate ARM template parameters file:

  • KeyVault Resource ID (Get-AzureRMKeyVault | select ResourceId)
  • Automation Account Endpoint (Get-AzureRmAutomationRegistrationInfo -ResourceGroupName Utility-RG -AutomationAccountName AzureAutomation | select Endpoint)

ARM template and instructions

ARM template expects following pieces of information

  • vmssName Name what scale set will be called and what each nodes will have as a prefix to their name
  • instanceCount Number of nodes to be created by default in VMSS
  • adminPassword Password which will be assigned to administrators account for worker nodes and swarm manager, by default pulled from KeyVault (KeyVault Resource ID above)
  • registrationURL URL to be used for DSC registration from steps above (Automation Account Endpoint)
  • registrationKey Key to register nodes with DSC pull server (PrimaryKey obtained above)
  • hostVMProfile What type of server to be used for virtual machine scale set
  • LicenseType Whether to use Hybrid Benefit for servers which are being deployed
  • AutomationAccountName Name of automation account obtained from steps above
  • AutomationAccountRGName Resource Group name of automation account
  • WorkerNodeDSCConfigURL This is URL DSC script which contains desired state for worker nodes
  • SwarmManagerNodeDSCConfigURL This is URL DSC script which contains desired state for swarm manager
  • swarmanagerdeploymenturi This is URL for nested deployment of swarm manager

Example of template file is below with relevant information filled in.

{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"vmssName": {
"value": "swarmhosts"
},
"instanceCount": {
"value": 2
},
"adminUsername": {
"value": "cloudadmin"
},
"adminPassword": {
"reference": {
"keyVault": {
"id": "/subscriptions/b55607ab-c703-4044-a526-72bd701b0d48/resourceGroups/Utility-RG/providers/Microsoft.KeyVault/vaults/GregKeyVault"
},
"secretName": "WindowsPasswordSecret"
}
},
"registrationUrl": {
"value": "https://scus-agentservice-prod-1.azure-automation.net/accounts/3459491c-abe7-4802-91d2-be4313be6272"
},
"registrationKey": {
"reference": {
"keyVault": {
"id": "/subscriptions/b55607ab-c703-4044-a526-72bd701b0d48/resourceGroups/Utility-RG/providers/Microsoft.KeyVault/vaults/GregKeyVault"
},
"secretName": "AzureAutomationKey"
}
},
"hostVMprofile": {
"value": {
"hostvmSku": "Standard_D2",
"windowsOSVersion": "Datacenter-Core-1709-with-Containers-smalldisk",
"offer": "WindowsServerSemiannual"
}
},
"LicenseType": {
"value": "Windows_Server"
},
"AutomationaccountName": {
"value": "AutomationAccount"
},
"AutomationaccountRGName": {
"value": "Utility-RG"
},
"WorkerNodeDSCConfigURL": {
"value": "https://raw.githubusercontent.com/artisticcheese/dockerswarmarm/master/swarmhost.ps1"
},
"SwarmManagerNodeDSCConfigURL": {
"value": "https://raw.githubusercontent.com/artisticcheese/dockerswarmarm/master/swarmmanager.ps1"
},
"swarmmanagerdeploymenturi": {
"value": "https://raw.githubusercontent.com/artisticcheese/dockerswarmarm/master/nestedtemplates/swarmmanagerdeployment.json"
}
}
}

Deployment script is below which creates initial resource group and then deploys entire solution into it.

#Requires -Version 3.0
Param(
[string] $ResourceGroupLocation = "southcentralus",
[string] $ResourceGroupName = 'Swarm-RG',
[string] $TemplateFile = 'azuredeploy.json',
[string] $TemplateParametersFile = 'azuredeploy.parameters-show.json'
)
If (-not (Get-AzureRmResourceGroup Name $ResourceGroupName ErrorAction SilentlyContinue)) {New-AzureRmResourceGroup Name $ResourceGroupName Location $ResourceGroupLocation Verbose }
New-AzureRmResourceGroupDeployment Name ((Get-ChildItem $TemplateFile).BaseName + '' + ((Get-Date).ToUniversalTime()).ToString('MMdd-HHmm')) `
ResourceGroupName $ResourceGroupName TemplateFile $TemplateFile `
TemplateParameterFile $TemplateParametersFile Force Verbose ErrorVariable ErrorMessages DeploymentDebugLogLevel All

view raw
deploy.ps1
hosted with ❤ by GitHub

Entire solution deployment takes about 1.5 hours due to requirement to do certain steps sequentially since you can not for example create VMSS for worker nodes before swarm manager is initialized etc.

Working with deployed swarm

You can find out if your entire swarm is properly deployed by examining Automation Account and verifying that all nodes have green checkmark next identifying successful verification of DSC configuration.

ApplicationFrameHost_2018-04-20_16-12-20

Log into manager and verify that your swarm looks healthy and showing 3 initial nodes.

PS C:\Users\cloudadmin> docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
iamctppujla19jing2j1jeyp2 swarmhost000000 Ready Active Leader
tz4kj2utkhra3e9tp3ooa5ks0 swarmhost000001 Ready Active Reachable
y1nrx7qu5mq0x9ydhdmjmpjgw * swarmmanager1 Ready Active Reachable

view raw
a.ps1
hosted with ❤ by GitHub

https://gist.github.com/artisticcheese/797b49a4de6b9a1fdf7119ea80892c0e#file-a-ps1

Since application gateway and Azure load balancer is only bound to VMSS then it’s necessary to drain primary swarm manager node to prevent it from hosting any containers

PS C:\Users\cloudadmin> docker node update availability drain swarmmanager1
swarmmanager1
PS C:\Users\cloudadmin> docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
iamctppujla19jing2j1jeyp2 swarmhost000000 Ready Active Leader
tz4kj2utkhra3e9tp3ooa5ks0 swarmhost000001 Ready Active Reachable
y1nrx7qu5mq0x9ydhdmjmpjgw * swarmmanager1 Ready Drain Reachable

view raw
a.ps1
hosted with ❤ by GitHub

Create service with global distribution mode and host based port mapping
docker service create –name iis –publish published=80,target=80,mode=host –mode global microsoft/iis:windowsservercore-1709

Verify that service was successfully created and distributed by checking number of active replicas (you shall be having 2/2 since out of 3 nodes only 2 are active)

PS C:\Users\cloudadmin> docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
odt8g82356ig iis global 2/2 microsoft/iis:windowsservercore1709

view raw
a.ps1
hosted with ❤ by GitHub

Check if worker nodes respond to HTTP calls

PS C:\Users\cloudadmin> Invoke-WebRequest swarmhost000001
StatusCode : 200
StatusDescription : OK
Content : <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta httpequiv="Content-Type" cont…
RawContent : HTTP/1.1 200 OK
AcceptRanges: bytes
ContentLength: 703
ContentType: text/html
Date: Fri, 20 Apr 2018 19:40:01 GMT
ETag: "b3cd316bd1d31:0"
LastModified: Tue, 10 Apr 2018 20:32:46 GMT
Server:…
Forms : {}
Headers : {[AcceptRanges, bytes], [ContentLength, 703], [ContentType, text/html], [Date, Fri, 20 Apr 2018 19:40:01 GMT]…}
Images : {@{innerHTML=; innerText=; outerHTML=<IMG alt=IIS src="iisstart.png" width=960 height=600>; outerText=; tagName=IMG; alt=IIS; src=iisstart.png; width=960; height=600}}
InputFields : {}
Links : {@{innerHTML=<IMG alt=IIS src="iisstart.png" width=960 height=600>; innerText=; outerHTML=<A href="http://go.microsoft.com/fwlink/?linkid=66138&amp;clcid=0x409"><IMG alt=IIS
src="iisstart.png" width=960 height=600></A>; outerText=; tagName=A; href=http://go.microsoft.com/fwlink/?linkid=66138&amp;clcid=0x409}}
ParsedHtml : System.__ComObject
RawContentLength : 703

view raw
a.ps1
hosted with ❤ by GitHub

Access those containers through provisioned Application Layer Gateway

PS C:\gd\Documents\dockerswarmarm> Get-AzureRmPublicIpAddress Name swarmhostsoei4cappgwpip ResourceGroupName swarmrg | select IPaddress
IpAddress
———
13.84.214.117
PS C:\gd\Documents\dockerswarmarm> Invoke-webrequest 13.84.214.117
StatusCode : 200
StatusDescription : OK
Content : <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta httpequiv="Content-Type" cont…
RawContent : HTTP/1.1 200 OK
AcceptRanges: bytes
ContentLength: 703
ContentType: text/html
Date: Fri, 20 Apr 2018 20:22:30 GMT
ETag: "b3cd316bd1d31:0"
LastModified: Tue, 10 Apr 2018 20:32:46 GMT
Server:…
Forms : {}
Headers : {[AcceptRanges, bytes], [ContentLength, 703], [ContentType, text/html], [Date, Fri, 20 Apr 2018 20:22:30 GMT]…}
Images : {@{innerHTML=; innerText=; outerHTML=<IMG alt=IIS src="iisstart.png" width=960 height=600>; outerText=; tagName=IMG; alt=IIS; src=iisstart.png; width=960; height=600}}
InputFields : {}
Links : {@{innerHTML=<IMG alt=IIS src="iisstart.png" width=960 height=600>; innerText=; outerHTML=<A href="http://go.microsoft.com/fwlink/?linkid=66138&amp;clcid=0x409"><IMG
alt=IIS src="iisstart.png" width=960 height=600></A>; outerText=; tagName=A; href=http://go.microsoft.com/fwlink/?linkid=66138&amp;clcid=0x409}}
ParsedHtml : mshtml.HTMLDocumentClass
RawContentLength : 703

view raw
a.ps1
hosted with ❤ by GitHub

Expand Virtual Machine Scale set to 5 members. Since new machines in virtual machines scale set are using the same ARM template as original 2 they will automatically provision all necessary software and join swarm.

PS C:\gd\Documents\dockerswarmarm> $vmss = (Get-AzureRmVmss ResourceGroupName SwarmRG Name swarmhost)
PS C:\gd\Documents\dockerswarmarm> $vmss.Sku.Capacity = 5
PS C:\gd\Documents\dockerswarmarm> Update-AzureRmVmss ResourceGroupName SwarmRG Name swarmhost VirtualMachineScaleSet $vmss

view raw
a.ps1
hosted with ❤ by GitHub

You can verify  that nodes joined to docker swarm and service is running on all nodes

PS C:\Users\cloudadmin> docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
iamctppujla19jing2j1jeyp2 swarmhost000000 Ready Active Leader
lhalb40ui3y8hguy41zhjwxtn swarmhost000008 Ready Active
tz4kj2utkhra3e9tp3ooa5ks0 swarmhost000001 Ready Active Reachable
vb2e2d3ax1mepjmnbdixv67m2 swarmhost000003 Ready Active
y1nrx7qu5mq0x9ydhdmjmpjgw * swarmmanager1 Ready Drain Reachable
zb0vx5hpmkni8qj98v4xspg10 swarmhost000002 Ready Active
PS C:\Users\cloudadmin> docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
odt8g82356ig iis global 5/5 microsoft/iis:windowsservercore1709

view raw
a.ps1
hosted with ❤ by GitHub