Moving Azure VM into availability set after VM creation

VMs created in Azure can be put into availability set (AS) only at creation time and require complete recreation from scratch if you want to add them to availability set if required.

Script below will allow to move VM into AS after it was created. There is bunch of similar script on internet but all of them relying on PS/CLI to attach/reattach NIC etc. Script below instead relies on native Azure functionality of exporting ARM representation of VM and modifying it with adding availability set to VM. Advantage of that method that a bunch of properties of resource is preserved which will be otherwise lost with PS/CLI approach (tags, extensions, caching info for disks etc)

File is available here or below (https://raw.githubusercontent.com/artisticcheese/artisticcheesecontainer/master/update-as.ps1)

#Requires -Version 7
#Script below would allow to add VM to availability set after VM was already deployed without it. 
#Process consists of exporting existing VM tempalte, modifying it's parameters and then importing it again
#Availability set shall already exist and be in the same resource group as VM resides
#Following paramters are are required: $vmName -> name of VM, $resourceGroupNam -> Name of resource group where VM and availability set is located, $availabilitySet -> name of availability set
#Once script is run Vm is removed and you are left with .\template.deploy.json file which you need to create a new deployment from with New-AzResourceGroupDeployment
#Example
#.\Update-AvailabilitySet.ps1 -vmName MyVm1 -resourceGroupName MyResourceGroup-RG -availabilitySet myAvailabilitySet
# New-AzResourceGroupDeployment -TemplateFile .\template.deploy.json -ResourceGroupName myResourceGroup-RG



[CmdletBinding()]
param (
   [Parameter(Mandatory = $true)] [string] $vmName,
   [Parameter(Mandatory = $true)] [string] $resourceGroupName,
   [Parameter(Mandatory = $true)] [string] $availabilitySet
)
$VerbosePreference = "Continue"
if ($null -eq (Get-AzContext)) { Login-AzAccount }
$ErrorActionPreference = "Stop"
$resource = Get-AzVM -ResourceGroupName $resourceGroupName -VMName $vmName 
$fileName = Join-Path (Get-Location) ".\template.json"
Export-AzResourceGroup -ResourceGroupName $resource.ResourceGroupName -Resource $resource.Id -IncludeParameterDefaultValue -IncludeComments -Path $fileName -Force
$templateTextFile = [System.IO.File]::ReadAllText($fileName)
$TemplateObject = ConvertFrom-Json $templateTextFile -AsHashtable
$computerObject = $TemplateObject.resources.where{ $_.type -eq "Microsoft.Compute/virtualMachines" }   
$computerObject[0].apiVersion = "2020-06-01"
if ($null -eq $computerObject.properties.availabilitySet) {
   $computerObject.properties.Add("availabilitySet", "")
}
$computerObject.properties.availabilitySet = @{ "id" = "[resourceId('Microsoft.Compute/availabilitySets', '$availabilitySet')]" }      
$computerObject.properties.storageProfile.dataDisks.ForEach{ $_.createOption = "Attach" }
$computerObject.properties.storageProfile.osDisk.createOption = "Attach"
$computerObject.properties.storageProfile.Remove("imageReference")
$computerObject.properties.storageProfile.osDisk.Remove("name")
$computerObject.properties.Remove("osProfile")
$TemplateObject | ConvertTo-Json -Depth 50 | Out-File -Path (Join-path (Get-Location) ".\template.deploy.json")
$resource | Stop-AzVM -Force
$resource | Remove-AzVM -Force
if ($env:POWERSHELL_DISTRIBUTION_CHANNEL -eq "CloudShell") {
   New-AzResourceGroupDeployment -TemplateFile (Join-path (Get-Location) ".\template.deploy.json") -ResourceGroupName $resourceGroupName
}

Circumventing Windows containers limitation on volume mounts

Windows containers can not map files into container (only directory mounting is allowed). This commonly required to provide sensitive data to container are run time (example are connection strings in web.config or any other values needed for application which shall not end up in container image).

Common solution to this is to provide environment variable which indicate the location of secret configuration file and then override ENTRYPOINT in image to copy/process information before application starts. Something like below which check presence of environment variable ConfigPath and if it’s present then copy that file dynamically at runtime to override base image configuration file.

if($env:ConfigPath){
    Copy-Item -path $env:ConfigPath -Destination c:\inetpub\wwwroot -Force
}

This approach has major drawback – any update to volume data will not be visible to application since this script only runs at container start up.

To curcumvent this limitation you can create symbolic link instead pointing to desired target location. This approach will still allow you to have base configuration file in place for local debuggin etc but it will be overriden at runtime with symlink to desired secret.

Steps and verification are below (based on Azure AKS implementation but this will work accross any kubernetes or other orchestration or local solution)

  1. Create AKS cluster with windows node pool
  2. Create deployment and secret based on this file
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secret-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: secret
  template:
    metadata:
      labels:
        app: secret
    spec:
      containers:
        - name: secret-container
          image: mcr.microsoft.com/dotnet/framework/aspnet:4.8-windowsservercore-ltsc2019
          env:
            - name: ConfigLocation
              value: c:\secret\web.config
          command: ["powershell"]
          args:
            - 'if ($env:ConfigLocation) {New-Item -Path C:\inetpub\wwwroot\Web.config -ItemType SymbolicLink -Value $env:ConfigLocation -force -Verbose}; & "C:\ServiceMonitor.exe" "w3svc"'
          volumeMounts:
            - name: secret
              mountPath: "secret"
      volumes:
        - name: secret
          secret:
            secretName: secretconfig
      nodeSelector:
        kubernetes.io/os: windows
      tolerations:
        - key: kubernetes.io/os
          operator: Equal
          value: windows
          effect: NoSchedule
---
apiVersion: v1
kind: Secret
metadata:
  name: secretconfig
type: Opaque
stringData:
  web.config: |
    <?xml version="1.0" encoding="utf-8"?>
    <configuration>
      <location path="." inheritInChildApplications="false">
        <system.webServer>
          <handlers>
            <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
          </handlers>
          <aspNetCore processPath=".\MyApp.exe"
                      stdoutLogEnabled="false"
                      stdoutLogFile=".\logs\stdout"
                      hostingModel="inprocess" />
        </system.webServer>
      </location>
    </configuration>

Deployment above creates secret with contents of desired web.config file, volume mapping into container and then overrides ENTRYPOINT to create symbolic link to secret.

Inside Kubernetes in fact similar concept is used to map secret name to a value. Executing below shows that actual secret web.config is actually a link to ..\data\web.config with directory ..data is in turn symbolink to dynamically named folder.

PS C:\Users\artis> kubectl exec deploy/secret-deployment powershell "get-childitem c:\secret | select name, linkType, target"
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

Name                            LinkType     Target
----                            --------     ------
..2021_05_02_17_05_33.369337563
..data                          SymbolicLink {..2021_05_02_17_05_33.369337563}
web.config                      SymbolicLink {..data\web.config}

You can also verify that web.config is in fact mapped to desired value in container

PS C:\Users\artis> kubectl exec deploy/secret-deployment powershell "get-content c:\inetpub\wwwroot\web.config"
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath=".\MyApp.exe"
                  stdoutLogEnabled="false"
                  stdoutLogFile=".\logs\stdout"
                  hostingModel="inprocess" />
    </system.webServer>
  </location>
</configuration>

Last thing to check if dynamic mapping of secret into running container. Changing slightly secret configuration file and redeploying it in fact shows new version of configuration item and in turn means IIS will pick up a change and will reload application domain.

PS C:\Users\artis> kubectl exec deploy/secret-deployment powershell "get-content c:\inetpub\wwwroot\web.config"
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  New version
  </location>
</configuration>

L2TP VPN via ARM template in Azure

Virtual Network Gateway in Azure (native Azure VPN solution) have numerous drawbacks vs hosted VPN solution. I would say following are main drawbacks of native VPN solution:

  • Long time to provision (per documentation you expect up to 45 minute provisioning time)
  • Expensive if you need higher bandwidth or bigger number of simultaneous clients
  • Takes at least /29 IP address space
  • VPN client can not be used remote gateway mode (split tunnel disabled). As a result:
    • Can not use Azure provided DNS service (168.63.129.16)
    • Can not use directly private endpoints by their names
    • Security risk since VPN client becomes a switch connecting external Internet with your VNET
  • Can not limit incoming connections based on IP addresses (NSGs are not allowed on VPN gateway subnets)
  • Can not use simple username/password authentication (requires either certificate based authentication or Azure AD based)
  • Requires to download and install file from Azure or Microsoft store

Solution outlined below deploys VPN server based on Windows 2019 Server core image with RRAS service installed with L2TP VPN. I tested it on smallest VM compute size (Standard_B1s) and had no issues reaching 200MBps. Server Core install runs surprisingly good on 1CPU/1GB RAM. Total monthly cost of config is $12 which is half the price of the cheapest tier of Azure VPN gateway with no licenses limitation for either bandwidth or number of concurrent sessions.

Windows 10 client requires following AssumeUDPEncapsulationContextOnSendRule with value of 2 and reboot. You can set this key with powershell below

Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\PolicyAgent AssumeUDPEncapsulationContextOnSendRule -Type DWord -Value 2 -Force

To force VPN connection to use VPN provided DNS service interface metric for Ethernet adapter of client shall be set to higher number then VPN client one. Substitute name of your adapter which you can get by executing Get-NetIPInterface | Select-Object -Property InterfaceAlias, InterfaceMetric

Set-NetIPInterface -InterfaceAlias "Ethernet" -InterfaceMetric 30

Complete ARM template is available at https://github.com/artisticcheese/Azure/tree/main/ARM-L2TP

Template consists of following resources:

  • Network security group which allows only L2TP VPN protocol ports
  • VM with Windows2019-ServerCore-smalldisk SKU
  • Custom script extension which configures server for VPN use
  • Template outputs FQDN for public IP of server for configuration of VPN client

Steps to deploy custom VPN server:

  1. Download template and parameters file from repo
  2. Modify parameters file which matches your network and with your desired password for preshared key and password for user account
  3. Execute powershell to start deployment
 New-AzResourceGroupDeployment -ResourceGroupName RRAS-RG -TemplateFile .\ARM-L2TP\template.json -TemplateParameterFile .\ARM-L2TP\template.parameters.json -Verbose
  1. As part of deployment server is rebooted and bearing in mind server is of smallest size you shall expect total execution time of about 10-20 minutes
  2. Once execution completes you need to configure Windows 10 VPN client:
  • Navigate start menu and type VPN, click Add VPN Connection button
  • Fill in the menu name, for server address use FQDN output of ARM template. For VPN type L2TP/IpSec with preshared key and username/password from parameters file

You shall be able now to connect to VPN and test functionality.

  1. Check that you route through VPN by checking what your connection appear as coming from when reaching Internet
PS C:\Users\artis> Invoke-RestMethod http://ipinfo.io | select Region

region
------
Virginia
  1. Check if you can resolve private endpoint IP addresses to internal addresses. I created storage account and created private endpoint for it which shall resolve to IP address on my VNET.
PS C:\Users\artis> resolve-dnsname rrasstorageaccount.blob.core.windows.net

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
rrasstorageaccount.blob.core.w CNAME  60    Answer     rrasstorageaccount.privatelink.blob.core.windows.net
indows.net

Name       : rrasstorageaccount.privatelink.blob.core.windows.net
QueryType  : A
TTL        : 1800
Section    : Answer
IP4Address : 10.0.0.5

ARM deploymentScripts resource is GA

There is new resource in Azure called Microsoft.Resources/deploymentScripts (https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-script-template) which fills void of ability to run custom code as part of deployment where ARM code on it’s own is not sufficient.

In the background Azure creates ACI container for you with Managed Identity and Az CLI/Powershell tools so you can pretty much do anything your heart desired which you could have accomplished from command line.

Example which I run into and which was solved with new resource is ability to calculate future date with specific time where built-in time functions of ARM templates are insufficient.

Azure automation account softwareUpdateConfigurations resource requires scheduleInfo startTime property to be specified in future only which on it’s own not difficult to implement in just ARM date functions but impossible to set it to specific time of the day. (Say 10 AM EST which is required for recurrence of patching cycle). This is where sample use of deploymentScripts is shining since powershell can easily tackle this setup to calculate tomorrows day at specific time.

Example below adds startTime property to be tomorrows day at specific time as required by passing parameter to powershell script.

Template is below with highligted line referencing output of deploymentScripts providing information about next date/time.

  {
         "type": "Microsoft.Automation/automationAccounts/softwareUpdateConfigurations",
         "apiVersion": "2017-05-15-preview",
         "copy": {
            "name": "deploymentScriptcopy",
            "count": "[length(parameters('other').schedule)]"
         },
         "name": "[concat(parameters('management').Automation.name, '/', parameters('other').schedule[copyIndex()].Name)]",
         "dependsOn": [
            "[resourceId('Microsoft.Automation/automationAccounts', parameters('management').Automation.name)]",
            "[concat('GetNextScheduledDate-script-', parameters('other').schedule[copyIndex()].Name)]"
         ],
         "properties": {
            "updateConfiguration": {
               "operatingSystem": "Windows",
               "windows": {
                  "includedUpdateClassifications": "Critical, Security, UpdateRollup, FeaturePack, ServicePack, Definition, Tools, Updates",
                  "rebootSetting": "IfRequired"
               },
               "targets": {
                  "azureQueries": [
                     {
                        "scope": "[parameters('other').subscriptionList]",
                        "tagSettings": {
                           "tags": {
                              "patchgroup": [
                                 "[parameters('other').schedule[copyIndex()].Name]"
                              ]
                           },
                           "filterOperator": "All"
                        },
                        "locations": []
                     }
                  ]
               },
               "duration": "PT2H"
            },
            "scheduleInfo": {
               "startTime": "[reference(concat('GetNextScheduledDate-script-', parameters('other').schedule[copyIndex()].Name)).outputs.text]",
               "expiryTime": "9999-12-31T17:59:00-06:00",
               "interval": 1,
               "frequency": "Month",
               "timeZone": "UTC",
               "advancedSchedule": {
                  "monthlyOccurrences": [
                     {
                        "occurrence": "[parameters('other').schedule[copyIndex()].weekNumber]",
                        "day": "[parameters('other').schedule[copyIndex()].weekDay]"
                     }
                  ]
               }
            }
         }
      }

DeploymentScripts code is below which takes as a parameter Hour variable and returns next date at that specific hour via $DeploymentScriptOutputs['text'] variable

 {
         "type": "Microsoft.Resources/deploymentScripts",
         "apiVersion": "2020-10-01",
         "copy": {
            "name": "deploymentScriptcopy",
            "count": "[length(parameters('other').schedule)]"
         },
         "name": "[concat('GetNextScheduledDate-script-', parameters('other').schedule[copyIndex()].Name)]",
         "location": "[resourceGroup().location]",
         "kind": "AzurePowerShell",
         "properties": {
            "forceUpdateTag": "1",
            "azPowerShellVersion": "5.0",
            "scriptContent": "
             param (
                [Parameter(Mandatory = $true)]
                $Hour
             )
             $output = (Get-Date -Hour $Hour -Minute 0 -Second 0).AddDays(1)
             $DeploymentScriptOutputs = @{}
             $DeploymentScriptOutputs['text'] = $output
              ",
            "arguments": "[concat(' -Hour ', parameters('other').schedule[copyIndex()].Hour)]",
            "timeout": "PT1H",
            "cleanupPreference": "OnSuccess",
            "retentionInterval": "P1D"
         }
      },