Integrating docker release management in TFS with webtests

TFS express 2017 provides free and fully integrated environment for continious integration and release management for docker windows containers. Post below only focuses on release management for docker containers via TFS and webtests produced by Visual Studio Enterprise edition.

My current setup of TFS build produces following artifacts available for release management.

  • docker-stack.yml

This file presented below provides information about build version of container image and in imagetag and additional information for running environment

version: '3.2'
target: 80
published: 8080
protocol: tcp
mode: host
type: volume
source: webredirect
target: c:\inetpub\logs\logfiles\host
type: volume
source: webredirect-freb
target: c:\inetpub\logs\logfiles\freb
mode: global
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
parallelism: 1
constraints: [node.role != manager]
external: true
external: true
external: true

view raw
hosted with ❤ by GitHub

  • tests folder with 3 files
    • local.testsettings
      • This file provides information which server webtest have to be running against
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Local" id="8912d4fb-1ed0-4d91-98cf-787f3ffb50aa" xmlns="">
<Description>These are default test settings for a local test run.</Description>
<Deployment enabled="false" />
<TestTypeSpecific >
<WebTestRunConfiguration testTypeId="4e7599fa-5ecb-43e9-a887-cd63cf72d207" simulateThinkTimes="true" />
<Property name="TestSettingsUIType" value="LoadTest" />

view raw
hosted with ❤ by GitHub

    • webtest1.webtest
      • This file provides actual steps which is tested
<?xml version="1.0" encoding="utf-8"?>
<WebTest Name="WebTest1" Id="2b0301fa-6605-4d91-9c11-9798ac868ba5" Owner="" Priority="2147483647" Enabled="True" CssProjectStructure="" CssIteration="" Timeout="0" WorkItemIds="" xmlns="" Description="" CredentialUserName="" CredentialPassword="" PreAuthenticate="True" Proxy="default" StopOnError="True" RecordedResultFile="" ResultsLocale="">
<Request Method="GET" Guid="1ffa095e-785d-4935-ac00-ecdf45081b34" Version="1.1" Url="{{EndpointName}}/" ThinkTime="0" Timeout="300" ParseDependentRequests="False" FollowRedirects="False" RecordResult="True" Cache="False" ResponseTimeGoal="0" Encoding="utf-8" ExpectedHttpStatusCode="301" ExpectedResponseUrl="" ReportingName="" IgnoreHttpStatusCode="False" />
<Request Method="GET" Guid="1ffa095e-785d-4935-ac00-ecdf45081b34" Version="1.1" Url="{{EndpointName}}/mssplus.txt" ThinkTime="0" Timeout="300" ParseDependentRequests="False" FollowRedirects="False" RecordResult="True" Cache="False" ResponseTimeGoal="0" Encoding="utf-8" ExpectedHttpStatusCode="200" ExpectedResponseUrl="" ReportingName="" IgnoreHttpStatusCode="False" />
<ContextParameter Name="EndpointName" Value="" />

view raw
hosted with ❤ by GitHub

  • runtest.ps1
    • This file which performs 2 functions: waits for swarm manager to bring latest version of image up and then run webtests tests on those.
$tool = "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\MSTest.exe",
$path = $psscriptroot,
$item = "$path\webtest1.webtest",
$results = "$path\webtest.trx",
$testsettings = "$path\Local.testsettings",
$expectedReleaseHeader = "webredirect:20170726_1050",
$urlTest = ""
while ($true)
$null = Invoke-WebRequest $urltest UseBasicParsing MaximumRedirection 0 OutVariable SLheaderOut ErrorAction SilentlyContinue
Write-Output $_
if ($Slheaderout)
$SLheader = $Slheaderout.headers.SL
$Slheader = $null
if ($SLheader -ieq $expectedReleaseHeader )
Write-Output "Found $SLheader while expecting $expectedReleaseHeader"
Start-Sleep 5
Write-Output "Found $Slheader on $urlTest, can proceed with the rest of tests"
${env:Test.EndpointName} = $urlTest
& $tool /TestContainer:$item /resultsfile:$Results /testsettings:$testsettings /noisolation
#Temporary measure untill VSTS is fixed
New-Item Name "webtest" Path $psscriptroot ItemType Directory
move-item Path (join-path path $path Childpath (get-childitem Path $path Name "in" Directory Recurse)) Destination $path\webtest
#Install evaluation version of VS 2017 Enterprise

view raw
hosted with ❤ by GitHub

Release pipeline itself contains of 3 steps for each environment. Screenshot below for staging environment.


First step is pushing newly built image to staging docker swarm. This step is using built-in Docker command tasks in TFS and passing information about swarm location and registry connection via built in TFS properties along with docker-stack.yml file above.


Second step is powershell script named runtest.ps1above. Variables  are passed to script to specify which server needs to be checked against as well current version of new build version of docker image


Last step is publishing results of those tests


Below is screenshot what failure in webtest looks like. Along with failure also attached webtestresult file which you can open in Visual Studio to inspect details of what has failed.


Please note to enable test run you need to have mstest.exe installed on TFS agent as part of Visual Studio Enterprise 2017 installation. You can use Evaluation Version to install it. You need to start Visual Studio at least once under user which agent is running under to fix the issue below (if your agent runs as System service then you need to run Visual Studio at least one a System). Test will fail otherwise with File extension specified .webtest is not a valid test extension.

Another thing to note is that there is currently a bug in TFS 2017 U2 which does not properly pull results form TRX file and hence additional logic was added to powershell script to put results in correct location to Publish Test Results tag can find results.


2 thoughts on “Integrating docker release management in TFS with webtests

  1. Thanks for the Article. It’s very useful. Do you need to install full license version of Visual Studio Enterprise 2017? Or evaluation version is enough?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s