How to Create a Monster Build Agent in Azure for Cheap!

Posted on by Pete

I recently created a VSTS Build and Release steps for my project but ran into an issue using the Hosted Build Agents…  During Release I was doing several Azure operations in a powershell script but unfortunately I was overflowing the 30 minute limit per task for Hosted Build and Release was failing most of the time.

To workaround this 30 minute limit, we have a few options:

  • Pay for Hosted Build in VSTS.  Costs $40 per month, details here.  This gives 360 minute duration before timing out
  • Spin up a private agent on-prem.  This is free (1st one) in VSTS, but it’s another machine (or VM) that I need to manage…
  • Spin up a private agent in Azure.  This is free (1st one) in VSTS but I need to pay for the compute hours in Azure

In this case, I only need build + release twice a week, which means that if I have a private agent I’ll be wasting machine-time.  BUT, let’s take a step back and think about the principles on leveraging computing in the cloud.  Why would I waste machine-time in Azure?  Azure machines can just be turned off when they’re not in use so I don’t pay compute hours, right?  Let’s see how we can make that work for a VSTS Build Agent!  That way we get the benefits of a private agent but DON’T get the price tag that goes with it.

OPTION 1:  Use DevTest Lab Startup/Shutdown Policies (NO-CODE Solution)

A great option for managing Virtual Machines is DevTest Labs.  DevTest Labs is a FREE service in Azure that acts as a management layer on top of Azure Virtual Machines.  You can find great information on the DevTest Labs Azure page here!  In our case, we will create a Virtual Machine to run the VSTS Agent inside the DevTest Lab and then use the Auto Startup and Auto Shutdown policies to ensure the Build Agent is running when we need it.

  1. Get an Azure Subscription, try Azure for free here: https://azure.microsoft.com/en-us/free/
  2. Create a new DevTest Lab (“+ New” in the upper left and choose Developer Services –> DevTest Lab.  Choose a name, location, and type of storage
  3. Once the lab is created (should be only a few seconds!), click on the lab to open it (either from the dashboard, or under “All Resources”)
  4. Click “+ Virtual machine” on the top to create a new Virtual Machine (when created this way, the VM will be managed by the DevTest Lab)

  1. Follow the UI to create a new Virtual Machine.  I used the following:
    • Base Image:  “Visual Studio 2015 Update 3 with Azure SDK 2.9.1” – I picked this since Visual Studio + Azure SDK is already installed, it’s a good start for a VSTS Build Agent
    • Virtual machine Name:  “MyMonsterAgent”
    • Username/Password:  <it’s a secret!>
    • Virtual Machine Size:  D3 (4 core, 14 GB ram, 200 GB local SSD) – I picked a beefy VM size since the VM will only run for a few hours a week and I want FAST builds
    • Artifacts:  Use the VSTS Build Agent artifact to install the build agent.  Alternatively, you can manually set this up with these step-by-step instructions.
  2. Update the Build Definition to point at the Agent Pool you used in the prior step.  Verify that the Build Definition is triggered by a schedule (so we can align schedules with DevTest Lab Policies)
  3. Configure the Startup/Shutdown policies in DevTest Labs to align with when you’ll need the Agent for builds.  This is found by opening the lab and scrolling down until you find the policies section (see screenshot below).  In my case I start my builds at 7:15am EST on Monday & Thursday, so I start the VM ahead a of time to make sure it’s ready to run the builds.

That’s it!  You now have a Virtual Machine that’s working as a VSTS Agent but only online when you need it to save compute hours (and money)!

OPTION 2:  Script Startup/Shutdown (Powershell Solution)

There are cases where the built-in solution above won’t work for your builds (if it’s setup with Continuous Integration for example, where you get automatic builds on checkins) or if there are other reasons where you won’t know the schedule of builds ahead of time.  In this case we want to startup the VSTS Build Agent before the build and shutdown after the build automatically (via powershell).  This lets us use the machine ONLY for the required time.  The issue in doing this is that a Build Definition in VSTS runs only on a single Build Agent – so we don’t have any place to insert some ‘extra’ scripts in the beginning to start the agent during VSTS Build.

The good news is that with the latest features of VSTS, the Build and Release definitions use the same underlying infrastructure, run against the same Agents and use the same tasks!  The other good news is that VSTS Release environments CAN run on separate agents AND we can configure environments to run consecutively!  Our Release Definition will look something like this:

To accomplish this, we need to move the existing Build to a Release, hook up our Azure Subscription to the VSTS Project and then wire up the Start/Stop Environments.  Here are the steps!

  1. Move the existing Build to a Release:  Start by creating a new Release Definition in the VSTS Portal.  The UI should be very familiar (same tasks available!) to recreate the same VSTS build as an Environment in the Release Definition
  2. Connect an Azure Subscription to our VSTS Release definition.  To do this, follow the steps in this blog post
  3. Check in the powershell script into Source Code Control – powershell script to start/stop a Virtual Machine is shared at the bottom of the post
  4. Follow the steps above (Option 1 above) to create a DevTest Lab and setup a Build Agent inside the lab
  5. Create two New Environments in your build definition (one at the beginning, one at the end)
  6. In both environments, create a new Azure Powershell task to execute the powershell script with a set of parameters.  Mine are below for reference!  NOTE:  I would definite encourage turning the parameters related to the Build Agent into variables for the Release Definition.  This way they can be easily updated to a new Build Agent at a later date if needed.
    • Azure Connection Type:  Azure Resource Manager (to access DevTest Labs)
    • Azure RM Subscription:  This is the Azure Endpoint that was setup as part of step 2
    • Script Path:  Location for the script in source code control (step 3).  If you don’t find the script, click on the “Artifacts” tab and create a new linked artifact to the source
    • Script Arguments:  Arguments for the script.  Copy the correct values for your lab from the Azure Portal – here are mine for reference:
      • Environment 1:  -ResourceGroupName vsts_agentsrg141219 -DevTestLabName VSTS_Agents -BuildAgent MyMonsterAgent -Action Start
      • Environment 5:  -ResourceGroupName vsts_agentsrg141219 –DevTestLabName VSTS_Agents -BuildAgent MyMonsterAgent –Action Stop

  1. Review the Agent Pools for all the Environments to make sure they’re setup correctly.  The first and last environments should be using Hosted Agent Pool and all the other Environments should use the private agent (Default or Named Agent Pool setup earlier)
  2. Review the Deployment Triggers for all the Environments to ensure that no other Environments are deployed until the 1st one (start private agent) is successful and the last one (stop the private agent) runs AFTER the final environment is complete.

That’s it for the scripted solution!  At this point you have a working pipeline that only has the Private VSTS Build/Release Agent running in Azure when it’s needed and shutdown otherwise!

Powershell script for Starting and Stopping the DevTest Labs Virtual Machine

param
(
    [Parameter(Mandatory=$true, HelpMessage="The name of the DevTest Lab resource group")]
    [string] $ResourceGroupName,
    
    [Parameter(Mandatory=$true, HelpMessage="The name of the DevTest Lab")]
    [string] $DevTestLabName,

	[Parameter(Mandatory=$true, HelpMessage="The name of the build agent")]
    [string] $BuildAgent,

	[Parameter(Mandatory=$true, HelpMessage="Either Start or Stop to apply an action to the Virtual Machine")]
    [string] $Action
)

# find the build agent in the subscription
$agentVM = Get-AzureRmResource `
                -ResourceGroupName $ResourceGroupName `
                -ResourceType Microsoft.DevTestLab/labs/virtualmachines `
                -ResourceName $DevTestLabName `
                -ApiVersion 2016-05-15 | Where-Object {$_.Name -eq $BuildAgent}

if ($agentVM -ne $null) {

    # Update the agent via DevTest Labs with the specified action (start or stop)
    $status = Invoke-AzureRmResourceAction `
                -ResourceGroupName $ResourceGroupName `
                -ResourceType Microsoft.DevTestLab/labs/virtualmachines `
                -ResourceName ($DevTestLabName + "/" + $BuildAgent) `
                -Action $Action -ApiVersion 2016-05-15 -Force

    if ($status.Status -eq 'Succeeded') {
        Write-Output "##[section] Successfully updated VSTS Build Agent: $BuildAgent , Action: $Action"
    }
    else {
        Write-Error "##[error]Failed to update the VSTS Build Agent: $BuildAgent , Action: $Action"
    }
}
else {
    Write-Error "##[error]$BuildAgent was not found in the DevTest Lab, unable to update the agent"
}