When you are running multiple projects you might start to lay awake at night with deep questions such as:
- «How do I get my solution back online after I, I mean, someone else deleted the wrong resource group?»
- «What if someone does a manual change and that change doesn’t carry up through the environments when doing a release?»
- «How can I quickly deploy a new environment easily without having to be a devops supergirl?»
As sleep is important, I thought I might offer one possible answer to these questions (and many variants to them) that we’ve implemented for our client, Statoil.
The point is basically to leave all the provisioning, configuration and deployment of an Azure application to a fully automated procedure.
Now, Azure provides a very powerful functionality for provisioning resources through the use of ARM templates. This gets you quite far with automatic deploys.
However, templates have their limitations and some tasks are still manual – either through scripting or doing some magic gestures in the Azure Portal.
This is something we worked on at the Microsoft Xamarin Hackfest and we got pretty far using PowerShell-scripts.
There was still some special knowledge required and certain tools to be installed on the machine doing the deployment.
We wanted to move away from running scripts manually and basically go to one-click deployments.
We also didn’t want everyone to have Owner roles on the Azure subscription in order to minimize human error when clicking around in the portal and suddenly bringing down applications in production.
This was finally made possible with the introduction of a new Azure AD permission.
This permission enables the application to not only create Azure AD Applications, but also query Azure AD for the information it needs to do certain tasks.
By granting this permission to a service principal, we could enable VSTS to perform all the tasks we previously had to do manually.
In order to run our custom tasks, we needed our fabled service principal.
Now, thats a lot of lines, but don’t worry! It’s not as complicated as you might think.
The service principal itself can be created automatically by VSTS, but we found that creating one manually gave us more control. Once you have that, you configure a new service endpoint in VSTS.
This service principal then needs the «Manage apps that this app creates or owns» permission in Azure AD added to it. To grant this permission, you need an Azure AD global admin – if thats not you, make sure you have your admin happy and available.
One thing to note is that it appears it can take up to 24 hours for this to take effect – or as little as 10 seconds, no idea why.
Next up! Create our own VSTS extension.
Basically, the custom tasks are just our previous scripts wrapped inside a VSTS task.
At the time of writing there are two tasks.
The first task we needed was one to automate the creation of a Azure AD Application which is used to integrate our applications with Azure AD for authentication.
This task will create or update an Azure AD Application and set a variable $(APPLICATIONID) which can be re-used in subsequent tasks.
It also allows for the creation of a service principal for the application using certificate credentials – so that the application can be granted access to Azure Key Vault and other resources.
Without credentials, an Azure AD Application is merely a entry in Azure AD you can request access tokens for. Once you add credentials, either password or certificate, you basically enable the application to become a user of sorts.
Next, we had an issue where we had deployed our ARM template, but there was still something missing. Since we deploy our Key Vault in the same ARM template, we don’t have an existing one with the certificate already in place – if we had, we could do something like this.
Instead, our goal was to get everything up and running from scratch fully automated. That meant after the Key Vault was created, we needed to upload the certificate to it and then configure the WebApp to use it. As far as I know, you cannot deploy a certificate to Key Vault using ARM templates. So we were in a chicken and egg kind of situation.
To automate this, you have to use the Azure REST API as described here.
So we wrapped it all up as a custom task instead.
The Build and Release definitions
In this particular instance, we are using ASPNet.Core and so the build definition is basically the standard VSTS template for ASPNet.Core with the added task of Copy Files which which copies the ARM template and its parameter files to the artifact drop point.
This is so that we can reference the ARM template in the same release definition as the actual deploy of the code.
You can separate these and then you don’t need the extra task on the build definition.
We included it here because of one special release definition which will deploy the actual Azure environment before deploying the code so that we can do nightly rebuilds of the development environments ensuring that only the configurations in the template and VSTS configuration are valid and the code runs (avoiding those pesky «I just played around with some settings in the Azure environment and now something isn’t working right»).
This kind of release definition can then also be used for disaster recovery and bringing more environments online in case of load balancing across geographic areas.
The release definition is where it gets interesting.
By including our custom tasks for creating the Azure AD Application we can then do a replacement in the configuration for our code so it will always have the correct OAuth audience when doing authentication with Azure AD.
As I said earlier, the Create Application task will expose a variable with the correct application id which then the Deploy Azure App Service task can use to do a JSON transform. By specifying a variable using JSONPath expression, the task is able to do a replace in your appsettings.json.
Our appsettings.json has a section like this:
Setting a VSTS variable named AzureAd.Audience to $(APPLICATIONID) will then ensure the configuration is correct.
Now you can configure your release definition with different approvals for each environment so that anyone re-deploy/update development environments, but only certain key individuals can approve the same for production. And since everything is fully automated you should not get any surprises moving from development up to production.
Depending on the resources in your ARM template, you can do a full provisioning of all your environments in a matter of minutes!