ARM Template Building Blocks and Reference Architectures

The AzureCAT team actively assists customers with the largest, most complex projects built on the Azure platform.  In addition to the super work the team does helping Azure customers the AzureCAT team have released a set of ARM templates which can be used as building blocks for ARM deployments.  The team have made these templates available on GitHub and have also released a set of reference architectures that use these ARM building blocks.

What’s really nice, is that the ARM building blocks the AzureCAT team have created, are based on the work they have done with real customers.  This means these templates are road tested and have a huge amount of customer learnings and good practices ‘baked in’ to them.

The AzureCAT team has made extensive use of nesting and arrays within their ARM templates to reduce some of the complexity inherent in ARM.

Even if you don’t use the templates directly, they are a great place to start when building your own templates and offer an interesting insight into learnings the AzureCAT team has gained over the last two years.

Current building blocks released are:

Building block Link Description
Virtual network vnet-n-subnet Used to create a virtual network with any number of subnets
Network security groups networkSecurityGroups Used to create any number of NSGs, and link them to any number of NICs and/or subnets
User defined routes userDefinedRoutes Used to create any number of UDR tables, and link them to any number of subnets
Gateway connection vpn-gateway-vpn-connection Used to create a VPN or ExpressRoute gateway and necessary connections to another network
Virtual machines multi-vm-n-nic-m-storage Used to create any number of VMs, each with any number of NICs, and any number of data disks
Load balanced workload loadBalancer-backend-n-vm Used to create a load balancer with a collection of VMs in the backend
DMZ dmz Used to create a DMZ between an Azure VNet and any other network, or the Internet

 

Using ARMclient to directly access Azure ARM REST API’s and list ARM Policy details

While Azure-PowerShell and the Azure xplat CLI are excellent tools, there are times when connecting directly to the Azure ARM REST API is just easier.   One of these times is when you want to find out the contents of an Azure Resource Manager Policy.  While Its easy to use PowerShell to find out the policies applied to a Resource Group.

get-azureRMpolicyassignment -Scope /subscriptions/$mysubscription/resourceGroups/$myresourcegroup

Getting the details as to the contents of an Azure Resource Manager Policy can be quite a bit trickier.  This is one situation where calling the ARM Policy provider via the ARM REST API’s directly is the easiest answer.

ARMclient is an OSS project which makes the task of connecting to Azure’s ARM REST API incredibly easy.  David Ebbo has a great blog article here on the simple steps to install ARMClient.

Once ARMclient is installed its a simple matter of logging in to the Azure tenant and then calling the provider.  In this case,  /Microsoft.authorization/policydefinitions to list the ARM Policies for a given subscription.

armclient login
armclient get "https://management.azure.com/subscriptions/<$YourSubscriptionID>/providers/Microsoft.authorization/policydefinitions?api-version=2016-04-01"

Depending on the number of ARM Policies in the subscription the output from the ARMclient can be quite large.  If you know the name of the ARM Policy you whish to see the details for you can append this to the call that you make to the API.

armclient get "https://management.azure.com/subscriptions/<$YourSubscriptionID>/providers/Microsoft.authorization/policydefinitions/tags-owner?api-version=2016-04-01"

This will display the contents of just that policy.

armclient get "https://management.azure.com/subscriptions/<$YourSubscriptionID>/providers/Microsoft.authorization/policydefinitions/tags-owner?api-version=2016-04-01"
{
  "properties": {
    "policyType": "Custom",
    "description": "Policy to set owner tags",
    "policyRule": {
      "if": {
        "allof": [
          {
            "field": "tags",
            "exists": "true"
          },
          {
            "field": "tags.owner",
            "exists": "false"
          }
        ]
      },
      "then": {
        "effect": "append",
        "details": [
          {
            "field": "tags.owner",
            "value": "Daniel"
          }
        ]
      }
    }
  },
  "id": "/subscriptions/<$YourSubscriptionID>/providers/Microsoft.Authorization/policyDefinitions/tags-owner",
  "type": "Microsoft.Authorization/policyDefinitions",
  "name": "tags-owner"
}

As you can see the output passed back by the API is exactly the same JSON format which is used when creating the ARM Policies .

There is heaps that you can do with the ARMclient, one of my favourites is to list out all the resources in an Azure subscription.

armclient get "https://management.azure.com/subscriptions/<@YourSubscriptionID>/resources?api-version=2014-04-01"

As the output is in JSON format its very easy to manipulate as required.  You can also convert the output from ARMclient to a PowerShell object using ConvertFrom-JSON in your PowerShell scripts.

The following command will list just the virtual machines in a subscription.

armclient get "https://management.azure.com/subscriptions/<$YourSubscriptionID>/providers/Microsoft.compute/virtualMachines?api-version=2016-03-30"

To get a list of all available ARM providers which you can call using the ARM REST API, you can run the following ARMclient command.

armclient get "https://management.azure.com/subscriptions/<$YourSubscriptionID>/providers?$expand=resourceTypes/aliases&api-version=2015-11-01"

This will also give you the API versions supported by the different providers.  An API version must be appended to the end of all calls to the ARM REST API.

As well as ‘get’ commands, the ARMclient can also be used to modify Azure ARM resources by using ‘post’, ‘put’ and ‘delete’ operators.  The website resources.azure.com exposes similar functionality to the ARMclient.

Azure spend now shows in subscription blade in Azure Portal

Microsoft have just released a nice little update to the Azure Portal to show your current spend on the subscription blade of the Azure portal. You used to have to go to the billing blade and be a billing administrator to see this information. The subscription blade only requires staff to have read access to the subscription to see the current and historical spend and they can’t modify the subscription in any way or see payment methods or billing address.

sub-blade-spend

The subscription blade can be found by clicking on more services at the bottom of the left hand menu and using the filter or scrolling down to the General category. Clicking the star next to subscription will add it to your quick access menu on the left hand side of the screen.

sub-blade

Microsoft have also enabled the ability to see the Azure Marketplace costs associated with each subscription. This information can be accessed under the External Service button which is visible when clicking on the subscription in the subscription blade.

sub-blade-detail

I really like the burn rate graph which is displayed on the detailed subscription blade, this is especially handy if you have an MSDN subscription as it gives you an indication if you are going to hit your cap or not ! For those that have not used an MSDN subscription you get a monthly Azure spend/cap and if you go over your cap everything gets shutdown (unless you put your credit card in) until your billing period roles over which can be embarrassing if you are in the middle of a demo :(.

Quickly switching between blades in Azure Portal

Since transitioning out of its beta status Microsoft has continued to update and develop portal.azure.com, which Microsoft staff commonly refer to as the Ibiza portal.  As more and more services which Microsoft call Resource Providers are added to the portal it can get quite confusing as to where you are and time consuming to move around between resource providers.

To make it easier to navigate, the Ibiza portal displays at the top of the page, the journey you have taken through the different blades associated with a resource.  you can back track to any blade by clicking on the name of that blade.

ibiza-journey

You can also quickly jump between resource provider blades by clicking on the chevron next to the Microsoft Azure logo at the top right of the page.  This will show you the resource providers that you have accessed during your current session.

ibiza-blade-history

You can quickly jump between resources providers by clicking on them.

ibiza-blade-history-2This makes it super easy for instance when you are testing ASR failover and want to check to see if your VM has shown up under virtual machines.  Or if you want to check Azure security center and then jump back to check on an Azure Automation job.

My final tip for navigating around the Ibiza portal is to use the minimise action to shrink blades back to a single bar.  This can be a good way clean up screen space with out having to do lots of left and right scrolling.

ibiza blade minimise.jpg

 

Azure Resource Manager Policy to add cost center and owner tags

Azure Resource Manager (ARM) allows you to enforce organisational standards through the use of custom policies.  There are a number of things that you can do with ARM policies.  These range from restricting the size of virtual machines and the location they can be deployed, through to ensuring standardised naming conventions.

ARM Policies are made up of conditions, logical operators and effects.  Policies can be applied to a subscription, resource group or resource.

{
  "if" : {
      <condition> | <logical operator>
  },
  "then" : {
      "effect" : "deny | audit | append"
  }
}

As it currently stands, if an ARM Policy is applied which requires specific ARM tags to present, deployment of resources via the Azure Portal is blocked by the ARM Policy.  The resource deployment blades in portal.azure.com do not allow the setting of tags as part of the deployment as such the ARM Policy stops the deployment.  For some customers this is not a problem as all resources are deployed using ARM templates and any template which does not have the right tags set will not be allowed to deploy. But for many customers ‘breaking’ the UI experience is really bad.

As ARM Policies have an effect of append as well as deny, a set of policies can be created to append default tags to resources as they are created, these can then be updated via the UI (or PowerShell or CLI).  This allows staff to continue using the UI for resource deployment but they will have to update the tags once the resource is provisioned.

To force all resources created in a resource group to have a default tag of ‘owner’ and ‘costcenter’ added (if not present) when the resource is being created, the following ARM Policies need to be created.

CostCenterTag.JSON

This Policy fires if tags are present for the resource but the costcenter tag is not present.  The policy appends the costcenter tag with a default value.

{
  "if": {
    "allOf": [
      {
        "field": "tags",
        "exists": "true"
      },
      {
        "field": "tags.costCenter",
        "exists": "false"
      }
    ]
  },
  "then": {
    "effect": "append",
    "details": [
      {
        "field": "tags.CostCenter",
        "value": "666"
      }
    ]
  }
}

OwnerTag.JSON

This Policy fires if tags are present for the resource but the owner tag is not present.  The policy appends the owner tag with a default value.

{
  "if": {
    "allof": [
      {
        "field": "tags",
        "exists": "true"
      },
      {
        "field": "tags.owner",
        "exists": "false"
      }
    ]
  },
  "then": {
    "effect": "append",
    "details": [
      {
        "field": "tags.owner",
        "value": "Daniel"
      }
    ]
  }
}
NoTagsPresent.JSON
This policy fires if the resource has no tags and adds the costcenter and owner tags with default values.
{
  "if": {
    "field": "tags",
    "exists": "false"
  },
  "then": {
    "effect": "append",
    "details": [
{
        "field": "tags",
        "value": {"costCenter":"666", "owner":"daniel"   }

      },
      
        ]
  }
}

All three policies need to be applied so that tags are added under the scenario that an ARM template is used that contains other tags and the scenario that a deployment is done via the UI or using an ARM template and no tags are specified .

These Policies and the PowerShell commands to deploy the policies can be downloaded from https://github.com/dbowbyes/ARM

A copy of the PowerShell script used to deploy the ARM Policies is included below.

$mysubscription = 
$myresourcegroup =


login-azurermaccount
Get-AzureRmSubscription
Select-AzureRmSubscription -SubscriptionId $mysubscription

$policy = New-AzureRmPolicyDefinition -Name tags-owner -Description "Policy to set owner tags" -Policy "tags-owner.json"
New-AzureRmPolicyAssignment -Name tags-owner -PolicyDefinition $policy -Scope /subscriptions/$mysubscription/resourceGroups/$myresourcegroup

$policy = New-AzureRmPolicyDefinition -Name tags-costcenter -Description "Policy to set costcenter" -Policy "tags-costcenter.json"
New-AzureRmPolicyAssignment -Name tags-costcenter -PolicyDefinition $policy -Scope /subscriptions/$mysubscription/resourceGroups/$myresourcegroup

$policy = New-AzureRmPolicyDefinition -Name tags-notags -Description "Policy to set costcenter" -Policy "tags-notags.json"
New-AzureRmPolicyAssignment -Name tags-notags -PolicyDefinition $policy -Scope /subscriptions/$mysubscription/resourceGroups/$myresourcegroup

The following command can be used to see what ARM Policies have been applied to a subscription or resource group

get-azureRMpolicyassignment -Scope /subscriptions/$mysubscription/resourceGroups/$myresourcegroup

More information and examples of ARM Policies can be found here.

Azure VNet Peering

VNet Peering has just been released in to public preview by Microsoft.  It allows VNets within the same region to communicate using the low latency, high bandwidth Azure backbone network .  Prior to VNet peering to allow communication between two VNet’s we had to setup VNet to VNet IKE connections which required VPN Gateways to be setup. These gateways cost money to run and inject potential throughput issues and other overheads inherent with traffic encapsulation.

Reading through the blog posts that Microsoft have released and having a play with VNet peering this evening there are a couple features that I really like:

  1. You can peer VNet’s across subscriptions and between ARM and ASM VNet’s
  2. You can control if the remote VNet can use your local gateway and if you want your VNet to use a remote VNet’s gateway
  3. All other networking features work with VNet peering, including UDR’s, Network Appliances, VPN Gateways and NSG’s
  4. VNet Peering can be setup even if VNet to VNet connections via gateways are in place. As soon as the VNet peering comes up it will be preferred over the VNet to VNet connection which allows it to be implemented with very little if any traffic impact.

VNetPeering

Microsoft have written some great blog articles on how to enable VNet peering, which can be managed via PowerShell as well as the Azure Portal.

https://azure.microsoft.com/en-us/documentation/articles/virtual-network-peering-overview/

One thing to note is that VNet peering needs to be enabled via PowerShell for each subscription that you want enable peering.  The PowerShell code below can be used to enable VNet peering for your subscription.

Login-AzureRmAccount
Get-AzureRmSubscription

Get-AzureRmSubscription -Subscriptionname "you sub name here" |Select-AzureRmSubscription

#register provider
Register-Azurermproviderfeature -FeatureName AllowVnetPeering -ProviderNamespace Microsoft.network
Register-AzureRmResourceProvider -ProviderNamespace Microsoft.Network

#show provider registration status
Get-AzureRmProviderFeature -FeatureName AllowVnetPeering -ProviderNamespace Microsoft.network

 

 


 

 

Using PowerShell to create a selfsigned cert

As an alternative to using certutil to create selfsigned certificates, the following PowerShell command can be used.  This command is part of the PKI PowerShell module which when using newer versions of PowerShell will be loaded automatically when you call the command.

New-SelfSignedCertificate -certstorelocation cert:\localmachine\my -dnsname myaddress.com

This is very handy if using PowerShell to script the deployment of AD FS for testing and you are happy to run the server with a selfsigned certificate.

 

Using Visual Studio to create and deploy Azure Resource Manager templates 

Azure Resource Manager (ARM) is Microsoft’s new way of provisioning Azure infrastructure and services.  One of the many great features of ARM is that it allows us to create JSON template files which describe the Azure services (and their relationships) we wish to deploy.   These templates can then be uploaded to Azure for deployment.
Once we have created an ARM template it is very easy to deploy the template across many  Azure subscriptions with out the potential errors that deploying the services by hand may result in.

There are many ways that ARM templates can be created but in this blog post I want to focus on using Visual Studio.  Why Visual Studio ? Well I am an IT Pro not a Programmer, so while I understand how to use the Azure portals and I get PowerShell and the Azure CLI.  I have never had any experience using Visual Studio and always thought it was a tool for Dev’s.   When I read that you could use Visual Studio to deploy ARM templates to Azure as well as create them, I was interested to find out more.

Now I don’t pretend to know how to use Visual Studio, and I am sure that I am only using a very small part of the software package, but what follows, are the steps that I have used to create an ARM template in Visual Studio and deploy the template to Azure.

First up you can download Visual Studio here

Once Visual Studio is installed select File, New, Project

Visual Studio New Project

Visual Studio Create New Project screen

The New Project screen will be displayed

Select Visual C#, Cloud, Azure Resource Group

VS-2

Visual Studio New Project popup

 

Next Visual Studio will display a list of pre made Azure Resource Manager templates, to select a blank template scroll to the bottom of the list and select Blank Template.

 

VS-3

Visual Studio Azure Template popup

 

Visual Studio will now create a new Azure Resource manager project.  On the right hand side of the screen the Solution Explorer box will show the contents of the project.  Under scripts is the PowerShell script that Visual Studio will use to deploy the template to Azure and in the templates folder you will find the ARM templates.  Double click on the DeploymentTemplate.json file to open it.

 

VS-4

Azure ARM Template

 

To add resources to the deployment template right click on the word resource as per the screenshot below and select ‘Add New Resource’

VS-5

Add New Resource to template

 

This will open the Add Resource pop up from which you can select the Azure Resources you wish to deploy.

VS-6

New Resource popup

 

Once you have added all the Azure Resources and linked them together you can test deploying your template to your Azure subscription.  To do this right click on the name of your project in the Solution Explorer and select Deploy, New Deployment.

VS-7

Deploy ARM Template to Azure

 

This will open the Deploy to Resource Group pop up and will prompt you to log in to your Azure subscription.  Clicking ‘Edit Parameters’ will bring up the Parameters dialog box which will allow you to add values for the parameters that you created when you made the ARM template.

VS-8

Subscription selection screen and parameters pop up window

 

Clicking Deploy will deploy your ARM template to the selected resource group in your Azure Subscription.

One last thing to note…..  Visual Studio uses PowerShell to deploy the ARM Template, which is all good, however with the release of Azure PowerShell 1.0 many of the Azure commands have changed.  Visual Studio creates a PowerShell deployment script that uses PowerShell 0.98 commands.  If you are using PowerShell 1.0 or greater you will need to update some of the commands in the PowerShell deployment script that gets created.  Check out the following blog post for more info

If an error is displayed stating that Switch-AzureMode is not recognised it is highly likely that you have PowerShell 1.0 installed and as such will need to follow the directions in the link above to update the deployment PowerShell script with the new 1.0 Azure commands.

Azure Security Center

This week Azure Security Center went from private preview to public preview, this new Azure service is designed to provide Azure administrators with a view of security across their Azure subscriptions.

The current public preview focuses on IaaS security, in particular VNets and virtual machines.  Azure Security Center comes with extensions that can be automatically installed in to your Azure VM’s (Windows and Ubuntu Linux with more distro’s supported in the future) which gives Azure Security center great visibility as to your security posture .

As well as reporting on the current security stance of the virtual machine the Azure Security Center also alerts if there are brute force attacks against your VM and if it is communicating with known malicious IP addresses.

Azure Security Center works with VM’s deployed using both Azure Service Manger (Classic VM’s) and Azure Resource Manager managed VM’s.

To find Azure Security Center log in to the new Azure portal http://portal.azure.com.  Using the navigation pane on the left hand side of the portal select Browse then scroll down and select Security Center.  (clicking the star will add Security Center to the left hand side navigation bar)

Azure Security Center 1

Once you have opened Security Center the first thing to do is enable the collection of information, clicking Security Policy will display your subscriptions and for each subscription you can enable the collection of security information, the storage account Security Center should store security logs for that subscription and the recommendations you wish to enable.

Enabling data collection will trigger the Security Center extension to be installed on all VM’s in that subscription.

Azure Security Center 2

Once the extensions are installed Security Center will show the security stance of your VM’s and recommend actions to remediate security issues.

Azure Security Center 3

Azure Security Center 4Some issues such as missing antimalware can be remediated from with in Azure Security Center.

Azure Security Center 6

Security Center will have more Azure services added to it over time and will be a key tool for monitoring the security of your Azure based services and infrastructure.

 

Azure – DevTest Labs

DevTestLab

One of the things about becoming an MVP is that you get notified about all sorts of interesting things coming in the world of Azure.  One new feature that caught my eye this week was Azure DevTest Labs, which is currently in preview.  Claude Remillard gave a great overview of DevTest Labs at the recent AzureCon event.

https://azure.microsoft.com/en-us/documentation/videos/azurecon-2015-introducing-azure-devtest-lab/

My interest in DevTest Labs (and the reason that I have registered for the preview) is the work the team has done to make it easy to spin up labs for testing purposes, they can be set to shutdown automatically when idle! (yay no more wasted spend!).

There are a couple of other functions that really caught my eye:

  • Ability to set quota’s for labs on machine sizes and spend plus usage monitoring so its easy to see if spend in the Lab is on track
  • Lab User security role which locks the user out of the rest of the Azure subscription and ensures they can only access the lab and its associated resources
  • Scheduled Lab shutdown (Scheduled Lab startup is coming)   — there is not point in my mind for paying for the running cost of machines that are not required or being used
  • Integration in to Visual Studio release pipeline so that code can easily be deployed to a Lab for testing

As well as Development teams I can see this being useful for Infrastructure teams.  One question I get asked  alot is how can we give our staff the ability to create a couple of VM’s in Azure but control the spend……Do we have to create a subscription for each of them?  I see DevTest Lab as solving this and allowing a single subscription to be used.  A customer can setup a Lab for each staff member with a quota attached, with the scheduled shutdown or shutdown on idle enabled compute costs would be hugely minimised.

If you are interested you can sign up for the private preview on the DevTest Labs page