Exploring – Azure Firewall Analytics

Azure Firewall is ever growing in popularity as a choice when it comes to perimeter protection for Azure networking. The introduction of additional SKUs (Premium and Basic) since its launch have made it both more functional while also increasing its appeal to a broader environment footprint.

For anyone who has used Azure Firewall since the beginning, troubleshooting and analysis of your logs has always had a steep-ish learning curve. On one hand, the logs are stored in Log Analytics and you can query them using Kusto, so there is familiarity. However, without context, their formatting can be challenging. The good news is, this is being improved with the introduction of a new format.

Previously logs we stored using the Azure Diagnostics mode, with this update, we will now see the use of Resource-Specific mode. This is something that will become more common across many Azure resources, and you should see it appear for several in the Portal already.

Destination table in the Portal

What difference will this make for Azure Firewall? This will mean individual tables in the selected workspace are created for each category selected in the diagnostic setting. This offers the following improvements:

  • Makes it much easier to work with the data in log queries
  • Makes it easier to discover schemas and their structure
  • Improves performance across both ingestion latency and query times
  • Allows you to grant Azure RBAC rights on a specific table

For Azure Firewall, the new resource specific tables are below:

  • Network rule log – Contains all Network Rule log data. Each match between data plane and network rule creates a log entry with the data plane packet and the matched rule’s attributes.
  • NAT rule log – Contains all DNAT (Destination Network Address Translation) events log data. Each match between data plane and DNAT rule creates a log entry with the data plane packet and the matched rule’s attributes.
  • Application rule log – Contains all Application rule log data. Each match between data plane and Application rule creates a log entry with the data plane packet and the matched rule’s attributes.
  • Threat Intelligence log – Contains all Threat Intelligence events.
  • IDPS log – Contains all data plane packets that were matched with one or more IDPS signatures.
  • DNS proxy log – Contains all DNS Proxy events log data.
  • Internal FQDN resolve failure log – Contains all internal Firewall FQDN resolution requests that resulted in failure.
  • Application rule aggregation log – Contains aggregated Application rule log data for Policy Analytics.
  • Network rule aggregation log – Contains aggregated Network rule log data for Policy Analytics.
  • NAT rule aggregation log – Contains aggregated NAT rule log data for Policy Analytics.

So, let’s start with getting logs enabled on your Azure Firewall. You can’t query your logs if there are none! And Azure Firewall does not enable this by default. I’d generally recommend enabling logs as part of your build process and I have an example of that using Bicep over on Github, (note this is Diagnostics mode, I will update it for Resource mode soon!) However, if already built, let’s look at simply doing this via the Portal.

So on our Azure Firewall blade, head to the Monitoring section and choose “Diagnostic settings”

Azure Firewall blade

We’re then going to choose all our new resource specific log options

New resource specific log categories in Portal

Next, we choose to send to a workspace, and make sure to switch to Resource specific.

Workspace option with Resource option chosen in Portal

Finally, give your settings a name, I generally use my resource convention here, and click Save.

It takes a couple of minutes for logs to stream through, so while that happens, let’s look at what is available for analysis on Azure Firewall out-of-the-box – Metrics.

While there are not many entries available, what is there can be quite useful to see what sort of strain your Firewall is under.

Metrics dropdown for Azure Firewall

Hit counts are straight forward, they can give you an insight into how busy the service is. Data Processed and Throughput are also somewhat interesting from an analytics perspective. However, it is Health State and SNAT that are most useful in my opinion. These are metrics you should enable alerts against.

For example, an alert rule for SNAT utilisation reaching an average of NN% can be very useful to ensure scale is working and within limits for your service and configuration of IPs.

Ok, back to our newly enabled Resource logs. When you open the logs tab on your Firewall, if you haven’t disabled it, you should see a queries screen pop-up as below:

Sample log queries

You can see there are now two sections, one specifically for Resource Specific tables. If I simply run the following query:

AZFWNetworkRule

I get a structured and clear output:

NerworkRule query output

To get a comparative output using Diagnostics Table, I need to run a query similar to the below:

// Network rule log data 
// Parses the network rule log data. 
AzureDiagnostics
| where Category == "AzureFirewallNetworkRule"
| where OperationName == "AzureFirewallNatRuleLog" or OperationName == "AzureFirewallNetworkRuleLog"
//case 1: for records that look like this:
//PROTO request from IP:PORT to IP:PORT.
| parse msg_s with Protocol " request from " SourceIP ":" SourcePortInt:int " to " TargetIP ":" TargetPortInt:int *
//case 1a: for regular network rules
| parse kind=regex flags=U msg_s with * ". Action\\: " Action1a "\\."
//case 1b: for NAT rules
//TCP request from IP:PORT to IP:PORT was DNAT'ed to IP:PORT
| parse msg_s with * " was " Action1b:string " to " TranslatedDestination:string ":" TranslatedPort:int *
//Parse rule data if present
| parse msg_s with * ". Policy: " Policy ". Rule Collection Group: " RuleCollectionGroup "." *
| parse msg_s with * " Rule Collection: "  RuleCollection ". Rule: " Rule 
//case 2: for ICMP records
//ICMP request from 10.0.2.4 to 10.0.3.4. Action: Allow
| parse msg_s with Protocol2 " request from " SourceIP2 " to " TargetIP2 ". Action: " Action2
| extend
SourcePort = tostring(SourcePortInt),
TargetPort = tostring(TargetPortInt)
| extend 
    Action = case(Action1a == "", case(Action1b == "",Action2,Action1b), split(Action1a,".")[0]),
    Protocol = case(Protocol == "", Protocol2, Protocol),
    SourceIP = case(SourceIP == "", SourceIP2, SourceIP),
    TargetIP = case(TargetIP == "", TargetIP2, TargetIP),
    //ICMP records don't have port information
    SourcePort = case(SourcePort == "", "N/A", SourcePort),
    TargetPort = case(TargetPort == "", "N/A", TargetPort),
    //Regular network rules don't have a DNAT destination
    TranslatedDestination = case(TranslatedDestination == "", "N/A", TranslatedDestination), 
    TranslatedPort = case(isnull(TranslatedPort), "N/A", tostring(TranslatedPort)),
    //Rule information
    Policy = case(Policy == "", "N/A", Policy),
    RuleCollectionGroup = case(RuleCollectionGroup == "", "N/A", RuleCollectionGroup ),
    RuleCollection = case(RuleCollection == "", "N/A", RuleCollection ),
    Rule = case(Rule == "", "N/A", Rule)
| project TimeGenerated, msg_s, Protocol, SourceIP,SourcePort,TargetIP,TargetPort,Action, TranslatedDestination, TranslatedPort, Policy, RuleCollectionGroup, RuleCollection, Rule

Obviously, there is a large visual difference in complexity! But there are also all of the benefits as described earlier for Resource Specific. I really like the simplicity of the queries. I also like the more structured approach. For example, take a look at the set columns that are supplied on the Application Rule table. You can now predict, understand, and manipulate queries with more detail than ever before. You can check out all the new tables by searching “AZFW” on this page.

Finally, a nice sample query to get you started. One that I use quite often when checking on new services added, or if there are reports of access issues. The below gives you a quick glance into web traffic being blocked and can allow you to spot immediate issues.

AZFWApplicationRule
| where Action == "Deny"
| distinct Fqdn
| sort by Fqdn asc

As usual, if there are any questions, get in touch!

How To – Enable Azure Firewall Resource Specific Diagnostics

There is a new format of logs coming to Azure resources. Currently most people are familiar with what is called Diagnostics Table logs. The resource log for each Azure service has a unique set of columns. The AzureDiagnostics table includes the most common columns used by Azure services. If a resource log includes a column that doesn’t already exist in the AzureDiagnostics table, that column is added the first time that data is collected. If the maximum number of 500 columns is reached, data for any additional columns is added to a dynamic column.

Resource Specific logs however are platform logs that provide insight into operations that were performed within an Azure resource. The content of resource logs varies by the Azure service and resource type. Resource logs aren’t collected by default.

So onto enabling them. Via the Portal, this is straight forward in terms of choice and is well documented here. However, when I went to include this enablement in a Bicep build that I have, I noticed there wasn’t anything clearly documented. So, here is an example using Azure Firewall.

Normally, my diagnostics resource looks like the below and this enables Diagnostics table logs:

resource azfwDiags 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: '${afwName}-diags'
  scope: azFW
  properties: {
    logs: [
      {
        category: 'AzureFirewallApplicationRule'
        enabled: true
        retentionPolicy: {
          days: 90
          enabled: true
        }
      }
      {
        category: 'AzureFirewallNetworkRule'
        enabled: true
        retentionPolicy: {
          days: 90
          enabled: true
        }
      }
    ]
    workspaceId: log
  }
}

However, to enable Resource Specific, a few changes are required. Obviously the category names are different however you also need to include the Property – logAnalyticsDestinationType as you see below on line 5.

resource azfwDiags 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: '${afwName}-diags'
  scope: azFW
  properties: {
    logAnalyticsDestinationType: 'Dedicated'
    logs: [
      {
        category: 'AZFWApplicationRule'
        enabled: true
        retentionPolicy: {
          days: 90
          enabled: true
        }
      }
      {
        category: 'AZFWNetworkRule'
        enabled: true
        retentionPolicy: {
          days: 90
          enabled: true
        }
      }
    ]
    workspaceId: log
  }
}

Using the resource above within your Bicep code will allow you to deploy Resource Specific diagnostics settings as needed.

As usual, if there are any questions get in touch!

How to – Deploy Azure Firewall Premium using Bicep

With the launch of a premium SKU for Azure Firewall, many people became interested in both testing and migrating to the SKU for new features. However, the migration path requires that you stop/deallocate your Azure Firewall (AFW), modify the SKU, then start up again.

With the outage required, it’s nice to have an infrastructure-as-code solution to allow for quick testing in advance/parallel. It’s also a worthwhile piece of effort to have your rules and config documented, as these can be kept somewhat separate to your SKU requirements, especially rules.

This post will go through what is required to have Azure Firewall Premium ready to deploy using Bicep. The repo includes some bare minimums to complete testing, but you can modify as required by simply modifying and reusing the AFW module for your own environment. An example would be adding your own certificate for TLS inspection testing.

All of the code discussed in this post is located on Github in this repo.

So I have split out the test resources into several modules, which allows for better organisation, but also allows me to reuse code blocks when I want to. As I go, I am building up a set of bicep files I can reuse as needed on other environment with minimal changes.

Code directory structure

As this post is about Azure Firewall, lets’ start there – afw.bicep

afw.bicep in visualiser

As you can see, the Azure Firewall module is quite simple. The rest of the core network resources required to actually build an Azure Firewall are in network.bicep. This module can be viewed as a grouping for your AFW settings.

The Firewall Policy resource – azFWPol – is where some complexity comes into play, especially the differences for Premium. You may need to consider conditional deployments here, if you wanted your code to be flexible depending on your tier. For example, if Dev, deploy Standard SKU etc. Although SKU is set within the Firewall resource – azFW.

Now, you might be asking (if you have looked at the code!) but where are my rules? I have moved these out to a separate module to allow for simplicity of changes. Which means we will rarely have to modify the AFW module, and our edit risks are reduced.

The module – rules.bicep – simply contains a single rule collection, with a single rule. But the premise is this, using Bicep, you can control and document all rules as code, making operation much easier. Where it can become quite complex is where you have complex, large scale rule collection groups. If this is the case, you may consider splitting these out into their own individual modules. This will depend on your environment.

However, the beauty of having this setup, and one of the reasons behind this post, is so that you can quickly test things if/when required. This whole build consistently takes under ten minutes deploying to North Europe using Microsoft hosted agents

Pipeline deploy runtime example

And that is it, the repo contains all you need to deploy Azure Firewall Premium, and edit to your specific requirements. Good luck with your testing, and as always if there are any questions – just ask!

How to – Migrate Azure Firewall from Classic Rules to Policy

If you have been using Azure Firewall since it went GA, you are most likely using the classic option. This means all rules are managed within the Azure Firewall resource itself. As a result, you’ve most likely noticed the below context menu pop up when accessing your resource:

The fact there is a portal driven option I personally think is great. Often “classic to new” scenarios require a rebuild, or several shell based commands. However, I found the docs a bit light in terms of details.

So this post will provide a bit more context. On the portal, you are presented with two options, migrate the existing rules to a new policy, or, attach an existing policy. Meaning you could build your policy from scratch and simply attach it, with the operation then removing the classic rules entirely.

My preference here is to attach an existing policy, however, I am not going to start from scratch. As part of creating a new firewall policy, on the rule tab, you can import your Azure Firewall rules.

This means you can capture your existing configuration, work on any changes in advance, then simply attach your newly updated policy

Two more clicks, and the Azure Firewall will replace the classic rules config with your policy. And this is the really important part – without any downtime. However, ensure you remember with my choice, I am building the policy in advance, if I make changes to that policy, they will be adhered to once live. So your changes may cause impact, but the operation of switching to policy will not.

That’s it, you’re done! A change such as this to Azure Firewall can be a concern, especially if it is handling all of your environment traffic. But this process is simple and straight forward.

As always, if any questions get in touch! Oh and if you would prefer to do this via Powershell, here are the details.

How to – Use Azure Firewall IP Groups

If you’re familiar with Azure Firewall you would know that the introduction of an IP Group resource is most welcome. IP Groups are still in preview at the moment, so as usual be cautious on production environments as there is no SLA. However, it’s always nice to try out a service to see if it can work for you, or make your life easier.

IP Groups themselves are a relatively simple resource. They can contain a single IP address, multiple IP addresses, or one or more IP address ranges. They can then be used for DNAT, Network, or Application rules in Azure Firewall.

They currently have some interesting limitations that are a little bit confusing at first. From Docs:

For 50 IP Groups or less, you can have a maximum of 5000 individual IP addresses each per firewall instance. For 51 to 100 IP Groups, you can have 500 individual IP address each per firewall instance.

What this means is that while your rules should already be scoped accurately, you may need to use a couple of extra IP groups if you’re working with large address ranges. A simple example is a /16 will simply not work in an IP Group, /20 is basically your limit per IP Group.

I actually tried this on my own sub and it appears to actually work for now. Expect that to change as preview progresses.

If you’ve worked with Azure Firewall, I’m sure you’ve already thought of several places these rules can really help. For me, it was within Network Rule Collections.

However, as the service is in preview, there are a few aspects to be ironed out. Unfortunately, one of those is the ability to add an IP Group as a destination within a network rule when using the Portal. See below

UPDATE: As expected, this is now resolved! However, read on to see how to do this at scale.

At this point, I am going to flag extreme caution if your Azure Firewall is in production and you are trying this. It is very easy to overwrite all of your collections, take your time and export them before making any changes!

I’m a Windows guy, so I’m going to explain how to do this with Powershell, but it also works for CLI. Similarly, I’m showing a Network Rule, same process works for Application Rules.

First up, you need to all of the details for your Azure Firewall as we will work with it’s config as a variable and finally update it.

#Get the AFW I want to edit
$afw = Get-AzFirewall -Name wda-afw-test -ResourceGroupName rg-wda-afw

#Save current Network Rule Collection to a variable for reference
$oldcol = $afw.NetworkRuleCollections

#Get the IP Group I want to use
$ipg = Get-AzIpGroup -Name wda-group1 -ResourceGroupName rg-wda-afw

#Create my new network rule
$newrule = New-AzFirewallNetworkRule -Name "rule2" -Protocol TCP -SourceAddress * -DestinationIpGroup $ipg.Id -DestinationPort 445

Now this is where it can get a bit tricky. Collections are stored as nested arrays. My AFW has two collections, I want to add my new rule to the second one which means I need to reference index 1. See the collections below, the one we’ll be editing is “collection2” which currently only has “rule2”

#view all collections
$afw.NetworkRuleCollections

#view the specific collection rules using place in array
$afw.NetworkRuleCollections[1].Rules | ft

#add my new rule to my collection
$afw.NetworkRuleCollections[1].AddRule($newrule)

#if you like, check it has updated as desired
$afw.NetworkRuleCollections[1].Rules | ft

#If as expected, update AFW
Set-AzFirewall -AzureFirewall $afw

The last command can take a minute or two to complete. Once it has, you can see the rule is now added to my collection2. The Portal will display it correctly, but you cannot edit correctly with the glitch.

And that’s it! You’ve successfully added an IP Group as a destination to your Azure Firewall. Again, please be careful, the above is only a guide and I cannot be responsible for your Azure Firewall 🙂