How to Monitor URL Filtering and Traffic on Palo Alto Firewall with Splunk

Learn how to use Splunk to monitor URL filtering categories and traffic on Palo Alto firewall, and how to troubleshoot common issues.

Palo Alto Networks is a leading provider of next-generation firewalls that offer advanced security features, such as URL filtering, threat prevention, and application control. URL filtering allows you to block or allow access to websites based on their content, features, and safety. However, to effectively manage and monitor your URL filtering policies, you need a powerful tool that can analyze and visualize the data from your firewall logs. That’s where Splunk comes in.

Splunk is a software platform that can collect, index, search, and analyze data from any source, such as logs, metrics, events, and alerts. Splunk can help you gain insights into your network and security operations, and troubleshoot issues faster.

In this article, you will learn how to use Splunk to monitor URL filtering categories and traffic on Palo Alto firewall, and how to troubleshoot common issues. You will also learn how to use Splunk’s dashboards, reports, and alerts to get a comprehensive view of your firewall activity.

Prerequisites

Before you start, make sure you have the following:

• A Palo Alto Networks firewall with URL filtering enabled and configured.
• A Splunk Enterprise or Splunk Cloud instance with the Palo Alto Networks App for Splunk and the Palo Alto Networks Add-on for Splunk installed and configured. These apps provide pre-built dashboards, reports, and alerts for Palo Alto Networks data, as well as data inputs and field extractions for Splunk.
• A network connection between your firewall and your Splunk instance, and a syslog server to forward the firewall logs to Splunk. You can use any syslog server that supports UDP, TCP, or SSL protocols, such as rsyslog, syslog-ng, or Kiwi Syslog.

How to monitor URL filtering categories and traffic on Palo Alto firewall with Splunk

Once you have everything set up, you can use Splunk to monitor URL filtering categories and traffic on your firewall. Here are the steps to do so:

Step 1: Access the Palo Alto Networks App for Splunk

To access the Palo Alto Networks App for Splunk, log in to your Splunk instance and click on the Apps menu on the top left corner. Then, select Palo Alto Networks from the list of apps. You will see the app’s home page, which gives you an overview of your firewall data, such as traffic, threats, URL filtering, WildFire, and GlobalProtect.

Step 2: Navigate to the URL Filtering dashboard

To see the URL filtering data from your firewall, click on the URL Filtering tab on the top menu bar. You will see the URL Filtering dashboard, which shows you various metrics and charts related to URL filtering, such as:

• Top URL categories by bytes and sessions
• Top URL subcategories by bytes and sessions
• Top URLs by bytes and sessions
• Top users by bytes and sessions
• Top source and destination IPs by bytes and sessions
• URL filtering actions by bytes and sessions
• URL filtering verdicts by bytes and sessions
• URL filtering threats by bytes and sessions

You can use the time range picker on the top right corner to adjust the time period for the dashboard. You can also use the search bar on the top left corner to filter the dashboard by any field, such as source IP, destination IP, user, URL category, URL subcategory, URL filtering action, URL filtering verdict, or URL filtering threat. For example, if you want to see the URL filtering data for a specific user, you can type user=”john.doe” in the search bar and hit enter.

Step 3: Drill down into the details

If you want to see more details about a specific metric or chart, you can click on the panel title or the magnifying glass icon on the top right corner of the panel. This will open a new tab with a detailed report or a search query for that metric or chart. For example, if you want to see the details of the top URL categories by bytes, you can click on the panel title or the magnifying glass icon, and you will see a new tab with a table that shows the URL category, bytes, sessions, and percentage of total bytes for each URL category.

You can also click on any value in the table or the chart to drill down further into that value. For example, if you want to see the details of the social-networking URL category, you can click on the value in the table or the chart, and you will see a new tab with a search query that filters the data by that URL category. You can then modify the search query or add more filters to refine your results.

Step 4: Create custom dashboards, reports, and alerts

If you want to create your own custom dashboards, reports, or alerts based on the URL filtering data, you can use the Splunk Search and Reporting app. To access the app, click on the Apps menu on the top left corner and select Search & Reporting from the list of apps. You will see the app’s home page, which allows you to search, analyze, and visualize your data.

To create a custom dashboard, report, or alert, you need to write a search query that specifies the data source, the fields, the filters, the aggregations, the transformations, and the visualizations you want to use. You can use the Splunk Search Reference and the Splunk Search Tutorial to learn how to write search queries. You can also use the search bar on the top left corner of the Palo Alto Networks App for Splunk to get some examples of search queries for URL filtering data.

For example, if you want to create a custom dashboard that shows the top 10 users who are hitting the unknown URL category, you can write the following search query:

index=pan_logs sourcetype=pan:threat url_category=unknown
| stats sum(bytes) as bytes by user
| sort -bytes
| head 10
| table user, bytes
| rename user as "User", bytes as "Bytes"

This search query does the following:

• It selects the data from the pan_logs index and the pan:threat sourcetype, which are the default data inputs for the Palo Alto Networks Add-on for Splunk.
• It filters the data by the url_category field, which contains the URL category assigned by the firewall. In this case, we only want to see the data for the unknown URL category, which means the firewall could not identify the content or the safety of the website.
• It calculates the sum of bytes for each user, which represents the amount of data transferred by the user for the unknown URL category.
• It sorts the results by the bytes field in descending order, which means the user with the highest bytes will be on the top of the list.
• It limits the results to the top 10 users, which means we only want to see the 10 users with the highest bytes for the unknown URL category.
• It displays the results in a table with two columns: user and bytes.
• It renames the user and bytes fields to User and Bytes, respectively, to make the table more readable.

Once you write the search query, you can click on the Search button to run it and see the results. You can also click on the Save As button to save the search query as a dashboard, a report, or an alert. You can then customize the dashboard, the report, or the alert according to your preferences. For example, you can change the title, the description, the permissions, the schedule, the actions, and the format of the dashboard, the report, or the alert. You can also add more panels, charts, tables, or visualizations to the dashboard or the report.

How to troubleshoot common issues with URL filtering and Splunk

Sometimes, you may encounter some issues with URL filtering and Splunk, such as:

• No data or incomplete data in the Palo Alto Networks App for Splunk
• Incorrect or inconsistent URL categories or subcategories in the firewall logs
• High volume of unknown URL categories or subcategories in the firewall logs
• High volume of URL filtering threats or verdicts in the firewall logs

Here are some tips on how to troubleshoot these issues:

No data or incomplete data in the Palo Alto Networks App for Splunk

If you don’t see any data or only see partial data in the Palo Alto Networks App for Splunk, you may have one of the following problems:

• Your firewall is not sending logs to Splunk. You need to check the firewall configuration and make sure the syslog server is set up correctly and the firewall is forwarding the logs to the syslog server. You can use the tail command on the syslog server to verify that the logs are arriving. You can also use the test log-receiver command on the firewall to test the log forwarding.

• Your Splunk instance is not receiving logs from the syslog server. You need to check the Splunk configuration and make sure the data inputs and the field extractions are set up correctly for the Palo Alto Networks data. You can use the index=pan_logs search query to verify that the data is indexed in Splunk. You can also use the | stats count by sourcetype search query to see the count of events by sourcetype. You can refer to the troubleshooting guide for the Palo Alto Networks Add-on for Splunk for more information on how to troubleshoot data input issues.
• Your Splunk instance is not parsing the logs correctly. You need to check the Splunk configuration and make sure the props.conf and the transforms.conf files are correct for the Palo Alto Networks data. You can use the | table _raw search query to see the raw data in Splunk. You can also use the | rex mode=sed “s/\<field\>/\<new_field\>/g” search query to rename any field that is incorrect or inconsistent. You can refer to the troubleshooting guide for the Palo Alto Networks Add-on for Splunk for more information on how to troubleshoot data parsing issues.

Incorrect or inconsistent URL categories or subcategories in the firewall logs

If you see incorrect or inconsistent URL categories or subcategories in the firewall logs, you may have one of the following problems:

• Your firewall is using an outdated URL filtering database. You need to check the firewall configuration and make sure the URL filtering database is updated regularly. You can use the show url-database status command on the firewall to see the status of the URL filtering database. You can also use the request url-filtering upgrade command on the firewall to manually upgrade the URL filtering database. You can refer to the official documentation for more information on how to update the URL filtering database on your firewall.
• Your firewall is using a custom URL category or subcategory. You need to check the firewall configuration and make sure the custom URL category or subcategory is defined correctly and applied to the appropriate URL filtering policy. You can use the show url-category command on the firewall to see the list of custom URL categories and subcategories. You can also use the set url-category command on the firewall to create or modify a custom URL category or subcategory. You can refer to the official documentation for more information on how to create and use custom URL categories and subcategories on your firewall.

High volume of unknown URL categories or subcategories in the firewall logs

If you see a high volume of unknown URL categories or subcategories in the firewall logs, you may have one of the following problems:

• Your firewall is not able to access the cloud-based URL filtering service. You need to check the firewall configuration and make sure the firewall has a valid license and a network connection to the cloud-based URL filtering service. You can use the show license command on the firewall to see the status of the license. You can also use the test url <url> command on the firewall to test the connectivity and the categorization of a URL. You can refer to the official documentation for more information on how to troubleshoot the cloud-based URL filtering service on your firewall.
• Your firewall is encountering a new or uncategorized URL. You need to check the firewall configuration and make sure the firewall is configured to handle new or uncategorized URLs appropriately. You can use the show url-filtering unknown-list command on the firewall to see the list of new or uncategorized URLs. You can also use the set url-filtering unknown-list command on the firewall to add or remove a URL from the list. You can refer to the official documentation for more information on how to manage new or uncategorized URLs on your firewall.

High volume of URL filtering threats or verdicts in the firewall logs

If you see a high volume of URL filtering threats or verdicts in the firewall logs, you may have one of the following problems:

• Your firewall is detecting malicious or suspicious URLs. You need to check the firewall configuration and make sure the firewall is configured to block or alert on malicious or suspicious URLs. You can use the show url-filtering threat-list command on the firewall to see the list of malicious or suspicious URLs. You can also use the set url-filtering threat-list command on the firewall to add or remove a URL from the list. You can refer to the official documentation for more information on how to manage malicious or suspicious URLs on your firewall.
• Your firewall is using WildFire or other threat prevention features. You need to check the firewall configuration and make sure the firewall is configured to use WildFire or other threat prevention features correctly. You can use the show wildfire status command on the firewall to see the status of WildFire. You can also use the show threat-prevention command on the firewall to see the status of other threat prevention features. You can refer to the official documentation for more information on how to use WildFire or other threat prevention features on your firewall.

Frequently Asked Questions (FAQs)

Here are some frequently asked questions about URL filtering and Splunk:

Question: How can I see the URL filtering logs in Splunk?

Answer: You can see the URL filtering logs in Splunk by using the index=pan_logs sourcetype=pan:threat search query. This will show you all the events related to URL filtering from your firewall. You can also use the URL Filtering dashboard in the Palo Alto Networks App for Splunk to see the URL filtering metrics and charts.

Question: How can I see the URL filtering policies in Splunk?

Answer: You can see the URL filtering policies in Splunk by using the index=pan_logs sourcetype=pan:config search query. This will show you all the events related to configuration changes on your firewall. You can then filter the events by the config_path field, which contains the path of the configuration object. For example, you can use the config_path=”*.url-filtering” filter to see the events related to URL filtering policies.

Question: How can I see the URL filtering statistics in Splunk?

Answer: You can see the URL filtering statistics in Splunk by using the index=pan_logs sourcetype=pan:system search query. This will show you all the events related to system information on your firewall. You can then filter the events by the event_id field, which contains the ID of the system event. For example, you can use the event_id=5200 filter to see the events related to URL filtering statistics.

Summary

In this article, you learned how to use Splunk to monitor URL filtering categories and traffic on Palo Alto firewall, and how to troubleshoot common issues. You also learned how to use Splunk’s dashboards, reports, and alerts to get a comprehensive view of your firewall activity. You can use Splunk to gain insights into your network and security operations, and improve your URL filtering performance and efficiency.

Disclaimer: This article is for informational purposes only and does not constitute professional advice. The author and the publisher are not liable for any damages or losses that may result from the use of the information in this article. The reader is responsible for verifying the accuracy and validity of the information in this article before applying it to their own situation. The reader is also responsible for complying with the terms and conditions of the Palo Alto Networks and Splunk products and services mentioned in this article.