Notebook Version: 2.0
Python Version: Python 3.8 (including Python 3.8 - AzureML)
Required Packages: msticpy, msticnb
Data Sources Required:
TI Proviers Used
This Notebooks brings together a series of tools and techniques to enable threat hunting within the context of a domain name or URL that has been identified as of interest. It provides a series of techniques to assist in determining whether a domain or URL is malicious. Once this has been established it provides an overview of the scope of the domain or URL across an environment, along with indicators of areas for further investigation such as hosts of interest.
Our broad initial hunting hypothesis is that a particular URL or domain might be malicious.
This notebook is designed to help you explore the data and identify if the URL is malicious and where the URL appears within the environment.
The next cell:
This should complete without errors. If you encounter errors or warnings look at the following two notebooks:
If you are running in the Microsoft Sentinel Notebooks environment (Azure Notebooks or Azure ML) you can run live versions of these notebooks:
You may also need to do some additional configuration to successfully use functions such as Threat Intelligence service lookup and Geo IP lookup.
There are more details about this in the ConfiguringNotebookEnvironment
notebook and in these documents:
from IPython.display import display, HTML, Image
from datetime import datetime, timedelta, timezone
display(HTML("<h3>Starting Notebook setup...</h3>"))
# If not using Azure Notebooks, install msticpy and msticnb with
# %pip install msticpy --upgrade
# %pip install msticnb --upgrade
import msticpy as mp
extra_imports = [
"msticpy.nbtools, observationlist",
"msticpy.sectools, domain_utils",
"pyvis.network, Network",
]
mp.init_notebook(
extra_imports=extra_imports,
additional_packages=["msticnb>=1.0"],
);
# papermill default parameters
ws_name = "Default"
url = ""
end = datetime.now(timezone.utc)
start = end - timedelta(days=2)
Use the following syntax if you are authenticating using an Azure Active Directory AppId and Secret:
%kql loganalytics://tenant(aad_tenant).workspace(WORKSPACE_ID).clientid(client_id).clientsecret(client_secret)
instead of
%kql loganalytics://code().workspace(WORKSPACE_ID)
Note: you may occasionally see a JavaScript error displayed at the end of the authentication - you can safely ignore this.
On successful authentication you should see a popup schema
button.
To find your Workspace Id go to Log Analytics. Look at the workspace properties to find the ID.
print(
"Configured workspaces: ",
", ".join(msticpy.settings.get_config("AzureSentinel.Workspaces").keys()),
)
import ipywidgets as widgets
ws_param = widgets.Combobox(
description="Workspace Name",
value=ws_name,
options=list(msticpy.settings.get_config("AzureSentinel.Workspaces").keys()),
)
ws_param
from msticpy.common.timespan import TimeSpan
# Authentication
qry_prov = QueryProvider(data_environment="MSSentinel")
qry_prov.connect(WorkspaceConfig(workspace=ws_param.value))
nb_timespan = TimeSpan(start, end)
qry_prov.query_time.timespan = nb_timespan
md("<hr>")
md("Confirm time range to search", "bold")
qry_prov.query_time
The notebook is expecting your Microsoft Sentinel Tenant ID and Workspace ID to be configured in one of the following places:
config.json
in the current foldermsticpyconfig.yaml
in the current folder or location specified by MSTICPYCONFIG
environment variable.For help with setting up your config.json
file (if this hasn't been done automatically) see the ConfiguringNotebookEnvironment
notebook in the root folder of your Azure-Sentinel-Notebooks project. This shows you how to obtain your Workspace and Subscription IDs from the Microsoft Sentinel Portal. You can use the SubscriptionID to find your Tenant ID). To view the current config.json
run the following in a code cell.
%pfile config.json
For help with setting up your msticpyconfig.yaml
see the Setup section at the end of this notebook and the ConfigureNotebookEnvironment notebook
This imports the msticnb package and the notebooklets classes.
These are needed for the notebook's operation.
import msticnb as nb
nb.init(query_provider=qry_prov)
Enter the domain or URL you wish to investigate. e.g. www.microsoft.com/index.html
url_txt = nbwidgets.GetText(
prompt="Enter the URL to investigate:", value=url
)
display(url_txt)
The following cell runs the URL Summary notebooklet the collects relevant information about the URL, its domain, and associated IPs.
Use the output to understand the context of the URL and where it appears in the environment. From here you can identify further areas and scopes to hunt on.
url_nb = nb.nblts.azsent.url.URLSummary()
url_result = url_nb.run(value=url_txt.value, timespan=qry_prov.query_time.timespan, options=["+screenshot"])
Once we have determined the nature of the domain or URL under investigation we want to see what the scope of impact is in our environment but identifying any presence of the domain or URL in our datasets.
If the domain has a high page rank score it is likely that it will be highly prevalent in a large environment, therefore you may wish to consider whether to run these cells for such a domain due to the data volumes involved.
if not url_result.dns_results.empty:
url_result.dns_results.mp_plot.timeline(title=f"DNS lookup events for {url_txt.value}")
else:
md(f"No dns events found relating to {url_txt.value}")
Understanding where a URL has appeared in alerts can help provide context on when a URL was first seen in an environment and its link to malicious activity.
url_result.display_alert_timeline()
url_result.browse_alerts()
As with alerts, understanding where a URL has appeared in bookmarks can help provide context on when a URL has previously been investigated within an environment.
if not url_result.bookmarks.empty:
url_result.bookmarks.mp_plot.timeline()
display(url_result.bookmarks)
else:
md(f"No bookmarks found relating to {url_txt.value}")
During the cells executed above we have identified hosts communicating with the URL in question. This cell provides a summary of these hosts.
These hosts are potential candidates for further investigation using Microsoft Sentinel or via the host entity explorer Notebook.
if url_result.hosts:
md(f"{len(url_result.hosts)} hosts observed connecting with {url_txt.value}")
for host in url_result.hosts:
md(host)
To dig further into data regarding the URL in the environment there are a number of data sources returned by the URL Summary notebooklet.
There are:
dns_results
: shows DNS lookup events for the domain.flows
: network flow logs for connections to the URL.They can be accessed by using url_result.[data_source]
in the cells below.
You may want to drill down on other entities in the Host data. You can use methods of the IpAddress or Host entities, for example, to look at these in more detail.
Run the ip_address_summary notebooklet pivot
IpAddress = entities.IpAddress
ip_result = IpAddress.nblt.ip_address_summary("157.56.162.53")
View the TI results
ip_result.browse_ti_results()