Notebook

Recipe 1a - Provisioning an Azure storage account using the Azure portal¶

Azure Blob storage is one of the four storage services available in Azure Storage. The other storage services are Table, Queue, and File Share. Table storage is used to store non-relational structured data as key-value pairs, queue storage is used to store messages as queues, and file share is used for creating file share directories/mount points that can be accessed using the NFS/SMB protocols. This recipe will focus on storing data using the Blob storage service.

In the Azure portal, select Create a resource and choose Storage account -- blob, file, table, queue (or search for storage account in the search bar; do not choose Storage accounts (classic)).
A new page, Create a storage account, will open. There are six tabs on the Create a storage account page -- Basics, Advanced, Networking, Data protection, Tags, and Review + create.
In the Basics tab, we need to provide the Azure Subscription, Resource group, Storage account name, Region, Performance, and Redundancy values.
In the Advanced tab, we need to select Enable hierarchical namespace under the Data Lake Storage Gen2 settings.
In the Networking tab, we need to provide the connectivity method.
In the Review + create tab, review the configuration settings and select Create to provision the Azure storage account.

The Azure storage account is deployed in the selected subscription, resource group, and location. The Performance tier can be either Standard or Premium. A Standard performance tier is a low-cost magnetic drive-backed storage. It's suitable for applications such as static websites and bulk storing flat files. The Premium tier is a high-cost SSD-backed storage service. The Premium tier can only be used with Azure virtual machine disks for I/O-intensive applications.

The Replication options available are Locally-redundant storage (LRS), Zone-redundant storage (ZRS), Geo-redundant storage (GRS), and Geo-zone-redundant storage (GZRS). Local redundancy stores three local copies within the data center and provides fault tolerance for failures within it. Zone-redundant storage provides fault tolerance by copying data to additional data centers within the same region, while geo-redundant storage maintains copies across regions. Geo-zone-redundant storage combines geo- and zone-redundant features and offers the highest fault tolerance. The default Geo-redundant storage (GRS) option was selected, as it provides fault tolerance across regions.

Azure storage accounts can be accessed publicly over the internet, through selected networks (selected IPs and IP ranges), and from private endpoints.

Recipe 1b - Provisioning an Azure storage account using PowerShell¶

PowerShell is a scripting language used to programmatically manage various tasks. In this recipe, we will learn how to provision an Azure storage account using PowerShell.

Before you start, you need to log in to the Azure subscription from the PowerShell console. To do this, execute the following command in a new PowerShell window:

In [ ]:

Connect-AzAccount

Execute the following command in a PowerShell window to create a new resource group. If you want to create the Azure storage account in an existing resource group, this step isn't required:

In [ ]:

New-AzResourceGroup -Name sparshde-powershell -Location 'East US'

Execute the following command to create a new Azure storage account in the sparshade-powershell resource group:

In [ ]:

New-AzStorageAccount -ResourceGroupName sparshde-powershell -Name sparshdestoragepowershellv2 -SkuName Standard_LRS -Location 'East US' -Kind StorageV2

There is a single command to create an Azure storage account using PowerShell -- New-AzStorageAccount. The SkuName parameter specifies the performance tier, and the Kind parameter specifies the account kind.

Recipe 2 - Creating containers and uploading files to Azure Blob storage using PowerShell¶

In this recipe, we will create a new container and upload files to Azure Blob storage using PowerShell.

Execute the following commands to create the container in an Azure storage account:

In [ ]:

$storageaccountname="sparshstorage1"
$containername="logfiles"
$resourcegroup="sparsh-resource-1"

#Get the Azure Storage account context
$storagecontext = (Get-AzStorageAccount -ResourceGroupName $resourcegroup -Name $storageaccountname).Context;

#Create a new container
New-AzStorageContainer -Name $containername -Context $storagecontext

Container creation is usually very quick. You should get the following output:

PS /Users/sparshagarwal/Desktop/projects/de> $storageaccountname="sparshstorage1"
PS /Users/sparshagarwal/Desktop/projects/de> $containername="logfiles"
PS /Users/sparshagarwal/Desktop/projects/de> $resourcegroup="sparsh-resource-1"
PS /Users/sparshagarwal/Desktop/projects/de> 
PS /Users/sparshagarwal/Desktop/projects/de> #Get the Azure Storage account context
PS /Users/sparshagarwal/Desktop/projects/de> $storagecontext = (Get-AzStorageAccount -ResourceGroupName $resourcegroup -Name $storageaccountname).Context;
PS /Users/sparshagarwal/Desktop/projects/de> 
PS /Users/sparshagarwal/Desktop/projects/de> #Create a new container
PS /Users/sparshagarwal/Desktop/projects/de> New-AzStorageContainer -Name $containername -Context $storagecontext

   Storage Account Name: sparshstorage1

Name                 PublicAccess         LastModified                   IsDeleted  VersionId
----                 ------------         ------------                   ---------  ---------
logfiles             Off                  2/10/2023 6:27:00 PM +00:00               

PS /Users/sparshagarwal/Desktop/projects/de>

Execute the following commands to upload a text file to an existing container:

In [ ]:

#upload single file to container

Set-AzStorageBlobContent -File ".\data\Logfiles\Logfile1.txt" -Context $storagecontext -Blob logfile1.txt -Container $containername

You should get an output similar to the following:

PS /Users/sparshagarwal/Desktop/projects/de> Set-AzStorageBlobContent -File ".\data\Logfiles\Logfile1.txt" -Context $storagecontext -Blob logfile1.txt -Container $containername

   AccountName: sparshstorage1, ContainerName: logfiles                                                                 

Name                 BlobType  Length          ContentType                    LastModified         AccessTier SnapshotTime                 IsDelete
                                                                                                                                           d
----                 --------  ------          -----------                    ------------         ---------- ------------                 --------
logfile1.txt         BlockBlob 48              application/octet-stream       2023-02-10 18:30:50Z Hot                                     False   

PS /Users/sparshagarwal/Desktop/projects/de>

Execute the following commands to upload all the files in a directory to an Azure container:

In [ ]:

#get files to be uploaded from the directory

$files = Get-ChildItem -Path ".\data\Logfiles";

#iterate through each file in the folder and upload it to the azure container

foreach($file in $files){

Set-AzStorageBlobContent -File $file.FullName -Context $storagecontext -Blob $file.BaseName -Container $containername -Force

}

You should get an output similar to the following:

PS /> #iterate through each file in the folder and upload it to the azure container
PS /> 
PS /> foreach($file in $files){
>> 
>> Set-AzStorageBlobContent -File $file.FullName -Context $storagecontext -Blob $file.BaseName -Container $containername -Force
>> 
>> }

   AccountName: sparshstorage1, ContainerName: logfiles                                                                 

Name                 BlobType  Length          ContentType                    LastModified         AccessTier SnapshotTime                 IsDelete
                                                                                                                                           d
----                 --------  ------          -----------                    ------------         ---------- ------------                 --------
Logfile1             BlockBlob 48              application/octet-stream       2023-02-10 18:35:09Z Hot                                     False   
Logfile2             BlockBlob 78              application/octet-stream       2023-02-10 18:35:09Z Hot                                     False   

PS />

The storage container is created using the New-AzStorageContainer command. It takes two parameters -- the container name and the storage context. The storage context can be set using the Get-AzStorageAccount command context property.

To upload files to the container, we used the Set-AzStorageBlobContent command. This command requires the storage context, a file path to be uploaded, and the container name. To upload multiple files, we can iterate through the folder and upload each file using the Set-AzStorageBlobContent command.

Recipe 3 - Managing blobs in Azure Storage using PowerShell¶

In this recipe, we will learn how to perform various management tasks on an Azure blob. We will perform operations such as copying, listing, modifying, deleting, and downloading files from Azure Blob storage.

Let's perform the following operations in this recipe:

Copy files/blobs between two blob storage containers.
List files from a blob container.
Modify the storage access tier of a blob from Hot to Cool.
Download a file/blob from a container.
Delete a file/blob from a container.

Copying blobs between containers¶

Execute the following commands to create a new container in an Azure storage account:

In [ ]:

#set the parameter values
$storageaccountname="sparshstorage1"
$resourcegroup="sparsh-resource-1"
$sourcecontainername="logfiles"
$destcontainername="textfiles"

#Get storage account context
$storagecontext = (Get-AzStorageAccount -ResourceGroupName $resourcegroup -Name $storageaccountname).Context

# create the container
$destcontainer = New-AzStorageContainer -Name $destcontainername -Context $storagecontext

$destcontainer

You should get an output similar to the following:

PS /> #set the parameter values                                               
PS /> $storageaccountname="sparshstorage1"
PS /> $resourcegroup="sparsh-resource-1"
PS /> $sourcecontainername="logfiles"
PS /> $destcontainername="textfiles"
PS /> 
PS /> #Get storage account context
PS /> $storagecontext = (Get-AzStorageAccount -ResourceGroupName $resourcegroup -Name $storageaccountname).Context
PS /> 
PS /> # create the container
PS /> $destcontainer = New-AzStorageContainer -Name $destcontainername -Context $storagecontext
PS /> 
PS /> $destcontainer

   Storage Account Name: sparshstorage1

Name                 PublicAccess         LastModified                   IsDeleted  VersionId
----                 ------------         ------------                   ---------  ---------
textfiles            Off                  2/10/2023 6:43:15 PM +00:00               

PS />

Execute the following command to copy the Logfile1 blob from the source container to the destination container:

In [ ]:

Start-CopyAzureStorageBlob -SrcBlob "Logfile1" -SrcContainer $sourcecontainername -DestContainer $destcontainername -Context $storagecontext -DestContext $storagecontext

You should get an output similar to the following:

PS /> Start-CopyAzureStorageBlob -SrcBlob "Logfile1" -SrcContainer $sourcecontainername -DestContainer $destcontainername -Context $storagecontext -DestContext $storagecontext

   AccountName: sparshstorage1, ContainerName: textfiles

Name                 BlobType  Length          ContentType                    LastModified         AccessTier SnapshotTime                 IsDeleted  VersionId
----                 --------  ------          -----------                    ------------         ---------- ------------                 ---------  ---------
Logfile1             BlockBlob 48              application/octet-stream       2023-02-10 18:46:34Z Hot                                     False      

PS />

Execute the following command to copy all the blobs from the source container to the destination container:

In [ ]:

Get-AzStorageBlob -Container $sourcecontainername -Context $storagecontext | Start-CopyAzureStorageBlob -DestContainer $destcontainername -DestContext $storagecontext -force

You should get an output similar to the following:


PS /> Get-AzStorageBlob -Container $sourcecontainername -Context $storagecontext | Start-CopyAzureStorageBlob -DestContainer $destcontainername -DestContext $storagecontext -force

   AccountName: sparshstorage1, ContainerName: textfiles

Name                 BlobType  Length          ContentType                    LastModified         AccessTier SnapshotTime                 IsDeleted  VersionId
----                 --------  ------          -----------                    ------------         ---------- ------------                 ---------  ---------
Logfile1             BlockBlob 48              application/octet-stream       2023-02-10 18:46:34Z Hot                                     False      
Logfile2             BlockBlob 78              application/octet-stream       2023-02-10 18:47:52Z Hot                                     False      
logfile1.txt         BlockBlob 48              application/octet-stream       2023-02-10 18:47:52Z Hot                                     False      

PS />

Listing blobs in an Azure storage container¶

Execute the following command to list the blobs from the destination container:

In [ ]:

(Get-AzStorageContainer -Name $destcontainername -Context $storagecontext).CloudBlobContainer.ListBlobs()

You should get an output similar to the following:

PS /> (Get-AzStorageContainer -Name $destcontainername -Context $storagecontext).CloudBlobContainer.ListBlobs()

   Container Uri: https://sparshstorage1.blob.core.windows.net/textfiles

Name                 BlobType        Length          IsDeleted  RemainingD ContentType                    LastModified         AccessTier SnapshotTime
                                                                aysBeforeP
                                                                ermanentDe
                                                                lete
----                 --------        ------          ---------  ---------- -----------                    ------------         ---------- ------------
Logfile1             BlockBlob       48              False                 application/octet-stream       2023-02-10 18:47:52Z            
Logfile2             BlockBlob       78              False                 application/octet-stream       2023-02-10 18:47:52Z            
logfile1.txt         BlockBlob       48              False                 application/octet-stream       2023-02-10 18:47:52Z            

PS />

Modifying a blob access tier¶

Execute the following commands to change the access tier of a blob:

In [ ]:

$blob = Get-AzStorageBlob -Blob *Logfile2* -Container $sourcecontainername -Context $storagecontext

#Get current access tier
$blob

#change access tier to cool
$blob.ICloudBlob.SetStandardBlobTier("cool")

#Get the modified access tier
Get-AzStorageBlob -Blob *Logfile2* -Container $sourcecontainername -Context $storagecontext

You should get an output similar to the following:

PS /> $blob = Get-AzStorageBlob -Blob *Logfile2* -Container $sourcecontainername -Context $storagecontext
PS /> 
PS /> #Get current access tier
PS /> $blob

   AccountName: sparshstorage1, ContainerName: logfiles

Name                 BlobType  Length          ContentType                    LastModified         AccessTier SnapshotTime                 IsDeleted  VersionId
----                 --------  ------          -----------                    ------------         ---------- ------------                 ---------  ---------
Logfile2             BlockBlob 78              application/octet-stream       2023-02-10 18:35:09Z Hot                                     False      

PS /> 
PS /> #change access tier to cool
PS /> $blob.ICloudBlob.SetStandardBlobTier("cool")
PS /> 
PS /> #Get the modified access tier
PS /> Get-AzStorageBlob -Blob *Logfile2* -Container $sourcecontainername -Context $storagecontext

   AccountName: sparshstorage1, ContainerName: logfiles

Name                 BlobType  Length          ContentType                    LastModified         AccessTier SnapshotTime                 IsDeleted  VersionId
----                 --------  ------          -----------                    ------------         ---------- ------------                 ---------  ---------
Logfile2             BlockBlob 78              application/octet-stream       2023-02-10 18:35:09Z Cool                                    False      

PS />

Execute the following commands to change the access tier of all the blobs in the container:

In [ ]:

$blobs = Get-AzStorageBlob -Container $destcontainername -Context $storagecontext

#change the access tier of all the blobs in the container
$blobs.icloudblob.setstandardblobtier("Cool")

#verify the access tier
Get-AzStorageBlob -Container $destcontainername -Context $storagecontext

You should get an output similar to the following:

PS /> $blobs = Get-AzStorageBlob -Container $destcontainername -Context $storagecontext
PS /> 
PS /> #change the access tier of all the blobs in the container
PS /> $blobs.icloudblob.setstandardblobtier("Cool")
PS /> 
PS /> #verify the access tier
PS /> Get-AzStorageBlob -Container $destcontainername -Context $storagecontext


   AccountName: sparshstorage1, ContainerName: textfiles

Name                 BlobType  Length          ContentType                    LastModified         AccessTier SnapshotTime                 IsDeleted  VersionId
----                 --------  ------          -----------                    ------------         ---------- ------------                 ---------  ---------
Logfile1             BlockBlob 48              application/octet-stream       2023-02-10 18:47:52Z Cool                                    False      
Logfile2             BlockBlob 78              application/octet-stream       2023-02-10 18:47:52Z Cool                                    False      
logfile1.txt         BlockBlob 48              application/octet-stream       2023-02-10 18:47:52Z Cool                                    False      

PS />

Downloading a blob¶

Execute the following commands to download a blob from Azure Storage to your local computer:

In [ ]:

$storagecontext = (Get-AzStorageAccount -ResourceGroupName
$resourcegroup -Name $storageaccountname).Context

#download the blob
Get-AzStorageBlobContent -Blob "Logfile1" -Container

$sourcecontainername -Destination .\data\Logfiles\ -Context $storagecontext -Force

Deleting a blob¶

Execute the following command to remove/delete a blob:

In [ ]:

$storagecontext = (Get-AzStorageAccount -ResourceGroupName
$resourcegroup -Name $storageaccountname).Context
Remove-AzStorageBlob -Blob "Logfile2" -Container
$sourcecontainername -Context $storagecontext

Copying blobs across containers in the same storage account or a different storage account can be done easily by the PowerShell Start-CopyAzureStorageBlob command. The command takes the source and destination blobs, the source and destination containers, and the source and destination storage accounts as parameters. To copy all blobs in a container, we can run Get-AzStorageBlob to get all the blobs in the container and pipe the blobs to the Start-CopyAzureStorageBlob command.

A blob access tier can be modified by first getting the reference to the blob object using Get-AzStorageBlob and then modifying the access tier using the setstandardblobtier property. There are three access tiers -- Hot, Cool, and Archive:

The Hot tier is suitable for files that are accessed frequently. It has a higher storage cost and low access cost.
The Cool tier is suitable for infrequently accessed files and has a lower access cost and a lower storage cost.
The Archive tier, as the name suggests, is used for long-term archival and should be used for files that are seldom required. It has the highest access cost and the lowest storage cost.

To download a blob from Azure to a local system, we use Get-AzStorageBlobContent. The command accepts the blob name, the container name, the local file path, and the storage context.

To delete a blob, run Remove-AzStorageBlob. Provide the blob name, the container name, and the storage context.

Recipe 4 - Configuring blob lifecycle management for blob objects using the Azure portal¶

Azure Storage provides different blob access tiers such as Hot, Cool, and Archive. Each access tier has a different storage and data transfer cost. Applying a proper lifecycle rule to move a blob among different access tiers helps optimize the cost. In this recipe, we will learn how to apply a lifecycle rule to a blob using the Azure portal.

In the Azure portal, find and open the Azure Storage case
Search for Data management and select Lifecycle Management under Data management
On the Add a rule page, create a rule to provide the lifecycle configuration. A lifecycle defines when to move a blob from a Hot to a Cool access tier, when to move a blob from a Cool to a Storage access tier, and when to delete the blob. Select Limit blobs with filters to create a lifecycle policy for a particular container. Click Next
Specify the condition you would like to use as a lifecycle policy. For example, a rule that moves the blobs that haven't been modified in the last 30 days to Cool storage and 60 days to Archive storage
In Filter set, specify the specific container or filename for which you need to apply the rule

A blob lifecycle management rule helps in managing storage costs by modifying the access tier of blobs as per the specified rule. Consider a log processing application that reads the log file from Azure Storage, analyzes it, and saves the result in a database. As the log file is read and processed, it may not be needed any further. Therefore, moving it to a Cool access tier from a Hot access tier will save on storage costs.

Blob lifecycle management helps in automating the access tier modification as per the application requirement and is, therefore, a must-have for any storage-based application.