Struggling with ABFS credentials¶

In [1]:

import pystac_client
import xarray as xr

In [2]:

from pystac_client.stac_api_io import StacApiIO
from pystac_client.client import Client
stac_api_io = StacApiIO()
catalog = Client.open("https://planetarycomputer.microsoft.com/api/stac/v1", stac_io=stac_api_io)

In [3]:

sas_token = stac_api_io.session.get("https://planetarycomputer.microsoft.com/api/sas/v1/token/cil-gdpcir-cc0").json()["token"]

In [4]:

catalog = catalog.get_collection("cil-gdpcir-cc0")

In [5]:

item = next(catalog.get_all_items())

In [6]:

asset = item.assets["pr"]
asset.href

Out[6]:

'abfs://cil-gdpcir/ScenarioMIP/INM/INM-CM5-0/ssp585/r1i1p1f1/day/pr/v1.1.zarr'

Storage options work here for xarray:¶

In [7]:

storage_options = {"account_name": "rhgeuwest", "sas_token": sas_token}

In [8]:

ds = xr.open_dataset(asset.href, engine="zarr", consolidated=True, chunks={}, 
                     storage_options=storage_options)

In [9]:

ds

Out[9]:

<xarray.Dataset>
Dimensions:  (lat: 720, lon: 1440, time: 31390)
Coordinates:
  * lat      (lat) float64 -89.88 -89.62 -89.38 -89.12 ... 89.38 89.62 89.88
  * lon      (lon) float64 -179.9 -179.6 -179.4 -179.1 ... 179.4 179.6 179.9
  * time     (time) object 2015-01-01 12:00:00 ... 2100-12-31 12:00:00
Data variables:
    pr       (time, lat, lon) float64 dask.array<chunksize=(365, 360, 360), meta=np.ndarray>
Attributes: (12/47)
    Conventions:                  CF-1.7 CMIP-6.2
    activity_id:                  ScenarioMIP
    contact:                      climatesci@rhg.com
    creation_date:                2019-07-23T13:02:14Z
    data_specs_version:           01.00.29
    dc6_bias_correction_method:   Quantile Delta Method (QDM)
    ...                           ...
    sub_experiment_id:            none
    table_id:                     day
    tracking_id:                  hdl:21.14100/ba34d30b-fca8-4737-887f-344ec5...
    variable_id:                  pr
    variant_label:                r1i1p1f1
    version_id:                   v20190724

Storage options don't work here for fsspec:¶

In [10]:

import fsspec

In [11]:

fs = fsspec.filesystem('abfs', storage_options=storage_options)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File /home/conda/users/222098f819851fd4ec9d4c9445d950452119627537a69aa8cd965d7dbb1ef7e2-20230602-144731-342097-199-pangeo/lib/python3.10/site-packages/adlfs/spec.py:448, in AzureBlobFileSystem.do_connect(self)
    447     else:
--> 448         raise ValueError(
    449             "Must provide either a connection_string or account_name with credentials!!"
    450         )
    452 except RuntimeError:

ValueError: Must provide either a connection_string or account_name with credentials!!

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[11], line 1
----> 1 fs = fsspec.filesystem('abfs', storage_options=storage_options)

File /home/conda/users/222098f819851fd4ec9d4c9445d950452119627537a69aa8cd965d7dbb1ef7e2-20230602-144731-342097-199-pangeo/lib/python3.10/site-packages/fsspec/registry.py:261, in filesystem(protocol, **storage_options)
    254     warnings.warn(
    255         "The 'arrow_hdfs' protocol has been deprecated and will be "
    256         "removed in the future. Specify it as 'hdfs'.",
    257         DeprecationWarning,
    258     )
    260 cls = get_filesystem_class(protocol)
--> 261 return cls(**storage_options)

File /home/conda/users/222098f819851fd4ec9d4c9445d950452119627537a69aa8cd965d7dbb1ef7e2-20230602-144731-342097-199-pangeo/lib/python3.10/site-packages/fsspec/spec.py:76, in _Cached.__call__(cls, *args, **kwargs)
     74     return cls._cache[token]
     75 else:
---> 76     obj = super().__call__(*args, **kwargs)
     77     # Setting _fs_token here causes some static linters to complain.
     78     obj._fs_token_ = token

File /home/conda/users/222098f819851fd4ec9d4c9445d950452119627537a69aa8cd965d7dbb1ef7e2-20230602-144731-342097-199-pangeo/lib/python3.10/site-packages/adlfs/spec.py:282, in AzureBlobFileSystem.__init__(self, account_name, account_key, connection_string, credential, sas_token, request_session, socket_timeout, blocksize, client_id, client_secret, tenant_id, anon, location_mode, loop, asynchronous, default_fill_cache, default_cache_type, version_aware, **kwargs)
    270 if (
    271     self.credential is None
    272     and self.anon is False
    273     and self.sas_token is None
    274     and self.account_key is None
    275 ):
    277     (
    278         self.credential,
    279         self.sync_credential,
    280     ) = self._get_default_azure_credential(**kwargs)
--> 282 self.do_connect()
    283 weakref.finalize(self, sync, self.loop, close_service_client, self)
    285 if self.credential is not None:

File /home/conda/users/222098f819851fd4ec9d4c9445d950452119627537a69aa8cd965d7dbb1ef7e2-20230602-144731-342097-199-pangeo/lib/python3.10/site-packages/adlfs/spec.py:458, in AzureBlobFileSystem.do_connect(self)
    455     self.do_connect()
    457 except Exception as e:
--> 458     raise ValueError(f"unable to connect to account for {e}")

ValueError: unable to connect to account for Must provide either a connection_string or account_name with credentials!!

In [ ]: