Notebook

Introduction¶

This tutorial will demonstrate how to use the QFieldCloud API to build custom applications and undertake analysis that use data stored and managed by a QFieldCloud instance.

We'll introduce the QFieldCloud API and demonstrate how to use the qfieldcloud-sdk, a Python package and client to make requests to the QFieldCloud API. We'll use these tools to complete two tasks:

Create interactive chart and web map visualisations using QFieldCloud data.
Use QFieldCloud data in an accuracy assessment of the ESA World Cover v200 dataset.

Along the way we'll provide code snippets that illustrate how to use the qfieldcloud-sdk that you can expand upon for your own applications.

QFieldCloud API¶

QFieldCloud comes with a REST API which can be used to interact with projects and data stored in QFieldCloud. This supports various use cases including building web apps on top of data collected in-the-field using QField.

The API for the hosted version of QFieldCloud can be found at: https://app.qfield.cloud/swagger/

The API for the QFieldCloud instance that will be used in this workshop can be found at: https://pgc.livelihoods-and-landscapes.com/swagger/

The QFieldCloud API has endpoints for authentication, managing users and teams, querying and managing projects, and querying and downloading project data.

QFieldCloud SDK¶

The qfieldcloud-sdk is the official client to connect to the QFieldCloud API. The qfieldcloud-sdk is a Python package and can be installed using pip:

pip install qfieldcloud-sdk

This makes it well suited for integrating QFieldCloud data in data analysis workflows that leverage other tools in the Python ecosystem (e.g. GeoPandas, sklearn) or web applications (e.g. Django, FastAPI; QFieldCloud is actually a Django application).

Both QFieldCloud and qfieldcloud-sdk are developed by OPENGIS.ch, the developers of QField.

Setup for Colab¶

If you are running this notebook using Google Colab you will need to uncomment the below lines and install qfieldcloud-sdk, geopandas, and rasterio.

In [ ]:

# !pip install qfieldcloud-sdk==0.6.1
# !pip install geopandas
# !pip install rasterio

Import packages¶

In [ ]:

import requests
import rasterio
import os
import plotly.express as px
import geopandas as gpd
from qfieldcloud_sdk import sdk
from pathlib import Path
from sklearn.metrics import accuracy_score
import plotly.io as pio
pio.renderers.default = "jupyterlab"

Create a Client object and login to QFieldCloud. The constructor function for a Client takes the URL for the QFieldCloud API as an argument.

Here, we pass in the URL for the api for QFieldCloud instance being used for this workshop: https://pgc.livelihoods-and-landscapes.com/api/v1/

In [ ]:

# Create a client object
client = sdk.Client(
    url="https://pgc.livelihoods-and-landscapes.com/api/v1/",
)

The Client object has methods for authentication and querying users, projects, and data stored in QFieldCloud. First, let's login to our QFieldCloud instance.

A successful login returns a token that can be used to make authenticated requests to the QFieldCloud API end points.

For non-demonstration purposes, don't pass in credentials as clear text!

In [ ]:

# Authenticate using the client's login() method
client.login(
    username="demo_user",
    password="demo_user"
)

Query projects¶

Call list_projects() on the authenticated Client object to get a list of the user's projects.

list_projects() returns a list of dictionary objects with project metadata including its id, name, owner, description, status, and the user's role on the project.

In [ ]:

projects = client.list_projects()

In [ ]:

projects

List project files¶

The list_remote_files() method can be used to list files associated with a project. The list_remote_files() method takes a project_id as an argument and returns a list of dictionary objects describing the project's files and the file versions.

In [ ]:

# list project files stored in QFieldCloud
files = client.list_remote_files(
    project_id="2127b0a8-ced6-4129-a56b-4e8edf332d3d"
)

In [ ]:

# print the first file object
files[0]

In [ ]:

# print filenames
for i in files:
    print(i["name"])

Download files¶

We can use the download_file() method to download a specified file from QFieldCloud.

The download_file() method has project_id, remote_filename, local_filename, download_type, and show_progress parameters.

The local_filename parameter expects a Path object from the pathlib module.

The qfieldcloud-sdk has a FileTransferType class which specifies whether we want the PROJECT or PACKAGE files. Here, we want the PROJECT files.

Let's download data.gpkg which stores the ground truth points collected in the field using QField.

In [ ]:

# Create a path object for the file to download
local_filename = Path(os.path.join(os.getcwd(), "data.gpkg"))

# Download the file from QFieldCloud
client.download_file(
    project_id="2127b0a8-ced6-4129-a56b-4e8edf332d3d",
    remote_filename="data.gpkg",
    local_filename=local_filename,
    download_type=sdk.FileTransferType.PROJECT,
    show_progress=False
)

In [ ]:

# Check data.gpkg downloaded OK
print(f"downloaded data.gpkg successfully: {'data.gpkg' in os.listdir()}")

Visualise data¶

Now we have downloaded data from the QFieldCloud API we can visualise and analyse it. First, let's explore the data using charts and web map widgets.

In [ ]:

# Read the data into a GeoPandas GeoDataFrame
gdf = gpd.read_file(os.path.join(os.getcwd(), "data.gpkg"))

First, let's inspect the data in data table.

In [ ]:

display(gdf)

Next, let's create interactive visualisations using the data downloaded from QFieldCloud. We'll use Plotly Express to create interactive figures and web maps.

We can use the px.histogram() function to create a bar plot of the counts of observations for each land cover class in our QFieldCloud project.

In [ ]:

color_discrete_map={
    "1": "#00097B",
    "2": "#04e3a5",
    "3": "#8a6d1d",
    "4": "#ffffff",
    "5": "#ff9143",
    "6": "#d0ff14",
    "7": "#a4c93f",
    "8": "#377d22"}

fig = px.histogram(
    gdf, 
    x="land_cover_class", 
    color="land_cover_class",
    color_discrete_map=color_discrete_map, 
    title="Number of observations per-land cover class",
    labels={"land_cover_class": "land cover class"}
)

fig.update_layout(
    xaxis = dict(
        tickmode = "array",
        tickvals = [1, 2, 3, 4, 5, 6, 7, 8],
        ticktext = ["water", "mangrove", "bare soil", "urban", "cropland", "grassland", "shrubland", "trees"]
    )
)

fig.show()

Next, let's visualise the data in our QFieldCloud project on a web map using the px.scatter_mapbox() function.

In [ ]:

color_discrete_map={
    "1": "#00097B",
    "2": "#04e3a5",
    "3": "#8a6d1d",
    "4": "#ffffff",
    "5": "#ff9143",
    "6": "#d0ff14",
    "7": "#a4c93f",
    "8": "#377d22"}

fig = px.scatter_mapbox(
    gdf,
    lat=gdf.geometry.y,
    lon=gdf.geometry.x,
    zoom=12,
    mapbox_style="open-street-map",
    color="land_cover_class",
    color_discrete_map=color_discrete_map
)

fig.show()

Accuracy Assessment¶

Here, we'll use the ground truth data that we've collected in the field using QField to perform a quick accuracy assessment of the ESA World Cover v200 land cover map.

Let's download clip of the ESA World Cover v200 land cover map that covers Suva and the surrounding area.

In [ ]:

!wget "https://github.com/livelihoods-and-landscapes/pacific-geo-conf/raw/main/esa-world-cover-v2-suva.tif"

Sample the ESA World Cover v2 land cover class at each location where we've collected a ground truth point. Append the predicted class as a column to our GeoDataFrame gdf.

In [ ]:

# based on https://geopandas.org/en/stable/gallery/geopandas_rasterio_sample.html
coord_list = [(x,y) for x,y in zip(gdf["geometry"].x , gdf["geometry"].y)]

with rasterio.open(os.path.join(os.getcwd(), "esa-world-cover-v2-suva.tif")) as src:
    meta = src.meta
    img = src.read(1)
    gdf["predicted_land_cover"] = [str(x[0]) for x in src.sample(coord_list)]

In [ ]:

display(gdf)

In [ ]:

# accuracy score
print(f"the accuracy score for the ESA World Cover v2 land cover map is {round(accuracy_score(gdf['land_cover_class'], gdf['predicted_land_cover']), 2)}")

In [ ]:

# quick look at the land cover map to check it seems OK
# px.imshow(img)

Finally, we can visualise a confusion matrix as a heatmap.

In [ ]:

fig = px.density_heatmap(
    gdf,
    x="land_cover_class",
    y="predicted_land_cover",
    text_auto=True,
    labels={"land_cover_class": "land cover class",
            "predicted_land_cover": "ESA World Cover prediction"}
)

fig.update_layout(
    xaxis = dict(
        tickmode = "array",
        tickvals = [1, 2, 3, 4, 5, 6, 7, 8],
        ticktext = ["water", "mangrove", "bare soil", "urban", "cropland", "grassland", "shrubland", "trees"]
    ),
    yaxis = dict(
        tickmode = "array",
        tickvals = [1, 2, 3, 4, 6, 8],
        ticktext = ["water", "mangrove", "bare soil", "urban", "grassland", "trees"]
    )
)

fig.show()

Introduction¶

QFieldCloud API¶

QFieldCloud SDK¶

Setup for Colab¶

Import packages¶

Login¶

Query projects¶

List project files¶

Download files¶

Visualise data¶

Accuracy Assessment¶