Example of DOV search methods for interpretations (gecodeerde lithologie)¶

Use cases explained below¶

Get 'gecodeerde lithologie' in a bounding box
Get 'gecodeerde lithologie' with specific properties within a distance from a point
Get 'gecodeerde lithologie' in a bounding box with specific properties
Get 'gecodeerde lithologie' based on fields not available in the standard output dataframe
Get 'gecodeerde lithologie' data, returning fields not available in the standard output dataframe

In [1]:

%matplotlib inline
import inspect, sys

In [2]:

# check pydov path
import pydov

Get information about the datatype 'Gecodeerde lithologie'¶

In [3]:

from pydov.search.interpretaties import GecodeerdeLithologieSearch
itp = GecodeerdeLithologieSearch()

A description is provided for the 'Gecodeerde lithologie' datatype:

In [4]:

itp.get_description()

Out[4]:

'Een gecodeerde lithologie van een boring is een geologische codering van een lithologische beschrijving op basis van een vaste DOV-standaard per diepte-interval. U vindt een overzicht van de gebruikte standaarden op onze website.'

The different fields that are available for objects of the 'Gecodeerde lithologie' datatype can be requested with the get_fields() method:

In [5]:

fields = itp.get_fields()

# print available fields
for f in fields.values():
    print(f['name'])

pkey_interpretatie
Type_proef
Proefnummer
pkey_boring
x
y
start_interpretatie_mtaw
diepte_tot_m
gemeente
Auteurs
Datum
Opdrachten
betrouwbaarheid_interpretatie
Geldig_van
Geldig_tot
diepte_laag_van
diepte_laag_tot
hoofdnaam1_grondsoort
hoofdnaam2_grondsoort
bijmenging1_plaatselijk
bijmenging1_hoeveelheid
bijmenging1_grondsoort
bijmenging2_plaatselijk
bijmenging2_hoeveelheid
bijmenging2_grondsoort
bijmenging3_plaatselijk
bijmenging3_hoeveelheid
bijmenging3_grondsoort

You can get more information of a field by requesting it from the fields dictionary:

name: name of the field
definition: definition of this field
cost: currently this is either 1 or 10, depending on the datasource of the field. It is an indication of the expected time it will take to retrieve this field in the output dataframe.
notnull: whether the field is mandatory or not
type: datatype of the values of this field

In [6]:

fields['Datum']

Out[6]:

{'name': 'Datum',
 'definition': 'De datum waarop de gecodeerde lithologie werd beschreven.',
 'type': 'date',
 'notnull': False,
 'query': True,
 'cost': 1}

Example use cases¶

Get 'Gecodeerde lithologie' in a bounding box¶

Get data for all the 'Gecodeerde lithologie' interpretations that are geographically located within the bounds of the specified box.

The coordinates are in the Belgian Lambert72 (EPSG:31370) coordinate system and are given in the order of lower left x, lower left y, upper right x, upper right y.

In [7]:

from pydov.util.location import Within, Box

df = itp.search(location=Within(Box(153145, 206930, 153150, 206935)))
df.head()

[000/001] .

Out[7]:

	pkey_interpretatie	pkey_boring	betrouwbaarheid_interpretatie	x	y	start_interpretatie_mtaw	diepte_laag_van	diepte_laag_tot	hoofdnaam1_grondsoort	hoofdnaam2_grondsoort	bijmenging1_plaatselijk	bijmenging1_hoeveelheid	bijmenging1_grondsoort	bijmenging2_plaatselijk	bijmenging2_hoeveelheid	bijmenging2_grondsoort	bijmenging3_plaatselijk	bijmenging3_hoeveelheid	bijmenging3_grondsoort
0	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1974...	goed	153147.0	206931.0	14.12	0.0	0.5	LE	NaN	False	N	XZ	NaN	NaN	NaN	NaN	NaN	NaN
1	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1974...	goed	153147.0	206931.0	14.12	0.5	1.0	LE	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1974...	goed	153147.0	206931.0	14.12	1.0	2.5	KL	NaN	False	N	XZ	NaN	NaN	NaN	NaN	NaN	NaN
3	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1974...	goed	153147.0	206931.0	14.12	2.5	3.5	FZ	NaN	False	N	GL	False	N	KL	NaN	NaN	NaN
4	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1974...	goed	153147.0	206931.0	14.12	3.5	7.0	FZ	NaN	False	N	GL	NaN	NaN	NaN	NaN	NaN	NaN

The dataframe contains one 'Gecodeerde lithologie' interpretation where five layers ('laag') were identified. The available data are flattened to represent unique attributes per row of the dataframe.

Using the pkey_interpretatie field one can request the details of this interpretation in a webbrowser:

In [8]:

for pkey_interpretatie in set(df.pkey_interpretatie):
    print(pkey_interpretatie)

https://www.dov.vlaanderen.be/data/interpretatie/2001-186544

Get 'Gecodeerde lithologie' with specific properties within a distance from a point¶

Next to querying interpretations based on their geographic location within a bounding box, we can also search for interpretations matching a specific set of properties. For this we can build a query using a combination of the 'Gecodeerde lithologie' fields and operators provided by the WFS protocol.

A list of possible operators can be found below:

In [9]:

[i for i,j in inspect.getmembers(sys.modules['owslib.fes'], inspect.isclass) if 'Property' in i]

Out[9]:

['PropertyIsBetween',
 'PropertyIsEqualTo',
 'PropertyIsGreaterThan',
 'PropertyIsGreaterThanOrEqualTo',
 'PropertyIsLessThan',
 'PropertyIsLessThanOrEqualTo',
 'PropertyIsLike',
 'PropertyIsNotEqualTo',
 'PropertyIsNull',
 'SortProperty']

In this example we build a query using the PropertyIsGreaterThan and PropertyIsEqualTo operators to find all interpretations that are at least 20 m deep, that are deemed appropriate for a range of 1 km from a defined point:

In [10]:

from owslib.fes import And, PropertyIsGreaterThan, PropertyIsEqualTo
from pydov.util.location import WithinDistance, Point

query = And([PropertyIsEqualTo(propertyname='Betrouwbaarheid',
                              literal='goed'),
            PropertyIsGreaterThan(propertyname='diepte_tot_m',
                                 literal='20'),
           ])
            
df = itp.search(query=query, 
                location=WithinDistance(Point(153145, 206930), 1000))

df.head()

[000/002] ..

Out[10]:

	pkey_interpretatie	pkey_boring	betrouwbaarheid_interpretatie	x	y	start_interpretatie_mtaw	diepte_laag_van	diepte_laag_tot	hoofdnaam1_grondsoort	hoofdnaam2_grondsoort	bijmenging1_plaatselijk	bijmenging1_hoeveelheid	bijmenging1_grondsoort	bijmenging2_plaatselijk	bijmenging2_hoeveelheid	bijmenging2_grondsoort	bijmenging3_plaatselijk	bijmenging3_hoeveelheid	bijmenging3_grondsoort
0	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1994...	goed	152565.03	207059.35	18.75	0.0	1.0	FZ	NaN	False	M	KA	NaN	NaN	NaN	NaN	NaN	NaN
1	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1994...	goed	152565.03	207059.35	18.75	1.0	2.0	FZ	NaN	False	M	SF	False	M	ST	NaN	NaN	NaN
2	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1994...	goed	152565.03	207059.35	18.75	2.0	4.0	FZ	NaN	False	M	SF	False	M	ST	NaN	NaN	NaN
3	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1994...	goed	152565.03	207059.35	18.75	4.0	18.0	FZ	NaN	False	M	GL	False	M	SF	False	M	ST
4	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1994...	goed	152565.03	207059.35	18.75	18.0	19.0	FZ	NaN	False	S	LE	NaN	NaN	NaN	NaN	NaN	NaN

Once again we can use the pkey_interpretatie as a permanent link to the information of these interpretations:

In [11]:

for pkey_interpretatie in set(df.pkey_interpretatie):
    print(pkey_interpretatie)

https://www.dov.vlaanderen.be/data/interpretatie/2016-294473
https://www.dov.vlaanderen.be/data/interpretatie/2016-294475

Get 'Gecodeerde lithologie' in a bounding box based on specific properties¶

We can combine a query on attributes with a query on geographic location to get the interpretations within a bounding box that have specific properties.

The following example requests the interpretations of boreholes only, within the given bounding box.

(Note that the datatype of the literal parameter should be a string, regardless of the datatype of this field in the output dataframe.)

In [12]:

from owslib.fes import PropertyIsEqualTo

query = PropertyIsEqualTo(
            propertyname='Type_proef',
            literal='Boring')

df = itp.search(
    location=Within(Box(153145, 206930, 154145, 207930)),
    query=query
    )

df.head()

[000/001] c

Out[12]:

	pkey_interpretatie	pkey_boring	betrouwbaarheid_interpretatie	x	y	start_interpretatie_mtaw	diepte_laag_van	diepte_laag_tot	hoofdnaam1_grondsoort	hoofdnaam2_grondsoort	bijmenging1_plaatselijk	bijmenging1_hoeveelheid	bijmenging1_grondsoort	bijmenging2_plaatselijk	bijmenging2_hoeveelheid	bijmenging2_grondsoort	bijmenging3_plaatselijk	bijmenging3_hoeveelheid	bijmenging3_grondsoort
0	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1974...	goed	153147.0	206931.0	14.12	0.0	0.5	LE	NaN	False	N	XZ	NaN	NaN	NaN	NaN	NaN	NaN
1	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1974...	goed	153147.0	206931.0	14.12	0.5	1.0	LE	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1974...	goed	153147.0	206931.0	14.12	1.0	2.5	KL	NaN	False	N	XZ	NaN	NaN	NaN	NaN	NaN	NaN
3	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1974...	goed	153147.0	206931.0	14.12	2.5	3.5	FZ	NaN	False	N	GL	False	N	KL	NaN	NaN	NaN
4	https://www.dov.vlaanderen.be/data/interpretat...	https://www.dov.vlaanderen.be/data/boring/1974...	goed	153147.0	206931.0	14.12	3.5	7.0	FZ	NaN	False	N	GL	NaN	NaN	NaN	NaN	NaN	NaN

We can look at one of the interpretations in a webbrowser using its pkey_interpretatie:

In [13]:

for pkey_interpretatie in set(df.pkey_interpretatie):
    print(pkey_interpretatie)

https://www.dov.vlaanderen.be/data/interpretatie/2001-186544

Get 'Gecodeerde lithologie' based on fields not available in the standard output dataframe¶

To keep the output dataframe size acceptable, not all available WFS fields are included in the standard output. However, one can use this information to select interpretations as illustrated below.

For example, make a selection of the interpretations in municipality the of Antwerp, before 1/1/1990:

!remark: mind that the municipality attribute is merely an attribute that is defined by the person entering the data. It can be ok, empty, outdated or wrong!

In [14]:

from owslib.fes import And, PropertyIsEqualTo, PropertyIsLessThan

query = And([PropertyIsEqualTo(propertyname='gemeente',
                               literal='Antwerpen'),
             PropertyIsLessThan(propertyname='Datum', 
                                 literal='2010-01-01')]
            )
df = itp.search(query=query,
                return_fields=('pkey_interpretatie', 'Datum'))
df.head()

Out[14]:

	pkey_interpretatie	Datum
0	https://www.dov.vlaanderen.be/data/interpretat...	2008-03-28
1	https://www.dov.vlaanderen.be/data/interpretat...	2001-07-04
2	https://www.dov.vlaanderen.be/data/interpretat...	2001-07-04
3	https://www.dov.vlaanderen.be/data/interpretat...	2001-07-16
4	https://www.dov.vlaanderen.be/data/interpretat...	2001-07-16

Get 'Gecodeerde lithologie' data, returning fields not available in the standard output dataframe¶

As denoted in the previous example, not all available fields are available in the default output frame to keep its size limited. However, you can request any available field by including it in the return_fields parameter of the search:

In [15]:

query = PropertyIsEqualTo(
            propertyname='gemeente',
            literal='Leuven')

df = itp.search(query=query,
                return_fields=('pkey_interpretatie', 'pkey_boring',
                               'x', 'y', 'Z_mTAW', 'gemeente', 'Auteurs', 'Proefnummer'))

df.head()

---------------------------------------------------------------------------
InvalidFieldError                         Traceback (most recent call last)
<ipython-input-15-54ebb308efde> in <module>
      3             literal='Leuven')
      4 
----> 5 df = itp.search(query=query,
      6                 return_fields=('pkey_interpretatie', 'pkey_boring',
      7                                'x', 'y', 'Z_mTAW', 'gemeente', 'Auteurs', 'Proefnummer'))

~/Sideprojects/pydov/pydov_git/pydov/search/abstract.py in search(self, location, query, sort_by, return_fields, max_features)
    838 
    839         """
--> 840         fts = self._search(location=location, query=query, sort_by=sort_by,
    841                            return_fields=return_fields,
    842                            max_features=max_features)

~/Sideprojects/pydov/pydov_git/pydov/search/abstract.py in _search(self, location, query, return_fields, sort_by, max_features, extra_wfs_fields)
    659 
    660         """
--> 661         self._pre_search_validation(location, query, sort_by, return_fields,
    662                                     max_features)
    663         self._init_namespace()

~/Sideprojects/pydov/pydov_git/pydov/search/abstract.py in _pre_search_validation(self, location, query, sort_by, return_fields, max_features)
    540                 if rf not in self._fields:
    541                     if rf in self._map_wfs_source_df:
--> 542                         raise InvalidFieldError(
    543                             "Unknown return field: "
    544                             "'{}'. Did you mean '{}'?".format(

InvalidFieldError: Unknown return field: 'Z_mTAW'. Did you mean 'start_interpretatie_mtaw'?

Visualize results¶

Using Folium, we can display the results of our search on a map.

In [16]:

# import the necessary modules (not included in the requirements of pydov!)
import folium
from folium.plugins import MarkerCluster
from pyproj import Transformer

In [17]:

# convert the coordinates to lat/lon for folium
def convert_latlon(x1, y1):
    transformer = Transformer.from_crs("epsg:31370", "epsg:4326", always_xy=True)
    x2,y2 = transformer.transform(x1, y1)
    return x2, y2

df['lon'], df['lat'] = zip(*map(convert_latlon, df['x'], df['y'])) 
# convert to list
loclist = df[['lat', 'lon']].values.tolist()

In [18]:

# initialize the Folium map on the centre of the selected locations, play with the zoom until ok
fmap = folium.Map(location=[df['lat'].mean(), df['lon'].mean()], zoom_start=12)
marker_cluster = MarkerCluster().add_to(fmap)
for loc in range(0, len(loclist)):
    folium.Marker(loclist[loc], popup=df['Proefnummer'][loc]).add_to(marker_cluster)
fmap

Out[18]:

Make this Notebook Trusted to load map: File -> Trust Notebook

In [ ]: