Get your data ready for training¶

This module defines the basic DataBunch object that is used inside Learner to train a model. This is the generic class, that can take any kind of fastai Dataset or DataLoader. You'll find helpful functions in the data module of every application to directly create this DataBunch for you.

In [1]:

from fastai.gen_doc.nbdoc import *
from fastai.basic_data import * 

In [2]:

show_doc(DataBunch, doc_string=False)

`class` `DataBunch`[source]

DataBunch(train_dl:DataLoader, valid_dl:DataLoader, test_dl:Optional[DataLoader]=None, device:device=None, tfms:Optional[Collection[Callable]]=None, path:PathOrStr='.', collate_fn:Callable='data_collate')

Bind together a train_dl, a valid_dl and optionally a test_dl, ensures they are on device and apply to them tfms as batch are drawn. path is used internally to store temporary files, collate_fn is passed to the pytorch Dataloader (replacing the one there) to explain how to collate the samples picked for a batch. By default, it applies data to the object sent (see in vision.image why this can be important).

An example of tfms to pass is normalization. train_dl, valid_dl and optionally test_dl will be wrapped in DeviceDataLoader.

In [3]:

show_doc(DataBunch.create, doc_string=False)

`create`[source]

create(train_ds:Dataset, valid_ds:Dataset, test_ds:Dataset=None, path:PathOrStr='.', bs:int=64, num_workers:int=4, tfms:Optional[Collection[Callable]]=None, device:device=None, collate_fn:Callable='data_collate') → DataBunch

Create a DataBunch from train_ds, valid_ds and optionally test_ds, with batch size bs and by using num_workers. tfms and device are passed to the init method.

In [4]:

show_doc(DataBunch.dl)

`dl`[source]

dl(ds_type:DatasetType=<DatasetType.Valid: 2>) → DeviceDataLoader

Returns appropriate Dataset for validation, training, or test (ds_type).

In [5]:

show_doc(DataBunch.add_tfm)

`add_tfm`[source]

add_tfm(tfm:Callable)

Adds a transform to all dataloaders.

In [6]:

show_doc(DeviceDataLoader, doc_string=False)

`class` `DeviceDataLoader`[source]

DeviceDataLoader(dl:DataLoader, device:device, tfms:List[Callable]=None, collate_fn:Callable='data_collate', skip_size1:bool=False)

Put the batches of dl on device after applying an optional list of tfms. collate_fn will replace the one of dl. All dataloaders of a DataBunch are of this type.

Factory method¶

In [7]:

show_doc(DeviceDataLoader.create, doc_string=False)

`create`[source]

create(dataset:Dataset, bs:int=64, shuffle:bool=False, device:device=device(type='cuda'), tfms:Collection[Callable]=None, num_workers:int=4, collate_fn:Callable='data_collate', kwargs:Any)

Create a DeviceDataLoader on device from a dataset with batch size bs, num_workersprocesses and a given collate_fn. The dataloader will shuffle the data if that flag is set to True, and tfms are passed to the init method. All kwargs are passed to the pytorch DataLoader class initialization.

Methods¶

In [8]:

show_doc(DeviceDataLoader.one_batch)

`one_batch`[source]

one_batch() → Collection[Tensor]

Get one batch from the data loader.

In [9]:

show_doc(DeviceDataLoader.add_tfm)

`add_tfm`[source]

add_tfm(tfm:Callable)

Add a transform (i.e. same as self.tfms.append(tfm)).

In [10]:

show_doc(DeviceDataLoader.remove_tfm)

`remove_tfm`[source]

remove_tfm(tfm:Callable)

Remove a transform.

In [11]:

show_doc(DatasetType, doc_string=False)

`DatasetType`

Enum = [Train, Valid, Test]

Internal enumerator to name the training, validation and test dataset/dataloader.

Generic classes¶

The first two last classes are just empty shell to be subclassed by one of the applications, the last one is there to create empty DataBunch (useful when we want a Learner on inference mode).

In [12]:

show_doc(DatasetBase, title_level=3)

`class` `DatasetBase`[source]

DatasetBase(c:int) :: Dataset

Base class for all fastai datasets.

In [13]:

show_doc(LabelDataset, title_level=3)

`class` `LabelDataset`[source]

LabelDataset(classes:Collection, class2idx:Dict[Any, int]=None) :: DatasetBase

Base class for fastai datasets that do classification, mapped according to classes.

In [14]:

show_doc(SingleClassificationDataset, title_level=3)

`class` `SingleClassificationDataset`[source]

SingleClassificationDataset(classes:StrList) :: DatasetBase

A Dataset that contains no data, only classes, mainly used for inference with set_item

Undocumented Methods - Methods moved below this line will intentionally be hidden¶

In [15]:

show_doc(DeviceDataLoader.proc_batch)

`proc_batch`[source]

proc_batch(b:Tensor) → Tensor

Proces batch b of TensorImage.

In [16]:

show_doc(DeviceDataLoader.collate_fn)

`data_collate`[source]

data_collate(batch:ItemsList) → Tensor

Convert batch items to tensor data.

New Methods - Please document or move to the undocumented section¶

In [ ]:

Get your data ready for training¶

class DataBunch[source]

create[source]

dl[source]

add_tfm[source]

class DeviceDataLoader[source]

Factory method¶

create[source]

Methods¶

one_batch[source]

add_tfm[source]

remove_tfm[source]

`DatasetType`

Generic classes¶

class DatasetBase[source]

class LabelDataset[source]

class SingleClassificationDataset[source]

Undocumented Methods - Methods moved below this line will intentionally be hidden¶

proc_batch[source]

data_collate[source]

New Methods - Please document or move to the undocumented section¶

`class` `DataBunch`[source]

`create`[source]

`dl`[source]

`add_tfm`[source]

`class` `DeviceDataLoader`[source]

`create`[source]

`one_batch`[source]

`add_tfm`[source]

`remove_tfm`[source]

`class` `DatasetBase`[source]

`class` `LabelDataset`[source]

`class` `SingleClassificationDataset`[source]

`proc_batch`[source]

`data_collate`[source]