DVC can already be useful even in the exploration phase: while you are trying out different models, you are probably also building a good training and validation dataset. You want to make this dataset available for all your quick & dirty experiments. Putting the dataset in the same repository as the code would be cumbersome in this case, as you would have to move your data all the time from one Git repository to another one, while adding/removing data at the same time. That’s why, in this phase, we opt for the strategy of creating data registries.

A data registry is simply a repository with a number of (related) datasets, plus some scripts for preparation and cleaning, that can be used to “feed” other projects. As we will see later, DVC makes this easy.

In the above picture, we have multiple datasets (or folders with annotated/labeled data) in our registry. We think this is good practise: as you organise data collection and assembly, you will have multiple initiatives going on (maybe data gathered from different locations or from different batches). We would suggest to keep them separate and only merge them on demand as part of DVC pipelines. This way, you can still isolate/remove one dataset if needed, or completely change your decisions on which data you will use for training and which data you will use for validation.

So, to summarize:

• We create a “data registry”: a Git repo with a DVC repo for every “data effort” (collection of related datasets that took us some work to assemble)
• The Git repositories with the model code will then import the results of the dataprep pipelines from these Git repos. We don’t need to duplicate the full dataprep pipeline (eg from CVAT to dataset) every time.

NOTE: 
When setting up DVC remote storage, map all the DVC contents of one projects to one storage container or bucket. No problem if you have multiple datasets in multiple Git repos sharing the same bucket: DVC uses content-addressable filenames so you cannot overwrite somebody elses data unless the files are identical.

NOTE: 
If you have tons of very small files (eg xml files with annotations), consider zipping them. It will make DVC much more efficient. In the tutorial below, we don’t do that, but adding it would be easy.

A data preparation pipeline

We have created a small python package called tinyvoc with some data preparation utilities . We will use them here. Contrary to datumaro our utilities only support Pascal VOC, but that makes the source code extremely simple and easy to modify.

Extracting individual images from the movie

In a first step of our data preparation, we will convert the movie to individual frames. Note that, for very large datasets, it might be better to do this right from your training code, but individual frames + XML Pascal VOC annotations is compatible with lots of GitHub repo’s containing models you might want to try.

Tinyvoc ships with a wrapper script video-to-frame that will call ffmpeg (you will need to install it) to convert the video to frames, and make sure the files are named in the same way as in the CVAT annotations.

Let’s use this to create a pipeline using DVC. We actually just run the script but we tell DVC how we run the script and what inputs it needs, so DVC can reproduce it later:

dvc run -n v2f_video1  \
  -d "data/movie1/movie.mkv" \
  -O "data/movie1/JPEGImages" \
  video-to-frame --source "data/movie1/movie.mkv" \
       --destination "data/movie1/JPEGImages"

This will not only run the script, but it will also save the commandline of the script with its dependencies in dvc.yaml so that you can rerun the script eg when you get a longer video. This is not too likely (when you get a new video, you maybe want to start a new dataset instead).

Note that here we use -O (capital letter) instead of -o which means that DVC will generate the output JPEGImages but it won’t upload it to our remote storage. Makes sense, since the transformation from the video is very speedy and the jpeg images dataset would be much bigger than the video (wasting space). As a disadvantage, we can’t DVC import a non-cached output from another project.

Preprocessing the annotations

We now want to preprocess annotations from CVAT to only get the objects we are interested in. We will use DVC parameters to do this, let’s create a params.yaml file:

annotations:
  # whether to add the "type" attribute (you can create it in CVAT) to the class names of the objects
  concat-type-attribute: False
  # valid labels
  valid-labels:
    - Pedestrian
    - Bicycle
    - Car

We can now read this yaml file from our Python scripts to find which labels we want to keep, and which ones we want to throw away. We could off course also do this using command line arguments. But as an extra bonus, we can tell DVC which parameters are used by our script. This way, DVC will rerun our script when the relevant section of this file changes.

dvc run \
  -p annotations -n filter_annotations \
  -d "data/movie1/JPEGImages" -d "data/movie1/cvat-pascalvoc.zip" -o "data/movie1/Annotations" \
  -o "data/movie1/ImageSets" -o "data/movie1/labelmap.txt" \
  -m "data/movie1/metrics.json" \
  prepare-annotations --parameters params.yaml \
     --root "data/movie1" --source "data/movie1/cvat-pascalvoc.zip" \
     --destination "data/movie1/Annotations" --export-imagesets
     --metrics "data/movie1/metrics.json" 

This is a whole lot!

• We have to put a lot of parameters twice, once as parameter for the tinyvoc prepare-annotations script, and once as metadata so that dvc can track the dependencies.
• We tell DVC to depend on the annotations parameters in our params.yaml using -p annotations.
• We tell DVC to depend on data/movie1/JPEGImages, which happens to be the output of our previous pipeline stage. so now we created a dependency between the two stages!
• We add a couple of outputs using -o. These outputs will be pushed to our remote because here we cache them. We could as well not cache them using -O.
• We let our script write simple metrics (basically the number of occurences of every class) to a metrics.json, and we tell DVC about this using -m. DVC will now nicely show statistics (and how they evolve) if we do dvc metrics show.

Merging multiple datasets

In our workflow, we imported and annotated several separate movies, each resulting in its own DVC pipeline. In the end, we can merge all these separate movies into one big training set and a validation set. This is the goal of the merge-annotations script in tinyvoc. The script is very simple, and only exists because of two reasons:

• Images are symlinked instead of copied to save space
• The filenames are made unique: otherwise, multiple movies could have frame001.jpeg

dvc run \
  -n merge_annotations \
  -d "data/movie1/JPEGImages" -d "data/movie1/Annotations" \
  -d "data/movie2/JPEGImages" -d "data/movie2/Annotations" \
  -O "data/merged/training"
  merge-annotations \
     --source "data/movie1" --source "data/movie2" \
     --destination "data/merged/training"