S-pot - a benchmark in spotting signs within continuous signing

We have collected, annotated and made available the S-pot benchmark data distribution for evaluating performance in spotting particular signs within continuous signing. The distribution builds on the over 5500 clip video material of the Suvi dictionary of Finnish Sign Language. In addition to those, the distribution includes annotations about the frame ranges of occurrences of target signs in the material determined by a native signer, and a wealth of annotations on other attributes of the videos, such as the signer clothing, handshapes, movement types etc. The benchmark rigorously specifies the sign spotting tasks in a few variants, as well as the exact performance metrics that should be used for evaluating the performance. A perl script is provided for calculating those metrics.

We have solved the sign spotting tasks of the benchmark with a DTW-based approach, and these results can be used as a reference. For those who do not wish to put their efforts in every detail of video and image processing, but still want to experiment with the benchmark data, we set available all the intermediate results that are produced in various stages of our processing pipeline. The benchmark distribution includes a file that lists the performances of all the methods for solving the tasks that we are made aware of.

Motivations and some details of the benchmark are documented in the paper published in the LREC 2014 conference. More details can be found in the README files in the distribution package. The files are collected also in this compilation so that they can be inspected prior to getting the permission to download the material.

The benchmark distribution contains lots of material that is not specifically tied to the sign spotting application. We thus believe that the distribution can be a valuable resource also for other purposes. The material is available for research purposes. The access to the material is controlled by the Finnish Association of Deaf. Please contact leena.savolainen@kuurojenliitto.fi if you want to download the material.

Please cite

inproceedings{
    LREC2014SPOT,
    author = "Viitaniemi, Ville and Jantunen, Tommi and Savolainen, Leena and Karppa, Matti and Laaksonen, Jorma",
    publisher = "European Language Resources Association",
    booktitle = "Proceedings of 9th Language Resources and Evaluation Conference (LREC 2014)",
    title = "S-pot -- a benchmark in spotting signs within continuous signing",
    year = "2014",
    month = "May",
    address = "Reykjav\'ik, Iceland",
}

if you use the benchmark in academic work.