Artifact Evaluation

For the second time, SPIN 2025 features artifact evaluation (AE), which is mandatory for all submissions to the category Full Tool Papers and strongly encouraged for papers in other categories as well.

Note that a submission to Full Tool Papers will be considered for acceptance only if its accompanying artifact receives at least the Functional Badge in artifact evaluation. (The evaluation results of a non-mandatory artifact will not influence the acceptance of the corresponding paper.)

Important Dates

There will be two AE rounds: one for mandatory artifacts, and the other for non-mandatory ones.

Mandatory AE for Full Tool Papers:

February 27, 2025 Artifact submission deadline
March 6, 2025 Smoke-test review
March 9, 2025 Author response to smoke test
March 24, 2025 Artifact notification

Voluntary AE for accepted papers in other categories:

April 7, 2025 Artifact submission deadline
April 14, 2025 Smoke-test review
April 17, 2025 Author response to smoke test
May 1, 2025 Artifact notification

Artifacts and Evaluation Criteria

An artifact is any additional material (software, data sets, machine-checkable proofs, etc.) that substantiates the claims made in the paper and ideally renders them fully reproducible. For example, an artifact might consist of a tool and its documentation, input files used for tool evaluation in the paper, and configuration files or documents describing parameters used in the experiments.

The Artifact Evaluation Committee (AEC) will evaluate the artifact according to the following criteria:

The evaluation will be based on the EAPLS guidelines, and the AEC will decide which of the badges — among Functional, Reusable, and Available — will be assigned to a given artifact. The corresponding badges will be added to the title page of the paper in case of acceptance.

Important

All AEC members: please read the reviewer instructions carefully.

We also recommend artifact authors go through the instructions so that you know how reviewers will evaluate your submission and can prepare a better artifact accordingly.

Submission Guidelines

Artifacts should be submitted via the EasyChair SPIN 2025 submission website: https://easychair.org/conferences/?conf=spin2025 in the track Artifacts.

The following information should be provided in your EasyChair submission:

  1. Artifact submission should have the same title and authors as the submitted/accepted paper.
  2. Upload a PDF file of the submitted/accepted paper.
  3. The abstract should summarize the content of the artifact and its relation to the paper. In particular, the abstract should:
    • provide a URL (preferably a DOI) to a publicly available zipped archive containing the artifact and all relevant files
    • provide a SHA256 checksum of the zipped archive file to ensure consistency; The checksum can be generated with:
      • Linux: sha256sum <file>
      • Windows: CertUtil -hashfile <file> SHA256
      • MacOS: shasum -a 256 <file>
    • (if required) discuss special requirements for running the artifact (specific hardware resources, installation of proprietary software, etc.)
    • clearly state which claims of the paper are supported by the artifact. If some claims cannot be reproduced, shortly justify the reasons.

Packaging Guidelines

The artifact should be a self-contained package runnable on the TACAS 23 Artifact Evaluation VM (VM) or a container image (Docker/Podman) provided by the authors. Authors should test their artifact on the VM or in the container prior to the submission and include all relevant instructions. Instructions should be clear and specify every step required to build and run the artifact, assuming no specific knowledge and including steps that the authors might consider trivial (e.g., building steps for the container).

In particular, the artifact should contain:

In principle, we require authors to supply a completely self-contained artifact to ensure long-term availability. Namely, no internet access is allowed in the VM or container. If internet access is strictly necessary for your artifact, please contact the AEC chairs.

VM

We recommend authors prepare the artifact based on the TACAS 23 Artifact Evaluation VM. The VM is based on an Ubuntu 22.04 LTS image with the following additional packages: build-essential, cmake, clang, mono-complete, openjdk-8-jdk, python3.10, pip3, ruby, and a 32-bit libc. VirtualBox guest additions are installed on the VM; it is therefore possible to connect a shared folder from the host computer.

The instructions must include all necessary steps for the installation and setup on the “clean” TACAS-23 VM (Do not submit the modified VM!). In the best case, the authors would provide a script to perform all or almost all steps for the setup of their tool in the VM automatically. In particular, authors should not rely on instructions provided by external tools.

In case of technical difficulties for your artifact to run on the TACAS VM, and you prefer submitting a whole VM image (OVF/OVA), please contact the AEC chairs in advance.

Container

A container-based artifact should include a Dockerfile that automates setup from a base image, such as Ubuntu. To ensure long-term availability and offline accessibility, please also submit the generated container image.

Documentation Guidelines

Additional Tips

Artifact Evaluation Committee

TBA