Overview
A scientific paper consists of a constellation of artifacts that extend beyond the document itself: software, hardware, evaluation data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on. In some cases, the quality of these artifacts is as important as that of the document itself, which is why WOOT organizes an optional artifact evaluation process, inspired by similar efforts in software engineering and security conferences.
Important Dates
All deadlines are at 23:59 AoE (Anywhere on Earth) time:
- Paper acceptance notifications to authors: Thursday, April 11, 2024
- Artifact submission deadline: Monday, April 29, 2024
- Discussion period: Thursday, May 2–Thursday, May 16, 2024
- Artifact decisions announced: Tuesday, May 21, 2024
- Practitioner track final papers due (including badges): Thursday, May 23, 2024
- Academic track final papers due (including badges): Thursday, May 30, 2024
Artifact Evaluation Committee
Artifact Evaluation Chair
Marius Muench, University of Birmingham
Artifact Evaluation Committee
Asmita, University of California, Davis
Tolga Atalay, Virginia Tech
Aurélien Hernandez, EURECOM
Adrian Herrera, Interrupt Labs
Jingmei Hu, Amazon
Doreen Joseph, University of California, Davis
Endong Liu, University of Birmingham
Jing Liu, University of California, Irvine
Zeyan Liu, University of Kansas
Sabrina Manickam, Vellore Institute of Technology and Max Planck Institute for Security and Privacy
Andrea Monzani, University of Milan
Paul Olivier, LAAS-CNRS
Samuel Pélissier, INSA Lyon, Inria
Davide Rusconi, University of Milan
Nathan Rutherford, Royal Holloway, University of London
Mahsa Saeidi, Oregon State university
Amit Samanta, University of Utah
Ryan Tsang, University of California, Davis
Billy Tsouvalas, Stony Brook University
Jayakrishna Menon Vadayath, Arizona State University
Nils Wiersma, Netherlands Forensic Institute
Lennert Wouters, KU Leuven
Yingao (Elaine) Yao, University of British Columbia
Matteo Zoia, University of Milan
Badges
Authors can choose to have their artifact evaluated against the following badges:
- Artifact Available. The artifact is publicly available.
- Artifacts Functional. The core functionality of the artifact could be confirmed by the AEC.
- Results Reproduced. The results, as reported in the paper, were reproduced by the AEC.
Notice that the three badges can be issued independently.
Benefits
We believe the dissemination of artifacts benefits our science and engineering as a whole, as well as the authors submitting them. Their availability improves replicability and reproducibility and enables authors to build on top of each other's work. It can also help more unambiguously resolve questions about cases not considered by the original authors. The authors receive recognition, leading to higher-impact papers, and also benefit themselves from making code reusable.
Process
Artifact evaluation is a separate process from paper reviews, and authors will be asked to submit their artifacts only after their papers have been (conditionally) accepted for publication at WOOT. After the artifact submission, at least one member of the AEC will download and install the artifact (where relevant) and evaluate it. Since we anticipate minor glitches with installation and usage, reviewers may communicate with authors to help resolve glitches while preserving reviewer anonymity. The AEC will complete its evaluation and notify the authors of the outcome.
For the camera-ready version, authors who have successfully passed the evaluation process will receive the awarded badges on their papers.
Artifact Details
To avoid excluding some papers, the AEC will try to accept any artifact that authors wish to submit. These can be software, hardware, data sets, survey results, test suites, mechanized proofs, etc. Given the experience in other communities, we decided not to accept paper proofs in the artifact evaluation process. The AEC lacks the time and often the expertise to carefully review paper proofs. Obviously, the better the artifact is packaged, the more likely the AEC can actually work with it during the evaluation process. Thus, we kindly ask authors to submit the artifact alongside setup scripts for reproducible software environments to build and run the artifact (if possible). This could, for instance, be build files for a Docker container or setup scripts for a Vagrant VM.
While we encourage open research, submission of an artifact does not include tacit permission to make its content public. All AEC members will be instructed not to publicize any part of your artifact during or after the evaluation, nor retain any part after evaluation. Thus, you are free to include, e.g., models, data files, or proprietary binaries in your artifact. Also, note that participating in the AEC experiment does not require you to publish your artifacts unless you apply for the Artifact Available badge. Still, we strongly encourage you to do so.
We recognize that some artifacts may attempt to perform malicious operations by design. These cases should be boldly and explicitly flagged in detail in the readme so AEC members can take appropriate precautions before installing and running these artifacts. The evaluation of exploits and similar results might lead to additional hurdles where we still need to collect experience on how to handle this best. Please contact the AE chair if you have concerns, for example, when evaluating bug-finding tools or other types of artifacts that need special requirements.
Submission Instructions
Authors of accepted papers who want to participate in the artifact evaluation are expected to submit the following via the artifact submission site.
- A PDF with an abstract for the artifact. The abstract should specify the core idea, the focus of the artifact, and what the evaluation should check.
- A PDF of the most recent version of the accepted paper.
- Documentation about the artifact. Describing what to test and/or how to reproduce the contributions of the paper.
- A link to the artifact. Accessible via a stable reference or DOI. For the purpose of publicly available artifacts, we recommend Zenodo. Since the artifact can evolve during the evaluation to address feedback from the reviewers, another (potentially different) stable reference will be later collected for the final version of the artifact.
Submissions are single-blind, i.e., artifacts do not need to be anonymized. However, authors must ensure that the evaluation process preserves reviewers' anonymity. If possible and applicable, we strongly encourage authors to submit their artifact in combination with setup scripts for reproducible software environments.
We expect submitted artifacts to be:
- consistent with the paper
- as complete as possible
- documented well
- easy to reuse, facilitating further research
For more details, please refer to Process and Artifact Details above. For any additional questions, please reach out to the AEC chair via woot24aec@usenix.org.