Submit your entries: NLC2CMD competition at NeurIPS 2020 is now open
NLC2CMD is comprised of a competition for automatic translation of English to Bash and a challenge for gathering data related to such translations.
The NLC2CMD (English to Bash) Competition at NeurIPS 2020 is now officially open for entries. The event is comprised of two main parts: the NLC2CMD Competition for automatic translation of English to Bash, and the NLC2CMD Challenge for gathering data related to such translations.
The NLC2CMD Competition solicits entries that can translate a given natural language utterance into a command to be executed on the Bash terminal shell. For example, “show me a list of all files” should produce something similar to “ls” as the predicted command. The competition features two tracks: The first is the accuracy track, which is measured in terms of whether the right utility (for example, “ls”) is predicted, along with the correct flags required for it to complete the required task. Full details of the metric used for evaluation in the accuracy track can be found here. The second track is the efficiency track — energy efficiency is increasingly an important consideration for AI and Machine Learning models, and the aim of this track is to encourage systems that are judicious in their energy consumption. The competition is hosted on the EvalAI platform.
The NLC2CMD Challenge is an open call to the community of Bash users to provide use cases for an automated system to translate. Submissions to the challenge will contribute to the data set that is used to test entries to the competition. The challenge — and the various systems submitted to the competition — are intended to drive the state of the art in command translation on the shell beyond the current baseline, which is the Tellina system. Input from the user community — developers, students, researchers, and all other users of the Bash terminal — is thus a key part of making this successful.
As part of a previous survey on the Command Line AI (CLAI) open source offering from IBM, we had a chance to collect some community feedback on features that are important in an AI assistant for the Bash terminal. We thank the community for this feedback, the full results of which can be viewed here. We ask the community to once again provide their input to help create a useful data set for the NLC2CMD task.
The top 10 contributors to the challenge (in order of number of valid entries submitted) will be awarded a prize of $100 each. For the competition, there are three prizes: The top 2 entries to the accuracy track will be awarded a team prize of $2,500 each, while the top entry to the efficiency track will be awarded a team prize of $2,000. Additionally, the top entry ordered by accuracy metric will be invited to replace the “tellina” skill in the CLAI skills catalog.
The results of the competition will be announced at the NeurIPS 2020 conference, which is scheduled for December 2020.