Dear Participants,
in recent days we found a serious issue in the evaluation data set. Due to a bug in our preprocessing code, a part of game states in the test data come from games that were included in the training set. This makes it possible to exploit the competition rules (even unconsciously) and generate solutions that are based on a simple game matching.
We regard this as a serious problem that threatens the integrity of the competition and makes its results useless. For this reason, we decided to undertake decisive actions that will be implemented in the next few days:
- We temporarily close the submission system. It will be opened again at latest on Friday, April 14.
- The current test data set will become an additional training data set. All labels for this data will become publically available.
- A new test data set will be uploaded to the Data files folder. It will have the same format as the current test data but will consist of game state descriptions obtained from a completely new set of games.
- The Leaderboard will be reset. All currently submitted solutions will become deprecated.
We realize that it can be a very inconvenient situation for many of you and we sincerely apologize. However, we do believe that it is the only reasonable solution.
Best regards,
Andrzej Janusz
