2 years ago

AAIA'17 Data Mining Challenge: Helping AI to Play Hearthstone

AAIA'17 Data Mining Challenge is the fourth data mining competition organized within the framework of International Symposium Advances in Artificial Intelligence and Applications (https://fedcsis.org/2017/aaia). This time, the task is to come up with an efficient prediction model which would help AI to play the game of Hearthstone: Heroes of Warcraft. The competition is kindly sponsored by Silver Bullet Solutions and Polish Information Processing Society (PTI).

Overview

Hearthstone: Heroes of Warcraft is a free-to-play online video game developed and published by Blizzard Entertainment. It is a turn-based collectible card game between two opponents, who use constructed decks of thirty cards along with a selected hero with a unique power. Players use mana points to cast spells or summon minions to attack the opponent, with the goal to reduce the opponent's health to zero. Simple general rules and appealing design made this game successful among casual players (according to Blizzard’s data, in 2016 the player-base of the game was about 50 million), as well as within the eSport community, with cash-prize tournaments and many international events every year.

Hearthstone has also become a testbed for AI research. A community of passionate players and developers has started the HearthSim (https://hearthsim.info/) project and created many tools that allow simulating the game for the purpose of AI and machine learning experiments. Together with Silver Bullet Solutions, we decided to utilize their work in our effort to develop a Hearthstone AI player that could successfully challenge a human opponent. We treat it as a proof-of-concept for our modular AI engine for video games (http://www.silverbullet.pl/demon/).

For the purpose of this challenge, we simulated a large number of Hearthstone gameplays. The task for participants is to construct a prediction model that can learn how to accurately evaluate particular intra-game states. In order to give participants a freedom of choosing a representation of the data which they want to use, apart from a preprocessed dataset in a tabular format, we provide raw JSON files that describe particular game states in more details.

More details regarding the task and a description of the competition data can be found in Task description section.

Special session at AAIA'17: As in previous years, a special session devoted to the competition will be held at the conference. We will invite authors of selected reports to extend them for publication in the conference proceedings (after reviews by Organizing Committee members) and presentation at the conference. The publications will be treated as short papers and will be indexed by IEEE Digital Library and Web of Science. The invited teams will be chosen based on their final rank, innovativeness of their approach and quality of the submitted report.

References:

http://eu.battle.net/hearthstone/en/

https://en.wikipedia.org/wiki/Hearthstone_(video_game)

https://hearthsim.info/



Terms & Conditions
 
 
FedCSIS 2017
Silver Bullet Solutions logo PTI logo

Contest Participation Rules:

  • The competition is open to all interested researchers, specialists and students. Only members of the Contest Organizing Committee and employees of Silver Bullet Solutions cannot participate.
  • Participants may submit solutions as teams made up of one or more persons.
  • Each team needs to designate a leader responsible for communication with the Organizers. A single person can be a leader of only one team.
  • One person may be incorporated in maximally 3 teams.
  • Each team needs to be composed of a different set of persons.
  • Each team is obliged to provide a short report describing their final solution. The report must contain information such as the name of the team, names of all team members and a brief overview of the used approach. The description should explain all data preprocessing steps and model construction steps. It should be submitted in the pdf format using our submission system by May 15, 2017 (23:59 GMT). Only submissions made by teams that provided the reports will qualify for the final evaluation.
  • Organizers hold the right to extend the deadlines for submitting solutions and/or reports. In such a case, they will inform participants about the change using the competition forum.
  • The final ranking of the competing teams will be done on the basis of the final evaluation results. In a case of draws in the evaluation scores, time of the submission will be taken into account.
  • Each report, paper and any other type of publication basing on the research where data from this competition is used should accredit both Knowledge Pit and Silver Bullet Solutions as the institutions that provided data for the research.
  • Organizers may reject any submission if they suspect that it was produced in an unfairly way or was submitted by a team which has broken the competition rules without providing any additional explanation.
  • By enrolling to this competition you grant the organizers rights to process your submissions and reports for the purpose of evaluation and post-competition research.

 In a case of questions related to the competition please contact us via email: aaia17challenge{_at_}silverbullet.pl or through the competition forum.

Please logIn to the system!

Data description and format: The data sets for this competition are provided in two different formats. The main one is a collection of JSON records which describe in details different game states. Each JSON record that can be used for training a prediction model is written in a single row of a text file. It contains information about a condition of each of the competing heroes, played minion cards, cards in the hand of the first player (it is assumed that the first player always starts the game) and much more. In particular, JSON records contain names of the cards in the hand of the first player and names of minions played by both players. It is allowed to use external knowledge bases about Hearthstone cards as long as they are publically available and their source is clearly stated in the submitted competition report. One example of such a source is the HearthPwn portal.

The training data is also available in a simpler tabular format, where each row corresponds to a different game state. Columns of the data tables correspond to the most important fields from the JSON records or to some aggregations of information from other fields, e.g. a sum of a maximal HP of the minions played by the first player, total mana cost of the minions played by the opponent or a number of spell cards in the hand of the first player. Please note that this is just an exemplary tabular representation of the available JSON data and it is likely to miss some important information.

The training data was compressed into two files, namely trainingData_tabular.zip and trainingData_JSON.zip, which correspond to the tabular representation and JSON representation, respectively. They can be downloaded from the Data files folder after a successful enrollment to the competition. In total, the training data contains descriptions of 2,000,000 game states, which are equally divided into four data chunks. The first column of tabular data and the gamestate_id field in the JSON records store unique identifiers of the game states. The second column and the decision field hold the information about the result of the game from which a given game state was extracted. The decision in the data is ‘1’ if the first player won the game and ‘0’ otherwise. The remaining columns/fields constitute a description of the game states. Their names are given in the data files and are rather self-explanatory.

The test data is available in the same format as the training sets, however, there is no information about the decisions. In the tabular format, the corresponding column stores no values and the decision field is missing in the JSON files. This is the target for predictions. In total, the test data consists of 1,250,000 records 750,000 records (see the post in the News section) divided into three chunks – 250,000 game states each. Please note that the training and test data sets contain game states from different play outs.

The format of submissions: The participants of the competition are asked to predict likelihoods of winning by the first player based on their representation of the data and send us their solutions using the submission system. Each solution should be sent in a single text file containing exactly 750,000 lines (files with an additional empty last line will also be accepted). In the consecutive lines, this file should contain exactly one real number corresponding to the predicted likelihood. The values do not need to be in a particular range, however, higher numerical values should indicate a higher chance of winning.

Evaluation of resultsThe submitted solutions will be evaluated on-line and the preliminary results will be published on the competition leaderboard. The preliminary score will be computed on a small subset of the test set, fixed for all participants. It will correspond to approximately 5% of the test data. The final evaluation will be performed after completion of the competition using the remaining part of the test data. Those results will also be published on-line. It is important to note that only teams which submit a report describing their approach before the end of the contest will qualify for the final evaluation. The winning teams will be officially announced during a special session devoted to this competition, which will be organized at the FedCSIS'17 conference. The assessment of solutions will be done using the Area Under the ROC Curve (AUC) measure.

Our AAIA'17 Data Mining Challenge: Helping AI to Play Hearthstone has ended. We would like to thank all participants for their involvement and hard work! 

The competition attracted 296 teams from 28 different countries. We received submissions from 188 teams, from which 114 shared a brief report describing their approach.

The official Winners:

  1. Łukasz Grad, University of Warsaw, Poland (team iwannabetheverybest)
  2. Quang Hieu Vu, ZALORAand Dymitr Ruta, EBTIC, Khalifa University, United Arab Emirates  (team hieuvq)
  3. Evgeny Patekha, Russia (team johnpateha)

Congratulation on your excellent results!

We would also like to distinguish six more teams:

 

  • jj: Jan Jakubik,
  • iran-amin: Amin Sadri,
  • pp332493: Przemysław Przybyszewski,
  • kulawykrul: Dominik Deja,
  • jaszczur: Szymon Dziewiątkowski, Sebastian Jaszczur, Mateusz Śmiech,
  • amorgun: Alexander Morgun

 

and invite them to contribute extended versions of their reports to our special session at the FedCSIS 2017 conference. We will be sending separate invitation letters shortly.

Finally, we would like to express our gratitude to Paweł Sienkowski who was the first to notice and report the possibility of a leak in the original test data. Thanks to his report, we were able to promptly identify the cause and react by providing the new test set.

We are planning to publish all competition data shortly after the conference. 

 

  • March 23, 2017: start of the competition, data sets and description become available,
  • May 14, 2017 (23:59 GMT): deadline for submitting the predictions,
  • May 15, 2017 (23:59 GMT): deadline for sending the reports, end of the competition,
  • May 22, 2017: on-line publication of the final results, sending invitations for submitting short papers for the special session at FedCSIS'17,
  • June 7, 2017: deadline for submissions of papers describing the selected solutions,
  • June 28, 2017: deadline for submissions of camera-ready papers selected for presentation at the FedCSIS'17.

Andrzej Janusz, University of Warsaw

Maciek Świechowski, Silver Bullet Solutions

Damian Zieniewicz, Silver Bullet Solutions

Krzysztof Stencel, University of Warsaw

Jacek Puczniewski, Silver Bullet Solutions

Jacek Mańdziuk, Warsaw University of Technology

Dominik Ślęzak, University of Warsaw & Infobright Inc.

  Discussion Author Replies Last post
Validation set Michał 0 by Michał
Thursday, November 02, 2017, 12:01:46
Final results Andrzej 0 by Andrzej
Sunday, May 21, 2017, 22:13:55
Final leaderboard Evgeny 2 by Evgeny
Tuesday, May 16, 2017, 09:20:46
Deadline for submitting predictions Sebastian 15 by Witold
Sunday, May 14, 2017, 09:01:13
Final prediction Łukasz 2 by Evgeny
Sunday, May 14, 2017, 17:48:48
The last day of AAIA'17 Data Mining Challenge Andrzej 0 by Andrzej
Saturday, May 13, 2017, 20:57:48
Deprecated labels matching Vasily 4 by Henry
Wednesday, May 10, 2017, 02:12:39
Post Competition Sharathkumar 1 by Sharathkumar
Friday, May 05, 2017, 09:54:07
Exception - Failed to connect to: 127.0.0.1:27017 Henry 7 by Henry
Tuesday, April 25, 2017, 10:46:39
AUC Basak Esin 1 by Basak Esin
Monday, April 24, 2017, 13:55:31
JSON files Amin 3 by Amin
Sunday, March 26, 2017, 05:49:45
The new test data set is available Andrzej 0 by Andrzej
Monday, April 17, 2017, 11:55:10
Completely new card in the test set? Yuri 1 by Yuri
Monday, April 17, 2017, 08:50:54
leaderboard precision Evgeny 1 by Andrzej
Sunday, April 16, 2017, 19:59:12
IMPORTANT ANNOUNCEMENT Andrzej 7 by Andrzej
Friday, April 14, 2017, 18:27:38
IMPORTANT ANNOUNCEMENT Andrzej 0 by Andrzej
Monday, April 10, 2017, 21:13:30
AI Quality Antoni 5 by Antoni
Monday, April 03, 2017, 17:37:57
Missing game information Piotr 2 by Piotr
Thursday, March 23, 2017, 11:59:40
Submission Order Confirmation Carlos 0 by Carlos
Monday, March 27, 2017, 15:12:42
Score 0 Carlos 6 by piotr
Thursday, March 23, 2017, 23:02:06
Leaderboard Andrzej 1 by Andrzej
Thursday, March 23, 2017, 09:55:31