1 year ago

AAIA'18 Data Mining Challenge: Predicting Win-rates of Hearthstone Decks

AAIA'18 Data Mining Challenge is the fifth competition organized within the framework of International Symposium Advances in Artificial Intelligence and Applications (https://fedcsis.org/2018/aaia). This time, the task is to assess win-rates of Hearthstone decks in games played between AI bots. The competition is kindly sponsored by Silver Bullet Labs, eSensei and Polish Information Processing Society (PTI).

Overview

This year's competition is a continuation of the topic from the previous year - data analytics related to video games. In particular, we focus on a popular collectible card video game Hearthstone: Heroes of Warcraft.

The purpose of this challenge is to discover reliable algorithms for predicting win-rates of Hearthstone decks. The task for participants is to construct a prediction model that can learn win chances of new decks, based on a history of match-ups between AI bots playing with similar decks. In order to give participants a freedom of choosing a representation of the data, apart from a preprocessed dataset in a tabular format, we provide raw JSON files that describe each game in more details - we are interested whether the data regarding the way in which cards are played during the game can be useful in the proposed task.

More details regarding the task and a description of the competition data can be found in Task description section.

Special session at AAIA'18: As in previous years, a special session devoted to the competition will be held at the conference. This year, it is called International Workshop on AI Methods in Data Mining Challenges (DMGATE). We will invite authors of selected reports to extend them for publication in the conference proceedings (after reviews by Organizing Committee members) and presentation at the conference. The publications will be treated as short papers and will be indexed by IEEE Digital Library and Web of Science. The invited teams will be chosen based on their final rank, innovativeness of their approach and quality of the submitted report. We also encourage regular paper submissions to this event, describing new approaches for analyzing data sets published during any of the challenges organized at Knowledge Pit.

References:

http://eu.battle.net/hearthstone/en/

https://en.wikipedia.org/wiki/Hearthstone_(video_game)

https://hearthsim.info/

Helping AI to Play Hearthstone: AAIA'17 Data Mining Challenge. FedCSIS 121-125
 
Terms & Conditions
 
 

Data description and format: The data for this competition is provided in two different formats. The main one is a collection of JSON files which describe in details games played between four different bots using 400 Hearthstone decks. These games can be used to estimate win-rate of the decks and learn how particular cards are used by the bots. Based on this knowledge, the task for participants is to estimate win-rates of a different set of 200 decks, played by the same bots.

The decks used in the provided collection of training games are given in a separate file named trainingDecks.json. Each row of this file corresponds to a different deck and stores a JSON with a deck identifier, hero identifier, and a list of card names along with their cardinality in the deck (there can be one or two cards of the same type in a deck). Analogically, the descriptions of test decks are provided in the same format in the file testDecks.json. It is allowed to use external knowledge bases about Hearthstone cards as long as they are publically available and their source is clearly stated in the submitted competition report. One example of such a source is the HearthPwn portal.

For the convenience of participants, we provide an additional table training_games.csv, whose rows correspond to JSON files describing the games between bots. Each row of this table gives a simple summary of the corresponding game. In particular, it provides ids of the playing bots, decks which they were using and the result of the game.

The format of submissions: The participants of the competition are asked to predict win-rates of 200 Hearthstone decks described in the file testDecks.json for the four bots which were used to generate the training data. The predictions should be sent using the submission system which will become available on April 13, 23:59 GMT. A file with a solution should consist of 800 lines, formatted in the same way as the exemplary solution provided as the file testSubmissionTemplate.csv. In each line, there should be an identifier of a bot, id of a deck and the predicted win-rate separated by semicolons. The win-rates should be expressed as percentages (real numbers between 0.0 and 100.0).

Evaluation of results: The submitted solutions will be evaluated online and the preliminary results will be published on the competition leaderboard. The preliminary score will be computed on a small subset of the test records, fixed for all participants. The final evaluation will be performed after completion of the competition using the remaining part of the test records. Those results will also be published online. It is important to note that only teams which submit a report describing their approach before the end of the contest will qualify for the final evaluation. The winning teams will be officially announced during a special session devoted to this competition, which will be organized at the FedCSIS'18 conference.

The assessment of solutions will be done using the RMSE measure. In order to keep the rule that the higher the score the better result, scores on the Leaderboard will correspond to minus RMSE values. For every pair bot-test deck, the reference win-rates were computed only based on games with the bots and decks from the training data.

The award ceremony will take place during the FedCSIS'18 conference (Sep 9-12, 2017, Poznań, Poland).

Rank Team Name Score Submission Date
1 hieuvq -5.573399 Saturday, May 5, 2018, 15:36:51
2 amy -5.648201 Sunday, May 6, 2018, 17:48:02
3 jj -5.667595 Monday, May 7, 2018, 01:04:51
4 dymitrruta -5.696228 Sunday, May 6, 2018, 23:10:06
5 amorgun -5.847379 Saturday, May 5, 2018, 09:17:02
6 a.ruta -5.860696 Sunday, April 22, 2018, 01:13:19
7 mrgrizz -5.934092 Tuesday, May 1, 2018, 14:30:09
8 kmichael08 -5.981266 Sunday, May 6, 2018, 21:48:39
9 towca -6.043132 Sunday, May 6, 2018, 21:07:39
10 adamwitkowski -6.349185 Tuesday, May 1, 2018, 23:17:34
11 karol -6.667479 Saturday, May 5, 2018, 17:44:58
12 db346864 -7.441551 Thursday, April 19, 2018, 01:42:19
13 tkacperek -7.543330 Friday, May 4, 2018, 18:19:52
14 tgarbus -7.612707 Thursday, April 26, 2018, 14:50:27
15 kabeem -7.742122 Monday, May 7, 2018, 00:16:51
16 leafproduction -7.779585 Sunday, May 6, 2018, 20:39:40
17 qwerty -7.840545 Monday, May 7, 2018, 01:10:15
18 tronowski -7.863463 Saturday, May 5, 2018, 20:22:14
19 isia -8.112951 Saturday, April 21, 2018, 19:02:13
20 siwy -8.116051 Saturday, April 28, 2018, 12:46:54
21 whatisgoingon -8.387689 Thursday, April 26, 2018, 23:15:44
22 subuk -8.392594 Sunday, May 6, 2018, 19:08:47
23 bandy -8.548265 Sunday, May 6, 2018, 23:56:20
24 bayes_brothers -8.695642 Saturday, April 28, 2018, 12:25:31
25 mw371854 -8.721913 Monday, April 30, 2018, 17:26:03
26 borysp -9.222077 Friday, May 4, 2018, 23:20:03
27 godul -9.404691 Sunday, May 6, 2018, 23:20:32
28 msonic -9.571263 Saturday, April 28, 2018, 18:31:32
29 janekf -10.535439 Sunday, May 6, 2018, 16:43:31
30 ip360730 -10.691340 Monday, May 7, 2018, 01:14:52
31 bearstrikesback -10.736522 Sunday, May 6, 2018, 16:09:19
32 kasrad -10.842473 Sunday, May 6, 2018, 23:24:04
33 optymista -12.168536 Friday, May 4, 2018, 21:25:10
34 qwymierne -12.518539 Sunday, May 6, 2018, 22:53:31
35 szarki -13.033823 Sunday, May 6, 2018, 23:54:47
36 adi_nar -13.152160 Monday, May 7, 2018, 01:03:33
37 raulmm7 -15.137684 Thursday, May 3, 2018, 14:21:21
38 csiluszyk -17.246092 Sunday, May 6, 2018, 23:50:31
39 sodar -17.246092 Monday, May 7, 2018, 00:04:37
40 gg370808 -49.867121 Sunday, May 6, 2018, 22:10:53
41 kobrar -49.967878 Monday, May 7, 2018, 01:55:18
42 tbb No report file found! Saturday, April 14, 2018, 12:51:09
43 alexionby No report file found! Wednesday, April 18, 2018, 11:44:38
44 czlowiekrakieta No report file found! Sunday, May 6, 2018, 22:48:43
45 alphapred No report file found! Thursday, April 26, 2018, 17:31:22
46 francoisw89 No report file found! Tuesday, April 24, 2018, 05:34:31
47 asztyber No report file found! Thursday, April 26, 2018, 21:10:30
48 dishonesty No report file found! Tuesday, May 1, 2018, 16:46:28
49 rekcahd No report file found! Monday, April 23, 2018, 17:43:07
50 dniwe No report file found! Tuesday, April 24, 2018, 14:01:01
51 iwannabetheverybest No report file found! Friday, April 13, 2018, 21:50:06
52 baseline_solution No report file found! Friday, April 13, 2018, 23:26:54
53 sadfsahfdasdf No report file found! Monday, April 30, 2018, 16:49:44
54 jvdputten No report file found! Monday, April 16, 2018, 13:00:59
55 lraszkiewicz No report file found! Tuesday, April 24, 2018, 12:39:49
56 marseel No report file found! Saturday, April 14, 2018, 17:29:16
57 gerstorger No report file found! Friday, April 27, 2018, 14:37:20
58 vadiaceu No report file found! Tuesday, April 17, 2018, 14:33:52
59 ducktile No report file found! Monday, April 23, 2018, 17:24:59
60 darkshadow No report file found! Tuesday, May 1, 2018, 04:29:19
61 grandgob No report file found! Tuesday, May 1, 2018, 05:04:49
62 windblade No report file found! Tuesday, May 1, 2018, 06:10:30
63 vexfight No report file found! Tuesday, May 1, 2018, 06:12:19
64 swiftroar No report file found! Tuesday, May 1, 2018, 06:32:56
65 vividfight No report file found! Tuesday, May 1, 2018, 06:48:26
66 skybinder No report file found! Tuesday, May 1, 2018, 07:56:00
67 grapplebelt No report file found! Tuesday, May 1, 2018, 08:22:13
68 dawnfather No report file found! Tuesday, May 1, 2018, 09:21:45
69 shiftbuttons No report file found! Tuesday, May 1, 2018, 09:58:59
70 randomseed19 No report file found! Tuesday, May 1, 2018, 16:57:11
71 ramich No report file found! Monday, May 7, 2018, 01:36:05
72 twsthomas No report file found! Monday, April 23, 2018, 16:42:56
73 doubleloop No report file found! Sunday, May 6, 2018, 23:32:35
74 rgod No report file found! Monday, May 7, 2018, 01:41:52
75 hjasud No report file found! Sunday, April 29, 2018, 01:35:32
76 mateuszjanczura No report file found! Sunday, May 6, 2018, 15:32:50
77 pp332493 No report file found! Saturday, May 5, 2018, 19:20:21
78 luki4824 No report file found! Saturday, April 28, 2018, 15:13:54
79 adambak No report file found! Saturday, April 21, 2018, 12:17:09
80 podludek No report file found! Thursday, April 19, 2018, 13:02:45
81 pabloxrl No report file found! Saturday, May 5, 2018, 11:58:53
82 mpk No report file found! Thursday, April 26, 2018, 14:52:53

AAIA'18 Data Mining Challenge: Predicting Win-rates of Hearthstone Decks has ended and we would like to thank all participants for their involvement and hard work! 

The official Winners:

  1. Quang Hieu Vu, ZALORAand Dymitr Ruta, EBTIC, Khalifa University, United Arab Emirates  (team hieuvq)
  2. Ling Cen, EBTIC, Khalifa University, United Arab Emirates, and Andrzej Ruta, ING Bank Slaski, Poland (team amy)
  3. Jan Jakubik, Wrocław University of Science and Technology, Poland (team jj)

Congratulation on your excellent results!

We would also like to distinguish two more teams:

  • kmichael08: Micha Kuźba and Ryszard Poklewski-Koziełł
  • adamwitkowski: Adam Witkowski, Jan Betley, and Anna Sztyber,

and invite them to contribute extended versions of their reports to our special session at the FedCSIS 2018 conference. We will be sending separate invitation letters shortly.

All participants of the challenge are also invited to submit regular conference papers describing their solutions to DMGATE'18 event which will be held at the FedCSIS'18 conference.

We are planning to publish all competition data shortly. 

  • April 3, 2018: start of the competition, training data become available,
  • May 6, 2018 (23:59 GMT): deadline for submitting the predictions,
  • May 7, 2018 (23:59 GMT): deadline for sending the reports, end of the competition,
  • May 15, 2018: online publication of the final results, sending invitations for submitting short papers for the special session at FedCSIS'18,

Authors of the top-ranked solutions (based on the final evaluation scores) will be awarded prizes funded by our sponsors:

  • First Prize: 1000 USD + one free FedCSIS'18 conference registration,
  • Second Prize: 500 USD + one free FedCSIS'18 conference registration,
  • Third Prize: one free FedCSIS'18 conference registration.

The award ceremony will take place during the FedCSIS'18 conference (Sep 9-12, 2017, Poznań, Poland).

  • Andrzej Janusz, University of Warsaw & eSensei
  • Maciek Świechowski, Warsaw University of Technology & Silver Bullet Labs
  • Tomasz Tajmajer, University of Warsaw & Silver Bullet Labs
  • Łukasz Grad, University of Warsaw & eSensei
  • Jacek Puczniewski, Silver Bullet Labs & eSensei
  • Dominik Ślęzak, University of Warsaw & eSensei

In case of any questions please post on the competition forum or write an email at janusza {at} mimuw.edu.pl.