1 year, 5 months ago

AAIA'18 Data Mining Challenge: Predicting Win-rates of Hearthstone Decks

AAIA'18 Data Mining Challenge is the fifth competition organized within the framework of International Symposium Advances in Artificial Intelligence and Applications (https://fedcsis.org/2018/aaia). This time, the task is to assess win-rates of Hearthstone decks in games played between AI bots. The competition is kindly sponsored by Silver Bullet Labs, eSensei and Polish Information Processing Society (PTI).

Overview

This year's competition is a continuation of the topic from the previous year - data analytics related to video games. In particular, we focus on a popular collectible card video game Hearthstone: Heroes of Warcraft.

The purpose of this challenge is to discover reliable algorithms for predicting win-rates of Hearthstone decks. The task for participants is to construct a prediction model that can learn win chances of new decks, based on a history of match-ups between AI bots playing with similar decks. In order to give participants a freedom of choosing a representation of the data, apart from a preprocessed dataset in a tabular format, we provide raw JSON files that describe each game in more details - we are interested whether the data regarding the way in which cards are played during the game can be useful in the proposed task.

More details regarding the task and a description of the competition data can be found in Task description section.

Special session at AAIA'18: As in previous years, a special session devoted to the competition will be held at the conference. This year, it is called International Workshop on AI Methods in Data Mining Challenges (DMGATE). We will invite authors of selected reports to extend them for publication in the conference proceedings (after reviews by Organizing Committee members) and presentation at the conference. The publications will be treated as short papers and will be indexed by IEEE Digital Library and Web of Science. The invited teams will be chosen based on their final rank, innovativeness of their approach and quality of the submitted report. We also encourage regular paper submissions to this event, describing new approaches for analyzing data sets published during any of the challenges organized at Knowledge Pit.

References:

  1. Helping AI to Play Hearthstone: AAIA'17 Data Mining Challenge. FedCSIS 121-125
  2. Maciej Swiechowski, Tomasz Tajmajer, Andrzej Janusz: Improving Hearthstone AI by Combining MCTS and Supervised Learning Algorithms. CIG 2018: 1-8
  3. Andrzej Janusz, Tomasz Tajmajer, Maciej Swiechowski, Lukasz Grad, Jacek Puczniewski, Dominik Slezak: Toward an Intelligent HS Deck Advisor: Lessons Learned from AAIA'18 Data Mining Competition. FedCSIS 2018: 189-192
  4. Andrzej Janusz, Lukasz Grad, Dominik Slezak: Utilizing Hybrid Information Sources to Learn Representations of Cards in Collectible Card Video Games. ICDM Workshops 2018: 422-429
  5. http://eu.battle.net/hearthstone/en/
  6. https://hearthsim.info/
Terms & Conditions
 
 

AAIA'18 Data Mining Challenge: Predicting Win-rates of Hearthstone Decks has ended and we would like to thank all participants for their involvement and hard work! 

The official Winners:

  1. Quang Hieu Vu, ZALORAand Dymitr Ruta, EBTIC, Khalifa University, United Arab Emirates  (team hieuvq)
  2. Ling Cen, EBTIC, Khalifa University, United Arab Emirates, and Andrzej Ruta, ING Bank Slaski, Poland (team amy)
  3. Jan Jakubik, Wrocław University of Science and Technology, Poland (team jj)

Congratulation on your excellent results!

We would also like to distinguish two more teams:

  • kmichael08: Micha Kuźba and Ryszard Poklewski-Koziełł
  • adamwitkowski: Adam Witkowski, Jan Betley, and Anna Sztyber,

and invite them to contribute extended versions of their reports to our special session at the FedCSIS 2018 conference. We will be sending separate invitation letters shortly.

All participants of the challenge are also invited to submit regular conference papers describing their solutions to DMGATE'18 event which will be held at the FedCSIS'18 conference.

All competition data files (including the ground truth win-rates) were published in the Data files section. If you are planning to use them in your research, please indicate KnowledgePit as the data source and include references to our papers related to the challenge. 

Please logIn to the system!

Data description and format: The data for this competition is provided in two different formats. The main one is a collection of JSON files which describe in details games played between four different bots using 400 Hearthstone decks. These games can be used to estimate win-rate of the decks and learn how particular cards are used by the bots. Based on this knowledge, the task for participants is to estimate win-rates of a different set of 200 decks, played by the same bots.

The decks used in the provided collection of training games are given in a separate file named trainingDecks.json. Each row of this file corresponds to a different deck and stores a JSON with a deck identifier, hero identifier, and a list of card names along with their cardinality in the deck (there can be one or two cards of the same type in a deck). Analogically, the descriptions of test decks are provided in the same format in the file testDecks.json. It is allowed to use external knowledge bases about Hearthstone cards as long as they are publically available and their source is clearly stated in the submitted competition report. One example of such a source is the HearthPwn portal.

For the convenience of participants, we provide an additional table training_games.csv, whose rows correspond to JSON files describing the games between bots. Each row of this table gives a simple summary of the corresponding game. In particular, it provides ids of the playing bots, decks which they were using and the result of the game.

The format of submissions: The participants of the competition are asked to predict win-rates of 200 Hearthstone decks described in the file testDecks.json for the four bots which were used to generate the training data. The predictions should be sent using the submission system which will become available on April 13, 23:59 GMT. A file with a solution should consist of 800 lines, formatted in the same way as the exemplary solution provided as the file testSubmissionTemplate.csv. In each line, there should be an identifier of a bot, id of a deck and the predicted win-rate separated by semicolons. The win-rates should be expressed as percentages (real numbers between 0.0 and 100.0).

Evaluation of results: The submitted solutions will be evaluated online and the preliminary results will be published on the competition leaderboard. The preliminary score will be computed on a small subset of the test records, fixed for all participants. The final evaluation will be performed after completion of the competition using the remaining part of the test records. Those results will also be published online. It is important to note that only teams which submit a report describing their approach before the end of the contest will qualify for the final evaluation. The winning teams will be officially announced during a special session devoted to this competition, which will be organized at the FedCSIS'18 conference.

The assessment of solutions will be done using the RMSE measure. In order to keep the rule that the higher the score the better result, scores on the Leaderboard will correspond to minus RMSE values. For every pair bot-test deck, the reference win-rates were computed only based on games with the bots and decks from the training data.

In order to download competition files you need to be enrolled.

The award ceremony will take place during the FedCSIS'18 conference (Sep 9-12, 2017, Poznań, Poland).

Rank Team Name Score Submission Date
1
hieuvq
-5.573399 2018-05-5 15:36:51
2
amy
-5.648201 2018-05-6 17:48:02
3
jj
-5.667595 2018-05-7 01:04:51
4
dymitrruta
-5.696228 2018-05-6 23:10:06
5
amorgun
-5.847379 2018-05-5 09:17:02
6
a.ruta
-5.860696 2018-04-22 01:13:19
7
mrgrizz
-5.934092 2018-05-1 14:30:09
8
kmichael08
-5.981266 2018-05-6 21:48:39
9
towca
-6.043132 2018-05-6 21:07:39
10
adamwitkowski
-6.349185 2018-05-1 23:17:34
11
karol
-6.667479 2018-05-5 17:44:58
12
db346864
-7.441551 2018-04-19 01:42:19
13
tkacperek
-7.543330 2018-05-4 18:19:52
14
tgarbus
-7.612707 2018-04-26 14:50:27
15
kabeem
-7.742122 2018-05-7 00:16:51
16
leafproduction
-7.779585 2018-05-6 20:39:40
17
qwerty
-7.840545 2018-05-7 01:10:15
18
tronowski
-7.863463 2018-05-5 20:22:14
19
isia
-8.112951 2018-04-21 19:02:13
20
siwy
-8.116051 2018-04-28 12:46:54
21
whatisgoingon
-8.387689 2018-04-26 23:15:44
22
subuk
-8.392594 2018-05-6 19:08:47
23
bandy
-8.548265 2018-05-6 23:56:20
24
bayes_brothers
-8.695642 2018-04-28 12:25:31
25
mw371854
-8.721913 2018-04-30 17:26:03
26
borysp
-9.222077 2018-05-4 23:20:03
27
godul
-9.404691 2018-05-6 23:20:32
28
msonic
-9.571263 2018-04-28 18:31:32
29
janekf
-10.535439 2018-05-6 16:43:31
30
ip360730
-10.691340 2018-05-7 01:14:52
31
bearstrikesback
-10.736522 2018-05-6 16:09:19
32
kasrad
-10.842473 2018-05-6 23:24:04
33
optymista
-12.168536 2018-05-4 21:25:10
34
qwymierne
-12.518539 2018-05-6 22:53:31
35
szarki
-13.033823 2018-05-6 23:54:47
36
adi_nar
-13.152160 2018-05-7 01:03:33
37
raulmm7
-15.137684 2018-05-3 14:21:21
38
csiluszyk
-17.246092 2018-05-6 23:50:31
39
sodar
-17.246092 2018-05-7 00:04:37
40
gg370808
-49.867121 2018-05-6 22:10:53
41
kobrar
-49.967878 2018-05-7 01:55:18
42
tbb
No report file found or report rejected. 2018-04-14 12:51:09
43
alexionby
No report file found or report rejected. 2018-04-18 11:44:38
44
czlowiekrakieta
No report file found or report rejected. 2018-05-6 22:48:43
45
alphapred
No report file found or report rejected. 2018-04-26 17:31:22
46
francoisw89
No report file found or report rejected. 2018-04-24 05:34:31
47
asztyber
No report file found or report rejected. 2018-04-26 21:10:30
48
dishonesty
No report file found or report rejected. 2018-05-1 16:46:28
49
rekcahd
No report file found or report rejected. 2018-04-23 17:43:07
50
dniwe
No report file found or report rejected. 2018-04-24 14:01:01
51
iwannabetheverybest
No report file found or report rejected. 2018-04-13 21:50:06
52
baseline_solution
No report file found or report rejected. 2018-04-13 23:26:54
53
sadfsahfdasdf
No report file found or report rejected. 2018-04-30 16:49:44
54
jvdputten
No report file found or report rejected. 2018-04-16 13:00:59
55
lraszkiewicz
No report file found or report rejected. 2018-04-24 12:39:49
56
marseel
No report file found or report rejected. 2018-04-14 17:29:16
57
gerstorger
No report file found or report rejected. 2018-04-27 14:37:20
58
vadiaceu
No report file found or report rejected. 2018-04-17 14:33:52
59
ducktile
No report file found or report rejected. 2018-04-23 17:24:59
60
darkshadow
No report file found or report rejected. 2018-05-1 04:29:19
61
grandgob
No report file found or report rejected. 2018-05-1 05:04:49
62
windblade
No report file found or report rejected. 2018-05-1 06:10:30
63
vexfight
No report file found or report rejected. 2018-05-1 06:12:19
64
swiftroar
No report file found or report rejected. 2018-05-1 06:32:56
65
vividfight
No report file found or report rejected. 2018-05-1 06:48:26
66
skybinder
No report file found or report rejected. 2018-05-1 07:56:00
67
grapplebelt
No report file found or report rejected. 2018-05-1 08:22:13
68
dawnfather
No report file found or report rejected. 2018-05-1 09:21:45
69
shiftbuttons
No report file found or report rejected. 2018-05-1 09:58:59
70
randomseed19
No report file found or report rejected. 2018-05-1 16:57:11
71
ramich
No report file found or report rejected. 2018-05-7 01:36:05
72
twsthomas
No report file found or report rejected. 2018-04-23 16:42:56
73
doubleloop
No report file found or report rejected. 2018-05-6 23:32:35
74
rgod
No report file found or report rejected. 2018-05-7 01:41:52
75
hjasud
No report file found or report rejected. 2018-04-29 01:35:32
76
mateuszjanczura
No report file found or report rejected. 2018-05-6 15:32:50
77
pp332493
No report file found or report rejected. 2018-05-5 19:20:21
78
luki4824
No report file found or report rejected. 2018-04-28 15:13:54
79
adambak
No report file found or report rejected. 2018-04-21 12:17:09
80
podludek
No report file found or report rejected. 2018-04-19 13:02:45
81
pabloxrl
No report file found or report rejected. 2018-05-5 11:58:53
82
mpk
No report file found or report rejected. 2018-04-26 14:52:53
  • April 3, 2018: start of the competition, training data become available,
  • May 6, 2018 (23:59 GMT): deadline for submitting the predictions,
  • May 7, 2018 (23:59 GMT): deadline for sending the reports, end of the competition,
  • May 15, 2018: online publication of the final results, sending invitations for submitting short papers for the special session at FedCSIS'18,

Authors of the top-ranked solutions (based on the final evaluation scores) will be awarded prizes funded by our sponsors:

  • First Prize: 1000 USD + one free FedCSIS'18 conference registration,
  • Second Prize: 500 USD + one free FedCSIS'18 conference registration,
  • Third Prize: one free FedCSIS'18 conference registration.

The award ceremony will take place during the FedCSIS'18 conference (Sep 9-12, 2017, Poznań, Poland).

  • Andrzej Janusz, University of Warsaw & eSensei
  • Maciek Świechowski, Warsaw University of Technology & Silver Bullet Labs
  • Tomasz Tajmajer, University of Warsaw & Silver Bullet Labs
  • Łukasz Grad, University of Warsaw & eSensei
  • Jacek Puczniewski, Silver Bullet Labs & eSensei
  • Dominik Ślęzak, University of Warsaw & eSensei

In case of any questions please post on the competition forum or write an email at janusza {at} mimuw.edu.pl.