3 days, 10 hours from now

IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty

The aim of the competition is to predict the difficulty of chess puzzles based on board configurations and moves that the solution to each puzzle consists of. The difficulty level is measured as the rating on the lichess platform. The top 3 solutions will be awarded prizes. IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty is the sixth data science competition organized in association with the IEEE International Conference on Big Data series (IEEE BigData 2024, https://www3.cs.stonybrook.edu/~ieeebigdata2024/index.html).

See the detailed program of our competition presentations at IEEE BigData 2024 -
https://qedsoftware.com/IEEE_BigData_2024_Chess_and_Granulation.pdf

Overview

A chess puzzle is a particular configuration of pieces on a chessboard, where the puzzle taker is instructed to assume the role of one of the players and continue the game from that position. The player has to find from one to several moves, until she delivers mate or obtains a decisive material advantage.

In the online setting, where these are often solved, the puzzle taker only makes moves from one side, while the puzzle publisher provides responses from the other side. One such puzzle solving service is Lichess Training

Solving puzzles is considered one of the primary ways to hone chess skills. However, currently the only way to reliably estimate puzzle difficulty is to present it to a wide variety of chess players and see if they manage to solve it. 

The goal of the contest is to predict how difficult a chess puzzle is just by looking at the board setup and the moves in the solution. Puzzle difficulty is measured by its Glicko-2 rating calibrated on the lichess.org website. In simplified terms, it means that lichess models the difficulty of a puzzle by assuming that every attempt at solving a puzzle is a “match”. If a user solves the puzzle correctly, she gains puzzle rating and the puzzle loses rating. The opposite happens when the user doesn’t find the full solution (partial solutions count as “losses”). Both user and puzzle ratings are initialized at 1500. More information about the Glicko rating can be found here.

Each chess puzzle is described by the initial position (using Forsyth–Edwards Notation, or FEN) and the moves included in the puzzle solution, starting with one move leading to the puzzle position and then alternating between the moves that the puzzle solver has to find and those made by the simulated “opponent”.

IEEE Big Data 2024: We will encourage the top 3 winners to submit papers describing their solutions. It is already agreed that the conference will provide the top 3 winners with free registrations. The QED Software’s team, just like in the previous years, intends to organize a workshop devoted to the competition outcomes. According to our experience, the ability to present workshop papers may be an extra incentive for participants to consider active involvement in the competition. 

The aim of the competition is to predict the difficulty of chess puzzles based on board configurations and moves that the solution to each puzzle consists of. The difficulty level is measured as the rating on the lichess platform. The top 3 solutions will be awarded prizes. IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty is the sixth data science competition organized in association with the IEEE International Conference on Big Data series (IEEE BigData 2024, https://www3.cs.stonybrook.edu/~ieeebigdata2024/index.html).

See the detailed program of our competition presentations at IEEE BigData 2024 - https://qedsoftware.com/IEEE_BigData_2024_Chess_and_Granulation.pdf

Terms & Conditions
 
 
Please log in to the system!
News

The Competition is Over!

Sincere thanks to all participants and congratulations to the winners!

The top3 teams will receive prizes and free full registration to IEEE Big Data conference.

In addition, selected teams (based on the score and interesting report) will be invited to participate in the special session during IEEE Big Data conference.

 

The data are provided as two .csv files, one for training dataset and one for testing dataset.

Each row of the testing dataset consists of the following fields:

Field name

Field description

Field type

Example value

PuzzleId

Unique puzzle ID

string

00sHx

FEN (link)

Standard notation for describing a particular board position of a chess game.

string

q3k1nr/1pp1nQpp/3p4/1P2p3/4P3/B1PP1b2/B5PP/5K2 b k - 0 17

Moves

Solution to the puzzle in Portable Game Notation (PGN). Includes the last move made before the puzzle position.

string

e8d7 a2e6 d7d8 f7f8

Based on the above data, the challenge contestants are expected to predict the Rating field (which will be kept secret).

Field name

Field description

Field type

Example value

Rating

Puzzle rating

int

1760

 

The training dataset contains all of the above fields, and also a few additional ones listed below. These fields are sometimes null in the training set and will not be provided for the test set:

RatingDeviation (int): Measure of uncertainty over puzzle’s difficulty.

Popularity (int): Users can ”upvote“ or “downvote” a puzzle. This value is the difference between the number of upvotes and downvotes.

NbPlays (int): Number of attempts at solving the puzzle.

Themes (str): Lichess allows choosing puzzles to solve based on different themes, such as tactical concepts, solution length or puzzle types (e.g. mates in x moves).

GameUrl (str): Lichess puzzles are generated based on games played on lichess.

OpeningTags (str): Information about the opening from which this puzzle originated.

Solution format 

Solutions in this competition should be submitted to the online evaluation system as a text file with exactly 2282 lines containing predictions for test instances. Each line in the submission should contain a single integer that indicates the predicted rating of the chess puzzle. The ordering of predictions should be the same as the ordering of the test set.

Evaluation

The quality of submissions will be evaluated using the mean squared error metric. 

Solutions will be evaluated online, and the preliminary results will be published on the public leaderboard. The public leaderboard will be available starting May 30th. The preliminary score will be computed on a small subset of the test records, fixed for all participants. The final evaluation will be performed after the completion of the competition using the remaining part of the test records. Those results will also be published online. It is important to note that only teams that submit a report describing their approach before the end of the challenge will qualify for the final evaluation.

 

There are two data files available to download.

 

Rank Team Name Score Submission Date
1
bread emoji
49141.5359 2024-08-20 18:06:14
2
anansch
58810.4586 2024-08-16 21:26:42
3
Andryyyyy
61381.3812 2024-08-4 19:20:02
4
ToDoFindATeamName
65136.8232 2024-08-29 19:13:44
5
JustEngine
67827.4254 2024-07-31 16:11:42
6
dymitr
69202.5691 2024-07-12 08:35:28
7
ousou
69890.9227 2024-08-25 10:04:33
8
Feiwyth
70792.7182 2024-08-30 21:33:00
9
NxGTR
73832.3591 2024-06-14 23:16:07
10
BigData2024
74135.4586 2024-08-30 15:59:23
11
alexmolas
74378.0110 2024-07-16 19:25:18
12
transformer_enjoyer
75995.0221 2024-08-4 20:32:12
13
MrAces
78837.4807 2024-07-6 18:03:22
14
deep
81429.3204 2024-06-20 13:57:20
15
neuralnite
82049.4972 2024-08-1 18:48:23
16
baellouf
82238.9890 2024-07-9 20:22:34
17
September
84712.2541 2024-08-30 08:54:44
18
scotchgame
85906.9503 2024-06-22 06:46:56
19
shoggoth
87533.5193 2024-07-10 09:53:59
20
Amy
91476.2762 2024-08-18 22:17:50
21
JKU-CODA
91664.9890 2024-07-16 09:22:48
22
DML
91728.0055 2024-08-24 18:02:17
23
AIBrain
91889.9503 2024-08-4 14:43:16
24
Spyridon Mouselinos
98151.4641 2025-01-13 23:03:42
25
Plats Bruts
98152.9503 2024-07-13 08:44:17
26
hieuvq
101897.2818 2024-06-7 15:04:56
27
checkmate
101972.7072 2024-06-26 12:38:39
28
witek0509x
102988.8177 2025-01-20 22:56:15
29
Marek
119067.6243 2024-08-23 21:00:30
30
kubapok
120870.0718 2024-07-21 20:20:47
31
Fontageau
122314.7127 2024-08-3 04:05:30
32
soksly
123260.3646 2024-06-27 12:58:01
33
bodenlos
129274.2431 2025-01-20 20:55:20
34
French_bestbytest
131666.9171 2024-06-27 08:43:31
35
tafhi
135894.2928 2024-08-24 12:30:48
36
fuzz
142167.5028 2024-07-7 00:28:25
37
Narcos
145032.0166 2025-01-21 09:12:38
38
Cavajah
149984.5028 2024-06-18 02:08:09
39
Tommaso and Riccardo
159697.1105 2025-01-20 16:59:21
40
LcWP
167116.2762 2024-06-27 04:20:01
41
undefined
169199.1215 2025-01-10 12:49:42
42
Azeezah
169580.3370 2024-06-15 23:43:27
43
OrganizerTest
187245.4199 2024-05-29 16:59:05
  • May 08, 2024: start of the competition, datasets become available, 
  • May 30, 2024: public leaderboard becomes available
  • August 31, 2024: deadline for submitting the solutions, 
  • September 12 (extended), 2024: deadline for sending the reports, end of the competition, 
  • September 15, 2024: online publication of the final results, sending invitations for submitting papers to the associated workshop at the IEEE Big Data 2024 conference, 
  • October 13, 2024: deadline for submitting invited papers,
  • October 28, 2024: notification of paper acceptance,
  • November 17, 2024: camera-ready of accepted papers due.

QED will sponsor the cash prizes:

  • 1000 USD for the winning solution
  • 500 USD for the 2nd place solution
  • 250 USD for the 3rd place solution

Additionally, the IEEE Big Data 2024 conference will provide the top 3 performers with free full registrations

  • Jan Zyśko
  • Katarzyna Jagieła
  • Maciej Świechowski
  • Sebastian Stawicki
  • Andrzej Janusz
  • Dominik Ślęzak
  • Zbigniew Pakleza
This forum is for all users to discuss matters related to the competition. Good manners apply!
  Discussion Author Replies Last post
Top score methods 0 by
Monday, December 23, 2024, 21:47:12
Paper Submission Anan 4 by
Wednesday, September 18, 2024, 14:55:26
Final Score 2 by
Wednesday, September 18, 2024, 14:54:17
Question about final 3 choices 3 by Maciej
Friday, September 13, 2024, 07:10:34
Dude, Report submission is working now, please submit your report before the new deadline M 0 by M
Wednesday, September 11, 2024, 17:59:04
Problem sending the report , chess 2 by
Wednesday, September 11, 2024, 15:12:59
could you please open the report submission again for at least one day? as the website was down for the past few days. M 5 by Anan
Tuesday, September 10, 2024, 12:30:44
When exactly is the deadline for submitting solutions? 8 by M
Monday, September 09, 2024, 09:08:20
Player initial rating deviation 1 by
Friday, August 23, 2024, 14:33:17
Player initial rating deviation 0 by
Friday, August 23, 2024, 14:08:56
Test set rating calculations Anan 4 by Anan
Thursday, August 22, 2024, 06:06:26
Final Evaluation Question 2 by
Wednesday, August 21, 2024, 15:55:48
Long evaluation time Szymon 1 by Maciej
Sunday, August 04, 2024, 17:19:29
How to add new team memebers in the team Abdul 1 by Maciej
Monday, July 15, 2024, 22:19:37
Test set MAROUANE 2 by MAROUANE
Thursday, July 04, 2024, 09:14:02
Is test set from the same distribution as train set? Alex 1 by Competition
Wednesday, June 26, 2024, 17:18:14
Add other users to my team Alex 1 by Maciej
Wednesday, June 26, 2024, 12:25:26
Evaluation is online! Maciej 3 by Maciej
Wednesday, June 26, 2024, 11:03:14
Use external information Alex 1 by Maciej
Wednesday, June 26, 2024, 10:52:48
Looking for teammates 2 by
Tuesday, June 25, 2024, 10:05:37
Puzzle taker vs simulated opponent Dymitr 3 by Dymitr
Wednesday, June 19, 2024, 18:04:47
Duplicate file in Your Team. Carlos 5 by Maciej
Monday, June 10, 2024, 13:12:04
Chess engine Michal 1 by Maciej
Tuesday, May 21, 2024, 13:57:38
Transfer learning Łukasz 1 by Maciej
Tuesday, May 14, 2024, 10:55:04