1 month ago

IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty

The aim of the competition is to predict the difficulty of chess puzzles based on board configurations and moves that the solution to each puzzle consists of. The difficulty level is measured as the rating on the lichess platform. The top 3 solutions will be awarded prizes. IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty is the sixth data science competition organized in association with the IEEE International Conference on Big Data series (IEEE BigData 2024, https://www3.cs.stonybrook.edu/~ieeebigdata2024/index.html).

Overview

A chess puzzle is a particular configuration of pieces on a chessboard, where the puzzle taker is instructed to assume the role of one of the players and continue the game from that position. The player has to find from one to several moves, until she delivers mate or obtains a decisive material advantage.

In the online setting, where these are often solved, the puzzle taker only makes moves from one side, while the puzzle publisher provides responses from the other side. One such puzzle solving service is Lichess Training

Solving puzzles is considered one of the primary ways to hone chess skills. However, currently the only way to reliably estimate puzzle difficulty is to present it to a wide variety of chess players and see if they manage to solve it.

The goal of the contest is to predict how difficult a chess puzzle is just by looking at the board setup and the moves in the solution. Puzzle difficulty is measured by its Glicko-2 rating calibrated on the lichess.org website. In simplified terms, it means that lichess models the difficulty of a puzzle by assuming that every attempt at solving a puzzle is a “match”. If a user solves the puzzle correctly, she gains puzzle rating and the puzzle loses rating. The opposite happens when the user doesn’t find the full solution (partial solutions count as “losses”). Both user and puzzle ratings are initialized at 1500. More information about the Glicko rating can be found here.

Each chess puzzle is described by the initial position (using Forsyth–Edwards Notation, or FEN) and the moves included in the puzzle solution, starting with one move leading to the puzzle position and then alternating between the moves that the puzzle solver has to find and those made by the simulated “opponent”.

IEEE Big Data 2024: We will encourage the top 3 winners to submit papers describing their solutions. It is already agreed that the conference will provide the top 3 winners with free registrations. The QED Software’s team, just like in the previous years, intends to organize a workshop devoted to the competition outcomes. According to our experience, the ability to present workshop papers may be an extra incentive for participants to consider active involvement in the competition.

Terms & Conditions

Competition Participation Rules for "IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty”

By entering this Competition you accept these official Competition rules.

1. Organizer

The Competition is organized by QED Software sp. z o. o. (the Organizer), registered at ul. Mazowiecka 11/49, 00-052 Warszawa. The Organizer’s website is: https://qed.pl/.
The Competition is sponsored by IEEE Big Data 2024 and QED Software sp. z o. o. (the Sponsors)
The Competition is organized via the KnowledgePit.ai platform (available at knowledgepit.ai or knowledgepit.ml) and any submissions made outside of the platform will not be admissible.

2. Entry

The Competition is open to all interested researchers, specialists, and students.
Members of the Contest Organizing Committee (see the dedicated section on the Competition page) and employees of the Organizer, the Sponsors, and their affiliated entities are not allowed to participate.
Persons who are residents of, or are affiliated with or employed by or otherwise contractually or legally tied to an organization, educational institution, company, or other entity of the Russian Federation or another state or territory that falls under the scope of international sanctions or controls for reasons of war, terrorism or otherwise, are not allowed to participate. The current list of such legal restrictions in force in the European Union is available at https://sanctionsmap.eu/#/main. If you have any doubt about whether such restriction may apply to you, the Organizer reserves the right to verify eligibility and to adjudicate your eligibility at any time.
Participants must be of legal age of majority (Check your country’s law, in Poland and in most countries it is 18 years of age).
Registration for the Competition is done via the KnowledgePit.ai platform (knowledgepit.ai or knowlegepit.ml).

3. Timeline

A detailed Competition time schedule is available in the Schedule section (below) of the Competition page.

4. Participants and Teams

A person may enrol in the Competition with only one KnowledgePit.ai user account. Using multiple accounts constitutes grounds for exclusion from the Competition.
Participants submit their solutions as members of teams made up of one or more persons.
Each participant may be a member of only one Team enrolled in the challenge.
Each Team must designate one of the Team Members as the Team Leader responsible for communication with the Organizer.
A single KnowledgePit.ai account can only be associated with one Team in a given competition. It is not possible to withdraw from a Team, but Teams can be merged.
Merging is done between Teams by their respective Team Leaders through the KnowledgePit.ai platform and requires the consent of both Teams.
Participation in the Competition is free and voluntary.
Privately sharing elements of the solution such as code or data outside of Teams is not permitted and may result in disqualification of the persons or Teams involved, however, it is permitted to share remarks and ideas with all participants on the KnowledgePit.ai Competition Forums. This is consistent with practices on most data science competition platforms.
Participants must use their own resources, particularly software and other necessary tools and equipment needed to prepare a solution to be submitted.

5. Solutions

Participants submit their solutions through the KnowledgePit.ai platform.
The submitted solution is checked for errors, and if none is found, the solution is immediately included on the public Leaderboard. The public Leaderboard will be made visible to the participants no later than May 30-th 2024.
There is a strict limit on the total number of solutions that can be submitted by a Team during the Competition and also, on the number of solutions that can be submitted daily.
For this Competition, those limits are set to 500 and 10, respectively.
The daily submission limit resets at 11:59 PM GMT.
When two Teams merge, their solutions accumulate, but the limits for the resulting Team remain unchanged.
If a Team exceeds the limit for the total number of submissions, the Team will be unable to submit new solutions until the Competition ends.
A team can select up to three solutions for the final evaluation, and the best of them will constitute the Team’s final score (Final Solution).
If a Team fails to select its 3 solutions for final evaluation, then this team’s solution with the highest score on the public Leaderboard is automatically selected as the Team’s Final Solution.

6. Evaluation

In order to be eligible for evaluation, each Team is obliged to provide a short Report describing their Final Solution. The Report must contain information such as the name of the Team, the names of all Team Members, and a brief overview of the approach used in the solution. The description should explain all data pre-processing steps and model construction steps. It should be submitted in the KnowledgePit.ai platform submission system by the Report Submission Date specified in the Schedule section of the Competition page (below).
The final evaluation takes place after the expiry of the Report Submission Date.
After the Final Evaluation, three top-ranked Teams will be asked to provide the source codes that can be used to reproduce their Final Solutions and Documentation that would allow running the code. If the code requires to be run within a complex environment (e.g. distributed Hadoop cluster), a detailed setup explanation should be provided as well. The source codes will be used to verify the legitimacy of the solutions and will be shared with the Competition Sponsors in accordance with the provisions of Section 9 below.
For a Team to be eligible for an Award in this Competition, a Team must be one of the three top-ranked teams. Moreover, a Team must provide the report and source codes upon request for the verification of their legitimacy.
The Organizer holds the right to extend the deadlines for submitting solutions and/or reports. In such a case, participants will be informed about such an extension through the KnowledgePit.ai platform Competition Forum.
The Organizer is not responsible for any consequences of technical issues related to the participant’s ability to access or submit to the KnowledgePit.ai platform, especially for issues relating to Internet connectivity. If due to technical problems within the Organizer’s control, the resulting delays or temporary unavailability of the platform or its components are likely to impact the participant’s ability to timely submit the Team’s solution, the challenge deadline will be extended by the Organizer by the time of such unavailability, and participants will be informed of such extension by email.
The final ranking of the competing Teams will be published based on the final evaluation results.
In the case of draws in the evaluation scores, the time of the submission will be taken into account.

7. Prizes

The Prizes are listed in the dedicated section of the Competition page.
Each prize comprises two components: a monetary amount and a non-monetary prize of one full IEEE BigData 2024 conference registration fee waiver. The Sponsors shall cover a maximum of one IEEE BigData 2024 registration fee per winning team and one monetary prize. The registration fee is a prize component that is non-monetary and cannot be converted into a monetary value.
Following the announcement of the winners, each winning Team must select their representative Team Member to participate in the 2024 IEEEE International Conference on Big Data and notify their selection to the Organizer within 14 days following the announcement. Once a representative has been appointed, the Team may change their representative by notifying the Organiser no later than 21 days before the IEEE BigData 2024 Conference. Any changes notified at a later date may be refused due to organisational constraints of the conference, however Organiser will use its best endeavours to include them.

4. If no Team Member is appointed by the winning Team to participate in the conference, the Team’s prize shall be deemed forfeited and shall pass to the next winning Team.

5. The monetary prizes shall be paid out in person during the IEEE BigData 2024 Conference directly to the representative of the winning Team appointed as per sec. 7.3. Should the appointed representative of a winning Team fail to appear at the conference, the prize shall be deemed forfeited and will not be allocated.

6. In order to receive the Prize, each prize recipient is required to submit to the Sponsors all documents necessary for the remittance of such prizes, such as their personal details, including bank account number, their certificate of tax residency issued by the tax authority of their place of residence or other documents required by law. If such necessary documents are not submitted within the time limit indicated in the notification of the award, the monetary prize cannot be paid out and shall deemed to have been forfeited by the winner. In such a case, the winning Team may select another of its members as the recipient of such a prize, provided however that the time limits specified in sec. 7.3 are observed.

8. Claims

Submissions are not admissible if they are in whole or part illegible, incomplete, damaged, altered, counterfeit, obtained through fraud, or late, or made or submitted in breach of these Rules.
Participant may be disqualified if the Organizer reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by using multiple accounts, cheating, deception, or other unfair playing practices or abuses, threats, or harassment towards other participants or the Organizer.
Organizer may at its discretion reject any submission or disqualify any participant if the Organizer reasonably believes that it was – respectively - produced in an unfair or illegitimate way or submitted by a person who has broken the challenge rules without providing any additional explanation.
If you think you were excluded without reason, or have any questions, please contact the Organizer at contact@knowledgepit.ai.

9. Data security, Privacy and Copyright

Each participant agrees to use reasonable and suitable measures to prevent persons who have not formally agreed to these Rules from gaining access to the Competition data.
You agree not to transmit, duplicate, publish, redistribute, or otherwise provide or make available the Competition data to any party not participating in the Competition. You agree to notify the Organizer immediately upon learning of any possible unauthorized transmission of or unauthorized access to the Competition data and agree to work with the Organizer to rectify any unauthorized transmission or access.
Each report, paper, and any other type of publication based on the team’s or participant’s research where data from this Competition is used should accredit KnowledgePit.ai, the Organizer and Sponsors as the institutions that provided data for the study.
The fact of accepting the award is equivalent to granting to the Organizers and Sponsors a worldwide, non-exclusive, sub-licensable, transferable, royalty-free, perpetual, and irrevocable right to use, reproduce, distribute, create derivative works of, publicly perform, publicly display, digitally perform, make, have made, sell, offer for sale and/or import, the winning solution submitted and the source code used to generate it, in any media now known or hereafter developed, for any purpose whatsoever, commercial or otherwise, without further approval, and without any payment to the participant or participants who authored or co-authored it. By accepting the award the participants also acknowledge that they have full and unrestricted rights to grant the aforementioned rights.
By enrolling in this Competition, the participant grants his/her consent for the processing of her/his registration data and the submissions and reports, and grants the Organizer the rights to use such data and submissions for the purpose of evaluation of solutions, competition administrative purposes and in post-competition research.
By accepting the award the winning participant grants the Organizer the right to process his or her personal data such as name, address, personal identification or security number, bank account or credit card number, and other necessary details provided for the purposes of prize processing and payment, including the payment of applicable taxes.
By accepting the award the participant grants the Organizer the right to use the participant's name, affiliation, and/or prize information for the purpose of informational and promotional purposes of the Competition and the KnowledgePit.ai platform in any medium without additional compensation.
Participant’s data is administered by the Organizer and shall be processed in accordance with the European data protection and privacy rules (The General Data Protection Regulation (EU) 2016/679). For more information check Organizer’s Privacy Policy.

10. Final arrangements

The Organizer reserves the right to modify the Rules of this Competition, including without limitation for the purpose of clarification, correcting obvious editing mistakes, extending the deadlines for the benefit of participants, or other minor amendments. In the event of any change to the Rules, participants will be informed of them via the Competition forum.
Unless otherwise provided in the Competition Rules above, all claims arising out of or relating to these Rules will be governed by Polish law and will be litigated in Poland. Participants consent to personal jurisdiction in those courts.
If any provision of these Rules is held to be invalid or unenforceable, all remaining provisions of the Rules will remain in full force and effect.

Enroll

Please log in to the system!

News

The Competition is Over!

Sincere thanks to all participants and congratulations to the winners!

The top3 teams will receive prizes and free full registration to IEEE Big Data conference.

In addition, selected teams (based on the score and interesting report) will be invited to participate in the special session during IEEE Big Data conference.

Task description

The data are provided as two .csv files, one for training dataset and one for testing dataset.

Each row of the testing dataset consists of the following fields:

Field name	Field description	Field type	Example value
PuzzleId	Unique puzzle ID	string	00sHx
FEN (link)	Standard notation for describing a particular board position of a chess game.	string	q3k1nr/1pp1nQpp/3p4/1P2p3/4P3/B1PP1b2/B5PP/5K2 b k - 0 17
Moves	Solution to the puzzle in Portable Game Notation (PGN). Includes the last move made before the puzzle position.	string	e8d7 a2e6 d7d8 f7f8

Based on the above data, the challenge contestants are expected to predict the Rating field (which will be kept secret).

Field name	Field description	Field type	Example value
Rating	Puzzle rating	int	1760

The training dataset contains all of the above fields, and also a few additional ones listed below. These fields are sometimes null in the training set and will not be provided for the test set:

RatingDeviation (int): Measure of uncertainty over puzzle’s difficulty.

Popularity (int): Users can ”upvote“ or “downvote” a puzzle. This value is the difference between the number of upvotes and downvotes.

NbPlays (int): Number of attempts at solving the puzzle.

Themes (str): Lichess allows choosing puzzles to solve based on different themes, such as tactical concepts, solution length or puzzle types (e.g. mates in x moves).

GameUrl (str): Lichess puzzles are generated based on games played on lichess.

OpeningTags (str): Information about the opening from which this puzzle originated.

Solution format

Solutions in this competition should be submitted to the online evaluation system as a text file with exactly 2282 lines containing predictions for test instances. Each line in the submission should contain a single integer that indicates the predicted rating of the chess puzzle. The ordering of predictions should be the same as the ordering of the test set.

Evaluation

The quality of submissions will be evaluated using the mean squared error metric.

Solutions will be evaluated online, and the preliminary results will be published on the public leaderboard. The public leaderboard will be available starting May 30th. The preliminary score will be computed on a small subset of the test records, fixed for all participants. The final evaluation will be performed after the completion of the competition using the remaining part of the test records. Those results will also be published online. It is important to note that only teams that submit a report describing their approach before the end of the challenge will qualify for the final evaluation.

Data files

There are two data files available to download.

Final results

Rank	Team Name	Is Report		Preliminary Score	Final Score	Submissions
1	bread emoji	True	True	49141.5359	104540.656891	47
2	anansch	True	True	58810.4586	120682.234604	38
3	ousou	True	True	69890.9227	123103.229717	50
4	Andryyyyy	True	True	61381.3812	129245.229228	56
5	ToDoFindATeamName	True	True	65136.8232	132631.424731	53
6	alexmolas	True	True	74378.0110	137839.668622	22
7	dymitr	True	True	69202.5691	141488.501466	50
8	Feiwyth	True	True	70792.7182	146729.234115	6
9	NxGTR	True	True	73832.3591	150757.042522	35
10	BigData2024	True	True	74135.4586	154905.759042	51
11	transformer_enjoyer	True	True	75995.0221	158292.081623	27
12	neuralnite	True	True	82049.4972	160677.509775	6
13	MrAces	True	True	78837.4807	163552.500000	43
14	DML	True	True	91728.0055	170533.507331	133
15	shoggoth	True	True	87533.5193	171309.505376	68
16	AIBrain	True	True	91889.9503	189554.737537	43
17	Marek	True	True	119067.6243	195828.732649	172
18	LcWP	True	True	167116.2762	244485.950147	4
19	JustEngine	False	True	67827.4254	No report file found or report rejected.	101
20	deep	False	True	81429.3204	No report file found or report rejected.	4
21	baellouf	False	True	82238.9890	No report file found or report rejected.	43
22	September	False	True	84712.2541	No report file found or report rejected.	19
23	Amy	False	True	91476.2762	No report file found or report rejected.	78
24	JKU-CODA	False	True	91664.9890	No report file found or report rejected.	20
25	Plats Bruts	False	True	98152.9503	No report file found or report rejected.	16
26	hieuvq	False	True	101897.2818	No report file found or report rejected.	10
27	checkmate	False	True	101972.7072	No report file found or report rejected.	4
28	scotchgame	False	True	85906.9503	No report file found or report rejected.	5
29	kubapok	False	True	120870.0718	No report file found or report rejected.	4
30	Fontageau	False	True	122314.7127	No report file found or report rejected.	2
31	soksly	False	True	123260.3646	No report file found or report rejected.	9
32	French_bestbytest	False	True	131666.9171	No report file found or report rejected.	13
33	tafhi	False	True	135894.2928	No report file found or report rejected.	2
34	fuzz	False	True	142167.5028	No report file found or report rejected.	3
35	Cavajah	False	True	149984.5028	No report file found or report rejected.	3
36	Azeezah	False	True	169580.3370	No report file found or report rejected.	1
37	OrganizerTest	False	True	187245.4199	No report file found or report rejected.	1

Schedule

May 08, 2024: start of the competition, datasets become available,
May 30, 2024: public leaderboard becomes available
August 31, 2024: deadline for submitting the solutions,
September 12 (extended), 2024: deadline for sending the reports, end of the competition,
September 15, 2024: online publication of the final results, sending invitations for submitting papers to the associated workshop at the IEEE Big Data 2024 conference,
October 13, 2024: deadline for submitting invited papers,
October 28, 2024: notification of paper acceptance,
November 17, 2024: camera-ready of accepted papers due.

Awards

QED will sponsor the cash prizes:

1000 USD for the winning solution
500 USD for the 2nd place solution
250 USD for the 3rd place solution

Additionally, the IEEE Big Data 2024 conference will provide the top 3 performers with free full registrations.

Contest organizing committee

Jan Zyśko
Katarzyna Jagieła
Maciej Świechowski
Sebastian Stawicki
Andrzej Janusz
Dominik Ślęzak
Zbigniew Pakleza

Forum

This forum is for all users to discuss matters related to the competition. Good manners apply!

Discussion	Author	Replies	Last post
Paper Submission	Anan	4	by Wednesday, September 18, 2024, 14:55:26
Final Score		2	by Wednesday, September 18, 2024, 14:54:17
Question about final 3 choices		3	by Maciej Friday, September 13, 2024, 07:10:34
Dude, Report submission is working now, please submit your report before the new deadline	M	0	by M Wednesday, September 11, 2024, 17:59:04
Problem sending the report ,	Abde	2	by Wednesday, September 11, 2024, 15:12:59
could you please open the report submission again for at least one day? as the website was down for the past few days.	M	5	by Anan Tuesday, September 10, 2024, 12:30:44
When exactly is the deadline for submitting solutions?		8	by M Monday, September 09, 2024, 09:08:20
Player initial rating deviation		1	by Friday, August 23, 2024, 14:33:17
Player initial rating deviation		0	by Friday, August 23, 2024, 14:08:56
Test set rating calculations	Anan	4	by Anan Thursday, August 22, 2024, 06:06:26
Final Evaluation Question		2	by Wednesday, August 21, 2024, 15:55:48
Long evaluation time	Szymon	1	by Maciej Sunday, August 04, 2024, 17:19:29
How to add new team memebers in the team	Abdul	1	by Maciej Monday, July 15, 2024, 22:19:37
Test set	MAROUANE	2	by MAROUANE Thursday, July 04, 2024, 09:14:02
Is test set from the same distribution as train set?	Alex	1	by Competition Wednesday, June 26, 2024, 17:18:14
Add other users to my team	Alex	1	by Maciej Wednesday, June 26, 2024, 12:25:26
Evaluation is online!	Maciej	3	by Maciej Wednesday, June 26, 2024, 11:03:14
Use external information	Alex	1	by Maciej Wednesday, June 26, 2024, 10:52:48
Looking for teammates		2	by Tuesday, June 25, 2024, 10:05:37
Puzzle taker vs simulated opponent	Dymitr	3	by Dymitr Wednesday, June 19, 2024, 18:04:47
Duplicate file in Your Team.	Carlos	5	by Maciej Monday, June 10, 2024, 13:12:04
Chess engine	Michal	1	by Maciej Tuesday, May 21, 2024, 13:57:38
Transfer learning	Łukasz	1	by Maciej Tuesday, May 14, 2024, 10:55:04