ICME 2023 Grand Challenge

1 year, 9 months ago

ICME 2023 Grand Challenge: Predicting Frags in Tactic Games

ICME 2023 Grand Challenge: Predicting Frags in Tactic Games is the first data science competition organized in association with the IEEE International Conference on Multimedia and Expo (https://www.2023.ieeeicme.org/) conference series at KnowledgePit. The task continues the topic from IEEE BigData 2021 Cup: Predicting Victories in Video Games. This time, we ask participants to predict the chances of scoring a frag in a tactical video game Tactical Troops: Anthracite Shift (http://tacticaltroops.net) based on game screenshots and auxiliary information about the game. The competition is sponsored by the IEEE ICME 2023 conference.

Overview

Tactical Troops: Anthracite Shift is a top-down, turn-based game that mixes tactical skills and the feeling of 80's sci-fi movies. Players can command a troop of soldiers to fight on one of 24 multi-player maps with artificial bots or human opponents. The goal of the challenge is to learn an efficient model for predicting chances that a player will score a frag in the next game turn (i.e., eliminate at least one troop of the opposing player) using screenshot data and basic information about the game state. A high-quality prediction model for this task will be used in a number of analytical tasks related to the game. It will also help us design better AI-controlled agents for future video games.

More details regarding the task and the description of the challenge datasets are available in the Task description section below. The competition started on February 12 and ended on March 19.

Special session at IEEE ICME 2023: A special session devoted to the challenge will be held at the IEEE ICME 2023 conference. We will invite authors of selected competition reports to extend them for publication in the conference proceedings (after reviews by Organizing Committee members) and presentation at the conference. The publications will be indexed in the same way as regular conference papers. The invited teams will be chosen based on their final rank, the innovativeness of their approach, and quality of the submitted report.

Terms & Conditions

Competition Participation Rules for “ICME 2023 Grand Challenge: Predicting Frags in Tactic Games”

By entering this Competition you accept these official Competition rules.

1. Organizer

The Competition is organized and sponsored by the IEEE International Conference on Multimedia and Expo 2023 conference (IEEE ICME 2023, https://www.2023.ieeeicme.org/), hereinafter referred to as the Organizer. The Organizer shall sponsor the Prizes and duly distribute them to the winners selected in accordance with these Rules.
The Competition is organized via the KnowledgePit platform (available at https://knowledgepit.ai or https://knowledgepit.ml) owned and operated by eSensei sp. z o. o., a Polish company belonging to QED Software Group (https://qed.pl/).
The Competition Organizing Committee shall include representatives of the QED Software Group who shall assist in managing the competition on behalf of the Organizer.
Any submissions made outside of the platform are not admissible.
All datasets used in the competition were prepared and are property of QED Software sp. z o. o.

2. Entry

The Competition is open to all interested researchers, specialists, and students.
Members of the Competition Organizing Committee (see the dedicated section on the Competition page) and employees of the Organizer, Sponsor, or other company involved in operating and organizing the Competition, and their affiliated entities are not allowed to participate.
Persons who are residents of, or are affiliated with or employed by or otherwise contractually or legally tied to an organization, educational institution, company, or other entity of any state or territory that falls under the scope of international sanctions or controls for reasons of war, terrorism or otherwise, are not allowed to participate. The current list of such legal restrictions in force in the European Union is available at https://sanctionsmap.eu/#/main. If you have any doubt about whether such restriction may apply to you, the Organizer reserves the right to verify eligibility and to adjudicate your eligibility at any time.
Participants must be of legal age of majority (Check your country’s law, in Poland and in most countries it is 18 years of age).
Registration for the Competition is done via the KnowledgePit platform (https://knowledgepit.ai or https://knowlegepit.ml).

3. Timeline

A detailed Competition time schedule is available in the Schedule section (below) of the Competition page.

4. Participants and Teams

A person may enroll in the Competition with only one KnowledgePit user account. Using multiple accounts constitutes grounds for exclusion from the Competition.
Participants submit their solutions as members of teams made up of one or more persons.
Each participant may be a member of only one Team enrolled in the Competition.
Each Team must designate one of the Team Members as the Team Leader responsible for communication with the Organizer.
A single KnowledgePit account can only be associated with one Team in a given competition. It is not possible to withdraw from a Team, but Teams can be merged.
Merging is done between Teams by their respective Team Leaders through the KnowledgePit platform and requires the consent of both Teams.
Participation in the Competition is free and voluntary.
Privately sharing elements of the solution such as code or data outside of Teams is not permitted and may result in disqualification of the persons or Teams involved, however, it is allowed to share remarks and ideas with all participants on the KnowledgePit Competition Forums.
Participants must use their own resources, particularly software and other necessary tools and equipment needed to prepare a solution to be submitted.

5. Solutions

Participants submit their solutions through the Knowledge Pit platform.
The submitted solution is checked for errors, and if none is found, the solution is immediately included on the public Leaderboard.
There is a strict limit on the total number of solutions that can be submitted by a Team during the Competition and also, on the number of solutions that can be submitted daily.
For this Competition, those limits are set to 500 and 25, respectively.
The daily submission limit resets at 11:59 PM GMT.
When two teams merge, their solutions accumulate, but the limits for the resulting Team remain unchanged.
If a Team exceeds the limit for the total number of submissions, the Team will be unable to submit new solutions until the Competition ends.
The Team can select up to two solutions for the final evaluation, and the best of them will constitute the Team’s final score (Final Solution).
If a Team fails to select its two solutions for the final evaluation, then this Team’s solution with the highest score on the public Leaderboard is automatically selected as the Team’s Final Solution.

6. Evaluation

Each Team is obliged to provide a short Report describing their Final Solution. The Report must contain information such as the name of the Team, the names of all Team Members, and a brief overview of the approach used in the solution. The description should explain all data pre-processing steps and model construction steps. It should be submitted in the KnowledgePit platform submission system by the Report Submission Date specified in the Schedule section of the Competition page (below).
The final evaluation takes place after the expiry of the Report Submission Date.
After the Final Evaluation, three top-ranked Teams will be asked to provide the source codes that can be used to reproduce their Final Solutions and Documentation that would allow running the code. If the code requires to be run within a complex environment (e.g. distributed Hadoop cluster), a detailed setup explanation should be provided as well. The source codes will be used to verify the legitimacy of the solutions.
For a Team to be eligible for an Award in this Competition, a Team must be one of the three top-ranked teams. Moreover, a Team must provide the report and source codes upon request for the verification of their legitimacy.
In this Competition, the teams are eligible for monetary prizes only if their final solution improves the baseline.
The Organizer holds the right to extend the deadlines for submitting solutions and/or reports. In such a case, participants will be informed about such an extension through the KnowledgePit platform Competition Forum.
The Organizer is not responsible for any consequences of technical issues related to the participant’s ability to access or submit to the KnowledgePit platform, especially for issues relating to Internet connectivity. If due to technical problems within the Organizer’s control, the resulting delays or temporary unavailability of the platform or its components are likely to impact the participant’s ability to timely submit the Team’s solution, the challenge deadline will be extended by the Organizer by the time of such unavailability, and participants will be informed of such extension by email.
The final ranking of the competing Teams will be published based on the final evaluation results.
In the case of draws in the evaluation scores, the time of the submission will be taken into account.

7. Prizes

The Prizes are listed in the dedicated section of the Competition page. The Prizes will be awarded to the teams with the highest ranking, that meet the additional conditions of submitting the competition report, as well as code and necessary documentation.
Each prize comprises two components: a monetary amount and the ability for a selected Team Member to participate in the IEEE ICME 2023 conference. The Organizer shall cover a maximum of one registration fee per winning team. The registration fee is a prize component that is non-monetary and cannot be converted into a monetary value.
Each winning Team must select their representative Team Member to participate in the IEEE International Conference on Multimedia and Expo 2023 and notify their selection to the Organizer. If no Team Member is selected by his or her Team to participate in the conference, the registration fee shall be deemed forfeited and not constitute part of the prize.
Following the announcement of the winners, The Organizer shall pay the monetary prizes to the winning teams. Each winning team shall notify the Organizer within 21 days following the announcement of how the Team wishes the monetary prize to be paid out (single payment, split payments, etc.).
In order to receive the Prize, each prize recipient is required to submit to the Organizer all documents necessary for the remittance of such prizes, such as their personal details, including bank account number, their certificate of tax residency issued by the tax authority of their place of residence or other documents required by law as requested by the Organizer. If such necessary documents are not submitted within the time limit indicated in the notification of the award, the monetary prize cannot be paid out and is deemed to have been forfeited by the winner.
The winning participant who fails or refuses to provide the necessary legal documentation such as defined above within the time limit specified by the Organizer in the notification of the prize award will be deemed to have forfeited the monetary prize, and such prize will not be paid out. In such a case, the winning Team may select another of its members as the recipient of such a prize.
In accordance with the law, the Organizer shall cover the tax payments required on the awarded prizes at the rates and on terms prescribed by law in the country of the Organizer.
The Organizer is not responsible in any way for any duties, levies, fees, or taxes that the winning participant may be subject to in his or her country of residence. It is the sole responsibility of the winning participant to inquire at the appropriate authorities about applicable documentation and/or tax payments that she or he may be required to submit.

8. Claims

Submissions are not admissible if they are in whole or in part illegible, incomplete, damaged, altered, counterfeit, obtained through fraud, or late, or made or submitted in breach of these Rules.
A Participant may be disqualified if the Organizer reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by using multiple accounts, cheating, deception, or other unfair playing practices or abuses, threats, or harassment toward other participants or the Organizer.
The Organizer may at its discretion reject any submission or disqualify any participant if the Organizer reasonably believes that it was – respectively - produced in an unfair or illegitimate way or submitted by a person who has broken the challenge rules without providing any additional explanation.
If you think you were excluded without reason, or have any questions, please contact the Organizer at contact@knowledgepit.ai.

9. Data security, Privacy and Copyright

Each participant agrees to use reasonable and suitable measures to prevent persons who have not formally agreed to these Rules from gaining access to the Competition data.
You agree not to transmit, duplicate, publish, redistribute or otherwise provide or make available the Competition data to any party not participating in the Competition. You agree to notify the Organizer immediately upon learning of any possible unauthorized transmission of or unauthorized access to the Competition data and agree to work with the Organizer to rectify any unauthorized transmission or access.
Each report, paper, and any other type of publication based on the team’s or participant’s research where data from this Competition is used should accredit KnowledgePit, Organizer, and QED Software as the institutions that provided data for the study.
The fact of accepting the award is equivalent to granting to the Organizers and administrators of KnowledgePit a worldwide, non-exclusive, sub-licensable, transferable, royalty-free, perpetual, and irrevocable right to use, reproduce, distribute, create derivative works of, publicly perform, publicly display, digitally perform, make, have made, sell, offer for sale and/or import, the winning solution submitted and the source code used to generate it, in any media now known or hereafter developed, for any purpose whatsoever, commercial or otherwise, without further approval, and without any payment to the participant or participants who authored or co-authored it. By accepting the award the participants also acknowledge that they have full and unrestricted rights to grant the aforementioned rights.
By enrolling in this Competition, the participant grants his/her consent for the processing of her/his registration data and the submissions and reports by the Organizer and administrators of KnowledgePit and grants them the rights to use such data and submissions for the purpose of evaluation of solutions, competition administrative purposes and in post-competition research.
By accepting the award the winning participant grants the Organizer the right to process his or her personal data such as name, address, personal identification or security number, bank account or credit card number, and other necessary details provided, for the purposes of prize processing and payment, including the payment of applicable taxes.
By accepting the award the participant grants the Organizer the right to use the participant's name, affiliation, and/or prize information for the purpose of informational and promotional purposes of the Competition and the KnowledgePit platform in any medium without additional compensation.
Participants’ data is administered by the Organizer and administrators of KnowledgePit for the purposes of running the competition and selecting its winners and shall be processed in accordance with the applicable privacy protection rules. For more information check Organizer’s Privacy Policy and KnowledgePit Privacy Policy.

10. Final arrangements

The Organizer reserves the right to modify the Rules of this Competition, including without limitation for the purpose of clarification, correcting obvious editing mistakes, extending the deadlines for the benefit of participants, or other minor amendments. In the event of any change to the Rules, participants will be informed of them via the Competition forum.
If any provision of these Rules is held to be invalid or unenforceable, all remaining provisions of the Rules will remain in full force and effect.

Final results

Rank	Team Name	Is Report		Preliminary Score	Final Score	Submissions
1	icme2023	True	True	0.8946	0.873600	18
2	NxGTR	True	True	0.8412	0.820600	92
3	Dymitr	True	True	0.8533	0.812300	441
4	The Lord of the Machine Learning	True	True	0.8385	0.806200	108
5	Cyan	True	True	0.8249	0.800800	108
6	baseline	True	True	0.8189	0.799400	1
7	goldboy	True	True	0.8185	0.797700	5
8	witteam	True	True	0.8144	0.793700	2
9	MB	True	True	0.8058	0.789200	3
10	sk	True	True	0.8120	0.785000	57
11	Climber	True	True	0.7829	0.776900	7
12	hieuvq	True	True	0.8184	0.773100	246
13	PSO	True	True	0.7822	0.761500	9
14	fulltank	True	True	0.6292	0.615100	1
15	kubapok	False	True	0.7742	No report file found or report rejected.	3
16	DeepTeam	False	True	0.7191	No report file found or report rejected.	1
17	TheArtificialTeam	False	True	0.5408	No report file found or report rejected.	3

Enroll

Please log in to the system!

Task description

The task in this competition is to learn how to predict whether a moving player can score a frag in his/her turn of Tactical Troops: Anthracite Shift game.

The available data has three different modalities, i.e., each data instance is characterized by a brief description of a game state at the beginning of the player’s turn, meta-data of the map on which the game is played, and a game screenshot from the beginning of the turn. Each screenshot covers the whole game arena and allows the extraction of valuable information, such as exact troop positioning with respect to terrain features, their class, and primary weapon type. Names of screenshot files correspond to the identifiers of training and test instances.

The meta-data of the game maps is composed of collections of bitmap masks and a small metadata.json file corresponding to each map in the game. The bitmap masks indicate special map features, such as static obstacles, impassible terrain, etc. The json file contains general information about the map size and enables translation between in-game coordinates and screenshot pixels. For example, if someone wanted to pre-train a model on a task of finding troop positions and their classes on game screenshots, the ground truth positions in pixel coordinates can be computed as:

px = (x - x_min) * game_to_pixel_scale
py = (y_max - y) * game_to_pixel_scale

where x and y are the in-game coordinates from the description of a game state.

The training data contains 50000 instances (game turns) and the test set is composed of additional 10000 cases from a different collection of games. For training data, information about the target variable, i.e., whether the player was able to score a frag, is available in the table describing the game states. It corresponds to the column named frag. In the test data, values from that column are missing.

Solution format: The predictions for the test instances should be submitted to the online evaluation system as a text file. The file should have exactly 10000 lines, and each line should contain exactly one number from the [0,1] interval indicating the chance that the moving player scores at least one frag during the turn. The ordering of predictions should be the same as the ordering of instances in the game_states_data_test.csv table.

Evaluation: the quality of submissions will be evaluated using the AUC measure. Solutions will be evaluated online and the preliminary results will be published on the public leaderboard. The preliminary score will be computed on a small (10%) subset of the test records, fixed for all participants. The final evaluation will be performed after the completion of the competition using the remaining part of the test cases. Those results will also be published online. It is important to note that only teams that submit a report describing their approach before the end of the challenge will qualify for the final evaluation and will be considered for the invitation to submit a paper for the special session at the IEEE ICME 2023 conference.

Data files

Schedule

February 12, 2023: start of the competition, datasets become available,
March 19, 2023: deadline for submitting the solutions,
March 19, 2023: deadline for sending the reports, end of the competition,
March 23, 2023: online publication of the final results, sending invitations for submitting papers to the associated workshop/special session at IEEE ICME 2023,
April 17, 2023: deadline for submitting invited papers,
April 24, 2023: notification of paper acceptance after reviews by the Competition Organizing Committee,
May 1, 2023: camera-ready of accepted papers due.

Awards

Authors of the top-ranked solutions (based on the final evaluation scores) will be awarded prizes funded by the IEEE ICME 2023 conference:

1,000 AUD for the winning solution (+ the cost of one IEEE ICME 2023 registration if the author attends the conference)
550 AUD for the 2nd place solution (+ the cost of one IEEE ICME 2023 registration if the author attends the conference)
300 AUD for the 3rd place solution (+ the cost of one IEEE ICME 2023 registration if the author attends the conference)

Organizing Committee

Andrzej Janusz, QED Software
Rafał Tyl, Silver Bullet Labs
Dominik Ślęzak, eSensei Sp. z o.o.

Forum

This forum is for all users to discuss matters related to the competition. Good manners apply!

Discussion	Author	Replies	Last post
Challenge report to be referenced in the paper	M	0	by M Wednesday, April 05, 2023, 11:39:02
Humble request for the three final evaluation entries in the future competitions	M	2	by M Wednesday, March 22, 2023, 05:51:45
Time Zone of the end of the competition March 19, 2023	M	2	by M Friday, March 17, 2023, 21:11:46
Deadline extension	Dymitr	1	by Andrzej Friday, March 17, 2023, 09:04:26
Whats the private sharing policy?	Carlos	2	by Carlos Thursday, March 09, 2023, 14:46:30
Data files	Karol	1	by Andrzej Monday, February 27, 2023, 14:27:33