10 months, 1 week ago

IEEE BigData 2019 Cup: Suspicious Network Event Recognition

Suspicious Network Event Recognition is a data mining challenge organized in association with IEEE BigData 2019 conference. The task is to decide which alerts should be regarded as suspicious based on information extracted from network traffic logs. The competition is kindly sponsored by Security On-Demand (https://www.securityondemand.com/) and QED Software (http://qed.pl/).

Cyber threat detection and analytics play a pivotal role in providing security to organizations that provide web services, and to their users. Importance of this field is continuously growing due to the increasing abundance of Internet services, wireless networks, smart devices, etc.   Since the cybersecurity domain is hugely complex, it is also one of the major challenges of the contemporary world.

In this challenge, the task is to detect truly suspicious events and false alarms within the set of so-called network traffic alerts, that the Security Operations Center (SOC) Team members @ SOD have to analyze on an everyday basis. An efficient classification model could help the SOC Team to optimize their operations significantly. It is worth adding that although the competition sponsor is entirely commercial, the knowledge and experience that can be gathered by the competition participants may be highly beneficial to improve the intelligent cybersecurity modules in many organizations.

More details regarding the task and a description of the challenge data can be found in the Task description section.

Special track at IEEE BigData 2019: A special session devoted to the challenge will be held at the IEEE BigData 2019 conference. We will invite authors of selected challenge reports to extend them for publication in the conference proceedings (after reviews by Organizing Committee members) and presentation at the conference. The publications will be indexed in the same way as regular conference papers. The invited teams will be chosen based on their final rank, innovativeness of their approach and quality of the submitted report. 

References:

  • A. Janusz, D. Kałuza, A. Chadzynska-Krasowska, B. Konarsk, J. Holland, D. Slezak: IEEE BigData 2019 Cup: Suspicious Network Event Recognition. BigData 2019.
  • D. Ślęzak, A. Chądzyńska-Krasowska, J. Holland, P. Synak, R. Glick, M. Perkowski: Scalable Cyber-security Analytics with a New Summary-based Approximate Query Engine. IEEE BigData 2017: 1840-1849
Terms & Conditions
 
 

IEEE BigData 2019 Cup: Suspicious Network Event Recognition has finished. We are happy to announce that the competition attracted a total of 249 teams which, in total, submitted over 2400 solutions. Thanks!

The official Winners:

  1.     Team hieuvq:

    Quang Hieu Vu, ZALORA
    Dymitr Ruta, EBTIC, Khalifa University
    Ling Cen, EBTIC, Khalifa University

  1.     Team test_123:

    Cong Dong, Institute of Information Engineering, Chinese Academy of Sciences
    Yu Fan Chen, College of Management and Economics, Tianjin University
    Dong Xu Han, Institute of Information Engineering, Chinese Academy of Sciences

    the team would also like to honorably mention their mentors:

    Song LiuBo Jiang, and Jun Rong Liu from Institute of Information Engineering, Chinese Academy of Sciences

  1.     Team HSOC

    Tian Wang, Institute of Information Engineering, Chinese Academy of Sciences
     Chen Zhang, Institute of Information Engineering, Chinese Academy of Sciences
     Zhigang Lu, Institute of Information Engineering, Chinese Academy of Sciences

We would like to thank all participants for their effort and invaluable contribution. In particular, we want to express our gratitude to all participants who decided to send us descriptions of their solutions in the form of brief reports. We have sent invitations to selected teams to extend their reports and submit conference papers for a special session at IEEE Big Data Conference. 

 

Rank Team Name Is Report Preliminary Score Final Score Submissions
1
hieuvq
True 0.9514 0.931743 299
2
test_123
True 0.9453 0.930295 175
3
HSOC
True 0.9429 0.926885 45
4
FightForPALI
True 0.9521 0.918421 131
5
bingogogogogo
True 0.9492 0.917840 25
6
DeepIf
True 0.9309 0.914965 84
7
VerifiedXiaoPAI
True 0.9431 0.910551 101
8
Pit
True 0.9304 0.910134 20
9
security
True 0.9403 0.910003 243
10
NSEC SJTU
True 0.9396 0.906779 244
11
extended baseline
True 0.9224 0.906375 1
12
AGW
True 0.9305 0.905450 36
13
maper1
True 0.9184 0.903533 30
14
Pisa
True 0.9257 0.902961 72
15
UMF Canada
True 0.9253 0.896850 25
16
IF
True 0.9119 0.889315 92
17
Chain
True 0.8943 0.888660 3
18
sink
True 0.9180 0.874392 8
19
baseline solution
True 0.8899 0.870464 5
20
CREDIT
True 0.8581 0.854441 44
21
TTUiDVL
True 0.8841 0.839764 35
22
M
True 0.8769 0.839644 13
23
hello_there
True 0.8557 0.838661 4
24
ISO_Project
True 0.8753 0.836666 10
25
InTensorty
True 0.8612 0.781533 122
26
melanzana
True 0.6311 0.668400 1
27
Lzxc
True 0.5327 0.535459 1
28
LiuZ
True 0.5419 0.530611 5
29
jinmeizhi
False 0.9430 No report file found or report rejected. 91
30
wt
False 0.9426 No report file found or report rejected. 5
31
tc
False 0.9410 No report file found or report rejected. 9
32
6e
False 0.9391 No report file found or report rejected. 5
33
perfect
False 0.9287 No report file found or report rejected. 35
34
dinesh
False 0.9238 No report file found or report rejected. 18
35
iddqd
False 0.9151 No report file found or report rejected. 19
36
r2
False 0.9126 No report file found or report rejected. 10
37
test
False 0.9106 No report file found or report rejected. 8
38
GoGoPowerRangers
False 0.9095 No report file found or report rejected. 12
39
piero
False 0.9082 No report file found or report rejected. 5
40
xyz
False 0.9066 No report file found or report rejected. 4
41
DennisShaw
False 0.9048 No report file found or report rejected. 4
42
Sinister Three
False 0.9045 No report file found or report rejected. 6
43
12213
False 0.8982 No report file found or report rejected. 18
44
shire
False 0.8978 No report file found or report rejected. 8
45
hopium
False 0.8966 No report file found or report rejected. 17
46
323_touch_fish
False 0.8934 No report file found or report rejected. 1
47
maa
False 0.8926 No report file found or report rejected. 9
48
michalm
False 0.8901 No report file found or report rejected. 17
49
Marshalls
False 0.8861 No report file found or report rejected. 13
50
Dan
False 0.8852 No report file found or report rejected. 4
51
MAGIX.AI
False 0.8836 No report file found or report rejected. 11
52
marcb
False 0.8760 No report file found or report rejected. 2
53
nightking
False 0.8753 No report file found or report rejected. 20
54
LIDIA_UDC
False 0.8632 No report file found or report rejected. 2
55
Rookie
False 0.8598 No report file found or report rejected. 14
56
Josh
False 0.8581 No report file found or report rejected. 27
57
agui
False 0.8350 No report file found or report rejected. 7
58
zhaohui
False 0.8345 No report file found or report rejected. 10
59
阿贵去哪了
False 0.8293 No report file found or report rejected. 28
60
roger
False 0.8246 No report file found or report rejected. 5
61
safak
False 0.8143 No report file found or report rejected. 4
62
Surrey
False 0.8095 No report file found or report rejected. 16
63
WASP
False 0.7990 No report file found or report rejected. 10
64
j
False 0.7925 No report file found or report rejected. 16
65
Mars
False 0.7774 No report file found or report rejected. 15
66
Sof
False 0.7679 No report file found or report rejected. 4
67
The artic lab
False 0.7216 No report file found or report rejected. 10
68
Toy
False 0.6388 No report file found or report rejected. 3
69
钢棍谢师傅
False 0.5542 No report file found or report rejected. 8
70
lol
False 0.5356 No report file found or report rejected. 1
71
Radosne Kurki
False 0.5356 No report file found or report rejected. 1
72
---
False 0.5356 No report file found or report rejected. 1
73
FuzzyMelon
False 0.5356 No report file found or report rejected. 3
74
Test Team
False 0.5356 No report file found or report rejected. 2
75
Kefi
False 0.5356 No report file found or report rejected. 1
76
WinTeam
False 0.5356 No report file found or report rejected. 3
77
LiuShuang-team
False 0.5356 No report file found or report rejected. 6
78
CBRL-TOPCODERS
False 0.5281 No report file found or report rejected. 1
79
mm_nitro
False 0.5200 No report file found or report rejected. 5
80
Cloud
False 0.5175 No report file found or report rejected. 1
81
J_theboss
False 0.5025 No report file found or report rejected. 1
82
panand
False 0.5000 No report file found or report rejected. 1
Please logIn to the system!

The data set available in the challenge consist of alerts investigated by a SOC team at SoD (called ‘investigated alerts’). Each record is described by various statistics selected based on experts’ knowledge, and a hierarchy of associated IP addresses (anonymized), called assets. For each alert in the ‘investigated alerts’ data tables, there is a history of related log events (a detailed set of network operations acquired by SoD, anonymized to ensure the safety of SoD clients).

In total, training and test data sets cover a period between October 1, 2018, and March 31, 2019. A description of columns from the ‘investigated alerts’ data is provided in a separate file column_descriptions.txt. The main data was divided into a training set and a test set based on alert timestamps. Approximately four months are used as the training set (the file cybersecurity_training.csv) and the remaining part is used as a test set (the file cybersecurity_test.csv). The format of those two files is the same - columns are separated by the '|' sign, however, the target column called 'notified' is missing in the test data.

There will also be a second data set available to participants of the challenge. Due to its large size, it will be hosted on an external platform. We will provide access to this data on request, to participants who exceed the baseline score on the public leaderboard. This data contains information about individual event logs associated with each of the alerts from the main data (both training and test parts). A more detailed description of this set will be provided at a later stage of the competition.

The task and the format of submissions: the task for participants of this challenge is to predict which of the investigated alerts were considered truly suspicious by the SOC team and led to issuing a notification to SoD’s clients. In the training data, this information is indicated by the column 'notified'. A submission should have a form of scores assigned to every record from the test data - each score in a separate line of a text file. An example of a correctly formatted submission file is provided in the Data files section.

Evaluation: the quality of submissions will be evaluated using the AUC measure. Solutions will be evaluated online and the preliminary results will be published on the public leaderboard. The preliminary score will be computed on a small subset of the test records, fixed for all participants. The final evaluation will be performed after completion of the competition using the remaining part of the test records. Those results will also be published online. It is important to note that only teams which submit a report describing their approach before the end of the challenge will qualify for the final evaluation. 

After the end of this challenge, all data sets remain available on request. Any registered team can get access to the data after receiving written consent from the organizers

If you are using the competition data in your publications, please consider adding a reference to our BigData 2019 paper:
Andrzej Janusz, Daniel Kałuza, Agnieszka Chadzynska-Krasowska, Bartek Konarsk, Joel Holland, Dominik Slezak: IEEE BigData 2019 Cup: Suspicious Network Event Recognition. BigData 2019.

In order to download competition files you need to be enrolled.
  • May 27, 2019: web site of the challenge opens, the task is revealed,
  • June 15, 2019 July 1, 2019: start of the competition, data become available,
  • September 29, 2019 (23:59 GMT): deadline for submitting the solutions,
  • October 2, 2019 (23:59 GMT): deadline for sending the reports, end of the competition,
  • October 7, 2019: online publication of the final results, sending invitations for submitting papers for the special track at the IEEE BigData 2019 conference,
  • October 28, 2019: deadline for submiting invited papers,
  • November 4, 2019: notification of paper acceptance,
  • November 15, 2019: camera-ready of accepted papers due,
  • December 9-12, 2019: the IEEE BigData 2019 conference (special track date TBA).

Authors of the top-ranked solutions (based on the final evaluation scores) will be awarded prizes funded by our sponsors:

  • First Prize: 1500 USD + one free IEEE BigData 2019 conference registration,
  • Second Prize: 1000 USD + one free IEEE BigData 2019 conference registration,
  • Third Prize: 500 USD + one free IEEE BigData 2019 conference registration.

The award ceremony will take place during the special track at IEEE BigData 2019 conference.

  • Dominik Ślęzak, QED Software & Security On-Demand & University of Warsaw
  • Agnieszka Chądzyńska-Krasowska, Security On-Demand & Polish-Japanese Academy of Information Technology
  • Joel Holland, Security On-Demand
  • Andrzej Janusz, QED Software & University of Warsaw
  • Daniel Kałuża, QED Software
  • Bartek Konarski, Security On-Demand
  • Agnieszka Sochal, QED Software

In case of any questions please post on the competition forum or write an email at contact {at} knowledgepit.ml 

             

This forum is for all users to discuss matters related to the competition. Good manners apply!
  Discussion Author Replies Last post
post-competition research Andrzej 0 by Andrzej
Thursday, January 02, 2020, 18:20:05
Data File Download Speed Fernando 0 by Fernando
Tuesday, October 01, 2019, 01:20:04
Can not change the final submissions 5 by cong
Monday, September 30, 2019, 11:41:15
issues related to team merging Andrzej 0 by Andrzej
Thursday, September 26, 2019, 12:16:46
the end of competition approaches Andrzej 4 by Andrzej
Thursday, September 26, 2019, 00:35:15
Page not found J 1 by Andrzej
Monday, September 23, 2019, 13:06:28
Baseline submission code release? Dan 1 by Andrzej
Tuesday, September 10, 2019, 16:29:12
maintenance of the evaluation system Andrzej 0 by Andrzej
Wednesday, August 28, 2019, 11:31:50
the first of additional data sets released Andrzej 2 by Daniel
Monday, August 19, 2019, 12:00:09
alert_time Tian 1 by Daniel
Monday, August 19, 2019, 11:31:36
Training set order Scott 5 by Scott
Wednesday, August 14, 2019, 12:42:43
Role of localized_alerts_data and submission score A 3 by Andrzej
Tuesday, August 06, 2019, 19:47:06
Additional data - event logs Andrzej 1 by Daniel
Friday, August 02, 2019, 18:45:25
why I keep getting this error jayesh 1 by jayesh
Monday, July 29, 2019, 23:22:23
The data sets were released! Andrzej 2 by Andrzej
Thursday, July 04, 2019, 00:32:06
The submission system is online! Andrzej 0 by Andrzej
Tuesday, July 02, 2019, 18:18:22
the submission system opens soon Andrzej 0 by Andrzej
Tuesday, July 02, 2019, 01:59:22
a delay in disclosure of the competition data Andrzej 2 by Andrzej
Monday, July 01, 2019, 21:42:55
No option to add team memeber after creating the group AMIT 1 by Andrzej
Thursday, May 30, 2019, 14:42:57