4 years, 5 months ago

AAIA'15 Data Mining Competition: Tagging Firefighter Activities at a Fire Scene

AAIA'15 Data Mining Competition: Tagging Firefighter Activities at a Fire Scene is a continuation of the last year's competition organized within the framework of International Symposium on Advances in Artificial Intelligence and Applications (AAIA'15, https://fedcsis.org/aaia). It is also an integral part of the 2nd Complex Events and Information Modelling workshop (CEIM'15 https://fedcsis.org/ceim) devoted to the fire protection engeneering. This time, the task is related to the problem of recognizing activities carried out by firefighters based on streams of information from body sensor networks. Prizes worth over 4,000 PLN will be awarded to the most successful teams. The contest is sponsored by Polish Information Processing Society (http://www.pti.org.pl/), with a support from University of Warsaw (http://www.mimuw.edu.pl/) and ICRA project (http://icra-project.org/).

Introduction

Firefighter wearing a smart jacket by Karol Krenski

A fire ground is considered to be one of the most challenging decision taking environment. In dynamically changing situations, such as those occurring at a fire scene, all decisions need to be taken in a very short time. Since wrong decisions might have severe consequences, a commander of the response team is forced to act under a huge psychological pressure. This fact, combined with incomplete or inaccurate information about the current situation, sometimes leads to committing serious mistakes [1].

There are several initiatives that investigate this complex problem. One of them is The National Near Miss program in the USA (www.nationalnearmiss.org/). It gathers and analyzes reports describing real-life dangerous situations, and tries to draw some conclusions regarding their causes. Based on several thousand of carefully analyzed reports, experts identified the “lack of situational awareness” as the main factor associated with major accidents among firefighters [2]. This observation is in accordance with the results of the previous edition of our data mining competition [3]. The situational awareness corresponds to the cautiousness of a commander and his understanding of the actual state of the environment. Conditions affecting the situational awareness can be broken down into three groups: a lack of information, a lack of knowledge and a lack of cognition [4]. In this context, it seems that an increase in the situational awareness of commanders would result in taking better decision and thus increasing the safety of firefighters. Studies on the causes for mortal accidents during the actions of firefighters were also conducted by the Department of Homeland Security of the United States [5]. One conclusion of their research is that over 43% of deaths at a fire scene was caused by the stress or overexertion. Therefore, another critical way of increasing the firefighter safety is by monitoring their kinematics and psychophysical condition during the course of fire & rescue actions. 

Our research team works on those problems within a frame of ICRA project (www.icra-project.org/). One of prototype tools developed as a result of this project is, so called, a "smart jacket". This device is a wearable set of body sensors that allows to automatically track a firefighter at a fire scene. It also enables real-time screening of firefighter’s vital functions and monitoring of ongoing activities at the scene. The later of those two tasks is the main scope of this year’s AAIA Data Mining Competition. We would like to ask participants to come up with efficient algorithms for labelling activities conducted by firefighters during their training exercises, based on provided data sets from our body sensor network. More details regarding the data and their acquisition process can be found in Task description section. Moreover, a good starting point for research on the problem of activity recognition based on body sensor data was described in [6]. We hope that your expertise and innovative ideas will become a valuable contribution in our effort to increase the safety of brave men and women serving in Polish State Fire Service.

Special session at CEIM'15: As in the previous year, a special session devoted to the competition will be held at 2nd Complex Events and Information Modelling workshop (CEIM'15 https://fedcsis.org/ceim) which is a part of 10th International Symposium on Advances in Artificial Intelligence and Applications (AAIA'15, https://fedcsis.org/aaia). We will invite authors of selected reports to extend them for publication in the conference proceedings (after reviews by Organizing Committee members) and presentation at the conference. The publications will be treated as regular papers (including indexation in the Thomson Reuters Web of Science, DBLP, Scopus and other portals). The invited teams will be chosen based on their final rank, innovativeness of their approach and quality of the submitted report.

Promotional video:

In case of any questions please post on the forum or write us an email: AAIA15Contest@mimuw.edu.pl

References:

  1. A. Krasuski: “A framework for Dynamic Analytical Risk Management at the emergency scene. From tribal to top down in the risk management maturity model”, FedCSIS 2014, pp. 323-330
  2. L. J. Grorud and D. Smith: “The National Fire Fighter Near-Miss Reporting. Annual Report 2008”, in An exclusive supplement to Fire & Rescue magazine. Elsevier Public Safety, 2008, pp. 1–24
  3. A. Janusz, A. Krasuski, S. Stawicki, M. Rosiak, D. Ślęzak, H. S. Nguyen: “Key Risk Factors for Polish State Fire Service: a Data Mining Competition at Knowledge Pit”, FedCSIS 2014, pp. 345-354
  4. A. Krasuski, A. Jankowski, A. Skowron, and D. Ślęzak: “From sensory data to decision making: A perspective on supporting a fire commander”, in Web Intelligence (WI) and Intelligent Agent Technologies (IAT), 2013 IEEE/WIC/ACM International Joint Conferences on, vol. 3. IEEE, 2013, pp. 229–236
  5. United States Fire Administration: “Annual report on firefighter fatalities in the United States”, http://apps.usfa.fema.gov/firefighter-fatalities/
  6. M. Meina, B. Celmer, K. Rykaczewski: “Towards Robust Framework for On-line Human Activity Reporting Using Accelerometer Readings”, AMT 2014, pp. 347-358
Terms & Conditions
 
 

Contest Participation Rules:

  • The competition is open for all interested researchers, specialists and students. Only members of the Contest Organizing Committee cannot participate.
  • Participants may submit solutions as teams made up of one or more persons.
  • Each team needs to designate a leader responsible for communication with the Organizers. A single person can be a leader of only one team.
  • One person may be incorporated in maximally 3 teams.
  • Each team needs to be composed of a different set of persons.
  • A winner of the competition is chosen on the basis of the final evaluation results. In a case of draws in the evaluation scores, time of the submission will be taken into account.
  • Each team is obliged to provide a short report describing their final solution. Reports must contain information such as the name of a team, names of all team members, the last preliminary evaluation score and a brief overview of the used approach. Their length should not exceed 2000 words and they should be submitted in the pdf format using our submission system by June 5, 2015. Only submissions made by teams that provided the reports will qualify for the final evaluation.
  • By enrolling to this competition you grant the organizers rights to process your submissions for the purpose of evaluation and post-competition research.

In case of questions related to the competition please contact us via email: AAIA15Contest@mimuw.edu.pl or through the competition forum.

Please logIn to the system!

Data format: We provide the data for this competition in a tabular format. The training data set, namely trainingData.csv, is a comma-separated values file. It contains 20,000 rows and 17,242 columns. Each row corresponds to a short time series (approximately 1.8 s long) of sensory readings. The first 42 columns represent aggregations of data from sensors monitoring firefighter’s vital functions. These measurements were obtained using Equivital Single Subject Kit (EQ-02-KIT-SU-4). The remaining columns are divided into 400 chunks that represent consecutive readings from sets of kinetic sensors attached to firefighter’s torso, hands, arms and legs (a total of seven sets). Each set is composed of an accelerometer (dynamic bandwith: +/- 16G) and a gyroscope (scale up to 2,000 deg/s). Therefore, a single chunk of columns consists of 43 numeric values, from which the first one is time from the beginning of the series and the following 42 values represent the readings from the accelerometers (measured in m/s^2) and gyroscopes (measured in deg/s, drift and temperature compensation was applied). More details regarding the description of data columns can be found in the column_info.txt file. An average time difference between consecutive sensory readings in the data is 4.5 ms. Labels for the training data are provided in a separate file, trainingLabels.csv. Each row in this file contains two labels for a corresponding row in the training data. The first label describes a posture of a firefighter and the second label describes his current main activity. Test data file, namely testData.csv, is in the same format as the training data set, however, the labels for the test series are hidden from participants. It is important to note that the training and test data sets consist of recordings which were obtained from different groups of firefighters.

Format of submissions: The participants of the competition are asked to predict labels of the time series from the available test set (testData.csv) and send us their solutions using the submission system. Each solution should be sent in a single text file containing exactly 20,000 lines. In the consecutive lines, this file should contain exactly two strings indicating label names for the posture and main activity of a firefighter, separated by a comma – the format of this file should be the same as the format of the trainingLabels.csv file.

Evaluation of results: The submitted solutions will be evaluated on-line and the preliminary results will be published on the competition leaderboard. The preliminary score will be computed on a random subset of the test set, fixed for all participants. It will correspond to approximately 10% of the test data. The final evaluation will be performed after completion of the competition using the remaining part of the test data. Those results will also be published on-line. It is important to note that only teams which submit a short report describing their approach before the end of the contest will qualify for the final evaluation. The winning teams will be officially announced during CEIM'15 workshop devoted to this competition (https://fedcsis.org/ceim) at the FedCSIS'15 conference.

The assessment of solutions will be done using the balanced accuracy measure which is defined as an average accuracy within all decision classes. It will be computed separately for the labels describing the posture and main activities of firefighters. The final score in the competition will be a weighted average of balanced accuracies computed for those two sets of labels. Namely, if for a vector of predictions preds and a vector of true labels labels we define the balance accuracy as: \[ACC_{i}(preds,labels) = \frac{|j : preds_{j} = labels_{j} = i|}{|j : labels_{j} = i|}\] \[BAC(preds,labels) = \left(\sum\limits_{i = 1}^l ACC_i(preds,labels)\right)/l\] and we denote by: $$ \begin{array}{ccl} BAC_{p} & - & \textrm{balanced accuracy for labels describing the posture}, \\ BAC_{a} & - & \textrm{ balanced accuracy for labels describing the main activity}, \end{array} $$ then the final score in the competition for a solution s will be computed as: \[score(s) = \left(BAC_{p}(s) + 2*BAC_{a}(s)\right)/3\]

Rank Team Name Score Submission Date
1
jan
0.839120 2015-06-1 23:08:02
2
zagorecki
0.829851 2015-05-8 12:29:10
3
nitekna
0.826101 2015-06-2 01:46:52
4
mathurin
0.804086 2015-06-1 07:39:37
5
lp319499
0.791378 2015-06-2 01:50:53
6
szefo617
0.789292 2015-06-2 00:36:00
7
rp335451
0.781516 2015-06-1 20:11:21
8
gszpak
0.776082 2015-05-31 01:06:18
9
wawrzyniaksz
0.772883 2015-05-31 20:03:12
10
dr319377
0.772385 2015-05-31 03:21:16
11
mg320637
0.771529 2015-05-31 16:19:21
12
bb319318
0.765402 2015-05-31 00:35:16
13
marcb
0.760082 2015-05-19 13:58:20
14
cl320813
0.754474 2015-05-31 23:05:07
15
maciekf
0.753849 2015-05-20 23:36:16
16
pa333836
0.728170 2015-05-31 14:11:24
17
pb305164
0.725677 2015-06-2 00:11:29
18
iran-amin
0.719647 2015-05-25 04:47:28
19
snm
0.717770 2015-05-22 16:24:01
20
ps319383
0.711079 2015-06-1 23:17:10
21
krisun17
0.698275 2015-05-31 15:48:21
22
lj320680
0.694065 2015-06-1 21:56:07
23
mb305053
0.693647 2015-06-1 18:09:05
24
pawols
0.684818 2015-05-31 21:02:26
25
ks321221
0.683289 2015-05-31 19:28:59
26
apedzich
0.683289 2015-06-1 19:54:42
27
ks335802
0.669289 2015-06-1 15:44:25
28
kc320722
0.664322 2015-05-29 20:26:23
29
jk334662
0.662903 2015-05-20 00:20:01
30
bartekm
0.660551 2015-03-14 16:43:33
31
mk334581
0.659863 2015-06-1 16:39:02
32
lp334978
0.656276 2015-05-31 20:19:00
33
masari
0.649739 2015-06-1 12:31:17
34
mp334965
0.644272 2015-06-1 10:58:46
35
dj306244
0.639310 2015-06-1 21:58:51
36
jk334678
0.622602 2015-05-31 00:40:58
37
ck337256
0.620957 2015-05-31 20:47:13
38
am334774
0.619994 2015-06-1 22:23:33
39
aj334557
0.618137 2015-05-23 21:35:14
40
baseline_solution
0.603620 2015-03-7 19:43:00
41
mk334582
0.598073 2015-05-31 22:53:52
42
ak334680_mimuw
0.593368 2015-05-31 23:25:51
43
ab320556
0.575392 2015-06-1 22:40:54
44
mg234048
0.573930 2015-06-1 20:52:17
45
jd334392
0.551167 2015-06-1 00:21:26
46
kp334969
0.517393 2015-06-1 18:10:01
47
ma333856
0.472210 2015-05-25 11:40:47
48
kk306256
0.471488 2015-06-1 23:55:12
49
katarzynki
0.437836 2015-06-1 18:49:21
50
tdziopa
0.433893 2015-05-31 16:22:06
51
ak306305
0.416782 2015-05-27 01:00:59
52
nd334438
0.358328 2015-05-31 21:10:27
53
jr337655
0.202479 2015-05-31 22:50:03
54
fzero
0.141081 2015-06-1 20:47:11
55
ks335487
0.140249 2015-06-1 17:49:18
56
jankmaka
0.000000 2015-06-1 23:33:23
57
mf
No report file found or report rejected. 2015-06-2 01:30:08
58
mf292605
No report file found or report rejected. 2015-05-29 14:21:03
59
korzenek
No report file found or report rejected. 2015-05-28 23:25:51
60
dymitrruta
No report file found or report rejected. 2015-03-15 14:20:47
61
rs335786
No report file found or report rejected. 2015-06-1 22:58:08
62
rozkminiacze
No report file found or report rejected. 2015-05-28 22:37:55
63
michalu
No report file found or report rejected. 2015-05-25 19:41:50
64
a
No report file found or report rejected. 2015-06-1 22:14:21
65
kkurach
No report file found or report rejected. 2015-05-12 01:24:37
66
mmarks
No report file found or report rejected. 2015-05-26 23:42:12
67
tw336034
No report file found or report rejected. 2015-06-1 07:40:04
68
michalg
No report file found or report rejected. 2015-05-30 21:20:56
69
datadeep
No report file found or report rejected. 2015-06-1 23:02:17
70
es335773
No report file found or report rejected. 2015-06-2 00:17:01
71
bfrackowiak
No report file found or report rejected. 2015-06-2 00:26:35
72
obus
No report file found or report rejected. 2015-04-18 21:19:05
73
rchrysler
No report file found or report rejected. 2015-05-17 21:50:28
74
lukaszp
No report file found or report rejected. 2015-06-2 01:36:49
75
piotr
No report file found or report rejected. 2015-04-1 12:19:45
76
ziom
No report file found or report rejected. 2015-04-8 02:36:32
77
pw294402
No report file found or report rejected. 2015-06-1 14:06:02
78
tomkoz1
No report file found or report rejected. 2015-05-28 01:37:48
79
pk334685
No report file found or report rejected. 2015-06-2 00:00:28
80
sylwia
No report file found or report rejected. 2015-03-26 21:08:27
81
customs
No report file found or report rejected. 2015-04-12 16:23:59
82
pt335957
No report file found or report rejected. 2015-06-2 01:54:51

Our data mining competition has finally finished. Thank you very much for your hard work! 

The Winners:

  1. Jan Lasek, Polish Academy of Sciences and Marek Gagolewski, Systems Research Institute, Polish Academy of Sciences and Warsaw University of Technology, Poland (team jan)
  2. Adam Zagórecki, Cranfield University, United Kingdom (team zagorecki)

  3. Eftim Zdravevski, Petre Lameski, Riste Mingov, Andrea Kulakov, University Cc. Cyril and Methodius, Macedonia and NI TEKNA Intelligent Technologies, and Dejan Gjorgjevikj, University Cc. Cyril and Methodius, Macedonia (team nitekna)

Congratulation on your excellent results!

In the next few days we will be inviting selected teams to extend their reports into conference papers submitted to AAIA'15 conference. Stay tuned!

  • March 9, 2015: start of the competition, data sets become available,
  • June 1, 2015: deadline for submitting the predictions,
  • June 5, 2015: deadline for sending the reports, end of the challenge,
  • June 8, 2015: on-line publication of final results, sending invitations for submitting short papers for the special session,
  • June 22, 2015: deadline for submissions of papers describing the selected solutions,
  • July 6, 2015: deadline for submissions of camera-ready papers selected for presentation at the CEIM'15 workshop.

Authors of the top ranked solutions (based on the final evaluation scores) will be awarded with prizes:

  • First Prize: 3000 PLN + one free FedCSIS'15 conference registration,
  • Second Prize: 1000 PLN + one free FedCSIS'15 conference registration,
  • Third Prize: one free FedCSIS'15 conference registration.

The award ceremony will take place during the FedCSIS'15 conference (September 13 - 16, Łódź). Traditionally, invited authors who decide to attend the conference will receive a diploma and a competition T-shirt.

Andrzej Janusz, University of Warsaw

Michał Meina, University of Warsaw

Adam Krasuski, Main School of Fire Service

Krzysztof Rykaczewski, University of Warsaw

Bartosz Celmer, Main School of Fire Service

Dominik Ślęzak, University of Warsaw & Infobright Inc.

  Discussion Author Replies Last post
test labels Bartosz 0 by Bartosz
Thursday, October 01, 2015, 19:06:01
The last week of AAIA’15 Data Mining Competition Andrzej 0 by Andrzej
Monday, May 25, 2015, 14:21:24
medical data Konrad 1 by Krzysztof
Monday, May 18, 2015, 13:47:32
Re: Number of unique firefighters? Andrzej 2 by Karol
Tuesday, May 12, 2015, 01:30:50
New competition at Knowledge Pit: IJCRS’15 Data Challenge Andrzej 0 by Andrzej
Monday, April 13, 2015, 13:27:59
Re: Number of unique firefighters? Andrzej 0 by Andrzej
Monday, April 13, 2015, 09:14:40
A problem with KnowledgePit server Andrzej 0 by Andrzej
Sunday, April 12, 2015, 16:59:27
Re: Number of possible submissions Andrzej 0 by Andrzej
Friday, April 10, 2015, 11:26:35
Data set representativeness and skewed accelerations Dymitr 1 by Dymitr
Thursday, April 02, 2015, 08:53:15
Change in terms and conditions Andrzej 0 by Andrzej
Friday, March 20, 2015, 17:18:13
A bug in the evaluation procedure Andrzej 0 by Andrzej
Friday, March 20, 2015, 13:30:51
Number of unique firefighters? Dymitr 4 by Dymitr
Thursday, March 19, 2015, 05:37:54
PTI becomes the main sponsor of AAIA'15 DMC! Andrzej 0 by Andrzej
Sunday, March 01, 2015, 13:56:47