benchmarks hboa

75% de zeros
1200 population
0.015

1110000010010100000010100000011010000100000011000000000001010000000000000000100110000000000000000000
sorting used : naive bayes
0 0.333333
sorting used : naive bayes
0.333333 0.666667
sorting used : naive bayes
0.666667 1
truePositive : 3242
trueNegative : 3175
falsePositive : 1825
falseNegative : 1758

15/19
64.17%

full vector
DecisionTable
3folds
about 1mn

=== Run information ===

Scheme: weka.classifiers.rules.DecisionTable -X 1 -S "weka.attributeSelection.BestFirst -D 1 -N 5"
Relation: cpu
Instances: 10000
Attributes: 101
[list of attributes omitted]
Test mode: 3-fold cross-validation

=== Classifier model (full training set) ===

Decision Table:

Number of training instances: 10000
Number of Rules : 238
Non matches covered by Majority class.
Best first.
Start set: no attributes
Search direction: forward
Stale search after 5 node expansions
Total number of subsets evaluated: 961
Merit of best subset found: 69.76
Evaluation (for feature selection): CV (leave one out)
Feature set: 30,36,45,58,76,101

Time taken to build model: 27.75 seconds

=== Stratified cross-validation ===
=== Summary ===

Correctly Classified Instances 6872 68.72 %
Incorrectly Classified Instances 3128 31.28 %
Kappa statistic 0.3744
Mean absolute error 0.4019
Root mean squared error 0.4508
Relative absolute error 80.3745 %
Root relative squared error 90.1597 %
Total Number of Instances 10000

=== Detailed Accuracy By Class ===

TP Rate FP Rate Precision Recall F-Measure ROC Area Class
0.71 0.336 0.679 0.71 0.694 0.749 c1
0.664 0.29 0.696 0.664 0.68 0.749 c2

=== Confusion Matrix ===

a b <-- classified as
3551 1449 | a = c1
1679 3321 | b = c2

68.72%















ideal vector
DecisionTable
3folds
about 15s

=== Run information ===

Scheme: weka.classifiers.rules.DecisionTable -X 1 -S "weka.attributeSelection.BestFirst -D 1 -N 5"
Relation: cpu
Instances: 10000
Attributes: 26
0
1
2
8
13
20
27
29
30
32
35
37
38
40
42
44
46
50
55
57
59
75
79
80
88
play
Test mode: 3-fold cross-validation

=== Classifier model (full training set) ===

Decision Table:

Number of training instances: 10000
Number of Rules : 238
Non matches covered by Majority class.
Best first.
Start set: no attributes
Search direction: forward
Stale search after 5 node expansions
Total number of subsets evaluated: 209
Merit of best subset found: 69.76
Evaluation (for feature selection): CV (leave one out)
Feature set: 8,11,16,20,22,26

Time taken to build model: 5.59 seconds

=== Stratified cross-validation ===
=== Summary ===

Correctly Classified Instances 6872 68.72 %
Incorrectly Classified Instances 3128 31.28 %
Kappa statistic 0.3744
Mean absolute error 0.4019
Root mean squared error 0.4508
Relative absolute error 80.3745 %
Root relative squared error 90.1597 %
Total Number of Instances 10000

=== Detailed Accuracy By Class ===

TP Rate FP Rate Precision Recall F-Measure ROC Area Class
0.71 0.336 0.679 0.71 0.694 0.749 c1
0.664 0.29 0.696 0.664 0.68 0.749 c2

=== Confusion Matrix ===

a b <-- classified as
3551 1449 | a = c1
1679 3321 | b = c2

68.72%


















18/23 vector
DecisionTable
3folds
about 15s

=== Run information ===

Scheme: weka.classifiers.rules.DecisionTable -X 1 -S "weka.attributeSelection.BestFirst -D 1 -N 5"
Relation: cpu
Instances: 10000
Attributes: 24
0
1
2
8
11
13
16
20
29
37
42
44
46
55
57
59
75
76
79
80
88
96
97
play
Test mode: 3-fold cross-validation

=== Classifier model (full training set) ===

Decision Table:

Number of training instances: 10000
Number of Rules : 353
Non matches covered by Majority class.
Best first.
Start set: no attributes
Search direction: forward
Stale search after 5 node expansions
Total number of subsets evaluated: 190
Merit of best subset found: 69.67
Evaluation (for feature selection): CV (leave one out)
Feature set: 2,9,12,15,17,24

Time taken to build model: 5.11 seconds

=== Stratified cross-validation ===
=== Summary ===

Correctly Classified Instances 6766 67.66 %
Incorrectly Classified Instances 3234 32.34 %
Kappa statistic 0.3532
Mean absolute error 0.4063
Root mean squared error 0.4538
Relative absolute error 81.2526 %
Root relative squared error 90.7529 %
Total Number of Instances 10000

=== Detailed Accuracy By Class ===

TP Rate FP Rate Precision Recall F-Measure ROC Area Class
0.689 0.336 0.672 0.689 0.681 0.742 c1
0.664 0.311 0.681 0.664 0.673 0.742 c2

=== Confusion Matrix ===

a b <-- classified as
3445 1555 | a = c1
1679 3321 | b = c2

67.66%

the goal of feature selection is 3fold

-improve prediction performance
- provide more cost-effective prediction
- better understanding of the underlying process that generated the data

a feature selection method generates candidates from the feature space and asses them based on some evaluation criterion to find the best feature subset