0x02 Performance (1)file sum = 100 serial: (format)time elapsed: 0.583396s (embed)time elapsed: 30.847788s (extract)time elapsed: 57.884814s spark happybase: (read table)time elapsed: 1.471087s (process)time elapsed: 29.364628s (write table)time elapsed: 9.114235s spark happybase compressed: (read table)time elapsed: 1.449816s (process)time elapsed: 27.079774s (write table)time elapsed: 8.836868s spark distributed: (process)time elapsed: 49.995647s spark distributed compressed: (process)time elapsed: 55.280739s (2)file sum = 200 serial: (format)time elapsed: 1.147411s (embed)time elapsed: 62.815709s (extract)time elapsed: 118.217859s spark happybase: (read table)time elapsed: 3.008692s (process)time elapsed: 37.920278s (write table)time elapsed: 19.589578s spark happybase compressed: (read table)time elapsed: 2.948173s (process)time elapsed: 38.491461s (write table)time elapsed: 19.783102s spark distributed: (process)time elapsed: 76.258928s spark distributed compressed: (process)time elapsed: 83.836657s (3)file sum = 500 serial: (format)time elapsed: 2.763806s (embed)time elapsed: 162.806317s (extract)time elapsed: 299.778606s spark happybase: (read table)time elapsed: 6.817050s (process)time elapsed: 88.291989s (write table)time elapsed: 73.446282s spark happybase compressed: (read table)time elapsed: 6.949767s (process)time elapsed: 88.241193s (write table)time elapsed: 57.661072s spark distributed: (process)time elapsed: 177.039349s spark distributed compressed: (process)time elapsed: 143.524813s (4)file sum = 1000 serial: (format)time elapsed: 6.372794s (embed)time elapsed: 329.023151s (extract)time elapsed: 600.438977s spark happybase: (read table)time elapsed: 15.644418s (process)time elapsed: 159.951099s (write table)time elapsed: 186.335413s spark happybase compressed: (read table)time elapsed: 10.591560s (process)time elapsed: 158.694248s (write table)time elapsed: 161.767895s spark distributed: (process)time elapsed: 467.126756s spark distributed compressed: (process)time elapsed: 315.578952s (5)file sum = 2000 serial: (format)time elapsed: 14.960961s (embed)time elapsed: 679.357936s (extract)time elapsed: 1256.341536s spark happybase: (read table)time elapsed: 31.699596s (process)time elapsed: 313.154748s (write table)time elapsed: 387.063702s spark happybase compressed: (read table)time elapsed: 30.227347s (process)time elapsed: 313.085707s (write table)time elapsed: 360.762862s spark distributed: (process)time elapsed: 802.138669s spark distributed compressed: (process)time elapsed: 734.133909s (6)file sum = 5000 serial: (format)time elapsed: 39.880657s (embed)time elapsed: 1652.537536s (extract)time elapsed: 3067.980390s spark happybase: (read table)time elapsed: 73.070203s (process)time elapsed: 694.454719s (write table)time elapsed: 898.458633s spark happybase compressed: (read table)time elapsed: 65.254218s (process)time elapsed: 717.603061 s (write table)time elapsed: 833.214014s spark distributed: (process)time elapsed: 1860.610954s spark distributed compressed: (process)time elapsed: 1677.044038s 0x03 training performance embed rate = 0.2, F5(k=1) test_size : 0.4 (1)data size = 2000 sklearn: (train)time elapsed: 0.596740s (test)time elapsed: 0.240658s accuracy: (0.053, 0.9714434060228453, 0.95899133978604179) libsvm: (train)time elapsed: 2.936864s (test)time elapsed: 0.497595s accuracy: (95.89913397860418, 0.04100866021395823, 0.8432814964024421) opencv: (train)time elapsed: 4.166550s (test)time elapsed: 0.114133s (2)data size = 4000 sklearn: (train)time elapsed: 1.929259s (test)time elapsed: 0.953919s libsvm: (train)time elapsed: 12.814600s (test)time elapsed: 1.902628s opencv: (train)time elapsed: 6.986697s (test)time elapsed: 0.169895s (3)data size = 10000 sklearn: (train)time elapsed: 11.268877s (test)time elapsed: 5.840925s libsvm: (train)time elapsed: 85.663243s (test)time elapsed: 4.935881s opencv: (train)time elapsed: 10.817804s (test)time elapsed: 0.415325s spark MLlib 100 iters): (train)time elapsed: 102.71s (test)time elapsed: 1.398681s 0x04 detecting performance online: 110p/min (1)data size = 1000 offline: (read&detect&write)time elapsed: 176.679302s (2)data size = 2000 offline: (read&detect&write)time elapsed: 333.238773s (3)data size = 5000 offline: (read&detect&write)time elapsed: 749.348792s 0x03 MPB accuracy sklearn (svm.SVC(C=4, kernel='linear', shrinking=False)) (FP,TP,AR) (1)data size = 2000 F5 1 0.2 (0.07506053268765134, 0.9662618083670715, 0.94447989789406506) F5 1 0.1 (0.168141592920354, 0.7255154639175257, 0.77919591576260372) F5 1 0.05 (0.28846153846153844, 0.3760217983651226, 0.55427841634738184) F5 2 0.2 (0.06892382103990327, 0.9702702702702702, 0.94958519463943847) F5 2 0.1 (0.10709677419354839, 0.8042929292929293, 0.8481174218251436) F5 2 0.05 (0.12626262626262627, 0.3152454780361757, 0.5977011494252874) (2)data size = 4000 F5 1 0.2 (0.07142857142857142, 0.9820971867007673, 0.95530012771392081) F5 1 0.1 (0.16876574307304787, 0.7927461139896373, 0.8122605363984674) F5 1 0.05 (0.2661782661782662, 0.4625668449197861, 0.60433950223356736) F5 2 0.2 (0.06234413965087282, 0.9790575916230366, 0.95785440613026818) F5 2 0.1 (0.09535452322738386, 0.8048128342245989, 0.85696040868454659) F5 2 0.05 (0.1964512040557668, 0.42287917737789205, 0.61455009572431396) (3)data size = 10000 F5 1 0.2 (0.05196770938446014, 0.970679012345679, 0.95924605196128376) F5 1 0.1 (0.13636363636363635, 0.8643451143451143, 0.86398369842078448) F5 1 0.05 (0.2801400700350175, 0.5962636222106902, 0.65919510952623539) F5 2 0.2 (0.03944083874188717, 0.9776391055642226, 0.96892511462047881) F5 2 0.1 (0.09855649576903933, 0.9181011997913406, 0.9095771777890983) F5 2 0.05 (0.23769269020387868, 0.7289817232375979, 0.7460519612837494) 0x03 Gets accuracy params = { dataset size:10000 crop:200*200 test_interval: 200 base_lr: 0.001 momentum: 0.9 weight_decay: 0.0005 gamma: 0.0001 power: 0.75 batch_size: 200 scale: 1.0 feature map:10,20 conv filter:3-1,8-4 pool:2*2,2*2 activation function:ReLU solver_mode: GPU } (1)embed rate = 0.05 F5(1) J(1) iters:10000 0.6811 1.42341 F5(1) J(2) iters:10000 0.7096 1.58573 F5(2) J(1) iters:10000 0.7408 0.867403 F5(2) J(2) iters:10000 0.6077 2.1331 (2)embed rate = 0.1 F5(1) J(1) iters:10000 0.8172 0.827023 F5(1) J(2) iters:10000 0.8396 0.65971 F5(2) J(1) iters:10000 0.9166 0.234032 F5(2) J(2) iters:10000 0.7919 0.99669 (3)embed rate = 0.2 F5(1) J(1) iters:10000 0.9426 0.164854 F5(1) J(2) iters:10000 0.9591 0.103902 F5(2) J(1) iters:10000 0.9852 0.0592344 F5(2) J(2) iters:10000 0.9629 0.106968 0x04 Environment A: CPU : Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz Memory : 16GB OS : CentOS release 6.6 (Final) Kernel : Linux 2.6.32-504.el6.x86_64 B: CPU : Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz Memory : 16GB OS : Scientific Linux release 6.6 (Carbon) Kernel : Linux 2.6.32-504.1.3.el6.x86_64 C: CPU : Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz Memory : 10GB OS : CentOS release 6.6 (Final) Kernel : Linux 2.6.32-71.el6.x86_64