84648488
 Chunk
reverted.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
|
0x00 IO performance
performance:
1000(131832 B):
read-write: 13.414876s
2000(270908 B):
read-write: 25.270484s
5000(664204 B):
read-write: 64.528306s
About 20MB/s
0x01 IO & Computing time benchmark (with embed_rate = 0.2)
(1)file sum = 100
serial:
A.
(format)time elapsed: 0.583396s
(embed)time elapsed: 30.847788s
(extract)time elapsed: 57.884814s
real 89.99s
user 89.58s
sys 0.37s
CPU 99%
B.
(format)time elapsed: 0.595019s
(embed)time elapsed: 30.959033s
(extract)time elapsed: 57.931772s
real 90.16s
user 89.68s
sys 0.46s
CPU 99%
C.
(format)time elapsed: 0.690753s
(embed)time elapsed: 31.222347s
(extract)time elapsed: 57.818952s
real 90.43s
user 89.80s
sys 0.47s
CPU 99%
spark:
A.
(format table)time elapsed: 6.776471s
(read table)time elapsed: 0.060646s
(process)time elapsed: 39.127508 s
real 59.50s
user 11.81s
sys 1.09s
CPU 21%
B.
(format table)time elapsed: 6.470915s
(read table)time elapsed: 1.542012s
(process)time elapsed: 35.019916 s
real 58.32s
user 11.82s
sys 1.11s
CPU 22%
C.
(format table)time elapsed: 5.734575s
(read table)time elapsed: 1.485326s
(process)time elapsed: 35.217565 s
real 57.13s
user 12.15s
sys 1.06s
CPU 23%
(2)file sum = 200
serial:
A.
(format)time elapsed: 1.147411s
(embed)time elapsed: 62.815709s
(extract)time elapsed: 118.217859s
real 182.86s
user 181.89s
sys 0.88s
CPU 99%
B.
(format)time elapsed: 1.201338s
(embed)time elapsed: 62.710831s
(extract)time elapsed: 117.897544s
real 182.49s
user 181.71s
sys 0.72s
CPU 99%
C.
(format)time elapsed: 1.131681s
(embed)time elapsed: 62.880462s
(extract)time elapsed: 118.124955s
real 182.82s
user 181.95s
sys 0.77s
CPU 99%
spark:
A.
(format table)time elapsed: 10.535134s
(read table)time elapsed: 2.932193s
(process)time elapsed: 47.566111 s
real 84.84s
user 13.98s
sys 1.30s
CPU 18%
B.
(format table)time elapsed: 29.667743s
(read table)time elapsed: 0.136938s
(process)time elapsed: 46.093956 s
real 135.58s
user 14.52s
sys 1.62s
CPU 11%
C.
(format table)time elapsed: 13.773469s
(read table)time elapsed: 3.047374s
(process)time elapsed: 45.717876 s
real 90.10s
user 13.60s
sys 1.43s
CPU 16%
(3)file sum = 500
serial:
A.
(format)time elapsed: 2.763806s
(embed)time elapsed: 162.806317s
(extract)time elapsed: 299.778606s
real 466.03s
user 462.83s
sys 2.85s
CPU 99%
B.
(format)time elapsed: 2.773738s
(embed)time elapsed: 160.579258s
(extract)time elapsed: 296.961467s
real 461.01s
user 458.86s
sys 1.80s
CPU 99%
C.
(format)time elapsed: 2.991173s
(embed)time elapsed: 161.909154s
(extract)time elapsed: 296.754535s
real 462.34s
user 460.02s
sys 2.00s
CPU 99%
spark:
A.
(format table)time elapsed: 18.399418s
(read table)time elapsed: 0.298133s
(process)time elapsed: 84.794703 s
real 143.67s
user 18.91s
sys 1.93s
CPU 14%
B.
(format table)time elapsed: 31.979529s
(read table)time elapsed: 7.316972s
(process)time elapsed: 82.664176 s
real 196.24s
user 19.40s
sys 2.49s
CPU 11%
C.
(format table)time elapsed: 27.639130s
(read table)time elapsed: 0.228689s
(process)time elapsed: 85.939402 s
real 187.12s
user 19.50s
sys 2.71s
CPU 11%
(4)file sum = 1000
serial:
A.
(format)time elapsed: 6.372794s
(embed)time elapsed: 329.023151s
(extract)time elapsed: 600.438977s
real 936.53s
user 931.93s
sys 3.11s
CPU 99%
B.
(format)time elapsed: 5.329308s
(embed)time elapsed: 327.279783s
(extract)time elapsed: 596.171131s
real 929.46s
user 925.83s
sys 3.09s
CPU 99%
C.
(format)time elapsed: 5.180158s
(embed)time elapsed: 318.844047s
(extract)time elapsed: 585.748167s
real 910.47s
user 907.03s
sys 2.98s
CPU 99%
spark:
A.
(format table)time elapsed: 70.172458s
(read table)time elapsed: 17.022568s
(process)time elapsed: 159.624198 s
real 431.68s
user 31.02s
sys 4.54s
CPU 8%
B.
(format table)time elapsed: 87.720660s
(read table)time elapsed: 15.644418s
(process)time elapsed: 176.987813 s
real 428.69s
user 29.12s
sys 3.91s
CPU 7%
C
(format table)time elapsed: 120.458491s
(read table)time elapsed: 0.586775s
(process)time elapsed: 163.598833 s
real 486.40s
user 28.65s
sys 5.33s
CPU 6%
(5)file sum = 2000
serial:
A.
(format)time elapsed: 14.960961s
(embed)time elapsed: 679.357936s
(extract)time elapsed: 1256.341536s
real 1951.35s
user 1942.26s
sys 3.25s
CPU 99%
B.
(format)time elapsed: 34.474327s
(embed)time elapsed: 660.667673s
(extract)time elapsed: 1232.833102s
real 1928.65s
user 1900.41s
sys 3.30s
CPU 98%
C.
(format)time elapsed: 10.912531s
(embed)time elapsed: 661.449268s
(extract)time elapsed: 1233.958305s
real 1907.00s
user 1901.39s
sys 3.19s
CPU 99%
spark:
A.
(format table)time elapsed: 176.126157s
(read table)time elapsed: 1.319059s
(process)time elapsed: 343.191293 s
real 956.85s
user 65.04s
sys 11.55s
CPU 8%
B.
(format table)time elapsed: 262.840257s
(read table)time elapsed: 76.712754s
(process)time elapsed: 368.793729 s
real 931.69s
user 51.49s
sys 6.66s
CPU 6%
C.
(format table)time elapsed: 210.607617s
(read table)time elapsed: 1.118159s
(process)time elapsed: 354.279433 s
real 995.37s
user 56.69s
sys 11.84s
CPU 6%
(6)file sum = 5000
serial:
A.
(format)time elapsed: 39.880657s
(embed)time elapsed: 1652.537536s
(extract)time elapsed: 3067.980390s
real 4761.09s
user 4727.25s
sys 7.79s
CPU 99%
B.
(format)time elapsed: 99.882375s
(embed)time elapsed: 1600.979427s
(extract)time elapsed: 2970.367912s
real 4671.91s
user 4578.77s
sys 7.62s
CPU 98%
C.
(format)time elapsed: 30.885324s
(embed)time elapsed: 1612.269242s
(extract)time elapsed: 2973.932865s
real 4617.78s
user 4596.83s
sys 7.40s
CPU 99%
spark:
A.
(format table)time elapsed: 958.231221s
(read table)time elapsed: 88.844213s
(process)time elapsed: 852.649969 s
real 3162.58s
user 101.27s
sys 30.68s
CPU 4%
B.
(format table)time elapsed: 486.504693s
(read table)time elapsed: 3.625127s
(process)time elapsed: 811.479755 s
real 2361.70s
user 126.35s
sys 36.82s
CPU 6%
C.
(format table)time elapsed: 715.228832s
(read table)time elapsed: 42.838806s
(process)time elapsed: 821.420032 s
real 2282.66s
user 138.61s
sys 30.21s
CPU 7%
0x02 model performance benchmark (with file sum = 5000)
(1)embed_rate = 0.2
serial:
spark:
|