ElasticSearch-Hadoop
ElasticSearch-Hadoop Connector를 이용하여 ElasticSearch의 실시간 검색 및 분석 기능과 Hadoop의 강력한 데이터 저장 및 처리 기능을 동시에 활용.
기본적으로 ElasticSearch는 Join SQL이 수행되지 않는데 ElasticSearch-Hadoop Connector를 이용하면 DF 를 생성해서 multi index 에 대한 Join SQL을 수행할 수 있을 것 같다. 테스트 해본 데이터 사이즈가 커서 그런지 쿼리 응답을 받지는 못함..
설치
wget https://artifacts.elastic.co/downloads/elasticsearch-hadoop/elasticsearch-hadoop-6.4.2.zip
unzip *.zip
또는
Download ES-Hadoop 페이지에서 *.jar libraries 다운로드
spark-shell 실행 명령
spark-shell --master yarn --jars /app/elastic/elasticsearch-hadoop/6.4.2/dist/elasticsearch-spark-20_2.11-6.4.2.jar
간단한 조회
import org.apache.spark.sql.SQLContext
import org.elasticsearch.spark.sql._
import org.apache.spark.SparkConf
// 기존 SparkContext 종료
sc.stop()
val conf = new SparkConf().setAppName("es-hadoop-app01")
conf.set("spark.driver.allowMultipleContexts", "true")
conf.set("es.index.auto.create", "true")
conf.set("es.nodes.discovery", "true")
conf.set("es.nodes", "192.168.179.147:9200")
// SparkContext & SQLConext 새로 만들기
val sc = new org.apache.spark.SparkContext(conf)
val sqlContext = new SQLContext(sc)
// dataframe 생성
val df = sqlContext.read.format("org.elasticsearch.spark.sql").load("lineitem/doc")
df.registerTempTable("tab")
sqlContext.sql("SELECT COUNT(*) FROM tab").show()
sqlContext.sql("SELECT r_comment FROM tab").show()
Join SQL 수행하기
import org.apache.spark.sql.SQLContext
import org.elasticsearch.spark.sql._
import org.apache.spark.SparkConf
// 기존 SparkContext 종료
sc.stop()
val conf = new SparkConf().setAppName("es-hadoop-app01")
conf.set("spark.driver.allowMultipleContexts", "true")
conf.set("es.index.auto.create", "true")
conf.set("es.nodes.discovery", "true")
conf.set("es.nodes", "192.168.179.147:9200")
// SparkContext & SQLConext 새로 만들기
val sc = new org.apache.spark.SparkContext(conf)
val sqlContext = new SQLContext(sc)
// dataframe 생성
val df = sqlContext.read.format("org.elasticsearch.spark.sql").load("customer/doc")
df.registerTempTable("customer")
val df = sqlContext.read.format("org.elasticsearch.spark.sql").load("orders/doc")
df.registerTempTable("orders")
val df = sqlContext.read.format("org.elasticsearch.spark.sql").load("lineitem/doc")
df.registerTempTable("lineitem")
sqlContext.sql(s"""
select
l_orderkey,
sum(l_extendedprice * (1 - l_discount)) as revenue,
o_orderdate,
o_shippriority
from
customer,
orders,
lineitem
where
c_mktsegment = 'BUILDING'
and c_custkey = o_custkey
and l_orderkey = o_orderkey
and o_orderdate < date '1995-03-15'
and l_shipdate > date '1995-03-15'
group by
l_orderkey,
o_orderdate,
o_shippriority
order by
revenue desc,
o_orderdate
""").show()
한세월 걸릴듯
scala> sqlContext.sql(s"""
| select
| l_orderkey,
| sum(l_extendedprice * (1 - l_discount)) as revenue,
| o_orderdate,
| o_shippriority
| from
| customer,
| orders,
| lineitem
| where
| c_mktsegment = 'BUILDING'
| and c_custkey = o_custkey
| and l_orderkey = o_orderkey
| and o_orderdate < date '1995-03-15'
| and l_shipdate > date '1995-03-15'
| group by
| l_orderkey,
| o_orderdate,
| o_shippriority
| order by
| revenue desc,
| o_orderdate
| """).show()
18/10/21 12:10:40 INFO CodeGenerator: Code generated in 189.721249 ms
18/10/21 12:10:40 INFO CodeGenerator: Code generated in 29.113081 ms
18/10/21 12:10:40 INFO CodeGenerator: Code generated in 128.626791 ms
18/10/21 12:10:40 INFO CodeGenerator: Code generated in 16.440151 ms
18/10/21 12:10:40 INFO CodeGenerator: Code generated in 12.789172 ms
18/10/21 12:10:40 INFO ContextCleaner: Cleaned accumulator 3
18/10/21 12:10:40 INFO ContextCleaner: Cleaned accumulator 1
18/10/21 12:10:40 INFO ContextCleaner: Cleaned accumulator 0
18/10/21 12:10:40 INFO ContextCleaner: Cleaned accumulator 2
18/10/21 12:10:40 INFO CodeGenerator: Code generated in 18.835156 ms
18/10/21 12:10:40 INFO CodeGenerator: Code generated in 8.916005 ms
18/10/21 12:10:40 INFO ScalaEsRowRDD: Reading from [customer/doc]
18/10/21 12:10:43 INFO CodeGenerator: Code generated in 13.688141 ms
18/10/21 12:10:43 INFO CodeGenerator: Code generated in 22.876018 ms
18/10/21 12:10:43 INFO ScalaEsRowRDD: Reading from [orders/doc]
18/10/21 12:10:46 INFO CodeGenerator: Code generated in 13.804155 ms
18/10/21 12:10:46 INFO CodeGenerator: Code generated in 23.088171 ms
18/10/21 12:10:46 INFO ScalaEsRowRDD: Reading from [lineitem/doc]
18/10/21 12:10:55 INFO SparkContext: Starting job: show at <console>:54
18/10/21 12:10:55 INFO DAGScheduler: Registering RDD 7 (show at <console>:54)
18/10/21 12:10:55 INFO DAGScheduler: Registering RDD 11 (show at <console>:54)
18/10/21 12:10:55 INFO DAGScheduler: Registering RDD 16 (show at <console>:54)
18/10/21 12:10:55 INFO DAGScheduler: Registering RDD 20 (show at <console>:54)
18/10/21 12:10:55 INFO DAGScheduler: Got job 0 (show at <console>:54) with 200 output
18/10/21 12:10:55 INFO DAGScheduler: Final stage: ResultStage 4 (show at <console>:54
18/10/21 12:10:55 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2,
18/10/21 12:10:55 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 2, Shuffle
18/10/21 12:10:55 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[7ng parents
18/10/21 12:10:55 INFO MemoryStore: Block broadcast_0 stored as values in memory (est
18/10/21 12:10:55 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memor
18/10/21 12:10:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on node7.
18/10/21 12:10:55 INFO SparkContext: Created broadcast 0 from broadcast at DAGSchedul
18/10/21 12:10:55 INFO DAGScheduler: Submitting 27 missing tasks from ShuffleMapStage4) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
18/10/21 12:10:55 INFO YarnScheduler: Adding task set 0.0 with 27 tasks
18/10/21 12:10:55 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[1ing parents
18/10/21 12:10:55 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, node4.d bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, node1.d bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, node5.d bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, node3.d bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, node6.d bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, node2.d bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, node4.d bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, node1.d bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, node5.d bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, node3.d bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, node6473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, node2473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, node4473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, node1473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, node5473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, node3473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, node6473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, node2473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, node4473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, node1473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, node5473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, node3473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, node6473 bytes)
18/10/21 12:10:55 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, node2473 bytes)
18/10/21 12:10:55 INFO MemoryStore: Block broadcast_1 stored as values in memory (est
18/10/21 12:10:55 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memor
18/10/21 12:10:55 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on node7.
18/10/21 12:10:55 INFO SparkContext: Created broadcast 1 from broadcast at DAGSchedul
18/10/21 12:10:55 INFO DAGScheduler: Submitting 1497 missing tasks from ShuffleMapSta>:54) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
18/10/21 12:10:55 INFO YarnScheduler: Adding task set 1.0 with 1497 tasks
18/10/21 12:10:55 INFO DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[2ing parents
18/10/21 12:10:55 INFO MemoryStore: Block broadcast_2 stored as values in memory (est
18/10/21 12:10:55 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memor
18/10/21 12:10:55 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on node7.
18/10/21 12:10:55 INFO SparkContext: Created broadcast 2 from broadcast at DAGSchedul
18/10/21 12:10:55 INFO DAGScheduler: Submitting 5999 missing tasks from ShuffleMapSta>:54) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
18/10/21 12:10:55 INFO YarnScheduler: Adding task set 3.0 with 5999 tasks
18/10/21 12:10:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on node4.
18/10/21 12:10:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on node2.
18/10/21 12:10:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on node5.
18/10/21 12:10:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on node3.
18/10/21 12:10:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on node1.
18/10/21 12:10:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on node6.
18/10/21 12:11:22 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 24, node4473 bytes)
18/10/21 12:11:22 INFO TaskSetManager: Finished task 18.0 in stage 0.0 (TID 18) in 27
18/10/21 12:11:23 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, node1473 bytes)
18/10/21 12:11:23 INFO TaskSetManager: Finished task 19.0 in stage 0.0 (TID 19) in 27
18/10/21 12:11:24 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 26, node2473 bytes)
18/10/21 12:11:24 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 29
18/10/21 12:11:24 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 27, node6.9 bytes)
18/10/21 12:11:24 INFO TaskSetManager: Finished task 22.0 in stage 0.0 (TID 22) in 29
18/10/21 12:11:24 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on node6.
18/10/21 12:11:27 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 28, node3.9 bytes)
18/10/21 12:11:27 INFO TaskSetManager: Finished task 21.0 in stage 0.0 (TID 21) in 32
18/10/21 12:11:27 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on node3.
18/10/21 12:11:28 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 29, node1.9 bytes)
18/10/21 12:11:28 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 33
18/10/21 12:11:28 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on node1.
18/10/21 12:11:29 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 30, node3.9 bytes)
18/10/21 12:11:29 INFO TaskSetManager: Finished task 15.0 in stage 0.0 (TID 15) in 34
18/10/21 12:11:29 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 31, node6.9 bytes)
18/10/21 12:11:29 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 34
18/10/21 12:11:32 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 32, node5.9 bytes)
18/10/21 12:11:32 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 3724
18/10/21 12:11:32 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on node5.
18/10/21 12:11:33 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 33, node3.9 bytes)
18/10/21 12:11:33 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 3815
18/10/21 12:11:34 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 34, node3.9 bytes)
18/10/21 12:11:34 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 3899
18/10/21 12:11:35 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 35, node6.9 bytes)
18/10/21 12:11:35 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 4039
18/10/21 12:11:36 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 36, node4.9 bytes)
18/10/21 12:11:36 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4087
18/10/21 12:11:36 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on node4.
18/10/21 12:11:36 INFO TaskSetManager: Starting task 10.0 in stage 1.0 (TID 37, node4519 bytes)
18/10/21 12:11:36 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 4135
18/10/21 12:11:36 INFO TaskSetManager: Starting task 11.0 in stage 1.0 (TID 38, node1519 bytes)
18/10/21 12:11:36 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 4160
18/10/21 12:11:37 INFO TaskSetManager: Starting task 12.0 in stage 1.0 (TID 39, node5519 bytes)
18/10/21 12:11:37 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 14) in 42
18/10/21 12:11:47 INFO TaskSetManager: Starting task 13.0 in stage 1.0 (TID 40, node1519 bytes)
18/10/21 12:11:47 INFO TaskSetManager: Finished task 25.0 in stage 0.0 (TID 25) in 24
18/10/21 12:11:48 INFO TaskSetManager: Starting task 14.0 in stage 1.0 (TID 41, node2519 bytes)
18/10/21 12:11:48 INFO TaskSetManager: Finished task 23.0 in stage 0.0 (TID 23) in 52
18/10/21 12:11:48 INFO TaskSetManager: Starting task 15.0 in stage 1.0 (TID 42, node6519 bytes)
18/10/21 12:11:48 INFO TaskSetManager: Finished task 16.0 in stage 0.0 (TID 16) in 52
18/10/21 12:11:48 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on node2.
18/10/21 12:11:48 INFO TaskSetManager: Starting task 16.0 in stage 1.0 (TID 43, node5519 bytes)
18/10/21 12:11:48 INFO TaskSetManager: Finished task 20.0 in stage 0.0 (TID 20) in 53
18/10/21 12:11:48 INFO TaskSetManager: Starting task 17.0 in stage 1.0 (TID 44, node5519 bytes)
18/10/21 12:11:48 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 5309
18/10/21 12:11:48 INFO TaskSetManager: Starting task 18.0 in stage 1.0 (TID 45, node1519 bytes)
18/10/21 12:11:48 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 5309
18/10/21 12:11:49 INFO TaskSetManager: Starting task 19.0 in stage 1.0 (TID 46, node4519 bytes)
18/10/21 12:11:49 INFO TaskSetManager: Finished task 24.0 in stage 0.0 (TID 24) in 26
18/10/21 12:11:56 INFO TaskSetManager: Starting task 20.0 in stage 1.0 (TID 47, node2519 bytes)
18/10/21 12:11:56 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 6076
18/10/21 12:11:56 INFO TaskSetManager: Starting task 21.0 in stage 1.0 (TID 48, node4519 bytes)
18/10/21 12:11:56 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 60
18/10/21 12:11:56 INFO TaskSetManager: Starting task 22.0 in stage 1.0 (TID 49, node2519 bytes)
18/10/21 12:11:56 INFO TaskSetManager: Finished task 17.0 in stage 0.0 (TID 17) in 60
18/10/21 12:12:21 INFO TaskSetManager: Starting task 23.0 in stage 1.0 (TID 50, node2519 bytes)
18/10/21 12:12:21 INFO TaskSetManager: Finished task 26.0 in stage 0.0 (TID 26) in 57
18/10/21 12:12:21 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all compl
18/10/21 12:12:21 INFO DAGScheduler: ShuffleMapStage 0 (show at <console>:54) finishe
18/10/21 12:12:21 INFO DAGScheduler: looking for newly runnable stages
18/10/21 12:12:21 INFO DAGScheduler: running: Set(ShuffleMapStage 1, ShuffleMapStage
18/10/21 12:12:21 INFO DAGScheduler: waiting: Set(ShuffleMapStage 2, ResultStage 4)
18/10/21 12:12:21 INFO DAGScheduler: failed: Set()
18/10/21 12:12:31 INFO TaskSetManager: Starting task 24.0 in stage 1.0 (TID 51, node3519 bytes)
18/10/21 12:12:31 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 28) in 638
18/10/21 12:12:35 INFO TaskSetManager: Starting task 25.0 in stage 1.0 (TID 52, node4519 bytes)
18/10/21 12:12:35 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID 37) in 58
18/10/21 12:12:54 INFO TaskSetManager: Starting task 26.0 in stage 1.0 (TID 53, node6519 bytes)
18/10/21 12:12:54 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 35) in 791
18/10/21 12:12:56 INFO TaskSetManager: Starting task 27.0 in stage 1.0 (TID 54, node2519 bytes)
18/10/21 12:12:56 INFO TaskSetManager: Finished task 22.0 in stage 1.0 (TID 49) in 60
18/10/21 12:12:56 INFO TaskSetManager: Starting task 28.0 in stage 1.0 (TID 55, node6519 bytes)
18/10/21 12:12:56 INFO TaskSetManager: Finished task 15.0 in stage 1.0 (TID 42) in 68
18/10/21 12:12:57 INFO TaskSetManager: Starting task 29.0 in stage 1.0 (TID 56, node3519 bytes)
18/10/21 12:12:57 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 33) in 836
18/10/21 12:13:00 INFO TaskSetManager: Starting task 30.0 in stage 1.0 (TID 57, node5519 bytes)
18/10/21 12:13:00 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 32) in 879
18/10/21 12:13:01 INFO TaskSetManager: Starting task 31.0 in stage 1.0 (TID 58, node2519 bytes)
18/10/21 12:13:01 INFO TaskSetManager: Finished task 14.0 in stage 1.0 (TID 41) in 73
18/10/21 12:13:02 INFO TaskSetManager: Starting task 32.0 in stage 1.0 (TID 59, node4519 bytes)
18/10/21 12:13:02 INFO TaskSetManager: Finished task 19.0 in stage 1.0 (TID 46) in 73
18/10/21 12:13:02 INFO TaskSetManager: Starting task 33.0 in stage 1.0 (TID 60, node1519 bytes)
18/10/21 12:13:02 INFO TaskSetManager: Finished task 11.0 in stage 1.0 (TID 38) in 86
18/10/21 12:13:03 INFO TaskSetManager: Starting task 34.0 in stage 1.0 (TID 61, node5519 bytes)
18/10/21 12:13:03 INFO TaskSetManager: Finished task 17.0 in stage 1.0 (TID 44) in 75
18/10/21 12:13:04 INFO TaskSetManager: Starting task 35.0 in stage 1.0 (TID 62, node3519 bytes)
18/10/21 12:13:04 INFO TaskSetManager: Finished task 24.0 in stage 1.0 (TID 51) in 32
18/10/21 12:13:07 INFO TaskSetManager: Starting task 36.0 in stage 1.0 (TID 63, node6519 bytes)
18/10/21 12:13:07 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 31) in 972
18/10/21 12:13:14 INFO TaskSetManager: Starting task 37.0 in stage 1.0 (TID 64, node2519 bytes)
18/10/21 12:13:14 INFO TaskSetManager: Finished task 23.0 in stage 1.0 (TID 50) in 53
18/10/21 12:13:20 INFO TaskSetManager: Starting task 38.0 in stage 1.0 (TID 65, node1519 bytes)
18/10/21 12:13:20 INFO TaskSetManager: Finished task 13.0 in stage 1.0 (TID 40) in 92
18/10/21 12:13:22 INFO TaskSetManager: Starting task 39.0 in stage 1.0 (TID 66, node1519 bytes)
18/10/21 12:13:22 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 29) in 113
18/10/21 12:13:22 INFO TaskSetManager: Starting task 40.0 in stage 1.0 (TID 67, node6519 bytes)
18/10/21 12:13:22 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 27) in 117
18/10/21 12:13:22 INFO TaskSetManager: Starting task 41.0 in stage 1.0 (TID 68, node2519 bytes)
18/10/21 12:13:22 INFO TaskSetManager: Finished task 20.0 in stage 1.0 (TID 47) in 86
18/10/21 12:13:27 INFO TaskSetManager: Starting task 42.0 in stage 1.0 (TID 69, node4519 bytes)
18/10/21 12:13:27 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 36) in 111
18/10/21 12:13:28 INFO TaskSetManager: Starting task 43.0 in stage 1.0 (TID 70, node5519 bytes)
18/10/21 12:13:28 INFO TaskSetManager: Finished task 12.0 in stage 1.0 (TID 39) in 11
18/10/21 12:13:32 INFO TaskSetManager: Starting task 44.0 in stage 1.0 (TID 71, node5519 bytes)
18/10/21 12:13:32 INFO TaskSetManager: Finished task 16.0 in stage 1.0 (TID 43) in 10
18/10/21 12:13:32 INFO TaskSetManager: Starting task 45.0 in stage 1.0 (TID 72, node4519 bytes)
18/10/21 12:13:32 INFO TaskSetManager: Finished task 25.0 in stage 1.0 (TID 52) in 57
18/10/21 12:13:37 INFO TaskSetManager: Starting task 46.0 in stage 1.0 (TID 73, node6519 bytes)
18/10/21 12:13:37 INFO TaskSetManager: Finished task 26.0 in stage 1.0 (TID 53) in 42
18/10/21 12:13:41 INFO TaskSetManager: Starting task 47.0 in stage 1.0 (TID 74, node1519 bytes)
18/10/21 12:13:41 INFO TaskSetManager: Finished task 33.0 in stage 1.0 (TID 60) in 38
18/10/21 12:13:43 INFO TaskSetManager: Starting task 48.0 in stage 1.0 (TID 75, node3519 bytes)
18/10/21 12:13:43 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 34) in 128
18/10/21 12:13:45 INFO TaskSetManager: Starting task 49.0 in stage 1.0 (TID 76, node5519 bytes)
18/10/21 12:13:45 INFO TaskSetManager: Finished task 34.0 in stage 1.0 (TID 61) in 41
18/10/21 12:13:48 INFO TaskSetManager: Starting task 50.0 in stage 1.0 (TID 77, node6519 bytes)
18/10/21 12:13:48 INFO TaskSetManager: Finished task 28.0 in stage 1.0 (TID 55) in 51
18/10/21 12:13:52 INFO TaskSetManager: Starting task 51.0 in stage 1.0 (TID 78, node1519 bytes)
18/10/21 12:13:52 INFO TaskSetManager: Finished task 18.0 in stage 1.0 (TID 45) in 12
18/10/21 12:13:52 INFO TaskSetManager: Starting task 52.0 in stage 1.0 (TID 79, node2519 bytes)
18/10/21 12:13:52 INFO TaskSetManager: Finished task 37.0 in stage 1.0 (TID 64) in 37
18/10/21 12:13:58 INFO TaskSetManager: Starting task 53.0 in stage 1.0 (TID 80, node4519 bytes)
18/10/21 12:13:58 INFO TaskSetManager: Finished task 21.0 in stage 1.0 (TID 48) in 12
18/10/21 12:14:00 INFO TaskSetManager: Starting task 54.0 in stage 1.0 (TID 81, node1519 bytes)
18/10/21 12:14:00 INFO TaskSetManager: Finished task 38.0 in stage 1.0 (TID 65) in 39
18/10/21 12:14:06 INFO TaskSetManager: Starting task 55.0 in stage 1.0 (TID 82, node5519 bytes)
18/10/21 12:14:06 INFO TaskSetManager: Finished task 43.0 in stage 1.0 (TID 70) in 38
18/10/21 12:14:07 INFO TaskSetManager: Starting task 56.0 in stage 1.0 (TID 83, node4519 bytes)
18/10/21 12:14:07 INFO TaskSetManager: Finished task 45.0 in stage 1.0 (TID 72) in 34
18/10/21 12:14:22 INFO TaskSetManager: Starting task 57.0 in stage 1.0 (TID 84, node3519 bytes)
18/10/21 12:14:22 INFO TaskSetManager: Finished task 48.0 in stage 1.0 (TID 75) in 38
18/10/21 12:14:25 INFO TaskSetManager: Starting task 58.0 in stage 1.0 (TID 85, node1519 bytes)
18/10/21 12:14:25 INFO TaskSetManager: Finished task 51.0 in stage 1.0 (TID 78) in 33
18/10/21 12:14:31 INFO TaskSetManager: Starting task 59.0 in stage 1.0 (TID 86, node2519 bytes)
18/10/21 12:14:31 INFO TaskSetManager: Finished task 52.0 in stage 1.0 (TID 79) in 38
18/10/21 12:14:34 INFO TaskSetManager: Starting task 60.0 in stage 1.0 (TID 87, node4519 bytes)
18/10/21 12:14:34 INFO TaskSetManager: Finished task 53.0 in stage 1.0 (TID 80) in 35
18/10/21 12:14:35 INFO TaskSetManager: Starting task 61.0 in stage 1.0 (TID 88, node2519 bytes)
18/10/21 12:14:35 INFO TaskSetManager: Finished task 31.0 in stage 1.0 (TID 58) in 93
18/10/21 12:14:35 INFO TaskSetManager: Starting task 62.0 in stage 1.0 (TID 89, node5519 bytes)
18/10/21 12:14:35 INFO TaskSetManager: Finished task 30.0 in stage 1.0 (TID 57) in 94
18/10/21 12:14:36 INFO TaskSetManager: Starting task 63.0 in stage 1.0 (TID 90, node2519 bytes)
18/10/21 12:14:36 INFO TaskSetManager: Finished task 27.0 in stage 1.0 (TID 54) in 99
18/10/21 12:14:41 INFO TaskSetManager: Starting task 64.0 in stage 1.0 (TID 91, node4519 bytes)
18/10/21 12:14:41 INFO TaskSetManager: Finished task 56.0 in stage 1.0 (TID 83) in 34
18/10/21 12:15:09 INFO TaskSetManager: Starting task 65.0 in stage 1.0 (TID 92, node4519 bytes)
18/10/21 12:15:09 INFO TaskSetManager: Finished task 60.0 in stage 1.0 (TID 87) in 35
18/10/21 12:15:10 INFO TaskSetManager: Starting task 66.0 in stage 1.0 (TID 93, node2519 bytes)
18/10/21 12:15:10 INFO TaskSetManager: Finished task 63.0 in stage 1.0 (TID 90) in 33
18/10/21 12:15:12 INFO TaskSetManager: Starting task 67.0 in stage 1.0 (TID 94, node4519 bytes)
18/10/21 12:15:12 INFO TaskSetManager: Finished task 64.0 in stage 1.0 (TID 91) in 30
18/10/21 12:15:24 INFO TaskSetManager: Starting task 68.0 in stage 1.0 (TID 95, node5519 bytes)
18/10/21 12:15:24 INFO TaskSetManager: Finished task 44.0 in stage 1.0 (TID 71) in 11
18/10/21 12:15:26 INFO TaskSetManager: Starting task 69.0 in stage 1.0 (TID 96, node6519 bytes)
18/10/21 12:15:26 INFO TaskSetManager: Finished task 50.0 in stage 1.0 (TID 77) in 98
18/10/21 12:15:42 INFO TaskSetManager: Starting task 70.0 in stage 1.0 (TID 97, node4519 bytes)
18/10/21 12:15:42 INFO TaskSetManager: Finished task 67.0 in stage 1.0 (TID 94) in 29
18/10/21 12:15:44 INFO TaskSetManager: Starting task 71.0 in stage 1.0 (TID 98, node5519 bytes)
18/10/21 12:15:44 INFO TaskSetManager: Finished task 55.0 in stage 1.0 (TID 82) in 97
18/10/21 12:15:47 INFO TaskSetManager: Starting task 72.0 in stage 1.0 (TID 99, node5519 bytes)
18/10/21 12:15:47 INFO TaskSetManager: Finished task 49.0 in stage 1.0 (TID 76) in 12
18/10/21 12:16:00 INFO TaskSetManager: Starting task 73.0 in stage 1.0 (TID 100, node3519 bytes)
18/10/21 12:16:00 INFO TaskSetManager: Finished task 35.0 in stage 1.0 (TID 62) in 17
18/10/21 12:16:00 INFO TaskSetManager: Starting task 74.0 in stage 1.0 (TID 101, node3519 bytes)
18/10/21 12:16:00 INFO TaskSetManager: Finished task 69.0 in stage 1.0 (TID 96) in 34
18/10/21 12:16:02 INFO TaskSetManager: Starting task 75.0 in stage 1.0 (TID 102, node3519 bytes)
18/10/21 12:16:02 INFO TaskSetManager: Finished task 68.0 in stage 1.0 (TID 95) in 37
18/10/21 12:16:09 INFO TaskSetManager: Starting task 76.0 in stage 1.0 (TID 103, node3519 bytes)
18/10/21 12:16:09 INFO TaskSetManager: Finished task 61.0 in stage 1.0 (TID 88) in 94
18/10/21 12:16:12 INFO TaskSetManager: Starting task 77.0 in stage 1.0 (TID 104, node3519 bytes)
18/10/21 12:16:12 INFO TaskSetManager: Finished task 62.0 in stage 1.0 (TID 89) in 97
18/10/21 12:16:14 INFO TaskSetManager: Starting task 78.0 in stage 1.0 (TID 105, node3519 bytes)
18/10/21 12:16:14 INFO TaskSetManager: Finished task 70.0 in stage 1.0 (TID 97) in 32
18/10/21 12:16:15 INFO TaskSetManager: Starting task 79.0 in stage 1.0 (TID 106, node3519 bytes)
18/10/21 12:16:15 INFO TaskSetManager: Finished task 57.0 in stage 1.0 (TID 84) in 11
18/10/21 12:16:26 INFO TaskSetManager: Starting task 80.0 in stage 1.0 (TID 107, node3519 bytes)
18/10/21 12:16:26 INFO TaskSetManager: Finished task 39.0 in stage 1.0 (TID 66) in 18
18/10/21 12:16:27 INFO TaskSetManager: Starting task 81.0 in stage 1.0 (TID 108, node3519 bytes)
18/10/21 12:16:27 INFO TaskSetManager: Finished task 72.0 in stage 1.0 (TID 99) in 39
18/10/21 12:16:29 INFO TaskSetManager: Starting task 82.0 in stage 1.0 (TID 109, node3519 bytes)
18/10/21 12:16:29 INFO TaskSetManager: Finished task 59.0 in stage 1.0 (TID 86) in 11
18/10/21 12:16:31 INFO TaskSetManager: Starting task 83.0 in stage 1.0 (TID 110, node3519 bytes)
18/10/21 12:16:31 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 30) in 302
18/10/21 12:16:32 INFO TaskSetManager: Starting task 84.0 in stage 1.0 (TID 111, node3519 bytes)
18/10/21 12:16:32 INFO TaskSetManager: Finished task 58.0 in stage 1.0 (TID 85) in 12
18/10/21 12:16:36 INFO TaskSetManager: Starting task 85.0 in stage 1.0 (TID 112, node3519 bytes)
18/10/21 12:16:36 INFO TaskSetManager: Finished task 42.0 in stage 1.0 (TID 69) in 18
18/10/21 12:16:42 INFO TaskSetManager: Starting task 86.0 in stage 1.0 (TID 113, node3519 bytes)
18/10/21 12:16:42 INFO TaskSetManager: Finished task 75.0 in stage 1.0 (TID 102) in 4
18/10/21 12:16:46 INFO TaskSetManager: Starting task 87.0 in stage 1.0 (TID 114, node3519 bytes)
18/10/21 12:16:46 INFO TaskSetManager: Finished task 78.0 in stage 1.0 (TID 105) in 3
18/10/21 12:16:49 INFO TaskSetManager: Starting task 88.0 in stage 1.0 (TID 115, node3519 bytes)
18/10/21 12:16:49 INFO TaskSetManager: Finished task 76.0 in stage 1.0 (TID 103) in 3
18/10/21 12:16:59 INFO TaskSetManager: Starting task 89.0 in stage 1.0 (TID 116, node3519 bytes)
18/10/21 12:16:59 INFO TaskSetManager: Finished task 40.0 in stage 1.0 (TID 67) in 21
18/10/21 12:17:00 INFO TaskSetManager: Starting task 90.0 in stage 1.0 (TID 117, node3519 bytes)
18/10/21 12:17:00 INFO TaskSetManager: Finished task 65.0 in stage 1.0 (TID 92) in 11
18/10/21 12:17:01 INFO TaskSetManager: Starting task 91.0 in stage 1.0 (TID 118, node3519 bytes)
18/10/21 12:17:01 INFO TaskSetManager: Finished task 47.0 in stage 1.0 (TID 74) in 20
18/10/21 12:17:06 INFO TaskSetManager: Starting task 92.0 in stage 1.0 (TID 119, node3519 bytes)
18/10/21 12:17:06 INFO TaskSetManager: Finished task 82.0 in stage 1.0 (TID 109) in 3
18/10/21 12:17:15 INFO TaskSetManager: Starting task 93.0 in stage 1.0 (TID 120, node3519 bytes)
18/10/21 12:17:15 INFO TaskSetManager: Finished task 71.0 in stage 1.0 (TID 98) in 91
18/10/21 12:17:32 INFO TaskSetManager: Starting task 94.0 in stage 1.0 (TID 121, node3519 bytes)
18/10/21 12:17:32 INFO TaskSetManager: Finished task 91.0 in stage 1.0 (TID 118) in 3
18/10/21 12:17:32 INFO TaskSetManager: Starting task 95.0 in stage 1.0 (TID 122, node3519 bytes)
18/10/21 12:17:32 INFO TaskSetManager: Finished task 54.0 in stage 1.0 (TID 81) in 21
18/10/21 12:17:34 INFO TaskSetManager: Starting task 96.0 in stage 1.0 (TID 123, node3519 bytes)
18/10/21 12:17:34 INFO TaskSetManager: Finished task 89.0 in stage 1.0 (TID 116) in 3
18/10/21 12:17:34 INFO TaskSetManager: Starting task 97.0 in stage 1.0 (TID 124, node3519 bytes)
18/10/21 12:17:34 INFO TaskSetManager: Finished task 81.0 in stage 1.0 (TID 108) in 6
18/10/21 12:17:41 INFO TaskSetManager: Starting task 98.0 in stage 1.0 (TID 125, node3519 bytes)
18/10/21 12:17:41 INFO TaskSetManager: Finished task 83.0 in stage 1.0 (TID 110) in 7
18/10/21 12:17:48 INFO TaskSetManager: Starting task 99.0 in stage 1.0 (TID 126, node3519 bytes)
18/10/21 12:17:48 INFO TaskSetManager: Finished task 77.0 in stage 1.0 (TID 104) in 9
18/10/21 12:17:52 INFO TaskSetManager: Starting task 100.0 in stage 1.0 (TID 127, nod 13519 bytes)
18/10/21 12:17:52 INFO TaskSetManager: Finished task 93.0 in stage 1.0 (TID 120) in 3
18/10/21 12:18:04 INFO TaskSetManager: Starting task 101.0 in stage 1.0 (TID 128, nod 13519 bytes)
18/10/21 12:18:04 INFO TaskSetManager: Finished task 96.0 in stage 1.0 (TID 123) in 2
18/10/21 12:18:07 INFO TaskSetManager: Starting task 102.0 in stage 1.0 (TID 129, nod 13519 bytes)
18/10/21 12:18:07 INFO TaskSetManager: Finished task 95.0 in stage 1.0 (TID 122) in 3
18/10/21 12:18:13 INFO TaskSetManager: Starting task 103.0 in stage 1.0 (TID 130, nod 13519 bytes)
18/10/21 12:18:13 INFO TaskSetManager: Finished task 98.0 in stage 1.0 (TID 125) in 3
18/10/21 12:18:22 INFO TaskSetManager: Starting task 104.0 in stage 1.0 (TID 131, nod 13519 bytes)
18/10/21 12:18:22 INFO TaskSetManager: Finished task 86.0 in stage 1.0 (TID 113) in 9
18/10/21 12:18:28 INFO TaskSetManager: Starting task 105.0 in stage 1.0 (TID 132, nod 13519 bytes)
18/10/21 12:18:28 INFO TaskSetManager: Finished task 88.0 in stage 1.0 (TID 115) in 9
18/10/21 12:18:29 INFO TaskSetManager: Starting task 106.0 in stage 1.0 (TID 133, nod 13519 bytes)
18/10/21 12:18:29 INFO TaskSetManager: Finished task 97.0 in stage 1.0 (TID 124) in 5
18/10/21 12:18:54 INFO TaskSetManager: Starting task 107.0 in stage 1.0 (TID 134, nod 13519 bytes)
18/10/21 12:18:54 INFO TaskSetManager: Finished task 90.0 in stage 1.0 (TID 117) in 1
18/10/21 12:19:09 INFO TaskSetManager: Starting task 108.0 in stage 1.0 (TID 135, nod 13519 bytes)
18/10/21 12:19:09 INFO TaskSetManager: Finished task 101.0 in stage 1.0 (TID 128) in
18/10/21 12:19:18 INFO TaskSetManager: Starting task 109.0 in stage 1.0 (TID 136, nod 13519 bytes)
18/10/21 12:19:18 INFO TaskSetManager: Finished task 103.0 in stage 1.0 (TID 130) in
18/10/21 12:19:26 INFO TaskSetManager: Starting task 110.0 in stage 1.0 (TID 137, nod 13519 bytes)
18/10/21 12:19:26 INFO TaskSetManager: Finished task 107.0 in stage 1.0 (TID 134) in
18/10/21 12:19:29 INFO TaskSetManager: Starting task 111.0 in stage 1.0 (TID 138, nod 13519 bytes)
18/10/21 12:19:29 INFO TaskSetManager: Finished task 100.0 in stage 1.0 (TID 127) in
18/10/21 12:19:32 INFO TaskSetManager: Starting task 112.0 in stage 1.0 (TID 139, nod 13519 bytes)
18/10/21 12:19:32 INFO TaskSetManager: Finished task 105.0 in stage 1.0 (TID 132) in
18/10/21 12:19:57 INFO TaskSetManager: Starting task 113.0 in stage 1.0 (TID 140, nod 13519 bytes)
18/10/21 12:19:57 INFO TaskSetManager: Finished task 110.0 in stage 1.0 (TID 137) in
18/10/21 12:19:57 INFO TaskSetManager: Starting task 114.0 in stage 1.0 (TID 141, nod 13519 bytes)
18/10/21 12:19:57 INFO TaskSetManager: Finished task 104.0 in stage 1.0 (TID 131) in
18/10/21 12:20:07 INFO TaskSetManager: Starting task 115.0 in stage 1.0 (TID 142, nod 13519 bytes)
18/10/21 12:20:07 INFO TaskSetManager: Finished task 112.0 in stage 1.0 (TID 139) in
18/10/21 12:20:09 INFO TaskSetManager: Starting task 116.0 in stage 1.0 (TID 143, nod 13519 bytes)
18/10/21 12:20:09 INFO TaskSetManager: Finished task 109.0 in stage 1.0 (TID 136) in
18/10/21 12:20:38 INFO TaskSetManager: Starting task 117.0 in stage 1.0 (TID 144, nod 13519 bytes)
18/10/21 12:20:38 INFO TaskSetManager: Finished task 116.0 in stage 1.0 (TID 143) in
18/10/21 12:20:42 INFO TaskSetManager: Starting task 118.0 in stage 1.0 (TID 145, nod 13519 bytes)
18/10/21 12:20:42 INFO TaskSetManager: Finished task 108.0 in stage 1.0 (TID 135) in
18/10/21 12:21:04 INFO TaskSetManager: Starting task 119.0 in stage 1.0 (TID 146, nod 13519 bytes)
18/10/21 12:21:04 INFO TaskSetManager: Finished task 111.0 in stage 1.0 (TID 138) in
18/10/21 12:21:10 INFO TaskSetManager: Starting task 120.0 in stage 1.0 (TID 147, nod 13519 bytes)
18/10/21 12:21:10 INFO TaskSetManager: Finished task 118.0 in stage 1.0 (TID 145) in
18/10/21 12:21:32 INFO TaskSetManager: Starting task 121.0 in stage 1.0 (TID 148, nod 13519 bytes)
18/10/21 12:21:32 INFO TaskSetManager: Finished task 113.0 in stage 1.0 (TID 140) in
18/10/21 12:21:34 INFO TaskSetManager: Starting task 122.0 in stage 1.0 (TID 149, nod 13519 bytes)
18/10/21 12:21:34 INFO TaskSetManager: Finished task 74.0 in stage 1.0 (TID 101) in 3
18/10/21 12:21:56 INFO TaskSetManager: Starting task 123.0 in stage 1.0 (TID 150, nod 13519 bytes)
18/10/21 12:21:56 INFO TaskSetManager: Finished task 119.0 in stage 1.0 (TID 146) in
18/10/21 12:22:06 INFO TaskSetManager: Starting task 124.0 in stage 1.0 (TID 151, nod 13519 bytes)
18/10/21 12:22:06 INFO TaskSetManager: Finished task 122.0 in stage 1.0 (TID 149) in
18/10/21 12:22:09 INFO TaskSetManager: Starting task 125.0 in stage 1.0 (TID 152, nod 13519 bytes)
18/10/21 12:22:09 INFO TaskSetManager: Finished task 117.0 in stage 1.0 (TID 144) in
18/10/21 12:22:23 INFO TaskSetManager: Starting task 126.0 in stage 1.0 (TID 153, nod 13519 bytes)
18/10/21 12:22:23 INFO TaskSetManager: Finished task 121.0 in stage 1.0 (TID 148) in
18/10/21 12:22:38 INFO TaskSetManager: Starting task 127.0 in stage 1.0 (TID 154, nod 13519 bytes)
18/10/21 12:22:38 INFO TaskSetManager: Finished task 120.0 in stage 1.0 (TID 147) in
18/10/21 12:22:42 INFO TaskSetManager: Starting task 128.0 in stage 1.0 (TID 155, nod 13519 bytes)
18/10/21 12:22:42 INFO TaskSetManager: Finished task 125.0 in stage 1.0 (TID 152) in
18/10/21 12:22:57 INFO TaskSetManager: Starting task 129.0 in stage 1.0 (TID 156, nod 13519 bytes)
18/10/21 12:22:57 INFO TaskSetManager: Finished task 126.0 in stage 1.0 (TID 153) in
18/10/21 12:23:07 INFO TaskSetManager: Starting task 130.0 in stage 1.0 (TID 157, nod 13519 bytes)
18/10/21 12:23:07 INFO TaskSetManager: Finished task 84.0 in stage 1.0 (TID 111) in 3
18/10/21 12:23:17 INFO TaskSetManager: Starting task 131.0 in stage 1.0 (TID 158, nod 13519 bytes)
18/10/21 12:23:17 INFO TaskSetManager: Finished task 128.0 in stage 1.0 (TID 155) in
18/10/21 12:23:29 INFO TaskSetManager: Starting task 132.0 in stage 1.0 (TID 159, nod 13519 bytes)
18/10/21 12:23:29 INFO TaskSetManager: Finished task 127.0 in stage 1.0 (TID 154) in
18/10/21 12:23:30 INFO TaskSetManager: Starting task 133.0 in stage 1.0 (TID 160, nod 13519 bytes)
18/10/21 12:23:30 INFO TaskSetManager: Finished task 124.0 in stage 1.0 (TID 151) in
18/10/21 12:23:31 INFO TaskSetManager: Starting task 134.0 in stage 1.0 (TID 161, nod 13519 bytes)
18/10/21 12:23:31 INFO TaskSetManager: Finished task 129.0 in stage 1.0 (TID 156) in
18/10/21 12:23:42 INFO TaskSetManager: Starting task 135.0 in stage 1.0 (TID 162, nod 13519 bytes)
18/10/21 12:23:42 INFO TaskSetManager: Finished task 130.0 in stage 1.0 (TID 157) in
18/10/21 12:24:00 INFO TaskSetManager: Starting task 136.0 in stage 1.0 (TID 163, nod 13519 bytes)
18/10/21 12:24:00 INFO TaskSetManager: Finished task 132.0 in stage 1.0 (TID 159) in
18/10/21 12:24:03 INFO TaskSetManager: Starting task 137.0 in stage 1.0 (TID 164, nod 13519 bytes)
18/10/21 12:24:03 INFO TaskSetManager: Finished task 133.0 in stage 1.0 (TID 160) in
18/10/21 12:24:04 INFO TaskSetManager: Starting task 138.0 in stage 1.0 (TID 165, nod 13519 bytes)
18/10/21 12:24:04 INFO TaskSetManager: Finished task 92.0 in stage 1.0 (TID 119) in 4
18/10/21 12:24:08 INFO TaskSetManager: Starting task 139.0 in stage 1.0 (TID 166, nod 13519 bytes)
18/10/21 12:24:08 INFO TaskSetManager: Finished task 134.0 in stage 1.0 (TID 161) in
18/10/21 12:24:17 INFO TaskSetManager: Starting task 140.0 in stage 1.0 (TID 167, nod 13519 bytes)
18/10/21 12:24:17 INFO TaskSetManager: Finished task 135.0 in stage 1.0 (TID 162) in
18/10/21 12:24:33 INFO TaskSetManager: Starting task 141.0 in stage 1.0 (TID 168, nod 13519 bytes)
18/10/21 12:24:33 INFO TaskSetManager: Finished task 137.0 in stage 1.0 (TID 164) in
18/10/21 12:24:54 INFO TaskSetManager: Starting task 142.0 in stage 1.0 (TID 169, nod 13519 bytes)
18/10/21 12:24:54 INFO TaskSetManager: Finished task 138.0 in stage 1.0 (TID 165) in
18/10/21 12:25:28 INFO TaskSetManager: Starting task 143.0 in stage 1.0 (TID 170, nod 13519 bytes)
18/10/21 12:25:28 INFO TaskSetManager: Finished task 142.0 in stage 1.0 (TID 169) in
18/10/21 12:30:02 INFO TaskSetManager: Starting task 144.0 in stage 1.0 (TID 171, nod 13519 bytes)
18/10/21 12:30:02 INFO TaskSetManager: Finished task 32.0 in stage 1.0 (TID 59) in 10
18/10/21 12:30:25 INFO TaskSetManager: Starting task 145.0 in stage 1.0 (TID 172, nod 13519 bytes)
18/10/21 12:30:25 INFO TaskSetManager: Finished task 36.0 in stage 1.0 (TID 63) in 10
18/10/21 12:30:52 INFO TaskSetManager: Starting task 146.0 in stage 1.0 (TID 173, nod 13519 bytes)
18/10/21 12:30:52 INFO TaskSetManager: Finished task 144.0 in stage 1.0 (TID 171) in
18/10/21 12:31:15 INFO TaskSetManager: Starting task 147.0 in stage 1.0 (TID 174, nod 13519 bytes)
18/10/21 12:31:15 INFO TaskSetManager: Finished task 145.0 in stage 1.0 (TID 172) in
18/10/21 12:31:24 INFO TaskSetManager: Starting task 148.0 in stage 1.0 (TID 175, nod 13519 bytes)
18/10/21 12:31:24 INFO TaskSetManager: Finished task 146.0 in stage 1.0 (TID 173) in
18/10/21 12:31:47 INFO TaskSetManager: Starting task 149.0 in stage 1.0 (TID 176, nod 13519 bytes)
18/10/21 12:31:47 INFO TaskSetManager: Finished task 147.0 in stage 1.0 (TID 174) in
18/10/21 12:32:45 INFO TaskSetManager: Starting task 150.0 in stage 1.0 (TID 177, nod 13519 bytes)
18/10/21 12:32:45 INFO TaskSetManager: Finished task 148.0 in stage 1.0 (TID 175) in
18/10/21 12:33:10 INFO TaskSetManager: Starting task 151.0 in stage 1.0 (TID 178, nod 13519 bytes)
18/10/21 12:33:10 INFO TaskSetManager: Finished task 149.0 in stage 1.0 (TID 176) in
18/10/21 12:33:45 INFO TaskSetManager: Starting task 152.0 in stage 1.0 (TID 179, node6.dat, executor 6, partition 152, RACK_LOCAL, 13519 bytes)
18/10/21 12:33:45 INFO TaskSetManager: Finished task 151.0 in stage 1.0 (TID 178) in 34546 ms on node6.dat (executor 6) (129/1497)
18/10/21 12:34:05 INFO TaskSetManager: Starting task 153.0 in stage 1.0 (TID 180, node4.dat, executor 2, partition 153, RACK_LOCAL, 13519 bytes)
18/10/21 12:34:05 INFO TaskSetManager: Finished task 150.0 in stage 1.0 (TID 177) in 79664 ms on node4.dat (executor 2) (130/1497)
18/10/21 12:34:13 INFO TaskSetManager: Starting task 154.0 in stage 1.0 (TID 181, node5.dat, executor 4, partition 154, RACK_LOCAL, 13519 bytes)
18/10/21 12:34:13 INFO TaskSetManager: Finished task 99.0 in stage 1.0 (TID 126) in 985445 ms on node5.dat (executor 4) (131/1497)
18/10/21 12:34:40 INFO TaskSetManager: Starting task 155.0 in stage 1.0 (TID 182, node4.dat, executor 2, partition 155, RACK_LOCAL, 13519 bytes)
18/10/21 12:34:40 INFO TaskSetManager: Finished task 153.0 in stage 1.0 (TID 180) in 34767 ms on node4.dat (executor 2) (132/1497)
18/10/21 12:35:27 INFO TaskSetManager: Starting task 156.0 in stage 1.0 (TID 183, node5.dat, executor 4, partition 156, RACK_LOCAL, 13519 bytes)
18/10/21 12:35:27 INFO TaskSetManager: Finished task 154.0 in stage 1.0 (TID 181) in 74281 ms on node5.dat (executor 4) (133/1497)
18/10/21 12:35:54 INFO TaskSetManager: Starting task 157.0 in stage 1.0 (TID 184, node2.dat, executor 1, partition 157, RACK_LOCAL, 13519 bytes)
18/10/21 12:35:54 INFO TaskSetManager: Finished task 66.0 in stage 1.0 (TID 93) in 1243700 ms on node2.dat (executor 1) (134/1497)
18/10/21 12:35:55 INFO TaskSetManager: Starting task 158.0 in stage 1.0 (TID 185, node5.dat, executor 4, partition 158, RACK_LOCAL, 13519 bytes)
18/10/21 12:35:55 INFO TaskSetManager: Finished task 156.0 in stage 1.0 (TID 183) in 27829 ms on node5.dat (executor 4) (135/1497)
18/10/21 12:36:22 INFO TaskSetManager: Starting task 159.0 in stage 1.0 (TID 186, node2.dat, executor 1, partition 159, RACK_LOCAL, 13519 bytes)
18/10/21 12:36:22 INFO TaskSetManager: Finished task 157.0 in stage 1.0 (TID 184) in 28083 ms on node2.dat (executor 1) (136/1497)
18/10/21 12:36:33 INFO TaskSetManager: Starting task 160.0 in stage 1.0 (TID 187, node1.dat, executor 3, partition 160, RACK_LOCAL, 13519 bytes)
18/10/21 12:36:33 INFO TaskSetManager: Finished task 102.0 in stage 1.0 (TID 129) in 1105869 ms on node1.dat (executor 3) (137/1497)
18/10/21 12:37:10 INFO TaskSetManager: Starting task 161.0 in stage 1.0 (TID 188, node2.dat, executor 1, partition 161, RACK_LOCAL, 13519 bytes)
18/10/21 12:37:10 INFO TaskSetManager: Finished task 159.0 in stage 1.0 (TID 186) in 48229 ms on node2.dat (executor 1) (138/1497)
18/10/21 12:37:13 INFO TaskSetManager: Starting task 162.0 in stage 1.0 (TID 189, node5.dat, executor 4, partition 162, RACK_LOCAL, 13519 bytes)
18/10/21 12:37:13 INFO TaskSetManager: Finished task 158.0 in stage 1.0 (TID 185) in 77503 ms on node5.dat (executor 4) (139/1497)
18/10/21 12:37:40 INFO TaskSetManager: Starting task 163.0 in stage 1.0 (TID 190, node2.dat, executor 1, partition 163, RACK_LOCAL, 13519 bytes)
18/10/21 12:37:40 INFO TaskSetManager: Finished task 161.0 in stage 1.0 (TID 188) in 30316 ms on node2.dat (executor 1) (140/1497)
18/10/21 12:37:51 INFO TaskSetManager: Starting task 164.0 in stage 1.0 (TID 191, node1.dat, executor 3, partition 164, RACK_LOCAL, 13519 bytes)
18/10/21 12:37:51 INFO TaskSetManager: Finished task 160.0 in stage 1.0 (TID 187) in 78075 ms on node1.dat (executor 3) (141/1497)
18/10/21 12:38:20 INFO TaskSetManager: Starting task 165.0 in stage 1.0 (TID 192, node1.dat, executor 3, partition 165, RACK_LOCAL, 13519 bytes)
18/10/21 12:38:20 INFO TaskSetManager: Finished task 164.0 in stage 1.0 (TID 191) in 28988 ms on node1.dat (executor 3) (142/1497)
18/10/21 12:38:31 INFO TaskSetManager: Starting task 166.0 in stage 1.0 (TID 193, node5.dat, executor 4, partition 166, RACK_LOCAL, 13519 bytes)
18/10/21 12:38:31 INFO TaskSetManager: Finished task 162.0 in stage 1.0 (TID 189) in 78786 ms on node5.dat (executor 4) (143/1497)
18/10/21 12:38:32 INFO TaskSetManager: Starting task 167.0 in stage 1.0 (TID 194, node2.dat, executor 1, partition 167, RACK_LOCAL, 13519 bytes)
18/10/21 12:38:32 INFO TaskSetManager: Finished task 163.0 in stage 1.0 (TID 190) in 51649 ms on node2.dat (executor 1) (144/1497)
18/10/21 12:38:55 INFO TaskSetManager: Starting task 168.0 in stage 1.0 (TID 195, node1.dat, executor 3, partition 168, RACK_LOCAL, 13519 bytes)
18/10/21 12:38:55 INFO TaskSetManager: Finished task 165.0 in stage 1.0 (TID 192) in 35627 ms on node1.dat (executor 3) (145/1497)
18/10/21 12:39:03 INFO TaskSetManager: Starting task 169.0 in stage 1.0 (TID 196, node5.dat, executor 4, partition 169, RACK_LOCAL, 13519 bytes)
18/10/21 12:39:03 INFO TaskSetManager: Finished task 166.0 in stage 1.0 (TID 193) in 31144 ms on node5.dat (executor 4) (146/1497)
18/10/21 12:39:05 INFO TaskSetManager: Starting task 170.0 in stage 1.0 (TID 197, node2.dat, executor 1, partition 170, RACK_LOCAL, 13519 bytes)
18/10/21 12:39:05 INFO TaskSetManager: Finished task 167.0 in stage 1.0 (TID 194) in 32882 ms on node2.dat (executor 1) (147/1497)
18/10/21 12:39:17 INFO TaskSetManager: Starting task 171.0 in stage 1.0 (TID 198, node1.dat, executor 3, partition 171, RACK_LOCAL, 13519 bytes)
18/10/21 12:39:17 INFO TaskSetManager: Finished task 80.0 in stage 1.0 (TID 107) in 1370537 ms on node1.dat (executor 3) (148/1497)
18/10/21 12:39:29 INFO TaskSetManager: Starting task 172.0 in stage 1.0 (TID 199, node1.dat, executor 3, partition 172, RACK_LOCAL, 13519 bytes)
18/10/21 12:39:29 INFO TaskSetManager: Finished task 168.0 in stage 1.0 (TID 195) in 34018 ms on node1.dat (executor 3) (149/1497)
18/10/21 12:39:33 INFO TaskSetManager: Starting task 173.0 in stage 1.0 (TID 200, node2.dat, executor 1, partition 173, RACK_LOCAL, 13519 bytes)
18/10/21 12:39:33 INFO TaskSetManager: Finished task 170.0 in stage 1.0 (TID 197) in 28749 ms on node2.dat (executor 1) (150/1497)
18/10/21 12:39:40 INFO BlockManagerInfo: Removed broadcast_0_piece0 on node7.dat:34097 in memory (size: 5.5 KB, free: 366.3 MB)
18/10/21 12:39:40 INFO BlockManagerInfo: Removed broadcast_0_piece0 on node1.dat:42919 in memory (size: 5.5 KB, free: 366.3 MB)
18/10/21 12:39:40 INFO BlockManagerInfo: Removed broadcast_0_piece0 on node2.dat:40882 in memory (size: 5.5 KB, free: 366.3 MB)
18/10/21 12:39:40 INFO BlockManagerInfo: Removed broadcast_0_piece0 on node5.dat:46826 in memory (size: 5.5 KB, free: 366.3 MB)
18/10/21 12:39:40 INFO BlockManagerInfo: Removed broadcast_0_piece0 on node3.dat:36391 in memory (size: 5.5 KB, free: 366.3 MB)
18/10/21 12:39:40 INFO BlockManagerInfo: Removed broadcast_0_piece0 on node4.dat:38904 in memory (size: 5.5 KB, free: 366.3 MB)
18/10/21 12:39:40 INFO BlockManagerInfo: Removed broadcast_0_piece0 on node6.dat:34166 in memory (size: 5.5 KB, free: 366.3 MB)
18/10/21 12:40:05 INFO TaskSetManager: Starting task 174.0 in stage 1.0 (TID 201, node1.dat, executor 3, partition 174, RACK_LOCAL, 13519 bytes)
18/10/21 12:40:05 INFO TaskSetManager: Finished task 171.0 in stage 1.0 (TID 198) in 48191 ms on node1.dat (executor 3) (151/1497)
18/10/21 12:40:05 INFO TaskSetManager: Starting task 175.0 in stage 1.0 (TID 202, node1.dat, executor 3, partition 175, RACK_LOCAL, 13519 bytes)
18/10/21 12:40:05 INFO TaskSetManager: Finished task 172.0 in stage 1.0 (TID 199) in 35958 ms on node1.dat (executor 3) (152/1497)
18/10/21 12:40:07 INFO TaskSetManager: Starting task 176.0 in stage 1.0 (TID 203, node2.dat, executor 1, partition 176, RACK_LOCAL, 13519 bytes)
18/10/21 12:40:07 INFO TaskSetManager: Finished task 173.0 in stage 1.0 (TID 200) in 33296 ms on node2.dat (executor 1) (153/1497)
18/10/21 12:40:16 INFO TaskSetManager: Starting task 177.0 in stage 1.0 (TID 204, node5.dat, executor 4, partition 177, RACK_LOCAL, 13519 bytes)
18/10/21 12:40:16 INFO TaskSetManager: Finished task 169.0 in stage 1.0 (TID 196) in 73972 ms on node5.dat (executor 4) (154/1497)
18/10/21 12:40:38 INFO TaskSetManager: Starting task 178.0 in stage 1.0 (TID 205, node1.dat, executor 3, partition 178, RACK_LOCAL, 13519 bytes)
18/10/21 12:40:38 INFO TaskSetManager: Finished task 174.0 in stage 1.0 (TID 201) in 33687 ms on node1.dat (executor 3) (155/1497)
18/10/21 12:40:49 INFO TaskSetManager: Starting task 179.0 in stage 1.0 (TID 206, node2.dat, executor 1, partition 179, RACK_LOCAL, 13519 bytes)
18/10/21 12:40:49 INFO TaskSetManager: Finished task 176.0 in stage 1.0 (TID 203) in 41942 ms on node2.dat (executor 1) (156/1497)
18/10/21 12:40:51 INFO TaskSetManager: Starting task 180.0 in stage 1.0 (TID 207, node5.dat, executor 4, partition 180, RACK_LOCAL, 13519 bytes)
18/10/21 12:40:51 INFO TaskSetManager: Finished task 177.0 in stage 1.0 (TID 204) in 34235 ms on node5.dat (executor 4) (157/1497)
18/10/21 12:40:52 INFO TaskSetManager: Starting task 181.0 in stage 1.0 (TID 208, node1.dat, executor 3, partition 181, RACK_LOCAL, 13519 bytes)
18/10/21 12:40:52 INFO TaskSetManager: Finished task 175.0 in stage 1.0 (TID 202) in 46490 ms on node1.dat (executor 3) (158/1497)
18/10/21 12:41:08 INFO TaskSetManager: Starting task 182.0 in stage 1.0 (TID 209, node1.dat, executor 3, partition 182, RACK_LOCAL, 13519 bytes)
18/10/21 12:41:08 INFO TaskSetManager: Finished task 178.0 in stage 1.0 (TID 205) in 29286 ms on node1.dat (executor 3) (159/1497)
18/10/21 12:41:24 INFO TaskSetManager: Starting task 183.0 in stage 1.0 (TID 210, node2.dat, executor 1, partition 183, RACK_LOCAL, 13519 bytes)
18/10/21 12:41:24 INFO TaskSetManager: Finished task 179.0 in stage 1.0 (TID 206) in 35669 ms on node2.dat (executor 1) (160/1497)
18/10/21 12:41:28 INFO TaskSetManager: Starting task 184.0 in stage 1.0 (TID 211, node1.dat, executor 3, partition 184, RACK_LOCAL, 13519 bytes)
18/10/21 12:41:28 INFO TaskSetManager: Finished task 181.0 in stage 1.0 (TID 208) in 35800 ms on node1.dat (executor 3) (161/1497)
18/10/21 12:41:41 INFO TaskSetManager: Starting task 185.0 in stage 1.0 (TID 212, node1.dat, executor 3, partition 185, RACK_LOCAL, 13519 bytes)
18/10/21 12:41:41 INFO TaskSetManager: Finished task 182.0 in stage 1.0 (TID 209) in 33506 ms on node1.dat (executor 3) (162/1497)
18/10/21 12:42:19 INFO TaskSetManager: Starting task 186.0 in stage 1.0 (TID 213, node1.dat, executor 3, partition 186, RACK_LOCAL, 13519 bytes)
18/10/21 12:42:19 INFO TaskSetManager: Finished task 184.0 in stage 1.0 (TID 211) in 51776 ms on node1.dat (executor 3) (163/1497)
18/10/21 12:42:36 INFO TaskSetManager: Starting task 187.0 in stage 1.0 (TID 214, node1.dat, executor 3, partition 187, RACK_LOCAL, 13519 bytes)
18/10/21 12:42:36 INFO TaskSetManager: Finished task 185.0 in stage 1.0 (TID 212) in 55025 ms on node1.dat (executor 3) (164/1497)
18/10/21 12:42:52 INFO TaskSetManager: Starting task 188.0 in stage 1.0 (TID 215, node1.dat, executor 3, partition 188, RACK_LOCAL, 13519 bytes)
18/10/21 12:42:52 INFO TaskSetManager: Finished task 94.0 in stage 1.0 (TID 121) in 1520200 ms on node1.dat (executor 3) (165/1497)
18/10/21 12:42:52 INFO TaskSetManager: Starting task 189.0 in stage 1.0 (TID 216, node3.dat, executor 5, partition 189, RACK_LOCAL, 13519 bytes)
18/10/21 12:42:52 INFO TaskSetManager: Finished task 29.0 in stage 1.0 (TID 56) in 1795929 ms on node3.dat (executor 5) (166/1497)
18/10/21 12:43:11 INFO TaskSetManager: Starting task 190.0 in stage 1.0 (TID 217, node1.dat, executor 3, partition 190, RACK_LOCAL, 13519 bytes)
18/10/21 12:43:11 INFO TaskSetManager: Finished task 186.0 in stage 1.0 (TID 213) in 51263 ms on node1.dat (executor 3) (167/1497)
18/10/21 12:43:42 INFO TaskSetManager: Starting task 191.0 in stage 1.0 (TID 218, node1.dat, executor 3, partition 191, RACK_LOCAL, 13519 bytes)
18/10/21 12:43:42 INFO TaskSetManager: Finished task 188.0 in stage 1.0 (TID 215) in 50003 ms on node1.dat (executor 3) (168/1497)
18/10/21 12:44:27 INFO TaskSetManager: Starting task 192.0 in stage 1.0 (TID 219, node1.dat, executor 3, partition 192, RACK_LOCAL, 13519 bytes)
18/10/21 12:44:27 INFO TaskSetManager: Finished task 190.0 in stage 1.0 (TID 217) in 76112 ms on node1.dat (executor 3) (169/1497)
.
.
.
