Spark Executor Command: "/usr/java8/jdk1.8.0_202/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx10240M" "-Dspark.driver.port=45295" "-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-XX:+HeapDumpOnOutOfMemoryError" "-Xloggc:/tmp/memLog.txt" "-XX:+UseCompressedOops" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:45295" "--executor-id" "3" "--hostname" "5.161.188.154" "--cores" "5" "--app-id" "app-20230924212957-0005" "--worker-url" "spark://Worker@5.161.188.154:46635"
========================================

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
23/09/24 21:31:10 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 266745@entity-resolution
23/09/24 21:31:10 INFO SignalUtils: Registering signal handler for TERM
23/09/24 21:31:10 INFO SignalUtils: Registering signal handler for HUP
23/09/24 21:31:10 INFO SignalUtils: Registering signal handler for INT
23/09/24 21:31:10 WARN Utils: Your hostname, entity-resolution resolves to a loopback address: 127.0.1.1; using 5.161.188.154 instead (on interface eth0)
23/09/24 21:31:10 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
23/09/24 21:31:11 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/09/24 21:31:11 INFO SecurityManager: Changing view acls to: root
23/09/24 21:31:11 INFO SecurityManager: Changing modify acls to: root
23/09/24 21:31:11 INFO SecurityManager: Changing view acls groups to: 
23/09/24 21:31:11 INFO SecurityManager: Changing modify acls groups to: 
23/09/24 21:31:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
23/09/24 21:31:11 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:45295 after 69 ms (0 ms spent in bootstraps)
23/09/24 21:31:11 INFO SecurityManager: Changing view acls to: root
23/09/24 21:31:11 INFO SecurityManager: Changing modify acls to: root
23/09/24 21:31:11 INFO SecurityManager: Changing view acls groups to: 
23/09/24 21:31:11 INFO SecurityManager: Changing modify acls groups to: 
23/09/24 21:31:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
23/09/24 21:31:11 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:45295 after 1 ms (0 ms spent in bootstraps)
23/09/24 21:31:11 INFO DiskBlockManager: Created local directory at /tmp/spark-b5306174-a655-4af8-962d-ed2dcefb130b/executor-9d03ff6a-c42a-4483-b16e-9d5949c0888e/blockmgr-e4cf6662-4c18-423b-9540-d44084dc64d5
23/09/24 21:31:11 INFO MemoryStore: MemoryStore started with capacity 5.2 GiB
23/09/24 21:31:12 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:45295
23/09/24 21:31:12 INFO WorkerWatcher: Connecting to worker spark://Worker@5.161.188.154:46635
23/09/24 21:31:12 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:46635 after 1 ms (0 ms spent in bootstraps)
23/09/24 21:31:12 INFO ResourceUtils: ==============================================================
23/09/24 21:31:12 INFO ResourceUtils: No custom resources configured for spark.executor.
23/09/24 21:31:12 INFO ResourceUtils: ==============================================================
23/09/24 21:31:12 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
23/09/24 21:31:12 INFO Executor: Starting executor ID 3 on host 5.161.188.154
23/09/24 21:31:12 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38099.
23/09/24 21:31:12 INFO NettyBlockTransferService: Server created on 5.161.188.154:38099
23/09/24 21:31:12 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/09/24 21:31:12 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(3, 5.161.188.154, 38099, None)
23/09/24 21:31:12 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(3, 5.161.188.154, 38099, None)
23/09/24 21:31:12 INFO BlockManager: Initialized BlockManager: BlockManagerId(3, 5.161.188.154, 38099, None)
23/09/24 21:31:12 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:45295/jars/zingg-0.3.4-SNAPSHOT.jar with timestamp 1695590996527
23/09/24 21:31:12 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:45295 after 1 ms (0 ms spent in bootstraps)
23/09/24 21:31:12 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:45295/jars/zingg-0.3.4-SNAPSHOT.jar to /tmp/spark-b5306174-a655-4af8-962d-ed2dcefb130b/executor-9d03ff6a-c42a-4483-b16e-9d5949c0888e/spark-e561401b-79f9-433f-a429-bd83405f88cc/fetchFileTemp7160761862292983977.tmp
23/09/24 21:31:12 INFO Utils: Copying /tmp/spark-b5306174-a655-4af8-962d-ed2dcefb130b/executor-9d03ff6a-c42a-4483-b16e-9d5949c0888e/spark-e561401b-79f9-433f-a429-bd83405f88cc/8061961251695590996527_cache to /opt/spark/work/app-20230924212957-0005/3/./zingg-0.3.4-SNAPSHOT.jar
23/09/24 21:31:12 INFO Executor: Adding file:/opt/spark/work/app-20230924212957-0005/3/./zingg-0.3.4-SNAPSHOT.jar to class loader
23/09/24 21:31:12 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:45295/jars/postgresql-42.6.0.jar with timestamp 1695590996527
23/09/24 21:31:12 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:45295/jars/postgresql-42.6.0.jar to /tmp/spark-b5306174-a655-4af8-962d-ed2dcefb130b/executor-9d03ff6a-c42a-4483-b16e-9d5949c0888e/spark-e561401b-79f9-433f-a429-bd83405f88cc/fetchFileTemp965660791987355377.tmp
23/09/24 21:31:12 INFO Utils: Copying /tmp/spark-b5306174-a655-4af8-962d-ed2dcefb130b/executor-9d03ff6a-c42a-4483-b16e-9d5949c0888e/spark-e561401b-79f9-433f-a429-bd83405f88cc/17346205461695590996527_cache to /opt/spark/work/app-20230924212957-0005/3/./postgresql-42.6.0.jar
23/09/24 21:31:12 INFO Executor: Adding file:/opt/spark/work/app-20230924212957-0005/3/./postgresql-42.6.0.jar to class loader
23/09/24 21:31:12 INFO CoarseGrainedExecutorBackend: Got assigned task 461
23/09/24 21:31:12 INFO CoarseGrainedExecutorBackend: Got assigned task 462
23/09/24 21:31:12 INFO CoarseGrainedExecutorBackend: Got assigned task 463
23/09/24 21:31:12 INFO CoarseGrainedExecutorBackend: Got assigned task 464
23/09/24 21:31:12 INFO CoarseGrainedExecutorBackend: Got assigned task 465
23/09/24 21:31:12 INFO Executor: Running task 22.1 in stage 3.0 (TID 462)
23/09/24 21:31:12 INFO Executor: Running task 58.1 in stage 3.0 (TID 463)
23/09/24 21:31:12 INFO Executor: Running task 61.1 in stage 3.0 (TID 461)
23/09/24 21:31:12 INFO Executor: Running task 94.1 in stage 3.0 (TID 465)
23/09/24 21:31:12 INFO Executor: Running task 76.1 in stage 3.0 (TID 464)
23/09/24 21:31:12 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache
23/09/24 21:31:12 INFO TorrentBroadcast: Started reading broadcast variable 6 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:12 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:46709 after 1 ms (0 ms spent in bootstraps)
23/09/24 21:31:12 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 26.8 KiB, free 5.2 GiB)
23/09/24 21:31:12 INFO TorrentBroadcast: Reading broadcast variable 6 took 88 ms
23/09/24 21:31:13 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 92.6 KiB, free 5.2 GiB)
23/09/24 21:31:14 INFO CodeGenerator: Code generated in 427.71059 ms
23/09/24 21:31:16 INFO CodeGenerator: Code generated in 108.945573 ms
23/09/24 21:31:16 INFO CodeGenerator: Code generated in 42.066556 ms
23/09/24 21:31:16 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 7784628224-7918845952, partition values: [empty row]
23/09/24 21:31:16 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 2952790016-3087007744, partition values: [empty row]
23/09/24 21:31:16 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 10200547328-10334765056, partition values: [empty row]
23/09/24 21:31:16 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 12616466432-12750684160, partition values: [empty row]
23/09/24 21:31:16 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 8187281408-8321499136, partition values: [empty row]
23/09/24 21:31:16 INFO CodeGenerator: Code generated in 40.848358 ms
23/09/24 21:31:16 INFO TorrentBroadcast: Started reading broadcast variable 5 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:16 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 24.8 KiB, free 5.2 GiB)
23/09/24 21:31:16 INFO TorrentBroadcast: Reading broadcast variable 5 took 7 ms
23/09/24 21:31:16 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 394.6 KiB, free 5.2 GiB)
23/09/24 21:31:16 INFO CodeGenerator: Code generated in 169.058391 ms
23/09/24 21:31:21 INFO MemoryStore: Block rdd_22_58 stored as values in memory (estimated size 116.8 MiB, free 5.0 GiB)
23/09/24 21:31:21 INFO CodeGenerator: Code generated in 20.354217 ms
23/09/24 21:31:21 INFO CodeGenerator: Code generated in 17.072204 ms
23/09/24 21:31:21 INFO CodeGenerator: Code generated in 6.897024 ms
23/09/24 21:31:21 INFO Executor: Finished task 58.1 in stage 3.0 (TID 463). 2495 bytes result sent to driver
23/09/24 21:31:21 INFO CoarseGrainedExecutorBackend: Got assigned task 494
23/09/24 21:31:21 INFO Executor: Running task 24.1 in stage 3.0 (TID 494)
23/09/24 21:31:21 INFO MemoryStore: Block rdd_22_76 stored as values in memory (estimated size 116.8 MiB, free 4.9 GiB)
23/09/24 21:31:21 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 3221225472-3355443200, partition values: [empty row]
23/09/24 21:31:21 INFO Executor: Finished task 76.1 in stage 3.0 (TID 464). 2452 bytes result sent to driver
23/09/24 21:31:21 INFO CoarseGrainedExecutorBackend: Got assigned task 496
23/09/24 21:31:21 INFO Executor: Running task 39.1 in stage 3.0 (TID 496)
23/09/24 21:31:21 INFO MemoryStore: Block rdd_22_94 stored as values in memory (estimated size 116.6 MiB, free 4.8 GiB)
23/09/24 21:31:21 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 5234491392-5368709120, partition values: [empty row]
23/09/24 21:31:21 INFO Executor: Finished task 94.1 in stage 3.0 (TID 465). 2452 bytes result sent to driver
23/09/24 21:31:21 INFO CoarseGrainedExecutorBackend: Got assigned task 497
23/09/24 21:31:21 INFO Executor: Running task 120.1 in stage 3.0 (TID 497)
23/09/24 21:31:21 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 16106127360-16240345088, partition values: [empty row]
23/09/24 21:31:21 INFO MemoryStore: Block rdd_22_22 stored as values in memory (estimated size 119.4 MiB, free 4.7 GiB)
23/09/24 21:31:21 INFO MemoryStore: Block rdd_22_61 stored as values in memory (estimated size 116.8 MiB, free 4.6 GiB)
23/09/24 21:31:21 INFO Executor: Finished task 22.1 in stage 3.0 (TID 462). 2452 bytes result sent to driver
23/09/24 21:31:21 INFO CoarseGrainedExecutorBackend: Got assigned task 498
23/09/24 21:31:21 INFO Executor: Running task 75.1 in stage 3.0 (TID 498)
23/09/24 21:31:21 INFO Executor: Finished task 61.1 in stage 3.0 (TID 461). 2452 bytes result sent to driver
23/09/24 21:31:21 INFO CoarseGrainedExecutorBackend: Got assigned task 499
23/09/24 21:31:21 INFO Executor: Running task 93.1 in stage 3.0 (TID 499)
23/09/24 21:31:22 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 10066329600-10200547328, partition values: [empty row]
23/09/24 21:31:22 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 12482248704-12616466432, partition values: [empty row]
23/09/24 21:31:25 INFO MemoryStore: Block rdd_22_39 stored as values in memory (estimated size 116.7 MiB, free 4.5 GiB)
23/09/24 21:31:25 INFO Executor: Finished task 39.1 in stage 3.0 (TID 496). 2452 bytes result sent to driver
23/09/24 21:31:25 INFO MemoryStore: Block rdd_22_120 stored as values in memory (estimated size 116.9 MiB, free 4.4 GiB)
23/09/24 21:31:25 INFO Executor: Finished task 120.1 in stage 3.0 (TID 497). 2452 bytes result sent to driver
23/09/24 21:31:25 INFO MemoryStore: Block rdd_22_75 stored as values in memory (estimated size 116.8 MiB, free 4.2 GiB)
23/09/24 21:31:25 INFO Executor: Finished task 75.1 in stage 3.0 (TID 498). 2452 bytes result sent to driver
23/09/24 21:31:25 INFO MemoryStore: Block rdd_22_93 stored as values in memory (estimated size 116.7 MiB, free 4.1 GiB)
23/09/24 21:31:25 INFO Executor: Finished task 93.1 in stage 3.0 (TID 499). 2452 bytes result sent to driver
23/09/24 21:31:25 INFO MemoryStore: Block rdd_22_24 stored as values in memory (estimated size 118.9 MiB, free 4.0 GiB)
23/09/24 21:31:26 INFO Executor: Finished task 24.1 in stage 3.0 (TID 494). 2452 bytes result sent to driver
23/09/24 21:31:26 INFO CoarseGrainedExecutorBackend: Got assigned task 502
23/09/24 21:31:26 INFO Executor: Running task 0.0 in stage 5.0 (TID 502)
23/09/24 21:31:26 INFO MapOutputTrackerWorker: Updating epoch to 3 and clearing cache
23/09/24 21:31:26 INFO TorrentBroadcast: Started reading broadcast variable 8 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:26 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:37905 after 1 ms (0 ms spent in bootstraps)
23/09/24 21:31:26 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 27.0 KiB, free 4.0 GiB)
23/09/24 21:31:26 INFO TorrentBroadcast: Reading broadcast variable 8 took 9 ms
23/09/24 21:31:26 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 75.0 KiB, free 4.0 GiB)
23/09/24 21:31:26 INFO Executor: Finished task 0.0 in stage 5.0 (TID 502). 1599 bytes result sent to driver
23/09/24 21:31:26 INFO CoarseGrainedExecutorBackend: Got assigned task 504
23/09/24 21:31:26 INFO Executor: Running task 1.0 in stage 6.0 (TID 504)
23/09/24 21:31:26 INFO TorrentBroadcast: Started reading broadcast variable 10 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:26 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 7.0 KiB, free 4.0 GiB)
23/09/24 21:31:26 INFO TorrentBroadcast: Reading broadcast variable 10 took 6 ms
23/09/24 21:31:26 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 14.7 KiB, free 4.0 GiB)
23/09/24 21:31:26 INFO CodeGenerator: Code generated in 8.175608 ms
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00049-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4280, partition values: [empty row]
23/09/24 21:31:26 INFO TorrentBroadcast: Started reading broadcast variable 9 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:26 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 25.1 KiB, free 4.0 GiB)
23/09/24 21:31:26 INFO TorrentBroadcast: Reading broadcast variable 9 took 7 ms
23/09/24 21:31:26 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 398.2 KiB, free 4.0 GiB)
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00019-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4211, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00131-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4138, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00043-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4128, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00035-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4116, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00123-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4104, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00027-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4103, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00092-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4099, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00012-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4098, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00085-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4098, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00052-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4095, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00116-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4090, partition values: [empty row]
23/09/24 21:31:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00032-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4080, partition values: [empty row]
23/09/24 21:31:26 INFO Executor: Finished task 1.0 in stage 6.0 (TID 504). 2058 bytes result sent to driver
23/09/24 21:31:26 INFO CoarseGrainedExecutorBackend: Got assigned task 506
23/09/24 21:31:26 INFO Executor: Running task 0.0 in stage 7.0 (TID 506)
23/09/24 21:31:26 INFO MapOutputTrackerWorker: Updating epoch to 4 and clearing cache
23/09/24 21:31:26 INFO TorrentBroadcast: Started reading broadcast variable 11 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:26 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 4.0 GiB)
23/09/24 21:31:26 INFO TorrentBroadcast: Reading broadcast variable 11 took 7 ms
23/09/24 21:31:26 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 10.1 KiB, free 4.0 GiB)
23/09/24 21:31:26 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 2, fetching them
23/09/24 21:31:26 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:45295)
23/09/24 21:31:27 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:31:27 INFO ShuffleBlockFetcherIterator: Getting 3 (180.0 B) non-empty blocks including 1 (60.0 B) local and 2 (120.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:31:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 13 ms
23/09/24 21:31:27 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:42443 after 2 ms (0 ms spent in bootstraps)
23/09/24 21:31:27 INFO CodeGenerator: Code generated in 6.298464 ms
23/09/24 21:31:27 INFO Executor: Finished task 0.0 in stage 7.0 (TID 506). 2450 bytes result sent to driver
23/09/24 21:31:27 INFO CoarseGrainedExecutorBackend: Got assigned task 508
23/09/24 21:31:27 INFO Executor: Running task 1.0 in stage 8.0 (TID 508)
23/09/24 21:31:27 INFO TorrentBroadcast: Started reading broadcast variable 13 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:27 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 4.0 GiB)
23/09/24 21:31:27 INFO TorrentBroadcast: Reading broadcast variable 13 took 9 ms
23/09/24 21:31:27 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 29.8 KiB, free 4.0 GiB)
23/09/24 21:31:27 INFO CodeGenerator: Code generated in 11.232716 ms
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00049-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4280, partition values: [empty row]
23/09/24 21:31:27 INFO TorrentBroadcast: Started reading broadcast variable 12 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:27 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 25.8 KiB, free 4.0 GiB)
23/09/24 21:31:27 INFO TorrentBroadcast: Reading broadcast variable 12 took 6 ms
23/09/24 21:31:27 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 398.2 KiB, free 4.0 GiB)
23/09/24 21:31:27 INFO CodecPool: Got brand-new decompressor [.snappy]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00019-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4211, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00131-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4138, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00043-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4128, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00035-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4116, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00123-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4104, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00027-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4103, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00092-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4099, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00012-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4098, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00085-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4098, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00052-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4095, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00116-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4090, partition values: [empty row]
23/09/24 21:31:27 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00032-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4080, partition values: [empty row]
23/09/24 21:31:27 INFO MemoryStore: Block rdd_44_1 stored as values in memory (estimated size 5.9 KiB, free 4.0 GiB)
23/09/24 21:31:27 INFO CodeGenerator: Code generated in 3.74675 ms
23/09/24 21:31:27 INFO CodeGenerator: Code generated in 12.062 ms
23/09/24 21:31:27 INFO CodeGenerator: Code generated in 6.665847 ms
23/09/24 21:31:27 INFO Executor: Finished task 1.0 in stage 8.0 (TID 508). 2131 bytes result sent to driver
23/09/24 21:31:28 INFO CoarseGrainedExecutorBackend: Got assigned task 510
23/09/24 21:31:28 INFO Executor: Running task 1.0 in stage 9.0 (TID 510)
23/09/24 21:31:28 INFO TorrentBroadcast: Started reading broadcast variable 15 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 4.0 GiB)
23/09/24 21:31:28 INFO TorrentBroadcast: Reading broadcast variable 15 took 6 ms
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 36.8 KiB, free 4.0 GiB)
23/09/24 21:31:28 INFO BlockManager: Found block rdd_44_1 locally
23/09/24 21:31:28 INFO CodeGenerator: Code generated in 6.711925 ms
23/09/24 21:31:28 INFO CodeGenerator: Code generated in 11.258033 ms
23/09/24 21:31:28 INFO CodeGenerator: Code generated in 19.750748 ms
23/09/24 21:31:28 INFO TorrentBroadcast: Started reading broadcast variable 14 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 1669.0 B, free 4.0 GiB)
23/09/24 21:31:28 INFO TorrentBroadcast: Reading broadcast variable 14 took 6 ms
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 64.0 MiB, free 3.9 GiB)
23/09/24 21:31:28 INFO Executor: Finished task 1.0 in stage 9.0 (TID 510). 2435 bytes result sent to driver
23/09/24 21:31:28 INFO CoarseGrainedExecutorBackend: Got assigned task 516
23/09/24 21:31:28 INFO Executor: Running task 1.0 in stage 11.0 (TID 516)
23/09/24 21:31:28 INFO MapOutputTrackerWorker: Updating epoch to 5 and clearing cache
23/09/24 21:31:28 INFO TorrentBroadcast: Started reading broadcast variable 17 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 10.5 KiB, free 3.9 GiB)
23/09/24 21:31:28 INFO TorrentBroadcast: Reading broadcast variable 17 took 7 ms
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 30.5 KiB, free 3.9 GiB)
23/09/24 21:31:28 INFO BlockManager: Found block rdd_44_1 locally
23/09/24 21:31:28 INFO Executor: Finished task 1.0 in stage 11.0 (TID 516). 2131 bytes result sent to driver
23/09/24 21:31:28 INFO CoarseGrainedExecutorBackend: Got assigned task 517
23/09/24 21:31:28 INFO Executor: Running task 1.0 in stage 12.0 (TID 517)
23/09/24 21:31:28 INFO TorrentBroadcast: Started reading broadcast variable 19 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 3.9 GiB)
23/09/24 21:31:28 INFO TorrentBroadcast: Reading broadcast variable 19 took 6 ms
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 36.8 KiB, free 3.9 GiB)
23/09/24 21:31:28 INFO BlockManager: Found block rdd_44_1 locally
23/09/24 21:31:28 INFO CodeGenerator: Code generated in 5.89847 ms
23/09/24 21:31:28 INFO CodeGenerator: Code generated in 9.073281 ms
23/09/24 21:31:28 INFO TorrentBroadcast: Started reading broadcast variable 18 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 1669.0 B, free 3.9 GiB)
23/09/24 21:31:28 INFO TorrentBroadcast: Reading broadcast variable 18 took 25 ms
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 64.0 MiB, free 3.9 GiB)
23/09/24 21:31:28 INFO Executor: Finished task 1.0 in stage 12.0 (TID 517). 2435 bytes result sent to driver
23/09/24 21:31:28 INFO CoarseGrainedExecutorBackend: Got assigned task 522
23/09/24 21:31:28 INFO Executor: Running task 1.0 in stage 14.0 (TID 522)
23/09/24 21:31:28 INFO MapOutputTrackerWorker: Updating epoch to 6 and clearing cache
23/09/24 21:31:28 INFO TorrentBroadcast: Started reading broadcast variable 21 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 10.5 KiB, free 3.9 GiB)
23/09/24 21:31:28 INFO TorrentBroadcast: Reading broadcast variable 21 took 5 ms
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 30.5 KiB, free 3.9 GiB)
23/09/24 21:31:28 INFO BlockManager: Found block rdd_44_1 locally
23/09/24 21:31:28 INFO Executor: Finished task 1.0 in stage 14.0 (TID 522). 2131 bytes result sent to driver
23/09/24 21:31:28 INFO CoarseGrainedExecutorBackend: Got assigned task 525
23/09/24 21:31:28 INFO Executor: Running task 1.0 in stage 15.0 (TID 525)
23/09/24 21:31:28 INFO TorrentBroadcast: Started reading broadcast variable 23 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 3.9 GiB)
23/09/24 21:31:28 INFO TorrentBroadcast: Reading broadcast variable 23 took 5 ms
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_23 stored as values in memory (estimated size 36.8 KiB, free 3.9 GiB)
23/09/24 21:31:28 INFO BlockManager: Found block rdd_44_1 locally
23/09/24 21:31:28 INFO TorrentBroadcast: Started reading broadcast variable 22 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 1669.0 B, free 3.9 GiB)
23/09/24 21:31:28 INFO TorrentBroadcast: Reading broadcast variable 22 took 6 ms
23/09/24 21:31:28 INFO MemoryStore: Block broadcast_22 stored as values in memory (estimated size 64.0 MiB, free 3.8 GiB)
23/09/24 21:31:28 INFO Executor: Finished task 1.0 in stage 15.0 (TID 525). 2435 bytes result sent to driver
23/09/24 21:31:33 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
23/09/24 21:31:33 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
tdown