3.1.3
stderr log page for app-20230924213329-0006/3
Showing 31041 Bytes: 0 - 31041 of 31041
Spark Executor Command: "/usr/java8/jdk1.8.0_202/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx10240M" "-Dspark.driver.port=46637" "-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-XX:+HeapDumpOnOutOfMemoryError" "-Xloggc:/tmp/memLog.txt" "-XX:+UseCompressedOops" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:46637" "--executor-id" "3" "--hostname" "5.161.188.154" "--cores" "5" "--app-id" "app-20230924213329-0006" "--worker-url" "spark://Worker@5.161.188.154:36849" ======================================== Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 23/09/24 21:34:43 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 267619@entity-resolution 23/09/24 21:34:43 INFO SignalUtils: Registering signal handler for TERM 23/09/24 21:34:43 INFO SignalUtils: Registering signal handler for HUP 23/09/24 21:34:43 INFO SignalUtils: Registering signal handler for INT 23/09/24 21:34:43 WARN Utils: Your hostname, entity-resolution resolves to a loopback address: 127.0.1.1; using 5.161.188.154 instead (on interface eth0) 23/09/24 21:34:43 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 23/09/24 21:34:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 23/09/24 21:34:44 INFO SecurityManager: Changing view acls to: root 23/09/24 21:34:44 INFO SecurityManager: Changing modify acls to: root 23/09/24 21:34:44 INFO SecurityManager: Changing view acls groups to: 23/09/24 21:34:44 INFO SecurityManager: Changing modify acls groups to: 23/09/24 21:34:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 23/09/24 21:34:44 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:46637 after 73 ms (0 ms spent in bootstraps) 23/09/24 21:34:44 INFO SecurityManager: Changing view acls to: root 23/09/24 21:34:44 INFO SecurityManager: Changing modify acls to: root 23/09/24 21:34:44 INFO SecurityManager: Changing view acls groups to: 23/09/24 21:34:44 INFO SecurityManager: Changing modify acls groups to: 23/09/24 21:34:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 23/09/24 21:34:44 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:46637 after 1 ms (0 ms spent in bootstraps) 23/09/24 21:34:44 INFO DiskBlockManager: Created local directory at /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-51db565d-a47f-48c6-89cc-a104ffa6f752/blockmgr-632b859c-3f34-4509-8704-2cbeb52f8dc3 23/09/24 21:34:44 INFO MemoryStore: MemoryStore started with capacity 5.2 GiB 23/09/24 21:34:45 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:46637 23/09/24 21:34:45 INFO WorkerWatcher: Connecting to worker spark://Worker@5.161.188.154:36849 23/09/24 21:34:45 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:36849 after 1 ms (0 ms spent in bootstraps) 23/09/24 21:34:45 INFO ResourceUtils: ============================================================== 23/09/24 21:34:45 INFO ResourceUtils: No custom resources configured for spark.executor. 23/09/24 21:34:45 INFO ResourceUtils: ============================================================== 23/09/24 21:34:45 INFO WorkerWatcher: Successfully connected to spark://Worker@5.161.188.154:36849 23/09/24 21:34:45 INFO CoarseGrainedExecutorBackend: Successfully registered with driver 23/09/24 21:34:45 INFO Executor: Starting executor ID 3 on host 5.161.188.154 23/09/24 21:34:45 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45879. 23/09/24 21:34:45 INFO NettyBlockTransferService: Server created on 5.161.188.154:45879 23/09/24 21:34:45 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 23/09/24 21:34:45 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(3, 5.161.188.154, 45879, None) 23/09/24 21:34:45 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(3, 5.161.188.154, 45879, None) 23/09/24 21:34:45 INFO BlockManager: Initialized BlockManager: BlockManagerId(3, 5.161.188.154, 45879, None) 23/09/24 21:34:45 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:46637/jars/zingg-0.3.4-SNAPSHOT.jar with timestamp 1695591208014 23/09/24 21:34:45 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:46637 after 1 ms (0 ms spent in bootstraps) 23/09/24 21:34:45 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:46637/jars/zingg-0.3.4-SNAPSHOT.jar to /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-51db565d-a47f-48c6-89cc-a104ffa6f752/spark-76f5b227-d963-4314-979f-575388588d42/fetchFileTemp6472622663099201418.tmp 23/09/24 21:34:45 INFO Utils: Copying /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-51db565d-a47f-48c6-89cc-a104ffa6f752/spark-76f5b227-d963-4314-979f-575388588d42/15642360401695591208014_cache to /opt/spark/work/app-20230924213329-0006/3/./zingg-0.3.4-SNAPSHOT.jar 23/09/24 21:34:45 INFO Executor: Adding file:/opt/spark/work/app-20230924213329-0006/3/./zingg-0.3.4-SNAPSHOT.jar to class loader 23/09/24 21:34:45 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:46637/jars/postgresql-42.6.0.jar with timestamp 1695591208014 23/09/24 21:34:45 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:46637/jars/postgresql-42.6.0.jar to /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-51db565d-a47f-48c6-89cc-a104ffa6f752/spark-76f5b227-d963-4314-979f-575388588d42/fetchFileTemp6882310875512236074.tmp 23/09/24 21:34:45 INFO Utils: Copying /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-51db565d-a47f-48c6-89cc-a104ffa6f752/spark-76f5b227-d963-4314-979f-575388588d42/17575690151695591208014_cache to /opt/spark/work/app-20230924213329-0006/3/./postgresql-42.6.0.jar 23/09/24 21:34:45 INFO Executor: Adding file:/opt/spark/work/app-20230924213329-0006/3/./postgresql-42.6.0.jar to class loader 23/09/24 21:34:45 INFO CoarseGrainedExecutorBackend: Got assigned task 461 23/09/24 21:34:45 INFO CoarseGrainedExecutorBackend: Got assigned task 462 23/09/24 21:34:45 INFO CoarseGrainedExecutorBackend: Got assigned task 463 23/09/24 21:34:45 INFO CoarseGrainedExecutorBackend: Got assigned task 464 23/09/24 21:34:45 INFO CoarseGrainedExecutorBackend: Got assigned task 465 23/09/24 21:34:45 INFO Executor: Running task 46.1 in stage 3.0 (TID 462) 23/09/24 21:34:45 INFO Executor: Running task 106.1 in stage 3.0 (TID 463) 23/09/24 21:34:45 INFO Executor: Running task 7.1 in stage 3.0 (TID 465) 23/09/24 21:34:45 INFO Executor: Running task 79.1 in stage 3.0 (TID 464) 23/09/24 21:34:45 INFO Executor: Running task 37.1 in stage 3.0 (TID 461) 23/09/24 21:34:45 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache 23/09/24 21:34:45 INFO TorrentBroadcast: Started reading broadcast variable 6 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:34:45 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:45925 after 1 ms (0 ms spent in bootstraps) 23/09/24 21:34:45 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 26.8 KiB, free 5.2 GiB) 23/09/24 21:34:45 INFO TorrentBroadcast: Reading broadcast variable 6 took 73 ms 23/09/24 21:34:46 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 92.6 KiB, free 5.2 GiB) 23/09/24 21:34:47 INFO CodeGenerator: Code generated in 394.352242 ms 23/09/24 21:34:48 INFO CodeGenerator: Code generated in 144.052238 ms 23/09/24 21:34:48 INFO CodeGenerator: Code generated in 36.965971 ms 23/09/24 21:34:49 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 6174015488-6308233216, partition values: [empty row] 23/09/24 21:34:49 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 939524096-1073741824, partition values: [empty row] 23/09/24 21:34:49 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 10603200512-10737418240, partition values: [empty row] 23/09/24 21:34:49 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 4966055936-5100273664, partition values: [empty row] 23/09/24 21:34:49 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 14227079168-14361296896, partition values: [empty row] 23/09/24 21:34:49 INFO CodeGenerator: Code generated in 53.776646 ms 23/09/24 21:34:49 INFO TorrentBroadcast: Started reading broadcast variable 5 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:34:49 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:41347 after 2 ms (0 ms spent in bootstraps) 23/09/24 21:34:49 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 24.8 KiB, free 5.2 GiB) 23/09/24 21:34:49 INFO TorrentBroadcast: Reading broadcast variable 5 took 17 ms 23/09/24 21:34:49 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 394.6 KiB, free 5.2 GiB) 23/09/24 21:34:49 INFO CodeGenerator: Code generated in 180.882412 ms 23/09/24 21:34:54 INFO MemoryStore: Block rdd_22_79 stored as values in memory (estimated size 116.9 MiB, free 5.0 GiB) 23/09/24 21:34:54 INFO CodeGenerator: Code generated in 38.42147 ms 23/09/24 21:34:54 INFO MemoryStore: Block rdd_22_37 stored as values in memory (estimated size 116.8 MiB, free 4.9 GiB) 23/09/24 21:34:54 INFO MemoryStore: Block rdd_22_46 stored as values in memory (estimated size 114.5 MiB, free 4.8 GiB) 23/09/24 21:34:54 INFO CodeGenerator: Code generated in 45.821822 ms 23/09/24 21:34:54 INFO CodeGenerator: Code generated in 21.022483 ms 23/09/24 21:34:54 INFO Executor: Finished task 46.1 in stage 3.0 (TID 462). 2495 bytes result sent to driver 23/09/24 21:34:54 INFO CoarseGrainedExecutorBackend: Got assigned task 490 23/09/24 21:34:54 INFO Executor: Running task 120.1 in stage 3.0 (TID 490) 23/09/24 21:34:54 INFO Executor: Finished task 79.1 in stage 3.0 (TID 464). 2495 bytes result sent to driver 23/09/24 21:34:54 INFO MemoryStore: Block rdd_22_106 stored as values in memory (estimated size 116.9 MiB, free 4.7 GiB) 23/09/24 21:34:54 INFO Executor: Finished task 37.1 in stage 3.0 (TID 461). 2495 bytes result sent to driver 23/09/24 21:34:54 INFO CoarseGrainedExecutorBackend: Got assigned task 491 23/09/24 21:34:54 INFO Executor: Running task 30.1 in stage 3.0 (TID 491) 23/09/24 21:34:54 INFO Executor: Finished task 106.1 in stage 3.0 (TID 463). 2452 bytes result sent to driver 23/09/24 21:34:54 INFO CoarseGrainedExecutorBackend: Got assigned task 492 23/09/24 21:34:54 INFO Executor: Running task 48.1 in stage 3.0 (TID 492) 23/09/24 21:34:54 INFO CoarseGrainedExecutorBackend: Got assigned task 493 23/09/24 21:34:54 INFO Executor: Running task 75.1 in stage 3.0 (TID 493) 23/09/24 21:34:54 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 6442450944-6576668672, partition values: [empty row] 23/09/24 21:34:54 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 16106127360-16240345088, partition values: [empty row] 23/09/24 21:34:54 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 10066329600-10200547328, partition values: [empty row] 23/09/24 21:34:54 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 4026531840-4160749568, partition values: [empty row] 23/09/24 21:34:54 INFO MemoryStore: Block rdd_22_7 stored as values in memory (estimated size 116.9 MiB, free 4.6 GiB) 23/09/24 21:34:55 INFO Executor: Finished task 7.1 in stage 3.0 (TID 465). 2452 bytes result sent to driver 23/09/24 21:34:55 INFO CoarseGrainedExecutorBackend: Got assigned task 495 23/09/24 21:34:55 INFO Executor: Running task 45.1 in stage 3.0 (TID 495) 23/09/24 21:34:55 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/blackice/ubo-20230604-mapped.json, range: 6039797760-6174015488, partition values: [empty row] 23/09/24 21:34:58 INFO MemoryStore: Block rdd_22_30 stored as values in memory (estimated size 117.2 MiB, free 4.5 GiB) 23/09/24 21:34:58 INFO Executor: Finished task 30.1 in stage 3.0 (TID 491). 2452 bytes result sent to driver 23/09/24 21:34:58 INFO MemoryStore: Block rdd_22_45 stored as values in memory (estimated size 107.9 MiB, free 4.4 GiB) 23/09/24 21:34:58 INFO Executor: Finished task 45.1 in stage 3.0 (TID 495). 2452 bytes result sent to driver 23/09/24 21:34:58 INFO MemoryStore: Block rdd_22_75 stored as values in memory (estimated size 116.8 MiB, free 4.3 GiB) 23/09/24 21:34:58 INFO Executor: Finished task 75.1 in stage 3.0 (TID 493). 2452 bytes result sent to driver 23/09/24 21:34:58 INFO MemoryStore: Block rdd_22_48 stored as values in memory (estimated size 114.0 MiB, free 4.1 GiB) 23/09/24 21:34:58 INFO Executor: Finished task 48.1 in stage 3.0 (TID 492). 2452 bytes result sent to driver 23/09/24 21:34:59 INFO MemoryStore: Block rdd_22_120 stored as values in memory (estimated size 116.9 MiB, free 4.0 GiB) 23/09/24 21:34:59 INFO Executor: Finished task 120.1 in stage 3.0 (TID 490). 2452 bytes result sent to driver 23/09/24 21:34:59 INFO CoarseGrainedExecutorBackend: Got assigned task 502 23/09/24 21:34:59 INFO Executor: Running task 0.0 in stage 5.0 (TID 502) 23/09/24 21:34:59 INFO MapOutputTrackerWorker: Updating epoch to 3 and clearing cache 23/09/24 21:34:59 INFO TorrentBroadcast: Started reading broadcast variable 8 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:34:59 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:41733 after 1 ms (0 ms spent in bootstraps) 23/09/24 21:34:59 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 27.0 KiB, free 4.0 GiB) 23/09/24 21:34:59 INFO TorrentBroadcast: Reading broadcast variable 8 took 10 ms 23/09/24 21:34:59 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 75.0 KiB, free 4.0 GiB) 23/09/24 21:34:59 INFO Executor: Finished task 0.0 in stage 5.0 (TID 502). 1599 bytes result sent to driver 23/09/24 21:34:59 INFO CoarseGrainedExecutorBackend: Got assigned task 504 23/09/24 21:34:59 INFO Executor: Running task 1.0 in stage 6.0 (TID 504) 23/09/24 21:34:59 INFO TorrentBroadcast: Started reading broadcast variable 10 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:34:59 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 7.0 KiB, free 4.0 GiB) 23/09/24 21:34:59 INFO TorrentBroadcast: Reading broadcast variable 10 took 7 ms 23/09/24 21:34:59 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 14.7 KiB, free 4.0 GiB) 23/09/24 21:34:59 INFO CodeGenerator: Code generated in 9.546348 ms 23/09/24 21:34:59 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00049-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4280, partition values: [empty row] 23/09/24 21:34:59 INFO TorrentBroadcast: Started reading broadcast variable 9 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:34:59 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 25.1 KiB, free 4.0 GiB) 23/09/24 21:34:59 INFO TorrentBroadcast: Reading broadcast variable 9 took 7 ms 23/09/24 21:34:59 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 398.2 KiB, free 4.0 GiB) 23/09/24 21:34:59 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00019-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4211, partition values: [empty row] 23/09/24 21:34:59 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00131-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4138, partition values: [empty row] 23/09/24 21:34:59 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00043-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4128, partition values: [empty row] 23/09/24 21:34:59 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00035-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4116, partition values: [empty row] 23/09/24 21:34:59 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00123-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4104, partition values: [empty row] 23/09/24 21:34:59 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00027-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4103, partition values: [empty row] 23/09/24 21:34:59 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00092-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4099, partition values: [empty row] 23/09/24 21:34:59 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00012-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4098, partition values: [empty row] 23/09/24 21:34:59 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00085-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4098, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00052-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4095, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00116-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4090, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00032-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4080, partition values: [empty row] 23/09/24 21:35:00 INFO Executor: Finished task 1.0 in stage 6.0 (TID 504). 2058 bytes result sent to driver 23/09/24 21:35:00 INFO CoarseGrainedExecutorBackend: Got assigned task 509 23/09/24 21:35:00 INFO Executor: Running task 2.0 in stage 8.0 (TID 509) 23/09/24 21:35:00 INFO MapOutputTrackerWorker: Updating epoch to 4 and clearing cache 23/09/24 21:35:00 INFO TorrentBroadcast: Started reading broadcast variable 13 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:00 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 4.0 GiB) 23/09/24 21:35:00 INFO TorrentBroadcast: Reading broadcast variable 13 took 6 ms 23/09/24 21:35:00 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 29.8 KiB, free 4.0 GiB) 23/09/24 21:35:00 INFO CodeGenerator: Code generated in 16.005253 ms 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00038-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4078, partition values: [empty row] 23/09/24 21:35:00 INFO TorrentBroadcast: Started reading broadcast variable 12 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:00 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 25.8 KiB, free 4.0 GiB) 23/09/24 21:35:00 INFO TorrentBroadcast: Reading broadcast variable 12 took 6 ms 23/09/24 21:35:00 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 398.2 KiB, free 4.0 GiB) 23/09/24 21:35:00 INFO CodecPool: Got brand-new decompressor [.snappy] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00044-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4078, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00015-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4075, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00009-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4062, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00085-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4057, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00105-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4037, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00004-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4030, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00064-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4018, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00082-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4010, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00071-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4009, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00101-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-3992, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00059-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-3992, partition values: [empty row] 23/09/24 21:35:00 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00000-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-1531, partition values: [empty row] 23/09/24 21:35:00 INFO MemoryStore: Block rdd_44_2 stored as values in memory (estimated size 5.3 KiB, free 4.0 GiB) 23/09/24 21:35:00 INFO CodeGenerator: Code generated in 4.007443 ms 23/09/24 21:35:00 INFO CodeGenerator: Code generated in 11.859049 ms 23/09/24 21:35:00 INFO CodeGenerator: Code generated in 6.563933 ms 23/09/24 21:35:01 INFO Executor: Finished task 2.0 in stage 8.0 (TID 509). 2047 bytes result sent to driver 23/09/24 21:35:01 INFO CoarseGrainedExecutorBackend: Got assigned task 511 23/09/24 21:35:01 INFO Executor: Running task 2.0 in stage 9.0 (TID 511) 23/09/24 21:35:01 INFO TorrentBroadcast: Started reading broadcast variable 15 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 4.0 GiB) 23/09/24 21:35:01 INFO TorrentBroadcast: Reading broadcast variable 15 took 5 ms 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 36.8 KiB, free 4.0 GiB) 23/09/24 21:35:01 INFO BlockManager: Found block rdd_44_2 locally 23/09/24 21:35:01 INFO CodeGenerator: Code generated in 5.75793 ms 23/09/24 21:35:01 INFO CodeGenerator: Code generated in 9.789537 ms 23/09/24 21:35:01 INFO CodeGenerator: Code generated in 14.225901 ms 23/09/24 21:35:01 INFO TorrentBroadcast: Started reading broadcast variable 14 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 1669.0 B, free 4.0 GiB) 23/09/24 21:35:01 INFO TorrentBroadcast: Reading broadcast variable 14 took 6 ms 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 64.0 MiB, free 4.0 GiB) 23/09/24 21:35:01 INFO Executor: Finished task 2.0 in stage 9.0 (TID 511). 2435 bytes result sent to driver 23/09/24 21:35:01 INFO CoarseGrainedExecutorBackend: Got assigned task 514 23/09/24 21:35:01 INFO Executor: Running task 2.0 in stage 11.0 (TID 514) 23/09/24 21:35:01 INFO MapOutputTrackerWorker: Updating epoch to 5 and clearing cache 23/09/24 21:35:01 INFO TorrentBroadcast: Started reading broadcast variable 17 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 10.5 KiB, free 4.0 GiB) 23/09/24 21:35:01 INFO TorrentBroadcast: Reading broadcast variable 17 took 6 ms 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 30.5 KiB, free 4.0 GiB) 23/09/24 21:35:01 INFO BlockManager: Found block rdd_44_2 locally 23/09/24 21:35:01 INFO Executor: Finished task 2.0 in stage 11.0 (TID 514). 2047 bytes result sent to driver 23/09/24 21:35:01 INFO CoarseGrainedExecutorBackend: Got assigned task 519 23/09/24 21:35:01 INFO Executor: Running task 2.0 in stage 12.0 (TID 519) 23/09/24 21:35:01 INFO TorrentBroadcast: Started reading broadcast variable 19 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 4.0 GiB) 23/09/24 21:35:01 INFO TorrentBroadcast: Reading broadcast variable 19 took 5 ms 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 36.8 KiB, free 4.0 GiB) 23/09/24 21:35:01 INFO BlockManager: Found block rdd_44_2 locally 23/09/24 21:35:01 INFO CodeGenerator: Code generated in 4.156964 ms 23/09/24 21:35:01 INFO CodeGenerator: Code generated in 9.282281 ms 23/09/24 21:35:01 INFO TorrentBroadcast: Started reading broadcast variable 18 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 1669.0 B, free 4.0 GiB) 23/09/24 21:35:01 INFO TorrentBroadcast: Reading broadcast variable 18 took 5 ms 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 64.0 MiB, free 3.9 GiB) 23/09/24 21:35:01 INFO Executor: Finished task 2.0 in stage 12.0 (TID 519). 2435 bytes result sent to driver 23/09/24 21:35:01 INFO CoarseGrainedExecutorBackend: Got assigned task 520 23/09/24 21:35:01 INFO Executor: Running task 0.0 in stage 13.0 (TID 520) 23/09/24 21:35:01 INFO MapOutputTrackerWorker: Updating epoch to 6 and clearing cache 23/09/24 21:35:01 INFO TorrentBroadcast: Started reading broadcast variable 20 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 3.9 GiB) 23/09/24 21:35:01 INFO TorrentBroadcast: Reading broadcast variable 20 took 4 ms 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 10.1 KiB, free 3.9 GiB) 23/09/24 21:35:01 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 4, fetching them 23/09/24 21:35:01 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:46637) 23/09/24 21:35:01 INFO MapOutputTrackerWorker: Got the output locations 23/09/24 21:35:01 INFO ShuffleBlockFetcherIterator: Getting 3 (180.0 B) non-empty blocks including 1 (60.0 B) local and 2 (120.0 B) host-local and 0 (0.0 B) remote blocks 23/09/24 21:35:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 9 ms 23/09/24 21:35:01 INFO CodeGenerator: Code generated in 5.669652 ms 23/09/24 21:35:01 INFO Executor: Finished task 0.0 in stage 13.0 (TID 520). 2450 bytes result sent to driver 23/09/24 21:35:01 INFO CoarseGrainedExecutorBackend: Got assigned task 523 23/09/24 21:35:01 INFO Executor: Running task 2.0 in stage 14.0 (TID 523) 23/09/24 21:35:01 INFO TorrentBroadcast: Started reading broadcast variable 21 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 10.5 KiB, free 3.9 GiB) 23/09/24 21:35:01 INFO TorrentBroadcast: Reading broadcast variable 21 took 9 ms 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 30.5 KiB, free 3.9 GiB) 23/09/24 21:35:01 INFO BlockManager: Found block rdd_44_2 locally 23/09/24 21:35:01 INFO Executor: Finished task 2.0 in stage 14.0 (TID 523). 2047 bytes result sent to driver 23/09/24 21:35:01 INFO CoarseGrainedExecutorBackend: Got assigned task 524 23/09/24 21:35:01 INFO Executor: Running task 2.0 in stage 15.0 (TID 524) 23/09/24 21:35:01 INFO TorrentBroadcast: Started reading broadcast variable 23 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 3.9 GiB) 23/09/24 21:35:01 INFO TorrentBroadcast: Reading broadcast variable 23 took 5 ms 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_23 stored as values in memory (estimated size 36.8 KiB, free 3.9 GiB) 23/09/24 21:35:01 INFO BlockManager: Found block rdd_44_2 locally 23/09/24 21:35:01 INFO TorrentBroadcast: Started reading broadcast variable 22 with 1 pieces (estimated total size 4.0 MiB) 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 1669.0 B, free 3.9 GiB) 23/09/24 21:35:01 INFO TorrentBroadcast: Reading broadcast variable 22 took 6 ms 23/09/24 21:35:01 INFO MemoryStore: Block broadcast_22 stored as values in memory (estimated size 64.0 MiB, free 3.8 GiB) 23/09/24 21:35:01 INFO Executor: Finished task 2.0 in stage 15.0 (TID 524). 2435 bytes result sent to driver 23/09/24 21:35:06 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown 23/09/24 21:35:06 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM tdown