Spark Executor Command: "/usr/java8/jdk1.8.0_202/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx10240M" "-Dspark.driver.port=39929" "-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-XX:+HeapDumpOnOutOfMemoryError" "-Xloggc:/tmp/memLog.txt" "-XX:+UseCompressedOops" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:39929" "--executor-id" "1" "--hostname" "5.161.188.154" "--cores" "5" "--app-id" "app-20230924201213-0003" "--worker-url" "spark://Worker@5.161.188.154:36849"
========================================

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
23/09/24 20:12:14 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 259349@entity-resolution
23/09/24 20:12:14 INFO SignalUtils: Registering signal handler for TERM
23/09/24 20:12:14 INFO SignalUtils: Registering signal handler for HUP
23/09/24 20:12:14 INFO SignalUtils: Registering signal handler for INT
23/09/24 20:12:14 WARN Utils: Your hostname, entity-resolution resolves to a loopback address: 127.0.1.1; using 5.161.188.154 instead (on interface eth0)
23/09/24 20:12:14 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
23/09/24 20:12:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/09/24 20:12:15 INFO SecurityManager: Changing view acls to: root
23/09/24 20:12:15 INFO SecurityManager: Changing modify acls to: root
23/09/24 20:12:15 INFO SecurityManager: Changing view acls groups to: 
23/09/24 20:12:15 INFO SecurityManager: Changing modify acls groups to: 
23/09/24 20:12:15 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
23/09/24 20:12:15 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:39929 after 66 ms (0 ms spent in bootstraps)
23/09/24 20:12:15 INFO SecurityManager: Changing view acls to: root
23/09/24 20:12:15 INFO SecurityManager: Changing modify acls to: root
23/09/24 20:12:15 INFO SecurityManager: Changing view acls groups to: 
23/09/24 20:12:15 INFO SecurityManager: Changing modify acls groups to: 
23/09/24 20:12:15 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
23/09/24 20:12:15 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:39929 after 1 ms (0 ms spent in bootstraps)
23/09/24 20:12:15 INFO DiskBlockManager: Created local directory at /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-490ab460-820e-4cda-a5fe-bab36225920e/blockmgr-45e67ab5-6bce-4ffa-a328-2363453eddc0
23/09/24 20:12:15 INFO MemoryStore: MemoryStore started with capacity 5.2 GiB
23/09/24 20:12:15 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:39929
23/09/24 20:12:15 INFO WorkerWatcher: Connecting to worker spark://Worker@5.161.188.154:36849
23/09/24 20:12:15 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:36849 after 2 ms (0 ms spent in bootstraps)
23/09/24 20:12:15 INFO WorkerWatcher: Successfully connected to spark://Worker@5.161.188.154:36849
23/09/24 20:12:15 INFO ResourceUtils: ==============================================================
23/09/24 20:12:15 INFO ResourceUtils: No custom resources configured for spark.executor.
23/09/24 20:12:15 INFO ResourceUtils: ==============================================================
23/09/24 20:12:15 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
23/09/24 20:12:15 INFO Executor: Starting executor ID 1 on host 5.161.188.154
23/09/24 20:12:15 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39049.
23/09/24 20:12:15 INFO NettyBlockTransferService: Server created on 5.161.188.154:39049
23/09/24 20:12:15 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/09/24 20:12:15 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, 5.161.188.154, 39049, None)
23/09/24 20:12:16 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, 5.161.188.154, 39049, None)
23/09/24 20:12:16 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, 5.161.188.154, 39049, None)
23/09/24 20:12:16 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:39929/jars/postgresql-42.6.0.jar with timestamp 1695586332587
23/09/24 20:12:16 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:39929 after 1 ms (0 ms spent in bootstraps)
23/09/24 20:12:16 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:39929/jars/postgresql-42.6.0.jar to /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-490ab460-820e-4cda-a5fe-bab36225920e/spark-43d0c9da-4199-4746-ba3b-03251ce377ef/fetchFileTemp3343093827529572982.tmp
23/09/24 20:12:16 INFO Utils: Copying /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-490ab460-820e-4cda-a5fe-bab36225920e/spark-43d0c9da-4199-4746-ba3b-03251ce377ef/-6435210351695586332587_cache to /opt/spark/work/app-20230924201213-0003/1/./postgresql-42.6.0.jar
23/09/24 20:12:16 INFO Executor: Adding file:/opt/spark/work/app-20230924201213-0003/1/./postgresql-42.6.0.jar to class loader
23/09/24 20:12:16 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:39929/jars/zingg-0.3.4-SNAPSHOT.jar with timestamp 1695586332587
23/09/24 20:12:16 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:39929/jars/zingg-0.3.4-SNAPSHOT.jar to /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-490ab460-820e-4cda-a5fe-bab36225920e/spark-43d0c9da-4199-4746-ba3b-03251ce377ef/fetchFileTemp7666621845408969432.tmp
23/09/24 20:12:16 INFO Utils: Copying /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-490ab460-820e-4cda-a5fe-bab36225920e/spark-43d0c9da-4199-4746-ba3b-03251ce377ef/-9240959261695586332587_cache to /opt/spark/work/app-20230924201213-0003/1/./zingg-0.3.4-SNAPSHOT.jar
23/09/24 20:12:16 INFO Executor: Adding file:/opt/spark/work/app-20230924201213-0003/1/./zingg-0.3.4-SNAPSHOT.jar to class loader
23/09/24 20:12:20 INFO CoarseGrainedExecutorBackend: Got assigned task 2
23/09/24 20:12:20 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
23/09/24 20:12:20 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache
23/09/24 20:12:20 INFO TorrentBroadcast: Started reading broadcast variable 4 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:20 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:36249 after 2 ms (0 ms spent in bootstraps)
23/09/24 20:12:20 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 17.8 KiB, free 5.2 GiB)
23/09/24 20:12:20 INFO TorrentBroadcast: Reading broadcast variable 4 took 71 ms
23/09/24 20:12:20 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 49.1 KiB, free 5.2 GiB)
23/09/24 20:12:21 INFO CodeGenerator: Code generated in 197.508135 ms
23/09/24 20:12:22 INFO CodeGenerator: Code generated in 27.874334 ms
23/09/24 20:12:22 INFO CodeGenerator: Code generated in 17.050073 ms
23/09/24 20:12:22 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/febrl/test.csv, range: 0-6561, partition values: [empty row]
23/09/24 20:12:22 INFO CodeGenerator: Code generated in 14.095715 ms
23/09/24 20:12:22 INFO TorrentBroadcast: Started reading broadcast variable 3 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:22 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 24.7 KiB, free 5.2 GiB)
23/09/24 20:12:22 INFO TorrentBroadcast: Reading broadcast variable 3 took 9 ms
23/09/24 20:12:22 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 394.8 KiB, free 5.2 GiB)
23/09/24 20:12:22 INFO CodeGenerator: Code generated in 52.974442 ms
23/09/24 20:12:22 INFO MemoryStore: Block rdd_18_0 stored as values in memory (estimated size 9.0 KiB, free 5.2 GiB)
23/09/24 20:12:22 INFO CodeGenerator: Code generated in 4.316858 ms
23/09/24 20:12:22 INFO CodeGenerator: Code generated in 18.941412 ms
23/09/24 20:12:22 INFO CodeGenerator: Code generated in 8.215136 ms
23/09/24 20:12:22 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 2494 bytes result sent to driver
23/09/24 20:12:23 INFO CoarseGrainedExecutorBackend: Got assigned task 4
23/09/24 20:12:23 INFO Executor: Running task 0.0 in stage 4.0 (TID 4)
23/09/24 20:12:23 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache
23/09/24 20:12:23 INFO TorrentBroadcast: Started reading broadcast variable 6 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 27.0 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO TorrentBroadcast: Reading broadcast variable 6 took 9 ms
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 75.0 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 1599 bytes result sent to driver
23/09/24 20:12:23 INFO CoarseGrainedExecutorBackend: Got assigned task 5
23/09/24 20:12:23 INFO Executor: Running task 0.0 in stage 5.0 (TID 5)
23/09/24 20:12:23 INFO TorrentBroadcast: Started reading broadcast variable 8 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 7.0 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO TorrentBroadcast: Reading broadcast variable 8 took 14 ms
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 14.7 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO CodeGenerator: Code generated in 17.552611 ms
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00056-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4526, partition values: [empty row]
23/09/24 20:12:23 INFO TorrentBroadcast: Started reading broadcast variable 7 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 25.1 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO TorrentBroadcast: Reading broadcast variable 7 took 18 ms
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 398.2 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00136-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4489, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00000-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4486, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00072-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4485, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00016-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4480, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00145-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4447, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00024-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4420, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00097-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4419, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00090-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4387, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00105-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4380, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00074-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4364, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00154-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4340, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00066-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4281, partition values: [empty row]
23/09/24 20:12:23 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 2057 bytes result sent to driver
23/09/24 20:12:25 INFO CoarseGrainedExecutorBackend: Got assigned task 11
23/09/24 20:12:25 INFO Executor: Running task 2.0 in stage 7.0 (TID 11)
23/09/24 20:12:25 INFO MapOutputTrackerWorker: Updating epoch to 3 and clearing cache
23/09/24 20:12:25 INFO TorrentBroadcast: Started reading broadcast variable 11 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:25 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.2 GiB)
23/09/24 20:12:25 INFO TorrentBroadcast: Reading broadcast variable 11 took 9 ms
23/09/24 20:12:25 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 29.8 KiB, free 5.2 GiB)
23/09/24 20:12:25 INFO CodeGenerator: Code generated in 17.356851 ms
23/09/24 20:12:25 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00038-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4078, partition values: [empty row]
23/09/24 20:12:25 INFO TorrentBroadcast: Started reading broadcast variable 10 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:25 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 25.8 KiB, free 5.2 GiB)
23/09/24 20:12:25 INFO TorrentBroadcast: Reading broadcast variable 10 took 16 ms
23/09/24 20:12:25 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 398.2 KiB, free 5.2 GiB)
23/09/24 20:12:25 INFO CodecPool: Got brand-new decompressor [.snappy]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00044-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4078, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00015-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4075, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00009-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4062, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00085-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4057, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00105-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4037, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00004-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4030, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00064-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4018, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00082-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4010, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00071-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4009, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00101-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-3992, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00059-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-3992, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00000-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-1531, partition values: [empty row]
23/09/24 20:12:26 INFO MemoryStore: Block rdd_40_2 stored as values in memory (estimated size 5.3 KiB, free 5.2 GiB)
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 5.384443 ms
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 12.913966 ms
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 5.378913 ms
23/09/24 20:12:26 INFO Executor: Finished task 2.0 in stage 7.0 (TID 11). 2090 bytes result sent to driver
23/09/24 20:12:26 INFO CoarseGrainedExecutorBackend: Got assigned task 12
23/09/24 20:12:26 INFO Executor: Running task 2.0 in stage 8.0 (TID 12)
23/09/24 20:12:26 INFO TorrentBroadcast: Started reading broadcast variable 13 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 5.2 GiB)
23/09/24 20:12:26 INFO TorrentBroadcast: Reading broadcast variable 13 took 8 ms
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 36.8 KiB, free 5.2 GiB)
23/09/24 20:12:26 INFO BlockManager: Found block rdd_40_2 locally
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 7.473277 ms
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 10.43146 ms
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 13.216858 ms
23/09/24 20:12:26 INFO TorrentBroadcast: Started reading broadcast variable 12 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 1669.0 B, free 5.2 GiB)
23/09/24 20:12:26 INFO TorrentBroadcast: Reading broadcast variable 12 took 8 ms
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 64.0 MiB, free 5.1 GiB)
23/09/24 20:12:26 INFO Executor: Finished task 2.0 in stage 8.0 (TID 12). 2434 bytes result sent to driver
23/09/24 20:12:26 INFO CoarseGrainedExecutorBackend: Got assigned task 17
23/09/24 20:12:26 INFO Executor: Running task 2.0 in stage 10.0 (TID 17)
23/09/24 20:12:26 INFO MapOutputTrackerWorker: Updating epoch to 4 and clearing cache
23/09/24 20:12:26 INFO TorrentBroadcast: Started reading broadcast variable 15 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 10.5 KiB, free 5.1 GiB)
23/09/24 20:12:26 INFO TorrentBroadcast: Reading broadcast variable 15 took 12 ms
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 30.5 KiB, free 5.1 GiB)
23/09/24 20:12:26 INFO BlockManager: Found block rdd_40_2 locally
23/09/24 20:12:26 INFO Executor: Finished task 2.0 in stage 10.0 (TID 17). 2047 bytes result sent to driver
23/09/24 20:12:27 INFO CoarseGrainedExecutorBackend: Got assigned task 21
23/09/24 20:12:27 INFO Executor: Running task 2.0 in stage 11.0 (TID 21)
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 17 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 5.1 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 17 took 8 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 36.8 KiB, free 5.1 GiB)
23/09/24 20:12:27 INFO BlockManager: Found block rdd_40_2 locally
23/09/24 20:12:27 INFO CodeGenerator: Code generated in 10.741606 ms
23/09/24 20:12:27 INFO CodeGenerator: Code generated in 7.826043 ms
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 16 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 1669.0 B, free 5.1 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 16 took 6 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 64.0 MiB, free 5.0 GiB)
23/09/24 20:12:27 INFO Executor: Finished task 2.0 in stage 11.0 (TID 21). 2434 bytes result sent to driver
23/09/24 20:12:27 INFO CoarseGrainedExecutorBackend: Got assigned task 25
23/09/24 20:12:27 INFO Executor: Running task 2.0 in stage 13.0 (TID 25)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Updating epoch to 5 and clearing cache
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 19 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 10.5 KiB, free 5.0 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 19 took 6 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 30.5 KiB, free 5.0 GiB)
23/09/24 20:12:27 INFO BlockManager: Found block rdd_40_2 locally
23/09/24 20:12:27 INFO Executor: Finished task 2.0 in stage 13.0 (TID 25). 2047 bytes result sent to driver
23/09/24 20:12:27 INFO CoarseGrainedExecutorBackend: Got assigned task 27
23/09/24 20:12:27 INFO Executor: Running task 2.0 in stage 14.0 (TID 27)
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 21 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 5.0 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 21 took 31 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 36.8 KiB, free 5.0 GiB)
23/09/24 20:12:27 INFO BlockManager: Found block rdd_40_2 locally
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 20 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:34301 after 1 ms (0 ms spent in bootstraps)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 1669.0 B, free 5.0 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 20 took 20 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 64.0 MiB, free 5.0 GiB)
23/09/24 20:12:27 INFO Executor: Finished task 2.0 in stage 14.0 (TID 27). 2434 bytes result sent to driver
23/09/24 20:12:27 INFO CoarseGrainedExecutorBackend: Got assigned task 30
23/09/24 20:12:27 INFO Executor: Running task 0.0 in stage 16.0 (TID 30)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Updating epoch to 6 and clearing cache
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 23 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 18.4 KiB, free 5.1 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 23 took 6 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_23 stored as values in memory (estimated size 52.6 KiB, free 5.1 GiB)
23/09/24 20:12:27 INFO BlockManager: Found block rdd_18_0 locally
23/09/24 20:12:27 INFO CodeGenerator: Code generated in 10.953484 ms
23/09/24 20:12:27 INFO CodeGenerator: Code generated in 9.605752 ms
23/09/24 20:12:27 INFO Executor: Finished task 0.0 in stage 16.0 (TID 30). 2369 bytes result sent to driver
23/09/24 20:12:27 INFO CoarseGrainedExecutorBackend: Got assigned task 33
23/09/24 20:12:27 INFO Executor: Running task 2.0 in stage 17.0 (TID 33)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Updating epoch to 7 and clearing cache
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 24 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 10.2 KiB, free 5.1 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 24 took 7 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_24 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 6, fetching them
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:28 INFO ShuffleBlockFetcherIterator: Getting 1 (1156.0 B) non-empty blocks including 1 (1156.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 11 ms
23/09/24 20:12:28 INFO MemoryStore: Block rdd_88_2 stored as values in memory (estimated size 3.4 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 12.205388 ms
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 6.653961 ms
23/09/24 20:12:28 INFO MemoryStore: Block rdd_94_2 stored as values in memory (estimated size 3.0 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO Executor: Finished task 2.0 in stage 17.0 (TID 33). 3204 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 35
23/09/24 20:12:28 INFO Executor: Running task 0.0 in stage 18.0 (TID 35)
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Updating epoch to 8 and clearing cache
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 25 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_25_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 25 took 4 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_25 stored as values in memory (estimated size 10.1 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 7, fetching them
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:28 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 20:12:28 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:38819 after 1 ms (0 ms spent in bootstraps)
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 6.107659 ms
23/09/24 20:12:28 INFO Executor: Finished task 0.0 in stage 18.0 (TID 35). 2450 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 36
23/09/24 20:12:28 INFO Executor: Running task 2.0 in stage 20.0 (TID 36)
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 26 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 26 took 6 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_26 stored as values in memory (estimated size 24.3 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_94_2 locally
23/09/24 20:12:28 INFO Executor: Finished task 2.0 in stage 20.0 (TID 36). 3321 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 41
23/09/24 20:12:28 INFO Executor: Running task 2.0 in stage 21.0 (TID 41)
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 27 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 11.6 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 27 took 8 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_27 stored as values in memory (estimated size 34.2 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_40_2 locally
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 9.893243 ms
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 6.877662 ms
23/09/24 20:12:28 INFO Executor: Finished task 2.0 in stage 21.0 (TID 41). 3786 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 43
23/09/24 20:12:28 INFO Executor: Running task 0.0 in stage 22.0 (TID 43)
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 29 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 15.2 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 29 took 5 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_29 stored as values in memory (estimated size 44.1 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_40_0 remotely
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 10.051052 ms
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 9.469764 ms
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 28 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_28_piece0 stored as bytes in memory (estimated size 5.9 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 28 took 5 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_28 stored as values in memory (estimated size 64.0 MiB, free 5.0 GiB)
23/09/24 20:12:28 INFO MemoryStore: Block rdd_117_0 stored as values in memory (estimated size 7.0 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 12.039544 ms
23/09/24 20:12:28 INFO BlockManager: Found block rdd_40_1 remotely
23/09/24 20:12:28 INFO MemoryStore: Block rdd_117_1 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_40_2 locally
23/09/24 20:12:28 INFO MemoryStore: Block rdd_117_2 stored as values in memory (estimated size 16.0 B, free 5.0 GiB)
23/09/24 20:12:28 INFO Executor: Finished task 0.0 in stage 22.0 (TID 43). 4298 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 44
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 45
23/09/24 20:12:28 INFO Executor: Running task 0.0 in stage 23.0 (TID 44)
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 46
23/09/24 20:12:28 INFO Executor: Running task 1.0 in stage 23.0 (TID 45)
23/09/24 20:12:28 INFO Executor: Running task 2.0 in stage 23.0 (TID 46)
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 30 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 16.9 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 30 took 4 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_30 stored as values in memory (estimated size 47.9 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_117_1 locally
23/09/24 20:12:28 INFO BlockManager: Found block rdd_117_2 locally
23/09/24 20:12:28 INFO BlockManager: Found block rdd_117_0 locally
23/09/24 20:12:28 INFO Executor: Finished task 1.0 in stage 23.0 (TID 45). 2601 bytes result sent to driver
23/09/24 20:12:28 INFO Executor: Finished task 2.0 in stage 23.0 (TID 46). 2558 bytes result sent to driver
23/09/24 20:12:28 INFO Executor: Finished task 0.0 in stage 23.0 (TID 44). 2601 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 47
23/09/24 20:12:28 INFO Executor: Running task 0.0 in stage 24.0 (TID 47)
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Updating epoch to 9 and clearing cache
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 31 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_31_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 31 took 4 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_31 stored as values in memory (estimated size 10.1 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 8, fetching them
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:28 INFO ShuffleBlockFetcherIterator: Getting 3 (180.0 B) non-empty blocks including 3 (180.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:28 INFO Executor: Finished task 0.0 in stage 24.0 (TID 47). 2450 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 49
23/09/24 20:12:28 INFO Executor: Running task 2.0 in stage 25.0 (TID 49)
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 32 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_32_piece0 stored as bytes in memory (estimated size 11.6 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 32 took 5 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_32 stored as values in memory (estimated size 34.2 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_40_2 locally
23/09/24 20:12:28 INFO Executor: Finished task 2.0 in stage 25.0 (TID 49). 3786 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 52
23/09/24 20:12:28 INFO Executor: Running task 2.0 in stage 26.0 (TID 52)
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 34 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_34_piece0 stored as bytes in memory (estimated size 16.9 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 34 took 5 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_34 stored as values in memory (estimated size 47.9 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_40_2 locally
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 16.611676 ms
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 33 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_33_piece0 stored as bytes in memory (estimated size 5.9 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 33 took 5 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_33 stored as values in memory (estimated size 64.0 MiB, free 5.0 GiB)
23/09/24 20:12:28 INFO MemoryStore: Block rdd_141_2 stored as values in memory (estimated size 6.6 KiB, free 5.0 GiB)
23/09/24 20:12:28 INFO Executor: Finished task 2.0 in stage 26.0 (TID 52). 2644 bytes result sent to driver
23/09/24 20:12:34 INFO BlockManager: Removing RDD 191
23/09/24 20:12:34 INFO BlockManager: Removing RDD 191
23/09/24 20:12:35 INFO CoarseGrainedExecutorBackend: Got assigned task 94
23/09/24 20:12:35 INFO Executor: Running task 2.0 in stage 68.0 (TID 94)
23/09/24 20:12:35 INFO MapOutputTrackerWorker: Updating epoch to 10 and clearing cache
23/09/24 20:12:35 INFO TorrentBroadcast: Started reading broadcast variable 114 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_114_piece0 stored as bytes in memory (estimated size 17.8 KiB, free 5.0 GiB)
23/09/24 20:12:35 INFO TorrentBroadcast: Reading broadcast variable 114 took 5 ms
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_114 stored as values in memory (estimated size 63.4 KiB, free 5.0 GiB)
23/09/24 20:12:35 INFO BlockManager: Found block rdd_88_2 locally
23/09/24 20:12:35 INFO CodeGenerator: Code generated in 66.417977 ms
23/09/24 20:12:35 INFO CodeGenerator: Code generated in 8.121209 ms
23/09/24 20:12:35 INFO Executor: Finished task 2.0 in stage 68.0 (TID 94). 2574 bytes result sent to driver
23/09/24 20:12:35 INFO CoarseGrainedExecutorBackend: Got assigned task 99
23/09/24 20:12:35 INFO Executor: Running task 1.0 in stage 69.0 (TID 99)
23/09/24 20:12:35 INFO MapOutputTrackerWorker: Updating epoch to 11 and clearing cache
23/09/24 20:12:35 INFO TorrentBroadcast: Started reading broadcast variable 115 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_115_piece0 stored as bytes in memory (estimated size 7.8 KiB, free 5.0 GiB)
23/09/24 20:12:35 INFO TorrentBroadcast: Reading broadcast variable 115 took 4 ms
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_115 stored as values in memory (estimated size 19.4 KiB, free 5.0 GiB)
23/09/24 20:12:35 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 10, fetching them
23/09/24 20:12:35 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:35 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:35 INFO ShuffleBlockFetcherIterator: Getting 3 (1118.0 B) non-empty blocks including 1 (228.0 B) local and 2 (890.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:35 INFO MemoryStore: Block rdd_254_1 stored as values in memory (estimated size 3.0 KiB, free 5.0 GiB)
23/09/24 20:12:35 INFO CodeGenerator: Code generated in 11.426187 ms
23/09/24 20:12:35 INFO CodeGenerator: Code generated in 5.854551 ms
23/09/24 20:12:35 INFO Executor: Finished task 1.0 in stage 69.0 (TID 99). 3253 bytes result sent to driver
23/09/24 20:12:35 INFO CoarseGrainedExecutorBackend: Got assigned task 104
23/09/24 20:12:35 INFO Executor: Running task 1.0 in stage 72.0 (TID 104)
23/09/24 20:12:35 INFO TorrentBroadcast: Started reading broadcast variable 117 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_117_piece0 stored as bytes in memory (estimated size 51.3 KiB, free 5.0 GiB)
23/09/24 20:12:35 INFO TorrentBroadcast: Reading broadcast variable 117 took 7 ms
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_117 stored as values in memory (estimated size 139.3 KiB, free 5.0 GiB)
23/09/24 20:12:35 INFO BlockManager: Found block rdd_254_1 locally
23/09/24 20:12:35 INFO CodeGenerator: Code generated in 10.183932 ms
23/09/24 20:12:35 INFO CodeGenerator: Code generated in 11.855007 ms
23/09/24 20:12:35 INFO TorrentBroadcast: Started reading broadcast variable 116 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_116_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 5.0 GiB)
23/09/24 20:12:35 INFO TorrentBroadcast: Reading broadcast variable 116 took 7 ms
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_116 stored as values in memory (estimated size 7.6 KiB, free 5.0 GiB)
23/09/24 20:12:35 INFO MemoryStore: Block rdd_265_1 stored as values in memory (estimated size 4.9 KiB, free 5.0 GiB)
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 17.120485 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 23.089082 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 6.14525 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 3.863964 ms
23/09/24 20:12:36 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
23/09/24 20:12:36 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
23/09/24 20:12:36 INFO MemoryStore: Block rdd_271_1 stored as values in memory (estimated size 5.1 KiB, free 5.0 GiB)
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 8.336696 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 8.537475 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 6.043106 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 3.024047 ms
23/09/24 20:12:36 INFO Executor: Finished task 1.0 in stage 72.0 (TID 104). 2849 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 106
23/09/24 20:12:37 INFO Executor: Running task 1.0 in stage 75.0 (TID 106)
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 118 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_118_piece0 stored as bytes in memory (estimated size 51.3 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 118 took 4 ms
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_118 stored as values in memory (estimated size 139.3 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO BlockManager: Found block rdd_271_1 locally
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 7.844937 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 8.011022 ms
23/09/24 20:12:37 INFO Executor: Finished task 1.0 in stage 75.0 (TID 106). 2817 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 110
23/09/24 20:12:37 INFO Executor: Running task 2.0 in stage 77.0 (TID 110)
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 119 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_119_piece0 stored as bytes in memory (estimated size 7.1 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 119 took 9 ms
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_119 stored as values in memory (estimated size 15.1 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO BlockManager: Found block rdd_88_2 locally
23/09/24 20:12:37 INFO Executor: Finished task 2.0 in stage 77.0 (TID 110). 3043 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 115
23/09/24 20:12:37 INFO Executor: Running task 1.0 in stage 80.0 (TID 115)
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 121 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 119
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_121_piece0 stored as bytes in memory (estimated size 51.9 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO Executor: Running task 1.0 in stage 82.0 (TID 119)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 121 took 5 ms
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 122 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_121 stored as values in memory (estimated size 140.2 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_122_piece0 stored as bytes in memory (estimated size 51.9 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 122 took 18 ms
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_122 stored as values in memory (estimated size 140.3 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 12.766598 ms
23/09/24 20:12:37 INFO BlockManager: Found block rdd_271_1 locally
23/09/24 20:12:37 INFO Executor: Finished task 1.0 in stage 80.0 (TID 115). 2997 bytes result sent to driver
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 11.19412 ms
23/09/24 20:12:37 INFO BlockManager: Found block rdd_271_1 locally
23/09/24 20:12:37 INFO Executor: Finished task 1.0 in stage 82.0 (TID 119). 3130 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 124
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 127
23/09/24 20:12:37 INFO Executor: Running task 2.0 in stage 81.0 (TID 124)
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 130
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 133
23/09/24 20:12:37 INFO Executor: Running task 8.0 in stage 81.0 (TID 130)
23/09/24 20:12:37 INFO Executor: Running task 5.0 in stage 81.0 (TID 127)
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Updating epoch to 12 and clearing cache
23/09/24 20:12:37 INFO Executor: Running task 11.0 in stage 81.0 (TID 133)
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 123 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_123_piece0 stored as bytes in memory (estimated size 53.6 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 123 took 5 ms
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_123 stored as values in memory (estimated size 147.6 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 135
23/09/24 20:12:37 INFO Executor: Running task 1.0 in stage 83.0 (TID 135)
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Updating epoch to 13 and clearing cache
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 124 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_124_piece0 stored as bytes in memory (estimated size 53.6 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 124 took 5 ms
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_124 stored as values in memory (estimated size 147.6 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 11, fetching them
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 11, fetching them
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 0 (0.0 B) local and 1 (368.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 0 (0.0 B) local and 1 (368.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 0 (0.0 B) local and 1 (368.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 0 (0.0 B) local and 1 (368.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 13, fetching them
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 0 (0.0 B) local and 1 (445.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 10.839972 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 20.87881 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 14.630676 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 17.643372 ms
23/09/24 20:12:37 INFO MemoryStore: Block rdd_283_11 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO MemoryStore: Block rdd_283_5 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO MemoryStore: Block rdd_283_2 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO MemoryStore: Block rdd_283_8 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 14.869556 ms
23/09/24 20:12:37 INFO Executor: Finished task 5.0 in stage 81.0 (TID 127). 4580 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 137
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_1 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO Executor: Running task 3.0 in stage 83.0 (TID 137)
23/09/24 20:12:37 INFO Executor: Finished task 8.0 in stage 81.0 (TID 130). 4580 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 138
23/09/24 20:12:37 INFO Executor: Running task 4.0 in stage 83.0 (TID 138)
23/09/24 20:12:37 INFO Executor: Finished task 2.0 in stage 81.0 (TID 124). 4580 bytes result sent to driver
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 6.172451 ms
23/09/24 20:12:37 INFO Executor: Finished task 11.0 in stage 81.0 (TID 133). 4580 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 139
23/09/24 20:12:37 INFO Executor: Finished task 1.0 in stage 83.0 (TID 135). 4580 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 140
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 141
23/09/24 20:12:37 INFO Executor: Running task 7.0 in stage 83.0 (TID 141)
23/09/24 20:12:37 INFO Executor: Running task 5.0 in stage 83.0 (TID 139)
23/09/24 20:12:37 INFO Executor: Running task 6.0 in stage 83.0 (TID 140)
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (405.0 B) non-empty blocks including 0 (0.0 B) local and 1 (405.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 0 (0.0 B) local and 1 (445.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_4 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 0 (0.0 B) local and 1 (445.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (405.0 B) non-empty blocks including 0 (0.0 B) local and 1 (405.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_7 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_6 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO Executor: Finished task 4.0 in stage 83.0 (TID 138). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO Executor: Finished task 7.0 in stage 83.0 (TID 141). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_5 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 142
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 143
23/09/24 20:12:37 INFO Executor: Running task 8.0 in stage 83.0 (TID 142)
23/09/24 20:12:37 INFO Executor: Running task 9.0 in stage 83.0 (TID 143)
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 1 (445.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO Executor: Finished task 5.0 in stage 83.0 (TID 139). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO Executor: Finished task 6.0 in stage 83.0 (TID 140). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 144
23/09/24 20:12:37 INFO Executor: Running task 10.0 in stage 83.0 (TID 144)
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 145
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_3 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO Executor: Running task 11.0 in stage 83.0 (TID 145)
23/09/24 20:12:37 INFO Executor: Finished task 3.0 in stage 83.0 (TID 137). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 146
23/09/24 20:12:37 INFO Executor: Running task 12.0 in stage 83.0 (TID 146)
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (405.0 B) non-empty blocks including 0 (0.0 B) local and 1 (405.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 0 (0.0 B) local and 1 (445.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (405.0 B) non-empty blocks including 0 (0.0 B) local and 1 (405.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 4 ms
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_11 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_10 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 0 (0.0 B) local and 1 (445.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO Executor: Finished task 11.0 in stage 83.0 (TID 145). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_9 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO Executor: Finished task 10.0 in stage 83.0 (TID 144). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_8 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 0 (0.0 B) local and 1 (445.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO Executor: Finished task 8.0 in stage 83.0 (TID 142). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO Executor: Finished task 9.0 in stage 83.0 (TID 143). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_12 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO Executor: Finished task 12.0 in stage 83.0 (TID 146). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 150
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 153
23/09/24 20:12:37 INFO Executor: Running task 0.0 in stage 84.0 (TID 150)
23/09/24 20:12:37 INFO Executor: Running task 3.0 in stage 84.0 (TID 153)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Updating epoch to 15 and clearing cache
23/09/24 20:12:38 INFO TorrentBroadcast: Started reading broadcast variable 125 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_125_piece0 stored as bytes in memory (estimated size 75.0 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO TorrentBroadcast: Reading broadcast variable 125 took 5 ms
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_125 stored as values in memory (estimated size 233.4 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 12, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 14, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 15 (6.3 KiB) non-empty blocks including 11 (4.6 KiB) local and 4 (1669.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 12 (4.3 KiB) non-empty blocks including 4 (1472.0 B) local and 8 (2.9 KiB) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 7.755849 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 7.386593 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 12.530662 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 8.828103 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 46.692734 ms
23/09/24 20:12:38 INFO MemoryStore: Block rdd_323_1 stored as values in memory (estimated size 7.4 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO MemoryStore: Block rdd_323_0 stored as values in memory (estimated size 6.6 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 24.5451 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 31.459561 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 5.215965 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 5.531462 ms
23/09/24 20:12:38 INFO TorrentBroadcast: Started reading broadcast variable 120 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_120_piece0 stored as bytes in memory (estimated size 3.7 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 20.680735 ms
23/09/24 20:12:38 INFO TorrentBroadcast: Reading broadcast variable 120 took 8 ms
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_120 stored as values in memory (estimated size 7.4 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO MemoryStore: Block rdd_333_1 stored as values in memory (estimated size 4.1 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 4.81553 ms
23/09/24 20:12:38 INFO Executor: Finished task 0.0 in stage 84.0 (TID 150). 7923 bytes result sent to driver
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 12.920157 ms
23/09/24 20:12:38 INFO Executor: Finished task 3.0 in stage 84.0 (TID 153). 7923 bytes result sent to driver
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 154
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 157
23/09/24 20:12:38 INFO Executor: Running task 0.0 in stage 91.0 (TID 154)
23/09/24 20:12:38 INFO Executor: Running task 3.0 in stage 91.0 (TID 157)
23/09/24 20:12:38 INFO TorrentBroadcast: Started reading broadcast variable 126 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_126_piece0 stored as bytes in memory (estimated size 75.1 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO TorrentBroadcast: Reading broadcast variable 126 took 3 ms
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_126 stored as values in memory (estimated size 234.1 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 4.266153 ms
23/09/24 20:12:38 INFO BlockManager: Found block rdd_323_0 locally
23/09/24 20:12:38 INFO BlockManager: Found block rdd_333_1 locally
23/09/24 20:12:38 INFO Executor: Finished task 0.0 in stage 91.0 (TID 154). 7948 bytes result sent to driver
23/09/24 20:12:38 INFO Executor: Finished task 3.0 in stage 91.0 (TID 157). 7948 bytes result sent to driver
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 160
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 163
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 166
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 169
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 172
23/09/24 20:12:38 INFO Executor: Running task 11.0 in stage 92.0 (TID 169)
23/09/24 20:12:38 INFO Executor: Running task 8.0 in stage 92.0 (TID 166)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Updating epoch to 16 and clearing cache
23/09/24 20:12:38 INFO Executor: Running task 14.0 in stage 92.0 (TID 172)
23/09/24 20:12:38 INFO Executor: Running task 5.0 in stage 92.0 (TID 163)
23/09/24 20:12:38 INFO TorrentBroadcast: Started reading broadcast variable 127 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:38 INFO Executor: Running task 2.0 in stage 92.0 (TID 160)
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_127_piece0 stored as bytes in memory (estimated size 129.5 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO TorrentBroadcast: Reading broadcast variable 127 took 5 ms
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_127 stored as values in memory (estimated size 379.7 KiB, free 5.0 GiB)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 15, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 15, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 15, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 15, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 15, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 1 (304.0 B) local and 1 (304.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 1 (304.0 B) local and 1 (304.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 1 (304.0 B) local and 1 (276.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 1 (304.0 B) local and 1 (304.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 1 (304.0 B) local and 1 (304.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 21.436123 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 5.156974 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 18.830943 ms
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:38 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:38 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:38 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:38 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:38 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:38 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:38 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:38 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:38 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:38 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:38 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:38 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:38 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:38 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:38 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:38 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:38 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:38 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:38 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:38 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:38 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:38 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:38 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:38 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:38 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:38 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:38 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:38 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:38 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:38 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:38 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:38 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 20:12:38 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 20:12:38 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 20:12:38 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 20:12:38 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 20:12:38 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 20:12:38 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 20:12:38 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 20:12:38 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 20:12:38 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 20:12:38 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 435
23/09/24 20:12:38 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 431
23/09/24 20:12:38 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 441
23/09/24 20:12:38 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 375
23/09/24 20:12:38 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 429
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012384828653413350235138_0092_m_000008_166' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012384828653413350235138_0092_m_000008
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012384828653413350235138_0092_m_000008_166: Committed
23/09/24 20:12:39 INFO Executor: Finished task 8.0 in stage 92.0 (TID 166). 10148 bytes result sent to driver
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012388651599059870845021_0092_m_000014_172' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012388651599059870845021_0092_m_000014
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012384658386015237725229_0092_m_000002_160' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012384658386015237725229_0092_m_000002
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012384658386015237725229_0092_m_000002_160: Committed
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012388651599059870845021_0092_m_000014_172: Committed
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012383475440681940373282_0092_m_000005_163' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012383475440681940373282_0092_m_000005
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012383475440681940373282_0092_m_000005_163: Committed
23/09/24 20:12:39 INFO CoarseGrainedExecutorBackend: Got assigned task 178
23/09/24 20:12:39 INFO Executor: Running task 20.0 in stage 92.0 (TID 178)
23/09/24 20:12:39 INFO Executor: Finished task 14.0 in stage 92.0 (TID 172). 10191 bytes result sent to driver
23/09/24 20:12:39 INFO Executor: Finished task 2.0 in stage 92.0 (TID 160). 10191 bytes result sent to driver
23/09/24 20:12:39 INFO Executor: Finished task 5.0 in stage 92.0 (TID 163). 10148 bytes result sent to driver
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012388157572300490573105_0092_m_000011_169' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012388157572300490573105_0092_m_000011
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012388157572300490573105_0092_m_000011_169: Committed
23/09/24 20:12:39 INFO Executor: Finished task 11.0 in stage 92.0 (TID 169). 10148 bytes result sent to driver
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202309242012383726433164696646411_0092_m_000020_178
23/09/24 20:12:39 INFO Executor: Finished task 20.0 in stage 92.0 (TID 178). 10061 bytes result sent to driver
23/09/24 20:12:39 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
23/09/24 20:12:39 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
tdown