3.1.3
stderr log page for app-20230924201213-0003/2
Showing 102400 Bytes: 3788 - 106188 of 106188
===========================================================
23/09/24 20:12:15 INFO WorkerWatcher: Successfully connected to spark://Worker@5.161.188.154:42679
23/09/24 20:12:15 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
23/09/24 20:12:15 INFO Executor: Starting executor ID 2 on host 5.161.188.154
23/09/24 20:12:15 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38819.
23/09/24 20:12:15 INFO NettyBlockTransferService: Server created on 5.161.188.154:38819
23/09/24 20:12:15 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/09/24 20:12:15 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(2, 5.161.188.154, 38819, None)
23/09/24 20:12:15 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(2, 5.161.188.154, 38819, None)
23/09/24 20:12:15 INFO BlockManager: Initialized BlockManager: BlockManagerId(2, 5.161.188.154, 38819, None)
23/09/24 20:12:16 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:39929/jars/postgresql-42.6.0.jar with timestamp 1695586332587
23/09/24 20:12:16 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:39929 after 1 ms (0 ms spent in bootstraps)
23/09/24 20:12:16 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:39929/jars/postgresql-42.6.0.jar to /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-66c5bcde-13ed-42b5-8a3a-cc630135331f/spark-66bfb3c2-cff9-46e1-bca6-38e27074efa4/fetchFileTemp7469939506899507016.tmp
23/09/24 20:12:16 INFO Utils: Copying /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-66c5bcde-13ed-42b5-8a3a-cc630135331f/spark-66bfb3c2-cff9-46e1-bca6-38e27074efa4/-6435210351695586332587_cache to /opt/spark/work/app-20230924201213-0003/2/./postgresql-42.6.0.jar
23/09/24 20:12:16 INFO Executor: Adding file:/opt/spark/work/app-20230924201213-0003/2/./postgresql-42.6.0.jar to class loader
23/09/24 20:12:16 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:39929/jars/zingg-0.3.4-SNAPSHOT.jar with timestamp 1695586332587
23/09/24 20:12:16 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:39929/jars/zingg-0.3.4-SNAPSHOT.jar to /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-66c5bcde-13ed-42b5-8a3a-cc630135331f/spark-66bfb3c2-cff9-46e1-bca6-38e27074efa4/fetchFileTemp8576817143440831630.tmp
23/09/24 20:12:16 INFO Utils: Copying /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-66c5bcde-13ed-42b5-8a3a-cc630135331f/spark-66bfb3c2-cff9-46e1-bca6-38e27074efa4/-9240959261695586332587_cache to /opt/spark/work/app-20230924201213-0003/2/./zingg-0.3.4-SNAPSHOT.jar
23/09/24 20:12:16 INFO Executor: Adding file:/opt/spark/work/app-20230924201213-0003/2/./zingg-0.3.4-SNAPSHOT.jar to class loader
23/09/24 20:12:18 INFO CoarseGrainedExecutorBackend: Got assigned task 0
23/09/24 20:12:18 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
23/09/24 20:12:18 INFO TorrentBroadcast: Started reading broadcast variable 1 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:18 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:36249 after 2 ms (0 ms spent in bootstraps)
23/09/24 20:12:18 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 7.9 KiB, free 5.2 GiB)
23/09/24 20:12:18 INFO TorrentBroadcast: Reading broadcast variable 1 took 104 ms
23/09/24 20:12:18 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 15.4 KiB, free 5.2 GiB)
23/09/24 20:12:19 INFO CodeGenerator: Code generated in 169.077972 ms
23/09/24 20:12:19 INFO FileScanRDD: Reading File path: file:///root/zingg/examples/febrl/test.csv, range: 0-6561, partition values: [empty row]
23/09/24 20:12:19 INFO CodeGenerator: Code generated in 9.215083 ms
23/09/24 20:12:19 INFO TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:19 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 24.7 KiB, free 5.2 GiB)
23/09/24 20:12:19 INFO TorrentBroadcast: Reading broadcast variable 0 took 10 ms
23/09/24 20:12:19 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 394.8 KiB, free 5.2 GiB)
23/09/24 20:12:19 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1886 bytes result sent to driver
23/09/24 20:12:19 INFO CoarseGrainedExecutorBackend: Got assigned task 1
23/09/24 20:12:19 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
23/09/24 20:12:19 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache
23/09/24 20:12:19 INFO TorrentBroadcast: Started reading broadcast variable 2 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:19 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.2 GiB)
23/09/24 20:12:19 INFO TorrentBroadcast: Reading broadcast variable 2 took 9 ms
23/09/24 20:12:19 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 10.1 KiB, free 5.2 GiB)
23/09/24 20:12:19 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them
23/09/24 20:12:19 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:19 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:19 INFO ShuffleBlockFetcherIterator: Getting 1 (60.0 B) non-empty blocks including 1 (60.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 9 ms
23/09/24 20:12:19 INFO CodeGenerator: Code generated in 10.743929 ms
23/09/24 20:12:19 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 2493 bytes result sent to driver
23/09/24 20:12:23 INFO CoarseGrainedExecutorBackend: Got assigned task 3
23/09/24 20:12:23 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
23/09/24 20:12:23 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache
23/09/24 20:12:23 INFO TorrentBroadcast: Started reading broadcast variable 5 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO TorrentBroadcast: Reading broadcast variable 5 took 11 ms
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 10.1 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 1, fetching them
23/09/24 20:12:23 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:23 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:23 INFO ShuffleBlockFetcherIterator: Getting 1 (60.0 B) non-empty blocks including 0 (0.0 B) local and 1 (60.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
23/09/24 20:12:23 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:39049 after 2 ms (0 ms spent in bootstraps)
23/09/24 20:12:23 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 2450 bytes result sent to driver
23/09/24 20:12:23 INFO CoarseGrainedExecutorBackend: Got assigned task 6
23/09/24 20:12:23 INFO Executor: Running task 1.0 in stage 5.0 (TID 6)
23/09/24 20:12:23 INFO TorrentBroadcast: Started reading broadcast variable 8 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 7.0 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO TorrentBroadcast: Reading broadcast variable 8 took 12 ms
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 14.7 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO CodeGenerator: Code generated in 30.057776 ms
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00049-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4280, partition values: [empty row]
23/09/24 20:12:23 INFO TorrentBroadcast: Started reading broadcast variable 7 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 25.1 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO TorrentBroadcast: Reading broadcast variable 7 took 29 ms
23/09/24 20:12:23 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 398.2 KiB, free 5.2 GiB)
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00019-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4211, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00131-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4138, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00043-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4128, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00035-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4116, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00123-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4104, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00027-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4103, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00092-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4099, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00012-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4098, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00085-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4098, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00052-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4095, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00116-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4090, partition values: [empty row]
23/09/24 20:12:23 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00032-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4080, partition values: [empty row]
23/09/24 20:12:24 INFO Executor: Finished task 1.0 in stage 5.0 (TID 6). 2057 bytes result sent to driver
23/09/24 20:12:25 INFO CoarseGrainedExecutorBackend: Got assigned task 8
23/09/24 20:12:25 INFO Executor: Running task 0.0 in stage 6.0 (TID 8)
23/09/24 20:12:25 INFO MapOutputTrackerWorker: Updating epoch to 3 and clearing cache
23/09/24 20:12:25 INFO TorrentBroadcast: Started reading broadcast variable 9 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:25 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.2 GiB)
23/09/24 20:12:25 INFO TorrentBroadcast: Reading broadcast variable 9 took 7 ms
23/09/24 20:12:25 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 10.1 KiB, free 5.2 GiB)
23/09/24 20:12:25 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 2, fetching them
23/09/24 20:12:25 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:25 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:25 INFO ShuffleBlockFetcherIterator: Getting 3 (180.0 B) non-empty blocks including 1 (60.0 B) local and 2 (120.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:25 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:34301 after 2 ms (0 ms spent in bootstraps)
23/09/24 20:12:25 INFO Executor: Finished task 0.0 in stage 6.0 (TID 8). 2450 bytes result sent to driver
23/09/24 20:12:25 INFO CoarseGrainedExecutorBackend: Got assigned task 9
23/09/24 20:12:25 INFO Executor: Running task 0.0 in stage 7.0 (TID 9)
23/09/24 20:12:25 INFO TorrentBroadcast: Started reading broadcast variable 11 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:25 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.2 GiB)
23/09/24 20:12:25 INFO TorrentBroadcast: Reading broadcast variable 11 took 8 ms
23/09/24 20:12:25 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 29.8 KiB, free 5.2 GiB)
23/09/24 20:12:25 INFO CodeGenerator: Code generated in 29.159779 ms
23/09/24 20:12:25 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00056-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4526, partition values: [empty row]
23/09/24 20:12:25 INFO TorrentBroadcast: Started reading broadcast variable 10 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:25 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 25.8 KiB, free 5.2 GiB)
23/09/24 20:12:25 INFO TorrentBroadcast: Reading broadcast variable 10 took 10 ms
23/09/24 20:12:25 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 398.2 KiB, free 5.2 GiB)
23/09/24 20:12:25 INFO CodecPool: Got brand-new decompressor [.snappy]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00136-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4489, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00000-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4486, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00072-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4485, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00016-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4480, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00145-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4447, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00024-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4420, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00097-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4419, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00090-95d092b2-cf46-43df-84c9-f5a022102ae0-c000.snappy.parquet, range: 0-4387, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00105-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4380, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00074-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4364, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00154-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4340, partition values: [empty row]
23/09/24 20:12:26 INFO FileScanRDD: Reading File path: file:///root/zingg/models/100/trainingData/marked/part-00066-08a344f4-94f9-408b-8983-a293c33c986e-c000.snappy.parquet, range: 0-4281, partition values: [empty row]
23/09/24 20:12:26 INFO MemoryStore: Block rdd_40_0 stored as values in memory (estimated size 5.4 KiB, free 5.2 GiB)
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 6.979264 ms
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 24.864903 ms
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 7.978802 ms
23/09/24 20:12:26 INFO Executor: Finished task 0.0 in stage 7.0 (TID 9). 2124 bytes result sent to driver
23/09/24 20:12:26 INFO CoarseGrainedExecutorBackend: Got assigned task 13
23/09/24 20:12:26 INFO Executor: Running task 0.0 in stage 8.0 (TID 13)
23/09/24 20:12:26 INFO TorrentBroadcast: Started reading broadcast variable 13 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 5.2 GiB)
23/09/24 20:12:26 INFO TorrentBroadcast: Reading broadcast variable 13 took 8 ms
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 36.8 KiB, free 5.2 GiB)
23/09/24 20:12:26 INFO BlockManager: Found block rdd_40_0 locally
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 9.856024 ms
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 16.281682 ms
23/09/24 20:12:26 INFO CodeGenerator: Code generated in 17.580303 ms
23/09/24 20:12:26 INFO TorrentBroadcast: Started reading broadcast variable 12 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 1669.0 B, free 5.2 GiB)
23/09/24 20:12:26 INFO TorrentBroadcast: Reading broadcast variable 12 took 8 ms
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 64.0 MiB, free 5.1 GiB)
23/09/24 20:12:26 INFO Executor: Finished task 0.0 in stage 8.0 (TID 13). 2434 bytes result sent to driver
23/09/24 20:12:26 INFO CoarseGrainedExecutorBackend: Got assigned task 18
23/09/24 20:12:26 INFO Executor: Running task 0.0 in stage 10.0 (TID 18)
23/09/24 20:12:26 INFO MapOutputTrackerWorker: Updating epoch to 4 and clearing cache
23/09/24 20:12:26 INFO TorrentBroadcast: Started reading broadcast variable 15 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 10.5 KiB, free 5.1 GiB)
23/09/24 20:12:26 INFO TorrentBroadcast: Reading broadcast variable 15 took 6 ms
23/09/24 20:12:26 INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 30.5 KiB, free 5.1 GiB)
23/09/24 20:12:26 INFO BlockManager: Found block rdd_40_0 locally
23/09/24 20:12:26 INFO Executor: Finished task 0.0 in stage 10.0 (TID 18). 2124 bytes result sent to driver
23/09/24 20:12:27 INFO CoarseGrainedExecutorBackend: Got assigned task 20
23/09/24 20:12:27 INFO Executor: Running task 0.0 in stage 11.0 (TID 20)
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 17 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 5.1 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 17 took 7 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 36.8 KiB, free 5.1 GiB)
23/09/24 20:12:27 INFO BlockManager: Found block rdd_40_0 locally
23/09/24 20:12:27 INFO CodeGenerator: Code generated in 13.563012 ms
23/09/24 20:12:27 INFO CodeGenerator: Code generated in 13.650235 ms
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 16 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 1669.0 B, free 5.1 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 16 took 15 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 64.0 MiB, free 5.0 GiB)
23/09/24 20:12:27 INFO Executor: Finished task 0.0 in stage 11.0 (TID 20). 2434 bytes result sent to driver
23/09/24 20:12:27 INFO CoarseGrainedExecutorBackend: Got assigned task 24
23/09/24 20:12:27 INFO Executor: Running task 0.0 in stage 13.0 (TID 24)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Updating epoch to 5 and clearing cache
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 19 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 10.5 KiB, free 5.0 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 19 took 6 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 30.5 KiB, free 5.0 GiB)
23/09/24 20:12:27 INFO BlockManager: Found block rdd_40_0 locally
23/09/24 20:12:27 INFO Executor: Finished task 0.0 in stage 13.0 (TID 24). 2124 bytes result sent to driver
23/09/24 20:12:27 INFO CoarseGrainedExecutorBackend: Got assigned task 26
23/09/24 20:12:27 INFO Executor: Running task 0.0 in stage 14.0 (TID 26)
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 21 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 5.0 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 21 took 137 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 36.8 KiB, free 5.0 GiB)
23/09/24 20:12:27 INFO BlockManager: Found block rdd_40_0 locally
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 20 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 1669.0 B, free 5.0 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 20 took 8 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 64.0 MiB, free 5.0 GiB)
23/09/24 20:12:27 INFO Executor: Finished task 0.0 in stage 14.0 (TID 26). 2477 bytes result sent to driver
23/09/24 20:12:27 INFO CoarseGrainedExecutorBackend: Got assigned task 29
23/09/24 20:12:27 INFO Executor: Running task 0.0 in stage 15.0 (TID 29)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Updating epoch to 6 and clearing cache
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 22 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.0 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 22 took 5 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_22 stored as values in memory (estimated size 10.1 KiB, free 5.0 GiB)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 5, fetching them
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:27 INFO ShuffleBlockFetcherIterator: Getting 3 (180.0 B) non-empty blocks including 1 (60.0 B) local and 2 (120.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 20:12:27 INFO CodeGenerator: Code generated in 7.588053 ms
23/09/24 20:12:27 INFO Executor: Finished task 0.0 in stage 15.0 (TID 29). 2450 bytes result sent to driver
23/09/24 20:12:27 INFO CoarseGrainedExecutorBackend: Got assigned task 32
23/09/24 20:12:27 INFO Executor: Running task 1.0 in stage 17.0 (TID 32)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Updating epoch to 7 and clearing cache
23/09/24 20:12:27 INFO TorrentBroadcast: Started reading broadcast variable 24 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 10.2 KiB, free 5.1 GiB)
23/09/24 20:12:27 INFO TorrentBroadcast: Reading broadcast variable 24 took 5 ms
23/09/24 20:12:27 INFO MemoryStore: Block broadcast_24 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 6, fetching them
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:27 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:27 INFO ShuffleBlockFetcherIterator: Getting 1 (1271.0 B) non-empty blocks including 0 (0.0 B) local and 1 (1271.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:27 INFO MemoryStore: Block rdd_88_1 stored as values in memory (estimated size 3.5 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 4.161525 ms
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 12.251466 ms
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 9.382128 ms
23/09/24 20:12:28 INFO MemoryStore: Block rdd_94_1 stored as values in memory (estimated size 3.1 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 9.523466 ms
23/09/24 20:12:28 INFO Executor: Finished task 1.0 in stage 17.0 (TID 32). 3204 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 38
23/09/24 20:12:28 INFO Executor: Running task 1.0 in stage 20.0 (TID 38)
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Updating epoch to 8 and clearing cache
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 26 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 26 took 6 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_26 stored as values in memory (estimated size 24.3 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 20:12:28 INFO Executor: Finished task 1.0 in stage 20.0 (TID 38). 3475 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 42
23/09/24 20:12:28 INFO Executor: Running task 0.0 in stage 21.0 (TID 42)
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 27 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 11.6 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 27 took 5 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_27 stored as values in memory (estimated size 34.2 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_40_0 locally
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 16.633075 ms
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 10.825845 ms
23/09/24 20:12:28 INFO Executor: Finished task 0.0 in stage 21.0 (TID 42). 3734 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 48
23/09/24 20:12:28 INFO Executor: Running task 0.0 in stage 25.0 (TID 48)
23/09/24 20:12:28 INFO MapOutputTrackerWorker: Updating epoch to 9 and clearing cache
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 32 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_32_piece0 stored as bytes in memory (estimated size 11.6 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 32 took 5 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_32 stored as values in memory (estimated size 34.2 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_40_0 locally
23/09/24 20:12:28 INFO Executor: Finished task 0.0 in stage 25.0 (TID 48). 3734 bytes result sent to driver
23/09/24 20:12:28 INFO CoarseGrainedExecutorBackend: Got assigned task 51
23/09/24 20:12:28 INFO Executor: Running task 0.0 in stage 26.0 (TID 51)
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 34 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_34_piece0 stored as bytes in memory (estimated size 16.9 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO TorrentBroadcast: Reading broadcast variable 34 took 5 ms
23/09/24 20:12:28 INFO MemoryStore: Block broadcast_34 stored as values in memory (estimated size 47.9 KiB, free 5.1 GiB)
23/09/24 20:12:28 INFO BlockManager: Found block rdd_40_0 locally
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 15.22021 ms
23/09/24 20:12:28 INFO CodeGenerator: Code generated in 20.552394 ms
23/09/24 20:12:28 INFO TorrentBroadcast: Started reading broadcast variable 33 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:29 INFO MemoryStore: Block broadcast_33_piece0 stored as bytes in memory (estimated size 5.9 KiB, free 5.1 GiB)
23/09/24 20:12:29 INFO TorrentBroadcast: Reading broadcast variable 33 took 7 ms
23/09/24 20:12:29 INFO MemoryStore: Block broadcast_33 stored as values in memory (estimated size 64.0 MiB, free 5.0 GiB)
23/09/24 20:12:29 INFO MemoryStore: Block rdd_141_0 stored as values in memory (estimated size 4.3 KiB, free 5.0 GiB)
23/09/24 20:12:29 INFO Executor: Finished task 0.0 in stage 26.0 (TID 51). 2644 bytes result sent to driver
23/09/24 20:12:29 INFO CoarseGrainedExecutorBackend: Got assigned task 54
23/09/24 20:12:29 INFO Executor: Running task 0.0 in stage 27.0 (TID 54)
23/09/24 20:12:29 INFO MapOutputTrackerWorker: Updating epoch to 10 and clearing cache
23/09/24 20:12:29 INFO TorrentBroadcast: Started reading broadcast variable 35 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:29 INFO MemoryStore: Block broadcast_35_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.0 GiB)
23/09/24 20:12:29 INFO TorrentBroadcast: Reading broadcast variable 35 took 4 ms
23/09/24 20:12:29 INFO MemoryStore: Block broadcast_35 stored as values in memory (estimated size 10.1 KiB, free 5.0 GiB)
23/09/24 20:12:29 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 9, fetching them
23/09/24 20:12:29 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:29 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:29 INFO ShuffleBlockFetcherIterator: Getting 3 (180.0 B) non-empty blocks including 1 (60.0 B) local and 2 (120.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:29 INFO Executor: Finished task 0.0 in stage 27.0 (TID 54). 2450 bytes result sent to driver
23/09/24 20:12:34 INFO BlockManager: Removing RDD 191
23/09/24 20:12:34 INFO BlockManager: Removing RDD 191
23/09/24 20:12:35 INFO CoarseGrainedExecutorBackend: Got assigned task 96
23/09/24 20:12:35 INFO Executor: Running task 1.0 in stage 68.0 (TID 96)
23/09/24 20:12:35 INFO TorrentBroadcast: Started reading broadcast variable 114 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_114_piece0 stored as bytes in memory (estimated size 17.8 KiB, free 5.1 GiB)
23/09/24 20:12:35 INFO TorrentBroadcast: Reading broadcast variable 114 took 6 ms
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_114 stored as values in memory (estimated size 63.4 KiB, free 5.1 GiB)
23/09/24 20:12:35 INFO BlockManager: Found block rdd_88_1 locally
23/09/24 20:12:35 INFO CodeGenerator: Code generated in 55.649672 ms
23/09/24 20:12:35 INFO CodeGenerator: Code generated in 8.799639 ms
23/09/24 20:12:35 INFO Executor: Finished task 1.0 in stage 68.0 (TID 96). 2574 bytes result sent to driver
23/09/24 20:12:35 INFO CoarseGrainedExecutorBackend: Got assigned task 98
23/09/24 20:12:35 INFO CoarseGrainedExecutorBackend: Got assigned task 101
23/09/24 20:12:35 INFO Executor: Running task 0.0 in stage 69.0 (TID 98)
23/09/24 20:12:35 INFO Executor: Running task 3.0 in stage 69.0 (TID 101)
23/09/24 20:12:35 INFO MapOutputTrackerWorker: Updating epoch to 11 and clearing cache
23/09/24 20:12:35 INFO TorrentBroadcast: Started reading broadcast variable 115 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_115_piece0 stored as bytes in memory (estimated size 7.8 KiB, free 5.1 GiB)
23/09/24 20:12:35 INFO TorrentBroadcast: Reading broadcast variable 115 took 5 ms
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_115 stored as values in memory (estimated size 19.4 KiB, free 5.1 GiB)
23/09/24 20:12:35 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 10, fetching them
23/09/24 20:12:35 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:35 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 10, fetching them
23/09/24 20:12:35 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:35 INFO ShuffleBlockFetcherIterator: Getting 4 (2.5 KiB) non-empty blocks including 1 (789.0 B) local and 3 (1733.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:35 INFO ShuffleBlockFetcherIterator: Getting 3 (861.0 B) non-empty blocks including 1 (334.0 B) local and 2 (527.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 20:12:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 20:12:35 INFO MemoryStore: Block rdd_254_0 stored as values in memory (estimated size 2.8 KiB, free 5.1 GiB)
23/09/24 20:12:35 INFO MemoryStore: Block rdd_254_3 stored as values in memory (estimated size 4.2 KiB, free 5.1 GiB)
23/09/24 20:12:35 INFO CodeGenerator: Code generated in 12.498612 ms
23/09/24 20:12:35 INFO CodeGenerator: Code generated in 10.647177 ms
23/09/24 20:12:35 INFO Executor: Finished task 0.0 in stage 69.0 (TID 98). 3008 bytes result sent to driver
23/09/24 20:12:35 INFO Executor: Finished task 3.0 in stage 69.0 (TID 101). 4256 bytes result sent to driver
23/09/24 20:12:35 INFO CoarseGrainedExecutorBackend: Got assigned task 102
23/09/24 20:12:35 INFO CoarseGrainedExecutorBackend: Got assigned task 105
23/09/24 20:12:35 INFO Executor: Running task 0.0 in stage 72.0 (TID 102)
23/09/24 20:12:35 INFO Executor: Running task 3.0 in stage 72.0 (TID 105)
23/09/24 20:12:35 INFO TorrentBroadcast: Started reading broadcast variable 117 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_117_piece0 stored as bytes in memory (estimated size 51.3 KiB, free 5.1 GiB)
23/09/24 20:12:35 INFO TorrentBroadcast: Reading broadcast variable 117 took 5 ms
23/09/24 20:12:35 INFO MemoryStore: Block broadcast_117 stored as values in memory (estimated size 139.3 KiB, free 5.1 GiB)
23/09/24 20:12:36 INFO BlockManager: Found block rdd_254_0 locally
23/09/24 20:12:36 INFO BlockManager: Found block rdd_254_3 locally
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 14.08717 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 13.422866 ms
23/09/24 20:12:36 INFO TorrentBroadcast: Started reading broadcast variable 116 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:36 INFO MemoryStore: Block broadcast_116_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 5.1 GiB)
23/09/24 20:12:36 INFO TorrentBroadcast: Reading broadcast variable 116 took 5 ms
23/09/24 20:12:36 INFO MemoryStore: Block broadcast_116 stored as values in memory (estimated size 7.6 KiB, free 5.1 GiB)
23/09/24 20:12:36 INFO MemoryStore: Block rdd_265_3 stored as values in memory (estimated size 7.2 KiB, free 5.1 GiB)
23/09/24 20:12:36 INFO MemoryStore: Block rdd_265_0 stored as values in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 12.282664 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 25.772216 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 7.66074 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 5.539386 ms
23/09/24 20:12:36 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
23/09/24 20:12:36 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
23/09/24 20:12:36 INFO MemoryStore: Block rdd_271_0 stored as values in memory (estimated size 5.2 KiB, free 5.1 GiB)
23/09/24 20:12:36 INFO MemoryStore: Block rdd_271_3 stored as values in memory (estimated size 7.7 KiB, free 5.1 GiB)
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 14.493496 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 11.669578 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 7.873532 ms
23/09/24 20:12:36 INFO CodeGenerator: Code generated in 4.615021 ms
23/09/24 20:12:36 INFO Executor: Finished task 3.0 in stage 72.0 (TID 105). 3068 bytes result sent to driver
23/09/24 20:12:36 INFO Executor: Finished task 0.0 in stage 72.0 (TID 102). 2919 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 107
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 109
23/09/24 20:12:37 INFO Executor: Running task 0.0 in stage 75.0 (TID 107)
23/09/24 20:12:37 INFO Executor: Running task 3.0 in stage 75.0 (TID 109)
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 118 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_118_piece0 stored as bytes in memory (estimated size 51.3 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 118 took 4 ms
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_118 stored as values in memory (estimated size 139.3 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO BlockManager: Found block rdd_271_0 locally
23/09/24 20:12:37 INFO BlockManager: Found block rdd_271_3 locally
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 8.806593 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 15.271335 ms
23/09/24 20:12:37 INFO Executor: Finished task 0.0 in stage 75.0 (TID 107). 2817 bytes result sent to driver
23/09/24 20:12:37 INFO Executor: Finished task 3.0 in stage 75.0 (TID 109). 3063 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 111
23/09/24 20:12:37 INFO Executor: Running task 1.0 in stage 77.0 (TID 111)
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 119 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_119_piece0 stored as bytes in memory (estimated size 7.1 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 119 took 9 ms
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_119 stored as values in memory (estimated size 15.1 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO BlockManager: Found block rdd_88_1 locally
23/09/24 20:12:37 INFO Executor: Finished task 1.0 in stage 77.0 (TID 111). 3197 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 116
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 117
23/09/24 20:12:37 INFO Executor: Running task 0.0 in stage 80.0 (TID 116)
23/09/24 20:12:37 INFO Executor: Running task 3.0 in stage 80.0 (TID 117)
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 121 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_121_piece0 stored as bytes in memory (estimated size 51.9 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 120
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 121
23/09/24 20:12:37 INFO Executor: Running task 0.0 in stage 82.0 (TID 120)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 121 took 5 ms
23/09/24 20:12:37 INFO Executor: Running task 3.0 in stage 82.0 (TID 121)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_121 stored as values in memory (estimated size 140.2 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 122 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_122_piece0 stored as bytes in memory (estimated size 51.9 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 122 took 5 ms
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_122 stored as values in memory (estimated size 140.3 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 12.766597 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 11.021955 ms
23/09/24 20:12:37 INFO BlockManager: Found block rdd_271_0 locally
23/09/24 20:12:37 INFO BlockManager: Found block rdd_271_0 locally
23/09/24 20:12:37 INFO BlockManager: Found block rdd_271_3 locally
23/09/24 20:12:37 INFO BlockManager: Found block rdd_271_3 locally
23/09/24 20:12:37 INFO Executor: Finished task 0.0 in stage 80.0 (TID 116). 3126 bytes result sent to driver
23/09/24 20:12:37 INFO Executor: Finished task 0.0 in stage 82.0 (TID 120). 3130 bytes result sent to driver
23/09/24 20:12:37 INFO Executor: Finished task 3.0 in stage 80.0 (TID 117). 3126 bytes result sent to driver
23/09/24 20:12:37 INFO Executor: Finished task 3.0 in stage 82.0 (TID 121). 3130 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 122
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 125
23/09/24 20:12:37 INFO Executor: Running task 0.0 in stage 81.0 (TID 122)
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 128
23/09/24 20:12:37 INFO Executor: Running task 3.0 in stage 81.0 (TID 125)
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 131
23/09/24 20:12:37 INFO Executor: Running task 9.0 in stage 81.0 (TID 131)
23/09/24 20:12:37 INFO Executor: Running task 6.0 in stage 81.0 (TID 128)
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Updating epoch to 12 and clearing cache
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 123 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_123_piece0 stored as bytes in memory (estimated size 53.6 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 123 took 6 ms
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_123 stored as values in memory (estimated size 147.6 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 136
23/09/24 20:12:37 INFO Executor: Running task 2.0 in stage 83.0 (TID 136)
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Updating epoch to 13 and clearing cache
23/09/24 20:12:37 INFO TorrentBroadcast: Started reading broadcast variable 124 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_124_piece0 stored as bytes in memory (estimated size 53.6 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO TorrentBroadcast: Reading broadcast variable 124 took 19 ms
23/09/24 20:12:37 INFO MemoryStore: Block broadcast_124 stored as values in memory (estimated size 147.6 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 11, fetching them
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 0 (0.0 B) local and 1 (368.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 0 (0.0 B) local and 1 (368.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (405.0 B) non-empty blocks including 1 (405.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 13, fetching them
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 1 (368.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 0 (0.0 B) local and 1 (445.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 25.991038 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 9.395345 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 13.449837 ms
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 27.941611 ms
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_2 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 20:12:37 INFO MemoryStore: Block rdd_283_0 stored as values in memory (estimated size 4.7 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO MemoryStore: Block rdd_283_3 stored as values in memory (estimated size 4.8 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO MemoryStore: Block rdd_283_6 stored as values in memory (estimated size 4.8 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO MemoryStore: Block rdd_283_9 stored as values in memory (estimated size 4.7 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 19.444613 ms
23/09/24 20:12:37 INFO Executor: Finished task 2.0 in stage 83.0 (TID 136). 4580 bytes result sent to driver
23/09/24 20:12:37 INFO CodeGenerator: Code generated in 27.590049 ms
23/09/24 20:12:37 INFO Executor: Finished task 3.0 in stage 81.0 (TID 125). 4580 bytes result sent to driver
23/09/24 20:12:37 INFO Executor: Finished task 9.0 in stage 81.0 (TID 131). 4580 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 147
23/09/24 20:12:37 INFO Executor: Running task 13.0 in stage 83.0 (TID 147)
23/09/24 20:12:37 INFO Executor: Finished task 6.0 in stage 81.0 (TID 128). 4580 bytes result sent to driver
23/09/24 20:12:37 INFO Executor: Finished task 0.0 in stage 81.0 (TID 122). 4580 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 148
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 149
23/09/24 20:12:37 INFO Executor: Running task 14.0 in stage 83.0 (TID 148)
23/09/24 20:12:37 INFO Executor: Running task 15.0 in stage 83.0 (TID 149)
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 1 (445.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 1 (445.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_13 stored as values in memory (estimated size 4.8 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO Executor: Finished task 13.0 in stage 83.0 (TID 147). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_14 stored as values in memory (estimated size 4.8 KiB, free 5.1 GiB)
23/09/24 20:12:37 INFO Executor: Finished task 14.0 in stage 83.0 (TID 148). 4537 bytes result sent to driver
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:37 INFO MemoryStore: Block rdd_302_15 stored as values in memory (estimated size 16.0 B, free 5.1 GiB)
23/09/24 20:12:37 INFO Executor: Finished task 15.0 in stage 83.0 (TID 149). 4408 bytes result sent to driver
23/09/24 20:12:37 INFO CoarseGrainedExecutorBackend: Got assigned task 151
23/09/24 20:12:37 INFO Executor: Running task 1.0 in stage 84.0 (TID 151)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Updating epoch to 15 and clearing cache
23/09/24 20:12:38 INFO TorrentBroadcast: Started reading broadcast variable 125 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_125_piece0 stored as bytes in memory (estimated size 75.0 KiB, free 5.1 GiB)
23/09/24 20:12:38 INFO TorrentBroadcast: Reading broadcast variable 125 took 4 ms
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_125 stored as values in memory (estimated size 233.4 KiB, free 5.1 GiB)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 14, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 15 (6.3 KiB) non-empty blocks including 3 (1335.0 B) local and 12 (5.0 KiB) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 7.694794 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 10.254204 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 14.519676 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 47.013047 ms
23/09/24 20:12:38 INFO MemoryStore: Block rdd_323_1 stored as values in memory (estimated size 7.5 KiB, free 5.1 GiB)
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 13.894807 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 4.24377 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 5.971852 ms
23/09/24 20:12:38 INFO TorrentBroadcast: Started reading broadcast variable 120 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_120_piece0 stored as bytes in memory (estimated size 3.7 KiB, free 5.1 GiB)
23/09/24 20:12:38 INFO TorrentBroadcast: Reading broadcast variable 120 took 5 ms
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_120 stored as values in memory (estimated size 7.4 KiB, free 5.1 GiB)
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 3.965405 ms
23/09/24 20:12:38 INFO Executor: Finished task 1.0 in stage 84.0 (TID 151). 7923 bytes result sent to driver
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 155
23/09/24 20:12:38 INFO Executor: Running task 1.0 in stage 91.0 (TID 155)
23/09/24 20:12:38 INFO TorrentBroadcast: Started reading broadcast variable 126 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_126_piece0 stored as bytes in memory (estimated size 75.1 KiB, free 5.1 GiB)
23/09/24 20:12:38 INFO TorrentBroadcast: Reading broadcast variable 126 took 4 ms
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_126 stored as values in memory (estimated size 234.1 KiB, free 5.1 GiB)
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 3.975765 ms
23/09/24 20:12:38 INFO BlockManager: Found block rdd_323_1 locally
23/09/24 20:12:38 INFO Executor: Finished task 1.0 in stage 91.0 (TID 155). 7948 bytes result sent to driver
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 158
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 161
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 164
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 167
23/09/24 20:12:38 INFO CoarseGrainedExecutorBackend: Got assigned task 170
23/09/24 20:12:38 INFO Executor: Running task 9.0 in stage 92.0 (TID 167)
23/09/24 20:12:38 INFO Executor: Running task 12.0 in stage 92.0 (TID 170)
23/09/24 20:12:38 INFO Executor: Running task 3.0 in stage 92.0 (TID 161)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Updating epoch to 16 and clearing cache
23/09/24 20:12:38 INFO TorrentBroadcast: Started reading broadcast variable 127 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 20:12:38 INFO Executor: Running task 0.0 in stage 92.0 (TID 158)
23/09/24 20:12:38 INFO Executor: Running task 6.0 in stage 92.0 (TID 164)
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_127_piece0 stored as bytes in memory (estimated size 129.5 KiB, free 5.1 GiB)
23/09/24 20:12:38 INFO TorrentBroadcast: Reading broadcast variable 127 took 6 ms
23/09/24 20:12:38 INFO MemoryStore: Block broadcast_127 stored as values in memory (estimated size 379.7 KiB, free 5.1 GiB)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 15, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:39929)
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 15, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 15, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 15, fetching them
23/09/24 20:12:38 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 0 (0.0 B) local and 2 (580.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 0 (0.0 B) local and 2 (580.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 1 (276.0 B) local and 1 (304.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 2 (552.0 B) non-empty blocks including 1 (276.0 B) local and 1 (276.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 0 (0.0 B) local and 2 (608.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 29.666006 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 16.392802 ms
23/09/24 20:12:38 INFO CodeGenerator: Code generated in 18.632951 ms
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:38 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:38 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:38 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:38 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:38 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:38 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:38 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:38 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:38 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:38 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:38 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:38 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:38 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:38 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:38 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:38 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:38 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:38 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:38 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:38 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:38 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:38 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:38 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:38 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:38 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:38 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:38 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:38 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:38 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:38 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:38 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:38 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "z_zid",
"type" : "long",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_cluster",
"type" : "string",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_prediction",
"type" : "double",
"nullable" : false,
"metadata" : {
"ml_attr" : {
"type" : "nominal",
"num_vals" : 2
}
}
}, {
"name" : "z_score",
"type" : "double",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_isMatch",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "fname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "lname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "stNo",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add1",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add2",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "city",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "areacode",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "state",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "dob",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "ssn",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_source",
"type" : "string",
"nullable" : false,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int64 z_zid;
required binary z_cluster (UTF8);
required double z_prediction;
optional double z_score;
required int32 z_isMatch;
optional binary fname (UTF8);
optional binary lname (UTF8);
optional binary stNo (UTF8);
optional binary add1 (UTF8);
optional binary add2 (UTF8);
optional binary city (UTF8);
optional binary areacode (UTF8);
optional binary state (UTF8);
optional binary dob (UTF8);
optional binary ssn (UTF8);
required binary z_source (UTF8);
}
23/09/24 20:12:38 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "z_zid",
"type" : "long",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_cluster",
"type" : "string",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_prediction",
"type" : "double",
"nullable" : false,
"metadata" : {
"ml_attr" : {
"type" : "nominal",
"num_vals" : 2
}
}
}, {
"name" : "z_score",
"type" : "double",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_isMatch",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "fname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "lname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "stNo",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add1",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add2",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "city",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "areacode",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "state",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "dob",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "ssn",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_source",
"type" : "string",
"nullable" : false,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int64 z_zid;
required binary z_cluster (UTF8);
required double z_prediction;
optional double z_score;
required int32 z_isMatch;
optional binary fname (UTF8);
optional binary lname (UTF8);
optional binary stNo (UTF8);
optional binary add1 (UTF8);
optional binary add2 (UTF8);
optional binary city (UTF8);
optional binary areacode (UTF8);
optional binary state (UTF8);
optional binary dob (UTF8);
optional binary ssn (UTF8);
required binary z_source (UTF8);
}
23/09/24 20:12:38 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "z_zid",
"type" : "long",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_cluster",
"type" : "string",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_prediction",
"type" : "double",
"nullable" : false,
"metadata" : {
"ml_attr" : {
"type" : "nominal",
"num_vals" : 2
}
}
}, {
"name" : "z_score",
"type" : "double",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_isMatch",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "fname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "lname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "stNo",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add1",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add2",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "city",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "areacode",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "state",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "dob",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "ssn",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_source",
"type" : "string",
"nullable" : false,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int64 z_zid;
required binary z_cluster (UTF8);
required double z_prediction;
optional double z_score;
required int32 z_isMatch;
optional binary fname (UTF8);
optional binary lname (UTF8);
optional binary stNo (UTF8);
optional binary add1 (UTF8);
optional binary add2 (UTF8);
optional binary city (UTF8);
optional binary areacode (UTF8);
optional binary state (UTF8);
optional binary dob (UTF8);
optional binary ssn (UTF8);
required binary z_source (UTF8);
}
23/09/24 20:12:38 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "z_zid",
"type" : "long",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_cluster",
"type" : "string",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_prediction",
"type" : "double",
"nullable" : false,
"metadata" : {
"ml_attr" : {
"type" : "nominal",
"num_vals" : 2
}
}
}, {
"name" : "z_score",
"type" : "double",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_isMatch",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "fname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "lname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "stNo",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add1",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add2",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "city",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "areacode",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "state",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "dob",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "ssn",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_source",
"type" : "string",
"nullable" : false,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int64 z_zid;
required binary z_cluster (UTF8);
required double z_prediction;
optional double z_score;
required int32 z_isMatch;
optional binary fname (UTF8);
optional binary lname (UTF8);
optional binary stNo (UTF8);
optional binary add1 (UTF8);
optional binary add2 (UTF8);
optional binary city (UTF8);
optional binary areacode (UTF8);
optional binary state (UTF8);
optional binary dob (UTF8);
optional binary ssn (UTF8);
required binary z_source (UTF8);
}
23/09/24 20:12:38 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "z_zid",
"type" : "long",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_cluster",
"type" : "string",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_prediction",
"type" : "double",
"nullable" : false,
"metadata" : {
"ml_attr" : {
"type" : "nominal",
"num_vals" : 2
}
}
}, {
"name" : "z_score",
"type" : "double",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_isMatch",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "fname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "lname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "stNo",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add1",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add2",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "city",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "areacode",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "state",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "dob",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "ssn",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_source",
"type" : "string",
"nullable" : false,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int64 z_zid;
required binary z_cluster (UTF8);
required double z_prediction;
optional double z_score;
required int32 z_isMatch;
optional binary fname (UTF8);
optional binary lname (UTF8);
optional binary stNo (UTF8);
optional binary add1 (UTF8);
optional binary add2 (UTF8);
optional binary city (UTF8);
optional binary areacode (UTF8);
optional binary state (UTF8);
optional binary dob (UTF8);
optional binary ssn (UTF8);
required binary z_source (UTF8);
}
23/09/24 20:12:38 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 20:12:38 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 20:12:38 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 20:12:38 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 20:12:38 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 20:12:38 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 333
23/09/24 20:12:38 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 418
23/09/24 20:12:38 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 438
23/09/24 20:12:38 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 436
23/09/24 20:12:38 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 451
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_20230924201238364040899019752049_0092_m_000012_170' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_20230924201238364040899019752049_0092_m_000012
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012387111620466771559972_0092_m_000009_167' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012387111620466771559972_0092_m_000009
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_20230924201238364040899019752049_0092_m_000012_170: Committed
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012384772723316017148804_0092_m_000000_158' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012384772723316017148804_0092_m_000000
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012387111620466771559972_0092_m_000009_167: Committed
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_20230924201238288529523948432609_0092_m_000003_161' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_20230924201238288529523948432609_0092_m_000003
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_20230924201238288529523948432609_0092_m_000003_161: Committed
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012384772723316017148804_0092_m_000000_158: Committed
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012382991396801171011127_0092_m_000006_164' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012382991396801171011127_0092_m_000006
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012382991396801171011127_0092_m_000006_164: Committed
23/09/24 20:12:39 INFO Executor: Finished task 6.0 in stage 92.0 (TID 164). 10148 bytes result sent to driver
23/09/24 20:12:39 INFO Executor: Finished task 12.0 in stage 92.0 (TID 170). 10191 bytes result sent to driver
23/09/24 20:12:39 INFO Executor: Finished task 0.0 in stage 92.0 (TID 158). 10148 bytes result sent to driver
23/09/24 20:12:39 INFO Executor: Finished task 9.0 in stage 92.0 (TID 167). 10148 bytes result sent to driver
23/09/24 20:12:39 INFO Executor: Finished task 3.0 in stage 92.0 (TID 161). 10148 bytes result sent to driver
23/09/24 20:12:39 INFO CoarseGrainedExecutorBackend: Got assigned task 173
23/09/24 20:12:39 INFO CoarseGrainedExecutorBackend: Got assigned task 174
23/09/24 20:12:39 INFO Executor: Running task 15.0 in stage 92.0 (TID 173)
23/09/24 20:12:39 INFO Executor: Running task 16.0 in stage 92.0 (TID 174)
23/09/24 20:12:39 INFO CoarseGrainedExecutorBackend: Got assigned task 175
23/09/24 20:12:39 INFO CoarseGrainedExecutorBackend: Got assigned task 176
23/09/24 20:12:39 INFO CoarseGrainedExecutorBackend: Got assigned task 177
23/09/24 20:12:39 INFO Executor: Running task 18.0 in stage 92.0 (TID 176)
23/09/24 20:12:39 INFO Executor: Running task 17.0 in stage 92.0 (TID 175)
23/09/24 20:12:39 INFO Executor: Running task 19.0 in stage 92.0 (TID 177)
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 1 (276.0 B) local and 1 (304.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:39 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:39 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:39 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:39 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:39 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:39 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:39 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:39 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:39 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "z_zid",
"type" : "long",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_cluster",
"type" : "string",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_prediction",
"type" : "double",
"nullable" : false,
"metadata" : {
"ml_attr" : {
"type" : "nominal",
"num_vals" : 2
}
}
}, {
"name" : "z_score",
"type" : "double",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_isMatch",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "fname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "lname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "stNo",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add1",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add2",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "city",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "areacode",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "state",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "dob",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "ssn",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_source",
"type" : "string",
"nullable" : false,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int64 z_zid;
required binary z_cluster (UTF8);
required double z_prediction;
optional double z_score;
required int32 z_isMatch;
optional binary fname (UTF8);
optional binary lname (UTF8);
optional binary stNo (UTF8);
optional binary add1 (UTF8);
optional binary add2 (UTF8);
optional binary city (UTF8);
optional binary areacode (UTF8);
optional binary state (UTF8);
optional binary dob (UTF8);
optional binary ssn (UTF8);
required binary z_source (UTF8);
}
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Getting 2 (552.0 B) non-empty blocks including 1 (276.0 B) local and 1 (276.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Getting 2 (527.0 B) non-empty blocks including 1 (251.0 B) local and 1 (276.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 0 (0.0 B) local and 2 (580.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 423
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:39 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:39 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:39 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:39 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:39 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:39 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:39 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:39 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:39 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:39 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:39 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:39 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:39 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:39 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:39 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:39 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:39 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:39 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:39 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:39 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:39 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Getting 2 (552.0 B) non-empty blocks including 0 (0.0 B) local and 2 (552.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 20:12:39 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:39 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:39 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 20:12:39 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "z_zid",
"type" : "long",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_cluster",
"type" : "string",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_prediction",
"type" : "double",
"nullable" : false,
"metadata" : {
"ml_attr" : {
"type" : "nominal",
"num_vals" : 2
}
}
}, {
"name" : "z_score",
"type" : "double",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_isMatch",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "fname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "lname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "stNo",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add1",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add2",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "city",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "areacode",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "state",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "dob",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "ssn",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_source",
"type" : "string",
"nullable" : false,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int64 z_zid;
required binary z_cluster (UTF8);
required double z_prediction;
optional double z_score;
required int32 z_isMatch;
optional binary fname (UTF8);
optional binary lname (UTF8);
optional binary stNo (UTF8);
optional binary add1 (UTF8);
optional binary add2 (UTF8);
optional binary city (UTF8);
optional binary areacode (UTF8);
optional binary state (UTF8);
optional binary dob (UTF8);
optional binary ssn (UTF8);
required binary z_source (UTF8);
}
23/09/24 20:12:39 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "z_zid",
"type" : "long",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_cluster",
"type" : "string",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_prediction",
"type" : "double",
"nullable" : false,
"metadata" : {
"ml_attr" : {
"type" : "nominal",
"num_vals" : 2
}
}
}, {
"name" : "z_score",
"type" : "double",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_isMatch",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "fname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "lname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "stNo",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add1",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add2",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "city",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "areacode",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "state",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "dob",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "ssn",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_source",
"type" : "string",
"nullable" : false,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int64 z_zid;
required binary z_cluster (UTF8);
required double z_prediction;
optional double z_score;
required int32 z_isMatch;
optional binary fname (UTF8);
optional binary lname (UTF8);
optional binary stNo (UTF8);
optional binary add1 (UTF8);
optional binary add2 (UTF8);
optional binary city (UTF8);
optional binary areacode (UTF8);
optional binary state (UTF8);
optional binary dob (UTF8);
optional binary ssn (UTF8);
required binary z_source (UTF8);
}
23/09/24 20:12:39 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "z_zid",
"type" : "long",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_cluster",
"type" : "string",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_prediction",
"type" : "double",
"nullable" : false,
"metadata" : {
"ml_attr" : {
"type" : "nominal",
"num_vals" : 2
}
}
}, {
"name" : "z_score",
"type" : "double",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_isMatch",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "fname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "lname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "stNo",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add1",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add2",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "city",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "areacode",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "state",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "dob",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "ssn",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_source",
"type" : "string",
"nullable" : false,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int64 z_zid;
required binary z_cluster (UTF8);
required double z_prediction;
optional double z_score;
required int32 z_isMatch;
optional binary fname (UTF8);
optional binary lname (UTF8);
optional binary stNo (UTF8);
optional binary add1 (UTF8);
optional binary add2 (UTF8);
optional binary city (UTF8);
optional binary areacode (UTF8);
optional binary state (UTF8);
optional binary dob (UTF8);
optional binary ssn (UTF8);
required binary z_source (UTF8);
}
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 20:12:39 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012385265624198656218264_0092_m_000019_177' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012385265624198656218264_0092_m_000019
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012385265624198656218264_0092_m_000019_177: Committed
23/09/24 20:12:39 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:39 INFO CodecConfig: Compression: SNAPPY
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 20:12:39 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 20:12:39 INFO ParquetOutputFormat: Dictionary is on
23/09/24 20:12:39 INFO ParquetOutputFormat: Validation is off
23/09/24 20:12:39 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 20:12:39 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 20:12:39 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 20:12:39 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 20:12:39 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 20:12:39 INFO Executor: Finished task 19.0 in stage 92.0 (TID 177). 10191 bytes result sent to driver
23/09/24 20:12:39 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 328
23/09/24 20:12:39 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "z_zid",
"type" : "long",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_cluster",
"type" : "string",
"nullable" : false,
"metadata" : { }
}, {
"name" : "z_prediction",
"type" : "double",
"nullable" : false,
"metadata" : {
"ml_attr" : {
"type" : "nominal",
"num_vals" : 2
}
}
}, {
"name" : "z_score",
"type" : "double",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_isMatch",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "fname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "lname",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "stNo",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add1",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "add2",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "city",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "areacode",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "state",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "dob",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "ssn",
"type" : "string",
"nullable" : true,
"metadata" : { }
}, {
"name" : "z_source",
"type" : "string",
"nullable" : false,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int64 z_zid;
required binary z_cluster (UTF8);
required double z_prediction;
optional double z_score;
required int32 z_isMatch;
optional binary fname (UTF8);
optional binary lname (UTF8);
optional binary stNo (UTF8);
optional binary add1 (UTF8);
optional binary add2 (UTF8);
optional binary city (UTF8);
optional binary areacode (UTF8);
optional binary state (UTF8);
optional binary dob (UTF8);
optional binary ssn (UTF8);
required binary z_source (UTF8);
}
23/09/24 20:12:39 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 328
23/09/24 20:12:39 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 435
23/09/24 20:12:39 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 343
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012385588034870762456689_0092_m_000017_175' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012385588034870762456689_0092_m_000017
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012385588034870762456689_0092_m_000017_175: Committed
23/09/24 20:12:39 INFO Executor: Finished task 17.0 in stage 92.0 (TID 175). 10191 bytes result sent to driver
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_20230924201238747853410619179763_0092_m_000016_174' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_20230924201238747853410619179763_0092_m_000016
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_20230924201238747853410619179763_0092_m_000016_174: Committed
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_202309242012383249739823008976281_0092_m_000015_173' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242012383249739823008976281_0092_m_000015
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_202309242012383249739823008976281_0092_m_000015_173: Committed
23/09/24 20:12:39 INFO Executor: Finished task 16.0 in stage 92.0 (TID 174). 10191 bytes result sent to driver
23/09/24 20:12:39 INFO Executor: Finished task 15.0 in stage 92.0 (TID 173). 10191 bytes result sent to driver
23/09/24 20:12:39 INFO FileOutputCommitter: Saved output of task 'attempt_20230924201238122923725294423608_0092_m_000018_176' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_20230924201238122923725294423608_0092_m_000018
23/09/24 20:12:39 INFO SparkHadoopMapRedUtil: attempt_20230924201238122923725294423608_0092_m_000018_176: Committed
23/09/24 20:12:39 INFO Executor: Finished task 18.0 in stage 92.0 (TID 176). 10191 bytes result sent to driver
23/09/24 20:12:39 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
23/09/24 20:12:39 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
tdown