9/24 21:18:19 INFO TorrentBroadcast: Started reading broadcast variable 19 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:19 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 10.5 KiB, free 5.0 GiB)
23/09/24 21:18:19 INFO TorrentBroadcast: Reading broadcast variable 19 took 7 ms
23/09/24 21:18:19 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 30.5 KiB, free 5.0 GiB)
23/09/24 21:18:19 INFO BlockManager: Found block rdd_40_1 locally
23/09/24 21:18:19 INFO Executor: Finished task 1.0 in stage 13.0 (TID 23). 2131 bytes result sent to driver
23/09/24 21:18:19 INFO CoarseGrainedExecutorBackend: Got assigned task 26
23/09/24 21:18:19 INFO Executor: Running task 1.0 in stage 14.0 (TID 26)
23/09/24 21:18:19 INFO TorrentBroadcast: Started reading broadcast variable 21 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:19 INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 13.7 KiB, free 5.0 GiB)
23/09/24 21:18:19 INFO TorrentBroadcast: Reading broadcast variable 21 took 11 ms
23/09/24 21:18:19 INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 36.8 KiB, free 5.0 GiB)
23/09/24 21:18:19 INFO BlockManager: Found block rdd_40_1 locally
23/09/24 21:18:19 INFO TorrentBroadcast: Started reading broadcast variable 20 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:19 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 1669.0 B, free 5.0 GiB)
23/09/24 21:18:19 INFO TorrentBroadcast: Reading broadcast variable 20 took 20 ms
23/09/24 21:18:19 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 64.0 MiB, free 5.0 GiB)
23/09/24 21:18:19 INFO Executor: Finished task 1.0 in stage 14.0 (TID 26). 2434 bytes result sent to driver
23/09/24 21:18:20 INFO CoarseGrainedExecutorBackend: Got assigned task 32
23/09/24 21:18:20 INFO Executor: Running task 1.0 in stage 17.0 (TID 32)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Updating epoch to 7 and clearing cache
23/09/24 21:18:20 INFO TorrentBroadcast: Started reading broadcast variable 24 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 10.2 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO TorrentBroadcast: Reading broadcast variable 24 took 6 ms
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_24 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 6, fetching them
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:20 INFO ShuffleBlockFetcherIterator: Getting 1 (1399.0 B) non-empty blocks including 0 (0.0 B) local and 1 (1399.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 11 ms
23/09/24 21:18:20 INFO MemoryStore: Block rdd_88_1 stored as values in memory (estimated size 3.5 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO CodeGenerator: Code generated in 4.605057 ms
23/09/24 21:18:20 INFO CodeGenerator: Code generated in 12.554025 ms
23/09/24 21:18:20 INFO CodeGenerator: Code generated in 9.828233 ms
23/09/24 21:18:20 INFO MemoryStore: Block rdd_94_1 stored as values in memory (estimated size 3.2 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO CodeGenerator: Code generated in 8.854596 ms
23/09/24 21:18:20 INFO CodeGenerator: Code generated in 5.562074 ms
23/09/24 21:18:20 INFO Executor: Finished task 1.0 in stage 17.0 (TID 32). 3204 bytes result sent to driver
23/09/24 21:18:20 INFO CoarseGrainedExecutorBackend: Got assigned task 38
23/09/24 21:18:20 INFO Executor: Running task 1.0 in stage 20.0 (TID 38)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Updating epoch to 8 and clearing cache
23/09/24 21:18:20 INFO TorrentBroadcast: Started reading broadcast variable 26 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 10.2 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO TorrentBroadcast: Reading broadcast variable 26 took 6 ms
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_26 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:20 INFO Executor: Finished task 1.0 in stage 20.0 (TID 38). 2989 bytes result sent to driver
23/09/24 21:18:20 INFO CoarseGrainedExecutorBackend: Got assigned task 40
23/09/24 21:18:20 INFO Executor: Running task 0.0 in stage 21.0 (TID 40)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Updating epoch to 9 and clearing cache
23/09/24 21:18:20 INFO TorrentBroadcast: Started reading broadcast variable 27 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO TorrentBroadcast: Reading broadcast variable 27 took 5 ms
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_27 stored as values in memory (estimated size 10.1 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 8, fetching them
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:20 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 21:18:20 INFO CodeGenerator: Code generated in 7.043398 ms
23/09/24 21:18:20 INFO Executor: Finished task 0.0 in stage 21.0 (TID 40). 2450 bytes result sent to driver
23/09/24 21:18:20 INFO CoarseGrainedExecutorBackend: Got assigned task 43
23/09/24 21:18:20 INFO Executor: Running task 1.0 in stage 25.0 (TID 43)
23/09/24 21:18:20 INFO TorrentBroadcast: Started reading broadcast variable 29 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 10.2 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO TorrentBroadcast: Reading broadcast variable 29 took 6 ms
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_29 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:20 INFO Executor: Finished task 1.0 in stage 25.0 (TID 43). 2989 bytes result sent to driver
23/09/24 21:18:20 INFO CoarseGrainedExecutorBackend: Got assigned task 46
23/09/24 21:18:20 INFO Executor: Running task 0.0 in stage 26.0 (TID 46)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Updating epoch to 10 and clearing cache
23/09/24 21:18:20 INFO TorrentBroadcast: Started reading broadcast variable 30 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO TorrentBroadcast: Reading broadcast variable 30 took 6 ms
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_30 stored as values in memory (estimated size 10.1 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 9, fetching them
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:20 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:20 INFO Executor: Finished task 0.0 in stage 26.0 (TID 46). 2450 bytes result sent to driver
23/09/24 21:18:20 INFO CoarseGrainedExecutorBackend: Got assigned task 49
23/09/24 21:18:20 INFO Executor: Running task 1.0 in stage 28.0 (TID 49)
23/09/24 21:18:20 INFO TorrentBroadcast: Started reading broadcast variable 31 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_31_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO TorrentBroadcast: Reading broadcast variable 31 took 6 ms
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_31 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:20 INFO Executor: Finished task 1.0 in stage 28.0 (TID 49). 2989 bytes result sent to driver
23/09/24 21:18:20 INFO CoarseGrainedExecutorBackend: Got assigned task 52
23/09/24 21:18:20 INFO Executor: Running task 1.0 in stage 31.0 (TID 52)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Updating epoch to 11 and clearing cache
23/09/24 21:18:20 INFO TorrentBroadcast: Started reading broadcast variable 33 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_33_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO TorrentBroadcast: Reading broadcast variable 33 took 6 ms
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_33 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:20 INFO Executor: Finished task 1.0 in stage 31.0 (TID 52). 2989 bytes result sent to driver
23/09/24 21:18:20 INFO CoarseGrainedExecutorBackend: Got assigned task 57
23/09/24 21:18:20 INFO Executor: Running task 1.0 in stage 34.0 (TID 57)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Updating epoch to 12 and clearing cache
23/09/24 21:18:20 INFO TorrentBroadcast: Started reading broadcast variable 35 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_35_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO TorrentBroadcast: Reading broadcast variable 35 took 5 ms
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_35 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:20 INFO Executor: Finished task 1.0 in stage 34.0 (TID 57). 2989 bytes result sent to driver
23/09/24 21:18:20 INFO CoarseGrainedExecutorBackend: Got assigned task 64
23/09/24 21:18:20 INFO Executor: Running task 1.0 in stage 37.0 (TID 64)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Updating epoch to 13 and clearing cache
23/09/24 21:18:20 INFO TorrentBroadcast: Started reading broadcast variable 37 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_37_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO TorrentBroadcast: Reading broadcast variable 37 took 8 ms
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_37 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:20 INFO Executor: Finished task 1.0 in stage 37.0 (TID 64). 2990 bytes result sent to driver
23/09/24 21:18:20 INFO CoarseGrainedExecutorBackend: Got assigned task 68
23/09/24 21:18:20 INFO Executor: Running task 1.0 in stage 40.0 (TID 68)
23/09/24 21:18:20 INFO MapOutputTrackerWorker: Updating epoch to 14 and clearing cache
23/09/24 21:18:20 INFO TorrentBroadcast: Started reading broadcast variable 39 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_39_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO TorrentBroadcast: Reading broadcast variable 39 took 5 ms
23/09/24 21:18:20 INFO MemoryStore: Block broadcast_39 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:20 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:20 INFO Executor: Finished task 1.0 in stage 40.0 (TID 68). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 71
23/09/24 21:18:21 INFO Executor: Running task 0.0 in stage 41.0 (TID 71)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 15 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 40 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_40_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 40 took 5 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_40 stored as values in memory (estimated size 10.1 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 14, fetching them
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:21 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:21 INFO Executor: Finished task 0.0 in stage 41.0 (TID 71). 2450 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 74
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 43.0 (TID 74)
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 41 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_41_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 41 took 5 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_41 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 43.0 (TID 74). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 77
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 46.0 (TID 77)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 16 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 43 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_43_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 43 took 4 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_43 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 46.0 (TID 77). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 81
23/09/24 21:18:21 INFO Executor: Running task 0.0 in stage 47.0 (TID 81)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 17 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 44 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_44_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 44 took 3 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_44 stored as values in memory (estimated size 10.1 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 16, fetching them
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:21 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:21 INFO Executor: Finished task 0.0 in stage 47.0 (TID 81). 2450 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 82
23/09/24 21:18:21 INFO Executor: Running task 0.0 in stage 49.0 (TID 82)
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 45 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_45_piece0 stored as bytes in memory (estimated size 9.7 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 45 took 4 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_45 stored as values in memory (estimated size 24.5 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 0.0 in stage 49.0 (TID 82). 3605 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 83
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 51.0 (TID 83)
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 46 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_46_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 46 took 7 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_46 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 51.0 (TID 83). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 90
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 54.0 (TID 90)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 18 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 48 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_48_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 48 took 6 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_48 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 54.0 (TID 90). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 92
23/09/24 21:18:21 INFO Executor: Running task 0.0 in stage 55.0 (TID 92)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 19 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 49 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_49_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 49 took 5 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_49 stored as values in memory (estimated size 10.1 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 18, fetching them
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:21 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:21 INFO Executor: Finished task 0.0 in stage 55.0 (TID 92). 2450 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 93
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 57.0 (TID 93)
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 50 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_50_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 50 took 4 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_50 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 57.0 (TID 93). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 100
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 60.0 (TID 100)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 20 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 52 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_52_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 52 took 5 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_52 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 60.0 (TID 100). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 103
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 63.0 (TID 103)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 21 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 54 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_54_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 54 took 4 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_54 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 63.0 (TID 103). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 110
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 66.0 (TID 110)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 22 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 56 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_56_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 56 took 4 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_56 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 66.0 (TID 110). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 112
23/09/24 21:18:21 INFO Executor: Running task 0.0 in stage 67.0 (TID 112)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 23 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 57 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_57_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 57 took 4 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_57 stored as values in memory (estimated size 10.1 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 22, fetching them
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:21 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:21 INFO Executor: Finished task 0.0 in stage 67.0 (TID 112). 2450 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 113
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 69.0 (TID 113)
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 58 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_58_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 58 took 4 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_58 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 69.0 (TID 113). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 120
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 72.0 (TID 120)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 24 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 60 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_60_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 60 took 4 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_60 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 72.0 (TID 120). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 124
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 77.0 (TID 124)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 25 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 63 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_63_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 63 took 5 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_63 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 77.0 (TID 124). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 129
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 80.0 (TID 129)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 26 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 65 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_65_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 65 took 4 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_65 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 80.0 (TID 129). 2990 bytes result sent to driver
23/09/24 21:18:21 INFO CoarseGrainedExecutorBackend: Got assigned task 136
23/09/24 21:18:21 INFO Executor: Running task 1.0 in stage 83.0 (TID 136)
23/09/24 21:18:21 INFO MapOutputTrackerWorker: Updating epoch to 27 and clearing cache
23/09/24 21:18:21 INFO TorrentBroadcast: Started reading broadcast variable 67 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_67_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO TorrentBroadcast: Reading broadcast variable 67 took 5 ms
23/09/24 21:18:21 INFO MemoryStore: Block broadcast_67 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:21 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:21 INFO Executor: Finished task 1.0 in stage 83.0 (TID 136). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 141
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 86.0 (TID 141)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 28 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 69 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_69_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 69 took 4 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_69 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 86.0 (TID 141). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 143
23/09/24 21:18:22 INFO Executor: Running task 0.0 in stage 87.0 (TID 143)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 29 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 70 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_70_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 70 took 3 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_70 stored as values in memory (estimated size 10.1 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 28, fetching them
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:22 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:22 INFO Executor: Finished task 0.0 in stage 87.0 (TID 143). 2450 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 146
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 89.0 (TID 146)
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 71 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_71_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 71 took 4 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_71 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 89.0 (TID 146). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 149
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 92.0 (TID 149)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 30 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 73 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_73_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 73 took 3 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_73 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 92.0 (TID 149). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 156
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 95.0 (TID 156)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 31 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 75 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_75_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 75 took 4 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_75 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 95.0 (TID 156). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 158
23/09/24 21:18:22 INFO Executor: Running task 0.0 in stage 96.0 (TID 158)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 32 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 76 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_76_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 76 took 3 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_76 stored as values in memory (estimated size 10.1 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 31, fetching them
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:22 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:22 INFO Executor: Finished task 0.0 in stage 96.0 (TID 158). 2450 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 160
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 98.0 (TID 160)
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 77 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_77_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 77 took 5 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_77 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 98.0 (TID 160). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 165
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 103.0 (TID 165)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 33 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 80 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_80_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 80 took 3 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_80 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 103.0 (TID 165). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 172
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 106.0 (TID 172)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 34 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 82 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_82_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 82 took 4 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_82 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 106.0 (TID 172). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 176
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 109.0 (TID 176)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 35 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 84 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_84_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 84 took 4 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_84 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 109.0 (TID 176). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 182
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 112.0 (TID 182)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 36 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 86 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_86_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 86 took 7 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_86 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 112.0 (TID 182). 3033 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 186
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 115.0 (TID 186)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 37 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 88 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_88_piece0 stored as bytes in memory (estimated size 10.2 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 88 took 4 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_88 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 115.0 (TID 186). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 191
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 118.0 (TID 191)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 38 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 90 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_90_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 90 took 3 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_90 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 118.0 (TID 191). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 196
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 121.0 (TID 196)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 39 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 92 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_92_piece0 stored as bytes in memory (estimated size 10.3 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 92 took 3 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_92 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 121.0 (TID 196). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 200
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 124.0 (TID 200)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 40 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 94 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_94_piece0 stored as bytes in memory (estimated size 10.2 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 94 took 5 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_94 stored as values in memory (estimated size 26.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_94_1 locally
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 124.0 (TID 200). 2990 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 204
23/09/24 21:18:22 INFO Executor: Running task 0.0 in stage 125.0 (TID 204)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Updating epoch to 41 and clearing cache
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 95 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_95_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 95 took 3 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_95 stored as values in memory (estimated size 10.1 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 40, fetching them
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:22 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:22 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:22 INFO Executor: Finished task 0.0 in stage 125.0 (TID 204). 2450 bytes result sent to driver
23/09/24 21:18:22 INFO CoarseGrainedExecutorBackend: Got assigned task 206
23/09/24 21:18:22 INFO Executor: Running task 1.0 in stage 126.0 (TID 206)
23/09/24 21:18:22 INFO TorrentBroadcast: Started reading broadcast variable 96 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_96_piece0 stored as bytes in memory (estimated size 11.6 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO TorrentBroadcast: Reading broadcast variable 96 took 3 ms
23/09/24 21:18:22 INFO MemoryStore: Block broadcast_96 stored as values in memory (estimated size 34.2 KiB, free 5.1 GiB)
23/09/24 21:18:22 INFO BlockManager: Found block rdd_40_1 locally
23/09/24 21:18:22 INFO CodeGenerator: Code generated in 16.702562 ms
23/09/24 21:18:22 INFO CodeGenerator: Code generated in 8.179974 ms
23/09/24 21:18:22 INFO Executor: Finished task 1.0 in stage 126.0 (TID 206). 4593 bytes result sent to driver
23/09/24 21:18:23 INFO CoarseGrainedExecutorBackend: Got assigned task 211
23/09/24 21:18:23 INFO Executor: Running task 0.0 in stage 130.0 (TID 211)
23/09/24 21:18:23 INFO TorrentBroadcast: Started reading broadcast variable 101 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:23 INFO MemoryStore: Block broadcast_101_piece0 stored as bytes in memory (estimated size 16.4 KiB, free 5.1 GiB)
23/09/24 21:18:23 INFO TorrentBroadcast: Reading broadcast variable 101 took 3 ms
23/09/24 21:18:23 INFO MemoryStore: Block broadcast_101 stored as values in memory (estimated size 47.7 KiB, free 5.1 GiB)
23/09/24 21:18:23 INFO CodeGenerator: Code generated in 6.663802 ms
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_0 remotely
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_1 remotely
23/09/24 21:18:23 WARN BlockManager: Failed to fetch remote block rdd_382_2 from BlockManagerId(0, 5.161.188.154, 37323, None) (failed attempt 1)
java.lang.IllegalStateException: Empty buffer received for non empty block
	at org.apache.spark.storage.BlockManager.fetchRemoteManagedBuffer(BlockManager.scala:1063)
	at org.apache.spark.storage.BlockManager.$anonfun$getRemoteBlock$8(BlockManager.scala:1005)
	at scala.Option.orElse(Option.scala:447)
	at org.apache.spark.storage.BlockManager.getRemoteBlock(BlockManager.scala:1005)
	at org.apache.spark.storage.BlockManager.getRemoteValues(BlockManager.scala:951)
	at org.apache.spark.storage.BlockManager.get(BlockManager.scala:1168)
	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1230)
	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.CoalescedRDD.$anonfun$compute$1(CoalescedRDD.scala:99)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithoutKey_1$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithoutKey_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:345)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
23/09/24 21:18:23 WARN BlockManager: Failed to fetch block after 2 fetch failures. Most recent failure cause:
java.lang.IllegalStateException: Empty buffer received for non empty block
	at org.apache.spark.storage.BlockManager.fetchRemoteManagedBuffer(BlockManager.scala:1063)
	at org.apache.spark.storage.BlockManager.$anonfun$getRemoteBlock$8(BlockManager.scala:1005)
	at scala.Option.orElse(Option.scala:447)
	at org.apache.spark.storage.BlockManager.getRemoteBlock(BlockManager.scala:1005)
	at org.apache.spark.storage.BlockManager.getRemoteValues(BlockManager.scala:951)
	at org.apache.spark.storage.BlockManager.get(BlockManager.scala:1168)
	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1230)
	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.CoalescedRDD.$anonfun$compute$1(CoalescedRDD.scala:99)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithoutKey_1$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithoutKey_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:345)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
23/09/24 21:18:23 INFO BlockManager: Found block rdd_40_2 remotely
23/09/24 21:18:23 INFO CodeGenerator: Code generated in 12.601964 ms
23/09/24 21:18:23 INFO CodeGenerator: Code generated in 13.211846 ms
23/09/24 21:18:23 INFO TorrentBroadcast: Started reading broadcast variable 97 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:23 INFO MemoryStore: Block broadcast_97_piece0 stored as bytes in memory (estimated size 5.9 KiB, free 5.1 GiB)
23/09/24 21:18:23 INFO TorrentBroadcast: Reading broadcast variable 97 took 4 ms
23/09/24 21:18:23 INFO MemoryStore: Block broadcast_97 stored as values in memory (estimated size 64.0 MiB, free 5.0 GiB)
23/09/24 21:18:23 INFO MemoryStore: Block rdd_382_2 stored as values in memory (estimated size 16.0 B, free 5.0 GiB)
23/09/24 21:18:23 INFO Executor: Finished task 0.0 in stage 130.0 (TID 211). 2363 bytes result sent to driver
23/09/24 21:18:23 INFO CoarseGrainedExecutorBackend: Got assigned task 214
23/09/24 21:18:23 INFO Executor: Running task 0.0 in stage 133.0 (TID 214)
23/09/24 21:18:23 INFO TorrentBroadcast: Started reading broadcast variable 104 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:23 INFO MemoryStore: Block broadcast_104_piece0 stored as bytes in memory (estimated size 16.4 KiB, free 5.0 GiB)
23/09/24 21:18:23 INFO TorrentBroadcast: Reading broadcast variable 104 took 3 ms
23/09/24 21:18:23 INFO MemoryStore: Block broadcast_104 stored as values in memory (estimated size 47.7 KiB, free 5.0 GiB)
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_0 remotely
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_1 remotely
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_2 locally
23/09/24 21:18:23 INFO Executor: Finished task 0.0 in stage 133.0 (TID 214). 2320 bytes result sent to driver
23/09/24 21:18:23 INFO CoarseGrainedExecutorBackend: Got assigned task 215
23/09/24 21:18:23 INFO Executor: Running task 0.0 in stage 134.0 (TID 215)
23/09/24 21:18:23 INFO TorrentBroadcast: Started reading broadcast variable 105 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:23 INFO MemoryStore: Block broadcast_105_piece0 stored as bytes in memory (estimated size 16.4 KiB, free 5.0 GiB)
23/09/24 21:18:23 INFO TorrentBroadcast: Reading broadcast variable 105 took 3 ms
23/09/24 21:18:23 INFO MemoryStore: Block broadcast_105 stored as values in memory (estimated size 47.7 KiB, free 5.0 GiB)
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_0 remotely
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_1 remotely
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_2 locally
23/09/24 21:18:23 INFO Executor: Finished task 0.0 in stage 134.0 (TID 215). 2320 bytes result sent to driver
23/09/24 21:18:23 INFO CoarseGrainedExecutorBackend: Got assigned task 218
23/09/24 21:18:23 INFO Executor: Running task 0.0 in stage 137.0 (TID 218)
23/09/24 21:18:23 INFO TorrentBroadcast: Started reading broadcast variable 108 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:23 INFO MemoryStore: Block broadcast_108_piece0 stored as bytes in memory (estimated size 16.4 KiB, free 5.0 GiB)
23/09/24 21:18:23 INFO TorrentBroadcast: Reading broadcast variable 108 took 3 ms
23/09/24 21:18:23 INFO MemoryStore: Block broadcast_108 stored as values in memory (estimated size 47.7 KiB, free 5.0 GiB)
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_0 remotely
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_1 remotely
23/09/24 21:18:23 INFO BlockManager: Found block rdd_382_2 locally
23/09/24 21:18:23 INFO Executor: Finished task 0.0 in stage 137.0 (TID 218). 2320 bytes result sent to driver
23/09/24 21:18:24 INFO CoarseGrainedExecutorBackend: Got assigned task 223
23/09/24 21:18:24 INFO Executor: Running task 0.0 in stage 142.0 (TID 223)
23/09/24 21:18:24 INFO TorrentBroadcast: Started reading broadcast variable 113 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:24 INFO MemoryStore: Block broadcast_113_piece0 stored as bytes in memory (estimated size 16.4 KiB, free 5.0 GiB)
23/09/24 21:18:24 INFO TorrentBroadcast: Reading broadcast variable 113 took 3 ms
23/09/24 21:18:24 INFO MemoryStore: Block broadcast_113 stored as values in memory (estimated size 47.7 KiB, free 5.0 GiB)
23/09/24 21:18:24 INFO BlockManager: Found block rdd_382_0 remotely
23/09/24 21:18:24 INFO BlockManager: Found block rdd_382_1 remotely
23/09/24 21:18:24 INFO BlockManager: Found block rdd_382_2 locally
23/09/24 21:18:24 INFO Executor: Finished task 0.0 in stage 142.0 (TID 223). 2320 bytes result sent to driver
23/09/24 21:18:24 INFO CoarseGrainedExecutorBackend: Got assigned task 229
23/09/24 21:18:24 INFO Executor: Running task 1.0 in stage 145.0 (TID 229)
23/09/24 21:18:24 INFO MapOutputTrackerWorker: Updating epoch to 42 and clearing cache
23/09/24 21:18:24 INFO TorrentBroadcast: Started reading broadcast variable 116 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:24 INFO MemoryStore: Block broadcast_116_piece0 stored as bytes in memory (estimated size 11.6 KiB, free 5.0 GiB)
23/09/24 21:18:24 INFO TorrentBroadcast: Reading broadcast variable 116 took 4 ms
23/09/24 21:18:24 INFO MemoryStore: Block broadcast_116 stored as values in memory (estimated size 34.2 KiB, free 5.0 GiB)
23/09/24 21:18:24 INFO BlockManager: Found block rdd_40_1 locally
23/09/24 21:18:24 INFO Executor: Finished task 1.0 in stage 145.0 (TID 229). 4593 bytes result sent to driver
23/09/24 21:18:24 INFO CoarseGrainedExecutorBackend: Got assigned task 231
23/09/24 21:18:24 INFO Executor: Running task 1.0 in stage 146.0 (TID 231)
23/09/24 21:18:24 INFO TorrentBroadcast: Started reading broadcast variable 118 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:24 INFO MemoryStore: Block broadcast_118_piece0 stored as bytes in memory (estimated size 16.8 KiB, free 5.0 GiB)
23/09/24 21:18:24 INFO TorrentBroadcast: Reading broadcast variable 118 took 3 ms
23/09/24 21:18:24 INFO MemoryStore: Block broadcast_118 stored as values in memory (estimated size 47.9 KiB, free 5.0 GiB)
23/09/24 21:18:24 INFO BlockManager: Found block rdd_40_1 locally
23/09/24 21:18:24 INFO CodeGenerator: Code generated in 15.998514 ms
23/09/24 21:18:24 INFO TorrentBroadcast: Started reading broadcast variable 117 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:24 INFO MemoryStore: Block broadcast_117_piece0 stored as bytes in memory (estimated size 5.9 KiB, free 5.0 GiB)
23/09/24 21:18:24 INFO TorrentBroadcast: Reading broadcast variable 117 took 3 ms
23/09/24 21:18:24 INFO MemoryStore: Block broadcast_117 stored as values in memory (estimated size 64.0 MiB, free 5.0 GiB)
23/09/24 21:18:24 INFO MemoryStore: Block rdd_497_1 stored as values in memory (estimated size 6.8 KiB, free 5.0 GiB)
23/09/24 21:18:24 INFO Executor: Finished task 1.0 in stage 146.0 (TID 231). 2645 bytes result sent to driver
23/09/24 21:18:27 INFO BlockManager: Removing RDD 547
23/09/24 21:18:28 INFO BlockManager: Removing RDD 547
23/09/24 21:18:28 INFO CoarseGrainedExecutorBackend: Got assigned task 275
23/09/24 21:18:28 INFO Executor: Running task 1.0 in stage 188.0 (TID 275)
23/09/24 21:18:28 INFO MapOutputTrackerWorker: Updating epoch to 43 and clearing cache
23/09/24 21:18:28 INFO TorrentBroadcast: Started reading broadcast variable 198 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_198_piece0 stored as bytes in memory (estimated size 17.8 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO TorrentBroadcast: Reading broadcast variable 198 took 4 ms
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_198 stored as values in memory (estimated size 63.4 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO BlockManager: Found block rdd_88_1 locally
23/09/24 21:18:28 INFO CodeGenerator: Code generated in 41.849274 ms
23/09/24 21:18:28 INFO CodeGenerator: Code generated in 10.943978 ms
23/09/24 21:18:28 INFO Executor: Finished task 1.0 in stage 188.0 (TID 275). 2574 bytes result sent to driver
23/09/24 21:18:28 INFO CoarseGrainedExecutorBackend: Got assigned task 279
23/09/24 21:18:28 INFO Executor: Running task 1.0 in stage 189.0 (TID 279)
23/09/24 21:18:28 INFO MapOutputTrackerWorker: Updating epoch to 44 and clearing cache
23/09/24 21:18:28 INFO TorrentBroadcast: Started reading broadcast variable 199 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_199_piece0 stored as bytes in memory (estimated size 7.9 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO TorrentBroadcast: Reading broadcast variable 199 took 4 ms
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_199 stored as values in memory (estimated size 19.4 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 43, fetching them
23/09/24 21:18:28 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:28 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:28 INFO ShuffleBlockFetcherIterator: Getting 4 (1739.0 B) non-empty blocks including 1 (593.0 B) local and 3 (1146.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:28 INFO MemoryStore: Block rdd_610_1 stored as values in memory (estimated size 3.3 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO CodeGenerator: Code generated in 12.443988 ms
23/09/24 21:18:28 INFO CodeGenerator: Code generated in 9.829134 ms
23/09/24 21:18:28 INFO Executor: Finished task 1.0 in stage 189.0 (TID 279). 3616 bytes result sent to driver
23/09/24 21:18:28 INFO CoarseGrainedExecutorBackend: Got assigned task 284
23/09/24 21:18:28 INFO Executor: Running task 1.0 in stage 192.0 (TID 284)
23/09/24 21:18:28 INFO TorrentBroadcast: Started reading broadcast variable 201 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_201_piece0 stored as bytes in memory (estimated size 51.3 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO TorrentBroadcast: Reading broadcast variable 201 took 4 ms
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_201 stored as values in memory (estimated size 139.3 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO BlockManager: Found block rdd_610_1 locally
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 13.243916 ms
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 15.856075 ms
23/09/24 21:18:30 INFO TorrentBroadcast: Started reading broadcast variable 200 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_200_piece0 stored as bytes in memory (estimated size 3.9 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO TorrentBroadcast: Reading broadcast variable 200 took 4 ms
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_200 stored as values in memory (estimated size 7.9 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO MemoryStore: Block rdd_621_1 stored as values in memory (estimated size 5.9 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 11.792268 ms
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 27.964007 ms
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 8.718739 ms
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 5.823718 ms
23/09/24 21:18:30 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
23/09/24 21:18:30 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
23/09/24 21:18:30 INFO MemoryStore: Block rdd_627_1 stored as values in memory (estimated size 6.2 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 8.253231 ms
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 13.113241 ms
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 10.164096 ms
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 4.782613 ms
23/09/24 21:18:30 INFO Executor: Finished task 1.0 in stage 192.0 (TID 284). 2930 bytes result sent to driver
23/09/24 21:18:30 INFO CoarseGrainedExecutorBackend: Got assigned task 286
23/09/24 21:18:30 INFO Executor: Running task 1.0 in stage 195.0 (TID 286)
23/09/24 21:18:30 INFO TorrentBroadcast: Started reading broadcast variable 202 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_202_piece0 stored as bytes in memory (estimated size 51.3 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO TorrentBroadcast: Reading broadcast variable 202 took 5 ms
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_202 stored as values in memory (estimated size 139.3 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO BlockManager: Found block rdd_627_1 locally
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 8.399096 ms
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 16.780619 ms
23/09/24 21:18:30 INFO Executor: Finished task 1.0 in stage 195.0 (TID 286). 2925 bytes result sent to driver
23/09/24 21:18:30 INFO CoarseGrainedExecutorBackend: Got assigned task 291
23/09/24 21:18:30 INFO Executor: Running task 1.0 in stage 197.0 (TID 291)
23/09/24 21:18:30 INFO TorrentBroadcast: Started reading broadcast variable 203 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_203_piece0 stored as bytes in memory (estimated size 7.1 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO TorrentBroadcast: Reading broadcast variable 203 took 4 ms
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_203 stored as values in memory (estimated size 15.1 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO BlockManager: Found block rdd_88_1 locally
23/09/24 21:18:30 INFO Executor: Finished task 1.0 in stage 197.0 (TID 291). 3329 bytes result sent to driver
23/09/24 21:18:30 INFO CoarseGrainedExecutorBackend: Got assigned task 295
23/09/24 21:18:30 INFO Executor: Running task 1.0 in stage 200.0 (TID 295)
23/09/24 21:18:30 INFO TorrentBroadcast: Started reading broadcast variable 205 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_205_piece0 stored as bytes in memory (estimated size 52.0 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 299
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 205 took 4 ms
23/09/24 21:18:31 INFO Executor: Running task 1.0 in stage 202.0 (TID 299)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_205 stored as values in memory (estimated size 140.3 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 206 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_206_piece0 stored as bytes in memory (estimated size 51.9 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 206 took 4 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_206 stored as values in memory (estimated size 140.2 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 17.12065 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 21.095972 ms
23/09/24 21:18:31 INFO BlockManager: Found block rdd_627_1 locally
23/09/24 21:18:31 INFO BlockManager: Found block rdd_627_1 locally
23/09/24 21:18:31 INFO Executor: Finished task 1.0 in stage 202.0 (TID 299). 3125 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 1.0 in stage 200.0 (TID 295). 3130 bytes result sent to driver
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 303
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 306
23/09/24 21:18:31 INFO Executor: Running task 1.0 in stage 203.0 (TID 303)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 309
23/09/24 21:18:31 INFO Executor: Running task 4.0 in stage 203.0 (TID 306)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 312
23/09/24 21:18:31 INFO Executor: Running task 7.0 in stage 203.0 (TID 309)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Updating epoch to 45 and clearing cache
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 207 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO Executor: Running task 10.0 in stage 203.0 (TID 312)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 313
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_207_piece0 stored as bytes in memory (estimated size 53.6 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO Executor: Running task 0.0 in stage 201.0 (TID 313)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 207 took 6 ms
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Updating epoch to 46 and clearing cache
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 208 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_207 stored as values in memory (estimated size 147.5 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_208_piece0 stored as bytes in memory (estimated size 53.6 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 208 took 5 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_208 stored as values in memory (estimated size 147.6 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 44, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 44, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 44, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 44, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 46, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (405.0 B) non-empty blocks including 0 (0.0 B) local and 1 (405.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (334.0 B) non-empty blocks including 0 (0.0 B) local and 1 (334.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 9 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 0 (0.0 B) local and 1 (368.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 1 (368.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 0 (0.0 B) local and 1 (368.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 11 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 9 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 16.509828 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 27.630108 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 20.63461 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 14.276183 ms
23/09/24 21:18:31 INFO MemoryStore: Block rdd_639_7 stored as values in memory (estimated size 4.7 KiB, free 4.9 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_639_4 stored as values in memory (estimated size 4.8 KiB, free 4.9 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_639_1 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_639_10 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_658_0 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 31.959045 ms
23/09/24 21:18:31 INFO Executor: Finished task 7.0 in stage 203.0 (TID 309). 4580 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 10.0 in stage 203.0 (TID 312). 4580 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 1.0 in stage 203.0 (TID 303). 4580 bytes result sent to driver
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 12.659033 ms
23/09/24 21:18:31 INFO Executor: Finished task 4.0 in stage 203.0 (TID 306). 4580 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 0.0 in stage 201.0 (TID 313). 4580 bytes result sent to driver
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 330
23/09/24 21:18:31 INFO Executor: Running task 1.0 in stage 204.0 (TID 330)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Updating epoch to 48 and clearing cache
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 209 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_209_piece0 stored as bytes in memory (estimated size 75.1 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 209 took 3 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_209 stored as values in memory (estimated size 233.4 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO BlockManager: Found block rdd_679_1 remotely
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 10.482547 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 3.692407 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 7.54228 ms
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 204 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_204_piece0 stored as bytes in memory (estimated size 3.7 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 204 took 3 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_204 stored as values in memory (estimated size 7.6 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 4.068856 ms
23/09/24 21:18:31 INFO Executor: Finished task 1.0 in stage 204.0 (TID 330). 8009 bytes result sent to driver
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 339
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 342
23/09/24 21:18:31 INFO Executor: Running task 2.0 in stage 212.0 (TID 339)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 345
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 348
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 351
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Updating epoch to 49 and clearing cache
23/09/24 21:18:31 INFO Executor: Running task 11.0 in stage 212.0 (TID 348)
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 211 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO Executor: Running task 5.0 in stage 212.0 (TID 342)
23/09/24 21:18:31 INFO Executor: Running task 8.0 in stage 212.0 (TID 345)
23/09/24 21:18:31 INFO Executor: Running task 14.0 in stage 212.0 (TID 351)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_211_piece0 stored as bytes in memory (estimated size 129.5 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 211 took 5 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_211 stored as values in memory (estimated size 379.7 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 48, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 0 (0.0 B) local and 2 (580.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 0 (0.0 B) local and 2 (608.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 0 (0.0 B) local and 2 (580.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 2 (638.0 B) non-empty blocks including 0 (0.0 B) local and 2 (638.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 0 (0.0 B) local and 2 (580.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 9 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 21.144492 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 3.984307 ms
23/09/24 21:18:32 INFO CodeGenerator: Code generated in 10.677806 ms
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 21:18:32 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 21:18:32 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 21:18:32 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 21:18:32 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 423
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 439
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 437
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 427
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 348
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118316205670933325577925_0212_m_000011_348' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118316205670933325577925_0212_m_000011
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118316205670933325577925_0212_m_000011_348: Committed
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118315799502888492005826_0212_m_000002_339' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118315799502888492005826_0212_m_000002
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118315799502888492005826_0212_m_000002_339: Committed
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118314588042370258416415_0212_m_000005_342' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118314588042370258416415_0212_m_000005
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118314588042370258416415_0212_m_000005_342: Committed
23/09/24 21:18:32 INFO Executor: Finished task 11.0 in stage 212.0 (TID 348). 10234 bytes result sent to driver
23/09/24 21:18:32 INFO Executor: Finished task 2.0 in stage 212.0 (TID 339). 10234 bytes result sent to driver
23/09/24 21:18:32 INFO Executor: Finished task 5.0 in stage 212.0 (TID 342). 10191 bytes result sent to driver
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118314745735459695416460_0212_m_000014_351' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118314745735459695416460_0212_m_000014
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118314745735459695416460_0212_m_000014_351: Committed
23/09/24 21:18:32 INFO CoarseGrainedExecutorBackend: Got assigned task 357
23/09/24 21:18:32 INFO Executor: Running task 20.0 in stage 212.0 (TID 357)
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118311903858892477630828_0212_m_000008_345' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118311903858892477630828_0212_m_000008
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118311903858892477630828_0212_m_000008_345: Committed
23/09/24 21:18:32 INFO Executor: Finished task 14.0 in stage 212.0 (TID 351). 10191 bytes result sent to driver
23/09/24 21:18:32 INFO Executor: Finished task 8.0 in stage 212.0 (TID 345). 10191 bytes result sent to driver
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202309242118317953962894896782203_0212_m_000020_357
23/09/24 21:18:32 INFO Executor: Finished task 20.0 in stage 212.0 (TID 357). 10061 bytes result sent to driver
23/09/24 21:18:32 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
23/09/24 21:18:32 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
tdown