Spark Executor Command: "/usr/java8/jdk1.8.0_202/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx8192M" "-Dspark.driver.port=40131" "-Dspark.rpc.message.maxSize=200" "-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-XX:+HeapDumpOnOutOfMemoryError" "-Xloggc:/tmp/memLog.txt" "-XX:+UseCompressedOops" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:40131" "--executor-id" "2" "--hostname" "5.161.188.154" "--cores" "5" "--app-id" "app-20230917181211-0002" "--worker-url" "spark://Worker@5.161.188.154:42679"
========================================

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
23/09/17 18:12:11 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 50212@entity-resolution
23/09/17 18:12:11 INFO SignalUtils: Registering signal handler for TERM
23/09/17 18:12:11 INFO SignalUtils: Registering signal handler for HUP
23/09/17 18:12:11 INFO SignalUtils: Registering signal handler for INT
23/09/17 18:12:12 WARN Utils: Your hostname, entity-resolution resolves to a loopback address: 127.0.1.1; using 5.161.188.154 instead (on interface eth0)
23/09/17 18:12:12 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
23/09/17 18:12:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/09/17 18:12:12 INFO SecurityManager: Changing view acls to: root
23/09/17 18:12:12 INFO SecurityManager: Changing modify acls to: root
23/09/17 18:12:12 INFO SecurityManager: Changing view acls groups to: 
23/09/17 18:12:12 INFO SecurityManager: Changing modify acls groups to: 
23/09/17 18:12:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
23/09/17 18:12:12 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:40131 after 69 ms (0 ms spent in bootstraps)
23/09/17 18:12:12 INFO SecurityManager: Changing view acls to: root
23/09/17 18:12:12 INFO SecurityManager: Changing modify acls to: root
23/09/17 18:12:12 INFO SecurityManager: Changing view acls groups to: 
23/09/17 18:12:12 INFO SecurityManager: Changing modify acls groups to: 
23/09/17 18:12:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
23/09/17 18:12:12 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:40131 after 2 ms (0 ms spent in bootstraps)
23/09/17 18:12:13 INFO DiskBlockManager: Created local directory at /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-7907a773-db12-4c89-b68d-f86c5554938b/blockmgr-03d6e781-ac32-497f-8037-dd00e52e8091
23/09/17 18:12:13 INFO MemoryStore: MemoryStore started with capacity 6.1 GiB
23/09/17 18:12:13 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:40131
23/09/17 18:12:13 INFO WorkerWatcher: Connecting to worker spark://Worker@5.161.188.154:42679
23/09/17 18:12:13 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:42679 after 2 ms (0 ms spent in bootstraps)
23/09/17 18:12:13 INFO WorkerWatcher: Successfully connected to spark://Worker@5.161.188.154:42679
23/09/17 18:12:13 INFO ResourceUtils: ==============================================================
23/09/17 18:12:13 INFO ResourceUtils: No custom resources configured for spark.executor.
23/09/17 18:12:13 INFO ResourceUtils: ==============================================================
23/09/17 18:12:13 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
23/09/17 18:12:13 INFO Executor: Starting executor ID 2 on host 5.161.188.154
23/09/17 18:12:13 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38071.
23/09/17 18:12:13 INFO NettyBlockTransferService: Server created on 5.161.188.154:38071
23/09/17 18:12:13 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/09/17 18:12:13 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(2, 5.161.188.154, 38071, None)
23/09/17 18:12:13 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(2, 5.161.188.154, 38071, None)
23/09/17 18:12:13 INFO BlockManager: Initialized BlockManager: BlockManagerId(2, 5.161.188.154, 38071, None)
23/09/17 18:12:13 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:40131/jars/postgresql-42.6.0.jar with timestamp 1694974330023
23/09/17 18:12:13 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:40131 after 1 ms (0 ms spent in bootstraps)
23/09/17 18:12:13 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:40131/jars/postgresql-42.6.0.jar to /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-7907a773-db12-4c89-b68d-f86c5554938b/spark-b40f3d1e-e4f4-4363-88fd-2271eac8eb24/fetchFileTemp6349941010545591381.tmp
23/09/17 18:12:13 INFO Utils: Copying /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-7907a773-db12-4c89-b68d-f86c5554938b/spark-b40f3d1e-e4f4-4363-88fd-2271eac8eb24/11246080281694974330023_cache to /opt/spark/work/app-20230917181211-0002/2/./postgresql-42.6.0.jar
23/09/17 18:12:13 INFO Executor: Adding file:/opt/spark/work/app-20230917181211-0002/2/./postgresql-42.6.0.jar to class loader
23/09/17 18:12:13 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:40131/jars/zingg-0.3.4-SNAPSHOT.jar with timestamp 1694974330023
23/09/17 18:12:13 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:40131/jars/zingg-0.3.4-SNAPSHOT.jar to /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-7907a773-db12-4c89-b68d-f86c5554938b/spark-b40f3d1e-e4f4-4363-88fd-2271eac8eb24/fetchFileTemp3384853000587768764.tmp
23/09/17 18:12:13 INFO Utils: Copying /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-7907a773-db12-4c89-b68d-f86c5554938b/spark-b40f3d1e-e4f4-4363-88fd-2271eac8eb24/-700982371694974330023_cache to /opt/spark/work/app-20230917181211-0002/2/./zingg-0.3.4-SNAPSHOT.jar
23/09/17 18:12:13 INFO Executor: Adding file:/opt/spark/work/app-20230917181211-0002/2/./zingg-0.3.4-SNAPSHOT.jar to class loader
23/09/17 18:12:15 INFO CoarseGrainedExecutorBackend: Got assigned task 2
23/09/17 18:12:15 INFO CoarseGrainedExecutorBackend: Got assigned task 5
23/09/17 18:12:15 INFO CoarseGrainedExecutorBackend: Got assigned task 8
23/09/17 18:12:15 INFO CoarseGrainedExecutorBackend: Got assigned task 11
23/09/17 18:12:15 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
23/09/17 18:12:15 INFO Executor: Running task 11.0 in stage 0.0 (TID 11)
23/09/17 18:12:15 INFO Executor: Running task 8.0 in stage 0.0 (TID 8)
23/09/17 18:12:15 INFO Executor: Running task 5.0 in stage 0.0 (TID 5)
23/09/17 18:12:15 INFO TorrentBroadcast: Started reading broadcast variable 1 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:12:15 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:42953 after 2 ms (0 ms spent in bootstraps)
23/09/17 18:12:15 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 7.9 KiB, free 6.1 GiB)
23/09/17 18:12:15 INFO TorrentBroadcast: Reading broadcast variable 1 took 86 ms
23/09/17 18:12:15 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 15.3 KiB, free 6.1 GiB)
23/09/17 18:12:16 INFO CodeGenerator: Code generated in 176.581028 ms
23/09/17 18:12:16 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 46137344-47316767, partition values: [empty row]
23/09/17 18:12:16 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 20971520-25165824, partition values: [empty row]
23/09/17 18:12:16 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 33554432-37748736, partition values: [empty row]
23/09/17 18:12:16 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 8388608-12582912, partition values: [empty row]
23/09/17 18:12:16 INFO CodeGenerator: Code generated in 12.612444 ms
23/09/17 18:12:16 INFO TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:12:16 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:39021 after 1 ms (0 ms spent in bootstraps)
23/09/17 18:12:16 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 24.9 KiB, free 6.1 GiB)
23/09/17 18:12:16 INFO TorrentBroadcast: Reading broadcast variable 0 took 16 ms
23/09/17 18:12:16 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 394.6 KiB, free 6.1 GiB)
23/09/17 18:12:17 INFO Executor: Finished task 11.0 in stage 0.0 (TID 11). 1886 bytes result sent to driver
23/09/17 18:12:17 INFO Executor: Finished task 8.0 in stage 0.0 (TID 8). 1886 bytes result sent to driver
23/09/17 18:12:17 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 1886 bytes result sent to driver
23/09/17 18:12:17 INFO Executor: Finished task 5.0 in stage 0.0 (TID 5). 1929 bytes result sent to driver
23/09/17 18:12:17 INFO CoarseGrainedExecutorBackend: Got assigned task 12
23/09/17 18:12:17 INFO Executor: Running task 0.0 in stage 1.0 (TID 12)
23/09/17 18:12:17 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache
23/09/17 18:12:17 INFO TorrentBroadcast: Started reading broadcast variable 2 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:12:17 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 6.1 GiB)
23/09/17 18:12:17 INFO TorrentBroadcast: Reading broadcast variable 2 took 8 ms
23/09/17 18:12:17 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 10.1 KiB, free 6.1 GiB)
23/09/17 18:12:17 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them
23/09/17 18:12:17 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:40131)
23/09/17 18:12:17 INFO MapOutputTrackerWorker: Got the output locations
23/09/17 18:12:17 INFO ShuffleBlockFetcherIterator: Getting 12 (720.0 B) non-empty blocks including 4 (240.0 B) local and 8 (480.0 B) host-local and 0 (0.0 B) remote blocks
23/09/17 18:12:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 10 ms
23/09/17 18:12:17 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:38751 after 1 ms (0 ms spent in bootstraps)
23/09/17 18:12:17 INFO CodeGenerator: Code generated in 11.393135 ms
23/09/17 18:12:17 INFO Executor: Finished task 0.0 in stage 1.0 (TID 12). 2450 bytes result sent to driver
23/09/17 18:12:17 INFO CoarseGrainedExecutorBackend: Got assigned task 13
23/09/17 18:12:17 INFO Executor: Running task 0.0 in stage 2.0 (TID 13)
23/09/17 18:12:17 INFO CoarseGrainedExecutorBackend: Got assigned task 16
23/09/17 18:12:17 INFO Executor: Running task 3.0 in stage 2.0 (TID 16)
23/09/17 18:12:17 INFO CoarseGrainedExecutorBackend: Got assigned task 19
23/09/17 18:12:17 INFO Executor: Running task 6.0 in stage 2.0 (TID 19)
23/09/17 18:12:17 INFO CoarseGrainedExecutorBackend: Got assigned task 22
23/09/17 18:12:17 INFO Executor: Running task 9.0 in stage 2.0 (TID 22)
23/09/17 18:12:17 INFO TorrentBroadcast: Started reading broadcast variable 4 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:12:17 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 16.5 KiB, free 6.1 GiB)
23/09/17 18:12:17 INFO TorrentBroadcast: Reading broadcast variable 4 took 18 ms
23/09/17 18:12:17 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 41.1 KiB, free 6.1 GiB)
23/09/17 18:12:18 INFO CodeGenerator: Code generated in 22.27673 ms
23/09/17 18:12:19 INFO CodeGenerator: Code generated in 21.662742 ms
23/09/17 18:12:19 INFO CodeGenerator: Code generated in 25.185134 ms
23/09/17 18:12:19 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 0-4194304, partition values: [empty row]
23/09/17 18:12:19 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 12582912-16777216, partition values: [empty row]
23/09/17 18:12:19 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 37748736-41943040, partition values: [empty row]
23/09/17 18:12:19 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 25165824-29360128, partition values: [empty row]
23/09/17 18:12:19 INFO CodeGenerator: Code generated in 10.964197 ms
23/09/17 18:12:19 INFO TorrentBroadcast: Started reading broadcast variable 3 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:12:19 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 24.9 KiB, free 6.1 GiB)
23/09/17 18:12:19 INFO TorrentBroadcast: Reading broadcast variable 3 took 9 ms
23/09/17 18:12:19 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 394.6 KiB, free 6.1 GiB)
23/09/17 18:12:19 INFO CodeGenerator: Code generated in 73.615203 ms
23/09/17 18:12:20 INFO MemoryStore: Block rdd_18_9 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB)
23/09/17 18:12:20 INFO MemoryStore: Block rdd_18_3 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB)
23/09/17 18:12:20 INFO MemoryStore: Block rdd_18_0 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB)
23/09/17 18:12:20 INFO CodeGenerator: Code generated in 5.211909 ms
23/09/17 18:12:20 INFO MemoryStore: Block rdd_18_6 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB)
23/09/17 18:12:20 INFO CodeGenerator: Code generated in 25.523944 ms
23/09/17 18:12:20 INFO Executor: Finished task 3.0 in stage 2.0 (TID 16). 2451 bytes result sent to driver
23/09/17 18:12:20 INFO Executor: Finished task 6.0 in stage 2.0 (TID 19). 2451 bytes result sent to driver
23/09/17 18:12:20 INFO Executor: Finished task 9.0 in stage 2.0 (TID 22). 2451 bytes result sent to driver
23/09/17 18:12:20 INFO Executor: Finished task 0.0 in stage 2.0 (TID 13). 2451 bytes result sent to driver
23/09/17 18:12:20 INFO CoarseGrainedExecutorBackend: Got assigned task 26
23/09/17 18:12:20 INFO Executor: Running task 0.0 in stage 4.0 (TID 26)
23/09/17 18:12:20 INFO CoarseGrainedExecutorBackend: Got assigned task 29
23/09/17 18:12:20 INFO CoarseGrainedExecutorBackend: Got assigned task 32
23/09/17 18:12:20 INFO Executor: Running task 3.0 in stage 4.0 (TID 29)
23/09/17 18:12:20 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache
23/09/17 18:12:20 INFO CoarseGrainedExecutorBackend: Got assigned task 35
23/09/17 18:12:20 INFO Executor: Running task 6.0 in stage 4.0 (TID 32)
23/09/17 18:12:20 INFO Executor: Running task 9.0 in stage 4.0 (TID 35)
23/09/17 18:12:20 INFO TorrentBroadcast: Started reading broadcast variable 6 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:12:20 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 16.8 KiB, free 6.1 GiB)
23/09/17 18:12:20 INFO TorrentBroadcast: Reading broadcast variable 6 took 19 ms
23/09/17 18:12:20 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 43.0 KiB, free 6.1 GiB)
23/09/17 18:12:20 INFO BlockManager: Found block rdd_18_3 locally
23/09/17 18:12:20 INFO BlockManager: Found block rdd_18_6 locally
23/09/17 18:12:20 INFO BlockManager: Found block rdd_18_9 locally
23/09/17 18:12:20 INFO BlockManager: Found block rdd_18_0 locally
23/09/17 18:12:21 INFO CodeGenerator: Code generated in 40.934174 ms
23/09/17 18:12:21 INFO CodeGenerator: Code generated in 15.014775 ms
23/09/17 18:12:21 INFO Executor: Finished task 3.0 in stage 4.0 (TID 29). 2369 bytes result sent to driver
23/09/17 18:12:21 INFO Executor: Finished task 0.0 in stage 4.0 (TID 26). 2369 bytes result sent to driver
23/09/17 18:12:21 INFO Executor: Finished task 9.0 in stage 4.0 (TID 35). 2369 bytes result sent to driver
23/09/17 18:12:21 INFO Executor: Finished task 6.0 in stage 4.0 (TID 32). 2369 bytes result sent to driver
23/09/17 18:12:21 INFO CoarseGrainedExecutorBackend: Got assigned task 40
23/09/17 18:12:21 INFO Executor: Running task 2.0 in stage 5.0 (TID 40)
23/09/17 18:12:21 INFO MapOutputTrackerWorker: Updating epoch to 3 and clearing cache
23/09/17 18:12:21 INFO TorrentBroadcast: Started reading broadcast variable 7 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:12:21 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 9.5 KiB, free 6.1 GiB)
23/09/17 18:12:21 INFO TorrentBroadcast: Reading broadcast variable 7 took 7 ms
23/09/17 18:12:21 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 23.7 KiB, free 6.1 GiB)
23/09/17 18:12:21 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 2, fetching them
23/09/17 18:12:21 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:40131)
23/09/17 18:12:21 INFO MapOutputTrackerWorker: Got the output locations
23/09/17 18:12:21 INFO ShuffleBlockFetcherIterator: Getting 12 (3.1 MiB) non-empty blocks including 4 (1136.6 KiB) local and 8 (2.0 MiB) host-local and 0 (0.0 B) remote blocks
23/09/17 18:12:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/17 18:12:21 INFO MemoryStore: Block rdd_35_2 stored as values in memory (estimated size 4.5 MiB, free 6.1 GiB)
23/09/17 18:12:21 INFO CodeGenerator: Code generated in 10.270418 ms
23/09/17 18:12:21 INFO MemoryStore: Block rdd_41_2 stored as values in memory (estimated size 4.5 MiB, free 6.1 GiB)
23/09/17 18:12:21 INFO Executor: Finished task 2.0 in stage 5.0 (TID 40). 3204 bytes result sent to driver
23/09/17 18:12:22 INFO CoarseGrainedExecutorBackend: Got assigned task 43
23/09/17 18:12:22 INFO Executor: Running task 2.0 in stage 8.0 (TID 43)
23/09/17 18:12:22 INFO MapOutputTrackerWorker: Updating epoch to 4 and clearing cache
23/09/17 18:12:22 INFO TorrentBroadcast: Started reading broadcast variable 9 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:12:22 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 9.1 KiB, free 6.1 GiB)
23/09/17 18:12:22 INFO TorrentBroadcast: Reading broadcast variable 9 took 9 ms
23/09/17 18:12:22 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 22.0 KiB, free 6.1 GiB)
23/09/17 18:12:22 INFO BlockManager: Found block rdd_41_2 locally
23/09/17 18:12:22 INFO MemoryStore: Block taskresult_43 stored as bytes in memory (estimated size 3.1 MiB, free 6.1 GiB)
23/09/17 18:12:22 INFO Executor: Finished task 2.0 in stage 8.0 (TID 43). 3234481 bytes result sent via BlockManager)
23/09/17 18:12:22 INFO CoarseGrainedExecutorBackend: Got assigned task 48
23/09/17 18:12:22 INFO CoarseGrainedExecutorBackend: Got assigned task 51
23/09/17 18:12:22 INFO Executor: Running task 0.0 in stage 9.0 (TID 48)
23/09/17 18:12:22 INFO Executor: Running task 3.0 in stage 9.0 (TID 51)
23/09/17 18:12:22 INFO CoarseGrainedExecutorBackend: Got assigned task 54
23/09/17 18:12:22 INFO CoarseGrainedExecutorBackend: Got assigned task 57
23/09/17 18:12:22 INFO Executor: Running task 6.0 in stage 9.0 (TID 54)
23/09/17 18:12:22 INFO Executor: Running task 9.0 in stage 9.0 (TID 57)
23/09/17 18:12:22 INFO TorrentBroadcast: Started reading broadcast variable 10 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:12:22 INFO CoarseGrainedExecutorBackend: Got assigned task 60
23/09/17 18:12:22 INFO Executor: Running task 0.0 in stage 10.0 (TID 60)
23/09/17 18:12:22 INFO TorrentBroadcast: Started reading broadcast variable 11 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:12:22 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 18.7 KiB, free 6.1 GiB)
23/09/17 18:12:22 INFO TorrentBroadcast: Reading broadcast variable 10 took 13 ms
23/09/17 18:12:22 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 50.4 KiB, free 6.1 GiB)
23/09/17 18:12:22 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 18.1 KiB, free 6.1 GiB)
23/09/17 18:12:22 INFO TorrentBroadcast: Reading broadcast variable 11 took 25 ms
23/09/17 18:12:22 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 46.5 KiB, free 6.1 GiB)
23/09/17 18:12:22 INFO BlockManager: Found block rdd_18_9 locally
23/09/17 18:12:22 INFO BlockManager: Found block rdd_18_3 locally
23/09/17 18:12:22 INFO BlockManager: Found block rdd_18_0 locally
23/09/17 18:12:22 INFO BlockManager: Found block rdd_18_6 locally
23/09/17 18:12:22 INFO CodeGenerator: Code generated in 24.085251 ms
23/09/17 18:12:23 INFO MemoryStore: Block rdd_59_6 stored as values in memory (estimated size 1506.3 KiB, free 6.1 GiB)
23/09/17 18:12:23 INFO MemoryStore: Block rdd_59_9 stored as values in memory (estimated size 1517.1 KiB, free 6.1 GiB)
23/09/17 18:12:23 INFO MemoryStore: Block rdd_59_3 stored as values in memory (estimated size 1478.4 KiB, free 6.1 GiB)
23/09/17 18:12:23 INFO MemoryStore: Block rdd_59_0 stored as values in memory (estimated size 1500.3 KiB, free 6.1 GiB)
23/09/17 18:12:23 WARN BlockManager: Block rdd_59_0 already exists on this machine; not re-adding it
23/09/17 18:12:23 INFO CodeGenerator: Code generated in 50.672568 ms
23/09/17 18:12:23 INFO CodeGenerator: Code generated in 23.834175 ms
23/09/17 18:12:23 INFO CodeGenerator: Code generated in 11.853503 ms
23/09/17 18:12:23 INFO Executor: Finished task 0.0 in stage 10.0 (TID 60). 2714 bytes result sent to driver
23/09/17 18:12:23 INFO CoarseGrainedExecutorBackend: Got assigned task 65
23/09/17 18:12:23 INFO Executor: Running task 3.0 in stage 10.0 (TID 65)
23/09/17 18:12:23 INFO BlockManager: Found block rdd_59_3 locally
23/09/17 18:12:23 INFO Executor: Finished task 3.0 in stage 9.0 (TID 51). 2776 bytes result sent to driver
23/09/17 18:12:23 INFO Executor: Finished task 3.0 in stage 10.0 (TID 65). 2672 bytes result sent to driver
23/09/17 18:12:23 INFO CoarseGrainedExecutorBackend: Got assigned task 68
23/09/17 18:12:23 INFO Executor: Running task 6.0 in stage 10.0 (TID 68)
23/09/17 18:12:23 INFO CoarseGrainedExecutorBackend: Got assigned task 69
23/09/17 18:12:23 INFO Executor: Running task 9.0 in stage 10.0 (TID 69)
23/09/17 18:12:23 INFO BlockManager: Found block rdd_59_6 locally
23/09/17 18:12:23 INFO BlockManager: Found block rdd_59_9 locally
23/09/17 18:12:23 INFO Executor: Finished task 0.0 in stage 9.0 (TID 48). 2819 bytes result sent to driver
23/09/17 18:12:23 INFO Executor: Finished task 6.0 in stage 9.0 (TID 54). 2776 bytes result sent to driver
23/09/17 18:12:23 INFO Executor: Finished task 9.0 in stage 9.0 (TID 57). 2776 bytes result sent to driver
23/09/17 18:12:23 INFO Executor: Finished task 9.0 in stage 10.0 (TID 69). 2672 bytes result sent to driver
23/09/17 18:12:23 INFO Executor: Finished task 6.0 in stage 10.0 (TID 68). 2672 bytes result sent to driver
23/09/17 18:20:19 INFO CoarseGrainedExecutorBackend: Got assigned task 73
23/09/17 18:20:19 INFO Executor: Running task 2.0 in stage 13.0 (TID 73)
23/09/17 18:20:19 INFO MapOutputTrackerWorker: Updating epoch to 6 and clearing cache
23/09/17 18:20:19 INFO TorrentBroadcast: Started reading broadcast variable 13 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:20:19 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 142.1 KiB, free 6.1 GiB)
23/09/17 18:20:19 INFO TorrentBroadcast: Reading broadcast variable 13 took 5 ms
23/09/17 18:20:19 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 546.8 KiB, free 6.1 GiB)
23/09/17 18:20:19 INFO BlockManager: Found block rdd_35_2 locally
23/09/17 18:20:19 INFO CodeGenerator: Code generated in 31.590813 ms
23/09/17 18:20:19 INFO CodeGenerator: Code generated in 7.602114 ms
23/09/17 18:20:19 INFO Executor: Finished task 2.0 in stage 13.0 (TID 73). 2574 bytes result sent to driver
23/09/17 18:20:19 INFO CoarseGrainedExecutorBackend: Got assigned task 78
23/09/17 18:20:19 INFO Executor: Running task 2.0 in stage 14.0 (TID 78)
23/09/17 18:20:19 INFO MapOutputTrackerWorker: Updating epoch to 7 and clearing cache
23/09/17 18:20:19 INFO TorrentBroadcast: Started reading broadcast variable 14 with 1 pieces (estimated total size 4.0 MiB)
23/09/17 18:20:19 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 16.7 KiB, free 6.1 GiB)
23/09/17 18:20:19 INFO TorrentBroadcast: Reading broadcast variable 14 took 5 ms
23/09/17 18:20:19 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 42.2 KiB, free 6.1 GiB)
23/09/17 18:20:19 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 6, fetching them
23/09/17 18:20:19 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:40131)
23/09/17 18:20:19 INFO MapOutputTrackerWorker: Got the output locations
23/09/17 18:20:19 INFO ShuffleBlockFetcherIterator: Getting 4 (2.4 MiB) non-empty blocks including 1 (609.1 KiB) local and 3 (1827.2 KiB) host-local and 0 (0.0 B) remote blocks
23/09/17 18:20:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/17 18:20:20 INFO MemoryStore: Block rdd_89_2 stored as values in memory (estimated size 2.8 MiB, free 6.1 GiB)
23/09/17 18:20:20 INFO CodeGenerator: Code generated in 33.202383 ms
23/09/17 18:20:20 INFO CodeGenerator: Code generated in 10.13062 ms
23/09/17 18:20:20 INFO CodeGenerator: Code generated in 6.256346 ms
23/09/17 18:20:20 INFO CodeGenerator: Code generated in 3.856963 ms
23/09/17 18:20:20 INFO BlockManager: Found block rdd_89_2 locally
23/09/17 18:20:20 INFO CodeGenerator: Code generated in 17.059655 ms
23/09/17 18:20:20 INFO CodeGenerator: Code generated in 12.633189 ms
23/09/17 18:20:20 INFO CodeGenerator: Code generated in 16.716088 ms
23/09/17 18:20:22 INFO MemoryStore: Block rdd_101_2 stored as values in memory (estimated size 13.0 MiB, free 6.0 GiB)
23/09/17 18:20:22 INFO Executor: Finished task 2.0 in stage 14.0 (TID 78). 3959 bytes result sent to driver
23/09/17 19:11:27 INFO CoarseGrainedExecutorBackend: Got assigned task 80
23/09/17 19:11:27 INFO Executor: Running task 3.1 in stage 14.0 (TID 80)
23/09/17 19:11:27 INFO BlockManager: Read rdd_89_3 from the disk of a same host executor is successful.
23/09/17 19:11:27 INFO BlockManager: Found block rdd_89_3 remotely
23/09/17 19:11:27 INFO BlockManager: Read rdd_89_3 from the disk of a same host executor is successful.
23/09/17 19:11:27 INFO BlockManager: Found block rdd_89_3 remotely
23/09/17 19:13:31 INFO MemoryStore: 15 blocks selected for dropping (31.7 MiB bytes)
23/09/17 19:13:31 INFO BlockManager: Dropping block broadcast_3_piece0 from memory
23/09/17 19:13:31 INFO BlockManager: Writing block broadcast_3_piece0 to disk
23/09/17 19:13:31 INFO BlockManager: Dropping block broadcast_3 from memory
23/09/17 19:13:31 INFO BlockManager: Writing block broadcast_3 to disk
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_41_2 from memory
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_18_9 from memory
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_18_3 from memory
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_18_0 from memory
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_18_6 from memory
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_59_0 from memory
23/09/17 19:13:32 INFO BlockManager: Writing block rdd_59_0 to disk
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_59_3 from memory
23/09/17 19:13:32 INFO BlockManager: Writing block rdd_59_3 to disk
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_59_6 from memory
23/09/17 19:13:32 INFO BlockManager: Writing block rdd_59_6 to disk
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_59_9 from memory
23/09/17 19:13:32 INFO BlockManager: Writing block rdd_59_9 to disk
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_35_2 from memory
23/09/17 19:13:32 INFO BlockManager: Dropping block broadcast_14_piece0 from memory
23/09/17 19:13:32 INFO BlockManager: Writing block broadcast_14_piece0 to disk
23/09/17 19:13:32 INFO BlockManager: Dropping block broadcast_14 from memory
23/09/17 19:13:32 INFO BlockManager: Writing block broadcast_14 to disk
23/09/17 19:13:32 INFO BlockManager: Dropping block rdd_89_2 from memory
23/09/17 19:13:32 INFO BlockManager: Writing block rdd_89_2 to disk
23/09/17 19:13:32 INFO MemoryStore: After dropping 15 blocks, free memory is 6.0 GiB
23/09/17 19:14:23 INFO MemoryStore: Will not store rdd_101_3
23/09/17 19:14:23 WARN MemoryStore: Not enough space to cache rdd_101_3 in memory! (computed 4.0 GiB so far)
23/09/17 19:14:23 INFO MemoryStore: Memory use = 13.0 MiB (blocks) + 4.0 GiB (scratch space shared across 1 tasks(s)) = 4.0 GiB. Storage limit = 6.0 GiB.
23/09/17 19:14:23 WARN BlockManager: Persisting block rdd_101_3 to disk instead.
23/09/17 19:28:31 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
23/09/17 19:28:31 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
tdown