Spark Executor Command: "/usr/java8/jdk1.8.0_202/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx8192M" "-Dspark.driver.port=43225" "-Dspark.rpc.message.maxSize=200" "-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-XX:+HeapDumpOnOutOfMemoryError" "-Xloggc:/tmp/memLog.txt" "-XX:+UseCompressedOops" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:43225" "--executor-id" "0" "--hostname" "5.161.188.154" "--cores" "5" "--app-id" "app-20230915162736-0001" "--worker-url" "spark://Worker@5.161.188.154:46635"
========================================

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
23/09/15 16:27:37 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 7142@entity-resolution
23/09/15 16:27:37 INFO SignalUtils: Registering signal handler for TERM
23/09/15 16:27:37 INFO SignalUtils: Registering signal handler for HUP
23/09/15 16:27:37 INFO SignalUtils: Registering signal handler for INT
23/09/15 16:27:37 WARN Utils: Your hostname, entity-resolution resolves to a loopback address: 127.0.1.1; using 5.161.188.154 instead (on interface eth0)
23/09/15 16:27:37 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
23/09/15 16:27:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/09/15 16:27:37 INFO SecurityManager: Changing view acls to: root
23/09/15 16:27:37 INFO SecurityManager: Changing modify acls to: root
23/09/15 16:27:37 INFO SecurityManager: Changing view acls groups to: 
23/09/15 16:27:37 INFO SecurityManager: Changing modify acls groups to: 
23/09/15 16:27:37 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
23/09/15 16:27:38 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:43225 after 61 ms (0 ms spent in bootstraps)
23/09/15 16:27:38 INFO SecurityManager: Changing view acls to: root
23/09/15 16:27:38 INFO SecurityManager: Changing modify acls to: root
23/09/15 16:27:38 INFO SecurityManager: Changing view acls groups to: 
23/09/15 16:27:38 INFO SecurityManager: Changing modify acls groups to: 
23/09/15 16:27:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
23/09/15 16:27:38 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:43225 after 2 ms (0 ms spent in bootstraps)
23/09/15 16:27:38 INFO DiskBlockManager: Created local directory at /tmp/spark-b5306174-a655-4af8-962d-ed2dcefb130b/executor-45383340-0225-489e-a9c2-100820a49510/blockmgr-9b3ab3ba-f1d9-45c4-bcc3-7349cb0355d7
23/09/15 16:27:38 INFO MemoryStore: MemoryStore started with capacity 6.1 GiB
23/09/15 16:27:38 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:43225
23/09/15 16:27:38 INFO WorkerWatcher: Connecting to worker spark://Worker@5.161.188.154:46635
23/09/15 16:27:38 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:46635 after 1 ms (0 ms spent in bootstraps)
23/09/15 16:27:38 INFO WorkerWatcher: Successfully connected to spark://Worker@5.161.188.154:46635
23/09/15 16:27:38 INFO ResourceUtils: ==============================================================
23/09/15 16:27:38 INFO ResourceUtils: No custom resources configured for spark.executor.
23/09/15 16:27:38 INFO ResourceUtils: ==============================================================
23/09/15 16:27:38 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
23/09/15 16:27:38 INFO Executor: Starting executor ID 0 on host 5.161.188.154
23/09/15 16:27:38 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41361.
23/09/15 16:27:38 INFO NettyBlockTransferService: Server created on 5.161.188.154:41361
23/09/15 16:27:38 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/09/15 16:27:38 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(0, 5.161.188.154, 41361, None)
23/09/15 16:27:38 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(0, 5.161.188.154, 41361, None)
23/09/15 16:27:38 INFO BlockManager: Initialized BlockManager: BlockManagerId(0, 5.161.188.154, 41361, None)
23/09/15 16:27:38 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:43225/jars/postgresql-42.6.0.jar with timestamp 1694795255238
23/09/15 16:27:38 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:43225 after 1 ms (0 ms spent in bootstraps)
23/09/15 16:27:38 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:43225/jars/postgresql-42.6.0.jar to /tmp/spark-b5306174-a655-4af8-962d-ed2dcefb130b/executor-45383340-0225-489e-a9c2-100820a49510/spark-7550ef00-03a4-4c5f-90e3-60d727ec1739/fetchFileTemp3188841938896609751.tmp
23/09/15 16:27:38 INFO Utils: Copying /tmp/spark-b5306174-a655-4af8-962d-ed2dcefb130b/executor-45383340-0225-489e-a9c2-100820a49510/spark-7550ef00-03a4-4c5f-90e3-60d727ec1739/4762760891694795255238_cache to /opt/spark/work/app-20230915162736-0001/0/./postgresql-42.6.0.jar
23/09/15 16:27:38 INFO Executor: Adding file:/opt/spark/work/app-20230915162736-0001/0/./postgresql-42.6.0.jar to class loader
23/09/15 16:27:38 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:43225/jars/zingg-0.3.4-SNAPSHOT.jar with timestamp 1694795255238
23/09/15 16:27:38 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:43225/jars/zingg-0.3.4-SNAPSHOT.jar to /tmp/spark-b5306174-a655-4af8-962d-ed2dcefb130b/executor-45383340-0225-489e-a9c2-100820a49510/spark-7550ef00-03a4-4c5f-90e3-60d727ec1739/fetchFileTemp3181251597245113474.tmp
23/09/15 16:27:38 INFO Utils: Copying /tmp/spark-b5306174-a655-4af8-962d-ed2dcefb130b/executor-45383340-0225-489e-a9c2-100820a49510/spark-7550ef00-03a4-4c5f-90e3-60d727ec1739/-589628741694795255238_cache to /opt/spark/work/app-20230915162736-0001/0/./zingg-0.3.4-SNAPSHOT.jar
23/09/15 16:27:38 INFO Executor: Adding file:/opt/spark/work/app-20230915162736-0001/0/./zingg-0.3.4-SNAPSHOT.jar to class loader
23/09/15 16:27:40 INFO CoarseGrainedExecutorBackend: Got assigned task 1
23/09/15 16:27:40 INFO CoarseGrainedExecutorBackend: Got assigned task 4
23/09/15 16:27:40 INFO CoarseGrainedExecutorBackend: Got assigned task 7
23/09/15 16:27:40 INFO CoarseGrainedExecutorBackend: Got assigned task 10
23/09/15 16:27:40 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
23/09/15 16:27:40 INFO Executor: Running task 10.0 in stage 0.0 (TID 10)
23/09/15 16:27:40 INFO Executor: Running task 4.0 in stage 0.0 (TID 4)
23/09/15 16:27:40 INFO Executor: Running task 7.0 in stage 0.0 (TID 7)
23/09/15 16:27:40 INFO TorrentBroadcast: Started reading broadcast variable 1 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:27:40 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:37097 after 2 ms (0 ms spent in bootstraps)
23/09/15 16:27:40 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 7.9 KiB, free 6.1 GiB)
23/09/15 16:27:40 INFO TorrentBroadcast: Reading broadcast variable 1 took 88 ms
23/09/15 16:27:40 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 15.3 KiB, free 6.1 GiB)
23/09/15 16:27:41 INFO CodeGenerator: Code generated in 165.726508 ms
23/09/15 16:27:41 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 4194304-8388608, partition values: [empty row]
23/09/15 16:27:41 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 29360128-33554432, partition values: [empty row]
23/09/15 16:27:41 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 41943040-46137344, partition values: [empty row]
23/09/15 16:27:41 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 16777216-20971520, partition values: [empty row]
23/09/15 16:27:41 INFO CodeGenerator: Code generated in 11.738714 ms
23/09/15 16:27:41 INFO TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:27:41 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:44199 after 4 ms (0 ms spent in bootstraps)
23/09/15 16:27:41 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 24.9 KiB, free 6.1 GiB)
23/09/15 16:27:41 INFO TorrentBroadcast: Reading broadcast variable 0 took 27 ms
23/09/15 16:27:41 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 394.6 KiB, free 6.1 GiB)
23/09/15 16:27:42 INFO Executor: Finished task 4.0 in stage 0.0 (TID 4). 1886 bytes result sent to driver
23/09/15 16:27:42 INFO Executor: Finished task 7.0 in stage 0.0 (TID 7). 1929 bytes result sent to driver
23/09/15 16:27:42 INFO Executor: Finished task 10.0 in stage 0.0 (TID 10). 1929 bytes result sent to driver
23/09/15 16:27:42 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1929 bytes result sent to driver
23/09/15 16:27:43 INFO CoarseGrainedExecutorBackend: Got assigned task 15
23/09/15 16:27:43 INFO Executor: Running task 2.0 in stage 2.0 (TID 15)
23/09/15 16:27:43 INFO CoarseGrainedExecutorBackend: Got assigned task 18
23/09/15 16:27:43 INFO Executor: Running task 5.0 in stage 2.0 (TID 18)
23/09/15 16:27:43 INFO CoarseGrainedExecutorBackend: Got assigned task 21
23/09/15 16:27:43 INFO Executor: Running task 8.0 in stage 2.0 (TID 21)
23/09/15 16:27:43 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache
23/09/15 16:27:43 INFO CoarseGrainedExecutorBackend: Got assigned task 24
23/09/15 16:27:43 INFO Executor: Running task 11.0 in stage 2.0 (TID 24)
23/09/15 16:27:43 INFO TorrentBroadcast: Started reading broadcast variable 4 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:27:43 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:36475 after 1 ms (0 ms spent in bootstraps)
23/09/15 16:27:43 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 16.6 KiB, free 6.1 GiB)
23/09/15 16:27:43 INFO TorrentBroadcast: Reading broadcast variable 4 took 22 ms
23/09/15 16:27:43 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 41.1 KiB, free 6.1 GiB)
23/09/15 16:27:43 INFO CodeGenerator: Code generated in 27.27561 ms
23/09/15 16:27:44 INFO CodeGenerator: Code generated in 23.922457 ms
23/09/15 16:27:44 INFO CodeGenerator: Code generated in 31.040549 ms
23/09/15 16:27:44 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 20971520-25165824, partition values: [empty row]
23/09/15 16:27:44 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 46137344-47316767, partition values: [empty row]
23/09/15 16:27:44 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 33554432-37748736, partition values: [empty row]
23/09/15 16:27:44 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 8388608-12582912, partition values: [empty row]
23/09/15 16:27:44 INFO CodeGenerator: Code generated in 20.714049 ms
23/09/15 16:27:44 INFO TorrentBroadcast: Started reading broadcast variable 3 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:27:44 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 24.9 KiB, free 6.1 GiB)
23/09/15 16:27:44 INFO TorrentBroadcast: Reading broadcast variable 3 took 12 ms
23/09/15 16:27:44 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 394.6 KiB, free 6.1 GiB)
23/09/15 16:27:44 INFO CodeGenerator: Code generated in 67.152131 ms
23/09/15 16:27:45 INFO MemoryStore: Block rdd_18_11 stored as values in memory (estimated size 966.3 KiB, free 6.1 GiB)
23/09/15 16:27:45 INFO CodeGenerator: Code generated in 5.175239 ms
23/09/15 16:27:45 INFO CodeGenerator: Code generated in 18.573973 ms
23/09/15 16:27:45 INFO Executor: Finished task 11.0 in stage 2.0 (TID 24). 2451 bytes result sent to driver
23/09/15 16:27:45 INFO MemoryStore: Block rdd_18_5 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB)
23/09/15 16:27:45 INFO MemoryStore: Block rdd_18_2 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB)
23/09/15 16:27:45 INFO Executor: Finished task 5.0 in stage 2.0 (TID 18). 2451 bytes result sent to driver
23/09/15 16:27:45 INFO MemoryStore: Block rdd_18_8 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB)
23/09/15 16:27:45 INFO Executor: Finished task 8.0 in stage 2.0 (TID 21). 2451 bytes result sent to driver
23/09/15 16:27:45 INFO Executor: Finished task 2.0 in stage 2.0 (TID 15). 2451 bytes result sent to driver
23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 27
23/09/15 16:27:46 INFO Executor: Running task 2.0 in stage 4.0 (TID 27)
23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 30
23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 33
23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 36
23/09/15 16:27:46 INFO Executor: Running task 8.0 in stage 4.0 (TID 33)
23/09/15 16:27:46 INFO Executor: Running task 11.0 in stage 4.0 (TID 36)
23/09/15 16:27:46 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache
23/09/15 16:27:46 INFO Executor: Running task 5.0 in stage 4.0 (TID 30)
23/09/15 16:27:46 INFO TorrentBroadcast: Started reading broadcast variable 6 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:27:46 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 16.8 KiB, free 6.1 GiB)
23/09/15 16:27:46 INFO TorrentBroadcast: Reading broadcast variable 6 took 10 ms
23/09/15 16:27:46 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 43.0 KiB, free 6.1 GiB)
23/09/15 16:27:46 INFO BlockManager: Found block rdd_18_2 locally
23/09/15 16:27:46 INFO BlockManager: Found block rdd_18_5 locally
23/09/15 16:27:46 INFO BlockManager: Found block rdd_18_8 locally
23/09/15 16:27:46 INFO BlockManager: Found block rdd_18_11 locally
23/09/15 16:27:46 INFO CodeGenerator: Code generated in 62.158886 ms
23/09/15 16:27:46 INFO CodeGenerator: Code generated in 17.49095 ms
23/09/15 16:27:46 INFO Executor: Finished task 11.0 in stage 4.0 (TID 36). 2326 bytes result sent to driver
23/09/15 16:27:46 INFO Executor: Finished task 2.0 in stage 4.0 (TID 27). 2326 bytes result sent to driver
23/09/15 16:27:46 INFO Executor: Finished task 8.0 in stage 4.0 (TID 33). 2326 bytes result sent to driver
23/09/15 16:27:46 INFO Executor: Finished task 5.0 in stage 4.0 (TID 30). 2326 bytes result sent to driver
23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 38
23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 41
23/09/15 16:27:46 INFO Executor: Running task 0.0 in stage 5.0 (TID 38)
23/09/15 16:27:46 INFO Executor: Running task 3.0 in stage 5.0 (TID 41)
23/09/15 16:27:46 INFO MapOutputTrackerWorker: Updating epoch to 3 and clearing cache
23/09/15 16:27:46 INFO TorrentBroadcast: Started reading broadcast variable 7 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:27:46 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 9.5 KiB, free 6.1 GiB)
23/09/15 16:27:46 INFO TorrentBroadcast: Reading broadcast variable 7 took 6 ms
23/09/15 16:27:46 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 23.7 KiB, free 6.1 GiB)
23/09/15 16:27:46 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 2, fetching them
23/09/15 16:27:46 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 2, fetching them
23/09/15 16:27:46 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:43225)
23/09/15 16:27:46 INFO MapOutputTrackerWorker: Got the output locations
23/09/15 16:27:46 INFO ShuffleBlockFetcherIterator: Getting 12 (3.1 MiB) non-empty blocks including 4 (934.7 KiB) local and 8 (2.2 MiB) host-local and 0 (0.0 B) remote blocks
23/09/15 16:27:46 INFO ShuffleBlockFetcherIterator: Getting 12 (3.1 MiB) non-empty blocks including 4 (934.7 KiB) local and 8 (2.2 MiB) host-local and 0 (0.0 B) remote blocks
23/09/15 16:27:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 12 ms
23/09/15 16:27:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 12 ms
23/09/15 16:27:47 INFO MemoryStore: Block rdd_35_3 stored as values in memory (estimated size 4.5 MiB, free 6.1 GiB)
23/09/15 16:27:47 INFO MemoryStore: Block rdd_35_0 stored as values in memory (estimated size 4.5 MiB, free 6.1 GiB)
23/09/15 16:27:47 INFO CodeGenerator: Code generated in 11.00997 ms
23/09/15 16:27:47 INFO MemoryStore: Block rdd_41_3 stored as values in memory (estimated size 4.5 MiB, free 6.1 GiB)
23/09/15 16:27:47 INFO Executor: Finished task 3.0 in stage 5.0 (TID 41). 3247 bytes result sent to driver
23/09/15 16:27:47 INFO MemoryStore: Block rdd_41_0 stored as values in memory (estimated size 4.5 MiB, free 6.1 GiB)
23/09/15 16:27:47 INFO Executor: Finished task 0.0 in stage 5.0 (TID 38). 3247 bytes result sent to driver
23/09/15 16:27:47 INFO CoarseGrainedExecutorBackend: Got assigned task 44
23/09/15 16:27:47 INFO Executor: Running task 0.0 in stage 8.0 (TID 44)
23/09/15 16:27:47 INFO CoarseGrainedExecutorBackend: Got assigned task 46
23/09/15 16:27:47 INFO Executor: Running task 3.0 in stage 8.0 (TID 46)
23/09/15 16:27:47 INFO MapOutputTrackerWorker: Updating epoch to 4 and clearing cache
23/09/15 16:27:47 INFO TorrentBroadcast: Started reading broadcast variable 9 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:27:47 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 9.1 KiB, free 6.1 GiB)
23/09/15 16:27:47 INFO TorrentBroadcast: Reading broadcast variable 9 took 8 ms
23/09/15 16:27:47 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 22.0 KiB, free 6.1 GiB)
23/09/15 16:27:47 INFO BlockManager: Found block rdd_41_0 locally
23/09/15 16:27:47 INFO BlockManager: Found block rdd_41_3 locally
23/09/15 16:27:47 INFO MemoryStore: Block taskresult_44 stored as bytes in memory (estimated size 3.1 MiB, free 6.1 GiB)
23/09/15 16:27:47 INFO MemoryStore: Block taskresult_46 stored as bytes in memory (estimated size 3.1 MiB, free 6.1 GiB)
23/09/15 16:27:47 INFO Executor: Finished task 0.0 in stage 8.0 (TID 44). 3235315 bytes result sent via BlockManager)
23/09/15 16:27:47 INFO Executor: Finished task 3.0 in stage 8.0 (TID 46). 3230515 bytes result sent via BlockManager)
23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 49
23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 52
23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 55
23/09/15 16:27:48 INFO Executor: Running task 5.0 in stage 9.0 (TID 52)
23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 58
23/09/15 16:27:48 INFO Executor: Running task 8.0 in stage 9.0 (TID 55)
23/09/15 16:27:48 INFO Executor: Running task 2.0 in stage 9.0 (TID 49)
23/09/15 16:27:48 INFO Executor: Running task 11.0 in stage 9.0 (TID 58)
23/09/15 16:27:48 INFO TorrentBroadcast: Started reading broadcast variable 10 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 61
23/09/15 16:27:48 INFO Executor: Running task 2.0 in stage 10.0 (TID 61)
23/09/15 16:27:48 INFO TorrentBroadcast: Started reading broadcast variable 11 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:27:48 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 18.1 KiB, free 6.1 GiB)
23/09/15 16:27:48 INFO TorrentBroadcast: Reading broadcast variable 10 took 9 ms
23/09/15 16:27:48 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 18.7 KiB, free 6.1 GiB)
23/09/15 16:27:48 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 46.5 KiB, free 6.1 GiB)
23/09/15 16:27:48 INFO TorrentBroadcast: Reading broadcast variable 11 took 8 ms
23/09/15 16:27:48 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 50.4 KiB, free 6.1 GiB)
23/09/15 16:27:48 INFO BlockManager: Found block rdd_18_11 locally
23/09/15 16:27:48 INFO BlockManager: Found block rdd_18_8 locally
23/09/15 16:27:48 INFO BlockManager: Found block rdd_18_5 locally
23/09/15 16:27:48 INFO BlockManager: Found block rdd_18_2 locally
23/09/15 16:27:48 INFO CodeGenerator: Code generated in 21.643561 ms
23/09/15 16:27:48 INFO MemoryStore: Block rdd_59_11 stored as values in memory (estimated size 422.9 KiB, free 6.1 GiB)
23/09/15 16:27:48 INFO CodeGenerator: Code generated in 10.562247 ms
23/09/15 16:27:48 INFO MemoryStore: Block rdd_59_2 stored as values in memory (estimated size 1512.8 KiB, free 6.1 GiB)
23/09/15 16:27:48 INFO MemoryStore: Block rdd_59_8 stored as values in memory (estimated size 1503.5 KiB, free 6.1 GiB)
23/09/15 16:27:48 INFO MemoryStore: Block rdd_59_5 stored as values in memory (estimated size 1493.0 KiB, free 6.1 GiB)
23/09/15 16:27:48 WARN BlockManager: Block rdd_59_2 already exists on this machine; not re-adding it
23/09/15 16:27:48 INFO CodeGenerator: Code generated in 32.227797 ms
23/09/15 16:27:48 INFO CodeGenerator: Code generated in 11.427527 ms
23/09/15 16:27:48 INFO Executor: Finished task 11.0 in stage 9.0 (TID 58). 2671 bytes result sent to driver
23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 62
23/09/15 16:27:48 INFO Executor: Running task 5.0 in stage 10.0 (TID 62)
23/09/15 16:27:48 INFO BlockManager: Found block rdd_59_5 locally
23/09/15 16:27:48 INFO Executor: Finished task 5.0 in stage 9.0 (TID 52). 2671 bytes result sent to driver
23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 63
23/09/15 16:27:48 INFO Executor: Running task 8.0 in stage 10.0 (TID 63)
23/09/15 16:27:48 INFO BlockManager: Found block rdd_59_8 locally
23/09/15 16:27:48 INFO Executor: Finished task 2.0 in stage 9.0 (TID 49). 2671 bytes result sent to driver
23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 64
23/09/15 16:27:48 INFO Executor: Running task 11.0 in stage 10.0 (TID 64)
23/09/15 16:27:48 INFO BlockManager: Found block rdd_59_11 locally
23/09/15 16:27:48 INFO Executor: Finished task 8.0 in stage 9.0 (TID 55). 2714 bytes result sent to driver
23/09/15 16:27:48 INFO Executor: Finished task 2.0 in stage 10.0 (TID 61). 2733 bytes result sent to driver
23/09/15 16:27:48 INFO Executor: Finished task 8.0 in stage 10.0 (TID 63). 2733 bytes result sent to driver
23/09/15 16:27:48 INFO Executor: Finished task 5.0 in stage 10.0 (TID 62). 2733 bytes result sent to driver
23/09/15 16:27:48 INFO Executor: Finished task 11.0 in stage 10.0 (TID 64). 2734 bytes result sent to driver
23/09/15 16:35:53 INFO CoarseGrainedExecutorBackend: Got assigned task 72
23/09/15 16:35:53 INFO CoarseGrainedExecutorBackend: Got assigned task 75
23/09/15 16:35:53 INFO Executor: Running task 0.0 in stage 13.0 (TID 72)
23/09/15 16:35:53 INFO Executor: Running task 3.0 in stage 13.0 (TID 75)
23/09/15 16:35:53 INFO MapOutputTrackerWorker: Updating epoch to 6 and clearing cache
23/09/15 16:35:53 INFO TorrentBroadcast: Started reading broadcast variable 13 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:35:53 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 144.3 KiB, free 6.1 GiB)
23/09/15 16:35:53 INFO TorrentBroadcast: Reading broadcast variable 13 took 6 ms
23/09/15 16:35:53 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 555.6 KiB, free 6.1 GiB)
23/09/15 16:35:53 INFO BlockManager: Found block rdd_35_3 locally
23/09/15 16:35:53 INFO BlockManager: Found block rdd_35_0 locally
23/09/15 16:35:53 INFO CodeGenerator: Code generated in 40.606749 ms
23/09/15 16:35:54 INFO CodeGenerator: Code generated in 10.128228 ms
23/09/15 16:35:54 INFO Executor: Finished task 0.0 in stage 13.0 (TID 72). 2617 bytes result sent to driver
23/09/15 16:35:54 INFO Executor: Finished task 3.0 in stage 13.0 (TID 75). 2617 bytes result sent to driver
23/09/15 16:35:54 INFO CoarseGrainedExecutorBackend: Got assigned task 78
23/09/15 16:35:54 INFO Executor: Running task 2.0 in stage 14.0 (TID 78)
23/09/15 16:35:54 INFO MapOutputTrackerWorker: Updating epoch to 7 and clearing cache
23/09/15 16:35:54 INFO TorrentBroadcast: Started reading broadcast variable 14 with 1 pieces (estimated total size 4.0 MiB)
23/09/15 16:35:54 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 16.7 KiB, free 6.1 GiB)
23/09/15 16:35:54 INFO TorrentBroadcast: Reading broadcast variable 14 took 5 ms
23/09/15 16:35:54 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 42.2 KiB, free 6.1 GiB)
23/09/15 16:35:54 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 6, fetching them
23/09/15 16:35:54 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:43225)
23/09/15 16:35:54 INFO MapOutputTrackerWorker: Got the output locations
23/09/15 16:35:54 INFO ShuffleBlockFetcherIterator: Getting 4 (2.4 MiB) non-empty blocks including 2 (1218.2 KiB) local and 2 (1218.2 KiB) host-local and 0 (0.0 B) remote blocks
23/09/15 16:35:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/15 16:35:54 INFO MemoryStore: Block rdd_89_2 stored as values in memory (estimated size 2.8 MiB, free 6.1 GiB)
23/09/15 16:35:54 INFO CodeGenerator: Code generated in 11.893868 ms
23/09/15 16:35:54 INFO CodeGenerator: Code generated in 8.163301 ms
23/09/15 16:35:54 INFO CodeGenerator: Code generated in 7.973043 ms
23/09/15 16:35:54 INFO CodeGenerator: Code generated in 5.853804 ms
23/09/15 16:35:54 INFO BlockManager: Found block rdd_89_2 locally
23/09/15 16:35:54 INFO CodeGenerator: Code generated in 22.058667 ms
23/09/15 16:35:54 INFO CodeGenerator: Code generated in 11.842082 ms
23/09/15 16:35:54 INFO CodeGenerator: Code generated in 26.591699 ms
23/09/15 16:35:56 INFO MemoryStore: Block rdd_101_2 stored as values in memory (estimated size 12.5 MiB, free 6.0 GiB)
23/09/15 16:35:56 INFO Executor: Finished task 2.0 in stage 14.0 (TID 78). 3959 bytes result sent to driver
23/09/15 16:51:19 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
23/09/15 16:51:19 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
tdown