3.1.3
stderr log page for app-20230915162736-0001/1
Showing 28499 Bytes: 0 - 28499 of 28499
Spark Executor Command: "/usr/java8/jdk1.8.0_202/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx8192M" "-Dspark.driver.port=43225" "-Dspark.rpc.message.maxSize=200" "-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-XX:+HeapDumpOnOutOfMemoryError" "-Xloggc:/tmp/memLog.txt" "-XX:+UseCompressedOops" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:43225" "--executor-id" "1" "--hostname" "5.161.188.154" "--cores" "5" "--app-id" "app-20230915162736-0001" "--worker-url" "spark://Worker@5.161.188.154:36849" ======================================== Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 23/09/15 16:27:37 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 7159@entity-resolution 23/09/15 16:27:37 INFO SignalUtils: Registering signal handler for TERM 23/09/15 16:27:37 INFO SignalUtils: Registering signal handler for HUP 23/09/15 16:27:37 INFO SignalUtils: Registering signal handler for INT 23/09/15 16:27:37 WARN Utils: Your hostname, entity-resolution resolves to a loopback address: 127.0.1.1; using 5.161.188.154 instead (on interface eth0) 23/09/15 16:27:37 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 23/09/15 16:27:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 23/09/15 16:27:37 INFO SecurityManager: Changing view acls to: root 23/09/15 16:27:37 INFO SecurityManager: Changing modify acls to: root 23/09/15 16:27:37 INFO SecurityManager: Changing view acls groups to: 23/09/15 16:27:37 INFO SecurityManager: Changing modify acls groups to: 23/09/15 16:27:37 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 23/09/15 16:27:38 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:43225 after 64 ms (0 ms spent in bootstraps) 23/09/15 16:27:38 INFO SecurityManager: Changing view acls to: root 23/09/15 16:27:38 INFO SecurityManager: Changing modify acls to: root 23/09/15 16:27:38 INFO SecurityManager: Changing view acls groups to: 23/09/15 16:27:38 INFO SecurityManager: Changing modify acls groups to: 23/09/15 16:27:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 23/09/15 16:27:38 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:43225 after 1 ms (0 ms spent in bootstraps) 23/09/15 16:27:38 INFO DiskBlockManager: Created local directory at /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-0670f1b0-df4e-480e-a391-86a62366550f/blockmgr-1286c95a-78c9-4ed2-9de1-a9865677869c 23/09/15 16:27:38 INFO MemoryStore: MemoryStore started with capacity 6.1 GiB 23/09/15 16:27:38 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:43225 23/09/15 16:27:38 INFO WorkerWatcher: Connecting to worker spark://Worker@5.161.188.154:36849 23/09/15 16:27:38 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:36849 after 1 ms (0 ms spent in bootstraps) 23/09/15 16:27:38 INFO WorkerWatcher: Successfully connected to spark://Worker@5.161.188.154:36849 23/09/15 16:27:38 INFO ResourceUtils: ============================================================== 23/09/15 16:27:38 INFO ResourceUtils: No custom resources configured for spark.executor. 23/09/15 16:27:38 INFO ResourceUtils: ============================================================== 23/09/15 16:27:38 INFO CoarseGrainedExecutorBackend: Successfully registered with driver 23/09/15 16:27:38 INFO Executor: Starting executor ID 1 on host 5.161.188.154 23/09/15 16:27:38 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44199. 23/09/15 16:27:38 INFO NettyBlockTransferService: Server created on 5.161.188.154:44199 23/09/15 16:27:38 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 23/09/15 16:27:38 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, 5.161.188.154, 44199, None) 23/09/15 16:27:38 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, 5.161.188.154, 44199, None) 23/09/15 16:27:38 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, 5.161.188.154, 44199, None) 23/09/15 16:27:38 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:43225/jars/postgresql-42.6.0.jar with timestamp 1694795255238 23/09/15 16:27:38 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:43225 after 1 ms (0 ms spent in bootstraps) 23/09/15 16:27:38 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:43225/jars/postgresql-42.6.0.jar to /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-0670f1b0-df4e-480e-a391-86a62366550f/spark-1c363e11-330d-484d-8e96-3286da41e9b2/fetchFileTemp2878011300893205546.tmp 23/09/15 16:27:38 INFO Utils: Copying /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-0670f1b0-df4e-480e-a391-86a62366550f/spark-1c363e11-330d-484d-8e96-3286da41e9b2/4762760891694795255238_cache to /opt/spark/work/app-20230915162736-0001/1/./postgresql-42.6.0.jar 23/09/15 16:27:38 INFO Executor: Adding file:/opt/spark/work/app-20230915162736-0001/1/./postgresql-42.6.0.jar to class loader 23/09/15 16:27:38 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:43225/jars/zingg-0.3.4-SNAPSHOT.jar with timestamp 1694795255238 23/09/15 16:27:38 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:43225/jars/zingg-0.3.4-SNAPSHOT.jar to /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-0670f1b0-df4e-480e-a391-86a62366550f/spark-1c363e11-330d-484d-8e96-3286da41e9b2/fetchFileTemp8752166844136631666.tmp 23/09/15 16:27:38 INFO Utils: Copying /tmp/spark-88076f2a-e5dc-4e36-a772-e443b2e354d0/executor-0670f1b0-df4e-480e-a391-86a62366550f/spark-1c363e11-330d-484d-8e96-3286da41e9b2/-589628741694795255238_cache to /opt/spark/work/app-20230915162736-0001/1/./zingg-0.3.4-SNAPSHOT.jar 23/09/15 16:27:38 INFO Executor: Adding file:/opt/spark/work/app-20230915162736-0001/1/./zingg-0.3.4-SNAPSHOT.jar to class loader 23/09/15 16:27:40 INFO CoarseGrainedExecutorBackend: Got assigned task 0 23/09/15 16:27:40 INFO CoarseGrainedExecutorBackend: Got assigned task 3 23/09/15 16:27:40 INFO CoarseGrainedExecutorBackend: Got assigned task 6 23/09/15 16:27:40 INFO CoarseGrainedExecutorBackend: Got assigned task 9 23/09/15 16:27:40 INFO Executor: Running task 3.0 in stage 0.0 (TID 3) 23/09/15 16:27:40 INFO Executor: Running task 9.0 in stage 0.0 (TID 9) 23/09/15 16:27:40 INFO Executor: Running task 6.0 in stage 0.0 (TID 6) 23/09/15 16:27:40 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) 23/09/15 16:27:40 INFO TorrentBroadcast: Started reading broadcast variable 1 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:40 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:37097 after 2 ms (0 ms spent in bootstraps) 23/09/15 16:27:40 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 7.9 KiB, free 6.1 GiB) 23/09/15 16:27:40 INFO TorrentBroadcast: Reading broadcast variable 1 took 75 ms 23/09/15 16:27:40 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 15.3 KiB, free 6.1 GiB) 23/09/15 16:27:41 INFO CodeGenerator: Code generated in 169.502038 ms 23/09/15 16:27:41 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 0-4194304, partition values: [empty row] 23/09/15 16:27:41 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 12582912-16777216, partition values: [empty row] 23/09/15 16:27:41 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 25165824-29360128, partition values: [empty row] 23/09/15 16:27:41 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 37748736-41943040, partition values: [empty row] 23/09/15 16:27:41 INFO CodeGenerator: Code generated in 13.209638 ms 23/09/15 16:27:41 INFO TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:41 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 24.9 KiB, free 6.1 GiB) 23/09/15 16:27:41 INFO TorrentBroadcast: Reading broadcast variable 0 took 11 ms 23/09/15 16:27:41 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 394.6 KiB, free 6.1 GiB) 23/09/15 16:27:42 INFO Executor: Finished task 6.0 in stage 0.0 (TID 6). 1929 bytes result sent to driver 23/09/15 16:27:42 INFO Executor: Finished task 9.0 in stage 0.0 (TID 9). 1929 bytes result sent to driver 23/09/15 16:27:42 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 1929 bytes result sent to driver 23/09/15 16:27:42 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1929 bytes result sent to driver 23/09/15 16:27:43 INFO CoarseGrainedExecutorBackend: Got assigned task 14 23/09/15 16:27:43 INFO Executor: Running task 1.0 in stage 2.0 (TID 14) 23/09/15 16:27:43 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache 23/09/15 16:27:43 INFO CoarseGrainedExecutorBackend: Got assigned task 17 23/09/15 16:27:43 INFO Executor: Running task 4.0 in stage 2.0 (TID 17) 23/09/15 16:27:43 INFO CoarseGrainedExecutorBackend: Got assigned task 20 23/09/15 16:27:43 INFO Executor: Running task 7.0 in stage 2.0 (TID 20) 23/09/15 16:27:43 INFO CoarseGrainedExecutorBackend: Got assigned task 23 23/09/15 16:27:43 INFO Executor: Running task 10.0 in stage 2.0 (TID 23) 23/09/15 16:27:43 INFO TorrentBroadcast: Started reading broadcast variable 4 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:43 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:36475 after 1 ms (0 ms spent in bootstraps) 23/09/15 16:27:43 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 16.6 KiB, free 6.1 GiB) 23/09/15 16:27:43 INFO TorrentBroadcast: Reading broadcast variable 4 took 26 ms 23/09/15 16:27:43 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 41.1 KiB, free 6.1 GiB) 23/09/15 16:27:43 INFO CodeGenerator: Code generated in 45.283735 ms 23/09/15 16:27:44 INFO CodeGenerator: Code generated in 28.830873 ms 23/09/15 16:27:44 INFO CodeGenerator: Code generated in 29.679243 ms 23/09/15 16:27:44 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 41943040-46137344, partition values: [empty row] 23/09/15 16:27:44 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 4194304-8388608, partition values: [empty row] 23/09/15 16:27:44 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 16777216-20971520, partition values: [empty row] 23/09/15 16:27:44 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 29360128-33554432, partition values: [empty row] 23/09/15 16:27:44 INFO CodeGenerator: Code generated in 22.008819 ms 23/09/15 16:27:44 INFO TorrentBroadcast: Started reading broadcast variable 3 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:44 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:41361 after 2 ms (0 ms spent in bootstraps) 23/09/15 16:27:44 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 24.9 KiB, free 6.1 GiB) 23/09/15 16:27:44 INFO TorrentBroadcast: Reading broadcast variable 3 took 27 ms 23/09/15 16:27:44 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 394.6 KiB, free 6.1 GiB) 23/09/15 16:27:44 INFO CodeGenerator: Code generated in 94.913547 ms 23/09/15 16:27:45 INFO MemoryStore: Block rdd_18_4 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB) 23/09/15 16:27:45 INFO CodeGenerator: Code generated in 4.654016 ms 23/09/15 16:27:45 INFO MemoryStore: Block rdd_18_1 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB) 23/09/15 16:27:45 INFO MemoryStore: Block rdd_18_10 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB) 23/09/15 16:27:45 INFO CodeGenerator: Code generated in 19.123689 ms 23/09/15 16:27:45 INFO MemoryStore: Block rdd_18_7 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB) 23/09/15 16:27:45 INFO Executor: Finished task 7.0 in stage 2.0 (TID 20). 2451 bytes result sent to driver 23/09/15 16:27:45 INFO Executor: Finished task 10.0 in stage 2.0 (TID 23). 2451 bytes result sent to driver 23/09/15 16:27:45 INFO Executor: Finished task 4.0 in stage 2.0 (TID 17). 2451 bytes result sent to driver 23/09/15 16:27:45 INFO Executor: Finished task 1.0 in stage 2.0 (TID 14). 2451 bytes result sent to driver 23/09/15 16:27:45 INFO CoarseGrainedExecutorBackend: Got assigned task 25 23/09/15 16:27:45 INFO Executor: Running task 0.0 in stage 3.0 (TID 25) 23/09/15 16:27:45 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache 23/09/15 16:27:45 INFO TorrentBroadcast: Started reading broadcast variable 5 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:45 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 6.1 GiB) 23/09/15 16:27:45 INFO TorrentBroadcast: Reading broadcast variable 5 took 7 ms 23/09/15 16:27:45 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 10.1 KiB, free 6.1 GiB) 23/09/15 16:27:45 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 1, fetching them 23/09/15 16:27:45 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:43225) 23/09/15 16:27:45 INFO MapOutputTrackerWorker: Got the output locations 23/09/15 16:27:46 INFO ShuffleBlockFetcherIterator: Getting 12 (720.0 B) non-empty blocks including 4 (240.0 B) local and 8 (480.0 B) host-local and 0 (0.0 B) remote blocks 23/09/15 16:27:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 11 ms 23/09/15 16:27:46 INFO CodeGenerator: Code generated in 9.627855 ms 23/09/15 16:27:46 INFO Executor: Finished task 0.0 in stage 3.0 (TID 25). 2450 bytes result sent to driver 23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 26 23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 29 23/09/15 16:27:46 INFO Executor: Running task 4.0 in stage 4.0 (TID 29) 23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 32 23/09/15 16:27:46 INFO Executor: Running task 7.0 in stage 4.0 (TID 32) 23/09/15 16:27:46 INFO TorrentBroadcast: Started reading broadcast variable 6 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:46 INFO Executor: Running task 1.0 in stage 4.0 (TID 26) 23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 35 23/09/15 16:27:46 INFO Executor: Running task 10.0 in stage 4.0 (TID 35) 23/09/15 16:27:46 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 16.8 KiB, free 6.1 GiB) 23/09/15 16:27:46 INFO TorrentBroadcast: Reading broadcast variable 6 took 21 ms 23/09/15 16:27:46 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 43.0 KiB, free 6.1 GiB) 23/09/15 16:27:46 INFO BlockManager: Found block rdd_18_10 locally 23/09/15 16:27:46 INFO BlockManager: Found block rdd_18_7 locally 23/09/15 16:27:46 INFO BlockManager: Found block rdd_18_4 locally 23/09/15 16:27:46 INFO BlockManager: Found block rdd_18_1 locally 23/09/15 16:27:46 INFO CodeGenerator: Code generated in 31.679203 ms 23/09/15 16:27:46 INFO CodeGenerator: Code generated in 19.415299 ms 23/09/15 16:27:46 INFO Executor: Finished task 7.0 in stage 4.0 (TID 32). 2326 bytes result sent to driver 23/09/15 16:27:46 INFO Executor: Finished task 4.0 in stage 4.0 (TID 29). 2326 bytes result sent to driver 23/09/15 16:27:46 INFO Executor: Finished task 10.0 in stage 4.0 (TID 35). 2326 bytes result sent to driver 23/09/15 16:27:46 INFO Executor: Finished task 1.0 in stage 4.0 (TID 26). 2326 bytes result sent to driver 23/09/15 16:27:46 INFO CoarseGrainedExecutorBackend: Got assigned task 40 23/09/15 16:27:46 INFO Executor: Running task 2.0 in stage 5.0 (TID 40) 23/09/15 16:27:46 INFO MapOutputTrackerWorker: Updating epoch to 3 and clearing cache 23/09/15 16:27:46 INFO TorrentBroadcast: Started reading broadcast variable 7 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:46 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 9.5 KiB, free 6.1 GiB) 23/09/15 16:27:46 INFO TorrentBroadcast: Reading broadcast variable 7 took 7 ms 23/09/15 16:27:46 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 23.7 KiB, free 6.1 GiB) 23/09/15 16:27:46 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 2, fetching them 23/09/15 16:27:46 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:43225) 23/09/15 16:27:46 INFO MapOutputTrackerWorker: Got the output locations 23/09/15 16:27:46 INFO ShuffleBlockFetcherIterator: Getting 12 (3.2 MiB) non-empty blocks including 4 (1165.0 KiB) local and 8 (2.0 MiB) host-local and 0 (0.0 B) remote blocks 23/09/15 16:27:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 23/09/15 16:27:47 INFO MemoryStore: Block rdd_35_2 stored as values in memory (estimated size 4.5 MiB, free 6.1 GiB) 23/09/15 16:27:47 INFO CodeGenerator: Code generated in 11.276452 ms 23/09/15 16:27:47 INFO MemoryStore: Block rdd_41_2 stored as values in memory (estimated size 4.5 MiB, free 6.1 GiB) 23/09/15 16:27:47 INFO Executor: Finished task 2.0 in stage 5.0 (TID 40). 3247 bytes result sent to driver 23/09/15 16:27:47 INFO CoarseGrainedExecutorBackend: Got assigned task 42 23/09/15 16:27:47 INFO Executor: Running task 0.0 in stage 6.0 (TID 42) 23/09/15 16:27:47 INFO MapOutputTrackerWorker: Updating epoch to 4 and clearing cache 23/09/15 16:27:47 INFO TorrentBroadcast: Started reading broadcast variable 8 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:47 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 6.1 GiB) 23/09/15 16:27:47 INFO TorrentBroadcast: Reading broadcast variable 8 took 7 ms 23/09/15 16:27:47 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 10.1 KiB, free 6.1 GiB) 23/09/15 16:27:47 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 3, fetching them 23/09/15 16:27:47 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:43225) 23/09/15 16:27:47 INFO MapOutputTrackerWorker: Got the output locations 23/09/15 16:27:47 INFO ShuffleBlockFetcherIterator: Getting 4 (240.0 B) non-empty blocks including 1 (60.0 B) local and 3 (180.0 B) host-local and 0 (0.0 B) remote blocks 23/09/15 16:27:47 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 23/09/15 16:27:47 INFO Executor: Finished task 0.0 in stage 6.0 (TID 42). 2450 bytes result sent to driver 23/09/15 16:27:47 INFO CoarseGrainedExecutorBackend: Got assigned task 45 23/09/15 16:27:47 INFO Executor: Running task 2.0 in stage 8.0 (TID 45) 23/09/15 16:27:47 INFO TorrentBroadcast: Started reading broadcast variable 9 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:47 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 9.1 KiB, free 6.1 GiB) 23/09/15 16:27:47 INFO TorrentBroadcast: Reading broadcast variable 9 took 7 ms 23/09/15 16:27:47 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 22.0 KiB, free 6.1 GiB) 23/09/15 16:27:47 INFO BlockManager: Found block rdd_41_2 locally 23/09/15 16:27:47 INFO MemoryStore: Block taskresult_45 stored as bytes in memory (estimated size 3.1 MiB, free 6.1 GiB) 23/09/15 16:27:47 INFO Executor: Finished task 2.0 in stage 8.0 (TID 45). 3233077 bytes result sent via BlockManager) 23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 47 23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 50 23/09/15 16:27:48 INFO Executor: Running task 1.0 in stage 9.0 (TID 47) 23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 53 23/09/15 16:27:48 INFO Executor: Running task 4.0 in stage 9.0 (TID 50) 23/09/15 16:27:48 INFO Executor: Running task 7.0 in stage 9.0 (TID 53) 23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 56 23/09/15 16:27:48 INFO Executor: Running task 10.0 in stage 9.0 (TID 56) 23/09/15 16:27:48 INFO TorrentBroadcast: Started reading broadcast variable 10 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 59 23/09/15 16:27:48 INFO Executor: Running task 1.0 in stage 10.0 (TID 59) 23/09/15 16:27:48 INFO TorrentBroadcast: Started reading broadcast variable 11 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:27:48 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 18.1 KiB, free 6.1 GiB) 23/09/15 16:27:48 INFO TorrentBroadcast: Reading broadcast variable 10 took 28 ms 23/09/15 16:27:48 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 46.5 KiB, free 6.1 GiB) 23/09/15 16:27:48 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 18.7 KiB, free 6.1 GiB) 23/09/15 16:27:48 INFO TorrentBroadcast: Reading broadcast variable 11 took 31 ms 23/09/15 16:27:48 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 50.4 KiB, free 6.1 GiB) 23/09/15 16:27:48 INFO BlockManager: Found block rdd_18_4 locally 23/09/15 16:27:48 INFO BlockManager: Found block rdd_18_7 locally 23/09/15 16:27:48 INFO BlockManager: Found block rdd_18_10 locally 23/09/15 16:27:48 INFO BlockManager: Found block rdd_18_1 locally 23/09/15 16:27:48 INFO CodeGenerator: Code generated in 13.236329 ms 23/09/15 16:27:48 INFO MemoryStore: Block rdd_59_10 stored as values in memory (estimated size 1502.0 KiB, free 6.1 GiB) 23/09/15 16:27:48 INFO MemoryStore: Block rdd_59_1 stored as values in memory (estimated size 1491.9 KiB, free 6.1 GiB) 23/09/15 16:27:48 WARN BlockManager: Block rdd_59_1 already exists on this machine; not re-adding it 23/09/15 16:27:48 INFO MemoryStore: Block rdd_59_7 stored as values in memory (estimated size 1495.6 KiB, free 6.1 GiB) 23/09/15 16:27:48 INFO MemoryStore: Block rdd_59_4 stored as values in memory (estimated size 1506.1 KiB, free 6.1 GiB) 23/09/15 16:27:48 INFO CodeGenerator: Code generated in 53.868684 ms 23/09/15 16:27:48 INFO CodeGenerator: Code generated in 29.893938 ms 23/09/15 16:27:48 INFO CodeGenerator: Code generated in 35.412915 ms 23/09/15 16:27:48 INFO Executor: Finished task 7.0 in stage 9.0 (TID 53). 2671 bytes result sent to driver 23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 65 23/09/15 16:27:48 INFO Executor: Running task 4.0 in stage 10.0 (TID 65) 23/09/15 16:27:48 INFO BlockManager: Found block rdd_59_4 locally 23/09/15 16:27:48 INFO Executor: Finished task 1.0 in stage 9.0 (TID 47). 2671 bytes result sent to driver 23/09/15 16:27:48 INFO Executor: Finished task 4.0 in stage 9.0 (TID 50). 2671 bytes result sent to driver 23/09/15 16:27:48 INFO Executor: Finished task 10.0 in stage 9.0 (TID 56). 2671 bytes result sent to driver 23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 69 23/09/15 16:27:48 INFO CoarseGrainedExecutorBackend: Got assigned task 70 23/09/15 16:27:48 INFO Executor: Finished task 1.0 in stage 10.0 (TID 59). 2733 bytes result sent to driver 23/09/15 16:27:48 INFO Executor: Running task 10.0 in stage 10.0 (TID 70) 23/09/15 16:27:48 INFO Executor: Running task 7.0 in stage 10.0 (TID 69) 23/09/15 16:27:48 INFO BlockManager: Found block rdd_59_7 locally 23/09/15 16:27:48 INFO BlockManager: Found block rdd_59_10 locally 23/09/15 16:27:48 INFO Executor: Finished task 4.0 in stage 10.0 (TID 65). 2777 bytes result sent to driver 23/09/15 16:27:48 INFO Executor: Finished task 10.0 in stage 10.0 (TID 70). 2777 bytes result sent to driver 23/09/15 16:27:48 INFO Executor: Finished task 7.0 in stage 10.0 (TID 69). 2777 bytes result sent to driver 23/09/15 16:35:53 INFO CoarseGrainedExecutorBackend: Got assigned task 74 23/09/15 16:35:53 INFO Executor: Running task 2.0 in stage 13.0 (TID 74) 23/09/15 16:35:53 INFO MapOutputTrackerWorker: Updating epoch to 6 and clearing cache 23/09/15 16:35:53 INFO TorrentBroadcast: Started reading broadcast variable 13 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:35:53 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 144.3 KiB, free 6.1 GiB) 23/09/15 16:35:53 INFO TorrentBroadcast: Reading broadcast variable 13 took 5 ms 23/09/15 16:35:53 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 555.6 KiB, free 6.1 GiB) 23/09/15 16:35:53 INFO BlockManager: Found block rdd_35_2 locally 23/09/15 16:35:53 INFO CodeGenerator: Code generated in 36.943648 ms 23/09/15 16:35:53 INFO CodeGenerator: Code generated in 8.263912 ms 23/09/15 16:35:54 INFO Executor: Finished task 2.0 in stage 13.0 (TID 74). 2574 bytes result sent to driver 23/09/15 16:35:54 INFO CoarseGrainedExecutorBackend: Got assigned task 77 23/09/15 16:35:54 INFO Executor: Running task 1.0 in stage 14.0 (TID 77) 23/09/15 16:35:54 INFO MapOutputTrackerWorker: Updating epoch to 7 and clearing cache 23/09/15 16:35:54 INFO TorrentBroadcast: Started reading broadcast variable 14 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 16:35:54 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 16.7 KiB, free 6.1 GiB) 23/09/15 16:35:54 INFO TorrentBroadcast: Reading broadcast variable 14 took 6 ms 23/09/15 16:35:54 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 42.2 KiB, free 6.1 GiB) 23/09/15 16:35:54 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 6, fetching them 23/09/15 16:35:54 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:43225) 23/09/15 16:35:54 INFO MapOutputTrackerWorker: Got the output locations 23/09/15 16:35:54 INFO ShuffleBlockFetcherIterator: Getting 4 (2.4 MiB) non-empty blocks including 1 (609.1 KiB) local and 3 (1827.2 KiB) host-local and 0 (0.0 B) remote blocks 23/09/15 16:35:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 23/09/15 16:35:54 INFO MemoryStore: Block rdd_89_1 stored as values in memory (estimated size 2.8 MiB, free 6.1 GiB) 23/09/15 16:35:54 INFO CodeGenerator: Code generated in 11.789382 ms 23/09/15 16:35:54 INFO CodeGenerator: Code generated in 7.385232 ms 23/09/15 16:35:54 INFO CodeGenerator: Code generated in 5.519213 ms 23/09/15 16:35:54 INFO CodeGenerator: Code generated in 3.666027 ms 23/09/15 16:35:54 INFO BlockManager: Found block rdd_89_1 locally 23/09/15 16:35:54 INFO CodeGenerator: Code generated in 8.17838 ms 23/09/15 16:35:54 INFO CodeGenerator: Code generated in 34.308367 ms 23/09/15 16:35:54 INFO CodeGenerator: Code generated in 36.74799 ms 23/09/15 16:35:56 INFO MemoryStore: Block rdd_101_1 stored as values in memory (estimated size 12.1 MiB, free 6.0 GiB) 23/09/15 16:35:56 INFO Executor: Finished task 1.0 in stage 14.0 (TID 77). 3959 bytes result sent to driver 23/09/15 16:51:19 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown 23/09/15 16:51:19 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM tdown