spark on swarm[standalone mode]
version:2.3.1
创建spark-master
|
1 2 3 4 5 6 7 8 |
docker service create \ --name spark-master \ --hostname spark-master \ --network bigdata \ --replicas 1 \ --detach=true \ --endpoint-mode dnsrr \ cppla/spark-docker master |
创建spark-slave
|
1 2 3 4 5 6 7 |
docker service create \ --name spark-slave \ --network bigdata \ --replicas 3 \ --detach=true \ --endpoint-mode dnsrr \ cppla/spark-docker slave spark://spark-master:7077 |
测试
|
1 2 3 4 5 6 7 |
spark-shell --master spark://spark-master:7077 val lines = sc.textFile("hdfs://hadoop-master:8020/user/hadoop/test/hello.txt") val words = lines.flatMap(_.split(",")) val wc = words.map(word => (word, 1)).reduceByKey(_ + _) wc.collect() val cnt = words.map(word => 1).reduce(_ + _) |
前几天有朋友需要快速搭建大数据测试集群（spark on swarm[standalone mode]），借花献佛 base: newnius Dockerfiles