Spark 3.5.0 伪分布式 Standalone 模式搭建Ubuntu 22.04 单机 2 节点配置与测试1. 环境准备与核心概念在单台物理机或虚拟机上模拟分布式集群环境是开发者验证Spark应用逻辑的高效方式。Standalone模式作为Spark内置的轻量级集群管理器无需依赖Hadoop YARN或Mesos特别适合快速搭建测试环境。以下是关键组件说明Master节点负责资源调度和集群管理默认监听7077端口RPC通信和8080端口Web UIWorker节点执行具体计算任务向Master汇报资源情况伪分布式模式通过单机多进程模拟多节点环境降低硬件需求系统要求Ubuntu 22.04 LTS内核版本5.15Java 8/11推荐OpenJDK 11至少4GB内存建议8GB20GB可用磁盘空间# 验证Java环境 java -version # 应输出类似内容 openjdk version 11.0.22 2024-01-16 OpenJDK Runtime Environment (build 11.0.227-post-Ubuntu-0ubuntu222.04.1)2. 安装与基础配置2.1 获取Spark安装包从Apache镜像站下载预编译版本选择与Hadoop无关的版本wget https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-without-hadoop.tgz sha512sum spark-3.5.0-bin-without-hadoop.tgz | grep -i $(curl -s https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-without-hadoop.tgz.sha512)解压并建立软链接tar -xzf spark-3.5.0-bin-without-hadoop.tgz -C /opt ln -s /opt/spark-3.5.0-bin-without-hadoop /opt/spark2.2 环境变量配置编辑/etc/profile.d/spark.shexport SPARK_HOME/opt/spark export PATH$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin export PYSPARK_PYTHONpython3立即生效配置source /etc/profile3. 伪分布式集群配置3.1 核心配置文件修改进入配置目录并复制模板文件cd $SPARK_HOME/conf cp spark-env.sh.template spark-env.sh cp workers.template workersspark-env.sh关键配置# 设置Master主机本机IP或主机名 export SPARK_MASTER_HOST$(hostname -I | awk {print $1}) # 资源配置根据实际硬件调整 export SPARK_WORKER_CORES2 export SPARK_WORKER_MEMORY2g export SPARK_DAEMON_MEMORY1g # Java环境 export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 # 日志配置可选 export SPARK_LOG_DIR/var/log/spark export SPARK_WORKER_DIR/tmp/spark/workerworkers文件配置localhost注意伪分布式模式下workers文件只需包含localhost即可Spark会自动启动多个Worker进程3.2 目录权限设置创建日志和工作目录sudo mkdir -p /var/log/spark /tmp/spark/worker sudo chown -R $USER:$USER /var/log/spark /tmp/spark4. 集群启动与验证4.1 启动集群服务使用内置脚本启动服务start-all.sh验证进程是否正常jps # 应包含以下进程 # Master # Worker4.2 Web UI访问通过浏览器访问Master的Web界面默认8080端口http://your-server-ip:8080正常界面应显示1个Alive的WorkerWorker的CPU和内存资源信息当前运行的Applications应为空4.3 命令行验证通过spark-shell连接集群spark-shell --master spark://$(hostname -I | awk {print $1}):7077在Scala REPL中执行测试val data 1 to 10000 val distData sc.parallelize(data) distData.map(_ * 2).reduce(_ _)5. 实战案例WordCount作业提交5.1 准备测试数据创建示例文件echo hello spark hello world /tmp/test.txt5.2 提交Python版WordCount创建wordcount.pyfrom pyspark.sql import SparkSession spark SparkSession.builder.appName(WordCount).getOrCreate() lines spark.read.text(/tmp/test.txt).rdd.map(lambda r: r[0]) counts lines.flatMap(lambda x: x.split( )) \ .map(lambda word: (word, 1)) \ .reduceByKey(lambda a, b: a b) output counts.collect() for (word, count) in output: print(%s: %i % (word, count)) spark.stop()提交作业spark-submit \ --master spark://$(hostname -I | awk {print $1}):7077 \ wordcount.py5.3 提交Scala版WordCount创建WordCount.scalaimport org.apache.spark.{SparkConf, SparkContext} object WordCount { def main(args: Array[String]) { val conf new SparkConf().setAppName(WordCount) val sc new SparkContext(conf) val textFile sc.textFile(/tmp/test.txt) val counts textFile.flatMap(line line.split( )) .map(word (word, 1)) .reduceByKey(_ _) counts.saveAsTextFile(/tmp/wordcount_output) sc.stop() } }编译打包后提交spark-submit \ --class WordCount \ --master spark://$(hostname -I | awk {print $1}):7077 \ target/scala-2.12/wordcount_2.12-1.0.jar6. 性能调优与问题排查6.1 常见配置优化参数推荐值说明spark.executor.memory1g-2g每个Executor内存spark.driver.memory1gDriver进程内存spark.default.parallelismWorker核心数×2默认并行度spark.sql.shuffle.partitions200SQL操作分区数在spark-defaults.conf中添加spark.executor.memory 2g spark.driver.memory 1g spark.default.parallelism 46.2 典型问题解决问题1端口冲突java.net.BindException: Address already in use解决方案修改spark-env.sh中的端口号或终止占用端口的进程问题2内存不足java.lang.OutOfMemoryError: Java heap space解决方案增加SPARK_WORKER_MEMORY值或减少并行任务数问题3Python依赖缺失ImportError: No module named pandas解决方案使用--py-files提交依赖包或在所有节点安装相同Python环境7. 集群管理与监控7.1 常用管理命令命令功能stop-all.sh停止所有服务start-history-server.sh启动历史服务器spark-class org.apache.spark.deploy.Client手动提交应用7.2 日志查看技巧Master日志$SPARK_HOME/logs/spark--master-*.outWorker日志$SPARK_HOME/logs/spark--worker-*.out应用日志Web UI的Executors标签页实时监控日志tail -f /var/log/spark/spark--org.apache.spark.deploy.master.Master-1-*.out8. 扩展配置8.1 启用历史服务器创建日志目录hdfs dfs -mkdir /spark-logs配置spark-defaults.confspark.eventLog.enabled true spark.eventLog.dir hdfs://localhost:9000/spark-logs spark.history.fs.logDirectory hdfs://localhost:9000/spark-logs启动服务start-history-server.sh8.2 集成Jupyter Notebook安装并配置pip install jupyter pyspark echo export PYSPARK_DRIVER_PYTHONjupyter ~/.bashrc echo export PYSPARK_DRIVER_PYTHON_OPTSnotebook --ip0.0.0.0 ~/.bashrc启动带集群支持的Notebookpyspark --master spark://$(hostname -I | awk {print $1}):7077