Mac搭建Hadoop源码阅读环境

1.安装maven，使用的版本是3.3.9,下载apache-maven-3.3.9-bin.tar,解压：
tar -zxvf apache-maven-3.3.9-bin.tar
进入 Maven 安装根目录 cd apache-maven-3.3.9
进入 Maven配置文件目录 cd conf
编辑settings.xml文件 vi settings.xml
设置Maven本地库的路径：这个路径要创建好。
同时在加入阿里云的Maven库

<mirror>
          <id>nexus-aliyun</id>
          <mirrorOf>*</mirrorOf>
          <name>Nexus aliyun</name>
          <url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

或者用开源的中国环境

<mirror>
<id>maven.oschina.net</id>
<mirrorOf>central</mirrorOf>
      <name>one of the central mirrors in China</name>
      <url>http://maven.oschina.net/content/groups/public/</url>
</mirror>

设置Maven环境变量：

export MAVEN_HOME=路径
export PATH=${PATH}:${MAVEN_HOME}/bin:
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
export PATH

更新环境变量 source ~/.bash_profile
mvn -version 能看到相应的版本，则Maven安装成功。

2.安装protobuf，下载protobuf-2.5.0.tar.gz，下载地址： http://pan.baidu.com/s/1pJlZubT
解压：tar -zxvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
依次执行

1 ./configure
2 make
3 make check
4 make install

最后输入 protoc --version命令，如显示libprotoc 2.5.0则安装成功
3、编译Hadoop源码
在Hadoop官网下载hadoop-2.6.0-src.tar.gz源码
解压：tar -zxvf hadoop-2.6.0-src.tar.gz

mvn package -Pdist -DskipTests -Dtar

最后输出：[INFO] Reactor Summary：

[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.040 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  4.994 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 11.060 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  9.013 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.132 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.940 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  9.764 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.025 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 30.464 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:29 min
[INFO] Finished at: 2017-09-08T22:45:03+08:00
[INFO] Final Memory: 187M/1529M

前几天想重新弄一下结果一致报错

JLSdeMacBook-Pro:hadoop-2.6.1-src jls$ mvn package -Pdist -DskipTests -Dstar
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-project:pom:2.6.1
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: javax.servlet.jsp:jsp-api:jar -> duplicate declaration of version 2.1 @ line 563, column 19
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.curator:curator-framework:jar -> duplicate declaration of version 2.6.0 @ line 915, column 18
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.curator:curator-test:jar -> duplicate declaration of version 2.6.0 @ line 920, column 18
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @ line 1154, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-project-dist:pom:2.6.1
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: javax.servlet.jsp:jsp-api:jar -> duplicate declaration of version 2.1 @ org.apache.hadoop:hadoop-project:2.6.1, /Users/jls/Learning/hadoop-src/hadoop-src-new/hadoop-2.6.1-src/hadoop-project/pom.xml, line 563, column 19
.....

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ hadoop-assemblies ---
[INFO] Using default encoding to copy filtered resources.
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-assemblies ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-assemblies ---
[INFO] Using default encoding to copy filtered resources.
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-assemblies ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hadoop-assemblies ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ hadoop-assemblies ---
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ hadoop-assemblies ---
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ hadoop-assemblies ---
[INFO] No sources in project. Archive not created.
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hadoop-assemblies ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-assemblies ---
[INFO] Building jar: /Users/jls/Learning/hadoop-src/hadoop-src-new/hadoop-2.6.1-src/hadoop-assemblies/target/hadoop-assemblies-2.6.1-javadoc.jar
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop Project Dist POM 2.6.1
[INFO] ------------------------------------------------------------------------
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-assembly-plugin/2.3/maven-assembly-plugin-2.3.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.258 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.036 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.304 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.264 s]
[INFO] Apache Hadoop Project Dist POM ..................... FAILURE [  1.234 s]
[INFO] Apache Hadoop Maven Plugins ........................ SKIPPED
[INFO] Apache Hadoop MiniKDC .............................. SKIPPED
[INFO] Apache Hadoop Auth ................................. SKIPPED
[INFO] Apache Hadoop Auth Examples ........................ SKIPPED
[INFO] Apache Hadoop Common ............................... SKIPPED
[INFO] Apache Hadoop NFS .................................. SKIPPED
[INFO] Apache Hadoop KMS .................................. SKIPPED
[INFO] Apache Hadoop Common Project ....................... SKIPPED
[INFO] Apache Hadoop HDFS ................................. SKIPPED
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] hadoop-yarn ........................................ SKIPPED
[INFO] hadoop-yarn-api .................................... SKIPPED
[INFO] hadoop-yarn-common ................................. SKIPPED
[INFO] hadoop-yarn-server ................................. SKIPPED
[INFO] hadoop-yarn-server-common .......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED
[INFO] hadoop-yarn-server-tests ........................... SKIPPED
[INFO] hadoop-yarn-client ................................. SKIPPED
[INFO] hadoop-yarn-applications ........................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED
[INFO] hadoop-yarn-site ................................... SKIPPED
[INFO] hadoop-yarn-registry ............................... SKIPPED
[INFO] hadoop-yarn-project ................................ SKIPPED
[INFO] hadoop-mapreduce-client ............................ SKIPPED
[INFO] hadoop-mapreduce-client-core ....................... SKIPPED
[INFO] hadoop-mapreduce-client-common ..................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle .................... SKIPPED
[INFO] hadoop-mapreduce-client-app ........................ SKIPPED
[INFO] hadoop-mapreduce-client-hs ......................... SKIPPED
[INFO] hadoop-mapreduce-client-jobclient .................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................. SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] hadoop-mapreduce ................................... SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Ant Tasks ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Client ............................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.434 s
[INFO] Finished at: 2018-07-09T10:45:11+08:00
[INFO] Final Memory: 60M/724M
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.apache.maven.plugins:maven-assembly-plugin:2.3 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-assembly-plugin:jar:2.3: Could not transfer artifact org.apache.maven.plugins:maven-assembly-plugin:pom:2.3 from/to central (https://repo.maven.apache.org/maven2): Received fatal alert: protocol_version -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

原来是maven的镜像配置没有配...
配置完成之后完美成功

一点一滴

Mac搭建Hadoop源码阅读环境

没有评论