一、Spark WordCount动手实践 我们通过Spark WordCount动手实践，编写单词计数代码；在wordcount.scala的基础上，从数据流动的视角深入分析Spark RDD的数据处理过程。
首先需要建立一个文本文件helloSpark.txt，helloSpark.txt的文本内容如下。
Hello Spark Hello Scala Hello Hadoop Hello Flink Spark is Awesome 然后在Eclipse中编写wordcount.scala的代码如下。
import org.apache.spark.SparkConf import org.apache.spark.SparkContext
object wordcount { def main(args: Array[String]): Unit = {
// 第1步：创建Spark的配置对象SparkConf，设置Spark程序运行时的配置信息， val conf = new SparkConf().setAppName("My First Spark APP").setMaster("local") // 第2步：创建SparkContext对象 val sc = new SparkContext(conf) // 第3步：根据具体的数据来源来创建RDD val lines = sc.textFile("helloSpark.txt", 1) // 第4步：对初始的RDD进行Transformation级别的处理，如通过map、filter等 val words = lines.flatMap{line=>line.split(" ")} val pairs = words.map{word=>(word,1)} val wordCountsOdered = pairs.reduceByKey(_+_).map( pair=>(pair._2,pair._1) ).sortByKey(false).map(pair=>(pair._2,pair._1)) wordCountsOdered.collect.foreach(wordNumberPair=>println(wordNumberPair._1+" : "+wordNumberPair._2)) sc.stop()} } 在Eclipse中运行程序，wordcount.scala的运行结果如下：
二、解析RDD生成的内部机制 下面详细解析一下wordcount.scala的运行原理。
（1）从数据流动视角解密WordCount，使用Spark作单词计数统计，搞清楚数据到底是怎么流动的。
（2）从RDD依赖关系的视角解密WordCount。Spark中的一切操作都是RDD，后面的RDD对前面的RDD有依赖关系。
（3）DAG与血统Lineage的思考。
在wordcount.scala的基础上，我们从数据流动的视角分析数据到底是怎么处理的。下面有一张WordCount数据处理过程图，由于图片较大，为了方便阅读，将原图分成两张图，如下面两张图所示。
数据在生产环境中默认在HDFS中进行分布式存储，如果在分布式集群中，我们的机器会分成不同的节点对数据进行处理，这里我们在本地测试，重点关注数据是怎么流动的。处理的第一步是获取数据，读取数据会生成HadoopRDD。
在WordCount.scala中，单击sc.textFile进入Spark框架，SparkContext.scala的textFile的源码如下。
/**
Read a text file from HDFS, a local file system (available on all nodes), or anyHadoop-supported file system URI, and return it as an RDD of Strings. */ def textFile( path: String, minPartitions: Int = defaultMinPartitions): RDD[String] = withScope { assertNotStopped() hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text], minPartitions).map(pair => pair._2.toString).setName(path) } 下面看一下hadoopFile的源码，HadoopRDD从Hdfs上读取分布式数据，并且以数据分片的方式存在于集群中。所谓的数据分片，就是把我们要处理的数据分成不同的部分，例如，在集群中有4个节点，粗略的划分可以认为将数据分成4个部分，4条语句就分成4个部分。例如，Hello Spark在第一台机器上，Hello Hadoop在第二台机器上，Hello Flink在第三台机器上，Spark is Awesome在第四台机器上。HadoopRDD帮助我们从磁盘上读取数据，计算的时候会分布式地放入内存中，Spark运行在Hadoop上，要借助Hadoop来读取数据。Spark的特点包括：分布式、基于内存（部分基于磁盘）、可迭代；默认分片策略Block多大，分片就多大。但这种说法不完全准确，因为分片记录可能跨两个Block，所以一个分片不会严格地等于Block的大小。例如，HDFS的Block大小是128MB的话，分片可能多几个字节或少几个字节。分片不一定小于128MB，因为如果最后一条记录跨两个Block，分片会把最后一条记录放在前一个分片中。这里，HadoopRDD用了4个数据分片，设想为128M左右。
hadoopFile的源码如下。
def hadoopFile[K, V]( path: String, inputFormatClass: Class[_ <: InputFormat[K, V]], keyClass: Class[K], valueClass: Class[V], minPartitions: Int = defaultMinPartitions): RDD[(K, V)] = withScope { assertNotStopped()
// This is a hack to enforce loading hdfs-site.xml. // See SPARK-11227 for details. FileSystem.getLocal(hadoopConfiguration)
// A Hadoop configuration can be about 10 KB, which is pretty big, so broadcast it. val confBroadcast = broadcast(new SerializableConfiguration(hadoopConfiguration)) val setInputPathsFunc = (jobConf: JobConf) => FileInputFormat.setInputPaths(jobConf, path) new HadoopRDD( this, confBroadcast, Some(setInputPathsFunc), inputFormatClass, keyClass, valueClass, minPartitions).setName(path) } SparkContext.scala的textFile源码中，调用hadoopFile方法后进行了map转换操作，map对读取的每一行数据进行转换，读入的数据是一个Tuple，Key值为索引，Value值为每行数据的内容，生成MapPartitionsRDD。这里，map(pair => pair._2.toString)是基于HadoopRDD产生的Partition去掉的行Key产生的Value，第二个元素是读取的每行数据内容。MapPartitionsRDD是Spark框架产生的，运行中可能产生一个RDD，也可能产生两个RDD。例如，textFile中Spark框架就产生了两个RDD，即HadoopRDD和MapPartitionsRDD。下面是map的源码。
/**
Return a new RDD by applying a function to all elements of this RDD. */ def map[U: ClassTag](f: T => U): RDD[U] = withScope { val cleanF = sc.clean(f) new MapPartitionsRDD[U, T](this, (context, pid, iter) => iter.map(cleanF)) } 我们再来看一下WordCount业务代码，对读取的每行数据进行flatMap转换。这里，flatMap对RDD中的每一个Partition的每一行数据内容进行单词切分，如有4个Partition分别进行单词切分，将“Hello Spark”切分成单词“Hello”和“Spark”，对每一个Partition中的每一行进行单词切分并合并成一个大的单词实例的集合。flatMap转换生成的仍然是MapPartitionsRDD：/**
Return a new RDD by first applying a function to all elements of thisRDD, and then flattening the results. */ def flatMap[U: ClassTag](f: T => TraversableOnce[U]): RDD[U] = withScope { val cleanF = sc.clean(f) new MapPartitionsRDD[U, T](this, (context, pid, iter) => iter.flatMap(cleanF)) } 继续WordCount业务代码，计数之后进行一个关键的reduceByKey操作，对全局的数据进行计数统计。reduceByKey对相同的Key进行Value的累计（包括Local和Reducer级别，同时Reduce）。reduceByKey在MapPartitionsRDD之后，在Local reduce级别本地进行了统计，这里也是MapPartitionsRDD。例如，在本地将（Hello，1），（Spark，1），（Hello，1），（Scala，1）汇聚成（Hello，2），（Spark，1），（Scala，1）。Shuffle之前的Local Reduce操作主要负责本地局部统计，并且把统计以后的结果按照分区策略放到不同的file。举一个简单的例子，如果下一个阶段Stage是3个并行度，每个Partition进行local reduce以后，将自己的数据分成3种类型，最简单的方式是根据HashCode按3取模。
PairRDDFunctions.scala的reduceByKey的源码如下。
/**
Merge the values for each key using an associative and commutative reduce function. This willalso perform the merging locally on each mapper before sending results to a reducer, similarlyto a “combiner” in MapReduce. Output will be hash-partitioned with the existing partitioner/parallelism level. */ def reduceByKey(func: (V, V) => V): RDD[(K, V)] = self.withScope { reduceByKey(defaultPartitioner(self), func) } 至此，前面所有的操作都是一个Stage，一个Stage意味着什么：完全基于内存操作。父Stage：Stage内部的操作是基于内存迭代的，也可以进行Cache，这样速度快很多。不同于Hadoop的Map Redcue，Hadoop Map Redcue每次都要经过磁盘。reduceByKey在Local reduce本地汇聚以后生成的MapPartitionsRDD仍属于父Stage；然后reduceByKey展开真正的Shuffle操作，Shuffle是Spark甚至整个分布式系统的性能瓶颈，Shuffle产生ShuffleRDD，ShuffledRDD就变成另一个Stage，为什么是变成另外一个Stage？因为要网络传输，网络传输不能在内存中进行迭代。
从WordCount业务代码pairs.reduceByKey(+)中看一下PairRDDFunctions.scala的reduceByKey的源码。
1 2 3 4 5 6 7 8 /**
Merge the values for each key using an associative and commutative reduce function. This willalso perform the merging locally on each mapper before sending results to a reducer, similarlyto a “combiner” in MapReduce. */ def reduceByKey(partitioner: Partitioner, func: (V, V) => V): RDD[(K, V)] = self.withScope { combineByKeyWithClassTag[V]((v: V) => v, func, func, partitioner) } reduceByKey内部调用了combineByKeyWithClassTag方法。下面看一下PairRDDFunctions. scala的combineByKeyWithClassTag的源码。1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 def combineByKeyWithClassTag[C]( createCombiner: V => C, mergeValue: (C, V) => C, mergeCombiners: (C, C) => C, partitioner: Partitioner, mapSideCombine: Boolean = true, serializer: Serializer = null)(implicit ct: ClassTag[C]): RDD[(K, C)] = self.withScope { require(mergeCombiners != null, “mergeCombiners must be defined”) // required as of Spark 0.9.0 if (keyClass.isArray) { if (mapSideCombine) { throw new SparkException(“Cannot use map-side combining with array keys.”) } if (partitioner.isInstanceOf[HashPartitioner]) { throw new SparkException(“Default partitioner cannot partition array keys.”) } } val aggregator = new Aggregator[K, V, C]( self.context.clean(createCombiner), self.context.clean(mergeValue), self.context.clean(mergeCombiners)) if (self.partitioner == Some(partitioner)) { self.mapPartitions(iter => { val context = TaskContext.get() new InterruptibleIterator(context, aggregator.combineValuesByKey(iter, context)) }, preservesPartitioning = true) } else { new ShuffledRDD[K, V, C](self, partitioner) .setSerializer(serializer) .setAggregator(aggregator) .setMapSideCombine(mapSideCombine) } } 在combineByKeyWithClassTag方法中就用new()函数创建了ShuffledRDD。
前面假设有4台机器并行计算，每台机器在自己的内存中进行迭代计算，现在产生Shuffle，数据就要进行分类，MapPartitionsRDD数据根据Hash已经分好类，我们就抓取MapPartitionsRDD中的数据。我们从第一台机器中获取的内容为（Hello，2），从第二台机器中获取的内容为（Hello，1），从第三台机器中获取的内容为（Hello，1），把所有的Hello都抓过来。同样，我们把其他的数据（Hadoop，1），（Flink，1）……都抓过来。
这就是Shuffle的过程，根据数据的分类拿到自己需要的数据。注意，MapPartitionsRDD属于第一个Stage，是父Stage，内部基于内存进行迭代，不需要操作都要读写磁盘，所以速度非常快；从计算算子的角度讲，reduceByKey发生在哪里？reduceByKey发生的计算过程包括两个RDD：一个是MapPartitionsRDD；一个是ShuffledRDD。ShuffledRDD要产生网络通信。
reduceByKey之后，我们将结果收集起来，进行全局级别的reduce，产生reduceByKey的最后结果，如将（Hello，2），（Hello，1），（Hello，1）在内部变成（Hello，4），其他数据也类似统计。这里reduceByKey之后，如果通过Collect将数据收集起来，就会产生MapPartitionsRDD。从Collect的角度讲，MapPartitionsRDD的作用是将结果收集起来发送给Driver；从saveAsTextFile输出到Hdfs的角度讲，例如输出（Hello，4），其中Hello是key，4是Value吗？不是！这里（Hello，4）就是value，这就需要设计一个key出来。
下面是RDD.scala的saveAsTextFile方法。
/**
Save this RDD as a text file, using string representations of elements. */ def saveAsTextFile(path: String): Unit = withScope { // https://issues.apache.org/jira/browse/SPARK-2075 // // NullWritable is a Comparable in Hadoop 1.+, so the compiler cannot find an implicit // Ordering for it and will use the default null. However, it’s a Comparable[NullWritable] // in Hadoop 2.+, so the compiler will call the implicit Ordering.ordered method to create an // Ordering for NullWritable. That’s why the compiler will generate different anonymous // classes for saveAsTextFile in Hadoop 1.+ and Hadoop 2.+. // // Therefore, here we provide an explicit Ordering null to make sure the compiler generate // same bytecodes for saveAsTextFile. val nullWritableClassTag = implicitly[ClassTag[NullWritable]] val textClassTag = implicitly[ClassTag[Text]] val r = this.mapPartitions { iter => val text = new Text() iter.map { x => text.set(x.toString) (NullWritable.get(), text) } } RDD.rddToPairRDDFunctions®(nullWritableClassTag, textClassTag, null) .saveAsHadoopFileTextOutputFormat[NullWritable, Text] } RDD.scala的saveAsTextFile方法中的iter.map {x=>text.set(x.toString) (NullWritable.get(), text)}，这里，key转换成Null，value就是内容本身（Hello，4）。saveAsHadoopFile中的TextOutputFormat要求输出的是key-value的格式，而我们处理的是内容。回顾一下，之前我们在textFile读入数据的时候，读入split分片将key去掉了，计算的是value。因此，输出时，须将丢失的key重新弄进来，这里key对我们没有意义，但key对Spark框架有意义，只有value对我们有意义。第一次计算的时候我们把key丢弃了，所以最后往HDFS写结果的时候需要生成key，这符合对称法则和能量守恒形式。 http://bbs.quyouji.com/topic.asp?id=1639 http://bbs.quyouji.com/topic.asp?id=1640 http://bbs.quyouji.com/topic.asp?id=1641 http://bbs.quyouji.com/topic.asp?id=1642 http://bbs.quyouji.com/topic.asp?id=1643 http://bbs.quyouji.com/topic.asp?id=1644 http://bbs.quyouji.com/topic.asp?id=1645 http://bbs.quyouji.com/topic.asp?id=1646 http://bbs.quyouji.com/topic.asp?id=1647 http://bbs.quyouji.com/topic.asp?id=1648 http://bbs.quyouji.com/topic.asp?id=1649 http://bbs.quyouji.com/topic.asp?id=1650 http://bbs.quyouji.com/topic.asp?id=1651 http://bbs.quyouji.com/topic.asp?id=1652 http://bbs.quyouji.com/topic.asp?id=1653 http://bbs.quyouji.com/topic.asp?id=1654 http://bbs.quyouji.com/topic.asp?id=1655 http://bbs.quyouji.com/topic.asp?id=1656 http://bbs.quyouji.com/topic.asp?id=1657 http://bbs.quyouji.com/topic.asp?id=1658 http://bbs.quyouji.com/topic.asp?id=1659 http://bbs.quyouji.com/topic.asp?id=1660 http://bbs.quyouji.com/topic.asp?id=1661 http://bbs.quyouji.com/topic.asp?id=1662 http://bbs.quyouji.com/topic.asp?id=1663 http://bbs.quyouji.com/topic.asp?id=1664 http://bbs.quyouji.com/topic.asp?id=1665 http://bbs.quyouji.com/topic.asp?id=1666 http://bbs.quyouji.com/topic.asp?id=1667 http://bbs.quyouji.com/topic.asp?id=1668 http://bbs.quyouji.com/topic.asp?id=1669 http://bbs.quyouji.com/topic.asp?id=1670 http://bbs.quyouji.com/topic.asp?id=1671 http://bbs.quyouji.com/topic.asp?id=1672 http://bbs.quyouji.com/topic.asp?id=1673 http://bbs.quyouji.com/topic.asp?id=1674 http://bbs.quyouji.com/topic.asp?id=1675 http://bbs.quyouji.com/topic.asp?id=1676 http://bbs.quyouji.com/topic.asp?id=1677 http://bbs.quyouji.com/topic.asp?id=1678 http://bbs.quyouji.com/topic.asp?id=1679 http://bbs.quyouji.com/topic.asp?id=1680 http://bbs.quyouji.com/topic.asp?id=1681 http://bbs.quyouji.com/topic.asp?id=1682 http://bbs.quyouji.com/topic.asp?id=1683 http://bbs.quyouji.com/topic.asp?id=1684 http://bbs.quyouji.com/topic.asp?id=1685 http://bbs.quyouji.com/topic.asp?id=1686 http://bbs.quyouji.com/topic.asp?id=1687 http://bbs.quyouji.com/topic.asp?id=1688 http://bbs.quyouji.com/topic.asp?id=1689 http://bbs.quyouji.com/topic.asp?id=1690 http://bbs.quyouji.com/topic.asp?id=1691 http://bbs.quyouji.com/topic.asp?id=1692 http://bbs.quyouji.com/topic.asp?id=1693 http://bbs.quyouji.com/topic.asp?id=1694 http://bbs.quyouji.com/topic.asp?id=1695 http://bbs.quyouji.com/topic.asp?id=1696 http://bbs.quyouji.com/topic.asp?id=1697 http://bbs.quyouji.com/topic.asp?id=1698 http://bbs.quyouji.com/topic.asp?id=1699 http://bbs.quyouji.com/topic.asp?id=1700 http://bbs.quyouji.com/topic.asp?id=1701 http://bbs.quyouji.com/topic.asp?id=1702 http://bbs.quyouji.com/topic.asp?id=1703 http://bbs.quyouji.com/topic.asp?id=1704 http://bbs.quyouji.com/topic.asp?id=1705 http://bbs.quyouji.com/topic.asp?id=1706 http://bbs.quyouji.com/topic.asp?id=1707 http://bbs.quyouji.com/topic.asp?id=1708 http://bbs.quyouji.com/topic.asp?id=1709 http://bbs.quyouji.com/topic.asp?id=1710 http://bbs.quyouji.com/topic.asp?id=1711 http://bbs.quyouji.com/topic.asp?id=1712 http://bbs.quyouji.com/topic.asp?id=1713 http://bbs.quyouji.com/topic.asp?id=1714 http://bbs.quyouji.com/topic.asp?id=1715 http://bbs.quyouji.com/topic.asp?id=1716 http://bbs.quyouji.com/topic.asp?id=1717 http://bbs.quyouji.com/topic.asp?id=1718 http://bbs.quyouji.com/topic.asp?id=1719 http://bbs.quyouji.com/topic.asp?id=1720 http://bbs.quyouji.com/topic.asp?id=1721 http://bbs.quyouji.com/topic.asp?id=1722 http://bbs.quyouji.com/topic.asp?id=1723 http://bbs.quyouji.com/topic.asp?id=1724 http://bbs.quyouji.com/topic.asp?id=1725 http://bbs.quyouji.com/topic.asp?id=1726 http://bbs.quyouji.com/topic.asp?id=1727 http://bbs.quyouji.com/topic.asp?id=1728 http://bbs.quyouji.com/topic.asp?id=1729 http://bbs.quyouji.com/topic.asp?id=1730 http://bbs.quyouji.com/topic.asp?id=1731 http://bbs.quyouji.com/topic.asp?id=1732 http://bbs.quyouji.com/topic.asp?id=1733 http://bbs.quyouji.com/topic.asp?id=1734 http://bbs.quyouji.com/topic.asp?id=1735 http://bbs.quyouji.com/topic.asp?id=1736 http://bbs.quyouji.com/topic.asp?id=1737 http://bbs.quyouji.com/topic.asp?id=1738 http://bbs.quyouji.com/topic.asp?id=1739 http://bbs.quyouji.com/topic.asp?id=1740 http://bbs.quyouji.com/topic.asp?id=1741 http://bbs.quyouji.com/topic.asp?id=1742 http://bbs.quyouji.com/topic.asp?id=1743 http://bbs.quyouji.com/topic.asp?id=1744 http://bbs.quyouji.com/topic.asp?id=1745 http://bbs.quyouji.com/topic.asp?id=1746 http://bbs.quyouji.com/topic.asp?id=1747 http://bbs.quyouji.com/topic.asp?id=1748 http://bbs.quyouji.com/topic.asp?id=1749 http://bbs.quyouji.com/topic.asp?id=1750 http://bbs.quyouji.com/topic.asp?id=1751 http://bbs.quyouji.com/topic.asp?id=1752 http://bbs.quyouji.com/topic.asp?id=1753 http://bbs.quyouji.com/topic.asp?id=1754 http://bbs.quyouji.com/topic.asp?id=1755 http://bbs.quyouji.com/topic.asp?id=1756 http://bbs.quyouji.com/topic.asp?id=1757 http://bbs.quyouji.com/topic.asp?id=1758 http://bbs.quyouji.com/topic.asp?id=1759 http://bbs.quyouji.com/topic.asp?id=1760 http://bbs.quyouji.com/topic.asp?id=1761 http://bbs.quyouji.com/topic.asp?id=1762 http://bbs.quyouji.com/topic.asp?id=1763 http://bbs.quyouji.com/topic.asp?id=1764 http://bbs.quyouji.com/topic.asp?id=1765 http://bbs.quyouji.com/topic.asp?id=1766 http://bbs.quyouji.com/topic.asp?id=1767 http://bbs.quyouji.com/topic.asp?id=1768 http://bbs.quyouji.com/topic.asp?id=1769 http://bbs.quyouji.com/topic.asp?id=1770 http://bbs.quyouji.com/topic.asp?id=1771 http://bbs.quyouji.com/topic.asp?id=1772 http://bbs.quyouji.com/topic.asp?id=1773 http://bbs.quyouji.com/topic.asp?id=1774 http://bbs.quyouji.com/topic.asp?id=1775 http://bbs.quyouji.com/topic.asp?id=1776 http://bbs.quyouji.com/topic.asp?id=1777 http://bbs.quyouji.com/topic.asp?id=1778 http://bbs.quyouji.com/topic.asp?id=1779 http://bbs.quyouji.com/topic.asp?id=1780 http://bbs.quyouji.com/topic.asp?id=1781 http://bbs.quyouji.com/topic.asp?id=1782 http://bbs.quyouji.com/topic.asp?id=1783 http://bbs.quyouji.com/topic.asp?id=1784 http://bbs.quyouji.com/topic.asp?id=1785 http://bbs.quyouji.com/topic.asp?id=1786 http://bbs.quyouji.com/topic.asp?id=1787 http://bbs.quyouji.com/topic.asp?id=1788 http://bbs.quyouji.com/topic.asp?id=1789 http://bbs.quyouji.com/topic.asp?id=1790 http://bbs.quyouji.com/topic.asp?id=1791 http://bbs.quyouji.com/topic.asp?id=1792 http://bbs.quyouji.com/topic.asp?id=1793 http://bbs.quyouji.com/topic.asp?id=1794 http://bbs.quyouji.com/topic.asp?id=1795 http://bbs.quyouji.com/topic.asp?id=1796 http://bbs.quyouji.com/topic.asp?id=1797 http://bbs.quyouji.com/topic.asp?id=1798 http://bbs.quyouji.com/topic.asp?id=1799 http://bbs.quyouji.com/topic.asp?id=1800 http://bbs.quyouji.com/topic.asp?id=1801 http://bbs.quyouji.com/topic.asp?id=1802 http://bbs.quyouji.com/topic.asp?id=1803 http://bbs.quyouji.com/topic.asp?id=1804 http://bbs.quyouji.com/topic.asp?id=1805 http://bbs.quyouji.com/topic.asp?id=1806 http://bbs.quyouji.com/topic.asp?id=1807 http://bbs.quyouji.com/topic.asp?id=1808 http://bbs.quyouji.com/topic.asp?id=1809 http://bbs.quyouji.com/topic.asp?id=1810 http://bbs.quyouji.com/topic.asp?id=1811 http://bbs.quyouji.com/topic.asp?id=1812 http://bbs.quyouji.com/topic.asp?id=1813 http://bbs.quyouji.com/topic.asp?id=1814 http://bbs.quyouji.com/topic.asp?id=1815 http://bbs.quyouji.com/topic.asp?id=1816 http://bbs.quyouji.com/topic.asp?id=1817 http://bbs.quyouji.com/topic.asp?id=1818 http://bbs.quyouji.com/topic.asp?id=1819 http://bbs.quyouji.com/topic.asp?id=1820 http://bbs.quyouji.com/topic.asp?id=1821 http://bbs.quyouji.com/topic.asp?id=1822 http://bbs.quyouji.com/topic.asp?id=1823 http://bbs.quyouji.com/topic.asp?id=1824 http://bbs.quyouji.com/topic.asp?id=1825 http://bbs.quyouji.com/topic.asp?id=1826 http://bbs.quyouji.com/topic.asp?id=1827 http://bbs.quyouji.com/topic.asp?id=1828 http://bbs.quyouji.com/topic.asp?id=1829 http://bbs.quyouji.com/topic.asp?id=1830 http://bbs.quyouji.com/topic.asp?id=1831 http://bbs.quyouji.com/topic.asp?id=1832 http://bbs.quyouji.com/topic.asp?id=1833 http://bbs.quyouji.com/topic.asp?id=1834 http://bbs.quyouji.com/topic.asp?id=1835 http://bbs.quyouji.com/topic.asp?id=1836 http://bbs.quyouji.com/topic.asp?id=1837 http://bbs.quyouji.com/topic.asp?id=1838 http://bbs.quyouji.com/topic.asp?id=1839 http://bbs.quyouji.com/topic.asp?id=1840 http://bbs.quyouji.com/topic.asp?id=1841 http://bbs.quyouji.com/topic.asp?id=1842 http://bbs.quyouji.com/topic.asp?id=1843 http://bbs.quyouji.com/topic.asp?id=1844 http://bbs.quyouji.com/topic.asp?id=1845 http://bbs.quyouji.com/topic.asp?id=1846 http://bbs.quyouji.com/topic.asp?id=1847 http://bbs.quyouji.com/topic.asp?id=1848 http://bbs.quyouji.com/topic.asp?id=1849 http://bbs.quyouji.com/topic.asp?id=1850 http://bbs.quyouji.com/topic.asp?id=1851 http://bbs.quyouji.com/topic.asp?id=1852 http://bbs.quyouji.com/topic.asp?id=1853 http://bbs.quyouji.com/topic.asp?id=1854 http://bbs.quyouji.com/topic.asp?id=1855 http://bbs.quyouji.com/topic.asp?id=1856 http://bbs.quyouji.com/topic.asp?id=1857 http://bbs.quyouji.com/topic.asp?id=1858 http://bbs.quyouji.com/topic.asp?id=1859 http://bbs.quyouji.com/topic.asp?id=1860 http://bbs.quyouji.com/topic.asp?id=1861 http://bbs.quyouji.com/topic.asp?id=1862 http://bbs.quyouji.com/topic.asp?id=1863 http://bbs.quyouji.com/topic.asp?id=1864 http://bbs.quyouji.com/topic.asp?id=1865 http://bbs.quyouji.com/topic.asp?id=1866 http://bbs.quyouji.com/topic.asp?id=1867 http://bbs.quyouji.com/topic.asp?id=1868 http://bbs.quyouji.com/topic.asp?id=1869 http://bbs.quyouji.com/topic.asp?id=1870 http://bbs.quyouji.com/topic.asp?id=1871 http://bbs.quyouji.com/topic.asp?id=1872 http://bbs.quyouji.com/topic.asp?id=1873 http://bbs.quyouji.com/topic.asp?id=1874 http://bbs.quyouji.com/topic.asp?id=1875 http://bbs.quyouji.com/topic.asp?id=1876 http://bbs.quyouji.com/topic.asp?id=1877 http://bbs.quyouji.com/topic.asp?id=1878 http://bbs.quyouji.com/topic.asp?id=1879 http://bbs.quyouji.com/topic.asp?id=1880 http://bbs.quyouji.com/topic.asp?id=1881 http://bbs.quyouji.com/topic.asp?id=1882 http://bbs.quyouji.com/topic.asp?id=1883 http://bbs.quyouji.com/topic.asp?id=1884 http://bbs.quyouji.com/topic.asp?id=1885 http://bbs.quyouji.com/topic.asp?id=1886 http://bbs.quyouji.com/topic.asp?id=1887 http://bbs.quyouji.com/topic.asp?id=1888 http://bbs.quyouji.com/topic.asp?id=1889 http://bbs.quyouji.com/topic.asp?id=1890 http://bbs.quyouji.com/topic.asp?id=1891 http://bbs.quyouji.com/topic.asp?id=1892 http://bbs.quyouji.com/topic.asp?id=1893 http://bbs.quyouji.com/topic.asp?id=1894 http://bbs.quyouji.com/topic.asp?id=1895 http://bbs.quyouji.com/topic.asp?id=1896 http://bbs.quyouji.com/topic.asp?id=1897 http://bbs.quyouji.com/topic.asp?id=1898 http://bbs.quyouji.com/topic.asp?id=1899 http://bbs.quyouji.com/topic.asp?id=1900 http://bbs.quyouji.com/topic.asp?id=1901 http://bbs.quyouji.com/topic.asp?id=1902 http://bbs.quyouji.com/topic.asp?id=1903 http://bbs.quyouji.com/topic.asp?id=1904 http://bbs.quyouji.com/topic.asp?id=1905 http://bbs.quyouji.com/topic.asp?id=1906 http://bbs.quyouji.com/topic.asp?id=1907 http://bbs.quyouji.com/topic.asp?id=1908 http://bbs.quyouji.com/topic.asp?id=1909 http://bbs.quyouji.com/topic.asp?id=1910 http://bbs.quyouji.com/topic.asp?id=1911 http://bbs.quyouji.com/topic.asp?id=1912 http://bbs.quyouji.com/topic.asp?id=1913 http://bbs.quyouji.com/topic.asp?id=1914 http://bbs.quyouji.com/topic.asp?id=1915 http://bbs.quyouji.com/topic.asp?id=1916 http://bbs.quyouji.com/topic.asp?id=1917 http://bbs.quyouji.com/topic.asp?id=1918 http://bbs.quyouji.com/topic.asp?id=1919 http://bbs.quyouji.com/topic.asp?id=1920 http://bbs.quyouji.com/topic.asp?id=1921 http://bbs.quyouji.com/topic.asp?id=1922 http://bbs.quyouji.com/topic.asp?id=1923 http://bbs.quyouji.com/topic.asp?id=1924 http://bbs.quyouji.com/topic.asp?id=1925 http://bbs.quyouji.com/topic.asp?id=1926 http://bbs.quyouji.com/topic.asp?id=1927 http://bbs.quyouji.com/topic.asp?id=1928 http://bbs.quyouji.com/topic.asp?id=1929 http://bbs.quyouji.com/topic.asp?id=1930 http://bbs.quyouji.com/topic.asp?id=1931 http://bbs.quyouji.com/topic.asp?id=1932 http://bbs.quyouji.com/topic.asp?id=1933 http://bbs.quyouji.com/topic.asp?id=1934 http://bbs.quyouji.com/topic.asp?id=1935 http://bbs.quyouji.com/topic.asp?id=1936 http://bbs.quyouji.com/topic.asp?id=1937 http://bbs.quyouji.com/topic.asp?id=1938 http://bbs.quyouji.com/topic.asp?id=1939 http://bbs.quyouji.com/topic.asp?id=1940 http://bbs.quyouji.com/topic.asp?id=1941 http://bbs.quyouji.com/topic.asp?id=1942 http://bbs.quyouji.com/topic.asp?id=1943 http://bbs.quyouji.com/topic.asp?id=1944 http://bbs.quyouji.com/topic.asp?id=1945 http://bbs.quyouji.com/topic.asp?id=1946 http://bbs.quyouji.com/topic.asp?id=1947 http://bbs.quyouji.com/topic.asp?id=1948 http://bbs.quyouji.com/topic.asp?id=1949 http://bbs.quyouji.com/topic.asp?id=1950 http://bbs.quyouji.com/topic.asp?id=1951 http://bbs.quyouji.com/topic.asp?id=1952 http://bbs.quyouji.com/topic.asp?id=1953 http://bbs.quyouji.com/topic.asp?id=1954 http://bbs.quyouji.com/topic.asp?id=1955 http://bbs.quyouji.com/topic.asp?id=1956 http://bbs.quyouji.com/topic.asp?id=1957 http://bbs.quyouji.com/topic.asp?id=1958 http://bbs.quyouji.com/topic.asp?id=1959 http://bbs.quyouji.com/topic.asp?id=1960 http://bbs.quyouji.com/topic.asp?id=1961 http://bbs.quyouji.com/topic.asp?id=1962 http://bbs.quyouji.com/topic.asp?id=1963 http://bbs.quyouji.com/topic.asp?id=1964 http://bbs.quyouji.com/topic.asp?id=1965 http://bbs.quyouji.com/topic.asp?id=1966 http://bbs.quyouji.com/topic.asp?id=1967 http://bbs.quyouji.com/topic.asp?id=1968 http://bbs.quyouji.com/topic.asp?id=1969 http://bbs.quyouji.com/topic.asp?id=1970 http://bbs.quyouji.com/topic.asp?id=1971 http://bbs.quyouji.com/topic.asp?id=1972 http://bbs.quyouji.com/topic.asp?id=1973 http://bbs.quyouji.com/topic.asp?id=1974 http://bbs.quyouji.com/topic.asp?id=1975 http://bbs.quyouji.com/topic.asp?id=1976 http://bbs.quyouji.com/topic.asp?id=1977 http://bbs.quyouji.com/topic.asp?id=1978 http://bbs.quyouji.com/topic.asp?id=1979 http://bbs.quyouji.com/topic.asp?id=1980 http://bbs.quyouji.com/topic.asp?id=1981 http://bbs.quyouji.com/topic.asp?id=1982 http://bbs.quyouji.com/topic.asp?id=1983 http://bbs.quyouji.com/topic.asp?id=1984 http://bbs.quyouji.com/topic.asp?id=1985 http://bbs.quyouji.com/topic.asp?id=1986 http://bbs.quyouji.com/topic.asp?id=1987 http://bbs.quyouji.com/topic.asp?id=1988 http://bbs.quyouji.com/topic.asp?id=1989 http://bbs.quyouji.com/topic.asp?id=1990 http://bbs.quyouji.com/topic.asp?id=1991 http://bbs.quyouji.com/topic.asp?id=1992 http://bbs.quyouji.com/topic.asp?id=1993 http://bbs.quyouji.com/topic.asp?id=1994 http://bbs.quyouji.com/topic.asp?id=1995 http://bbs.quyouji.com/topic.asp?id=1996 http://bbs.quyouji.com/topic.asp?id=1997 http://bbs.quyouji.com/topic.asp?id=1998 http://bbs.quyouji.com/topic.asp?id=1999 http://bbs.quyouji.com/topic.asp?id=2000 http://bbs.quyouji.com/topic.asp?id=2001 http://bbs.quyouji.com/topic.asp?id=2002 http://bbs.quyouji.com/topic.asp?id=2003 http://bbs.quyouji.com/topic.asp?id=2004 http://bbs.quyouji.com/topic.asp?id=2005 http://bbs.quyouji.com/topic.asp?id=2006 http://bbs.quyouji.com/topic.asp?id=2007 http://bbs.quyouji.com/topic.asp?id=2008 http://bbs.quyouji.com/topic.asp?id=2009 http://bbs.quyouji.com/topic.asp?id=2010 http://bbs.quyouji.com/topic.asp?id=2011 http://bbs.quyouji.com/topic.asp?id=2012 http://bbs.quyouji.com/topic.asp?id=2013 http://bbs.quyouji.com/topic.asp?id=2014 http://bbs.quyouji.com/topic.asp?id=2015 http://bbs.quyouji.com/topic.asp?id=2016 http://bbs.quyouji.com/topic.asp?id=2017 http://bbs.quyouji.com/topic.asp?id=2018 http://bbs.quyouji.com/topic.asp?id=2019 http://bbs.quyouji.com/topic.asp?id=2020 http://bbs.quyouji.com/topic.asp?id=2021 http://bbs.quyouji.com/topic.asp?id=2022 http://bbs.quyouji.com/topic.asp?id=2023 http://bbs.quyouji.com/topic.asp?id=2024 http://bbs.quyouji.com/topic.asp?id=2025 http://bbs.quyouji.com/topic.asp?id=2026 http://bbs.quyouji.com/topic.asp?id=2027 http://bbs.quyouji.com/topic.asp?id=2028 http://bbs.quyouji.com/topic.asp?id=2029 http://bbs.quyouji.com/topic.asp?id=2030 http://bbs.quyouji.com/topic.asp?id=2031 http://bbs.quyouji.com/topic.asp?id=2032 http://bbs.quyouji.com/topic.asp?id=2033 http://bbs.quyouji.com/topic.asp?id=2034 http://bbs.quyouji.com/topic.asp?id=2035 http://bbs.quyouji.com/topic.asp?id=2036 http://bbs.quyouji.com/topic.asp?id=2037 http://bbs.quyouji.com/topic.asp?id=2038 http://bbs.quyouji.com/topic.asp?id=2039 http://bbs.quyouji.com/topic.asp?id=2040 http://bbs.quyouji.com/topic.asp?id=2041 http://bbs.quyouji.com/topic.asp?id=2042 http://bbs.quyouji.com/topic.asp?id=2043 http://bbs.quyouji.com/topic.asp?id=2044 http://bbs.quyouji.com/topic.asp?id=2045 http://bbs.quyouji.com/topic.asp?id=2046 http://bbs.quyouji.com/topic.asp?id=2047 http://bbs.quyouji.com/topic.asp?id=2048 http://bbs.quyouji.com/topic.asp?id=2049 http://bbs.quyouji.com/topic.asp?id=2050 http://bbs.quyouji.com/topic.asp?id=2051 http://bbs.quyouji.com/topic.asp?id=2052 http://bbs.quyouji.com/topic.asp?id=2053 http://bbs.quyouji.com/topic.asp?id=2054 http://bbs.quyouji.com/topic.asp?id=2055 http://bbs.quyouji.com/topic.asp?id=2056 http://bbs.quyouji.com/topic.asp?id=2057 http://bbs.quyouji.com/topic.asp?id=2058 http://bbs.quyouji.com/topic.asp?id=2059 http://bbs.quyouji.com/topic.asp?id=2060 http://bbs.quyouji.com/topic.asp?id=2061 http://bbs.quyouji.com/topic.asp?id=2062 http://bbs.quyouji.com/topic.asp?id=2063 http://bbs.quyouji.com/topic.asp?id=2064 http://bbs.quyouji.com/topic.asp?id=2065 http://bbs.quyouji.com/topic.asp?id=2066 http://bbs.quyouji.com/topic.asp?id=2067 http://bbs.quyouji.com/topic.asp?id=2068 http://bbs.quyouji.com/topic.asp?id=2069 http://bbs.quyouji.com/topic.asp?id=2070 http://bbs.quyouji.com/topic.asp?id=2071 http://bbs.quyouji.com/topic.asp?id=2072 http://bbs.quyouji.com/topic.asp?id=2073 http://bbs.quyouji.com/topic.asp?id=2074 http://bbs.quyouji.com/topic.asp?id=2075 http://bbs.quyouji.com/topic.asp?id=2076 http://bbs.quyouji.com/topic.asp?id=2077 http://bbs.quyouji.com/topic.asp?id=2078 http://bbs.quyouji.com/topic.asp?id=2079 http://bbs.quyouji.com/topic.asp?id=2080 http://bbs.quyouji.com/topic.asp?id=2081 http://bbs.quyouji.com/topic.asp?id=2082 http://bbs.quyouji.com/topic.asp?id=2083 http://bbs.quyouji.com/topic.asp?id=2084 http://bbs.quyouji.com/topic.asp?id=2085 http://bbs.quyouji.com/topic.asp?id=2086 http://bbs.quyouji.com/topic.asp?id=2087 http://bbs.quyouji.com/topic.asp?id=2088 http://bbs.quyouji.com/topic.asp?id=2089 http://bbs.quyouji.com/topic.asp?id=2090 http://bbs.quyouji.com/topic.asp?id=2091 http://bbs.quyouji.com/topic.asp?id=2092 http://bbs.quyouji.com/topic.asp?id=2093 http://bbs.quyouji.com/topic.asp?id=2094 http://bbs.quyouji.com/topic.asp?id=2095 http://bbs.quyouji.com/topic.asp?id=2096 http://bbs.quyouji.com/topic.asp?id=2097 http://bbs.quyouji.com/topic.asp?id=2098 http://bbs.quyouji.com/topic.asp?id=2099 http://bbs.quyouji.com/topic.asp?id=2100 http://bbs.quyouji.com/topic.asp?id=2101 http://bbs.quyouji.com/topic.asp?id=2102 http://bbs.quyouji.com/topic.asp?id=2103 http://bbs.quyouji.com/topic.asp?id=2104 http://bbs.quyouji.com/topic.asp?id=2105 http://bbs.quyouji.com/topic.asp?id=2106 http://bbs.quyouji.com/topic.asp?id=2107 http://bbs.quyouji.com/topic.asp?id=2108 http://bbs.quyouji.com/topic.asp?id=2109 http://bbs.quyouji.com/topic.asp?id=2110 http://bbs.quyouji.com/topic.asp?id=2111 http://bbs.quyouji.com/topic.asp?id=2112 http://bbs.quyouji.com/topic.asp?id=2113三、WordCount总结： 第一个Stage有哪些RDD？HadoopRDD、MapPartitionsRDD、MapPartitionsRDD、MapPartitionsRDD、MapPartitionsRDD。
第二个Stage有哪些RDD？ShuffledRDD、MapPartitionsRDD。
