Flink 算子Operators总结

Operator作用流的转换map将一个元素转换成另外一个元素DataStream → DataStream本flapmap将几个的一个元素转换为零个，一个或者多个DataStream → DataStreamfilter保留集合中返回true的元素DataStream → DataStreamkeyBy对数据流进行逻辑分区，相同的key在同一分区DataStream → KeyedStreamreduce遍历集合，依次合并元素最终生产一个元素KeyedStream → DataStreamfold遍历结合从第一个元素到最后一个元素依次连接起来KeyedStream → DataStreamAggregationsemmmmKeyedStream → DataStreamWindow基于已经分区的stream，将元素划分窗口KeyedStream → WindowedStreamWindowAll基于未分区的stream，将所有元素集中到一个taskDataStream → AllWindowedStreamApply(Window)自定义函数处理窗口内所有的元素WindowedStream → DataStream AllWindowedStream → DataStreamWindow Reduce窗口内所有元素reduce到一个结果WindowedStream → DataStreamWindow Fold同stream的foldWindowedStream → DataStreamAggregations on windows同stream的AggregationsWindowedStream → DataStreamUnion将两个流合并DataStream* → DataStreamWindow Join两个流join成一个流，指定分区key，在指定window，窗口是必须的DataStream,DataStream → DataStreamInterval Join流2 join 流1中一段时间的元素KeyedStream,KeyedStream → DataStreamWindow CoGroup双流join，指定窗口DataStream,DataStream → DataStreamConnect联合两个流，保留各种stateDataStream,DataStream → ConnectedStreamsCoMap, CoFlatMap同map, CoFlatMapConnectedStreams → DataStreamSplit流拆分DataStream → SplitStreamSelect从SplitStream分离出DataStreamSplitStream → DataStreamIterate-DataStream → IterativeStream → DataStream---Extract Timestamps设置event timeDataStream → DataStream map 将每个元素乘以2

 

DataStream<Integer> dataStream = //... dataStream.map(new MapFunction<Integer, Integer>() { @Override public Integer map(Integer value) throws Exception { return 2 * value; } }); flatMap 单词分隔

 

dataStream.flatMap(new FlatMapFunction<String, String>() { @Override public void flatMap(String value, Collector<String> out) throws Exception { for(String word: value.split(" ")){ out.collect(word); } } }); filter 保留value=0的元素

 

dataStream.filter(new FilterFunction<Integer>() { @Override public boolean filter(Integer value) throws Exception { return value != 0; } }); keyby

 

dataStream.keyBy("someKey") // Key by field "someKey" dataStream.keyBy(0) // Key by the first element of a Tuple reduce 求和

 

keyedStream.reduce(new ReduceFunction<Integer>() { @Override public Integer reduce(Integer value1, Integer value2) throws Exception { return value1 + value2; } }); fold A fold function that, when applied on the sequence (1,2,3,4,5), emits the sequence "start-1", "start-1-2", "start-1-2-3", ..

 

DataStream<String> result = keyedStream.fold("start", new FoldFunction<Integer, String>() { @Override public String fold(String current, Integer value) { return current + "-" + value; } }); Aggregations

 

keyedStream.sum(0); keyedStream.sum("key"); keyedStream.min(0); keyedStream.min("key"); keyedStream.max(0); keyedStream.max("key"); keyedStream.minBy(0); keyedStream.minBy("key"); keyedStream.maxBy(0); keyedStream.maxBy("key"); Window Join

 

dataStream.join(otherStream) .where(<key selector>).equalTo(<key selector>) .window(TumblingEventTimeWindows.of(Time.seconds(3))) .apply (new JoinFunction () {...}); Interval Join

 

// this will join the two streams so that // key1 == key2 && leftTs - 2 < rightTs < leftTs + 2 keyedStream.intervalJoin(otherKeyedStream) .between(Time.milliseconds(-2), Time.milliseconds(2)) // lower and upper bound .upperBoundExclusive(true) // optional .lowerBoundExclusive(true) // optional .process(new IntervalJoinFunction() {...}); Split

 

SplitStream<Integer> split = someDataStream.split(new OutputSelector<Integer>() { @Override public Iterable<String> select(Integer value) { List<String> output = new ArrayList<String>(); if (value % 2 == 0) { output.add("even"); } else { output.add("odd"); } return output; } });