Real-time real-time online data analysis

Data mining (user behavior analysis, interest division, regional display)

Data Aggregation (Daily/Weekly user clicks, click ranking)

Non-real-time analysis (log analysis, statistical analysis)

ConsumerMainly for data consumption

ConsumerNeed to specify when consuming dataTopic

multipleConsumerconfigure the samegroup.idwhen theseConsumerConsumed data will not be duplicated

newConsumerAPIConsumption data does not need to be specifiedZooKeeperaddress

available fromHDFSdirectly into thehivesurface

available fromHiveServerThe local hard disk of the node is directly loaded into thehivesurface

It can be directly loaded from the local hard disk of the node where the client is located.hivesurface

HiveThe data loading process will not parse the specific file content, mainly the process of file transfer

mergemapThe result of the stage has the samekeyvalue key-value pair

put allmapThe results of the stages are merged to get the final result

rightmapThe results of the stages are sorted and merged

Split the input data into individualsplit, and willsplitfurther split into < key,value > yes, then asmapletter

input of numbers

SupervisorAfter the process exits, it can beNimbusAutomatically identify and pull up without affecting the running business

WorkerCan be used after abnormal exitsupervisorsPulls up automatically without manual intervention

When a node fails, the tasks on the node will be reassigned to other normal nodes without manual intervention

NimbusAfter the failure, the backupNimbusIt will take over automatically without affecting the running business

Flume

Loader

HBase

MapReduce

Both are available externallySQLor classSQLability

Both can run onMapReduceon the computational framework

Both are available via table fields andHBaseAccess by field mappingHBasedata on

Both can run onSparkon the computational framework

Client

ApplicationMaster

Executor

Driver

True

False

broadcast variable in everyTasksave a copy

broadcast variable in eachExecutorsave a copy

Broadcast variables are read-only

broadcast variable viaSparkContext.broadcast() method to generate

hive.exec.reducers.bytes.per.reducer

hive.exec-ducers.max

hive.exec.max.dynamic.partitions

hive.exec.max.dynamic.partitions.pernode

lines.flatMap(_.split??)).map(word= > (word,1)).reduce(_+_).print ( )

lines.flatMap(_.split??)).map(word= > (word,word.length())).reduceByKey(_+_).print ( )

lines.flatMap(_.split??)).map(word= > (word,1)).reduceByKey(_+_).print ( )

lines.flatMap(_.split??)).flatmap(word= > (word,1)).groupByKey(_+_).print ( )

True

False

hdfs fsck /test.txt–files

hdfs fsck /text.txt–locations

hdfs fsck /test.txt–blocks

hdfs fsck /test.txt–list–corruptfileblocks

True

False