综合论文：Cecho_Ting_SqoopBigDataTechCon（英文）.pdfVIP

Mastering Sqoop for Data Transfer for Big Data Jarek Jarcec Cecho | Kathleen Ting 1 Who Are We? •  Jarek Jarcec Cecho •  Apache Sqoop Commier, PMC Member •  Soware Engineer, Cloudera •  jarcec@apache.org •  Kathleen Ting •  Apache Sqoop Commier, PMC Member •  Customer Operaons Engineering Manager, Cloudera •  kathleen@apache.org, @kate_ng 2 What is Sqoop? •  Apache Top-Level Project •  SQl to hadOOP •  Tool to transfer data from relaonal databases •  Teradata, MySQL, PostgreSQL, Oracle, Netezza •  To Hadoop ecosystem •  HDFS (text, sequence file), Hive, HBase, Avro •  And vice versa 3 Why Sqoop? •  Efficient/Controlled resource ulizaon •  Concurrent connecons, Time of operaon •  Datatype mapping and conversion •  Automac, and User override •  Metadata propagaon •  Sqoop Record •  Hive Metastore •  Avro 4 Sqoop 1 5 Sqoop 1 •  Based on Connectors •  Responsible for Metadata lookups, and Data Transfer •  Majority of connectors are JDBC based •  Non-JDBC (direct) connectors for opmized data transfer •  Connectors responsible for all supported funconality •  HBase Import, Avro Support, ... 6 Sqoop 1 Challenges •  Crypc, contextual command line arguments •  Security concerns •  Type mapping is not clearly defined •  Client needs access to Hadoop binaries/configuraon and database •  JDBC model is enforced 7 Sqoop 1 Challenges •  Non-uniform funconality •  Different connectors support different capabilies •  Overlapped/Duplicated funconality