Sunday, July 30, 2017

Sqoop : Export


scoop export --connect jdbc:mysql://localhost/db --username root 
--table employee --export-dir /emp/emp_data  

  • Table must already exist in db
  • Input files are read and parsed as per user provided inputs
  • Default is to generate insert statement , can use update  mode to generate update statements.
  • If sqoop attempt to insert violates PK constraint , the export will fail ?? whole or partial ??
  • Inserts are performed by multiple sqoop threads , each thread uses a seperate connection & transaction
  • Every 100 statement sqoop commits transaction ( so if 1 insert fail in 1 thread ... rest of them will commit ??)
  • So export is not an atomic operation , some commits will be visible before others.

scoop export --connect jdbc:mysql://localhost/DBNAME -username root 
-password root --export-dir /input/abc --table test 
--fields-terminated-by "," --columns "id,name,age"  

can use -update-key as well

scoop export--create --connect jdbc:mysql://localhost/DBNAME -username root
-password root --export-dir /input/abc --table test --fields-terminated-by "," 
--columns "id,name,age" --update-key id

No comments: