大數(shù)據(jù)集群間數(shù)據(jù)由hbase到impala

2024-07-21 02:51:47

字體：大中小

供稿：網(wǎng)友

10.0.0.11創(chuàng)建測(cè)試表create 'LV_TEST',{NAME =>'cf' ,BLOOMFILTER => 'ROW',COMPRESSION => 'SNAPPY',VERSIONS => 1}插入測(cè)試數(shù)據(jù)put 'LV_TEST','rowkey001','cf:name','zhangsan'put 'LV_TEST','rowkey001','cf:age','30'get 'LV_TEST','rowkey001', {COLUMN=>'cf:name'}get 'LV_TEST','rowkey001'put 'LV_TEST','rowkey002','cf:name','lisi'put 'LV_TEST','rowkey002','cf:age','40'get 'LV_TEST','rowkey002'查看測(cè)試表scan 'LV_TEST'導(dǎo)出到HDFSfind ./ /opt -name hbasecd /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/binhbase org.apache.hadoop.hbase.mapreduce.Export LV_TEST /data/LV_TESThadoop fs -ls /data/LV_TESThadoop fs -get /data/LV_TEST然后通過(guò)FTP下載到本地上傳到另一服務(wù)器192.168.51.251上傳到HDFSsu - hdfshadoop fs -mkdir /datahadoop fs -put /LV_TEST /data/LV_TESThadoop fs -ls /data/LV_TEST創(chuàng)建Hbase表create 'LV_TEST_COPY',{NAME =>'cf' ,BLOOMFILTER => 'ROW',COMPRESSION => 'SNAPPY',VERSIONS => 1}scan 'LV_TEST_COPY'hbase org.apache.hadoop.hbase.mapreduce.Import LV_TEST_COPY /data/LV_TEST建立hive外表hiveCREATE EXTERNAL TABLE LV_TEST_COPY(id string,name string,age string)ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key, cf:name, cf:age")TBLPROPERTIES("hbase.table.name" = "LV_TEST_COPY");impalaimpala-shell -i node90同步元數(shù)據(jù)INVALIDATE METADATA;SQL查詢結(jié)果

select * from LV_TEST_COPY;

最后將外部表數(shù)據(jù)復(fù)制到基于parques的impala內(nèi)部表，完成數(shù)據(jù)轉(zhuǎn)換。

insert into ***() select * from *****

上一篇：利用 ELK 處理 Percona 審計(jì)日志

下一篇：Redis的主從復(fù)制