絕了!阿裏這款開源神器,高級Java開發技術

隔壁的老郭 2021-09-18 07:12:22 阅读数:924

神器 java
CREATE USER canal IDENTIFIED BY 'canal';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;

  • 1.
  • 2.
  • 3.
  • 創建好測試用的數據庫canal-test,之後創建一張商品錶product,建錶語句如下。
CREATE TABLE `product` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`title` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
`sub_title` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
`price` decimal(10, 2) NULL DEFAULT NULL,
`pic` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL,
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 2 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.

canal-server使用

將我們下載好的壓縮包canal.deployer-1.1.5-SNAPSHOT.tar.gz上傳到Linux服務器,然後解壓到指定目錄/mydata/canal-server,可使用如下命令解壓;

tar -zxvf canal.deployer-1.1.5-SNAPSHOT.tar.gz

  • 1.

解壓完成後目錄結構如下;

├── bin
│?? ├── restart.sh
│?? ├── startup.bat
│?? ├── startup.sh
│?? └── stop.sh
├── conf
│?? ├── canal_local.properties
│?? ├── canal.properties
│?? └── example
│?? ?? └── instance.properties
├── lib
├── logs
│?? ├── canal
│?? │?? └── canal.log
│?? └── example
│?? ├── example.log
│?? └── example.log
└── plugin

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.

修改配置文件conf/example/instance.properties,按如下配置即可,主要是修改數據庫相關配置;

# 需要同步數據的MySQL地址
canal.instance.master.address=127.0.0.1:3306
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=
# 用於同步數據的數據庫賬號
canal.instance.dbUsername=canal
# 用於同步數據的數據庫密碼
canal.instance.dbPassword=canal
# 數據庫連接編碼
canal.instance.connectionCharset = UTF-8
# 需要訂閱binlog的錶過濾正則錶達式
canal.instance.filter.regex=.*\\..*

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.

使用startup.sh脚本啟動canal-server服務;

sh bin/startup.sh

  • 1.

啟動成功後可使用如下命令查看服務日志信息;

tail -f logs/canal/canal.log

  • 1.
2020-10-26 16:18:13.354 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[172.17.0.1(172.17.0.1):11111]
2020-10-26 16:18:19.978 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......

  • 1.
  • 2.

啟動成功後可使用如下命令查看instance日志信息;

tail -f logs/example/example.log

  • 1.
2020-10-26 16:18:16.056 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2020-10-26 16:18:16.061 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2020-10-26 16:18:18.259 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2020-10-26 16:18:18.282 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2020-10-26 16:18:18.282 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql\.slave_.*$
2020-10-26 16:18:19.543 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2020-10-26 16:18:19.578 [main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2020-10-26 16:18:19.912 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just last position
{"identity":{"slaveId":-1,"sourceAddress":{"address":"localhost","port":3306}},"postion":{"gtid":"","included":false,"journalName":"mall-mysql-bin.000006","position":2271,"serverId":101,"timestamp":1603682664000}}
2020-10-26 16:18:22.435 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mall-mysql-bin.000006,position=2271,serverId=101,gtid=,timestamp=1603682664000] cost : 2768ms , the next step is binlog dump

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.

如果想要停止canal-server服務可以使用如下命令。

sh bin/stop.sh

  • 1.

canal-adapter使用

將我們下載好的壓縮包canal.adapter-1.1.5-SNAPSHOT.tar.gz上傳到Linux服務器,然後解壓到指定目錄/mydata/canal-adpter,解壓完成後目錄結構如下;

├── bin
│?? ├── adapter.pid
│?? ├── restart.sh
│?? ├── startup.bat
│?? ├── startup.sh
│?? └── stop.sh
├── conf
│?? ├── application.yml
│?? ├── es6
│?? ├── es7
│?? │?? ├── biz_order.yml
│?? │?? ├── customer.yml
│?? │?? └── product.yml
│?? ├── hbase
│?? ├── kudu
│?? ├── logback.xml
│?? ├── META-INF
│?? │?? └── spring.factories
│?? └── rdb
├── lib
├── logs
│?? └── adapter
│?? └── adapter.log
└── plugin

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.

修改配置文件conf/application.yml,按如下配置即可,主要是修改canal-server配置、數據源配置和客戶端適配器配置;

canal.conf:
mode: tcp # 客戶端的模式,可選tcp kafka rocketMQ
flatMessage: true # 扁平message開關, 是否以json字符串形式投遞數據, 僅在kafka/rocketMQ模式下有效
zookeeperHosts: # 對應集群模式下的zk地址
syncBatchSize: 1000 # 每次同步的批數量
retries: 0 # 重試次數, -1為無限重試
timeout: # 同步超時時間, 單比特毫秒
accessKey:
secretKey:
consumerProperties:
# canal tcp consumer
canal.tcp.server.host: 127.0.0.1:11111 #設置canal-server的地址
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 500
canal.tcp.username:
canal.tcp.password:
srcDataSources: # 源數據庫配置
defaultDS:
url: jdbc:mysql://127.0.0.1:3306/canal_test?useUnicode=true
username: canal
password: canal
canalAdapters: # 適配器列錶
- instance: example # canal實例名或者MQ topic名
groups: # 分組列錶
- groupId: g1 # 分組id, 如果是MQ模式將用到該值
outerAdapters:
- name: logger # 日志打印適配器
- name: es7 # ES同步適配器
hosts: 127.0.0.1:9200 # ES連接地址
properties:
mode: rest # 模式可選transport(9300) 或者 rest(9200)
# security.auth: test:123456 # only used for rest mode
cluster.name: elasticsearch # ES集群名稱

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.

添加配置文件canal-adapter/conf/es7/product.yml,用於配置MySQL中的錶與Elasticsearch中索引的映射關系;

dataSourceKey: defaultDS # 源數據源的key, 對應上面配置的srcDataSources中的值
destination: example # canal的instance或者MQ的topic
groupId: g1 # 對應MQ模式下的groupId, 只會同步對應groupId的數據
esMapping:
_index: canal_product # es 的索引名稱
_id: _id # es 的_id, 如果不配置該項必須配置下面的pk項_id則會由es自動分配
sql: "SELECT
p.id AS _id,
p.title,
p.sub_title,
p.price,
p.pic
FROM
product p" # sql映射
etlCondition: "where a.c_time>={}" #etl的條件參數
commitBatch: 3000 # 提交批大小

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.

使用startup.sh脚本啟動canal-adapter服務;

sh bin/startup.sh

  • 1.

啟動成功後可使用如下命令查看服務日志信息;

tail -f logs/adapter/adapter.log

  • 1.
20-10-26 16:52:55.148 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed
2020-10-26 16:52:57.005 [main] INFO c.a.o.c.client.adapter.es.core.config.ESSyncConfigLoader - ## Start loading es mapping config ...
2020-10-26 16:52:57.376 [main] INFO c.a.o.c.client.adapter.es.core.config.ESSyncConfigLoader - ## ES mapping config loaded
2020-10-26 16:52:58.615 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es7 succeed
2020-10-26 16:52:58.651 [main] INFO c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: /mydata/canal-adapter/plugin
2020-10-26 16:52:59.043 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: example-g1 succeed
2020-10-26 16:52:59.044 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2020-10-26 16:52:59.057 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: example <=============
2020-10-26 16:52:59.100 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2020-10-26 16:52:59.153 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2020-10-26 16:52:59.590 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2020-10-26 16:52:59.626 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 31.278 seconds (JVM running for 33.99)
2020-10-26 16:52:59.930 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: example succeed <=============

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.

如果需要停止canal-adapter服務可以使用如下命令。

sh bin/stop.sh

  • 1.

數據同步演示

經過上面的一系列步驟,canal的數據同步功能已經基本可以使用了,下面我們來演示下數據同步功能。

首先我們需要在Elasticsearch中創建索引,和MySQL中的product錶相對應,直接在Kibana的Dev Tools中使用如下命令創建即可;

PUT canal_product
{
"mappings": {
"properties": {
"title": {
"type": "text"
},
"sub_title": {
"type": "text"
},
"pic": {
"type": "text"
},
"price": {
# 總結
無論是哪家公司,都很重視高並發高可用的技術,重視基礎,重視JVM。面試是一個雙向選擇的過程,不要抱著畏懼的心態去面試,不利於自己的發揮。同時看中的應該不止薪資,還要看你是不是真的喜歡這家公司,是不是能真的得到鍛煉。其實我寫了這麼多,只是我自己的總結,並不一定適用於所有人,相信經過一些面試,大家都會有這些感觸。
最後我整理了一些面試真題資料,技術知識點剖析教程,還有和廣大同仁一起交流學習共同進步,還有一些職業經驗的分享。
**[CodeChina開源項目:【一線大廠Java面試題解析+核心總結學習筆記+最新講解視頻】](https://ali1024.coding.net/public/P7/Java/git)**
![絕了!阿裏這款開源神器,高級Java開發技術_Java](https://s7.51cto.com/images/20210918/1631919907992414.jpg)

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
版权声明:本文为[隔壁的老郭]所创,转载请带上原文链接,感谢。 https://gsmany.com/2021/09/20210918071221748w.html