1 前言
之前我們用兩篇文章講解了Spring Cloud Data Flow,例子都是用UI操作的,但我們在Linux系統上經常是無法提供介面來操作,整合在Jenkins上也無法使用UI。好在官方提供了Data Flow Shell工具,可以在命令列模式下進行操作,非常方便。
相關文章可參考:
Spring Cloud Data Flow Server提供了可操作的REST API,所以這個Shell工具的本質還是通過呼叫REST API來互動的。
2 常用操作2.1 啟動首先要確保我們已經安裝有Java環境和下載了可執行的jar包:spring-cloud-dataflow-shell-2.5.3.RELEASE.jar
然後啟動如下:
預設是連線了http://localhost:9393的Server,可以通過--dataflow.uri=地址來指定。如果需要認證資訊,需要加上--dataflow.username=使用者 --dataflow.password=密碼。
比如我們連線之前安裝在Kubernetes上的Server如下:
$ java -jar spring-cloud-dataflow-shell-2.5.3.RELEASE.jar --dataflow.uri=http://localhost:30093
2.2 Application操作
介紹一下Application相關操作:
列出所有目前註冊的app:
dataflow:>app list╔═══╤══════╤═════════╤════╤════════════════════╗║app│source│processor│sink│ task ║╠═══╪══════╪═════════╪════╪════════════════════╣║ │ │ │ │composed-task-runner║║ │ │ │ │timestamp-batch ║║ │ │ │ │timestamp ║╚═══╧══════╧═════════╧════╧════════════════════╝
檢視某個app的資訊:
dataflow:>app info --type task timestamp
清除app註冊資訊:
dataflow:>app unregister --type task timestampSuccessfully unregistered application 'timestamp' with type 'task'.
清除所有app註冊資訊:
dataflow:>app all unregisterSuccessfully unregistered applications.dataflow:>app list No registered apps.You can register new apps with the 'app register' and 'app import' commands.
註冊一個app:
dataflow:>app register --name timestamp-pkslow --type task --uri docker:springcloudtask/timestamp-task:2.1.1.RELEASESuccessfully registered application 'task:timestamp-pkslow'dataflow:>app list╔═══╤══════╤═════════╤════╤════════════════╗║app│source│processor│sink│ task ║╠═══╪══════╪═════════╪════╪════════════════╣║ │ │ │ │timestamp-pkslow║╚═══╧══════╧═════════╧════╧════════════════╝
批量匯入app,可以從一個URL或一個properties檔案匯入:
dataflow:>app import https://dataflow.spring.io/task-docker-latestSuccessfully registered 3 applications from [task.composed-task-runner, task.timestamp.metadata, task.composed-task-runner.metadata, task.timestamp-batch.metadata, task.timestamp-batch, task.timestamp]
需要注意的是,在註冊或匯入app時,如果重複的話,預設是無法匯入的,不會覆蓋。如果想要覆蓋,可以加引數--force。
dataflow:>app register --name timestamp-pkslow --type task --uri docker:springcloudtask/timestamp-task:2.1.1.RELEASECommand failed org.springframework.cloud.dataflow.rest.client.DataFlowClientException: The 'task:timestamp-pkslow' application is already registered as docker:springcloudtask/timestamp-task:2.1.1.RELEASEThe 'task:timestamp-pkslow' application is already registered as docker:springcloudtask/timestamp-task:2.1.1.RELEASEdataflow:>app register --name timestamp-pkslow --type task --uri docker:springcloudtask/timestamp-task:2.1.1.RELEASE --forceSuccessfully registered application 'task:timestamp-pkslow'
2.3 Task操作列出task:
dataflow:>task list╔════════════════╤════════════════════════════════╤═══════════╤═══════════╗║ Task Name │ Task Definition │description│Task Status║╠════════════════╪════════════════════════════════╪═══════════╪═══════════╣║timestamp-pkslow│timestamp │ │COMPLETE ║║timestamp-two │<t1: timestamp || t2: timestamp>│ │ERROR ║║timestamp-two-t1│timestamp │ │COMPLETE ║║timestamp-two-t2│timestamp │ │COMPLETE ║╚════════════════╧════════════════════════════════╧═══════════╧═══════════╝
刪除一個task,這裡我們刪除的是一個組合task,所以會把子task也一併刪除了:
dataflow:>task destroy timestamp-twoDestroyed task 'timestamp-two'
刪除所有task,會有風險提示:
dataflow:>task all destroy Really destroy all tasks? [y, n]: yAll tasks destroyeddataflow:>task list╔═════════╤═══════════════╤═══════════╤═══════════╗║Task Name│Task Definition│description│Task Status║╚═════════╧═══════════════╧═══════════╧═══════════╝
建立一個task:
dataflow:>task create timestamp-pkslow-t1 --definition "timestamp --format=\\"yyyy\\"" --description "pkslow timestamp task"Created new task 'timestamp-pkslow-t1'
啟動一個task並檢視狀態,啟動時需要記錄執行ID,然後通過執行ID來查詢狀態:
dataflow:>task launch timestamp-pkslow-t1Launched task 'timestamp-pkslow-t1' with execution id 8dataflow:>task execution status 8
檢視所有task執行並檢視執行日誌:
dataflow:>task execution list dataflow:>task execution log 8 . ____ _ __ _ _ /\\\\ / ___'_ __ _ _(_)_ __ __ _ \\ \\ \\ \\( ( )\\___ | '_ | '_| | '_ \\/ _` | \\ \\ \\ \\ \\\\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.1.13.RELEASE)2020-08-01 17:20:51.626 INFO 1 --- [ Thread-5] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...2020-08-01 17:20:51.633 INFO 1 --- [ Thread-5] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
2.4 Http請求可以進行http請求:
dataflow:>http get https://www.pkslow.comdataflow:>http post --target https://www.pkslow.com --data "data"> POST (text/plain) https://www.pkslow.com data> 405 METHOD_NOT_ALLOWEDError sending data 'data' to 'https://www.pkslow.com'
2.5 讀取並執行檔案
先準備一個指令碼檔案,用來放Data Flow Shell命令,檔名為pkslow.shell,內容如下:
versiondateapp list
執行與結果如下:
dataflow:>script pkslow.shellversion2.5.3.RELEASEdateSunday, August 2, 2020 1:59:34 AM CSTapp list╔═══╤══════╤═════════╤════╤════════════════════╗║app│source│processor│sink│ task ║╠═══╪══════╪═════════╪════╪════════════════════╣║ │ │ │ │timestamp-pkslow ║║ │ │ │ │composed-task-runner║║ │ │ │ │timestamp-batch ║║ │ │ │ │timestamp ║╚═══╧══════╧═════════╧════╧════════════════════╝Script required 0.045 seconds to executedataflow:>
但其實我們在CI/CD的pipeline中,並不想先啟動一個shell命令列,然後再執行一個指令碼。我們想一步到位,直接執行,執行完畢後退出shell命令列。這也是有辦法的,可以在啟動的時候通過--spring.shell.commandFile指定檔案,如果有多個檔案則用逗號,分隔。如下所示:
$ java -jar spring-cloud-dataflow-shell-2.5.3.RELEASE.jar --dataflow.uri=http://localhost:30093 --spring.shell.commandFile=pkslow.shellSuccessfully targeted http://localhost:300932020-08-02T02:03:49+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - 2.5.3.RELEASE2020-08-02T02:03:49+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - Sunday, August 2, 2020 2:03:49 AM CST2020-08-02T02:03:49+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:309 - ╔═══╤══════╤═════════╤════╤════════════════════╗║app│source│processor│sink│ task ║╠═══╪══════╪═════════╪════╪════════════════════╣║ │ │ │ │timestamp-pkslow ║║ │ │ │ │composed-task-runner║║ │ │ │ │timestamp-batch ║║ │ │ │ │timestamp ║╚═══╧══════╧═════════╧════╧════════════════════╝$
執行完畢後,不會在shell命令列模式裡,而是退回linux的終端。這正是我們所需要的。
我們來準備一個註冊應用——建立任務——執行任務的指令碼試試:
versiondateapp register --name pkslow-app-1 --type task --uri docker:springcloudtask/timestamp-task:2.1.1.RELEASEtask create pkslow-task-1 --definition "pkslow-app-1"task launch pkslow-task-1
執行與結果如下:
$ java -jar spring-cloud-dataflow-shell-2.5.3.RELEASE.jar --dataflow.uri=http://localhost:30093 --spring.shell.commandFile=pkslow.shellSuccessfully targeted http://localhost:300932020-08-02T02:06:41+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - 2.5.3.RELEASE2020-08-02T02:06:41+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - Sunday, August 2, 2020 2:06:41 AM CST2020-08-02T02:06:41+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - Successfully registered application 'task:pkslow-app-1'2020-08-02T02:06:42+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - Created new task 'pkslow-task-1'2020-08-02T02:06:51+0800 INFO main o.s.c.d.s.DataflowJLineShellComponent:311 - Launched task 'pkslow-task-1' with execution id 9
這樣,我們就可以實現自動化打包與部署運行了。
3 一些使用技巧強大的shell工具提供了許多命令,其實不用一一記住,可以通過help命令檢視所有命令:
dataflow:>help
如果只對特定的一類命令感興趣,可以通過help xxx的方式獲取幫助:
dataflow:>help version* version - Displays shell versiondataflow:>help app* app all unregister - Unregister all applications* app default - Change the default application version* app import - Register all applications listed in a properties file* app info - Get information about an application* app list - List all registered applications* app register - Register a new application* app unregister - Unregister an application
shell還支援tab鍵補全命令。
4 總結本文的命令比較多,不想造成冗長,部分執行結果就不貼出來了。
多讀書,多分享;多寫作,多整理。